{"id":469,"date":"2013-09-11T09:46:13","date_gmt":"2013-09-11T09:46:13","guid":{"rendered":"http:\/\/anna.ps\/blog\/?p=469"},"modified":"2017-03-16T16:28:36","modified_gmt":"2017-03-16T16:28:36","slug":"statistics-find-nicest-food-at-waitrose","status":"publish","type":"post","link":"https:\/\/anna.ps\/blog\/statistics-find-nicest-food-at-waitrose","title":{"rendered":"Using statistics to find the nicest (and nastiest) food at Waitrose"},"content":{"rendered":"

Using the Wilson score interval to identify the most delicious, and disgusting, foods at Britain’s best online supermarket. Jump to the results<\/a>.<\/em><\/p>\n

The hardest bit of cooking, for me, has always been choosing what to cook<\/em>. Sure, it’s fine if you only make dinner once a week – then you can flick through a designer cookbook and pick the prettiest picture. <\/p>\n

But actual cooking, every day, without it taking up all your time – that’s tougher. You need food that is tasty, healthy, and affordable. Finding this is hard, so it’s easy to end up cooking the same things again and again. <\/p>\n

And online shopping makes meal-planning even less inspiring – you can’t smell a tomato to see if it’s ripe. That’s why I was so excited when I realised I could use the power of statistics to find the overall most delicious – and the most disgusting – things you can buy at Ocado<\/a>. <\/p>\n

(For non-UK readers, Ocado is an online supermarket chain. It mostly sells food from Waitrose, the best British supermarket. And I have no affiliation with either company.)<\/p>\n

How not to sort by average rating<\/strong><\/p>\n

I was mulling the above recently when I came across Evan Miller’s How Not To Sort By Average Rating<\/a>. It’s a great article, and I realised I could use it to make my own shopping easier. <\/p>\n

Ocado have great reviews on their site, with rich comments and star ratings, but they commit the second type of sin mentioned by Evan – they rank their groceries by mean rating. This means that you can’t reliably tell which groceries are actually the most popular. <\/p>\n

Let me show you why this matters. Say I want soup for Friday lunch. I will find the “soup” category on the Ocado site, and then sort by customer rating. These are the top results: <\/p>\n

<\/div>\n

As noted above, Ocado ranks by the average – mean<\/em> – star rating for each product. This throws up some weird anomalies. <\/p>\n

For example, in third place is Swedish blueberry soup, with just two reviews. Both those reviews are five-star, so it has a mean rating of five stars. Swedish blueberry soup may well be delicious, but with only two reviews, I’m unwilling to take a chance. <\/p>\n

Much further down, in 10th place, is some gazpacho, with 48 ratings, of which 47 are positive and 46 are five-star. That means it has a slightly lower mean ranking, of 4.91 stars, so it comes further down the list. But 46 people loved it enough both to review it and give it five stars. <\/p>\n

<\/div>\n

I want to try that gazpacho! But to work out it was popular, I had to click every soup to check the number of reviews. I can’t do that with every single thing on my shopping list. <\/p>\n

Estimating true popularity with the Wilson interval<\/strong><\/p>\n

So how can we unleash the full potential of Ocado’s reviews? Evan’s article explains how to trade off a high average rating against the overall number of reviews. We can calculate a confidence interval<\/em> for each soup’s true popularity. <\/p>\n

Here is Wilson’s interval in full: <\/p>\n

<\/a><\/div>\n

The maths looks complicated, but the premise is this (as articulated by a Hacker News comment<\/a>): “If we rounded up the entire population and forced every single person to carefully review this item and issue a rating, what’s our best guess as to the percentage of people who would rate it positively?”<\/strong><\/p>\n

The clever thing about the Wilson interval is that it looks at the number of ratings as well as the value of the reviews. If few people have reviewed a product, our confidence interval is wide. As more people review it, the confidence interval narrows – because we’re more confident about how good or bad it really is. <\/p>\n

So, we can now rank all the foods listed on the Ocado site. I wrote a Python script<\/a> to scrape them all. For each item, I recorded the total number of reviews, and the proportion of reviewers who would recommend the product. <\/p>\n

Then I wrote more Python code (based on Evan’s Ruby example<\/a>) to calculate the Wilson score interval for each product. Here is the full script<\/a> – you are welcome to use it for your own projects. <\/p>\n

The results: sugar and convenience good…<\/strong><\/span><\/p>\n

Without more ado, here are the definitive results: the most popular of the 18,229 foods that you can buy at Ocado, ranked by the lower bound of the 95% Wilson score interval. <\/p>\n