Not the jam study again

Go to any behavioural science conference, event or presentation, and there is a high probability you will hear about “the jam study”. Last week’s excellent MSiX was no exception, with at least three references I can recall. The story is wonderfully simple and I have, at times, been mildly sympathetic to the idea. However, it is time for this story to be retired or heavily qualified with the research that has occurred in the intervening years.

As a start, what is the jam study? In 2000, Mark Lepper and Sheena Iyengar published their findings (ungated pdf) about the response of consumers to displays of jam in an upmarket grocery store. Their paper also contained similar experiments involving choice of chocolate and essay questions, but those experiments have not gained the same reputation.

On two Saturdays, they set up tasting displays of either six or 24 jars of jam. Consumers could taste as many jams as they wished, and if they approached the tasting table, also received a $1 discount coupon to buy the jam. For attracting initial interest, the large display of 24 jams did a better job, with 60 per cent of people who passed the display stopping. Forty per cent stopped at the six jam display. But only three per cent of those who stopped at the 24 jam display purchased any of the jam, compared with almost 30 per cent who stopped at the six jam display.

This result has been one of the centrepieces of the argument that more choice is not necessarily good. The larger display seemed to reduce consumer motivation to buy the product. The theories around this concept and the associated idea that more choice does not make us happy are often labelled the choice overload hypothesis or the paradox of choice.

Fast-forward 10 years to another paper, this one by Benjamin Scheibehenne, Rainer Greifeneder and Peter Todd. They surveyed the literature on the choice overload hypothesis - there is plenty. And across the basket of studies, evidence of choice overload does not emerge so clearly. In some cases, choice increases purchases. In others it reduces them. Scheibehenne and friends determined that the mean effect size of changing the number of choices across the studies was effectively zero.

More pointedly, the reviewed studies included a few attempts to replicate the jam study results. An experiment using jam in an upscale German supermarket found no effect. Other experiments found no effect of choice size using chocolates or jelly beans. There were small differences in study design between these and the original jam study (as original authors are always quick to point out when replications fail), but if studies are so sensitive to study design and hard to replicate, it seems foolhardy to extrapolate the results of the original study too far.

That is not to say that there is not something interesting going on here. Scheibehenne and friends suggest that there may be a set of restrictive conditions under which choice overload occurs. These conditions might involve the complexity (and not the size) of the choice, the lack of dominant alternatives, assortment of options, time pressure or the distribution of product quality. These considerations are not issues of the size of the choice itself but the way the choice is undertaken. And since the jam study appears tough to replicate, these conditions might be particularly narrow. Still, the common refrain of making it easy for customers - as recommended for dealing with choice overload issues - holds for most of them. But they suggest different and more subtle solutions than simply reducing choice.

There are a lot of interesting studies floating around on choice overload - from decisions about turning off life-support (ungated pdf) to retirement savings (ungated pdf) - and the message is not always the same. But reading through them makes it clear that the jam study is just the tip of an iceberg and not necessarily representative of what lies beneath.

Finally, when Tim Harford wrote about these studies several years ago, he pointed out another often neglected argument about the importance of choice. It is only because we have choice that we are offered any good products at all, with companies incentivised to compete for us as customers. Even if choice has negative consequences, a world without choice might be worse.