Re-reading Kahneman’s Thinking, Fast and Slow

A bit over four years ago I wrote a glowing review of Daniel Kahneman’s Thinking, Fast and Slow. I described it as a “magnificent book” and “one of the best books I have read”. I praised the way Kahneman threaded his story around the System 1 / System 2 dichotomy, and the coherence provided  by prospect theory.

What a difference four years makes. I will still describe Thinking, Fast and Slow as an excellent book - possibly the best behavioural science book available. But during that time a combination of my learning path and additional research in the behavioural sciences has led me to see Thinking, Fast and Slow as a book with many flaws.

First, there is the list of studies that simply haven’t held up through the “replication crisis” of the last few years. The first substantive chapter of Thinking, Fast and Slow is on priming, so many of these studies are right up the front. These include the Florida effect, money priming, the idea that making a test harder to read can increase test results, and ego depletion (I touch on each of these in my recent talk at the Sydney Behavioural Economics and Behavioural Science Meetup).

It’s understandable that Kahneman was somewhat caught out by the replication crisis that has enveloped this literature. But what does not sit so well was the confidence with which Kahneman made his claims. For example, he wrote:

When I describe priming studies to audiences, the reaction is often disbelief . . . The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.

I am surprised at the blind spot I had when first reading it - Kahneman’s overconfidence didn’t register with me.

As I was also, Kahneman is a fan of the hot hand studies. Someone who believes in the hot hand believes that a sportsperson such as a basketball player is more likely to make a shot if they made their previous one. Kahneman wrote:

The hot hand is entirely in the eye of the beholders, who are consistently too quick to perceive order and causality in randomness. The hot hand is a massive and widespread cognitive illusion. [Could the same be said about much of the priming literature?]

The public reaction to this research is part of the story. The finding was picked up by the press because of its surprising conclusion, and the general response was disbelief. When the celebrated coach of the Boston Celtics, Red Auerbach, heard of Gilovich and his study, he responded, “Who is this guy? So he makes a study. I couldn’t care less.” The tendency to see patterns in randomness is overwhelming - certainly more impressive than a guy making a study.

And now it seems there is a hot hand. The finding that there was no hot hand the consequence of a statistical error (also covered in my recent talk). The disbelief was appropriate, and Auerbach did himself a favour by ignoring the study.

As I’ve picked on Dan Ariely for the way he talks about organ donation rates, here’s Kahneman on that same point:

A directive about organ donation in case of accidental death is noted on an individual’s driver licence in many countries. The formulation of that directive is another case in which one frame is clearly superior to the other. Few people would argue that the decision of whether or not to donate one’s organs is unimportant, but there is strong evidence that most people make their choice thoughtlessly. The evidence comes from a comparison of organ donation rates in European countries, which reveals startling differences between neighbouring and culturally similar countries. An article published in 2003 noted that the organ donation rate was closer to 100% in Austria but only 12% in Germany, 86% in Sweden but only 4% in Denmark.

These enormous differences are a framing effect, which is caused by the format of the critical question. The high-donation countries have an opt-out form, where individuals who wish not to donate must check an appropriate box. Unless they take this simple action, they are considered willing donors. The low-contribution countries have an opt-in form: you must check a box to become a donor. That is all. The best single predictor of whether or not people will donate their organs is the designation of the default option that will be adopted without having to check a box. …

When the role of formulation is acknowledged, a policy question arises: Which formulation should be adopted. In this case, the answer is straightforward. If you believe that a large supply of donated organs is good for society, you will not be neutral between a formulation that yields almost 100% donations and another formulation that elicits donations from 4% of drivers.

As Ariely does, Kahneman describes the difference between European countries as being due to differences in form design, when in fact those European countries with high “donor rates” never ask their citizens whether they wish to be donors. The form described does not exist in the high-donation countries. They are simply presumed to consent to donation. (The paper that these numbers come from, Do Defaults Save Lives?, might have been better titled “Does not asking if you can take people’s organs save lives?”. That could have saved some confusion.)

Further, Kahneman talks about the gap between 100% and 4% as donation rates, when these numbers refer to those who are presumed to consent in the high-donation countries. Actual donation rates and the gap between the different types of countries are much lower.

All the above points are minor in themselves. But together the shaky science, overconfidence and lazy storytelling add up to something substantial.

What I also find less satisfying now is the attempt to construct a framework around the disparate findings in behavioural science. I once saw prospect theory as a great framework for thinking about many of the findings, but it is as unrealistic a decision making model as that for the perfectly rational man - the maths involved is even more complicated. It’s might be a useful descriptive or predictive model (if you could work out what the reference point actually is) but no one makes decisions in that way. (One day I will write a post on this.)

It will be interesting to see how Thinking, Fast and Slow stands up after another five years.