Failure to replicate: ego depletion edition
Ego depletion is the idea that we have a limited supply of willpower. As we use it through the day, we become depleted and more likely to experience a willpower failure.
There is a mountain of published experiments providing evidence of ego depletion. Meta-analyses of the studies have supported the concept. The typical trick in these experiments is to get someone to engage in an ego depleting task - such as resisting chocolate - and then you watch them cave in more quickly on a later task than those who haven’t been subject to the earlier ego depletion.
But now the evidence is looking shaky. A pre-registered replication involving 23 labs and over 2,000 subjects will be published in Psychological Science. A smaller scale attempt to replicate was also published in PLOS One. The result? If there is any effect of ego depletion, it is close to zero.
Daniel Engber at Slate has the full story. One of the interesting points is how the meta-analysis didn’t show any problems:
To figure out what went wrong, Carter reviewed the 2010 meta-analysis—the study using data from 83 studies and 198 experiments. The closer he looked at the paper, though, the less he believed in its conclusions. First, the meta-analysis included only published studies, which meant the data would be subject to a standard bias in favor of positive results. Second, it included studies with contradictory or counterintuitive measures of self-control. One study, for example, suggested that depleted subjects would give more money to charity while another said depleted subjects would spend less time helping a stranger. When he and his adviser, Michael McCullough, reanalyzed the 2010 paper’s data using state-of-the-art analytic methods, they found no effect. For a second paper published last year, Carter and McCullough completed a second meta-analysis that included different studies, including 48 experiments that had never been published. Again, they found “very little evidence” of a real effect.
Roy Baumeister, one of the founders on this work on ego depletion, provided a response to Slate. It’s typical of many responses to this growing replication ‘crisis’ in psychology - suggest that those replicating the experiments haven’t captured all the experimental nuances, or that the effect is context specific.
In his lab, Baumeister told me, the letter e task [the task used in the replication] would have been handled differently. First, he’d train his subjects to pick out all the words containing e, until that became an ingrained habit. Only then would he add the second rule, about ignoring words with e’s and nearby vowels. That version of the task requires much more self-control, he says.
Second, he’d have his subjects do the task with pen and paper, instead of on a computer. It might take more self-control, he suggested, to withhold a gross movement of the arm than to stifle a tap of the finger on a keyboard.
If the replication showed us anything, Baumeister says, it’s that the field has gotten hung up on computer-based investigations. “In the olden days there was a craft to running an experiment. You worked with people, and got them into the right psychological state and then measured the consequences. There’s a wish now to have everything be automated so it can be done quickly and easily online.” These days, he continues, there’s less and less actual behavior in the science of behavior. “It’s just sitting at a computer and doing readings.”
Engber nicely points out the consequence of this line of defence. The big idea - and you only need to read Willpower to see that Baumeister and friends sell ego depletion as a big idea - loses its power:
One of the idea’s major selling points is its flexibility: Ego depletion applied not just to experiments involving chocolate chip cookies and radishes, but to those involving word games, conversations between white people and black people, decisions on whether to purchase soap, and even the behavior of dogs. In fact, the incredible range of the effect has often been cited in its favor. How could so many studies, performed in so many different ways, have all been wrong?
Yet now we know that ego depletion might be very fragile. It might be so sensitive to how a test is run that switching from a pen and paper to a keyboard and screen would be enough to make it disappear. If that’s the case, then why should we trust all those other variations on the theme? If that’s the case, then the Big Idea has shrunk to something very small.
Personally, I don’t believe that this is a case of experimental outcomes being subject to specific experimental context. Rather, the ‘experimental context’ is the ‘garden of forking paths’, p-hacking and publication bias.