Bankers are more honest than the rest of us

Well, probably not. But that’s one interpretation you could take from a the oft-quoted and cited Nature paper by Cohn and colleagues Business culture and dishonesty in the banking industry. That bankers are more honest is as plausible as the interpretation of the experiment provided by the authors.

As background to this paper, here’s an extract from the abstract:

[W]e show that employees of a large, international bank behave, on average, honestly in a control condition. However, when their professional identity as bank employees is rendered salient, a significant proportion of them become dishonest. … Our results thus suggest that the prevailing business culture in the banking industry weakens and undermines the honesty norm, implying that measures to re-establish an honest culture are very important.

I’ve known of this paper since it was first published (plenty of media and tweets), but have always placed it in the basket of likely not true and unlikely to be replicated. Show me some pre-registered replications and I would pay attention. As a result, I didn’t investigate any further.

But recently Koen Smets pointed me toward a working paper from Jean-Michel Hupé that critiqued the statistical analysis. That paper in turn pointed to a critique by Vranka and Houdek, Many faces of bankers’ identity: how (not) to study dishonesty.

These critiques caused me to go back to the Nature paper – and importantly, to the supplementary materials – and read it in detail. It has a host of problems besides being unlikely to replicate. The most interesting of these could lead us to ask whether bankers are actually more honest.

The experiment

Cohn and friends recruited 128 bank employees and randomly split them into two groups, the treatment and control. Before undertaking the experimental task, the treatment group was “primed” with a series of questions that reminded them that they were a bank employee (e.g. At which bank are you presently employed?). The control group were asked questions unrelated to their professional identity.

The experimenters then asked each member of these two groups to flip a coin 10 times, reporting the result via a computer. No-one else could see what they had flipped. For each flip that came up the right way, the experimenters paid them (approximately) $20 (or more precisely, they would be paid $20 per flip if they equalled or outperformed a randomly selected colleague). Ten correct flips and you could have $200 coming your way.

So how can we know if any particular person is telling the truth? You can’t. But across a decent sized group, you know the distribution of results that you would expect (a binomial distribution with a mean of 0.5). You would expect, on average, 50% heads and 50% tails. Someone getting 10 heads is a 1 in a thousand event. By comparing the distribution of the results to what you would expect, you can infer the level of cheating.

So, how did the bankers go? In the control group, 51.6% of coin flips were successful. It’s slightly more than 50%, but within the realms of chance for a group of honest coin flippers. The bankers primed with their professional identity reported 58.2% successful flips, 6.6 percentage points more than the control group. The dishonest bandits.

But how do we know that this result is particular to bankers? What if we primed other professionals with their profession? What if we took a group with no connection to the banking industry and primed them with banking concepts?

Cohn and friends answered these questions directly. When they primed a group of non-banking professionals with their professional identity, they reported 3 percentage points fewer successful coin flips than those in a control condition. Students primed with banking concepts also reported fewer successes, around 1.5%. These differences weren’t statistically significant and could have happened by chance, with no detectable effect from the primes.

These experimental outcomes are the centrepiece behind the conclusion that the prevailing culture in banking weakens and undermines the honesty norm.

But now let’s go to the supplementary materials and learn a bit more about these non-banking professionals and students.

An alternative interpretation

I have only reported the differences in successful coin flips above – as did the authors in the main paper (in a chart, Figure 3a). So how many successes did these non-banking professionals and students have?

In the control condition, the non-banking professionals reported 59.8% successful flips. This dropped to 55.8% when primed with their professional identity. The students were also dishonest bandits, reporting 57.9% successful flips in the control condition, and 56.4% in the banking prime condition.

So looking across the three groups (bankers, non-banking professionals and students), the only honest group we have come across are the bankers in the control condition.

This raises the question of what the appropriate reference point for this analysis is. Should we be asking if banking primes induce banker dishonesty? Or should we be asking whether the control primes – which were designed to be innocuous – can induce honesty? To accept that the banking prime induces bankers to cheat more, we also need to have a starting point that bankers, on the whole, cheat less.

I don’t see a great deal of value in trying to interpret this result and determine which of these frames are correct, as the result is just noise. It is unlikely to replicate. But once you look at these numbers, the interpretation by Cohn and friends appears little more than an overly keen attempt to get the results to fit their “theoretical framework”.

Other problems

I’ve just picked my favourite problem, but the two critiques I linked above argue that there are others. Vranka and Houdek suggest that there are many other ways to interpret the results. I agree with that overarching premise, but am less convinced by some of their suggested alternatives, such as the presence of stereotype or money primes. Those primes seem as robust as this banking prime is likely to be.

Hupé critiques the statistical approach, with which I also have some sympathy, but I haven’t spent enough time thinking about it to agree with his suggested alternative approach.

A quick afterthought

That this experimental result is bunk is not a reason to dismiss the idea that banking culture is poor or that exposure to that culture increases dishonesty. The general problem with the priming literature is that it attempts to elicit differences through primes that are insignificant relative to the actual environments people face.

For example, there is a large difference between answering a few questions about banking and working in a bank. In the latter, you are surrounded by other people, interacting with them daily, seeing what they do. Just because a few questions do not produce an effect doesn’t mean that months of exposure to a your work environment won’t change behaviour. Unfortunately, experiments such as this add approximately zero useful information as to whether this is actually the case.

Comments welcome

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s