How likely is “likely”?

From Andrew Mauboussin and Michael Mauboussin:

In a famous example (at least, it’s famous if you’re into this kind of thing), in March 1951, the CIA’s Office of National Estimates published a document suggesting that a Soviet attack on Yugoslavia within the year was a “serious possibility.” Sherman Kent, a professor of history at Yale who was called to Washington, D.C. to co-run the Office of National Estimates, was puzzled about what, exactly, “serious possibility” meant. He interpreted it as meaning that the chance of attack was around 65%. But when he asked members of the Board of National Estimates what they thought, he heard figures from 20% to 80%. Such a wide range was clearly a problem, as the policy implications of those extremes were markedly different. Kent recognized that the solution was to use numbers, noting ruefully, “We did not use numbers…and it appeared that we were misusing the words.”

Not much has changed since then. Today people in the worlds of business, investing, and politics continue to use vague words to describe possible outcomes.

To examine this problem in more depth, team Mauboussin asked 1700 people to attach probabilities to a range of words or phrases. For instance, if a future event is likely to happen, what percentage of the time would you estimate it ends up happening? Or what if the future event has a real possibility of happening?

Unsurprisingly, the answers are all over the place. The HBR article has a nice chart of the distribution of responses, and you see more detailed results here. (You can also take the survey there too).

What is the range of answers for an event that is “likely”? The 90% probability range for “likely” – that is the range that 90% of the answers fell within (and 5% of the answers were above, and 5% below) was 55% to 90%. “Real possibility” had a probability range of between 20% and 80% – the phrase in near meaningless. Even “always” is ambiguous, with a probability range of 90% to 100%.

An interesting finding of the survey was that men and women differ in their interpretations. Women are more likely to take a phrase as indicating a higher probability.

So what does team Mauboussin suggest we should do? Use numbers. Pin down those subjective probabilities using objective benchmarks. Practice.

And to close with another piece of Sherman Kent wisdom:

Said R. Jack Smith:  Sherm, I don’t like what I see in our recent papers. A 2-to-1 chance of this; 50-50 odds on that. You are turning us into the biggest bookie shop in town.

Replied Kent:  R.J., I’d rather be a bookie than a [blank-blank] poet.

One thought on “How likely is “likely”?

  1. Many years ago, as a first year grad student in psycholinguistics, my adviser suggested that I study hedged phrases like “real possiblity” and “very likely” from the perspective of the “fuzzy logic” of Lofti Zadeh. I didn’t realize that Zadeh was an engineer and didn’t really care that his system wasn’t well-defined or fully self-consistent, so I declined that project. Seeing these observations, I’m glad that I did. Nowdays, trajectories through activation landscapes in a high-dimensional “neural network” vector space would make make much more sense.

    But the background study that I put in at that time remains with me in a large amount of sympathy for the agonizingly precisely imprecise definitions of climate change event likelihoods used by the Intergovernmental Panel on Climate Change in their reports.

Comments welcome

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s