Overconfidence in Estimates: Why Our Ranges Are Too Narrow and How to Calibrate

Try this on a friend. Ask them for a range they are 98% sure contains some number neither of you knows, such as the length of a famous river or the year a building was finished. Ask ten such questions. If they are well calibrated, the truth should fall outside their range about once in fifty questions. In practice, for most people, it falls outside several times in ten.

That gap between how sure we feel and how often we are right is one of the most reliable findings in the psychology of judgement. It matters for anyone who makes estimates, and it matters in a very direct way in What's Your Guess?, where every guess comes with an invisible range attached and the run ends the first time the truth falls outside it.

In the late 1960s Marc Alpert and Howard Raiffa asked students at Harvard Business School for estimates of quantities they could not know offhand. Rather than a single number, each student gave a spread: a value they thought was 99% likely to be too high, one 99% likely to be too low, and points in between. The outer pair made a 98% range. If the students were calibrated, about 2% of the true answers would fall outside it.

About 40% did. The ranges were far too narrow. When the students were shown the result and asked to try again, their ranges widened and the misses fell, but they stayed far above 2%. Their paper circulated for years before it was published in 1982 in Judgment under Uncertainty: Heuristics and Biases, the collection edited by Daniel Kahneman, Paul Slovic and Amos Tversky, and it has been repeated in one form or another ever since.

Later studies have changed the questions, the people and the wording, and the result keeps coming back. A 2005 paper by Karl Halvor Teigen and Magne Jørgensen sums it up in its title: When 90% confidence intervals are 50% certain.

In 2008 Don Moore and Paul Healy, writing in Psychological Review, sorted overconfidence into three kinds. Overestimation is thinking you are better than you are. Overplacement is thinking you are better than others. Overprecision is being too sure that you know the truth, which shows up as ranges that are too narrow. Moore and his colleagues have since called overprecision the most robust of the three: the other two come and go with the task, but overly narrow ranges turn up almost everywhere.

We anchor on our best guess. The usual way to make a range is to think of a likely number and then stretch it outwards a bit each way. In their famous 1974 paper in Science, Tversky and Kahneman showed how stubborn a starting number is: when a rigged wheel of fortune landed on 10 or 65 before people estimated the share of African countries in the United Nations, the median estimates were 25% and 45%. A meaningless number moved the answer. A best guess, which is far from meaningless, holds a range even more tightly.

We size the range for the wrong uncertainty. We tend to allow for small errors inside our picture of the problem, a figure a little high or low, and forget the chance that the picture itself is wrong. A chain of four steps, each honestly uncertain by a factor of two, can be out by a factor of sixteen when the errors line up. Our ranges rarely allow for that.

Hard questions make it worse. Sarah Lichtenstein and Baruch Fischhoff reported in 1977 what became known as the hard–easy effect: people tend to be overconfident on difficult questions and, if anything, a little underconfident on easy ones. The less we know, the less we realise how much we do not know.

Narrow sounds like knowing. A precise answer sounds expert, and a wide one sounds like a shrug. Socially, overprecision is rewarded, which is one reason it survives.

The encouraging part is that calibration responds to practice, as long as the practice comes with feedback. The standard example is weather forecasting. When Allan Murphy and Robert Winkler studied American forecasters in the 1970s, they found that when the forecasters said there was a 70% chance of rain, it rained about 70% of the time. Forecasters get exactly the conditions calibration needs: many judgements, a quick and unarguable outcome for each, and a running score.

The same idea runs through the training Douglas Hubbard describes in How to Measure Anything, which uses the equivalent bet. Suppose you could win a prize either if the truth lies inside your 90% range, or by spinning a wheel that wins nine times in ten. If you would rather spin the wheel, your range is not really 90%: widen it until you could not choose between them. In Superforecasting, Philip Tetlock and Dan Gardner describe forecasters who became better calibrated by keeping score of their predictions year after year.

Before you commit to a number, a few questions sort real knowledge from a feeling of knowledge.

• Did you remember it, or build it? A figure you have read and can place ("about 8 billion people, from the UN") is different from one you just constructed. Both are fine. Only the first deserves a narrow range.
• How many links are in the chain? Every uncertain step widens the range, and the widening multiplies. Two steps each uncertain by a factor of three can be out by nine together.
• Do two routes agree? Estimate the same quantity from two different starting points. If they land close together, you can trust a narrower range. If they do not, you have found the weak link.
• Did the number arrive too fast? A figure that springs to mind at once is sometimes the answer to a neighbouring question: a country's population when the question was about its capital, a total when it asked about a year.
• Would you take the bet? Run the equivalent bet in your head. If the wheel looks tempting, widen.

Every guess in What's Your Guess? is really a range in disguise. It survives if the answer is anywhere from a tenth of the guess to ten times it, so each guess covers a window a hundred times wide. Close is a narrower window, from half the guess to double it, and Bullseye is narrower still.

That changes how to think about a hard question. Do not start by picking a number. Start by asking for your honest range, wide enough that you would take the equivalent bet. If it is less than a hundred times wide, the middle of it on a log scale cannot lose. If it is wider, no guess is safe, and the right move is to find the piece of the problem that makes it so wide and think harder about that piece before you type. The method for finding that middle is in how to get better at estimating.

Overconfidence tends to show up in runs in a recognisable way. Often the guess that ends a run is not one you agonised over but one that felt obvious, typed quickly, built on a number that came to mind too easily. And because runs get harder from the ninth question on, the hard–easy effect bites exactly where streaks are made or lost.

You can turn the game into calibration training with a pencil and paper.

1. Before you check each guess, write down how confident you are that it will land Close or better, within a factor of two: sure (about 8 in 10), maybe (about 5 in 10) or no idea (about 2 in 10).
2. After each reveal, note whether it did.
3. After 30 or 40 questions, count. Of your "sure" guesses, how many were Close? Of your "no idea" guesses?

If your "sure" guesses land Close far less than eight times in ten, you are overprecise, like nearly everyone. The end-of-run screen lists every question with how many times off you were, which makes the counting quick. Keep it up over a few sessions, adjusting as you go, and your "sure" should start to mean what it says.

The other remedy is not to trust one mind at all. Combining independent guesses, even two of your own, cancels some of the error that makes single ranges too tight, which is the story of the wisdom of crowds.