How to Check a Number: Sanity Checks for Any Statistic

Numbers arrive with an air of authority. A figure in a headline, a slide or a viral post looks like a fact simply because it is precise. Most are fine. Some are wrong by a factor of a thousand, and a few minutes of rough arithmetic is usually enough to tell which is which.

The checks below need no special knowledge, only a handful of reference figures and the habit of stopping for a moment. They are the same moves that make What's Your Guess? work, played in reverse: instead of building an estimate from scratch, you build one to test a number someone else has given you.

Before judging a number, be sure what it measures. Households or people? Adults or everyone? Per day or per year? Sold, made or owned? The economist and broadcaster Tim Harford, in How to Make the World Add Up (2020), calls skipping this step premature enumeration: arguing about how big a number is before anyone has agreed what it counts.

A surprising number of disagreements about statistics are really disagreements about definitions. Two honest sources can give very different figures for "the number of homeless people" in a city if one counts people sleeping outdoors on a single night and the other counts everyone placed in temporary housing over a year. Neither is wrong. They are answering different questions.

The single most useful check is to share a total out per person. Big totals are hard to judge; amounts per person are easy, because you are a person and have a feel for what one person does.

Suppose you read that a city of 500,000 people uses 2 billion litres of water a day. Divide: that is 4,000 litres per person per day. Homes in the UK use around 140 to 150 litres per person a day. Even allowing for offices, factories and leaks, 4,000 is far too high. Perhaps the figure is per year rather than per day, or it covers a whole region, or the units were gallons or cubic metres somewhere along the way. Something has slipped, and you found it in ten seconds.

Hans Rosling, in Factfulness (2018), called the failure to do this the size instinct: a lonely number always looks big. His remedy was to compare it with something and to divide it by something. Per person, per year, per pound spent: a rate makes a number meaningful.

The same idea works with time. A figure per year often becomes clear per day, and a figure per day becomes clear per minute.

If a claim says the average person spends a certain number of years of their life doing something, convert it back. Two years over an 80-year life is about 9 days a year, or about 36 minutes a day. That is easy to judge against your own day. Suppose it had said twelve years: that is three and a half hours a day, which might be plausible for television and is not plausible for, say, waiting at traffic lights.

Unit errors are the most common way for a number to be wrong by a large, clean factor, and they have caused real disasters.

• Pounds and kilograms. In 1983 an Air Canada Boeing 767 was fuelled using pounds where the calculation needed kilograms, so it carried less than half the fuel its crew thought. It ran dry at 41,000 feet over north-western Ontario, and its pilots glided it to a landing on the runway of a former air force base at Gimli, Manitoba; all 69 people on board survived. It became known as the Gimli Glider.
• Imperial and metric. In 1999 NASA lost the Mars Climate Orbiter because one piece of software reported thruster data in pound-force seconds while another expected newton-seconds, a factor of about 4.45. The spacecraft approached Mars far too low.
• Million and billion. One letter apart and a thousand times different. A council budget, a company's profit or a country's debt quoted in the wrong one is a story in itself. The words have even changed meaning: in older British usage a billion was a million million.
• Power and energy. A kilowatt is a rate; a kilowatt-hour is an amount. A battery's capacity quoted in kilowatts, or a household's yearly use quoted in kilowatts, has mixed the two up.
• Length, area and volume. Doubling a length quadruples an area and multiplies a volume by eight. A claim that something "twice the size" holds twice as much may be comparing different dimensions.

Whenever a number seems ten, a thousand or a million times off, look first for a unit.

A part cannot be bigger than the whole it belongs to, and that simple rule catches a lot. The whole world makes around 90 million motor vehicles a year, so a claim that one factory makes 50 million is impossible before you check anything else. If a survey says 40% of people do one thing, 35% do a second and 45% do a third, and the three were meant to be exclusive, the shares add up to 120% and something is wrong.

Keep a few totals in your head for this: the world's population, a country's, the world economy, the hours in a year. Our pocket list of numbers worth knowing has the main ones, rounded to remember.

When you have a few minutes, rebuild the number from parts. If a figure claims that a city has 5,000 taxis, estimate it from population, how many trips people take by taxi and how many trips one taxi can do in a day. If your estimate lands within a factor of two or three, the number is probably sound. If it lands a hundred times away, one of you has made a mistake, and finding out which is the interesting part.

Better still, estimate it two different ways: once from the top down (the whole population, then the share who use something) and once from the bottom up (one shop, one worker, one day, then multiply). Two independent routes that agree are much stronger evidence than one.

Changes in rates are a favourite source of confusion.

• Relative and absolute. "Doubles your risk" sounds alarming. If the risk goes from 1 in 10,000 to 2 in 10,000, it has doubled, and it is still 1 in 5,000. Ask for the starting figure.
• Percent and percentage points. If a rate rises from 10% to 15%, it has gone up by 5 percentage points, and by 50 per cent. Both are true; one sounds much bigger.
• The missing denominator. "10,000 people affected" means little until you know out of how many.

The last check is not arithmetic at all. Ask where the number came from.

• Who published it, and why? A national statistics office, a peer-reviewed study, a company press release and a campaign group can all be right, but they have different reasons to choose how a figure is framed.
• How was it measured? A count, a survey, a model or a guess? A survey of 200 people is a different thing from a census.
• When? An old figure quoted as new is one of the most common errors there is. Populations, prices and technology figures go stale within a few years.
• Is this the original? Numbers mutate as they are copied. Follow the chain back to the first place it was published and see whether it still says the same thing.
• Who is missing? Harford's question. A figure about "workers" may leave out the self-employed; one about "users" may leave out everyone who tried something once and left.

The quickest way to get good at checking numbers is to see a lot of numbers being checked. That is what the reveal in What's Your Guess? shows every time: the answer, the working that produces it with every figure filled in, and the publisher of each figure, linked so you can read the original. Before you open How it adds up, predict which figures the working will need. When an answer surprises you, open its source and see how the number was actually measured.

Do that for a while and a slipped zero in a headline starts to look as obvious as a misspelling. If the trouble is less about checking than about feeling the size of big numbers at all, start with why big numbers are hard to grasp.