A Number Walks Into a Boardroom
The slide is already on the screen when the trustees take their seats. An adolescent-nutrition programme in eastern Uttar Pradesh: anaemia among girls in the programme blocks was 58 per cent at baseline, 41 per cent at endline. Seventeen points in two years. The chair beams. There is a proposal on the table to triple the budget and expand to three new states, and this one number is about to carry the vote.
Nobody in the room is being dishonest. The survey was real, the fieldwork careful, the fall in anaemia genuine. And yet the slide does not show what everyone in the room believes it shows. In those same two years, the state ran an iron-and-folic-acid campaign through every government school, a good harvest lowered food prices, and the district rolled out fortified rice through the public distribution system. Some of those seventeen points belong to the programme. Some belong to everything else. The slide cannot tell you which.
The Question Inside Every Impact Claim
Every impact claim — every "we reduced", "we increased", "we transformed" — quietly contains a comparison. It compares the world as it is with a world that never happened: the one where the programme did not run. Evaluators call that invisible world the counterfactual, and the entire craft of impact evaluation is the art of estimating it honestly.
This means the question is never "did things improve?" Things improve, and worsen, for a hundred reasons — seasons turn, economies grow, children age, governments act. The question is always "compared to what would have happened anyway?" Impact is not the change you observed. Impact is the gap between what happened and what would have happened without you. The change you observed is merely one side of that subtraction; the other side is invisible, and it has to be estimated from somewhere.
None of this is exotic. You already think counterfactually a dozen times a day — "the traffic would have been worse on the ring road", "she'd have recovered without the antibiotics anyway". What the development sector struggles with is applying the same reflex to its own numbers, partly because reporting formats reward before-after arithmetic: baseline column, endline column, arrow pointing up. The counterfactual habit is not a statistical technique. It is a discipline of asking, before celebrating or despairing, where the invisible column would have landed — and being suspicious of anyone whose answer is simply "wherever the baseline was".
Four Ways the Comparison Lies to You
When nobody builds the counterfactual deliberately, one gets smuggled in by accident — and the accidental ones are systematically flattering. Four traps account for most of the damage. (Our new Counterfactual game lets you practise dodging all of them, plus a few more, in ten minutes.)
1. Regression to the mean
An after-school reading programme in Lucknow enrols the children who read worst on a fluency screening — the bottom of every class list. One term later, the same screen shows most of them reading noticeably better, and the annual report celebrates. But children land at the bottom of a single test partly through bad luck: illness that day, a noisy hall, one misread passage. Test them again and the unlucky ones drift back towards their true level with or without tuition. Whenever you select people because they scored at an extreme, their next measurement improves on average by statistical gravity alone — and a before-after comparison hands the programme the credit.
2. Self-selection
A federation promoting women's self-help groups in coastal Odisha finds that members' households save more and their children miss fewer school days than non-member households in the same panchayats. Proof that the groups work? Not yet. The women who walked into a savings meeting and stayed were already different from those who did not — more forward-planning, better supported at home, less consumed by crisis. Those same traits drive savings and school attendance directly. Comparing joiners with non-joiners measures who joins, not what joining does: the two groups differed before the first meeting was ever held.
3. Survivorship
An apparel-sector training centre in Indore reports an 82 per cent job-placement rate — calculated among trainees who finished the four-month course. The 500 who left midway, many pulled out by marriage, migration or a family illness, appear nowhere in the denominator. Yet the forces that pulled them out are the same forces that make steady factory employment hard, so the completers were always the trainees most likely to be placed, course or no course. Judge a programme only by its survivors and you are studying persistence, not training. The honest denominator is everyone who started.
4. Confounding shocks
A micro-irrigation subsidy scheme in Bundelkhand runs its endline the year the monsoon fails. Farm incomes come in below baseline, and the review meeting turns funereal — until someone asks what happened to incomes on the unsubsidised farms next door, which fell further. A before-after number silently attributes everything that occurred in between — drought, price crashes, a pandemic, an election-year subsidy — to the programme. The same arithmetic cuts both ways: had that endline landed in a bumper year, a mediocre scheme would have been crowned a triumph by the rain.
The Hierarchy of Honest Answers
None of this means every programme needs a randomised trial. It means every claim should be built on the strongest counterfactual that is feasible in its context — and that we should be honest about which rung of the ladder we are standing on.
- Before-after. The weakest rung. Good enough only when almost nothing else plausibly moves the outcome — counting toilets your programme built, yes; claiming income gains in a living economy, no.
- Matched comparison. Compare participants with non-participants who look similar on observable traits. Good enough when you understand why people got the programme and that reason is largely captured in your data — say, an administrative rule about district eligibility — rather than in unmeasured motivation.
- Difference-in-differences. Compare the change in participants with the change in a comparison group, netting out both fixed differences and common shocks like that Bundelkhand drought. Good enough when you hold baseline data for both groups and their pre-programme trends ran parallel; our post on quasi-experimental designs walks through the mechanics.
- Natural experiments. Let an eligibility cutoff, a lottery, or a staggered rollout do the assigning for you. Good enough — often excellent — when the rule that decided who got the programme had nothing to do with who was likely to succeed.
- Randomisation. The cleanest counterfactual there is, because chance guarantees the two groups start identical on average. Good enough by construction, but only worth it when the rollout genuinely permits it — an oversubscribed programme, a phased expansion — and when it is done well: a leaky RCT with heavy attrition tells you less than a careful difference-in-differences.
The stance worth internalising: the goal is not the gold standard, it is the strongest feasible design. A pipeline comparison that rides an expansion you were doing anyway beats an imaginary trial you will never run. The full logic — from theory of change to power calculations to ethics — is in our Causal Inference & Impact Evaluation course.
The Habit
You do not need statistics to practise counterfactual thinking. You need a reflex — three questions asked, politely and every time, of any number that claims impact.
Three questions for any impact claim
- Who was compared to whom? If the answer is "the same people, earlier", you have a before-after number, not an impact estimate.
- Why might the groups have differed before the programme? Volunteers, completers, and the lowest scorers are never a random slice of anyone.
- What else changed over the same period? Monsoons, markets, elections, and other people's programmes do not pause for your evaluation.
Ask them in that Uttar Pradesh boardroom and the conversation changes. Perhaps the seventeen points survive scrutiny — programmes do work, and honest counterfactuals often vindicate them, as the Bundelkhand irrigation scheme discovered in reverse. Or perhaps the board delays the vote and commissions a comparison-block survey first. Either way, the decision now rests on impact rather than arithmetic coincidence.
The habit sharpens with use. Try the Counterfactual game against eight boardroom-ready claims, take on the Stress-Test an Impact Claim brief among our Live Case Challenges, or work through the Critiquing Evidence practice pack with your team. The next time a triumphant slide goes up, be the person who asks, kindly: compared to what would have happened anyway?