Every impact claim casts a shadow question: what would have happened anyway? The village that got the programme is easy to see. The counterfactual village — its ghostly twin — must be found.
Illustration in the Warli tradition of Maharashtra — the art of the circle dance.
Counterfactual
The Evaluation Game — can you find what would have happened anyway?
You are the new Evaluation Lead at the Sambhav Collaborative, a consortium of Indian NGOs and funders. Eight programme teams are queued outside your office. Each arrives with a confident impact claim — and a decision to make about how to test it.
Your job is not to be a cynic. It is to choose, for each situation, the evaluation design that gets closest to the true counterfactual given the real constraints — budget, timing, ethics, and politics. Sometimes that's a randomised trial. Often it isn't.
8 rounds, ~10 minutes
Each round is a real-world scenario drawn from Indian development practice. Read the claim, weigh the constraints, pick a design.
Score 0–3 per round
3 points for the strongest feasible design, partial credit for defensible ones, 0 for the classic trap. Maximum: 24.
8 traps to dodge
Every scenario hides one classic threat to validity — regression to the mean, self-selection, survivorship, spillovers, and friends.
No jargon required
You don't need econometrics. You need the counterfactual habit of mind. The feedback explains every design in plain language.
What do you recommend?