Every impact claim casts a shadow question: what would have happened anyway? The village that got the programme is easy to see. The counterfactual village, its ghostly twin, must be found.
Illustration in the Warli tradition of Maharashtra: the art of the circle dance.
Counterfactual
The Evaluation Game: can you find what would have happened anyway?
You are the new Evaluation Lead at the Sambhav Collaborative, a consortium of Indian NGOs and funders. Eight programme teams are queued outside your office. Each arrives with a confident impact claim, and a decision to make about how to test it.
Your job is not to be a cynic. It is to choose, for each situation, the evaluation design that gets closest to the true counterfactual given the real constraints: budget, timing, ethics, and politics. Sometimes that's a randomised trial. Often it isn't.
8 rounds, ~10 minutes
Each round is a real-world scenario drawn from Indian development practice. Read the claim, weigh the constraints, pick a design.
Score 0–3 per round
3 points for the strongest feasible design, partial credit for defensible ones, 0 for the classic trap. Maximum: 24.
8 traps to dodge
Every scenario hides one classic threat to validity: regression to the mean, self-selection, survivorship, spillovers, and friends.
No jargon required
You don't need econometrics. You need the counterfactual habit of mind. The feedback explains every design in plain language.
What do you recommend?