Move from "did our programme work?" to a defensible causal claim. Frame the question and counterfactual, choose an identification strategy, size your sample and power, and stress-test the threats to validity — the companion tool to the Causal Inference for Development course.
Frame the Causal Question
Impact is a comparison against a counterfactual — what would have happened to the same units without the intervention. Since we never observe both states for one unit (the fundamental problem of causal inference), the whole job is to build a credible stand-in for that missing counterfactual.
Potential outcomes (the Rubin causal model). Each unit has two potential outcomes: Y(1) if treated and Y(0) if not. The individual treatment effect is Y(1) − Y(0), but only one is ever observed. Evaluation designs differ mainly in how they estimate the average of the unobserved side.
State it as a causal question about an effect on an outcome for a population.
The single measurable variable your claim rests on.
What exactly is delivered, to whom, at what dose?
Who or what will stand in for the untreated state? Be honest about how comparable they really are.
ITT vs treatment-on-the-treated. In real programmes not everyone offered the treatment takes it up. Intention-to-treat answers the policy-relevant question "what happens if we offer this?" and is protected by randomisation. The effect of treatment on the treated (via ATT or an instrumental-variables LATE) answers "what happens to those who actually comply?" — a larger number, but one that leans on stronger assumptions. Decide which your decision-maker needs before collecting data.
Choose an Identification Strategy
There is no universally "best" design — each buys its counterfactual with a different, testable assumption. Pick the one whose key assumption is most plausible in your setting. Tap a card to see when to use it, what it assumes, and how it can break.
The credibility test: for whichever design you pick, ask "what would have to be true for this to recover the real counterfactual — and can I test or defend that?" If the key assumption is untestable and implausible in your context, a bigger sample will not save the estimate.
Sample Size & Statistical Power
An under-powered study can miss a real effect and wrongly conclude "no impact". Before fielding, work out how many units you need to detect the smallest effect worth acting on — the minimum detectable effect (MDE).
Absolute change in the outcome mean you want to detect.
Control-group rate of the outcome. MDE above is the absolute change in this rate.
Individuals measured per cluster (e.g. pupils per school).
0 = no clustering; 0.05–0.20 is common for education/health outcomes.
The formula (two-arm comparison of means). Required sample per arm is
n ≈ (z1−α/2 + z1−β)² · 2σ² / Δ²,
where Δ is the MDE, σ the outcome SD, and the z-values come from the chosen α and power. For a binary outcome, 2σ² is replaced by p1(1−p1) + p2(1−p2). For cluster designs the per-arm figure is inflated by the design effectDEFF = 1 + (m − 1)ρ.
What this estimate assumes. It is a normal-approximation, equal-allocation, single-outcome, two-sided formula — a defensible first-pass number, not a substitute for a statistician. It ignores expected attrition (inflate for it), covariate adjustment / baseline controls (which can shrink it), multiple hypotheses (which raise it), and unequal group sizes. Treat σ and ρ as assumptions drawn from a pilot or comparable published study, and revisit them against your own baseline. First-draft sizing
Threats to Validity
Internal validity asks whether your estimate is really the causal effect in your sample. External validity asks whether it will hold elsewhere or at scale. Tick each threat you have a credible plan for, and note that plan — the notes flow into your exported memo.
Internal validity
External validity
Robustness score
Tick the threats you have addressed, then score your design's coverage.
Design is a set of trade-offs, not a checklist to ace. Some threats are unavoidable given your context — the goal is to know which ones bite, address what you can, and be transparent about the rest in your report. A credible evaluation names its own weaknesses.
Export your evaluation design memo
Download a plain-text memo summarising your question, chosen design and its assumptions, power calculation, and validity plan — ready to share with your team or a reviewing statistician.
You have a design on paper
A written question, a named identification strategy with its key assumption, a power target, and an honest list of threats — that is the backbone of a pre-analysis plan.
Interactive Lab