ImpactMojoImpact Evaluation 101www.impactmojo.in

ImpactMojo 101 Series · Free Forever

Impact
Evaluation
101

Designing & Commissioning Credible Evaluations — a Foundational Course for Development Programme & MEL Practitioners in South Asia

Practice-FocusedSouth Asia Focus100 SlidesFree Access

ImpactMojoImpact Evaluation 101www.impactmojo.in

Agenda

What We Cover

What Impact Evaluation Is

Slides 3–11

The Counterfactual & Causal Inference

Slides 12–20

Theory of Change & Evaluation Questions

Slides 21–29

Randomised Experiments (RCTs)

Slides 30–38

Quasi-Experimental Designs

Slides 39–48

Choosing a Method

Slides 49–57

Sampling & Statistical Power

Slides 58–66

Measurement & Data Collection

Slides 67–75

Analysis & Interpreting Results

Slides 76–84

External Validity, Cost & Ethics

Slides 85–91

Using IE for Decisions & Further Reading

Slides 92–99

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section One

What Impact Evaluation Is

ImpactMojoImpact Evaluation 101www.impactmojo.in

Definition

Measuring the change a programme caused

An impact evaluation answers one demanding question: how much of the change we observe was actually caused by our programme — and not by everything else happening at the same time?

Impact evaluation

A study that estimates the causal effect of a programme, policy or intervention on an outcome — the difference between what happened with the programme and what would have happened without it.

Every other M&E question can be useful. This one tells you whether the programme is worth running at all.

The question requires	Which most reports lack
A comparison group	Only the treated are measured
Measurement before and after	Only endline exists
A stated identification strategy	None is offered
A pre-specified outcome	Outcomes chosen afterwards

Any one of these missing means the study can describe a change and cannot attribute it, which is a different and much weaker claim.

That is not a reason to dismiss such reports. It is a reason to describe them accurately: they report outcomes, not impact.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Distinction

Impact is not the same as monitoring

	Monitoring	Impact evaluation
Asks	Are we doing what we planned?	Did it cause the change?
Tracks	Inputs, activities, outputs	Outcomes vs a counterfactual
Timing	Continuous, routine	Periodic, designed in advance
Example	1,200 toilets built	Did diarrhoea actually fall because of them?
Answers	Implementation	Attribution

Monitoring tells you the programme ran. Impact evaluation tells you whether it worked. You need both — they answer different questions.

	Monitoring	Impact evaluation
Question	Are we on plan?	Did we cause it?
Frequency	Continuous	Once or twice
Needs a comparison	No	Yes
Cost	Built into delivery	A separate budget
Can correct delivery	Yes	Usually too late

The last row is why monitoring is not a lesser version of evaluation: it is the only one of the two that can change what happens during the programme.

A programme with excellent monitoring and no impact evaluation is often better managed than the reverse.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Results Chain

Where impact sits in the results chain

INPUTS: budget, staff, training

→

ACTIVITIES: deliver the programme

→

OUTPUTS: toilets built, classes held

→

OUTCOMES: behaviour, take-up

→

IMPACT: the caused change in wellbeing

Monitoring lives on the left of this chain; impact evaluation targets the far right — and insists on attribution, not just association.

Level	Monitored by	Evaluated by
Inputs and activities	Routine systems	—
Outputs	Routine systems	Process evaluation
Outcomes	Periodic survey	Impact evaluation
Impact	Rarely	Impact evaluation, if at all

Monitoring covers the top of the chain and impact evaluation the bottom, and the middle is where the two overlap and are most often confused.

Most programmes claim results at the outcome level with data from the output level, which is the commonest reporting error in the sector.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Attribution Question

Compared to what?

A village had the programme; outcomes improved. Tempting to claim success — but outcomes might have improved anyway, through a good monsoon, rising incomes, or other schemes. The honest question is always: compared to what?

The fundamental question of impact evaluation is not 'did things get better?' but 'did things get better because of us?'

— a working principle of evaluation practice

Outcomes improved because	Unless you rule it out
The programme worked	You cannot claim it
The monsoon was good	A trend, not your effect
Incomes were rising anyway	A secular trend
Another scheme arrived	A confounder
You selected the promising villages	Selection bias

Each alternative explanation has to be excluded by design rather than by argument, which is what an identification strategy is for.

The last row is the one under your own control and the one most often responsible: programmes are placed where they are likely to succeed.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Why It Matters

What credible evidence buys you

When you have it

Scale what genuinely works
Stop what does not, and free up budget
Defend funding with real evidence
Learn why, not just whether

When you don't

Scale failures by mistake
Credit yourself for trends you didn't cause
Cut effective programmes blindly
Repeat the same expensive errors

With credible evidence	Without it
Scale what works	Scale a failure
Stop what does not	Keep funding it
Defend the budget	Defend it on anecdote
Learn the mechanism	Repeat the design

The second row is the one that saves the most money and is the hardest institutionally, because stopping a programme has an identifiable loser.

That is a political-economy problem rather than an evidence problem, and no amount of rigour resolves it on its own.

ImpactMojoImpact Evaluation 101www.impactmojo.in

A Caution

Impact evaluation is not always the answer

Impact evaluation is powerful but costly and slow. It is one tool in the MEL toolkit — not a stamp of legitimacy to apply to everything. Much of the time, good monitoring and process evaluation serve you better.

This course is as much about knowing when to commission an impact evaluation as about how to design one.

Do an impact evaluation when	Do something else when
The question is causal and open	It is already well established
A real decision hangs on it	Nobody will act on the answer
The programme is stable	It changes weekly
A credible comparison exists	Everyone is treated
You can afford it	Monitoring would serve better

Impact evaluation is costly and slow, and applying it as a badge of rigour to programmes that need operational information wastes both money and time.

Process evaluation and good monitoring answer most practitioner questions, and they answer them while the programme is still running.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Your Role

You don't have to run it to commission it well

Most practitioners will commission an impact evaluation rather than estimate it themselves. That still demands real literacy: framing the question, choosing a design, judging a proposal, and reading the findings critically.

This deck is written for the commissioner and the manager — the person who must ask hard questions of an evaluator, not necessarily write the regression.

Commissioning requires you to	Which needs
Frame the question	Knowing what decision it informs
Judge a design	Knowing what each design assumes
Read a power calculation	Understanding MDE
Interpret the finding	Distinguishing null from inconclusive

You do not need to run the analysis and you do need to be able to challenge it, because an evaluator will otherwise make these choices for you.

The single most valuable thing a commissioner does is bring the evaluator in before the rollout design is fixed.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Family of M&E

Impact evaluation among its cousins

Activity	Question it answers
Needs assessment	What is the problem, and for whom?
Process evaluation	Was the programme delivered as designed?
Monitoring	Are we on track against the plan?
Impact evaluation	Did the programme cause the change?
Cost-effectiveness analysis	Was the impact worth the cost?

Impact evaluation is one row in this table, not the whole table. It shines only when paired with the others — especially process evaluation, which tells you whether a null means the idea or the delivery failed.

Activity	Answers	Timing
Needs assessment	What is the problem?	Before design
Process evaluation	Was it delivered as designed?	During
Monitoring	Are we on track?	Continuous
Impact evaluation	Did it cause the change?	After, designed before
Cost-effectiveness	Was it worth it?	With the impact estimate

The fourth row’s timing note carries the whole practical lesson: an impact evaluation is executed afterwards and must be designed before the programme starts.

The commonest commissioning failure is asking for an impact evaluation of a programme that has already run everywhere.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Two

The Counterfactual & Causal Inference

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Big Idea

The counterfactual: the road not taken

Counterfactual

What would have happened to the very same people, in the same period, had the programme not existed. It is the benchmark against which impact is measured.

Impact = outcome with the programme − outcome without it, for the same group. The first we observe; the second we never can. Everything in this course is a strategy to estimate that missing half.

The counterfactual is	It is not
What would have happened to the same people	What happened to different people
In the same period	Before the programme
Absent the programme	A national average
Unobservable, always	Something you can measure directly

Because it is unobservable in principle, every design is a way of constructing a credible substitute for it. That is the whole subject.

When reading any evaluation, find the sentence that says what the counterfactual is. If there is no such sentence, there is no impact estimate.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Core Problem

The fundamental problem of causal inference

A person is either treated or not. We can never observe the same person in both states at once, so the true individual counterfactual is permanently missing. This is the fundamental problem of causal inference.

The solution is not to find each person's missing twin, but to build a comparison group that, on average, stands in for what the treated group would have looked like untreated.

We cannot observe	So we estimate
The same person, treated and untreated	A group average
An individual effect	An average treatment effect
The true counterfactual	A credible substitute for it

Holland’s formulation of the fundamental problem is that causal inference is a missing-data problem, and every method here is a way of filling that gap.

It also explains why individual-level claims are not available: an evaluation tells you about an average, not about what happened to any one participant.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Wrong Counterfactuals

Two tempting comparisons that mislead

Before vs after

Compare the same group before and after. But the world moved on too — you confuse the programme with everything else that changed (the monsoon, prices, other schemes).

Participants vs non-participants

Compare those who joined to those who didn't. But joiners often differ — more motivated, better placed. That difference is selection bias, not impact.

Both are easy, cheap and wrong. A credible counterfactual is the whole game.

Wrong comparison	What it actually measures
Before versus after	The programme plus everything else that changed
Participants versus non-participants	Who chose to join
Treated villages versus national average	Differences in the villages
Completers versus dropouts	Who could complete

All four are used routinely in programme reporting, and all four systematically overstate the effect in the same direction.

The fourth is the most seductive: comparing those who finished a training with those who left measures persistence, not the training.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Central Threat

Selection bias, defined

Selection bias

When the people who receive a programme differ systematically from those who don't — in ways that also affect the outcome — so a naive comparison mixes the programme's effect with those pre-existing differences.

If a skills programme enrols the most motivated youth, their later success partly reflects motivation, not training. Selection bias is the single biggest reason naive evaluations overstate impact.

Selection can come from	Example
Self-selection	The motivated volunteer
Programme placement	Villages chosen for likely success
Administrative targeting	The needlest selected
Attrition	Who remains at endline

Note that the third row biases in the opposite direction: if the programme targets the worst-off, a naive comparison understates its effect.

Knowing the direction of the likely bias is useful even when you cannot remove it, because it tells you whether your estimate is an upper or a lower bound.

ImpactMojoImpact Evaluation 101www.impactmojo.in

See It

Why volunteers make a misleading comparison

Illustrative: who self-selects into a training programme

Illustrative example, not real data

Joiners already differ before the programme. Comparing their later outcomes to non-joiners' measures who they were, not what the programme did.

Volunteers differ in	Which also affects
Motivation	The outcome
Prior education	The outcome

The right column is what makes it selection bias rather than mere difference: the characteristic drives both participation and the result.

This is why a comparison of joiners with non-joiners measures who joined, and why programmes with voluntary take-up need a design that handles it.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Goal

What a good comparison group must satisfy

It looks like the treatment group before the programme — same characteristics, same trajectory
It is exposed to the same outside forces — weather, prices, other schemes
The only systematic difference is the programme itself

Get this right and the comparison group's outcomes are a credible estimate of the treatment group's counterfactual. Every method that follows is a different way to build such a group.

A good comparison group	Test it by
Looks alike at baseline	A balance table
Shared the same prior trend	Pre-period data
Faces the same outside forces	Same region, same period
Differs only in the programme	Ruling out other schemes

The last row is checked least and matters often: a comparison village receiving a different scheme is not a counterfactual for yours.

Ask the evaluator directly what else was happening in the comparison area during the study period. It is a question with a knowable answer.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Two Estimands

ATE and ATT: effect on whom?

ATE

Average Treatment Effect — the effect if everyone in the population were treated.

ATT

Average Treatment effect on the Treated — the effect among those who actually received the programme. Often the policy-relevant one.

They differ when the programme works differently for participants than for everyone. Always ask which one a study reports — and which one your decision needs.

Estimand	Answers	Relevant when
ATE	Effect if everyone were treated	Considering universal rollout
ATT	Effect on those actually treated	Assessing the current programme
LATE	Effect on those an instrument moved	IV designs

The three can differ substantially, and the policy question determines which one you want — a scale-up decision needs the ATE, not the ATT.

Regression discontinuity estimates an effect local to the cutoff, which is a further variant: valid for units near the threshold and not for everyone eligible.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Logic in One Line

Every method, one promise

Strip away the jargon and every design in this course makes the same promise: my comparison group is a credible stand-in for the treatment group's counterfactual. Methods differ only in how they make that promise believable.

RCT: a lottery makes the groups alike

→

RD: the cutoff makes near-neighbours alike

→

DiD: a shared trend makes changes comparable

→

Matching: observed traits make pairs alike

Judge any study by asking: how believable is its version of that promise — and what would break it?

Design	The promise it makes
RCT	Chance made the groups comparable
RD	Units either side of the cutoff are alike
DiD	The trends would have been parallel
Matching	Observed characteristics capture the difference
IV	The instrument affects the outcome only through take-up

Every row is an assumption you can name, examine and often test partially. That is the practical skill this course is building.

Ask any evaluator to state their assumption in one sentence and say what evidence supports it. A design whose assumption cannot be stated is not identified.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Three

Theory of Change & Evaluation Questions

ImpactMojoImpact Evaluation 101www.impactmojo.in

Start Here

No theory of change, no evaluation

Theory of change

An explicit map of how and why a programme's activities are expected to lead to its intended outcomes — the chain of cause and effect, plus the assumptions at each link.

Before measuring impact, you must state what impact you expect and through what pathway. A vague programme cannot be rigorously evaluated.

Without a theory of change you cannot	Because
Choose outcomes	You do not know what should move
Interpret a null	You cannot say which link failed
Time the measurement	You do not know when to expect it
Test mechanisms	There are none stated

A null result without a theory of change is uninterpretable: it says the outcome did not move and nothing about why.

The interpretability of a null is one of the strongest practical arguments for spending time on the theory before commissioning anything.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Causal Chain

Map the pathway, link by link

Cash transfer to mothers

→

more household income

→

more spent on food & care

→

better child nutrition

→

lower stunting

Each arrow is an assumption that can fail. If transfers are spent elsewhere, or food prices rise, the chain breaks — and that is exactly what evaluation should test.

Link in the chain	The assumption
Transfer to income	It arrives, and is retained
Income to spending on food	It is spent that way
Spending to nutrition	The food improves diet quality
Nutrition to stunting	The window is right; enough time passes

Each arrow can fail independently, and a null result at the end tells you nothing about which one did unless you measured along the way.

Measuring intermediate outcomes is what turns an evaluation into a diagnosis rather than a verdict, and it costs little once you are collecting data anyway.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Assumptions

The arrows are where programmes fail

A theory of change makes assumptions visible so you can check them. Most programmes that 'fail' did not fail because the idea was wrong — they failed at a specific, checkable link.

Implementation failure

The chain was never delivered — training didn't happen, transfers didn't arrive. The theory is untested.

Theory failure

Everything was delivered, but an assumed link did not hold — the idea itself was wrong.

Impact evaluation paired with monitoring lets you tell these two apart — a null result means little if the programme never actually reached people.

Failure type	What it means	What to do
Implementation failure	The chain broke in delivery	Fix delivery; the theory is untested
Theory failure	Delivery worked; the link did not	Change the design

Distinguishing these two is what a process evaluation alongside the impact evaluation is for, and it is the difference between abandoning a good idea and persisting with a bad one.

Most programmes that fail an impact evaluation failed at implementation, which means the evaluation has told you less about the idea than it appears.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Framing

Turn the theory into evaluation questions

Impact question: did the programme change the outcome, by how much, for whom?
Mechanism question: through which link in the chain?
Process question: was it delivered as designed?
Cost question: was the impact worth what it cost?

Only the first is strictly an impact question — but a good evaluation usually answers several, because 'did it work?' is rarely enough to act on.

Question type	Method that answers it
Did it work, and by how much?	Impact evaluation
Through which link?	Mechanism analysis; qualitative
Was it delivered as designed?	Process evaluation
Was it worth the money?	Cost-effectiveness
For whom did it work?	Heterogeneity analysis

Commissioning only the first leaves you with a number and no explanation, which is why the strongest evaluations bundle at least the first three.

The cost row is the cheapest addition: costing data usually exists already and turns an effect size into a comparable figure.

ImpactMojoImpact Evaluation 101www.impactmojo.in

A Good Question

What a sharp evaluation question looks like

A usable impact question names the intervention, the population, the outcome, the comparison and the timeframe.

Weak: 'Does our programme help?' Strong: 'Does 18 months of monthly cash transfers to mothers of under-2s in Koraput reduce child stunting, relative to comparable villages without transfers?'

A sharp question names	Example
The intervention	18 months of monthly cash transfers
The population	Mothers of children under two
The outcome	Height-for-age z-score
The comparison	Versus no transfer
The timeframe	Measured at 24 months

Writing the question this way forces most of the design decisions into the open before anyone is commissioned, which is where they belong.

A question that cannot be written in this form is usually not yet a question. "Does our programme help?" fails on all five counts.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Scope

Choose the right question, not every question

You cannot evaluate everything. A focused evaluation that answers one decision well beats a sprawling one that answers many vaguely. Let the decision you face dictate the question.

Ask up front: what would we do differently if the answer were yes, versus no? If nothing changes either way, the evaluation may not be worth running.

Ask before scoping	To find
What decision does this inform?	Whether it is worth doing
Who will act on the answer?	Whether it will be used
By when do they need it?	The feasible design
What answer would change the decision?	The MDE

The last question is the most useful and the least asked: it converts a vague interest in impact into a specific minimum detectable effect.

If no answer would change the decision, the evaluation is a documentation exercise and should be described as one.

ImpactMojoImpact Evaluation 101www.impactmojo.in

When NOT to do it

When an impact evaluation is the wrong call

The programme is still changing weekly — there is nothing stable to evaluate
The intervention is already proven and the question is purely operational
No credible comparison group is possible (e.g. a universal, simultaneous rollout)
The budget and timeline cannot support a rigorous design — a weak IE is worse than none
The decision is already made and evidence will not change it

A badly underpowered or poorly identified impact evaluation can mislead more than no evaluation at all. Knowing when to walk away is a skill.

Do not commission when	Instead
The programme changes weekly	Stabilise first
The question is operational	Process evaluation
No comparison group is possible	Contribution analysis
The result cannot arrive in time	Rapid learning methods
Nobody will act on it	Do not do it

Each row has a legitimate alternative, so declining an impact evaluation is not declining to learn — it is choosing the method that fits the question.

The third row is common in policy and advocacy work, where universal coverage makes a comparison group impossible by construction.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Evaluability

Run an evaluability check first

An evaluability assessment asks, before you commit: is this programme well-defined, stable and measurable enough to evaluate credibly — and will anyone act on the result?

Clear theory of change?

→

Measurable outcomes?

→

A feasible counterfactual?

→

A decision waiting on the answer?

Evaluability check	Fails if
Clear theory of change	Nobody can state the causal chain
Measurable outcomes	The outcome is undefined
A credible comparison	Everyone is treated
Stable implementation	The design keeps changing
Someone will use it	The decision is already made

An evaluability assessment costs a few days and regularly saves an entire evaluation budget by identifying that the study cannot answer the question.

Do it before writing the terms of reference. Discovering the problem after an evaluator is contracted is expensive and awkward.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Four

Randomised Experiments (RCTs)

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Gold Standard

Randomisation: let the lottery build the counterfactual

Randomised controlled trial (RCT)

A study in which eligible units are assigned to treatment or control by chance, so the two groups are comparable in expectation and the difference in their outcomes estimates the causal effect.

If a coin flip decides who gets the programme, the treatment and control groups have no systematic reason to differ — so any later difference in outcomes can be attributed to the programme.

Randomisation gives you	Provided
Comparable groups in expectation	Enough units
Balance on unobservables	The randomisation is genuine
A clean causal claim	Compliance and no spillover
A simple estimator	Attrition is low and balanced

The right column is where field RCTs actually fail. Randomisation is easy to specify and hard to protect through a real implementation.

"Comparable in expectation" is doing work: in any single trial the groups can differ by chance, which is why the balance table is checked.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Why It Works

Chance balances what you can and cannot see

This is the deep power of randomisation: with enough units, it balances the two groups on observed characteristics (age, caste, income) and on unobserved ones (motivation, ability, family support) — in expectation.

No other method can claim to balance the unobservable factors. That is why a well-run RCT has the strongest claim to internal validity.

Balanced by randomisation	Not balanced by any other design
Age, income, education	Also balanced by matching
Motivation	Unobserved
Ability and family support	Unobserved
Everything you did not think of	Unobserved

The bottom three rows are the unique claim of randomisation, and no observational design can make it. That is the whole basis of the credibility ranking.

It holds in expectation and over repeated trials rather than in any one sample, which is why sample size and balance checks still matter.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Check Balance

Show the groups started alike

Illustrative baseline balance: treatment vs control

Illustrative example, not real data

A balance table like this is the first thing to check in any RCT: if randomisation worked, the groups look alike at baseline on every measured characteristic.

In a balance table, check	A warning sign
Means on key characteristics	Large differences
Baseline outcome levels	Imbalance on the outcome itself

Imbalance across many variables is expected; imbalance on the baseline outcome is what should give pause.

A paper that does not report a balance table on an RCT is withholding the first thing a reader should see.

ImpactMojoImpact Evaluation 101www.impactmojo.in

J-PAL

Who champions the RCT in development

The Abdul Latif Jameel Poverty Action Lab (J-PAL), based at MIT, has run hundreds of randomised evaluations of anti-poverty programmes worldwide, including across South Asia. Its founders, Abhijit Banerjee and Esther Duflo, shared the 2019 Nobel Prize in Economics (with Michael Kremer) for this experimental approach.

Its sister network 3ie (the International Initiative for Impact Evaluation) funds and synthesises impact evaluations across the Global South.

J-PAL	Detail
Based at	MIT, with regional offices including South Asia
Method	Randomised evaluations of anti-poverty programmes
Founders	Banerjee, Duflo, and colleagues
Recognition	The 2019 Nobel Memorial Prize, shared with Kremer

The prize citation was for the experimental approach to alleviating global poverty, which marks how far the method moved from the margins in two decades.

J-PAL’s evidence reviews and cost-effectiveness comparisons are freely available and are a useful first stop before commissioning anything.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Growth

How randomised evaluations took off

Illustrative: the rise of randomised evaluations in development

Illustrative trend, schematic only

Schematic, not a precise count — but the direction is real: randomised impact evaluation went from rare to mainstream in development over two decades.

The rise brought	And a critique
Credible causal estimates	Narrow questions, chosen for tractability
A generalisable method	Weak external validity

The critique is substantial and comes from within economics as much as outside it, principally that what can be randomised is not always what matters most.

Both readings can be held: the method is the strongest available for the questions it fits, and it does not fit every question worth asking.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Unit of Randomisation

Randomise individuals, or whole clusters?

Individual

Each person assigned separately. Most statistically efficient, but risky when treated and control people mix and influence each other.

Cluster

Whole villages, schools or clinics assigned together. Needed when the programme operates at group level or spillovers are likely — but you need many clusters.

Cluster designs cost statistical power: 40 villages of 50 people is far weaker than 2,000 individually randomised people. Power is driven by the number of clusters, not just total people.

	Individual randomisation	Cluster randomisation
Efficiency	Higher	Lower for the same n
Spillover risk	High if they interact	Lower
Sample needed	Smaller	Larger
Fits	Isolated individual treatments	Village or school-level programmes

The choice is driven by the intervention: a village-level water scheme cannot be randomised within a village, and a training might be.

Cluster designs cost power, and the cost depends on how similar people are within clusters, which is what the intra-cluster correlation measures.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Spillovers

When treatment leaks to the control group

If treated and control units interact, the programme can spill over — a dewormed child stops infecting an untreated neighbour. The control group is no longer a clean counterfactual, and the effect is understated.

Cluster randomisation — treating whole villages — is the usual defence, keeping treated and control units far enough apart that they don't contaminate each other.

Spillover type	Effect on the estimate
Positive — control benefits too	Understates the impact
Negative — control loses out	Overstates it
Learning across the boundary	Understates
Migration between groups	Contaminates both

The deworming literature is the standard example of positive spillover: an untreated child benefits from treated neighbours, so the within-village comparison understates the effect.

Designs handle it by randomising at a higher level, by leaving buffer areas, or by measuring spillovers deliberately as a separate estimate.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Ethics

Equipoise: the ethics of the lottery

Equipoise

Genuine uncertainty about whether the programme works. When we honestly do not know, randomly choosing who gets it first is defensible — and the evaluation resolves the uncertainty for everyone.

Random phase-in: everyone gets it eventually; the lottery just sets the order
Randomise only when demand exceeds supply — a lottery is already fair
Never withhold a known, effective, life-saving intervention to run a trial

Ethically defensible when	Not when
Genuine uncertainty exists	The programme is known to work
Resources cannot cover everyone	Denial is arbitrary and avoidable
Randomised phase-in is used	Some are permanently excluded
A committee has reviewed it	Nobody independent has

Randomised phase-in is the design that resolves most of the ethical objection: everyone receives the programme, and the order is randomised.

Equipoise is the condition. Where the intervention is already established, withholding it for a study is not defensible on any framing.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Five

Quasi-Experimental Designs

ImpactMojoImpact Evaluation 101www.impactmojo.in

When You Can't Randomise

Finding a counterfactual in the real world

Often you cannot randomise — the programme is already running, or rolled out to everyone, or randomisation is infeasible. Quasi-experimental designs exploit natural variation to approximate a control group.

They can be highly credible — but each rests on an assumption you must argue for, not just an untouchable coin flip. Know the assumption behind every design.

You cannot randomise when	Quasi-experimental option
The programme already ran	DiD, if you have pre-period data
Eligibility uses a cutoff	Regression discontinuity
Rollout was phased	DiD or staggered adoption designs
Take-up was nudged by something external	Instrumental variables

Each design exploits a specific feature of how the programme was allocated, which is why the first question is always how people came to be treated.

If allocation was purely discretionary and undocumented, none of these applies, and the honest answer is that the effect cannot be identified.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Diff-in-Diff

Difference-in-differences: subtract the trend

Difference-in-differences (DiD) compares the change in a treated group with the change in a comparison group over the same period. Subtracting the comparison group's change removes whatever trend would have happened anyway.

Treated: before → after

→

Comparison: before → after

→

Impact = (treated change) − (comparison change)

DiD removes	It does not remove
Fixed differences between groups	Differential trends
Common time shocks	Shocks hitting one group only
Level differences at baseline	Anything changing differently

The estimator is a double subtraction: the change in the treated group minus the change in the comparison group over the same period.

That is a strong design when the parallel-trends assumption holds and a weak one when it does not, which is the subject of the next slide.

ImpactMojoImpact Evaluation 101www.impactmojo.in

See It

DiD: the gap that opens up

Illustrative DiD: outcome before and after, treated vs comparison

Illustrative example, not real data

The grey dashed line is the assumed counterfactual: where the treated group would have gone, tracking the comparison group's trend. The estimated impact is the gap above it (~14 points).

Read the DiD plot for	Which tells you
Pre-period lines	Whether trends were parallel
The gap after treatment	The estimated effect

The dashed counterfactual is an assumption drawn as if it were data.

Several pre-periods are much stronger evidence than one. A single pre-period cannot show a trend at all.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Key Assumption

DiD identifies only under parallel trends

Parallel trends

The assumption that, absent the programme, the treated and comparison groups would have moved in parallel — the same trend over time. DiD is only valid if this holds.

If the groups were already diverging before the programme, DiD attributes that pre-existing divergence to the programme — and overstates impact. Always check trends in the pre-period first.

Support parallel trends by	Which is
Plotting several pre-periods	The standard evidence
Testing for pre-trends	Necessary, not sufficient
Choosing a similar comparison	Judgement
Event-study specification	Showing the timing

Parallel trends is an assumption about the unobserved counterfactual period and therefore cannot be tested directly. Pre-period parallelism is supporting evidence, not proof.

Ask for the pre-trend plot. If an evaluation reports a DiD estimate without one, the key assumption has not been examined.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Regression Discontinuity

RD: compare just either side of a cutoff

Many programmes use a sharp eligibility cutoff — a poverty score below 0.33, a test mark above 60. People just above and just below the line are nearly identical, except one gets the programme. That tiny band is a natural experiment.

Regression discontinuity (RD)

A design that estimates impact from the jump in outcomes exactly at an eligibility threshold, comparing units narrowly above and below the cutoff.

RD requires	And gives you
A sharp, enforced cutoff	A credible local comparison
No manipulation of the score	Units alike either side
Enough observations near it	Precision
A continuous running variable	A local effect only

The local nature is the main limitation: an RD estimates the effect for units near the threshold, which may not be the effect for those far from it.

Manipulation is the main threat. If people can adjust their score to fall on the favourable side, the units either side are no longer comparable.

ImpactMojoImpact Evaluation 101www.impactmojo.in

See It

RD: read the jump at the cutoff

Illustrative RD: outcome jumps at the eligibility threshold

Illustrative example, not real data

The vertical jump at the cutoff (~14 points) is the estimated effect. Crucially, RD gives a local effect — valid for units near the threshold, not for everyone.

RD gives	And does not give
A credible local effect	The effect far from the cutoff
A visual, checkable design	Precision without enough observations
A defence against selection	Protection if the score is manipulated

The locality is not a defect but a limitation to state: an RD on a poverty-score cutoff estimates the effect for households near that line.

For a scale-up decision covering everyone eligible, that is not the estimate you need, and saying so is part of reading it honestly.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Matching / PSM

Matching: build a statistical twin

Matching pairs each treated unit with one or more untreated units that look alike on observed characteristics. Propensity-score matching (PSM) matches on a single number: the estimated probability of being treated, given those characteristics.

The fatal limit: matching can only balance what you measured. If treated and untreated differ on something unobserved — motivation, hidden need — matching cannot fix it. Weaker than an RCT.

Matching assumes	Which fails when
Selection is on observables only	Motivation drives take-up
Common support exists	Treated units have no comparable untreated ones
The matching variables are the right ones	Something unmeasured matters

This is the weakest assumption in the toolkit because it is the one that cannot be examined: the whole concern is about variables you did not measure.

Matching improves a naive comparison and does not resolve selection bias. Describing a PSM estimate as causal without heavy caveats overstates it.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Instrumental Variables

IV: borrow some random-like variation

An instrumental variable is an outside factor that nudges people into the programme but has no other path to the outcome. It isolates the slice of programme take-up that behaves as if random — e.g. distance to a facility shifting who enrols.

A valid instrument needs two things: relevance (it genuinely shifts take-up) and the exclusion restriction (it affects the outcome only through take-up). The second can never be fully proven — only argued.

A valid instrument must	Which is
Predict take-up	Testable
Affect the outcome only through take-up	Untestable, and the crux
Be as good as random	Argued, not proven

The exclusion restriction — the second row — is an assumption defended by argument, and it is where most IV papers are challenged.

Distance to a facility is a common instrument and a contested one: distance may affect outcomes through other paths, such as access to markets.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Compare

The quasi-experimental toolkit at a glance

Design	Exploits	Key assumption
Diff-in-differences	Before/after × treated/comparison	Parallel trends
Regression discontinuity	A sharp eligibility cutoff	Units similar across the cutoff
Matching / PSM	Observed similarity	No unobserved confounders
Instrumental variables	An external nudge into take-up	Relevance + exclusion restriction

Each is only as credible as its assumption. A commissioner's job is to ask: what must be true for this to identify the effect — and is it?

Design	Assumption	How to probe it
DiD	Parallel trends	Pre-period plots
RD	No manipulation; smoothness	Density test at the cutoff
Matching	Selection on observables	Cannot be tested
IV	Exclusion restriction	Argument; falsification tests

Three of the four have at least a partial diagnostic, and the one without any is the one most often used because it needs no special allocation feature.

Learn the diagnostic for each. Asking for it is how a non-specialist commissioner exerts real quality control over an evaluation.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Six

Choosing a Method

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Trade-Off

There is no single best method

The 'best' design is the most credible one that is feasible and ethical for your situation. Method choice is a negotiation between three forces, not a ranking to memorise.

Validity

How credible is the causal claim?

Feasibility

Can you actually do it, in time and budget?

Ethics

Is it fair to those involved?

Force	Pulls toward
Validity	Randomisation
Feasibility	Whatever the rollout allows
Ethics	Not withholding a known benefit
Timeliness	Faster, simpler designs

The best design is the most credible one that is feasible and ethical here, which is a negotiation rather than a ranking to apply.

Stating the trade-off you made, in the report, is a mark of a serious evaluation. Pretending there was none is not.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Internal Validity

Internal validity: is the answer right here?

Internal validity

The degree to which a study correctly identifies the causal effect for the people and place it studied — free of selection bias and confounding.

A well-run RCT typically has the strongest internal validity; a before-after comparison the weakest. Internal validity is the first hurdle: an answer that is wrong here is useless everywhere.

Internal validity is threatened by	Which design handles it
Selection bias	Randomisation
Confounding trends	DiD
Reverse causation	Timing and design
Differential attrition	No design — only good execution

The last row is the important one: attrition arises after assignment, so no allocation method prevents it and only fieldwork quality controls it.

A well-run RCT typically has the strongest internal validity available, and that is a statement about the design rather than a guarantee about the study.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Credibility Ladder

Designs ranked by credibility

Design	Counterfactual quality	Credibility
Randomised controlled trial	Strongest — balances unobservables	Highest
Regression discontinuity	Strong, but local to the cutoff	High
Difference-in-differences	Good if parallel trends hold	Medium-high
Instrumental variables	Depends on a defensible instrument	Conditional
Matching / PSM	Only as good as observed variables	Medium
Before-after / naive comparison	No real counterfactual	Lowest

Climb as high as feasibility and ethics allow — but a credible quasi-experiment beats a badly run RCT. Execution matters as much as the rung.

Design	Counterfactual quality	Main vulnerability
RCT	Strongest	Execution: attrition, spillover, compliance
RD	Strong, local	Manipulation; local only
DiD	Good if trends parallel	Differential trends
IV	Depends on the instrument	Exclusion restriction
Matching	Weakest	Unobservables

The ranking is about the credibility of the identifying assumption rather than about the sophistication of the statistics.

Read the third column alongside the second. A well-executed DiD can be more trustworthy than a badly executed RCT, which is the next slide.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Feasibility

What makes a design practical

Is there a stage of rollout where you can still assign or compare?
Are baseline data and a comparison group available?
Do you have the sample size, budget and time the design needs?
Will the result arrive in time to inform the decision?

The single biggest feasibility lever is timing: the best designs are built in before a programme starts, not retrofitted afterwards.

Feasibility question	If the answer is no
Is there still a stage to assign or compare?	The design space has closed
Do baseline data exist?	Power costs rise
Is the sample large enough?	Reconsider the MDE
Will the result arrive in time?	Choose a faster method

The first question is decided by when you were consulted, which is why the commissioning advice in Section 11 keeps returning to timing.

Answer all four before writing terms of reference. Two of them can invalidate the whole exercise and both are knowable in advance.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Design In Early

Plan the evaluation before the programme rolls out

A staggered or phased rollout — which most large programmes need anyway — is a gift to evaluators. It creates a natural comparison group (those not yet reached) at no extra cost.

Lesson for commissioners: involve the evaluator at the design stage. Retrofitting rigour onto a finished programme is the hardest and least credible path.

Phased rollout gives you	At no extra cost
A comparison group	Those not yet reached
A defensible allocation rule	Order can be randomised
An ethical answer	Everyone eventually receives it
Baseline opportunity	Before the later phases

Most large programmes phase their rollout for operational reasons anyway, which makes this the single most available opportunity for credible evaluation in practice.

It requires only that someone is asked before the phasing order is fixed. After that, the opportunity is gone.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Mixed Methods

Quantitative tells you whether; qualitative tells you why

Quantitative IE

Estimates the size of the effect and how sure we are. Answers did it work, and by how much?

Qualitative work

Explains the mechanism, the surprises, the implementation. Answers why, how, and for whom?

The strongest evaluations combine both: a number you can trust, and a story that explains it. Neither alone is enough to act.

Quantitative answers	Qualitative answers
Did it work?	Why or why not
By how much?	For whom, and how
How confident are we?	What surprised us
On average	What the average conceals

A null result with qualitative work alongside can tell you which link broke. Without it, a null result tells you only that the outcome did not move.

Collect the qualitative material before the quantitative result is known, or the accounts become explanations for a conclusion already in hand.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Decision Guide

A rough rule for choosing

Can you assign at random, ethically? → RCT

→

Is there a sharp eligibility cutoff? → RD

→

Phased rollout with baseline data? → DiD

→

Only an external nudge into take-up? → IV

→

None of the above? → reconsider whether to run an IE

If	Then
You can assign randomly, ethically	RCT
There is a sharp eligibility cutoff	RD
Phased rollout and baseline data exist	DiD
Only an external nudge to take-up	IV
None of these	Reconsider the question

The last row is a real option and is regularly the right answer. Not every question can be answered credibly with the allocation that occurred.

Where none applies, contribution analysis and process tracing give a reasoned, evidenced causal claim without a counterfactual estimate. Say which you are doing.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Execution Beats Elegance

A well-run simple design beats a botched fancy one

The credibility ladder ranks designs, but in the field, execution decides everything. An RCT wrecked by attrition, spillovers and broken randomisation can be less trustworthy than a careful difference-in-differences.

So weigh the design and the team, the timeline and the field conditions together. Ambition you cannot execute is worse than modesty you can.

Execution problem	Effect
High attrition	Groups no longer comparable
Differential attrition	Bias, in an unknown direction
Spillovers	Estimate biased toward zero, usually
Non-compliance	ITT and ToT diverge
Broken randomisation	It is now an observational study

Each of these can turn a well-designed RCT into a study weaker than a careful quasi-experiment, which is why execution is reported as carefully as design.

Ask for the attrition rate by group before reading anything else. It is the fastest single check on whether the design survived contact with the field.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Seven

Sampling & Statistical Power

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Stakes

Why an underpowered evaluation misleads

If your sample is too small, even a real, useful effect can fail to reach statistical significance. You then conclude 'no impact' — when the truth is you simply could not detect it. This is the curse of the underpowered study.

An underpowered evaluation wastes money and can kill a programme that actually works. Power must be planned before data collection — it cannot be fixed afterwards.

Underpowered study produces	Which is read as
A wide confidence interval	"No significant effect"
An imprecise estimate	"The programme did not work"
A failure to detect	A finding of no effect

The distinction between a precise zero and an imprecise nothing is the single most misread thing in evaluation reporting.

Report the confidence interval, always. It is what lets a reader tell the two apart, and a point estimate alone cannot.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Definition

Statistical power, defined

Statistical power

The probability that a study will detect a real effect of a given size, if one truly exists. By convention, evaluations aim for 80% power or more.

Power of 80% means that if the programme really has the effect you assumed, you have an 80% chance of finding a statistically significant result — and a 20% chance of missing it.

Power of 80% means	And implies
A real effect is detected 4 times in 5	It is missed once in 5
Conditional on the assumed effect size	A smaller true effect is missed more often
A convention, not a law	Higher power costs sample

The 80% convention is exactly that: a norm rather than a scientific threshold, and for a high-stakes decision higher power may be worth the cost.

Power is calculated for an assumed effect size. If the true effect is smaller than assumed, the study is less powered than it says.

ImpactMojoImpact Evaluation 101www.impactmojo.in

MDE

Minimum detectable effect: what can you even see?

Minimum detectable effect (MDE)

The smallest true effect a study is reliably able to detect, given its sample size, the outcome's variability and the desired power.

Flip the question round: with this sample, what is the smallest impact I could detect? If your MDE is a 10-point gain but the programme can realistically deliver 3, the study is doomed before it starts.

MDE reframes the question	From	To
Sample	How many do we need?	What can we see with what we have?
Decision	Is it significant?	Is the MDE smaller than a useful effect?

This is the most useful concept for a commissioner. If your MDE is larger than any effect the programme could plausibly produce, the study cannot succeed.

Ask for the MDE in the units of the outcome and compare it with what would matter. A study with an MDE of ten percentage points cannot detect a realistic five.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Drivers

What determines how big a sample you need

Factor	Effect on required sample
Smaller expected effect	Much larger sample needed
Higher outcome variability	Larger sample needed
Higher power target (e.g. 90%)	Larger sample needed
Clustered design	Larger sample — driven by number of clusters
A good baseline to control for	Smaller sample needed

The headline driver is effect size: detecting small effects is expensive. Be honest about how big an impact is plausible.

Factor	Direction
Smaller expected effect	Much larger sample
Higher outcome variability	Larger sample
Higher power target	Larger sample
Clustering	Larger sample
Expected attrition	Larger sample
A baseline measurement	Smaller sample needed

The first row is non-linear and often surprising: halving the detectable effect roughly quadruples the sample required.

The last row is the one that saves money. A baseline reduces the variance to be explained and can substantially cut the sample needed.

ImpactMojoImpact Evaluation 101www.impactmojo.in

See It

How power rises with sample size

Illustrative power curve: power vs sample size

Illustrative, schematic shape only

Power climbs steeply at first, then flattens as it approaches 1. The dashed convention of 80% (~1,000 per arm here) is where most evaluations aim — beyond it, extra sample buys little.

Power against sample size	Behaves
At small n	Rises steeply
Approaching 80%	Still rising usefully

The curvature is the practical point: doubling a small sample buys a great deal, and doubling a large one buys little.

Where budget is constrained, the better lever is usually reducing outcome variance — a baseline, a better instrument — rather than more respondents.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Clustering

Why clusters cost you power

When you randomise whole villages, people within a village resemble each other — they share the same school, water, weather. These correlated responses carry less independent information than the same number of unrelated individuals.

So power is driven by the number of clusters, not the total headcount. Forty large villages can be weaker than a hundred small ones. Add clusters, not just people.

Clustering costs power because	Measured by
People within a cluster are alike	Intra-cluster correlation
Each adds less new information	The design effect
The effective sample is smaller	n divided by the design effect

For village-level outcomes the intra-cluster correlation can be substantial, which means a cluster-randomised study needs many more people than an individual one.

More clusters generally buys more power than more people per cluster, which is the practical rule when designing within a fixed budget.

ImpactMojoImpact Evaluation 101www.impactmojo.in

In Practice

Run a power calculation before you commit

State the smallest effect worth detecting (your MDE)
Estimate outcome variability from prior data or a pilot
Account for clustering, attrition and partial take-up
Solve for the sample size that gives at least 80% power

Tools like J-PAL's resources and free software make this routine. Insist on seeing the power calculation in any evaluation proposal — a design without one is a red flag.

Power calculation step	What people skip
State the MDE	Assuming an implausibly large effect
Estimate variability	Using no prior data
Account for clustering	Ignoring it entirely
Allow for attrition	Planning to the minimum
Allow for partial take-up	ITT dilution

The commonest failure is assuming an effect size larger than any comparable programme achieved, which makes the calculation produce a comfortable and useless sample.

Ask what effect size was assumed and where it came from. A calculation resting on an optimistic assumption is not a safeguard.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Two Errors

False positives and false negatives

Type I (false positive)

Concluding the programme worked when it didn't. Controlled by the significance level — conventionally a 5% risk.

Type II (false negative)

Missing a real effect. Its risk is 1 − power — the very thing a good sample size guards against.

Underpowered studies are dominated by Type II error: they scream 'no effect' when they simply could not see one. Power is how you buy down that risk.

	Type I	Type II
Error	Claiming an effect that is not there	Missing one that is
Controlled by	Significance level, usually 5%	Power, usually 80%
Cost	Scaling a failure	Abandoning something that works
Attention received	A great deal	Very little

The asymmetry in the last row is a real problem in this field: the false-negative risk is set four times higher than the false-positive risk by convention and is rarely discussed.

For a decision where abandoning a working programme is costly, that conventional balance may be the wrong one.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Eight

Measurement & Data Collection

ImpactMojoImpact Evaluation 101www.impactmojo.in

Outcomes First

Decide what to measure before how

The outcome indicators flow straight from the theory of change. Choose them before the evaluation starts, define them precisely, and commit to them — so you cannot fish for whichever outcome happens to look good afterwards.

Distinguish primary outcomes (the one or two you will judge success on) from secondary ones. Pre-specifying the primary outcome is a core discipline of credible IE.

Decide before the study	Because afterwards
The primary outcome	You can pick the one that moved
How it is defined	Definitions can be adjusted
The analysis specification	Specifications can be searched
Subgroups of interest	Subgroups can be mined

Committing in advance is what distinguishes a test from a search, and pre-registration is the mechanism that makes the commitment visible.

It also protects the evaluator. A pre-specified null is defensible; a null after twenty specifications invites the question of what else was tried.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Good Indicators

What makes an outcome indicator usable

Valid

Captures the concept you actually care about

Reliable

Gives the same reading on repeat measurement

Sensitive

Moves when the real outcome moves

Feasible

Can be measured affordably in the field

Property	Failing indicator
Valid	Measures something adjacent
Reliable	Different reading on retest
Sensitive	Cannot move within the study period
Feasible	Requires a survey you cannot fund

Sensitivity is the one that most often defeats an evaluation: an outcome that cannot plausibly move in eighteen months guarantees a null.

Check that the primary outcome could move by more than the MDE within the study period. If it could not, redesign before collecting anything.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Baseline / Endline

Measure before, measure after

BASELINE: measure outcomes before the programme

→

RANDOMISE / ASSIGN: form treatment & comparison

→

DELIVER: run the programme

→

ENDLINE: re-measure the same outcomes

A baseline does double duty: it confirms the groups started balanced, and it sharpens power by letting you control for starting levels. Skip it only if you must.

A baseline gives you	Which is worth
Balance verification	Confidence in the design
Variance reduction	A smaller required sample
Heterogeneity by baseline status	Who it worked for
A fallback if randomisation fails	A DiD option

The last row is undervalued: a baseline turns a compromised RCT into something still analysable, which is real insurance for a modest cost.

Where budget is tight, a baseline on a subsample can deliver much of the variance reduction at a fraction of the cost.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Data Sources

Surveys and administrative data

Survey data

Collected for the evaluation — exactly the outcomes you need, but costly, and prone to recall and reporting error.

Administrative data

Already generated by the system — school registers, HMIS, MGNREGA records. Cheap and continuous, but may not measure quite what you want.

Linking your evaluation to existing administrative systems can slash cost and enable long-term follow-up — where the data quality is good enough to trust.

	Survey data	Administrative data
Fit to the question	Exact	Approximate
Cost	High	Low
Coverage	Your sample	Everyone in the system
Error type	Recall and reporting	System errors, definitions
Risk	Courtesy bias	Differential recording by treatment

The last row is the specific danger with administrative data in evaluation: if the programme changes how records are kept, the measure moves without the outcome moving.

Where both are available, using them together is the strongest option, and disagreement between them is itself informative.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Measurement Bias

How measurement quietly corrupts a finding

Social desirability: people report what they think you want to hear
Recall error: hazy memory of past income, illness, spending
Surveyor effects: who asks, and how, shifts the answer
Differential measurement: measuring treatment and control groups differently

The last is the most dangerous: if treated respondents are surveyed more enthusiastically than controls, you manufacture an effect out of thin air. Measure both groups identically.

Bias	Guard
Social desirability	Neutral wording; private setting
Recall error	Short reference periods; anchors
Surveyor effects	Training; rotation; matched characteristics
Differential measurement	Blind enumerators to treatment status

The last row is the one that biases an impact estimate specifically: if enumerators know who was treated, expectation can shape what is recorded.

Blinding is often impossible in the field and worth attempting where it is not, particularly for observed or scored outcomes.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Good Instruments

Build the survey to avoid bias

Pilot every instrument before the real round — without exception
Use neutral wording; avoid leading and double-barrelled questions
Blind enumerators to treatment status where you possibly can
Use the same instrument, timing and team for both groups

Where feasible, prefer objective measures (test scores, anthropometry, biomarkers, admin records) over self-report — they are harder to bias.

Instrument discipline	Why
Pilot every instrument	Comprehension failures are invisible otherwise
Neutral wording	Leading questions produce the expected answer
Blind enumerators where possible	Expectation shapes recording
Same instrument at both rounds	A changed tool destroys the comparison

The last row is violated routinely and with good intentions: the instrument is improved between baseline and endline, and the comparison is lost.

If the tool must change, run both at the changeover round. Otherwise the change in the measure and the change in the world cannot be separated.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Attrition

When people drop out of the study

Attrition

The loss of study participants between baseline and endline — through migration, refusal or death — who cannot be measured at follow-up.

Attrition is dangerous when it differs between groups, or relates to the outcome — if the worst-off in the treatment group leave, the survivors look artificially good. Track it, report it, and test whether it is balanced.

Attrition is dangerous when	Check
It is high overall	The rate
It differs between groups	The rate by arm
It relates to the outcome	Baseline characteristics of leavers
It is unreported	Ask

Differential attrition destroys comparability just as selection bias does, and it arises after randomisation, so no design protects against it.

Bounding exercises — assuming the worst and best cases for those lost — are the standard response, and their width tells you how much attrition mattered.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Timing

When you measure shapes what you find

Measure too early and the effect has not yet appeared — a nutrition programme needs months to move stunting. Measure too late and a real effect may have faded, or comparison villages may have caught up.

Let the theory of change set the clock: when should this outcome plausibly respond? Endline timing is a design choice, not an afterthought — and a second follow-up tells you whether effects last.

Measure too early	Measure too late
The effect has not appeared	It may have faded
A false null	Comparison areas may have caught up
Nutrition, learning, income all lag	Programmes end; effects decay

The theory of change should dictate the timing, and a programme claiming an effect on stunting measured at six months has not allowed the mechanism to operate.

Where budget allows, two follow-ups are far more informative than one, because they distinguish a delayed effect from a fading one.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Nine

Analysis & Interpreting Results

ImpactMojoImpact Evaluation 101www.impactmojo.in

Effect Size

How big, not just whether

The first thing to read is the effect size: how much did the outcome change? Report it in units a decision-maker understands — percentage points, rupees, days of schooling — not just a coefficient.

Then ask the practical question: is an effect of this size worth the programme's cost? A real but tiny effect may not justify the spend.

Report the effect as	Not as
Percentage points	A coefficient
Rupees per household per month	A standardised score alone
Additional days of schooling	An effect size
Children per thousand	A log point

Standardised effect sizes are useful for comparison across studies and are meaningless to a decision-maker who has to choose between two programmes.

Report both: the natural unit for the decision, and the standardised size for comparability. It costs one extra column.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Significance

Significant is not the same as important

Statistical significance

A result unlikely to have arisen by chance alone if the programme truly had no effect. It speaks to confidence, not to the size or importance of the effect.

With a huge sample, a trivially small effect can be 'statistically significant'. With a small sample, a large, real effect can miss significance. Always read the effect size and the uncertainty — never the p-value alone.

Significant but trivial	Large but uncertain
A huge sample; a tiny effect	A small sample; a big point estimate
Statistically detectable	Statistically indistinguishable from zero
Practically irrelevant	Possibly important
Reported as a success	Reported as a null

Both rows of the bottom line are misreadings, and they run in opposite directions, which is why significance alone is a poor guide to what to do.

Ask what effect would justify the cost, then look at whether the confidence interval includes it. That is the decision-relevant question.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Confidence Intervals

Report the range, not just the point

An estimated impact of '6 percentage points' is shorthand for a range — say 2 to 10 — the confidence interval. The width of that range tells you how precisely the effect was estimated.

If the interval comfortably excludes zero, the effect is reasonably firm. If it straddles zero, the evaluation cannot rule out 'no effect'. Read intervals, not just stars.

If the interval	Then
Excludes zero and includes a useful effect	Reasonable evidence of a worthwhile impact
Excludes zero, all values trivial	Real but not worth it
Includes zero, is narrow	Probably a true small effect
Includes zero, is wide	Uninformative — underpowered

These four readings are all available from the same interval and none is available from the point estimate or the p-value alone.

The bottom two rows are what distinguishes a real null from an inconclusive study, and they are routinely reported identically.

ImpactMojoImpact Evaluation 101www.impactmojo.in

ITT vs ToT

Intention-to-treat vs treatment-on-the-treated

Intention-to-treat (ITT)

The effect of being offered the programme — everyone assigned to treatment, whether or not they took it up. Reflects real-world rollout, where take-up is never 100%.

Treatment-on-treated (ToT)

The effect on those who actually took up the programme. Usually larger than ITT, since it strips out the non-participants.

Keep groups by original assignment to preserve the randomisation. For a policy that can only offer (not force) a programme, ITT is often the more honest, decision-relevant number.

	ITT	ToT
Measures	Effect of being offered	Effect of actually receiving
Reflects	Real-world rollout	The intervention itself
Size	Smaller if take-up is partial	Larger
Use for	Policy decisions on rollout	Understanding the mechanism

The ITT is usually the policy-relevant estimate because a government offering a programme will also face incomplete take-up.

A report quoting the ToT without the take-up rate is presenting the more flattering number without the context that qualifies it.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Heterogeneity

Did it work differently for different people?

An average effect can hide wide variation. A programme might help women but not men, or the poorest but not the better-off. Heterogeneity analysis looks for these differences — and is often where the most useful learning lives.

But beware: test enough subgroups and one will look 'significant' by chance. Pre-specify the subgroups you care about, and treat surprising ones as hypotheses to test next time, not conclusions.

Heterogeneity analysis	Discipline required
By sex, caste, wealth, region	Pre-specify the subgroups
By baseline level of the outcome	Beware regression to the mean
By implementation quality	Not randomised — descriptive
By anything, post hoc	This is searching, not testing

Subgroup analysis is where the most useful findings and the most spurious ones both come from, which is why the pre-specification discipline matters here most.

The sample is also smaller in every subgroup, so subgroup estimates are less precise than the main one by construction.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Null Results

A null result is not a failure

Finding no significant impact is genuine, valuable knowledge — it can stop a wasteful programme or redirect resources. But distinguish a true zero from an inconclusive one.

A real null

Well-powered study, tight interval around zero: the programme genuinely did little. Act on it.

An empty null

Underpowered, wide interval: you simply couldn't detect an effect. This tells you almost nothing.

A real null	An inconclusive study
Well powered	Underpowered
Tight interval around zero	Wide interval
Implementation verified	Delivery unknown
Take-up adequate	Almost nobody participated

A null result is genuine knowledge and can stop a wasteful programme, which makes distinguishing it from an inconclusive one consequential rather than pedantic.

Check implementation before concluding anything from a null. A programme that was not delivered has not been tested.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Honesty

Read against the threats you learned

Was the design well identified — did the key assumption hold?
Were the groups balanced at baseline?
Was attrition low and balanced across groups?
Is the primary outcome the pre-specified one, or a convenient substitute?
Is the effect size practically meaningful, not just significant?

Read against	Ask
Identification	Did the key assumption hold?
Balance	Were the groups alike at baseline?
Attrition	Low, and balanced?
Outcomes	Is this the pre-specified primary one?
Take-up	Did anyone actually receive it?

Five questions, answerable from any competent evaluation report, and they will identify most of the problems a specialist reviewer would find.

The fourth is the one that catches the most: a switched primary outcome is common and is rarely flagged in a summary.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Mind the Multiple

Test twenty outcomes and one will 'work'

Measure enough outcomes and subgroups and, by chance alone, roughly one in twenty will look 'significant' at the usual threshold. Reporting only the winners — p-hacking — manufactures findings that will not replicate.

Defences: pre-specify the primary outcome, limit the number of tests, and adjust for multiple comparisons. Treat a lone surprising result as a hypothesis for the next study, not a conclusion.

Testing many things means	Handle by
Some will appear significant by chance	Pre-specifying the primary outcome
Subgroups multiply the problem	Limiting and declaring them
Specifications can be searched	Reporting robustness
Only the winners get reported	Multiple-comparison adjustment

With twenty independent outcomes at the usual threshold, about one will appear significant even when the programme does nothing at all.

The practical safeguard for a commissioner is simple: insist on one pre-specified primary outcome and treat everything else as exploratory.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Ten

External Validity, Cost & Ethics

ImpactMojoImpact Evaluation 101www.impactmojo.in

Will It Travel?

External validity: will it work elsewhere?

External validity

The degree to which a result found in one place, time and population holds in another. A programme that worked in Bihar may not work the same way in Tamil Nadu — or at national scale.

Internal validity asks 'is the answer right here?'; external validity asks 'does it transfer there?' A perfectly identified RCT can still mislead if you scale it into a different context.

Internal validity	External validity
Is the answer right here?	Will it hold elsewhere?
Established by design	Established by argument and replication
An RCT is strongest	An RCT is not automatically strong
A property of the study	A property of the claim

The two are frequently confused, and a well-identified result from one district is not a general finding, however clean the design.

External validity is built by replication across contexts and by understanding the mechanism, not by a bigger sample in one place.

ImpactMojoImpact Evaluation 101www.impactmojo.in

What Breaks Transfer

Why results don't always generalise

Context: different markets, culture, institutions
Implementation: a research-grade pilot run far better than a government scale-up
Scale effects: general-equilibrium changes once everyone is treated
Population: the studied group differs from the target group

Don't ask only 'did it work?' Ask 'why did it work, and are those conditions present where I want to use it?' Mechanism travels better than a headline number.

Transfer breaks because of	Which you can check
Different context	Are the preconditions present?
Implementation quality	Was the pilot research-grade?
Scale effects	Does it work when everyone has it?
Different population	Who was in the original sample?

The second row is the most common and least discussed: a pilot run by a research team with intensive supervision is not the programme a government will deliver.

The third is a real economic effect: a training programme that raises wages for a few may not when everyone is trained and the labour market adjusts.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Cost-Effectiveness

Impact per rupee, not impact alone

Impact is only half the decision. Cost-effectiveness asks how much outcome you buy per rupee — letting you compare very different programmes chasing the same goal.

A smaller-impact programme that costs a tenth as much may be the better buy. Always pair the effect size with a credible cost figure — the question is impact per rupee, not impact alone.

Cost-effectiveness lets you	Requires
Compare unlike programmes	A common outcome unit
Rank options within a budget	Full cost data
Argue for scale-up	Costs at scale, not pilot costs
Defend a smaller effect	If it is far cheaper

Pilot costs systematically understate scale-up costs and sometimes overstate them; either way they are not the number for a scale-up decision.

Collect cost data during the evaluation. Reconstructing it afterwards from accounting codes is difficult and usually incomplete.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Do No Harm

Ethics: the programme and the study both

Informed consent: participants understand and may refuse, without penalty
Do no harm: the study itself must not worsen anyone's situation
Privacy: protect respondents' data, especially the marginalised
Ethics review: an independent board (IRB) approves the design

Recall equipoise: random assignment is ethical only when we genuinely do not know what works — never withhold a proven, essential intervention to run a trial.

Ethical obligation	In an evaluation specifically
Informed consent	Including the right to refuse the study, not the programme
Do no harm	The study itself, not only the programme
Privacy	Small-sample re-identification
Ethics review	Before any data collection

The first row carries a distinction worth being precise about: refusing to participate in the research should not affect access to the programme.

Evaluation adds data collection to a population already being surveyed, and that burden is part of what the ethics review should weigh.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Transparency

Pre-registration and open evidence

Pre-registration

Publicly recording your hypotheses, primary outcome and analysis plan before collecting or seeing the data — so you cannot quietly change the goalposts to fit the result.

Registries like the AEA RCT Registry and 3ie's records make evaluation honest and cumulative. Pre-registration is the main defence against cherry-picking and p-hacking.

Pre-registration records	Preventing
The hypothesis	Retrofitting one to the result
The primary outcome	Selecting the one that moved
The analysis plan	Specification searching
Subgroups	Post-hoc mining

Registries such as the AEA RCT Registry and the 3ie registry make the plan public and time-stamped, which is what gives the commitment force.

For a commissioner, requiring pre-registration in the terms of reference costs nothing and substantially raises the credibility of whatever comes back.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Publication Bias

The file-drawer problem

Studies that find big, positive effects get published and shared; null results often languish in a file drawer. So the published record can overstate what works — you see the hits, not the misses.

Trust systematic reviews and replications over any single splashy study. Bodies like 3ie and the Campbell Collaboration synthesise across studies precisely to correct this bias.

Published record over-represents	Because
Large effects	They are more publishable
Positive results	Nulls are shelved
Novel findings	Replications are harder to publish
Small studies with big estimates	Chance plus selection

The consequence is that the average published effect is larger than the true average effect, in a direction and by an amount that varies by field.

Trust systematic reviews and registries over individual studies, and check whether a review searched for unpublished work.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Section Eleven

Using IE for Decisions & Further Reading

ImpactMojoImpact Evaluation 101www.impactmojo.in

Commissioning

How to commission an impact evaluation well

Define the decision the evidence will inform

→

Write a sharp evaluation question

→

Bring the evaluator in at design stage

→

Demand a power calculation & identification strategy

→

Pre-register; plan to act on whatever you find

Your leverage as commissioner is greatest before the contract is signed. Ask the hard questions then, not when the report lands.

Commissioning step	Skipped at what cost
Define the decision	An unused study
Write a sharp question	A vague answer
Bring the evaluator in early	No credible design left
Demand a power calculation	An underpowered null
Require pre-registration	Findings that cannot be trusted

The third row is the one that determines whether the rest is possible. Once the rollout is fixed, the design space is whatever the allocation happened to allow.

Commissioners hold more influence over evaluation quality than evaluators do, because most of the quality is decided before the contract is signed.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Reading a Proposal

Questions to ask any evaluator

What is the counterfactual, and how credible is it?
What must be true for this design to identify the effect?
What is the MDE, and is it smaller than a plausible impact?
How will you handle attrition, spillovers and partial take-up?
Is the primary outcome pre-specified, and will you pre-register?

Ask the evaluator	A weak answer
What is the counterfactual?	Vague or absent
What must be true to identify the effect?	Cannot state it
What is the MDE?	Not calculated
How will you handle attrition and spillover?	Not considered
Will you pre-register?	Reluctance

Five questions in a proposal meeting will distinguish a serious evaluator from one selling a methods label, and none requires technical training to ask.

Weak answers are not necessarily disqualifying. They are the conversation to have before the design is fixed rather than after.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Limits

The limits of 'what works'

Impact evaluation tells you whether a specific programme worked in a specific place. It cannot, by itself, tell you what to value, how to navigate trade-offs, or whether the result will hold at scale.

Evidence informs judgement; it does not replace it. The number is an input to a decision, never the decision itself.

— a working principle for evidence use

Impact evaluation can tell you	It cannot tell you
Whether this worked here	What to value
By how much	How to weigh trade-offs
For whom, if powered for it	Whether it holds at scale
Compared with a specific alternative	Whether a different design is better

Evidence informs judgement rather than replacing it, and treating an effect size as a decision skips the value questions that a number cannot answer.

The last row is worth remembering when a null arrives: the finding is about this version of the programme, not about the idea.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Evidence Into Action

Why good evidence still goes unused

Findings arrive after the decision was already made
Results are framed for academics, not for managers
No one owns turning the finding into a change in practice
Inconvenient nulls are quietly shelved

Plan use from the start: agree who decides what, by when, and commit to act on the answer — including if it is disappointing.

Why evidence goes unused	Fix
Arrives after the decision	Align the timeline to the decision
Framed for academics	Commission a practitioner brief too
Nobody owns the follow-through	Name an owner at commissioning
Inconvenient nulls are shelved	Agree publication terms in advance

The last row is a governance question rather than a communication one, and it should be settled in the contract before any result exists.

Naming who will act on the finding, at commissioning, is the single strongest predictor of whether anything happens afterwards.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Cornerstone

The one book to read: Gertler et al.

Impact Evaluation in Practice by Paul Gertler, Sebastian Martinez, Patrick Premand, Laura Rawlings and Christel Vermeersch (World Bank) is the standard, free, practitioner-friendly handbook — the natural next step after this deck.

It walks through counterfactuals, every design in this course, and a worked case study, in plain language. Download it free from the World Bank's Open Knowledge Repository.

Gertler et al. covers	Useful because
Every design in this course	Worked, with examples
Power calculation, step by step	Practically usable
Commissioning and management	Written for practitioners
Free to download	No barrier

Impact Evaluation in Practice is the standard practitioner handbook and is available free from the World Bank, which makes it the natural next step from this deck.

Read the design chapters and the power chapter. Those two carry most of what a commissioner needs beyond this course.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Where to Learn More

Trusted sources and resources

Source	What it offers	Note
J-PAL	Randomised evaluations, training, evidence reviews	MIT-based; strong South Asia office
3ie	Impact evaluations & systematic reviews of the Global South	Searchable evidence portal
World Bank DIME	Methods, guidance, the Gertler et al. handbook	Free handbook & toolkits
Campbell Collaboration	Systematic reviews of social interventions	Synthesis across studies
AEA RCT Registry	Pre-registered trial protocols	Check what was promised

Source	Best for
J-PAL	Randomised evaluations; training
3ie	Impact evaluations and systematic reviews of the Global South
Campbell Collaboration	Systematic reviews in social policy
AEA RCT Registry	Checking pre-registration
World Bank DIME	Methods and country work

3ie is the one most directly relevant to South Asian practice, and its evidence gap maps are a fast way to see what has and has not been studied in a sector.

Before commissioning anything, check whether a systematic review already answers your question. It is the cheapest possible evaluation.

ImpactMojoImpact Evaluation 101www.impactmojo.in

The Takeaways

If you remember five things

Always ask 'compared to what?' — the counterfactual is everything
Randomisation balances the unobservable, in expectation — no other method can claim that
Every quasi-experiment rests on an assumption — name it and test it
Plan power before, read effect size after — significance is not importance
Know when NOT to run an IE — and pre-register when you do

Pair this deck with ImpactMojo's Econometrics, Data Literacy and Research Ethics 101 courses.

Takeaway	The question it becomes
Compared to what?	What is the counterfactual here?
Randomisation balances unobservables	Was assignment genuinely random?
Every quasi-experiment has an assumption	What is it, and does it hold?
Power decides what you can see	What is the MDE?
Significance is not importance	What does the interval include?

Five questions, askable of any evaluation report or proposal, and none of them requires you to run an analysis.

If you carry one, carry the first. It is the whole subject compressed, and most weak impact claims fail on it immediately.

ImpactMojoImpact Evaluation 101www.impactmojo.in

Impact Evaluation 101 · Complete

Now go ask:
compared to what?

You don't have to run the regression to commission an evaluation well — you need to frame the question, choose a credible counterfactual, and read the findings honestly. Explore the rest of the ImpactMojo 101 Series, free forever.

More 101 Courses Explore ImpactMojo Dataverse

Continue your learning

Impact Evaluation Studio Causal Inference Flagship Counterfactual Game

CC BY-NC-ND 4.0·Free Forever·ImpactMojo 101 Series