Most development professionals read research papers for conclusions. This pack teaches you to read for design -- so you can assess whether the conclusions are actually supported by the evidence presented.
4 modules~100 minInteractive
Your progress
0% complete
Your Capstone
1-Page Critique of a Paper of Your Choice
A structured critique covering design, validity, statistical claims, and practical implications -- ready to share with colleagues or use in a journal club.
Module 1 -- ~25 min
Reading for design (RCT, quasi-experimental, qualitative)
Before evaluating findings, identify the research design. The design determines what claims the paper can and cannot make. A common over-claim is a strong causal claim resting on a weak design.
The design hierarchy (for causal claims)
Randomised Controlled Trial (RCT) -- random assignment to treatment/control. Strongest causal claim. Check: was randomisation actually random? Was there attrition? Were there spillovers?
Quasi-experimental -- comparison group but no random assignment. Difference-in-differences, regression discontinuity, propensity score matching. Check: how was the comparison group selected? Could selection bias remain?
Pre-post (no comparison) -- measures change over time in treatment group only. Cannot separate programme effect from time, maturation, or other changes. Check: does the paper acknowledge this limitation?
Cross-sectional -- snapshot at one point. Can describe associations but not causation or change. Check: does the paper claim change without time-series data?
Qualitative -- explores mechanisms, experiences, meaning. Not designed for causal claims. Check: does the paper stay within its design or over-reach?
Worked example -- Identifying design
Paper: "Impact of MGNREGA on rural livelihoods in Telangana" (hypothetical).
Method section says: "We surveyed 500 MGNREGA cardholders and compared their outcomes with 300 non-cardholders in the same mandals."
Design: Cross-sectional with comparison group. Not quasi-experimental because there is no pre-period and no method to address selection bias (people who get MGNREGA cards may differ systematically from those who do not). The paper can describe associations but cannot claim MGNREGA "caused" the differences.
Your Paper -- Design Identification
Choose a paper relevant to your work and identify its design.
Saved
Self-check
A paper titled "Impact of mid-day meals on school attendance" uses a pre-post design with no comparison group. Can it claim "impact"?
Yes -- pre-post measures change
No -- without a comparison group, the change could be due to other factors (time trends, policy changes, seasonal effects). It can document change but not attribute it.
Only if the sample is large enough
Only if the effect is very large
Correct. "Impact" implies causation, which requires a credible counterfactual (what would have happened without the programme). Pre-post designs cannot provide this. The attendance increase might be due to a new state policy, teacher recruitment, or seasonal patterns.
Module 2 -- ~25 min
Assessing internal and external validity
Internal validity: did the study actually measure what it claims to have measured? Were there alternative explanations the design did not rule out?
External validity: do the findings generalise beyond the study population? Would the same intervention produce the same results in a different context?
Internal validity threats
Selection bias -- treatment and comparison groups differ systematically before the intervention
Attrition -- if 30% of the sample drops out, the remaining sample may be systematically different
Spillover/contamination -- control group receives indirect benefits from the treatment
Hawthorne effect -- participants behave differently because they know they are being studied
Measurement error -- instruments are unreliable or biased
External validity questions
Was the study population representative of the broader target population?
Was the context (governance, culture, market access) similar to where you want to apply findings?
Was the implementer typical or exceptional? (Research-led pilots often have stronger implementation than government scale-up; Banerjee et al. (2016) document this for Teaching at the Right Level in India.)
The India generalisability problem
Many influential development studies are conducted in one or two Indian states. A study from Tamil Nadu may not generalise to Bihar -- governance capacity, social structures, and economic conditions differ enormously. When reading India-based papers, always check: which states? Urban or rural? Which population? The answer to "does it work?" is almost always "it depends on where and for whom."
Your Validity Assessment
Does the paper acknowledge them? How serious are they?
Saved
Self-check
An RCT of a livelihoods programme in Bangladesh shows strong effects. Your organisation wants to replicate it in Odisha. What is the primary concern?
Internal validity -- the RCT may have been poorly designed
External validity -- Bangladesh's context (NGO density, microfinance infrastructure, social norms) may differ significantly from Odisha
Statistical significance -- the sample may have been too small
The language barrier
Correct. RCTs have strong internal validity by design. The question for replication is always external validity: will the same intervention work in a different context with different implementing capacity, social norms, and market structures?