The P-Value Answers a Question You Probably Aren’t Asking
The ImpactMojo Team
August 19, 2026
8 min read
statisticsp-valuesevaluation
Every programme evaluation eventually produces a line like this one: the intervention raised handwashing-with-soap at critical times by 4.1 percentage points (p = 0.03). The room hears "p is under 0.05", writes down "it worked", and moves to the next slide.
That reading skips the question the meeting actually came to answer. A p-value is a statement about how surprising the data would be in a world where the programme did nothing. It is silent on how big the effect is, and a decision about whether to scale a programme is almost entirely a question about how big the effect is.
The same analysis, read two ways. Only one of them tells a programme manager whether the effect is large enough to be worth the money.
The two questions, precisely
A p-value asks: if the null hypothesis were true — if the programme genuinely changed nothing — how often would we see data at least this extreme? At p = 0.03, the answer is about three times in a hundred. That is a statement about data, conditional on an assumption. It says nothing about the probability that the assumption is true.
A confidence interval asks a different question: given this data, which values of the effect are compatible with it? A 95% interval of 0.3 to 7.9 percentage points says the data would not be very surprising under any true effect in that range. The "95%" describes the procedure — repeat the study many times and about 95% of the intervals you construct will contain the true value. It is not the probability that this particular interval contains it.
Why the distinction changes the decision
Take the handwashing result above, and suppose the programme costs ₹340 per household reached. The p-value is 0.03 in both of the following cases, and they call for opposite decisions.
Interval A: +3.8 to +4.4 pp. The effect is precisely estimated and comfortably above the level at which the cost per additional practising household is defensible. Scale it.
Interval B: +0.3 to +7.9 pp. Same point estimate, same p-value, far more uncertainty. The data is consistent with an effect large enough to be a flagship result and with one so small the programme is worse value than the alternatives. The honest answer is that this study cannot tell you which, and the next step is a better-powered evaluation rather than a scale-up.
Reporting only "4.1 pp, p = 0.03" makes A and B indistinguishable. The interval is what separates them.
The interval in the diagram is Interval B, which is the more common situation in field evaluations in South Asia — cluster designs, modest sample sizes, and intracluster correlation that widens intervals considerably. A result can clear the significance threshold and still be almost uninformative about magnitude.
Four misreadings worth naming
The following are documented misinterpretations, not strawmen. Greenland and colleagues catalogue twenty-five of them in a paper written precisely because they recur in published work.
"p = 0.03 means a 97% chance the effect is real"
The p-value is computed assuming there is no effect. It cannot also be the probability that there is one. Getting from a p-value to a probability about the hypothesis requires a prior, which is what Bayesian analysis supplies and frequentist testing does not.
"p > 0.05 means no effect"
It means the data is compatible with no effect. It is usually also compatible with a substantial effect. An underpowered study reliably produces non-significant results whether or not the programme works, which is why "no significant difference" and "no difference" are not interchangeable.
"The 95% interval has a 95% chance of containing the truth"
The true effect is a fixed number; it is either inside this interval or it is not. The 95% is a property of the procedure across repetitions, which is why the term of art is confidence rather than probability.
"Two studies disagree because one was significant and one was not"
Two studies with heavily overlapping intervals can easily land on opposite sides of 0.05. Comparing significance verdicts is not the same as comparing effect estimates, and treating it as such manufactures contradictions that the underlying numbers do not support.
What to do instead
None of this makes p-values useless. It makes them insufficient on their own. Three changes to how results are reported do most of the work.
Lead with the interval. Write "+4.1 pp (95% CI: 0.3 to 7.9)" rather than "+4.1 pp, p = 0.03". The interval carries the point estimate, the uncertainty, and the significance verdict together — if it excludes zero, p is below 0.05.
Say what would change the decision before you see the result. If a 2 percentage point gain would not justify the cost, decide that during design. A pre-specified threshold turns an interval into an answer, and stops the effect size being reverse-engineered from whatever the study found.
Describe non-significant results by their interval, not their verdict. "We can rule out effects larger than 3 points" is informative. "No significant effect" is not, and is routinely read as evidence of absence.
The habit worth building is small: whenever a p-value appears without an interval beside it, ask for the interval. In most reports the number is already in the regression output and was simply not carried into the summary.
Valentin Amrhein, Sander Greenland and Blake McShane, "Scientists rise up against statistical significance" (Nature, 2019). The argument for retiring the significant/non-significant dichotomy, signed by more than eight hundred researchers.