ImpactMojoSurvey Design 101www.impactmojo.in

ImpactMojo 101 Series · Free Forever

Survey
Design
101

Designing, Writing & Running Trustworthy Field Surveys — a Foundational Course for Development & MEL Practitioners in South Asia

Research-BackedSouth Asia Focus100 SlidesFree Access

ImpactMojoSurvey Design 101www.impactmojo.in

Agenda

What We Cover

What Surveys Are & When to Use One

Slides 3–10

The Survey Lifecycle

Slides 11–19

From Concept to Question

Slides 20–27

Writing Good Questions

Slides 28–36

Question & Response Types

Slides 37–45

Questionnaire Structure & Flow

Slides 46–54

Sampling for Surveys

Slides 55–63

Modes of Data Collection

Slides 64–72

Translation & Cultural Adaptation

Slides 73–81

Pretesting, Piloting & Fieldwork Quality

Slides 82–90

Ethics, Data & Practice

Slides 91–99

If you are	Start at
Commissioning a survey	Sections 1–3
Writing the instrument	Sections 4–6
Running fieldwork	Sections 7, 8, 10

No statistics background is assumed: where a formula would appear, the deck gives the decision it informs.

ImpactMojoSurvey Design 101www.impactmojo.in

Section One

What Surveys Are & When to Use One

This section answers	Slide
What a survey actually is	4
When one is the right tool — and when it is not	5–6
Why to exhaust free data first	7
What a survey can and cannot give you	8–9
What you owe the respondent	10

The commonest error in this section is procedural, not technical: a survey is commissioned because a survey was budgeted, before anyone established that the question needs one.

ImpactMojoSurvey Design 101www.impactmojo.in

Definition

A survey is a measurement instrument

A survey systematically collects standardised information from a sample of people, using the same questions in the same way, so answers can be counted and compared. Its power is standardisation; its discipline is design.

Survey

A method of gathering quantitative information by asking a defined sample of respondents a fixed set of questions under controlled conditions, so the results can be aggregated and generalised to a population.

A survey is not 'a list of questions'. It is an instrument that turns a concept into a number — and like any instrument, a badly built one produces confident nonsense.

Property	What it buys you	What it costs
Standardisation	Answers are comparable across people	Anything not anticipated is unrecordable
Sampling	A few thousand describe millions	An estimate with a margin, not a fact
Closed options	Fast, countable data	You have decided the answer space in advance
Repetition	Change over time is measurable	Improving the question breaks the trend

That last row is the trap nobody expects. The moment you fix a badly worded question, this round is no longer comparable with the last one — so you either keep a flawed item for continuity or accept a break in the series and document it loudly.

A survey does not discover what matters; it counts what someone already decided matters. The design work happens before a single question is drafted.

ImpactMojoSurvey Design 101www.impactmojo.in

When To Use One

Surveys answer 'how many' and 'how much'

A survey fits when…

You need numbers you can generalise
The concept is well understood already
You want to compare groups or track change
The population is too large to ask everyone

A survey misfits when…

You don't yet know the right questions
You need depth, meaning or 'why'
The topic is too sensitive for fixed answers
Good administrative data already exists

Reach for a survey to measure something you already understand — not to discover what matters.

The question you have	Right tool
What share of households have a toilet?	Survey
Why do people with toilets not use them?	Qualitative
How many children enrolled last year?	Administrative (UDISE+)
Did our programme cause the change?	Survey plus a design (control, baseline)

A survey with no comparison group can describe a situation but cannot attribute it. ‘Beneficiaries report higher income’ is not evidence a programme raised income — it is consistent with the programme having selected people whose income was rising anyway.

Two other misfits: a concept nobody has yet defined well enough to ask about, and a population so small you should simply speak to all of them.

ImpactMojoSurvey Design 101www.impactmojo.in

Three Data Sources

Survey, qualitative or administrative?

	Survey	Qualitative	Administrative
Answers	How many, how much	Why, how, meaning	What the system records
Strength	Generalisable, comparable	Depth, context, mechanism	Cheap, continuous
Sample	Representative sample	Small, purposive	Everyone served
Cost	High — fieldwork	Moderate	Already collected
Blind spot	Misses the 'why'	Cannot generalise	Misses who is not served

These are partners. A survey tells you that outcomes differ; qualitative work tells you why; admin data tells you what your programme already knows for free.

Failure mode	Survey	Qualitative	Administrative
Wrong people	Frame gaps, non-response	Convenience selection	Only those the system touched
Wrong answers	Bad wording, desirability	Interviewer steer	Reporting incentives
Cost of fixing	High — re-field	Low — ask again	None — you get what exists

The three sources are complementary, not ranked. The strongest designs sequence them: qualitative work to find out what to ask, administrative data to size the problem, a survey to measure it comparably.

Administrative data has an incentive problem surveys do not: the people who record it are often judged on it. Attendance registers, delivery counts and grievance logs all bend toward the target.

ImpactMojoSurvey Design 101www.impactmojo.in

Exhaust Free Data First

Don't survey what already exists

Before commissioning a survey, check whether the answer is already in secondary data — the Census, NFHS, PLFS/NSS, or your own HMIS, UDISE+ and MGNREGA records. Re-collecting wastes budget and, worse, wastes respondents' time.

NFHS-6

Health, nutrition, fertility down to district

IIPS / MoHFW, 2023–24

PLFS

Employment & unemployment, annual since 2017–18

MoSPI

HMIS

Your own facility data, updated monthly

Rule of thumb: a new survey is justified only for what existing data genuinely cannot answer at the level you need.

Source	Covers	Watch for
Census	Every household, decadal	2011 is the last full round; badly dated
NFHS	Health, nutrition, women’s status; district level	Sample sizes thin below district
PLFS / NSS	Employment, consumption; quarterly urban	Definitions of ‘work’ are technical
HMIS · UDISE+ · MGNREGA MIS	Continuous, administrative	Records the system, not the population

Check the secondary source first even when you are sure you will still need to collect. It often gives you a benchmark to sanity-check your own estimate against, which is worth having whatever you find.

Survey fatigue is real and compounding. In heavily studied blocks, households have been interviewed repeatedly with nothing returned to them, and refusal rates show it.

ImpactMojoSurvey Design 101www.impactmojo.in

Strengths

What a good survey buys you

Generalisation: a few thousand can describe millions
Comparability: identical questions across people, places, time
Quantification: attach a number and a margin of error to a claim
Coverage: reach groups and topics no register captures
Transparency: a documented instrument others can scrutinise

Claim you can make	Only because
‘38% of households, ±3 points’	The sample was drawn probabilistically
‘Higher in Block A than Block B’	The same instrument was used in both
‘Up from 31% in 2023’	The question and reference period did not change

Each of these claims rests on a design choice that had to be made before fieldwork. None can be repaired afterwards by analysis.

ImpactMojoSurvey Design 101www.impactmojo.in

Limits

What a survey can never give you

Meaning: fixed options flatten messy human reality
The unasked: a closed question can only return its own choices
Truth on tap: people misremember, please, and self-present
The missing: whoever the frame and fieldwork miss stays invisible

A survey measures what you thought to ask, from whoever you managed to reach. Both boundaries are design choices — make them deliberately.

Limit	What it looks like in a report
Meaning is flattened	‘Satisfied’ hides four different reasons
Only the asked is known	No line for the thing you did not anticipate
Self-report	‘Reported open defecation’ ≠ observed
The missing	Migrants, homeless, institutionalised — absent, uncounted

The honest fix is not to abandon surveys but to say what the number is. Write ‘share of respondents who reported X’ rather than ‘share who did X’ — the difference is the whole of measurement error.

Whoever the frame and the fieldwork miss are usually the poorest and most mobile. A survey’s blind spot is rarely random.

ImpactMojoSurvey Design 101www.impactmojo.in

Mindset

The respondent is doing you a favour

Asking a question is asking for time, attention and trust from someone who owes you none. Design as if you will have to sit through your own survey.

— a field researcher's rule of thumb

Every needless question, every confusing scale, every minute of length is a cost paid by a real person — often a busy, poor, over-surveyed one. Respect is good methodology.

Question you are about to add	Ask instead
‘Might be useful for analysis’	Which planned table needs it?
‘The donor may want it’	Will anyone read it? What decision changes?
‘We have always asked it’	Has it ever appeared in a report?

Every item costs respondent time, enumerator attention, translation, testing, cleaning and analysis. An instrument grows by accretion because each single addition looks cheap and none of the costs land on the person adding it.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Two

The Survey Lifecycle

ImpactMojoSurvey Design 101www.impactmojo.in

The Whole Journey

Eight stages from question to use

OBJECTIVES: the decision the survey must serve

→

INDICATORS: what to measure, defined precisely

→

INSTRUMENT: the questionnaire that captures it

→

SAMPLE: whom to ask, and how many

→

FIELDWORK: collecting data well in the field

→

DATA: clean, weighted, documented

→

ANALYSIS: turning answers into findings

→

USE: feeding a real decision

Most survey failures are designed in at the start — not in the analysis. Time spent on the first four stages is repaid many times over in the last four.

Stage	The mistake that is expensive here
Objectives	Starting from questions, not the decision
Indicators	Leaving the definition to the analyst
Instrument	Skipping cognitive pretesting
Sample	A frame nobody inspected
Fieldwork	No back-checks while data is still collectable
Cleaning & analysis	Discovering a missing variable

Errors get roughly ten times costlier at each stage. A wording problem caught in pretesting costs an afternoon; the same problem found in analysis costs the indicator.

ImpactMojoSurvey Design 101www.impactmojo.in

Start From The Decision

Objectives before questions

The first question is never 'what should we ask?' but 'what decision will this survey inform?' Work backwards from the decision to the findings you need, then to the indicators, then to the questions.

DECISION: where to scale the nutrition programme?

→

FINDING NEEDED: which blocks have highest stunting?

→

INDICATOR: % children under 5 stunted, by block

→

QUESTION: child's height & age, measured directly

If you cannot name the decision a question serves, cut the question. 'Nice to know' is the enemy of a usable survey.

Work backwards	Example
Decision	Whether to extend the cash transfer to Block C
Finding needed	Whether Block C’s food-insecurity rate resembles A and B
Indicator	FIES moderate-or-severe, past 12 months, household
Questions	The eight standard FIES items, translated and tested

If you cannot name the decision, the survey has no stopping rule for what to include — which is how a 20-minute instrument becomes 90 minutes.

Use an existing validated module (FIES, WEAI, PHQ-9, DHS women’s modules) wherever one exists. It is tested, translated, and comparable to other people’s numbers.

ImpactMojoSurvey Design 101www.impactmojo.in

Research Questions

Sharpen vague aims into answerable questions

Too vague

“Understand women's livelihoods in the district.” — unmeasurable; no population, no indicator, no time frame.

Answerable

“What share of women aged 18–45 in Block X earned cash income in the last 30 days, and from what source?”

A good research question names who, what, where and over what period. If it doesn't, the questionnaire can't either.

Component	Vague version	Answerable version
Population	‘women in the district’	women aged 18–45 in Block X
Indicator	‘livelihoods’	earned any cash income
Reference period	—	in the last 30 days
Comparison	—	vs the same measure in 2024

A research question is answerable when you can state, before fieldwork, what number would answer it and roughly what precision you need.

ImpactMojoSurvey Design 101www.impactmojo.in

Scope

Decide the unit and the reference period

Unit of analysis: the household? the individual? the child? the enterprise? Mixing units quietly breaks your numbers.
Respondent vs subject: a mother may report for a child — name who answers about whom.
Reference period: last 7 days? last month? last year? The window changes both recall and the indicator itself.

‘Household income’ and ‘respondent's income’ are different surveys. Decide the unit before you write a single question.

Choice	If you get it wrong
Unit of analysis	Household and individual rows mixed — denominators wrong
Respondent vs subject	‘Proxy reporting’ unlabelled — mother’s report read as the child’s
Reference period	‘Usually’ means something different to every respondent
Inclusion rule	Who counts as a household member differs by enumerator

‘Household member’ is the classic silent breakage. A migrant son who sends money — is he a member? Different enumerators will answer differently unless the rule is written and trained.

ImpactMojoSurvey Design 101www.impactmojo.in

Plan The Whole Chain

Design the analysis before fieldwork

Sketch your dummy tables — the exact tables and charts you will produce — before collecting a single response. If a planned table needs a variable no question captures, you found the gap in time to fix it.

“Which table will this question feed?” is the single best test of whether a question belongs in the instrument.

Dummy table	Variables it demands
Food insecurity by block, 2024 vs 2026	FIES score, block, round
Income by sex of respondent	Cash income, sex, reference period
Toilet use by caste group	Use item, social group, weights

Sketch the tables and charts of the final report before drafting the instrument. Any table that needs a variable no question captures has just found you a gap while it is still free to fix.

ImpactMojoSurvey Design 101www.impactmojo.in

Resources

Budget, time and team shape the design

Constraint	Pushes toward	Watch-out
Tight budget	Shorter instrument, phone mode	Coverage & quality loss
Short timeline	Smaller sample, fewer items	Underpowered estimates
Few trained staff	Simpler skips, CAPI logic	Enumerator error
Remote terrain	Cluster sampling, offline tools	Travel cost & fatigue

Constraints are not excuses for sloppiness — they are inputs to design. A realistic small survey beats an ambitious one that collapses in the field.

Constraint	Honest response	Dishonest response
Tight budget	Cut items, keep the sample and training	Keep everything, cut training
Short timeline	Smaller sample, state the wider margin	Same sample, rushed fieldwork
Few trained staff	Simpler instrument, CAPI logic	Complex paper skips

Training is the line item that gets cut because its absence is invisible in the deliverable. It shows up instead as unexplainable variation between enumerators, which is discovered months later, if ever.

ImpactMojoSurvey Design 101www.impactmojo.in

Documentation

Write it down as you go

A survey protocol: objectives, population, sample, methods
A questionnaire with variable names and routing
A field manual for enumerators and supervisors
A data dictionary: every variable, code and unit

These four documents are the difference between a survey someone can trust and re-use, and a one-off whose meaning dies with the team that ran it.

Document	Answers	Who needs it
Protocol	Why, whom, how many, how	Reviewers, ethics, successors
Questionnaire with variable names	What was asked, in what order	Analyst
Field manual	What to do at every edge case	Enumerators, supervisors
Data dictionary	What each variable, code and unit means	Anyone reusing the data

Write these as you go, not at the end. Reconstructed documentation is guesswork with a confident tone, and the person reconstructing is usually not the person who decided.

ImpactMojoSurvey Design 101www.impactmojo.in

Iterate

The lifecycle loops, it doesn't end

Pretesting sends you back to rewrite questions. Piloting sends you back to fix routing and sample. Analysis reveals what the next round should ask. A survey programme is a cycle of learning, not a single shot.

No questionnaire survives first contact with respondents. The good ones are the most rewritten.

— survey methodology folklore

What sends you back	To fix
Cognitive pretest	Wording, recall, option sets
Pilot	Routing, length, logistics, sample
Back-checks	Enumerator practice, mid-field
Analysis	What the next round should ask

If the survey will be repeated, decide now which items are frozen for comparability and which are open to revision, and record that decision in the protocol.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Three

From Concept to Question

ImpactMojoSurvey Design 101www.impactmojo.in

The Core Problem

You cannot ask 'empowerment' directly

Most things surveys care about — empowerment, food security, trust, wellbeing — are concepts, not facts you can simply request. The questionnaire is the bridge from an abstract concept to a concrete, observable answer.

CONCEPT: food security

→

DIMENSIONS: access, sufficiency, anxiety

→

INDICATOR: days household ate fewer meals last month

→

QUESTION: “In the last 30 days, on how many days…?”

Concept	A concrete stand-in	What the stand-in misses
Food security	FIES: 8 experience items, 12 months	Quality and dietary diversity
Empowerment	Who decides on large purchases	Whether she wanted to decide
Trust in the panchayat	Would you approach it with a problem	Whether approaching it works

The gap between concept and indicator never closes. Naming it in the report is the difference between a measure and a claim.

ImpactMojoSurvey Design 101www.impactmojo.in

Operationalising

Turn the concept into a measurement rule

Operationalisation

The precise rule that converts a concept into something countable: exactly what to ask, of whom, over what period, in what units, and how the answer maps to the indicator.

‘Is the household poor?’ only becomes measurable once you fix the definition — income below a line? deprived on a set of MPI indicators? — and write the exact questions that test it.

Rule must fix	Example
Exactly what to ask	‘In the last 30 days, did you or anyone…’
Of whom	The most knowledgeable adult member
Over what period	30 days, ending yesterday
In what units	Rupees; local units converted at entry
How it maps to the indicator	Any ‘yes’ → coded 1

Two teams operationalising ‘is the household food-insecure?’ differently will produce different rates from the same villages, and both will be defensible. That is why the rule, not the concept, belongs in the protocol.

ImpactMojoSurvey Design 101www.impactmojo.in

The Instrument Measures

The questionnaire IS the measurement

There is no underlying 'true survey' that a questionnaire merely records. The wording, order, options and translation are the instrument — change them and you change what you measure. Two questionnaires on the 'same' topic can yield different numbers, both correct for their wording.

This is why comparability demands identical instruments. Surveys compare answers to questions, not to reality directly — so the question must hold still.

Change this	And you have changed
A word in the stem	The concept respondents answer about
The option set	What answers are possible at all
The order	What the previous question primed
The translation	The instrument, in that language

This is why questionnaire changes between rounds are treated like an instrument recalibration in a laboratory: documented, dated, and flagged in every table that spans the break.

It also means a ‘small improvement’ requested late by a reviewer is not small. Price it as a break in the series.

ImpactMojoSurvey Design 101www.impactmojo.in

Total Survey Error

A map of everything that can go wrong

Groves' Total Survey Error (TSE) framework organises every source of error in a survey into two families: errors of representation (the wrong people) and errors of measurement (the wrong answers). Good design manages both at once.

Total Survey Error

The accumulated difference between a survey estimate and the true population value, from all sources — sampling and non-sampling, representation and measurement — considered together (Groves et al.).

Branch	Error	Concrete example
Representation	Coverage	Frame is the ration-card list; the unlisted are invisible
Representation	Sampling	Chance variation in who was drawn
Representation	Non-response	Working men never at home in the day
Measurement	Validity	The item does not capture the concept
Measurement	Response	Desirability, recall, interviewer effect
Measurement	Processing	Coding and entry errors

TSE’s discipline is that it forces you to name where your budget went, and to notice that most of it usually went to reducing the one error that is easiest to compute.

ImpactMojoSurvey Design 101www.impactmojo.in

The TSE Tree

Two branches, many leaves

ImpactMojoSurvey Design 101www.impactmojo.in

Two Families

Representation vs measurement error

	Representation (wrong people)	Measurement (wrong answers)
Coverage	Frame misses part of the population	—
Sampling	Chance variation in who is drawn	—
Non-response	Those who answer differ from those who don't	—
Validity	—	Question doesn't capture the concept
Response	—	Recall, social desirability, bad wording
Processing	—	Data entry & coding mistakes

Sampling error shrinks with sample size. The other five do not — they need design, not a bigger n.

Error	Fixable after fieldwork?
Sampling	Yes — reported as a margin of error
Coverage	No — the missing were never eligible to be drawn
Non-response	Partly — weights, if you know who is missing
Response bias	No — it is baked into the answers
Processing	Yes — if the raw export was kept

Only one row of that table is what a confidence interval describes. A report that shows ±3% and says nothing about the other four rows is presenting its smallest error as its total error.

ImpactMojoSurvey Design 101www.impactmojo.in

Trade-Offs

You manage error within a fixed budget

TSE's practical lesson: a fixed budget forces trade-offs. Spending everything on a huge sample (less sampling error) may starve enumerator training (more response error). Total error is what matters — not any one component.

Ask of every design choice: which errors does it reduce, which does it grow, and is the total smaller? A smaller but well-run survey often beats a larger, sloppier one.

Extra rupee spent on	Reduces	At the cost of
A bigger sample	Sampling error	Training, follow-up, quality
Enumerator training	Response & processing error	Sample size
Revisits & refusal conversion	Non-response bias	Coverage of new areas
Pretesting	Validity & response error	Fieldwork days

Where the marginal rupee does most good depends on which error currently dominates — and you usually know that from the last round, if anyone wrote it down.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Four

Writing Good Questions

Fault	Slide
Double-barrelled	30
Leading	31
Loaded or assuming	32
Jargon	33
Vague reference period	34–35

Every fault in this section is invisible to the person who wrote the question and obvious to a respondent. That asymmetry is the entire argument for cognitive pretesting.

ImpactMojoSurvey Design 101www.impactmojo.in

The Golden Rule

Clear, specific, one idea

A good survey question is understood the same way by every respondent and the question-writer. Three tests cover most failures: is it clear, is it specific, does it ask one thing?

CLEAR: plain words the respondent uses

→

SPECIFIC: a definite thing, time and unit

→

ONE IDEA: a single answerable question

→

NEUTRAL: no nudge toward an answer

Test	Failing version	Passing version
Clear	‘Do you avail ANC services?’	‘Did you go for check-ups when pregnant?’
Specific	‘Do you earn much?’	‘Last month, how much did you earn in cash?’
One idea	‘Clean and well-staffed?’	Two separate questions

A fourth test is worth adding: could two reasonable people answer the same true situation differently? If yes, the question is under-specified.

ImpactMojoSurvey Design 101www.impactmojo.in

Double-Barrelled

One question, one idea — never two

Bad

“Is the health centre clean and well-staffed?” — clean but understaffed? The respondent can't answer; you can't interpret.

Good

“Is the health centre clean?”
then
“Is the health centre adequately staffed?” — two questions, two clean answers.

Watch for the word and — it is the commonest sign a question is secretly two.

Hidden second idea	Split into
‘Clean and well-staffed?’	Clean? / Adequately staffed?
‘Safe and affordable?’	Safe? / Affordable?
‘Do you save and invest?’	Save? / Invest?

Double-barrelled items usually survive drafting because the author has one situation in mind where both halves are true. They break on the respondent for whom exactly one half is true — which is the case you most wanted to detect.

ImpactMojoSurvey Design 101www.impactmojo.in

Leading

Don't tell the respondent what to say

Leading

“Don't you agree the new clinic has improved care?” — pushes a yes; politeness does the rest.

Neutral

“Since the new clinic opened, has the care you receive become better, stayed the same, or become worse?”

Offer the full range of answers in the question itself. A balanced stem signals that any answer is acceptable.

Leading device	Neutral repair
‘Don’t you agree…’	‘Do you agree or disagree…’
One-sided stem	Name both directions in the stem
Unbalanced options	Equal number either side of the midpoint
Prestige cue (‘experts say’)	Delete it

Politeness does most of the work of a leading question, and it is strongest exactly where power differences are largest — an outsider with a tablet asking a poor respondent about a government service.

ImpactMojoSurvey Design 101www.impactmojo.in

Loaded & Assuming

Don't smuggle in assumptions

Loaded

“How much did the corrupt official demand?” — assumes corruption and a demand. “How many children do you still want?” assumes she wants more.

Open the assumption

“Did you pay anything beyond the official fee?” — then ask how much. Let a filter establish the fact before you ask about it.

A loaded question forces a false premise. Use a filter question to establish the fact before asking about it.

Question	Assumption smuggled in	Open it
‘How much did the official demand?’	That a demand was made	‘Did you pay anything beyond the official fee?’
‘How many children do you still want?’	That she wants more	‘Do you want any more children?’ → filter
‘When did you stop attending?’	That she attended, and stopped	Two filtered questions

The pattern is the same each time: a filter question first, then the quantity question only for those it applies to.

ImpactMojoSurvey Design 101www.impactmojo.in

Jargon

Use the respondent's words, not yours

Insider language

“Do you utilise institutional delivery services and avail ANC?” — acronyms and jargon no respondent uses.

Plain

“When you were pregnant, did you go for check-ups? Where did you give birth — at home or at a facility?”

If a question needs the enumerator to explain a word, the word is wrong. Write for the least-schooled respondent in your sample.

Insider term	What respondents say
institutional delivery	gave birth at the hospital / at home
ANC	check-ups when you were pregnant
livelihood diversification	other work besides farming
open defecation	going outside / to the field

The rule of thumb: if the phrase would not appear in a conversation in a courtyard, it does not belong in the question. Keep the technical term in the variable name, not the stem.

Test this in the language of administration, not only in English. Jargon reappears in translation when the translator is a programme person.

ImpactMojoSurvey Design 101www.impactmojo.in

Reference Periods

Anchor every question in time

Vague	Anchored	Why it's better
“Do you usually…?”	“In the last 7 days…?”	Defines 'usually' for everyone
“recently”	“in the last 30 days”	Same window for all respondents
“your income”	“income last month”	Fixes the unit of time
“often sick”	“ill in the past 2 weeks”	Countable, comparable

The right window balances recall against rarity: short enough to remember, long enough to capture the event. Health uses 2 weeks; consumption, 30 days; big purchases, a year.

Period	Suits	Risk
7 days	Frequent, small events (meals, wage work)	Seasonal events missed entirely
30 days	Income, expenditure, health visits	Telescoping in from outside
12 months	Rare events (birth, migration, shock)	Heavy forgetting
Since a local landmark	Recall-friendly anchoring	Date differs slightly by respondent

Short periods measure precisely but may capture an unrepresentative window; long periods cover the cycle but decay. Where it matters, do both and reconcile.

ImpactMojoSurvey Design 101www.impactmojo.in

Recall Error

Memory fades — and bends

Recall error

The gap between what actually happened and what a respondent remembers and reports — events forgotten, dates blurred, or 'telescoped' in from outside the reference period.

Forgetting: small, routine events vanish first
Telescoping: a memorable event is pulled into the window
Salience: big events (a death, a wedding) recalled far better
Fix: shorter windows, anchoring to landmark dates & festivals

Recall failure	Direction of the error
Forgetting small routine events	Under-report
Telescoping (pulling events in)	Over-report
Rounding and heaping (0, 5, 10)	Distribution distorted, mean roughly intact
Reconstructing from a rule	Smoothed, less variance than reality

Heaping at round numbers is the easiest quality check you can run on your own data: a spike at 0, 5, 10, 100 and 500 in a continuous variable tells you the answers were estimated, not recalled.

ImpactMojoSurvey Design 101www.impactmojo.in

A Rewrite Drill

Spot every flaw, then fix it

Before

“Don't you think you should usually spend more on your children's education and health?”

Leading + double-barrelled + vague + loaded — all four faults in one line.

After

“In the last 30 days, about how much did your household spend on schooling?” then “…on health?”

Read every question aloud and ask: am I leading, doubling up, assuming, or using jargon? Most bad questions fail more than one test at once.

Fault	Where it is in the ‘before’
Leading	‘Don’t you think you should…’
Double-barrelled	education and health
Vague	‘usually’, ‘more’
Loaded	presumes current spending is too low

Run this drill on your own instrument with a colleague who did not write it. Faults are nearly invisible to the author, who knows what was meant.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Five

Question & Response Types

ImpactMojoSurvey Design 101www.impactmojo.in

Open vs Closed

Let them speak, or give them choices?

	Open-ended	Closed-ended
Answer	In their own words	Pick from fixed options
Best for	Exploring, unknown answers	Counting, comparing
Analysis	Slow — needs coding	Fast — ready to tabulate
Risk	Vague, hard to compare	Misses the unlisted answer
Use when	Few cases, discovery	Most quantitative items

A closed question can only ever return the options you wrote. Pretest open-ended first to learn the real answer set, then close it for scale.

Use open when	Use closed when
You do not yet know the answer space	The options are known and stable
Piloting, to build the option list	Fielding at scale
The point is the respondent’s words	The point is a comparable count

The productive sequence is to run open in the pilot, code what comes back, and field the resulting list as closed with an ‘other, specify’ escape.

Open items at scale are expensive in a way that is easy to forget: someone must read, code and reconcile thousands of free-text answers, in several languages, consistently.

ImpactMojoSurvey Design 101www.impactmojo.in

Closed Done Well

Exhaustive and mutually exclusive

Response options must cover every possible answer (exhaustive) and never overlap (mutually exclusive). A respondent should always find exactly one box that fits.

Overlapping

Age: 0–18, 18–30, 30–50 — where does an 18- or 30-year-old go? And what about 60+?

Clean

0–17, 18–29, 30–49, 50–64, 65+ — no gaps, no overlaps, a home for everyone.

Fault	Example	Fix
Overlapping	0–18, 18–30, 30–50	0–17, 18–29, 30–49
Not exhaustive	Married / Unmarried	+ Widowed, Separated, Divorced
Unbalanced	Excellent–Good–Fair	Equal points either side
Missing escape	Fixed list only	+ Other (specify)

Collect ages and amounts as continuous numbers and band them at analysis. Banding in the instrument throws away information you can never recover, and locks you into someone’s guess about the useful cut-points.

ImpactMojoSurvey Design 101www.impactmojo.in

Likert Scales

Measuring agreement and attitude

Likert scale

A symmetric, ordered set of response options — typically 5 or 7 points — running from strong disagreement to strong agreement, with a neutral midpoint and balanced wording on each side.

Example: Strongly disagree · Disagree · Neither · Agree · Strongly agree. Keep the steps balanced, label every point (not just the ends), and use the same scale throughout a block.

Design choice	Convention
Points	5 or 7, odd if a neutral is genuine
Labels	Label every point, not just the ends
Balance	Same number and intensity either side
Direction	Keep it constant within a block

Likert data is ordinal. Averaging ‘strongly agree’=5 across items is common and usually tolerable for a scale that was validated that way; doing it to a single home-made item and reporting ‘mean satisfaction 3.7’ means very little.

Reversing a few items to catch straight-lining is standard, but reverse-worded items are harder to translate and often behave differently. Use sparingly and test them.

ImpactMojoSurvey Design 101www.impactmojo.in

Scale Choices

How many points, and a midpoint?

Choice	Pro	Con
5 points	Simple, fast, fits small screens	Less fine-grained
7 points	More discrimination	Harder to read aloud
With midpoint	Allows a genuine neutral	Some hide there to avoid choosing
Forced (no midpoint)	Pushes a stance	Fabricates an opinion that isn't there

For face-to-face fieldwork with mixed literacy, a labelled 5-point scale read aloud usually travels best. Whatever you pick, keep it consistent across the instrument.

Question	Choose	Because
Read aloud by an enumerator?	5 points	7 is hard to hold in memory
Self-completed on screen?	5 or 7	Respondent can see the scale
Is neutral a real position?	Keep the midpoint	Forcing hides genuine indifference
Is neutral an escape here?	Consider forced choice + explicit DK	Separates indifference from avoidance

Whatever you choose, keep it identical across rounds. A 5-point scale replaced by a 7-point scale produces a level shift that will be read as real change.

ImpactMojoSurvey Design 101www.impactmojo.in

Rating & Ranking

Rate each, or order them?

Rating

Score each item on its own scale (rate each service 1–5). Easy, but everything can end up 'good'.

Ranking

Force an order (rank your top 3 priorities). Reveals trade-offs, but is cognitively harder — keep the list short.

Ranking more than 4–5 items overloads respondents in the field. If you must, ask only for the top few, not a full order.

	Rating	Ranking
Cognitive load	Low	High — grows fast with list length
Reveals trade-offs	No — all items can be ‘important’	Yes
List length	Can be long	Cap at 4–5 items
Analysis	Straightforward	Ordinal, awkward to average

If you need priorities rather than approval, rank — but rank a short list, and read only the top one or two positions, which is all respondents reliably discriminate.

ImpactMojoSurvey Design 101www.impactmojo.in

Don't Know

When 'don't know' is data, not laziness

A genuine ‘Don't know’ or ‘Refused’ option prevents respondents from guessing or enumerators from inventing answers. But offered too readily, it becomes an easy escape that hollows out your data.

Keep DK/Refused available but not read aloud as a default
Distinguish ‘don't know’ from ‘not applicable’ from ‘refused’
Code them separately — never silently as missing or zero

Response	Means	Handle as
Don’t know	Genuinely lacks the information	Valid answer — report the share
Refused	Declines to say	Valid; often informative on sensitive items
Not applicable	Filtered out by design	Structural missing, not missing data
Blank	Enumerator skipped it	A data-quality problem

Keep DK and Refused distinct in the codebook, and do not read them aloud as options unless the item warrants it. On CAPI, make them reachable but not the first thing on screen.

A rising DK rate mid-fieldwork usually means enumerator fatigue rather than respondent ignorance. It is one of the most useful things a supervisor can monitor daily.

ImpactMojoSurvey Design 101www.impactmojo.in

Filters & Skips

Ask only what applies

A filter (gateway) question routes each respondent to only the relevant items via skip logic — so a man is never asked about his last pregnancy, and a non-farmer skips the whole farming module.

FILTER: Did anyone in the household farm last season?

→

IF NO → skip the entire agriculture module

→

IF YES → ask crops, area, inputs, yield

→

RESULT: shorter interview, cleaner data

Filter	Routes to	Trap
Any land cultivated?	Farming module	‘Cultivated’ undefined — leased in? homestead?
Ever been pregnant?	Maternal module	Asked of the wrong respondent
Any member migrated?	Migration module	‘Member’ rule not trained

Every filter is a place where a respondent can be routed out of a whole module by a single mis-keyed answer. Those are the items to check hardest in the pilot — an error there is silent and total.

ImpactMojoSurvey Design 101www.impactmojo.in

Routing On CAPI

Why skip logic loves a screen

On paper, complex skips invite enumerator error — missed branches, contradictory entries. On CAPI (computer-assisted personal interviewing), the device enforces routing automatically and can run real-time consistency checks.

Build skips, ranges and logic checks into the digital instrument so impossible answers (a 6-year-old with children) are caught at the doorstep, not in cleaning weeks later.

CAPI gives you	Paper cannot
Enforced routing	— skips are the enumerator’s job
Real-time range and consistency checks	— caught weeks later, if ever
Timestamps and GPS per interview	—
Same-day sync for back-checks	—

CAPI removes routing error and adds new ones: a hard constraint that rejects a true-but-unusual value, and a screen that hides context the paper page showed at a glance. Set range checks as soft warnings rather than blocks wherever a legitimate outlier is possible.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Six

Questionnaire Structure & Flow

ImpactMojoSurvey Design 101www.impactmojo.in

The Whole Arc

A questionnaire has a beginning, middle, end

Order is not cosmetic — it shapes answers and completion. A well-built instrument moves the respondent through a deliberate arc: welcome, easy questions, the core, the sensitive, then the close.

INTRO: consent, purpose, confidentiality

→

WARM-UP: easy, non-threatening items

→

CORE: the main substantive modules

→

SENSITIVE: income, health, beliefs — late

→

CLOSE: demographics, thanks

Stage	Purpose	Typical share of the interview
Introduction & consent	Explain, obtain consent	2–3 minutes
Warm-up	Easy, relevant, builds rapport	Short
Core modules	The substance	The bulk
Sensitive items	Placed late, after trust	Short, careful
Close	Thanks, contact, next steps	1 minute

The close matters more than it looks. A respondent who knows what happens next, and who to contact, is a respondent who will answer the follow-up round.

ImpactMojoSurvey Design 101www.impactmojo.in

The Questionnaire Map

A flow diagram of one instrument

ImpactMojoSurvey Design 101www.impactmojo.in

Logical Order

Group by topic, move from general to specific

Cluster related items into clearly signposted modules
Within a module, go from general to specific
Keep the same response scale within a block to build rhythm
Don't jump topics erratically — context-switching tires and confuses

A respondent who can follow the logic stays engaged and answers more accurately. Disordered questionnaires leak data through fatigue and confusion.

Ordering rule	Why
Group by topic, signpost each module	Reduces context-switching cost
General before specific	Specific items narrow later general answers
Same scale within a block	Builds rhythm, cuts errors
Filter before the module it gates	Avoids asking then discarding

Signposting aloud (‘Now I will ask about your work’) is not padding — it tells the respondent which frame to answer in and measurably reduces misinterpretation.

ImpactMojoSurvey Design 101www.impactmojo.in

Question Order Effects

Earlier questions colour later answers

Asking about local crime, then overall life satisfaction, drags satisfaction down — the first question primes the second. Order effects are real and measurable.

Put general attitude questions before specific ones on the same topic, and keep the order identical across rounds so any priming is at least constant and comparable.

Order	Effect
Crime questions, then life satisfaction	Satisfaction reported lower
Specific items, then a general one	General answer narrows to what was just asked
Programme questions, then trust in government	Trust reflects the programme

Order effects are real, measurable, and not removable by analysis. The only defences are to put general attitude items before specific ones, and to keep the order identical across rounds and arms.

ImpactMojoSurvey Design 101www.impactmojo.in

Warm Up First

Open with easy, relevant questions

The first questions should be easy, non-threatening and obviously relevant — they build rapport and signal that the survey is safe. Never open with income, caste or a long grid; you'll lose people at the door.

Save demographics like income and caste for late in the interview, once trust is established — not as an intimidating opening gate.

Open with	Never open with
Household roster, simple facts	Income
An easy, obviously relevant item	Caste
Something the respondent can answer confidently	A long grid

Most refusals happen in the first ninety seconds. What is asked there is deciding whether the interview happens at all, not measuring anything.

ImpactMojoSurvey Design 101www.impactmojo.in

Sensitive Questions

Handle the hard topics with care

Place them late, after rapport is built
Normalise: “Many people find…” lowers the threat
Offer private modes (self-completion, sealed response) for stigmatised topics
Always allow refusal without penalty or pressure

Income, health status, violence, caste, sexual behaviour and political views all invite social-desirability bias. Design reduces it; clumsy placement amplifies it.

Technique	What it does
Place late	Trades on rapport already built
Normalising preamble	‘Many people find…’ lowers the threat
Self-completion / sealed response	Removes the interviewer from the answer
Explicit right to refuse	Makes refusal safe, so answers given are truer

On violence, stigma or illegal activity, follow the established protocols rather than improvising: private setting, matched interviewer, a referral list in hand, and no interview if privacy cannot be secured.

ImpactMojoSurvey Design 101www.impactmojo.in

Length & Fatigue

Every extra minute costs quality

Illustrative: data quality declines as the interview lengthens

Illustrative pattern (respondent fatigue)

As fatigue sets in, respondents ‘straight-line’ grids, pick the first option, and say 'don't know' to escape. Ruthlessly cut every question that doesn't feed a planned table.

Fatigue shows as	Detect it by
Straight-lining a grid	Zero variance across a block

Monitor interview duration and per-block variance daily. Both are free on CAPI.

ImpactMojoSurvey Design 101www.impactmojo.in

Layout

Design the page, not just the words

Number questions and variables consistently for clean data entry
Make instructions to the enumerator visually distinct from the question text
Show skip instructions at the branching question, in bold
On CAPI, one screen per question reduces missed items and accidental skips

A cramped, ambiguous layout produces enumerator errors that no amount of cleaning fully repairs. Layout is part of measurement.

Element	Rule
Variable names	Fixed, meaningful, in the instrument itself
Enumerator instructions	Visually distinct from what is read aloud
Skip instructions	At the branching question, in bold
CAPI screens	One idea per screen; no scrolling grids

Deciding variable names at design time rather than at cleaning time saves days and prevents the classic mismatch where the questionnaire, the export and the data dictionary each call the same item something different.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Seven

Sampling for Surveys

ImpactMojoSurvey Design 101www.impactmojo.in

Why Sample

A good sample stands in for millions

You rarely need to ask everyone. A well-drawn sample of a few thousand can describe a population of millions — the principle behind NFHS, PLFS and every credible poll. The magic is not size, it is representativeness.

Data Literacy 101 covers the basics of sampling; here we go deeper into the choices a survey designer actually makes in the field.

Belief	Reality
‘We need 10% of the population’	Precision depends on n, not the share sampled
‘Bigger is always better’	A biased sample gets more confidently wrong
‘We surveyed everyone available’	That is a convenience sample, not a sample

Sampling fraction almost never matters at the scales development surveys work at. A sample of 1,000 gives about the same precision for a district of 200,000 as for a state of 60 million.

ImpactMojoSurvey Design 101www.impactmojo.in

Probability Methods

Let chance choose — it removes bias

Method	How	Use when
Simple random	Every unit equal chance	You have a full list
Systematic	Every k-th unit from a list	Ordered list, no hidden cycle
Stratified	Split into groups, sample each	Must represent subgroups
Cluster	Sample whole groups (villages)	People geographically spread
Multistage	Clusters, then units within	Large national surveys (NFHS)

Only probability sampling lets you compute a margin of error and generalise honestly. Everything else can describe, but not infer.

Method	Needs	Fails when
Simple random	A complete list	No usable frame exists
Systematic	An ordered list	The list has a hidden cycle
Stratified	Strata known in advance	Strata variable is missing or wrong
Cluster / multi-stage	Village or PSU list only	Design effect ignored in analysis
PPS	Population sizes per unit	Sizes are stale (2011 Census)

Almost every field survey in South Asia is multi-stage: PSUs (villages or urban blocks) selected with probability proportional to size, then households listed and drawn within. That design has a design effect, and ignoring it understates the margin of error — often by a factor of 1.5 to 2.5.

ImpactMojoSurvey Design 101www.impactmojo.in

The Sampling Frame

Your survey is only as good as its list

Sampling frame

The actual list of units from which you draw your sample — the operational stand-in for the target population. Whoever the frame omits, your survey can never reach.

TARGET POPULATION: whom you want to learn about

→

SAMPLING FRAME: the list you can actually draw from

→

SAMPLE: who you end up selecting

→

RESPONDENTS: who actually answers

The frame is often the weakest link. A list of phone numbers is not a list of citizens; a voter roll misses children and recent migrants. Coverage error starts here.

Frame	Systematically omits
Ration-card list	The unlisted, recent migrants, the newly poor
SHG membership register	Non-members — often the poorest
Voter roll	Under-18s, the recently moved
Programme beneficiary list	Everyone the programme did not reach

A beneficiary list is the most tempting frame and the most dangerous. It answers questions about participants only, and cannot support a statement about the population — including the statement that the programme reached the right people.

Where no good frame exists, the standard fix is a household listing exercise in the selected PSUs. It costs a fieldwork round and it is usually worth it.

ImpactMojoSurvey Design 101www.impactmojo.in

Stratify To Compare

Over-sample small groups on purpose

If you need reliable estimates for a small subgroup — say, a tribal block or a minority community — a proportional sample may yield too few of them. Stratify and deliberately over-sample that group, then correct with weights.

Design choice: representing every district equally, or every person equally? You usually can't have both at once — decide which your analysis needs, and weight accordingly.

Situation	Do
Small subgroup, need its own estimate	Over-sample the stratum; weight back
Comparing two blocks	Allocate roughly equally, not proportionally
Rare population	Screen, or use a specialised design

Over-sampling is a design decision that must be recorded and undone at analysis. An over-sampled stratum reported unweighted turns a deliberate choice into a bias.

ImpactMojoSurvey Design 101www.impactmojo.in

Sample Size

How many do I actually need?

Margin of error vs sample size (95% confidence, p=0.5)

Standard sampling theory

Landmark numbers (95% CI, p=0.5): n≈384 gives about ±5%; n≈1,067 gives about ±3%. Halving the error roughly quadruples the sample — precision gets expensive fast.

n	Margin (95%, p=0.5)
384 · 1,067	±5% · ±3%

Inflate for clustering: multiply by the design effect before quoting a margin.

ImpactMojoSurvey Design 101www.impactmojo.in

Drivers Of Size

What changes the number you need

If you want…	You need…	Note
Tighter margin of error	A larger sample	±3% needs ~3× the n of ±5%
Estimates for subgroups	More per subgroup	Each cell needs its own n
To detect a small change	More power	Effect size drives this
Cluster (not random) design	An inflation factor	The 'design effect'

Cluster sampling saves travel cost but inflates required size via the design effect — people in one village resemble each other, so each adds less new information.

You want	Cost
±3% instead of ±5%	About 3× the sample
A separate estimate per district	A full sample per district
To detect a 3-point change, not 10	Roughly 10× the sample
To sample clusters, not households	Inflate n by the design effect

Most underpowered surveys became underpowered when someone added a subgroup breakdown late. Every extra cell needs its own n; the total is not shared out.

ImpactMojoSurvey Design 101www.impactmojo.in

Bias Beats Size

The errors a bigger sample can't fix

Coverage bias: the frame systematically misses people
Non-response bias: refusers differ from responders
Selection bias: the draw method favours some over others

A bigger biased sample is just a more confident wrong answer. Sample size shrinks sampling error only — never bias. Fix bias with design, not with n.

Bias	A bigger n does
Coverage	Nothing — the missing were never in the frame
Non-response	Nothing — unless the refusers change
Selection	Nothing — it repeats the same skew
Sampling error	Reduces it — this one only

The 1936 Literary Digest poll had 2.4 million responses and got the US election wrong; a sample of a few thousand, drawn properly, got it right. Size is a precision instrument, not an accuracy one.

ImpactMojoSurvey Design 101www.impactmojo.in

Weights

Why survey results come 'weighted'

When groups are over-sampled by design or respond at different rates, surveys apply weights so each respondent represents the right number of real people. Unit data from NFHS or PLFS gives wrong totals if you ignore the weights.

Weighting is the bridge back to the population. (Data Literacy 101 covers how to read weighted estimates; here the point is to plan for them at the design stage.)

Weight component	Corrects for
Design weight	Unequal probability of selection
Non-response adjustment	Groups that answered less
Post-stratification	Drift from known population totals

NFHS and PLFS unit-level data are weighted, clustered and stratified. Computing a simple unweighted mean on them gives a number that is wrong twice over — wrong level, and a standard error that is too small.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Eight

Modes of Data Collection

ImpactMojoSurvey Design 101www.impactmojo.in

Mode Matters

How you ask changes what you hear

The mode — face-to-face, phone, web, self-completion — is not a neutral pipe. It shapes who you reach, how long they'll stay, how honest they are, and what you can ask. Choose it as carefully as the questions.

Mode

The channel through which questions are delivered and answers captured. Each mode carries its own coverage, cost, length limit and pattern of response bias.

Mode shapes	How
Who you reach	Phone ownership, internet access, being at home
How long they stay	Phone drops off sharply past 20 minutes
How honest they are	Less desirability bias without a human present
What you can ask	Show-cards, observation and long grids need presence

Choose the mode before writing questions, not after. An instrument designed for face-to-face rarely survives being read down a phone line intact.

ImpactMojoSurvey Design 101www.impactmojo.in

Face-to-Face & CAPI

The South Asian workhorse

Most rigorous development surveys in South Asia still rely on face-to-face interviews, increasingly via CAPI — computer-assisted personal interviewing — on a tablet or phone running KoboToolbox, ODK or SurveyCTO.

Strengths

Reaches low-literacy & offline areas
Long, complex instruments possible
Built-in skips & range checks
Direct measurement (height, weight, GPS)

Costs

Expensive — travel & staff
Slow to field at scale
Interviewer effects & bias
Device & charging logistics

CAPI in practice	Note
KoboToolbox · ODK · SurveyCTO	Offline-capable; sync when a signal appears
Enforced routing	Removes the largest source of paper error
GPS & timestamps	Enable back-checks and duration monitoring
Device logistics	Charging, theft, breakage, data plans

Budget for spare devices and power banks explicitly. Fieldwork in areas with unreliable electricity loses more days to charging than to weather.

Interview duration per enumerator is the single most informative quality metric CAPI gives you free. An enumerator consistently 40% faster than the team is not more efficient.

ImpactMojoSurvey Design 101www.impactmojo.in

Phone & IVR

Fast and cheap — but who picks up?

Phone surveys and IVR (interactive voice response, automated calls) are cheap, fast and safe in a crisis — widely used during COVID-19. But they must be short, and they systematically miss people without phones.

In rural South Asia, phone ownership skews male, younger, richer and more urban. A phone frame quietly excludes the poorest women in the remotest places — precisely those many programmes target.

Phone survey	Constraint
Length	15–20 minutes is the practical ceiling
Who answers	Phone owner ≠ household; often the man
Who is missed	No phone, no network, no charge, no literacy for IVR
Response rates	Low; and refusers differ systematically

COVID-19 normalised phone surveys and also demonstrated their skew: they reached the connected and reasonably well-off far better than the people whose situation the surveys were meant to describe.

ImpactMojoSurvey Design 101www.impactmojo.in

Web

Cheapest reach, narrowest coverage

Web surveys cost almost nothing to send to thousands and allow rich self-completion of sensitive items. But for general populations in South Asia, internet access is far from universal — coverage error is severe.

Web works well for connected, literate audiences — staff, professionals, urban youth. It badly misrepresents the general public, especially the poor and elderly.

Web works for	Web fails for
Staff, students, professionals	General rural populations
Lists with verified emails/numbers	Open links shared onward
Sensitive self-completion	Anyone without a device or data

An open web link is not a sample. It has no frame, no known selection probability and no way to weight — so it supports description of respondents and nothing about a population.

ImpactMojoSurvey Design 101www.impactmojo.in

Trade-Offs

Four modes, four profiles

Mode	Cost	Coverage	Length	Best for
Face-to-face / CAPI	High	Broadest	Long	Rigorous, rural, complex
Phone / IVR	Low	Phone owners	Short	Speed, crises, monitoring
Web	Lowest	Connected only	Medium	Staff & literate audiences
Self-completion	Medium	Literate	Medium	Sensitive topics

There is no best mode — only the right trade-off for your population, budget, topic and timeline. Match the mode to the people you must reach.

Mode	Roughly, per completed interview	Typical response rate
Face-to-face / CAPI	Highest	Highest
Phone / IVR	Low	Low to moderate
Web	Lowest	Lowest

Read the cost column with the coverage column, never alone. The cheapest completed interview is not cheap if the people it reaches are not the people you needed.

ImpactMojoSurvey Design 101www.impactmojo.in

Coverage By Mode

Which mode reaches whom

Illustrative: share of a rural population reachable, by mode

Illustrative pattern for rural South Asia

Illustrative, not measured — but the pattern is real: as cost falls, coverage narrows. Cheaper modes systematically shed the hardest-to-reach.

As cost falls	Coverage narrows to
Face-to-face → phone → web	Everyone → phone owners → the connected

The people excluded by the cheaper modes are usually the subject of the survey.

ImpactMojoSurvey Design 101www.impactmojo.in

Mode Effects

The same question, a different answer

Mode effect

A systematic difference in responses caused purely by the mode of administration — for example, people report stigmatised behaviour more honestly to a screen than to a human interviewer's face.

This wrecks comparability. If round one was face-to-face and round two was by phone, an apparent change may be a mode effect, not real. Hold the mode constant across rounds wherever you can.

Item type	Reported more honestly to
Stigmatised behaviour	A screen, not a person
Socially approved behaviour	Over-reported to a person
Complex recall	A person who can probe
Household detail	A person who can see the dwelling

Mode effects are why a mid-project switch from face-to-face to phone can produce an apparent trend that is entirely artefact. If you must switch, overlap the modes for one round so the shift can be estimated.

ImpactMojoSurvey Design 101www.impactmojo.in

Mixed Mode

Combining modes — carefully

Mixed-mode designs (web first, phone follow-up, face-to-face for refusers) can boost response and cut cost. But they blend the modes' different biases, so disentangling real change from mode effects gets harder.

If you mix modes, record the mode for every response so you can test and adjust for mode effects in analysis — never leave it invisible.

Mixed-mode gains	Mixed-mode costs
Higher response	Blended, hard-to-separate biases
Lower average cost	Mode effects confounded with real differences
Reaches refusers	Complex weighting

If you mix modes, record the mode on every record and test for mode effects before pooling. Reporting a single pooled figure without that check is where mixed-mode designs go wrong.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Nine

Translation & Cultural Adaptation

ImpactMojoSurvey Design 101www.impactmojo.in

Multilingual Reality

South Asia is not monolingual

A survey in Bihar may need Hindi, Maithili, Bhojpuri and Urdu; one across India, a dozen languages and scripts. The instrument respondents actually hear is the translated one — so translation is instrument design, not an afterthought.

languages in the Eighth Schedule of India's Constitution

100+

languages with substantial speaker bases

instrument must mean the same thing in all of them

Decision	Consequence
Which languages to field in	Who can answer at all
Who translates	Whether jargon returns
Whether translation is tested	Whether it measures the same thing
On-the-spot oral translation	Every enumerator becomes a translator

The last row is the common reality and the most damaging. If Maithili is not in the instrument but respondents speak Maithili, each enumerator is improvising a different instrument, and no record of it exists.

ImpactMojoSurvey Design 101www.impactmojo.in

Equivalence Is The Goal

Translate meaning, not just words

A literally correct translation can still measure something different. The aim is conceptual equivalence — the translated question evokes the same concept, with the same difficulty and connotations, as the original.

‘Do you feel empowered?’ has no clean equivalent in many languages. Adapt to a locally meaningful idea (‘Can you decide this on your own?’) rather than translating the abstract word.

Equivalence type	Question to ask
Semantic	Do the words mean the same?
Conceptual	Does it evoke the same idea?
Difficulty	Is it equally easy to answer?
Connotation	Does it carry the same social charge?

‘Depressed’ translates semantically into most Indian languages and conceptually into rather few. Validated mental-health instruments exist in Indian languages precisely because the literal translation does not work.

ImpactMojoSurvey Design 101www.impactmojo.in

Back-Translation

The standard quality check

Back-translation

One translator renders the source into the target language; an independent second translator, blind to the original, translates it back. Comparing the back-translation to the original surfaces meaning lost or shifted.

SOURCE: English question

→

FORWARD: translate to Hindi (translator A)

→

BACK: Hindi → English (translator B, blind)

→

RECONCILE: compare, fix discrepancies

Step	Who
Forward translation	Translator A, into the target language
Back-translation	Translator B, blind to the original
Comparison	The design team, against the source
Reconciliation	All three, resolving each discrepancy

Back-translation catches literal drift well. It does not catch a translation that is accurate but stilted, formal, or in a register no respondent uses — which is the commoner failure.

ImpactMojoSurvey Design 101www.impactmojo.in

Beyond Back-Translation

Committee & team approaches

Back-translation catches literal errors but can miss awkwardness and cultural misfit. A committee or team approach — bilingual experts, translators and field staff reviewing together — produces more natural, usable wording.

Best practice combines both: forward translation, expert committee review, back-translation, then cognitive testing with real speakers of the target language.

Method	Catches
Back-translation	Literal errors and omissions
Committee review	Register, naturalness, cultural misfit
Field-staff review	What is actually sayable aloud
Cognitive pretest in the target language	How respondents really interpret it

Do the cognitive pretest in every fielded language, not only the source. A question can pass in Hindi and fail in Bhojpuri for reasons no amount of desk review will surface.

ImpactMojoSurvey Design 101www.impactmojo.in

Local Categories

Use categories people recognise

Imported

Western occupation lists, ‘nuclear vs extended family’, or income bands that ignore how people are actually paid (daily wage, in kind, seasonal).

Local

Categories drawn from how the community describes work, household and kinship — refined during pretesting in each setting.

Caste, kinship and livelihood categories vary enormously across regions. A code list that fits Tamil Nadu may be meaningless in Assam.

Imported category	Local reality
‘Employed / unemployed’	Daily wage, seasonal, unpaid family work, several at once
‘Nuclear / extended family’	Joint households that split and re-form seasonally
Monthly income bands	Paid daily, weekly, in kind, or at harvest
‘Head of household’	Contested; often nominal rather than actual

Build the option list from pilot open-ended answers rather than from an international template. Then map your local categories to the standard ones at analysis, keeping both.

ImpactMojoSurvey Design 101www.impactmojo.in

Units & Measures

Ask in units people use

Land in local units (bigha, katha, acre) — then convert
Time by the agricultural or festival calendar, not just dates
Quantities in market units (a seer, a tin, a bundle)
Money as it is actually earned — daily, weekly, per task, in kind

Forcing respondents to convert into unfamiliar units in their heads adds error. Capture the local unit, record the conversion factor, and convert later in analysis.

Ask in	Convert to
Bigha, katha, guntha	Acres or hectares — with a district-specific factor
Seer, tin, bundle	Kilograms, at listed local weights
‘Since kharif harvest’	A dated reference period
Daily or piece-rate wage	A monthly equivalent, showing the assumption

Local land units vary by district and sometimes by village. Build the conversion table into the CAPI form keyed to location, and store the raw local value as well as the converted one.

ImpactMojoSurvey Design 101www.impactmojo.in

Calendars & Anchors

Anchor recall in the local year

‘In the last 12 months’ is abstract; ‘since last Diwali’ or ‘since the kharif harvest’ is vivid. A local events calendar built into the instrument sharpens recall and aligns reference periods across respondents.

Build the calendar of festivals, harvests and landmark local events with field staff before fieldwork — it is one of the cheapest accuracy gains available.

Anchor	Fixes recall to
‘Since Diwali’	A shared, vivid date
‘Since the kharif harvest’	The agricultural cycle
‘Since the school reopened’	A local, verifiable event

Build the events calendar during the pilot with local staff, and print it in the field manual. The gain in recall accuracy is one of the cheapest available in survey design.

ImpactMojoSurvey Design 101www.impactmojo.in

Adapt, Then Standardise

Same concept, locally clothed

The discipline is to adapt the wording, examples and units to each setting while holding the concept and the indicator identical. That is what keeps a multilingual survey both locally meaningful and nationally comparable.

Translate the meaning, standardise the measurement. Lose either and the comparison collapses.

— a cross-cultural survey principle

Hold identical	Adapt freely
The concept	Wording and idiom
The indicator definition	Examples used
The reference period	Units, then converted
The response scale	Labels in the local language

This is the line that keeps a multilingual survey both locally meaningful and nationally comparable. Cross it in the wrong direction — adapting the indicator instead of the wording — and the state-level total means nothing.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Ten

Pretesting, Piloting & Fieldwork Quality

ImpactMojoSurvey Design 101www.impactmojo.in

Test Before You Trust

No instrument is ready on the first draft

Between ‘finished’ questionnaire and real fieldwork sit two distinct checks: pretesting (do the questions work?) and piloting (does the whole operation work?). Skipping them is the most expensive false economy in survey work.

PRETEST: cognitive interviews on the questions

→

REVISE: fix wording, options, routing

→

PILOT: a dry run of the whole operation

→

FINALISE: lock the instrument & protocol

	Pretest	Pilot
Asks	Do the questions work?	Does the operation work?
Scale	8–20 respondents	A realistic mini-round
Method	Think-aloud, probing	Full protocol, end-to-end
Output	Rewritten questions	Fixed routing, timings, logistics

These are two different activities and skipping either is the most expensive economy in survey work. A wording problem found in analysis costs the indicator; found in a pretest it costs an afternoon.

ImpactMojoSurvey Design 101www.impactmojo.in

Cognitive Interviews

Find out how respondents actually think

Cognitive interviewing

A pretesting method where a few respondents answer questions while thinking aloud, and the interviewer probes how they understood the words, recalled the facts, and chose their answer.

It reveals what a draft cannot: that ‘household’ means different things to different people, that a recall window is impossible, that an option everyone needs is missing. A dozen good cognitive interviews save a thousand bad responses.

Probe	Reveals
‘What did that question mean to you?’	Comprehension
‘How did you work out the answer?’	Recall strategy and estimation
‘Was any option hard to choose between?’	Whether the scale discriminates
‘Would that be awkward to answer honestly?’	Sensitivity and desirability

Eight to twelve cognitive interviews find most serious comprehension problems. Recruit them to look like your respondents — testing on colleagues finds nothing, because colleagues share your assumptions.

ImpactMojoSurvey Design 101www.impactmojo.in

The Pilot

Rehearse the entire operation

Run the full instrument end-to-end in real conditions
Time it — is the interview length tolerable?
Test the CAPI routing, GPS, and data sync
Check logistics: travel, access, consent, refusal handling
Review the pilot data: are variables populating as expected?

The pilot tests the system, not just the questions: sampling, fieldwork, devices, supervision and data flow, all at once, before they fail at scale.

Pilot check	Pass condition
Interview length	Within the planned ceiling for most respondents
Routing	Every branch reached by at least one case
Sync & GPS	Data arrives complete and located
Consent & refusal	Handled per protocol, recorded
Pilot data run through the analysis	Every dummy table populates

Run the pilot data all the way through your analysis scripts. It is the only way to discover that a variable you need was never captured while there is still time.

ImpactMojoSurvey Design 101www.impactmojo.in

Enumerator Training

Your data is collected by people

However good the instrument, it is administered by enumerators. Standardised training — on the questions, the skips, neutral probing, consent and edge cases — is what makes 'the same question asked the same way' actually true in the field.

Read every question verbatim — no improvised paraphrase
Probe neutrally; never suggest an answer
Handle ‘don't know’ and refusals correctly
Practice with role-plays and mock interviews before going live

Training must cover	Or you get
Each question’s intent	Enumerators inventing interpretations
Neutral probing	Leading, and answers that match expectations
Skips and edge cases	Silent routing errors
Consent and refusal	Ethics failures and coerced participation
Practice interviews, observed	Errors discovered in the field

Include a written test and a supervised practice interview before certifying anyone to field. It is the point at which the person who cannot do the job is cheapest to identify.

ImpactMojoSurvey Design 101www.impactmojo.in

Interviewer Effects

Who asks can change the answer

Interviewer effects

Systematic differences in responses arising from the interviewer — their gender, caste, manner, or subtle cues — rather than from the respondent's true situation.

On gender-based violence or reproductive health, a woman enumerator usually elicits franker answers from women. Match interviewer characteristics to the topic, and standardise manner through training.

Interviewer attribute	Where it matters most
Gender	Violence, reproductive health, autonomy
Caste / community	Discrimination, access to services
Age	Deference and youth topics
Manner and dress	Whether the respondent reads you as an official

Match interviewer to respondent where the topic requires it, and record enumerator ID on every interview so the effect can be estimated afterwards. Without the ID it is invisible.

ImpactMojoSurvey Design 101www.impactmojo.in

Social Desirability

People answer how they think they should

Social-desirability bias

The tendency to give answers that present the respondent favourably — over-reporting good behaviour (hand-washing, school attendance) and under-reporting the stigmatised (drinking, violence, open defecation).

Normalise: “Many people…” before a sensitive item
Use private self-completion for stigmatised topics
Assure confidentiality — and mean it
Ask about behaviour, not identity, where possible

Direction	Typical items
Over-reported	Hand-washing, toilet use, school attendance, voting
Under-reported	Alcohol, tobacco, violence, income, caste practice

Mitigations: normalising preambles, self-completion for the sensitive block, list experiments where the design supports them, and never reading a socially approved option first.

Where feasible, validate against something observed — a soap-and-water check at the handwashing station beats a reported figure and often differs from it sharply.

ImpactMojoSurvey Design 101www.impactmojo.in

Back-Checks

Verify the fieldwork while it happens

Supervisors re-contact a sample of respondents (back-checks) and observe live interviews (spot-checks) to confirm the interview happened, key answers match, and protocol was followed — catching error and fabrication early.

Back-check

Re-ask a few stable questions by phone or revisit; compare to the original to detect errors or faked interviews.

Spot-check

A supervisor sits in on live interviews to observe technique, neutrality and consent in real time.

Check	Sample	Catches
Back-check call/visit	5–10% of interviews	Fabrication; key answers not matching
Spot-check (live observation)	A few per enumerator	Protocol drift, leading, skipped items
High-frequency data checks	All records, daily	Duration outliers, heaping, DK spikes

These only work while fieldwork is running. Run the daily checks from day one, not at the end — the point is to correct an enumerator in week one rather than discard their week-six data.

ImpactMojoSurvey Design 101www.impactmojo.in

Non-Response

Chase response — and study the refusers

Non-response — refusals, not-at-homes, ineligibles — bites hardest when those who don't answer differ from those who do. The poorest, the busiest and the most private are often the hardest to reach, biasing the result.

Make repeat visits at different times of day
Track and report the response rate honestly
Compare responders to the frame to test for bias
Adjust with weights — but weights can't fully cure deep non-response

Record	Why it matters
Refusal	May differ systematically from responders
Not at home after 3 visits	Working households under-represented
Ineligible	Affects the frame, not the response rate
Partial completion	Item non-response, treated separately

Keep a contact record for every attempt. It is what lets you compute a real response rate, describe the refusers using frame variables, and defend the estimate — or admit its limit.

ImpactMojoSurvey Design 101www.impactmojo.in

Section Eleven

Ethics, Data & Practice

This section covers	Slide
Why ethics is respect made operational	92
Consent as a floor, not a formality	93
What the DPDP Act, 2023 asks of you	94
De-identification and small-cell risk	95
Clean data, documentation and tools	96–97

These are not the closing formalities of a survey. Consent wording, retention and de-identification are design decisions that constrain the instrument, and they belong in the protocol alongside the sample.

ImpactMojoSurvey Design 101www.impactmojo.in

Behind Every Row

Every respondent is a person, not a data point

In a survey, each row is someone who gave you their time and trust — often a poor, busy, over-surveyed person with little power over how their answers are used. Survey ethics is respect made operational, not a compliance form.

The respondent owes you nothing. Everything they give, they give as a gift — treat it like one.

— a field ethics principle

Respondent gives	You owe them
30–60 minutes of time	An instrument with nothing spare in it
Personal information	Storage, limits, deletion
Trust in a stranger	Honesty about what happens next
Nothing in return, usually	At minimum, findings shared back

Over-surveyed communities notice when nothing ever comes back. The falling response rates that result are both an ethics problem and a data-quality problem, and they are the same problem.

ImpactMojoSurvey Design 101www.impactmojo.in

Informed Consent

Consent is the floor, not a formality

State plainly what you collect and why
Explain how it will be stored, used, shared, and for how long
Make clear they may refuse or stop, with no penalty
Deliver it in a language and form they genuinely understand

A thumbprint on an unexplained form is not consent. For children and other vulnerable respondents, extra safeguards and guardian consent apply.

Consent must state	Commonly skipped
What is collected and why	Rushed into one sentence
How it is stored, shared, for how long	Omitted entirely
That refusal carries no penalty	Implied but not said
Who to contact afterwards	No contact given

Consent for a child or an adult who cannot consent independently needs the guardian’s permission and the person’s own assent, recorded separately.

Written consent is not automatically better than oral. Where literacy is low or a signature signals officialdom, recorded oral consent to a read script is both more ethical and more honest.

ImpactMojoSurvey Design 101www.impactmojo.in

Data Protection & DPDP

India's data law reaches your survey

The Digital Personal Data Protection Act, 2023 sets duties for anyone handling personal digital data in India — including NGOs and researchers running CAPI surveys on tablets and phones.

Collect only what you need, for a stated purpose (purpose limitation)
Obtain free, informed, specific consent
Protect data with reasonable security safeguards
Stronger protections for children's data

‘We're a small NGO’ is not an exemption. Know your obligations before the first interview, not after a breach.

DPDP duty	In a CAPI survey
Purpose limitation	Collect only what the protocol names
Notice & consent	The consent script, in the respondent’s language
Security safeguards	Device encryption, access control, secure sync
Retention limits	A deletion date, and someone responsible for it
Rights of the data principal	A route to correction and erasure

The Act applies to NGOs and researchers handling personal digital data in India, not only to companies. A tablet full of named household records is squarely within it.

This is an orientation, not legal advice. Where a survey handles sensitive categories or children’s data at scale, get the protocol reviewed properly.

ImpactMojoSurvey Design 101www.impactmojo.in

Keep It Safe

Anonymisation is harder than deleting names

Removing names is not enough. Village + age + caste + occupation can re-identify one person in a small area. Survey data needs deliberate de-identification, encryption and access control — especially with GPS coordinates attached.

Direct IDs

Name, Aadhaar, phone, GPS — separate & protect

Quasi-IDs

Age + place + caste can re-identify — aggregate or coarsen

Identifier	Risk
Name, phone, exact GPS	Direct — remove from the analysis file
Village + age + caste + occupation	Indirect — can single out one person
Small-cell tables	A cell of 1 or 2 is an identification
Free-text answers	Often contain names and places

Keep identifiers in a separate, encrypted linking file with restricted access, and work from a de-identified analysis file. Suppress or aggregate cells below a threshold before publishing.

ImpactMojoSurvey Design 101www.impactmojo.in

From Survey To Data

Design for clean data, then document it

A good instrument hands the analyst clean, well-named data. Build validation into the CAPI form, keep the raw export untouched, and ship a data dictionary so every variable, code and unit is self-explanatory months later.

A survey is not done when fieldwork ends. It is done when someone else can open your documented dataset and understand exactly what every column means and how it was made.

Practice	Payoff
Validation in the CAPI form	Errors caught at the doorstep
Raw export kept untouched	Every cleaning step is reproducible
Cleaning done in a script, not by hand	You can say what you changed and why
Data dictionary shipped with the data	The file survives its author

Manual edits in a spreadsheet are the point at which a dataset stops being reproducible. Nobody, including you in six months, can reconstruct what was changed.

ImpactMojoSurvey Design 101www.impactmojo.in

The Tools

Software for modern field surveys

Tool	Good for	Note
KoboToolbox	Humanitarian & NGO CAPI	Free, offline, widely used
ODK (Open Data Kit)	Open-source mobile collection	Free, flexible, technical
SurveyCTO	Rigorous research surveys	Paid; strong quality controls
Survey Solutions	Large official surveys	Free (World Bank)

All build skip logic, range checks and offline collection into the instrument — turning design discipline into fewer field errors. Tools matter less than the design habits behind them.

Tool	Best fit	Watch
KoboToolbox	NGO and humanitarian CAPI	Free, offline, large user base
ODK	Open-source, self-hosted	Flexible; needs technical capacity
SurveyCTO	Rigorous research at scale	Paid; strong quality-control features
CSPro	Census-style large operations	Heavier, statistical-office lineage

Choose for the field team, not the analyst. The right tool is the one your enumerators can use offline on the devices you actually own, with a support person you can reach.

ImpactMojoSurvey Design 101www.impactmojo.in

Keep Learning

A short, serious reading list

Survey Methodology — Groves, Fowler, Couper et al. (the TSE bible)
Survey Research Methods — Floyd J. Fowler (concise, practical)
Improving Survey Questions — Floyd J. Fowler (question design)
Asking Questions — Bradburn, Sudman & Wansink
DIME / J-PAL / IPA field guides — practical survey toolkits

Pair this deck with ImpactMojo's Data Literacy 101, Qualitative Methods and Research Ethics courses for the full toolkit.

Read	For
Groves et al., Survey Methodology	Total Survey Error, in full
Fowler, Improving Survey Questions	Question wording, practically
Bradburn, Sudman & Wansink, Asking Questions	Worked examples across topic types
NFHS · PLFS instruments and reports	How this is done at national scale

Reading a live national instrument and its report side by side is the fastest available lesson: you can see exactly how each published table traces back to a numbered question.

ImpactMojoSurvey Design 101www.impactmojo.in

The Takeaways

If you remember six things

Start from the decision — objectives before questions
One question, one idea — clear, specific, neutral
The questionnaire is the measurement — wording is data
Manage total survey error — not just sample size
Pretest, pilot, back-check — never trust a first draft
Behind every row is a person — consent and care

Takeaway	What it rules out
Objectives before questions	Drafting an instrument first
One question, one idea	‘Clean and well-staffed?’
The questionnaire is the measurement	Treating wording as presentation
Manage total error	Reporting only the sampling margin
Pretest and pilot	Trusting a first draft
The respondent is a person	An instrument with spare questions in it

None of these needs a bigger budget. All of them need decisions taken before fieldwork, which is the only time most survey errors can still be prevented.

ImpactMojoSurvey Design 101www.impactmojo.in

Survey Design 101 · Complete

Now go design
a survey worth answering.

A trustworthy survey is built, not improvised — from the decision it serves to the last back-check in the field. Explore the rest of the ImpactMojo 101 Series, free forever.

More 101 Courses Explore ImpactMojo Dataverse

Continue your learning

Survey Design Studio Sampling Design Studio

CC BY-NC-ND 4.0·Free Forever·ImpactMojo 101 Series