fullscreen
ImpactMojoSurvey Design 101www.impactmojo.in
ImpactMojo 101 Series · Free Forever
Survey
Design
101
Designing, Writing & Running Trustworthy Field Surveys — a Foundational Course for Development & MEL Practitioners in South Asia
Research-BackedSouth Asia Focus100 SlidesFree Access
ImpactMojoSurvey Design 101www.impactmojo.in
What We Cover
01
What Surveys Are & When to Use One
Slides 3–10
02
The Survey Lifecycle
Slides 11–19
03
From Concept to Question
Slides 20–27
04
Writing Good Questions
Slides 28–36
05
Question & Response Types
Slides 37–45
06
Questionnaire Structure & Flow
Slides 46–54
07
Sampling for Surveys
Slides 55–63
08
Modes of Data Collection
Slides 64–72
09
Translation & Cultural Adaptation
Slides 73–81
10
Pretesting, Piloting & Fieldwork Quality
Slides 82–90
11
Ethics, Data & Practice
Slides 91–99
ImpactMojoSurvey Design 101www.impactmojo.in
01
Section One
What Surveys Are & When to Use One
ImpactMojoSurvey Design 101www.impactmojo.in
A survey is a measurement instrument
A survey systematically collects standardised information from a sample of people, using the same questions in the same way, so answers can be counted and compared. Its power is standardisation; its discipline is design.
Survey
A method of gathering quantitative information by asking a defined sample of respondents a fixed set of questions under controlled conditions, so the results can be aggregated and generalised to a population.
A survey is not 'a list of questions'. It is an instrument that turns a concept into a number — and like any instrument, a badly built one produces confident nonsense.
ImpactMojoSurvey Design 101www.impactmojo.in
Surveys answer 'how many' and 'how much'
A survey fits when…
  • You need numbers you can generalise
  • The concept is well understood already
  • You want to compare groups or track change
  • The population is too large to ask everyone
A survey misfits when…
  • You don't yet know the right questions
  • You need depth, meaning or 'why'
  • The topic is too sensitive for fixed answers
  • Good administrative data already exists
Reach for a survey to measure something you already understand — not to discover what matters.
ImpactMojoSurvey Design 101www.impactmojo.in
Survey, qualitative or administrative?
SurveyQualitativeAdministrative
AnswersHow many, how muchWhy, how, meaningWhat the system records
StrengthGeneralisable, comparableDepth, context, mechanismCheap, continuous
SampleRepresentative sampleSmall, purposiveEveryone served
CostHigh — fieldworkModerateAlready collected
Blind spotMisses the 'why'Cannot generaliseMisses who is not served
These are partners. A survey tells you that outcomes differ; qualitative work tells you why; admin data tells you what your programme already knows for free.
ImpactMojoSurvey Design 101www.impactmojo.in
Don't survey what already exists
Before commissioning a survey, check whether the answer is already in secondary data — the Census, NFHS, PLFS/NSS, or your own HMIS, UDISE+ and MGNREGA records. Re-collecting wastes budget and, worse, wastes respondents' time.
NFHS-5
Health, nutrition, fertility down to district
IIPS / MoHFW, 2019–21
PLFS
Employment & unemployment, annual since 2017–18
MoSPI
HMIS
Your own facility data, updated monthly
Rule of thumb: a new survey is justified only for what existing data genuinely cannot answer at the level you need.
ImpactMojoSurvey Design 101www.impactmojo.in
What a good survey buys you
  • Generalisation: a few thousand can describe millions
  • Comparability: identical questions across people, places, time
  • Quantification: attach a number and a margin of error to a claim
  • Coverage: reach groups and topics no register captures
  • Transparency: a documented instrument others can scrutinise
ImpactMojoSurvey Design 101www.impactmojo.in
What a survey can never give you
  • Meaning: fixed options flatten messy human reality
  • The unasked: a closed question can only return its own choices
  • Truth on tap: people misremember, please, and self-present
  • The missing: whoever the frame and fieldwork miss stays invisible
A survey measures what you thought to ask, from whoever you managed to reach. Both boundaries are design choices — make them deliberately.
ImpactMojoSurvey Design 101www.impactmojo.in
The respondent is doing you a favour
Asking a question is asking for time, attention and trust from someone who owes you none. Design as if you will have to sit through your own survey.
— a field researcher's rule of thumb
Every needless question, every confusing scale, every minute of length is a cost paid by a real person — often a busy, poor, over-surveyed one. Respect is good methodology.
ImpactMojoSurvey Design 101www.impactmojo.in
02
Section Two
The Survey Lifecycle
ImpactMojoSurvey Design 101www.impactmojo.in
Eight stages from question to use
01
OBJECTIVES: the decision the survey must serve
02
INDICATORS: what to measure, defined precisely
03
INSTRUMENT: the questionnaire that captures it
04
SAMPLE: whom to ask, and how many
05
FIELDWORK: collecting data well in the field
06
DATA: clean, weighted, documented
07
ANALYSIS: turning answers into findings
08
USE: feeding a real decision
Most survey failures are designed in at the start — not in the analysis. Time spent on the first four stages is repaid many times over in the last four.
ImpactMojoSurvey Design 101www.impactmojo.in
Objectives before questions
The first question is never 'what should we ask?' but 'what decision will this survey inform?' Work backwards from the decision to the findings you need, then to the indicators, then to the questions.
01
DECISION: where to scale the nutrition programme?
02
FINDING NEEDED: which blocks have highest stunting?
03
INDICATOR: % children under 5 stunted, by block
04
QUESTION: child's height & age, measured directly
If you cannot name the decision a question serves, cut the question. 'Nice to know' is the enemy of a usable survey.
ImpactMojoSurvey Design 101www.impactmojo.in
Sharpen vague aims into answerable questions
Too vague
“Understand women's livelihoods in the district.” — unmeasurable; no population, no indicator, no time frame.
Answerable
“What share of women aged 18–45 in Block X earned cash income in the last 30 days, and from what source?”
A good research question names who, what, where and over what period. If it doesn't, the questionnaire can't either.
ImpactMojoSurvey Design 101www.impactmojo.in
Decide the unit and the reference period
  • Unit of analysis: the household? the individual? the child? the enterprise? Mixing units quietly breaks your numbers.
  • Respondent vs subject: a mother may report for a child — name who answers about whom.
  • Reference period: last 7 days? last month? last year? The window changes both recall and the indicator itself.
‘Household income’ and ‘respondent's income’ are different surveys. Decide the unit before you write a single question.
ImpactMojoSurvey Design 101www.impactmojo.in
Design the analysis before fieldwork
Sketch your dummy tables — the exact tables and charts you will produce — before collecting a single response. If a planned table needs a variable no question captures, you found the gap in time to fix it.
“Which table will this question feed?” is the single best test of whether a question belongs in the instrument.
ImpactMojoSurvey Design 101www.impactmojo.in
Budget, time and team shape the design
ConstraintPushes towardWatch-out
Tight budgetShorter instrument, phone modeCoverage & quality loss
Short timelineSmaller sample, fewer itemsUnderpowered estimates
Few trained staffSimpler skips, CAPI logicEnumerator error
Remote terrainCluster sampling, offline toolsTravel cost & fatigue
Constraints are not excuses for sloppiness — they are inputs to design. A realistic small survey beats an ambitious one that collapses in the field.
ImpactMojoSurvey Design 101www.impactmojo.in
Write it down as you go
  • A survey protocol: objectives, population, sample, methods
  • A questionnaire with variable names and routing
  • A field manual for enumerators and supervisors
  • A data dictionary: every variable, code and unit
These four documents are the difference between a survey someone can trust and re-use, and a one-off whose meaning dies with the team that ran it.
ImpactMojoSurvey Design 101www.impactmojo.in
The lifecycle loops, it doesn't end
Pretesting sends you back to rewrite questions. Piloting sends you back to fix routing and sample. Analysis reveals what the next round should ask. A survey programme is a cycle of learning, not a single shot.
No questionnaire survives first contact with respondents. The good ones are the most rewritten.
— survey methodology folklore
ImpactMojoSurvey Design 101www.impactmojo.in
03
Section Three
From Concept to Question
ImpactMojoSurvey Design 101www.impactmojo.in
You cannot ask 'empowerment' directly
Most things surveys care about — empowerment, food security, trust, wellbeing — are concepts, not facts you can simply request. The questionnaire is the bridge from an abstract concept to a concrete, observable answer.
01
CONCEPT: food security
02
DIMENSIONS: access, sufficiency, anxiety
03
INDICATOR: days household ate fewer meals last month
04
QUESTION: “In the last 30 days, on how many days…?”
ImpactMojoSurvey Design 101www.impactmojo.in
Turn the concept into a measurement rule
Operationalisation
The precise rule that converts a concept into something countable: exactly what to ask, of whom, over what period, in what units, and how the answer maps to the indicator.
‘Is the household poor?’ only becomes measurable once you fix the definition — income below a line? deprived on a set of MPI indicators? — and write the exact questions that test it.
ImpactMojoSurvey Design 101www.impactmojo.in
The questionnaire IS the measurement
There is no underlying 'true survey' that a questionnaire merely records. The wording, order, options and translation are the instrument — change them and you change what you measure. Two questionnaires on the 'same' topic can yield different numbers, both correct for their wording.
This is why comparability demands identical instruments. Surveys compare answers to questions, not to reality directly — so the question must hold still.
ImpactMojoSurvey Design 101www.impactmojo.in
A map of everything that can go wrong
Groves' Total Survey Error (TSE) framework organises every source of error in a survey into two families: errors of representation (the wrong people) and errors of measurement (the wrong answers). Good design manages both at once.
Total Survey Error
The accumulated difference between a survey estimate and the true population value, from all sources — sampling and non-sampling, representation and measurement — considered together (Groves et al.).
ImpactMojoSurvey Design 101www.impactmojo.in
Two branches, many leaves
Total Survey ErrorRepresentationMeasurementCoverage error(frame gaps)Sampling error(chance of the draw)Non-response(refusals, not-at-home)Validity(asks wrong thing)Response error(recall, bias, wording)Processing error(entry, coding)
ImpactMojoSurvey Design 101www.impactmojo.in
Representation vs measurement error
Representation (wrong people)Measurement (wrong answers)
CoverageFrame misses part of the population
SamplingChance variation in who is drawn
Non-responseThose who answer differ from those who don't
ValidityQuestion doesn't capture the concept
ResponseRecall, social desirability, bad wording
ProcessingData entry & coding mistakes
Sampling error shrinks with sample size. The other five do not — they need design, not a bigger n.
ImpactMojoSurvey Design 101www.impactmojo.in
You manage error within a fixed budget
TSE's practical lesson: a fixed budget forces trade-offs. Spending everything on a huge sample (less sampling error) may starve enumerator training (more response error). Total error is what matters — not any one component.
Ask of every design choice: which errors does it reduce, which does it grow, and is the total smaller? A smaller but well-run survey often beats a larger, sloppier one.
ImpactMojoSurvey Design 101www.impactmojo.in
04
Section Four
Writing Good Questions
ImpactMojoSurvey Design 101www.impactmojo.in
Clear, specific, one idea
A good survey question is understood the same way by every respondent and the question-writer. Three tests cover most failures: is it clear, is it specific, does it ask one thing?
01
CLEAR: plain words the respondent uses
02
SPECIFIC: a definite thing, time and unit
03
ONE IDEA: a single answerable question
04
NEUTRAL: no nudge toward an answer
ImpactMojoSurvey Design 101www.impactmojo.in
One question, one idea — never two
Bad
“Is the health centre clean and well-staffed?” — clean but understaffed? The respondent can't answer; you can't interpret.
Good
“Is the health centre clean?”
then
“Is the health centre adequately staffed?” — two questions, two clean answers.
Watch for the word and — it is the commonest sign a question is secretly two.
ImpactMojoSurvey Design 101www.impactmojo.in
Don't tell the respondent what to say
Leading
“Don't you agree the new clinic has improved care?” — pushes a yes; politeness does the rest.
Neutral
“Since the new clinic opened, has the care you receive become better, stayed the same, or become worse?”
Offer the full range of answers in the question itself. A balanced stem signals that any answer is acceptable.
ImpactMojoSurvey Design 101www.impactmojo.in
Don't smuggle in assumptions
Loaded
“How much did the corrupt official demand?” — assumes corruption and a demand. “How many children do you still want?” assumes she wants more.
Open the assumption
“Did you pay anything beyond the official fee?” — then ask how much. Let a filter establish the fact before you ask about it.
A loaded question forces a false premise. Use a filter question to establish the fact before asking about it.
ImpactMojoSurvey Design 101www.impactmojo.in
Use the respondent's words, not yours
Insider language
“Do you utilise institutional delivery services and avail ANC?” — acronyms and jargon no respondent uses.
Plain
“When you were pregnant, did you go for check-ups? Where did you give birth — at home or at a facility?”
If a question needs the enumerator to explain a word, the word is wrong. Write for the least-schooled respondent in your sample.
ImpactMojoSurvey Design 101www.impactmojo.in
Anchor every question in time
VagueAnchoredWhy it's better
“Do you usually…?”“In the last 7 days…?”Defines 'usually' for everyone
“recently”“in the last 30 days”Same window for all respondents
“your income”“income last month”Fixes the unit of time
“often sick”“ill in the past 2 weeks”Countable, comparable
The right window balances recall against rarity: short enough to remember, long enough to capture the event. Health uses 2 weeks; consumption, 30 days; big purchases, a year.
ImpactMojoSurvey Design 101www.impactmojo.in
Memory fades — and bends
Recall error
The gap between what actually happened and what a respondent remembers and reports — events forgotten, dates blurred, or 'telescoped' in from outside the reference period.
  • Forgetting: small, routine events vanish first
  • Telescoping: a memorable event is pulled into the window
  • Salience: big events (a death, a wedding) recalled far better
  • Fix: shorter windows, anchoring to landmark dates & festivals
ImpactMojoSurvey Design 101www.impactmojo.in
Spot every flaw, then fix it
Before
“Don't you think you should usually spend more on your children's education and health?”
Leading + double-barrelled + vague + loaded — all four faults in one line.
After
“In the last 30 days, about how much did your household spend on schooling?” then “…on health?”
Read every question aloud and ask: am I leading, doubling up, assuming, or using jargon? Most bad questions fail more than one test at once.
ImpactMojoSurvey Design 101www.impactmojo.in
05
Section Five
Question & Response Types
ImpactMojoSurvey Design 101www.impactmojo.in
Let them speak, or give them choices?
Open-endedClosed-ended
AnswerIn their own wordsPick from fixed options
Best forExploring, unknown answersCounting, comparing
AnalysisSlow — needs codingFast — ready to tabulate
RiskVague, hard to compareMisses the unlisted answer
Use whenFew cases, discoveryMost quantitative items
A closed question can only ever return the options you wrote. Pretest open-ended first to learn the real answer set, then close it for scale.
ImpactMojoSurvey Design 101www.impactmojo.in
Exhaustive and mutually exclusive
Response options must cover every possible answer (exhaustive) and never overlap (mutually exclusive). A respondent should always find exactly one box that fits.
Overlapping
Age: 0–18, 18–30, 30–50 — where does an 18- or 30-year-old go? And what about 60+?
Clean
0–17, 18–29, 30–49, 50–64, 65+ — no gaps, no overlaps, a home for everyone.
ImpactMojoSurvey Design 101www.impactmojo.in
Measuring agreement and attitude
Likert scale
A symmetric, ordered set of response options — typically 5 or 7 points — running from strong disagreement to strong agreement, with a neutral midpoint and balanced wording on each side.
Example: Strongly disagree · Disagree · Neither · Agree · Strongly agree. Keep the steps balanced, label every point (not just the ends), and use the same scale throughout a block.
ImpactMojoSurvey Design 101www.impactmojo.in
How many points, and a midpoint?
ChoiceProCon
5 pointsSimple, fast, fits small screensLess fine-grained
7 pointsMore discriminationHarder to read aloud
With midpointAllows a genuine neutralSome hide there to avoid choosing
Forced (no midpoint)Pushes a stanceFabricates an opinion that isn't there
For face-to-face fieldwork with mixed literacy, a labelled 5-point scale read aloud usually travels best. Whatever you pick, keep it consistent across the instrument.
ImpactMojoSurvey Design 101www.impactmojo.in
Rate each, or order them?
Rating
Score each item on its own scale (rate each service 1–5). Easy, but everything can end up 'good'.
Ranking
Force an order (rank your top 3 priorities). Reveals trade-offs, but is cognitively harder — keep the list short.
Ranking more than 4–5 items overloads respondents in the field. If you must, ask only for the top few, not a full order.
ImpactMojoSurvey Design 101www.impactmojo.in
When 'don't know' is data, not laziness
A genuine ‘Don't know’ or ‘Refused’ option prevents respondents from guessing or enumerators from inventing answers. But offered too readily, it becomes an easy escape that hollows out your data.
  • Keep DK/Refused available but not read aloud as a default
  • Distinguish ‘don't know’ from ‘not applicable’ from ‘refused’
  • Code them separately — never silently as missing or zero
ImpactMojoSurvey Design 101www.impactmojo.in
Ask only what applies
A filter (gateway) question routes each respondent to only the relevant items via skip logic — so a man is never asked about his last pregnancy, and a non-farmer skips the whole farming module.
01
FILTER: Did anyone in the household farm last season?
02
IF NO → skip the entire agriculture module
03
IF YES → ask crops, area, inputs, yield
04
RESULT: shorter interview, cleaner data
ImpactMojoSurvey Design 101www.impactmojo.in
Why skip logic loves a screen
On paper, complex skips invite enumerator error — missed branches, contradictory entries. On CAPI (computer-assisted personal interviewing), the device enforces routing automatically and can run real-time consistency checks.
Build skips, ranges and logic checks into the digital instrument so impossible answers (a 6-year-old with children) are caught at the doorstep, not in cleaning weeks later.
ImpactMojoSurvey Design 101www.impactmojo.in
06
Section Six
Questionnaire Structure & Flow
ImpactMojoSurvey Design 101www.impactmojo.in
A questionnaire has a beginning, middle, end
Order is not cosmetic — it shapes answers and completion. A well-built instrument moves the respondent through a deliberate arc: welcome, easy questions, the core, the sensitive, then the close.
01
INTRO: consent, purpose, confidentiality
02
WARM-UP: easy, non-threatening items
03
CORE: the main substantive modules
04
SENSITIVE: income, health, beliefs — late
05
CLOSE: demographics, thanks
ImpactMojoSurvey Design 101www.impactmojo.in
A flow diagram of one instrument
Consentscreen / refuse → endWarm-upeasy itemsFilter: farms?gateway questionAgriculture moduleSkip moduleCore modulesmain outcomesSensitive blockincome, healthDemographics& close / thanksyesno
ImpactMojoSurvey Design 101www.impactmojo.in
Group by topic, move from general to specific
  • Cluster related items into clearly signposted modules
  • Within a module, go from general to specific
  • Keep the same response scale within a block to build rhythm
  • Don't jump topics erratically — context-switching tires and confuses
A respondent who can follow the logic stays engaged and answers more accurately. Disordered questionnaires leak data through fatigue and confusion.
ImpactMojoSurvey Design 101www.impactmojo.in
Earlier questions colour later answers
Asking about local crime, then overall life satisfaction, drags satisfaction down — the first question primes the second. Order effects are real and measurable.
Put general attitude questions before specific ones on the same topic, and keep the order identical across rounds so any priming is at least constant and comparable.
ImpactMojoSurvey Design 101www.impactmojo.in
Open with easy, relevant questions
The first questions should be easy, non-threatening and obviously relevant — they build rapport and signal that the survey is safe. Never open with income, caste or a long grid; you'll lose people at the door.
Save demographics like income and caste for late in the interview, once trust is established — not as an intimidating opening gate.
ImpactMojoSurvey Design 101www.impactmojo.in
Handle the hard topics with care
  • Place them late, after rapport is built
  • Normalise: “Many people find…” lowers the threat
  • Offer private modes (self-completion, sealed response) for stigmatised topics
  • Always allow refusal without penalty or pressure
Income, health status, violence, caste, sexual behaviour and political views all invite social-desirability bias. Design reduces it; clumsy placement amplifies it.
ImpactMojoSurvey Design 101www.impactmojo.in
Every extra minute costs quality
Illustrative: data quality declines as the interview lengthens
Illustrative pattern (respondent fatigue)
As fatigue sets in, respondents ‘straight-line’ grids, pick the first option, and say 'don't know' to escape. Ruthlessly cut every question that doesn't feed a planned table.
ImpactMojoSurvey Design 101www.impactmojo.in
Design the page, not just the words
  • Number questions and variables consistently for clean data entry
  • Make instructions to the enumerator visually distinct from the question text
  • Show skip instructions at the branching question, in bold
  • On CAPI, one screen per question reduces missed items and accidental skips
A cramped, ambiguous layout produces enumerator errors that no amount of cleaning fully repairs. Layout is part of measurement.
ImpactMojoSurvey Design 101www.impactmojo.in
07
Section Seven
Sampling for Surveys
ImpactMojoSurvey Design 101www.impactmojo.in
A good sample stands in for millions
You rarely need to ask everyone. A well-drawn sample of a few thousand can describe a population of millions — the principle behind NFHS, PLFS and every credible poll. The magic is not size, it is representativeness.
Data Literacy 101 covers the basics of sampling; here we go deeper into the choices a survey designer actually makes in the field.
ImpactMojoSurvey Design 101www.impactmojo.in
Let chance choose — it removes bias
MethodHowUse when
Simple randomEvery unit equal chanceYou have a full list
SystematicEvery k-th unit from a listOrdered list, no hidden cycle
StratifiedSplit into groups, sample eachMust represent subgroups
ClusterSample whole groups (villages)People geographically spread
MultistageClusters, then units withinLarge national surveys (NFHS)
Only probability sampling lets you compute a margin of error and generalise honestly. Everything else can describe, but not infer.
ImpactMojoSurvey Design 101www.impactmojo.in
Your survey is only as good as its list
Sampling frame
The actual list of units from which you draw your sample — the operational stand-in for the target population. Whoever the frame omits, your survey can never reach.
01
TARGET POPULATION: whom you want to learn about
02
SAMPLING FRAME: the list you can actually draw from
03
SAMPLE: who you end up selecting
04
RESPONDENTS: who actually answers
The frame is often the weakest link. A list of phone numbers is not a list of citizens; a voter roll misses children and recent migrants. Coverage error starts here.
ImpactMojoSurvey Design 101www.impactmojo.in
Over-sample small groups on purpose
If you need reliable estimates for a small subgroup — say, a tribal block or a minority community — a proportional sample may yield too few of them. Stratify and deliberately over-sample that group, then correct with weights.
Design choice: representing every district equally, or every person equally? You usually can't have both at once — decide which your analysis needs, and weight accordingly.
ImpactMojoSurvey Design 101www.impactmojo.in
How many do I actually need?
Margin of error vs sample size (95% confidence, p=0.5)
Standard sampling theory
Landmark numbers (95% CI, p=0.5): n≈384 gives about ±5%; n≈1,067 gives about ±3%. Halving the error roughly quadruples the sample — precision gets expensive fast.
ImpactMojoSurvey Design 101www.impactmojo.in
What changes the number you need
If you want…You need…Note
Tighter margin of errorA larger sample±3% needs ~3× the n of ±5%
Estimates for subgroupsMore per subgroupEach cell needs its own n
To detect a small changeMore powerEffect size drives this
Cluster (not random) designAn inflation factorThe 'design effect'
Cluster sampling saves travel cost but inflates required size via the design effect — people in one village resemble each other, so each adds less new information.
ImpactMojoSurvey Design 101www.impactmojo.in
The errors a bigger sample can't fix
  • Coverage bias: the frame systematically misses people
  • Non-response bias: refusers differ from responders
  • Selection bias: the draw method favours some over others
A bigger biased sample is just a more confident wrong answer. Sample size shrinks sampling error only — never bias. Fix bias with design, not with n.
ImpactMojoSurvey Design 101www.impactmojo.in
Why survey results come 'weighted'
When groups are over-sampled by design or respond at different rates, surveys apply weights so each respondent represents the right number of real people. Unit data from NFHS or PLFS gives wrong totals if you ignore the weights.
Weighting is the bridge back to the population. (Data Literacy 101 covers how to read weighted estimates; here the point is to plan for them at the design stage.)
ImpactMojoSurvey Design 101www.impactmojo.in
08
Section Eight
Modes of Data Collection
ImpactMojoSurvey Design 101www.impactmojo.in
How you ask changes what you hear
The mode — face-to-face, phone, web, self-completion — is not a neutral pipe. It shapes who you reach, how long they'll stay, how honest they are, and what you can ask. Choose it as carefully as the questions.
Mode
The channel through which questions are delivered and answers captured. Each mode carries its own coverage, cost, length limit and pattern of response bias.
ImpactMojoSurvey Design 101www.impactmojo.in
The South Asian workhorse
Most rigorous development surveys in South Asia still rely on face-to-face interviews, increasingly via CAPI — computer-assisted personal interviewing — on a tablet or phone running KoboToolbox, ODK or SurveyCTO.
Strengths
  • Reaches low-literacy & offline areas
  • Long, complex instruments possible
  • Built-in skips & range checks
  • Direct measurement (height, weight, GPS)
Costs
  • Expensive — travel & staff
  • Slow to field at scale
  • Interviewer effects & bias
  • Device & charging logistics
ImpactMojoSurvey Design 101www.impactmojo.in
Fast and cheap — but who picks up?
Phone surveys and IVR (interactive voice response, automated calls) are cheap, fast and safe in a crisis — widely used during COVID-19. But they must be short, and they systematically miss people without phones.
In rural South Asia, phone ownership skews male, younger, richer and more urban. A phone frame quietly excludes the poorest women in the remotest places — precisely those many programmes target.
ImpactMojoSurvey Design 101www.impactmojo.in
Cheapest reach, narrowest coverage
Web surveys cost almost nothing to send to thousands and allow rich self-completion of sensitive items. But for general populations in South Asia, internet access is far from universal — coverage error is severe.
Web works well for connected, literate audiences — staff, professionals, urban youth. It badly misrepresents the general public, especially the poor and elderly.
ImpactMojoSurvey Design 101www.impactmojo.in
Four modes, four profiles
ModeCostCoverageLengthBest for
Face-to-face / CAPIHighBroadestLongRigorous, rural, complex
Phone / IVRLowPhone ownersShortSpeed, crises, monitoring
WebLowestConnected onlyMediumStaff & literate audiences
Self-completionMediumLiterateMediumSensitive topics
There is no best mode — only the right trade-off for your population, budget, topic and timeline. Match the mode to the people you must reach.
ImpactMojoSurvey Design 101www.impactmojo.in
Which mode reaches whom
Illustrative: share of a rural population reachable, by mode
Illustrative pattern for rural South Asia
Illustrative, not measured — but the pattern is real: as cost falls, coverage narrows. Cheaper modes systematically shed the hardest-to-reach.
ImpactMojoSurvey Design 101www.impactmojo.in
The same question, a different answer
Mode effect
A systematic difference in responses caused purely by the mode of administration — for example, people report stigmatised behaviour more honestly to a screen than to a human interviewer's face.
This wrecks comparability. If round one was face-to-face and round two was by phone, an apparent change may be a mode effect, not real. Hold the mode constant across rounds wherever you can.
ImpactMojoSurvey Design 101www.impactmojo.in
Combining modes — carefully
Mixed-mode designs (web first, phone follow-up, face-to-face for refusers) can boost response and cut cost. But they blend the modes' different biases, so disentangling real change from mode effects gets harder.
If you mix modes, record the mode for every response so you can test and adjust for mode effects in analysis — never leave it invisible.
ImpactMojoSurvey Design 101www.impactmojo.in
09
Section Nine
Translation & Cultural Adaptation
ImpactMojoSurvey Design 101www.impactmojo.in
South Asia is not monolingual
A survey in Bihar may need Hindi, Maithili, Bhojpuri and Urdu; one across India, a dozen languages and scripts. The instrument respondents actually hear is the translated one — so translation is instrument design, not an afterthought.
22
languages in the Eighth Schedule of India's Constitution
100+
languages with substantial speaker bases
1
instrument must mean the same thing in all of them
ImpactMojoSurvey Design 101www.impactmojo.in
Translate meaning, not just words
A literally correct translation can still measure something different. The aim is conceptual equivalence — the translated question evokes the same concept, with the same difficulty and connotations, as the original.
‘Do you feel empowered?’ has no clean equivalent in many languages. Adapt to a locally meaningful idea (‘Can you decide this on your own?’) rather than translating the abstract word.
ImpactMojoSurvey Design 101www.impactmojo.in
The standard quality check
Back-translation
One translator renders the source into the target language; an independent second translator, blind to the original, translates it back. Comparing the back-translation to the original surfaces meaning lost or shifted.
01
SOURCE: English question
02
FORWARD: translate to Hindi (translator A)
03
BACK: Hindi → English (translator B, blind)
04
RECONCILE: compare, fix discrepancies
ImpactMojoSurvey Design 101www.impactmojo.in
Committee & team approaches
Back-translation catches literal errors but can miss awkwardness and cultural misfit. A committee or team approach — bilingual experts, translators and field staff reviewing together — produces more natural, usable wording.
Best practice combines both: forward translation, expert committee review, back-translation, then cognitive testing with real speakers of the target language.
ImpactMojoSurvey Design 101www.impactmojo.in
Use categories people recognise
Imported
Western occupation lists, ‘nuclear vs extended family’, or income bands that ignore how people are actually paid (daily wage, in kind, seasonal).
Local
Categories drawn from how the community describes work, household and kinship — refined during pretesting in each setting.
Caste, kinship and livelihood categories vary enormously across regions. A code list that fits Tamil Nadu may be meaningless in Assam.
ImpactMojoSurvey Design 101www.impactmojo.in
Ask in units people use
  • Land in local units (bigha, katha, acre) — then convert
  • Time by the agricultural or festival calendar, not just dates
  • Quantities in market units (a seer, a tin, a bundle)
  • Money as it is actually earned — daily, weekly, per task, in kind
Forcing respondents to convert into unfamiliar units in their heads adds error. Capture the local unit, record the conversion factor, and convert later in analysis.
ImpactMojoSurvey Design 101www.impactmojo.in
Anchor recall in the local year
‘In the last 12 months’ is abstract; ‘since last Diwali’ or ‘since the kharif harvest’ is vivid. A local events calendar built into the instrument sharpens recall and aligns reference periods across respondents.
Build the calendar of festivals, harvests and landmark local events with field staff before fieldwork — it is one of the cheapest accuracy gains available.
ImpactMojoSurvey Design 101www.impactmojo.in
Same concept, locally clothed
The discipline is to adapt the wording, examples and units to each setting while holding the concept and the indicator identical. That is what keeps a multilingual survey both locally meaningful and nationally comparable.
Translate the meaning, standardise the measurement. Lose either and the comparison collapses.
— a cross-cultural survey principle
ImpactMojoSurvey Design 101www.impactmojo.in
10
Section Ten
Pretesting, Piloting & Fieldwork Quality
ImpactMojoSurvey Design 101www.impactmojo.in
No instrument is ready on the first draft
Between ‘finished’ questionnaire and real fieldwork sit two distinct checks: pretesting (do the questions work?) and piloting (does the whole operation work?). Skipping them is the most expensive false economy in survey work.
01
PRETEST: cognitive interviews on the questions
02
REVISE: fix wording, options, routing
03
PILOT: a dry run of the whole operation
04
FINALISE: lock the instrument & protocol
ImpactMojoSurvey Design 101www.impactmojo.in
Find out how respondents actually think
Cognitive interviewing
A pretesting method where a few respondents answer questions while thinking aloud, and the interviewer probes how they understood the words, recalled the facts, and chose their answer.
It reveals what a draft cannot: that ‘household’ means different things to different people, that a recall window is impossible, that an option everyone needs is missing. A dozen good cognitive interviews save a thousand bad responses.
ImpactMojoSurvey Design 101www.impactmojo.in
Rehearse the entire operation
  • Run the full instrument end-to-end in real conditions
  • Time it — is the interview length tolerable?
  • Test the CAPI routing, GPS, and data sync
  • Check logistics: travel, access, consent, refusal handling
  • Review the pilot data: are variables populating as expected?
The pilot tests the system, not just the questions: sampling, fieldwork, devices, supervision and data flow, all at once, before they fail at scale.
ImpactMojoSurvey Design 101www.impactmojo.in
Your data is collected by people
However good the instrument, it is administered by enumerators. Standardised training — on the questions, the skips, neutral probing, consent and edge cases — is what makes 'the same question asked the same way' actually true in the field.
  • Read every question verbatim — no improvised paraphrase
  • Probe neutrally; never suggest an answer
  • Handle ‘don't know’ and refusals correctly
  • Practice with role-plays and mock interviews before going live
ImpactMojoSurvey Design 101www.impactmojo.in
Who asks can change the answer
Interviewer effects
Systematic differences in responses arising from the interviewer — their gender, caste, manner, or subtle cues — rather than from the respondent's true situation.
On gender-based violence or reproductive health, a woman enumerator usually elicits franker answers from women. Match interviewer characteristics to the topic, and standardise manner through training.
ImpactMojoSurvey Design 101www.impactmojo.in
People answer how they think they should
Social-desirability bias
The tendency to give answers that present the respondent favourably — over-reporting good behaviour (hand-washing, school attendance) and under-reporting the stigmatised (drinking, violence, open defecation).
  • Normalise: “Many people…” before a sensitive item
  • Use private self-completion for stigmatised topics
  • Assure confidentiality — and mean it
  • Ask about behaviour, not identity, where possible
ImpactMojoSurvey Design 101www.impactmojo.in
Verify the fieldwork while it happens
Supervisors re-contact a sample of respondents (back-checks) and observe live interviews (spot-checks) to confirm the interview happened, key answers match, and protocol was followed — catching error and fabrication early.
Back-check
Re-ask a few stable questions by phone or revisit; compare to the original to detect errors or faked interviews.
Spot-check
A supervisor sits in on live interviews to observe technique, neutrality and consent in real time.
ImpactMojoSurvey Design 101www.impactmojo.in
Chase response — and study the refusers
Non-response — refusals, not-at-homes, ineligibles — bites hardest when those who don't answer differ from those who do. The poorest, the busiest and the most private are often the hardest to reach, biasing the result.
  • Make repeat visits at different times of day
  • Track and report the response rate honestly
  • Compare responders to the frame to test for bias
  • Adjust with weights — but weights can't fully cure deep non-response
ImpactMojoSurvey Design 101www.impactmojo.in
11
Section Eleven
Ethics, Data & Practice
ImpactMojoSurvey Design 101www.impactmojo.in
Every respondent is a person, not a data point
In a survey, each row is someone who gave you their time and trust — often a poor, busy, over-surveyed person with little power over how their answers are used. Survey ethics is respect made operational, not a compliance form.
The respondent owes you nothing. Everything they give, they give as a gift — treat it like one.
— a field ethics principle
ImpactMojoSurvey Design 101www.impactmojo.in
Consent is the floor, not a formality
  • State plainly what you collect and why
  • Explain how it will be stored, used, shared, and for how long
  • Make clear they may refuse or stop, with no penalty
  • Deliver it in a language and form they genuinely understand
A thumbprint on an unexplained form is not consent. For children and other vulnerable respondents, extra safeguards and guardian consent apply.
ImpactMojoSurvey Design 101www.impactmojo.in
India's data law reaches your survey
The Digital Personal Data Protection Act, 2023 sets duties for anyone handling personal digital data in India — including NGOs and researchers running CAPI surveys on tablets and phones.
  • Collect only what you need, for a stated purpose (purpose limitation)
  • Obtain free, informed, specific consent
  • Protect data with reasonable security safeguards
  • Stronger protections for children's data
‘We're a small NGO’ is not an exemption. Know your obligations before the first interview, not after a breach.
ImpactMojoSurvey Design 101www.impactmojo.in
Anonymisation is harder than deleting names
Removing names is not enough. Village + age + caste + occupation can re-identify one person in a small area. Survey data needs deliberate de-identification, encryption and access control — especially with GPS coordinates attached.
Direct IDs
Name, Aadhaar, phone, GPS — separate & protect
Quasi-IDs
Age + place + caste can re-identify — aggregate or coarsen
ImpactMojoSurvey Design 101www.impactmojo.in
Design for clean data, then document it
A good instrument hands the analyst clean, well-named data. Build validation into the CAPI form, keep the raw export untouched, and ship a data dictionary so every variable, code and unit is self-explanatory months later.
A survey is not done when fieldwork ends. It is done when someone else can open your documented dataset and understand exactly what every column means and how it was made.
ImpactMojoSurvey Design 101www.impactmojo.in
Software for modern field surveys
ToolGood forNote
KoboToolboxHumanitarian & NGO CAPIFree, offline, widely used
ODK (Open Data Kit)Open-source mobile collectionFree, flexible, technical
SurveyCTORigorous research surveysPaid; strong quality controls
Survey SolutionsLarge official surveysFree (World Bank)
All build skip logic, range checks and offline collection into the instrument — turning design discipline into fewer field errors. Tools matter less than the design habits behind them.
ImpactMojoSurvey Design 101www.impactmojo.in
A short, serious reading list
  • Survey Methodology — Groves, Fowler, Couper et al. (the TSE bible)
  • Survey Research Methods — Floyd J. Fowler (concise, practical)
  • Improving Survey Questions — Floyd J. Fowler (question design)
  • Asking Questions — Bradburn, Sudman & Wansink
  • DIME / J-PAL / IPA field guides — practical survey toolkits
Pair this deck with ImpactMojo's Data Literacy 101, Qualitative Methods and Research Ethics courses for the full toolkit.
ImpactMojoSurvey Design 101www.impactmojo.in
If you remember six things
  • Start from the decision — objectives before questions
  • One question, one idea — clear, specific, neutral
  • The questionnaire is the measurement — wording is data
  • Manage total survey error — not just sample size
  • Pretest, pilot, back-check — never trust a first draft
  • Behind every row is a person — consent and care
ImpactMojoSurvey Design 101www.impactmojo.in
Survey Design 101 · Complete
Now go design
a survey worth answering.
CC BY-NC-ND 4.0·Free Forever·ImpactMojo 101 Series