ImpactMojoExploratory Data Analysis 101www.impactmojo.in

ImpactMojo 101 Series · Free Forever

Exploratory
Data
Analysis 101

Inspecting, Cleaning & Questioning Household Survey Data — a Foundational Course for Development Practitioners in South Asia

Household SurveysSouth Asia Focus~90 SlidesFree Access

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Agenda

What We Cover

What EDA Is & Why

Slides 3–10

Household Survey Data in India

Slides 11–21

The EDA Workflow

Slides 22–29

Knowing Your Variables

Slides 30–38

Univariate Exploration

Slides 39–48

Spotting Trouble

Slides 49–58

Bivariate & Multivariate

Slides 59–68

Survey Weights & Design

Slides 69–77

Visual EDA Done Well

Slides 78–85

From EDA to Questions

Slides 86–93

Tools & Reproducibility

Slides 94–99

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section One

What EDA Is & Why

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Definition

Look before you leap

Before you fit a model, build an index or write a finding, you must look at the data. Exploratory Data Analysis (EDA) is the open-minded first pass — describing, plotting and questioning a dataset to understand its shape, gaps and surprises.

Exploratory Data Analysis (EDA)

An attitude and a toolkit for examining data with few prior assumptions — using summaries and pictures to reveal structure, spot problems and generate questions, before any formal testing or modelling.

EDA is detective work, not decoration. You are interrogating the data to find out what it can — and cannot — honestly say.

EDA asks	Modelling asks
What is in this file?	Does X predict Y?
What is wrong with it?	How large is the effect?
What shape is this variable?	Is it significant?
Who is missing, and why?	Controlling for what?

The order matters more than it sounds. A model fitted to uninspected data will run, produce coefficients and stars, and tell you nothing about the 30% of women whose income variable was never filled in.

EDA is also the cheapest stage to be wrong in. A misread code costs minutes here and a retracted finding later.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Origin

Tukey and the case for exploring

Exploratory data analysis is an attitude, a flexibility, and a reliance on display, not a bundle of techniques.

— John W. Tukey, who named EDA in 1977

Tukey argued that statistics had become obsessed with confirming hypotheses and neglected the prior, humbler task of finding them. EDA restored looking, sketching and questioning to the centre of data work.

Tukey's objection	What it changed
Statistics had become confirm-only	Made exploration a legitimate stage
Display was treated as decoration	Made plotting central, not cosmetic
Assumptions went unchecked	Look at the data before assuming its shape
Robustness was neglected	Median and IQR over mean and SD for messy data

Exploratory Data Analysis (1977) also gave us the box plot, the stem-and-leaf display and the five-number summary — tools designed to be drawn by hand, fast, on real data.

Read the quotation carefully: an attitude and a reliance on display. Tukey is not describing a checklist you can complete, which is why "we did EDA" is not an answer to "what does this variable look like".

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Confirm vs Explore

Two modes of data work

Exploratory

Open-ended. What is going on here? Generates questions and hypotheses. Forgiving of surprises. Where every project should start.

Confirmatory

Pre-specified. Is this specific claim true? Tests a hypothesis fixed in advance. Where EDA leads, but is not the same step.

Danger: do not let exploration masquerade as confirmation. A pattern you found by digging is a hypothesis to test on fresh data, not a proven result.

	Exploratory	Confirmatory
Hypothesis	Generated by the analysis	Fixed before the analysis
p-values	Not meaningful	The point
Data	This dataset	Ideally fresh data
Surprises	Welcome	A warning sign
Output	Questions	Answers

The two modes are not stages of the same activity so much as different logics, and the damage comes from mixing them silently — exploring until a pattern appears, then reporting it with a p-value as though it had been specified in advance.

If you must do both on one dataset, say which findings came from which. That sentence is the entire safeguard.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Let It Speak

Let the data speak first

Practitioners often arrive with a story they want the data to confirm. EDA asks you to hold that story loosely and let the numbers talk back — to notice the variable that is half missing, the district that behaves oddly, the impossible age.

The greatest value of a picture is when it forces us to notice what we never expected to see.

— John W. Tukey

The story you arrived with	What EDA might find
"The scheme improved nutrition"	The outcome is missing for a third of the treated
"District X is an outlier"	District X has 40 households in the sample
"Incomes rose"	The mean rose; the median did not
"Coverage is 90%"	90% of those the question applied to

None of these findings say the story is wrong. They say the data cannot yet carry it — a different and more useful conclusion than either confirmation or refutation.

Practitioners are not uniquely prone to this. The pressure to arrive with a story comes from the report deadline and the funder's theory of change, and it does not go away because you have noticed it.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Understand First

Understand before you model

EXPLORE: what is in this dataset? what is wrong with it?

→

DESCRIBE: distributions, gaps, relationships

→

QUESTION: what hypotheses does this suggest?

→

ONLY THEN: model, test, conclude

Skipping EDA is how analysts end up modelling a coding error, averaging a top-coded variable, or reporting on a subgroup that is mostly missing data.

Skipped step	What ships as a result
No inspection	A weight variable read as a measurement
No cleaning	Age 999 in the mean
No description	A mean reported for a bimodal variable
No plotting	A curved relationship summarised as r = 0.1
No questioning	A pattern reported as a finding

Each of these has a common feature: nothing errors, nothing is flagged, and the output looks exactly like a correct one. That is why the discipline has to be a habit rather than a response to a warning.

The reverse also holds. Time spent on EDA is the best-predicted time in a project, because you find the problems while there is still time to solve them.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Why It Pays

What good EDA catches early

Errors — impossible ages, negative incomes, duplicated households
Gaps — a variable missing for 30% of women, not at random
Shape — consumption is skewed, so the mean misleads
Structure — the survey is clustered and weighted, not a simple random sample
Surprises — the outlier district that is the real story

Every hour of EDA saves a day of rework — and prevents a wrong number reaching a decision.

Catch	Cost of missing it
Impossible values	A mean that no reader can reproduce
Non-random gaps	A finding that holds only for those who answered
Skew	A "typical" household nobody resembles
Clustering and weights	Confidence intervals far too narrow
Duplicate IDs	Households counted twice, silently

The list is ordered roughly by how easy each is to find and how expensive each is to discover late. Duplicates are last because they are the ones most often found by a reader rather than the analyst.

Keep a written log of what EDA caught. It is the cheapest possible evidence of data quality when a reviewer asks, and it is otherwise entirely undocumented work.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Roadmap

How this course is built

Foundations

Indian household survey data
The EDA workflow
Variables, codebooks & measurement

Practice

Univariate, bivariate & multivariate views
Missingness, outliers, weights
Honest visuals & reproducible tools

Throughout, the data you meet is patterned on India's NSS, PLFS and NFHS — the surveys you will actually open at work.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Two

Household Survey Data in India

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Unit

What is a household survey?

Household survey

A survey that samples dwellings (households), then collects information about the household as a whole and about each member — the workhorse design behind NSS, PLFS, NFHS and most development data.

Because so much of welfare — consumption, sanitation, who eats, who decides — happens at the household level, the household is the natural sampling unit. But many questions are about people within it.

Design choice	What it means for your analysis
Dwellings are sampled, not people	People in institutions and the homeless are out of frame
One respondent often answers for all	Members' details are proxy-reported
Household defined by shared cooking	Definitions differ between surveys
Members listed by relation to head	"Head" is a reported role, not a legal one

The sampling frame is the quiet limitation. A household survey cannot tell you about people who do not live in a sampled dwelling: migrant workers in employer accommodation, people in institutions, the street homeless.

Where a group of policy interest is systematically outside the frame, the survey's silence about them will look identical to their absence. Say so explicitly.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Structure

Two rosters: households and members

Household roster

One row per household: assets, dwelling type, water source, ration card, total members. Keyed by a household ID.

Member roster

One row per person: age, sex, relation to head, education, work, health. Keyed by household ID + person line number.

EDA must respect this hierarchy. Household-level and person-level variables live in different files and must be merged on the household ID before you can analyse them together.

Question about	Lives in	Keyed by
Assets, water, sanitation	Household roster	Household ID
Age, sex, education, work	Member roster	HH ID + person line
Fertility, anthropometry	Woman / child file	HH ID + line
Consumption items	Item-level file	HH ID + item code

Nearly every real analysis needs a merge, and nearly every merge bug is the same one: joining on the household ID alone when the correct key is household ID plus person line, which silently multiplies rows.

Check the row count before and after every merge, and check it against what you expected. A join that grows your data is telling you something.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Long & Wide

Hierarchical data is nested

HH ID	Person line	Age	Sex	Education
10231	1	44	Male	Secondary
10231	2	40	Female	Primary
10231	3	16	Female	Secondary
10231	4	11	Male	Primary
10232	1	67	Female	None

Four rows, one household (10231). To count households you need unique HH IDs; to count people you count rows. Confusing the two is a classic early error.

You want	You must
Number of households	Count distinct household IDs, not rows
Mean household size	Collapse to household level first
% of women literate	Restrict to women, then weight by person weight
Household consumption per head	Divide by members, not by rows

The nesting is also why a household-level variable repeated on every member row will overstate its frequency: five members in a household with a toilet becomes five toilets if you tabulate at member level.

State the level of every table you produce, in the table. "Unit: households (n = 8,140)" prevents a whole class of misreading, including your own six months later.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Big Three

India's major household surveys

Survey	What it covers	Who runs it	Frequency
NSS (consumption, etc.)	Consumption expenditure, employment, social consumption	NSSO / MoSPI	Rounds (subject rotates)
PLFS	Labour force: work, unemployment, wages	NSSO / MoSPI	Annual since 2017–18
NFHS	Health, nutrition, fertility, anaemia, women's status	IIPS / MoHFW	~5 yrs (NFHS-5: 2019–21)
CMIE-CPHS	Household income, consumption, sentiment	CMIE (private)	Continuous, 3 waves/yr

Know these by name and by job. The right survey for a question depends on what it measures and how recently.

Survey	Best used for	Watch out for
NSS	Consumption, employment, social consumption	Subject rotates by round; comparability across rounds
PLFS	Annual and quarterly labour estimates	Urban panel design differs from rural
NFHS	Health, nutrition, gender, district estimates	Cross-sectional; not a panel
Census	Complete counts, small areas	Decennial; 2021 round delayed

Choose by unit and resolution, not by familiarity. NFHS gives district estimates; NSS consumption rounds are designed to state level, so a district figure derived from them carries a precision it does not have.

Where two sources disagree on the same quantity, the difference is usually definitional — reference period, universe, or question wording — before it is error.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

NSS & PLFS

NSS and PLFS: the official workhorses

The National Sample Survey (NSS) has measured consumption and employment for decades. The Periodic Labour Force Survey (PLFS) took over labour statistics in 2017–18, giving annual rural and quarterly urban estimates.

NSS

Consumption & expenditure rounds underpin official poverty estimates

NSSO / MoSPI

Annual

PLFS gives the labour force participation and unemployment rates since 2017–18

PLFS, MoSPI

	NSS employment rounds	PLFS
Started	Quinquennial rounds	2017-18
Frequency	Every ~5 years	Annual rural, quarterly urban
Panel element	None	Urban households revisited
Comparability	—	Breaks with earlier NSS series

PLFS did not simply continue the NSS employment series at higher frequency: the design, the questionnaire and the sampling changed. Splicing PLFS onto the old rounds to make a long trend line is a well-known error.

If you need a series across the break, present it as two series, and say why. A single line implies a continuity the data does not have.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

NFHS

NFHS: health, nutrition and gender

The National Family Health Survey is India's Demographic and Health Survey. NFHS-5 (2019–21) covered the health, nutrition and demographic situation of women, children and households across every district.

~636,000

households interviewed in NFHS-5

NFHS-5, 2019–21

707

districts covered

IIPS / MoHFW

rounds since 1992–93

IIPS

NFHS-5 gives you	It does not give you
District-level estimates	Block or village estimates
Anthropometry and biomarkers	Consumption or income
Women 15-49 and men 15-54	A panel over time
Two phases (2019-20, 2020-21)	A clean pre- and post-pandemic split

The two-phase fieldwork is a real analytical issue, not a footnote: phase 1 was completed before the pandemic and phase 2 largely after it, and the states are not evenly split between them.

Where a comparison spans the two phases, check which states sit in which. It is one of the more common quiet confounders in NFHS-5 analyses.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

CMIE-CPHS

CMIE-CPHS: high-frequency private data

The Consumer Pyramids Household Survey (CPHS), run by the private firm CMIE, tracks a large panel of households continuously, giving fast readings on income, spending and unemployment between official rounds.

Useful for timeliness, but debated on sampling and representativeness. Treat it as complementary to — not a replacement for — the official surveys, and read the methodology critics.

CPHS strength	CPHS caution
High frequency — waves every four months	Not an official statistical product
A genuine panel of households	Attrition accumulates across waves
Fast reading on income and unemployment	Sampling frame criticised for under-covering the poorest
Fills gaps between official rounds	Access is commercial, so replication is limited

The sampling-frame critique is substantive and has been argued in print by economists on both sides; treat CPHS levels with more caution than CPHS trends, which is roughly where the debate has settled.

For EDA the practical point is narrower: if you cannot share the data, you cannot let a reader check your numbers, and you should say which claims rest on it.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Unit of Analysis

Decide your unit before you compute

The unit of analysis is the entity each row of your working table represents — a household, a person, a child under five, a woman aged 15–49. Every statistic is implicitly 'per' some unit.

A rate like the anaemia prevalence is per eligible person, not per household. Computing it on the household file, or without restricting to the eligible group, gives the wrong answer.

Statistic	Unit it belongs to	Denominator
Anaemia prevalence	Women 15-49	Women tested, not all women
Stunting rate	Children under 5	Children measured
MPCE	Household, per head	Household members
Unemployment rate	Person in labour force	Labour force, not population

Almost every widely misquoted development statistic is a denominator error rather than a numerator one. The unemployment rate is the classic case: its denominator is the labour force, so it falls when people stop looking for work.

Write the denominator into the variable name while you work — anaemia_pct_women_tested is ugly and has never once been misread.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Design

These are not simple random samples

STRATIFY: split by state × rural/urban

→

STAGE 1: sample villages / urban blocks (clusters)

→

STAGE 2: sample households within each cluster

→

RESULT: a stratified, multistage, clustered sample

Because selection happens in stages and some groups are over-sampled, every household carries a survey weight. We return to weights in Section Eight — they change your EDA.

Design feature	Consequence for EDA
Stratification	Estimates must respect strata
Clustering (PSUs)	Observations within a village are correlated
Unequal selection probability	Raw sample proportions are not population ones
Non-response adjustment	Weights carry more than selection

Clustering is the one that surprises people. Two households in the same village are more alike than two at random, so each adds less information than an independent observation would — which is what the design effect measures.

The practical rule for exploration: unweighted plots are fine for finding data errors, and misleading for describing the population. Know which you are doing.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

First Look

What to check the moment a file opens

How many rows, and what is one row — a household or a person?
Is there a household ID, a person line number, a weight variable?
Which file is the household roster, which the member roster?
What does the documentation say the unit and reference period are?

Read the survey's report and documentation before the data. Five minutes there saves hours of confusion later.

Open the file and answer	Before you compute anything
What is one row?	Household, person, item, or birth
What is the key?	Which columns uniquely identify a row
Where is the weight?	And is it a household or person weight
What is the universe?	Who was asked this block at all
Which codes mean missing?	From the codebook, not by guessing

Answer these five in writing, at the top of your script, as comments. It takes ten minutes and it is the single highest-return habit in this course.

If you cannot answer one of them from the documentation, that is a finding: note it, and treat any number depending on it as provisional.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Three

The EDA Workflow

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Loop

Six steps, repeated

LOAD: read the data correctly

→

INSPECT: structure, types, size

→

CLEAN: codes, gaps, impossible values

→

DESCRIBE: summaries per variable

→

VISUALISE: plot distributions & relations

→

QUESTION: form hypotheses, then loop back

EDA is iterative, not linear. Each plot raises a question that sends you back to inspect or clean. Expect to go round this loop many times.

Step	Done when
Load	Row and column counts match the documentation
Inspect	Every column has a known type and range
Clean	Every recode is in a script, with a reason
Describe	No variable is unexamined
Visualise	Every continuous variable has been plotted once
Question	The open questions are written down

It is a loop rather than a pipeline. Visualising almost always sends you back to cleaning, and questioning sends you back to the codebook.

The step people skip is the last one. Without it EDA ends with a folder of plots and nothing anyone can act on.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Load

Step 1 — load it right

Read the right file format — fixed-width, CSV, .dta, .sav, .csv.gz
Make sure ID and code columns load as text, not numbers (leading zeros vanish otherwise)
Preserve special missing codes; do not let them become real values
Check the row count against the documentation

A household ID like 0457 silently becoming 457 will break every merge. Type matters from the very first line.

Load trap	Symptom
IDs read as numbers	Leading zeros gone; merges fail
Missing codes read as values	Mean age of 143
Wrong encoding	Mojibake in state names
Fixed-width read with wrong layout	Every column shifted by one
Only the first file loaded	Half the sample missing, no error

Read ID and code columns as text explicitly. Every statistical package will helpfully convert 07 to 7, and the merge that then fails will fail quietly, dropping rows rather than erroring.

Compare your loaded row count against the figure in the survey report before doing anything else. It is one line and it catches the worst class of error.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Inspect

Step 2 — inspect the skeleton

Question	What you are checking
How many rows & columns?	Size and whether the file is complete
What type is each column?	Numeric, text, date, categorical codes
What is the range of each?	Min, max, plausibility
How many distinct values?	Categorical levels, accidental duplicates
How many missing per column?	Where the gaps are

This is the data-equivalent of a doctor's first examination — vital signs before any diagnosis.

Inspect	Red flag
Row count	Suspiciously round, or half the documented total
Column types	A continuous variable stored as text
Min and max	Values outside anything possible
Distinct values	A "continuous" variable with six levels
Missing count	Zero missing on a question with a skip pattern

Zero missing where a skip pattern exists is a stronger warning than a high missing count. It usually means the not-applicable code has been read as a real value.

Do this for every column, not the ones you plan to use. The columns you were not planning to use are where the surprises live.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Clean

Step 3 — clean transparently

Cleaning means standardising codes, recoding missing values, fixing types and flagging impossible entries — in a script, never by hand-editing the raw file.

Golden rule: keep the raw extract read-only. Every change lives in code, so it is documented, reversible and reproducible. We expand on this in Section Eleven.

Never	Instead
Edit the raw file	Read it read-only; write a cleaned copy
Delete a suspicious row	Flag it and decide explicitly
Overwrite a variable	Create a new one and keep both
Recode silently	Comment the reason next to the line

The reason for keeping the raw extract read-only is not tidiness. It is that six months later somebody will ask why a number changed, and the only way to answer is to re-run from an unaltered source.

A cleaning script that runs end to end from raw extract to analysis table is also the only honest answer to "can you reproduce this?"

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Describe

Step 4 — describe every variable

For continuous variables: min, max, mean, median, quartiles, SD, % missing
For categorical variables: a frequency table of every level
Flag anything that looks impossible or surprising for a second look
Always describe before you visualise — numbers anchor the eye

Variable type	Describe with	Also report
Continuous, symmetric	Mean, SD	% missing
Continuous, skewed	Median, IQR	Min, max, % missing
Categorical	Frequency table	Levels with zero counts
Binary	Percentage	Denominator
Coded ID	Count of distinct values	Duplicates

Describing every variable, including the ones you will not use, is what turns a data extract into something you understand. It is also where the codebook errors surface.

Report percent missing next to every statistic. A mean over 60% of the sample is a different object from a mean over 99%, and printing them identically hides that.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Visualise

Step 5 — plot to see the shape

Summaries can hide as much as they reveal. A histogram shows skew and bimodality a mean cannot; a scatter shows a curved relationship a correlation cannot. Always plot.

Numerical calculations are exact, but graphs are rough.

— John W. Tukey — and we need both

Summary that hides it	Plot that shows it
Mean and SD	Bimodality — two peaks
Correlation	A curved or U-shaped relation
Median	A spike at a top-code
Range	Heaping on round numbers
Percent missing	Missingness concentrated in one region

The classic demonstration is Anscombe's quartet: four datasets with identical means, variances, correlation and regression line, and four completely different shapes.

In exploration a plot does not need to be presentable. Draw it fast, look, and throw it away — the polish belongs to the final chart, not this one.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Question

Step 6 — turn findings into questions

Good EDA ends not with answers but with sharper questions: why is consumption bimodal here? why is this variable missing more for women? is the district outlier real or an error?

Write the questions down. They become your analysis plan — and the line between what EDA suggested and what a later test confirmed.

Pattern noticed	Question it becomes
Consumption is bimodal in one state	Are two populations pooled here?
A variable is missing more for women	Is it a skip pattern or non-response?
One district is an outlier	Sample size, coding error, or real?
Two variables correlate strongly	Is one derived from the other?

The last one catches a surprisingly common embarrassment: a strong correlation between a total and one of its own components, discovered and reported as a finding.

Write the questions in the script, dated. They are the record of what you noticed, and the shortlist for whatever formal analysis comes next.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Four

Knowing Your Variables

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Key

The data dictionary is your map

Data dictionary / codebook

The document that defines every variable: its name, meaning, units, allowed values, the codes for each category, the special missing codes, and who it applies to.

A survey dataset without its codebook is a locked box. The number '2' could mean female, urban, 'no', or a missing code — only the dictionary tells you which.

Without the codebook you cannot know	So you cannot
What 9 means in this column	Compute any mean safely
Who was asked the question	State a denominator
The reference period	Compare with another survey
Which weight applies	Describe the population

A dataset without its codebook is not a hard dataset — it is an uninterpretable one. No amount of inspection recovers what a code was supposed to mean.

Download the codebook with the data, into the same folder, and keep both. Recovering it later from a changed government portal is a real and recurring problem.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Read It First

What the codebook tells you

Codebook field	Why EDA needs it
Variable label	What it actually measures
Value codes	1 = yes, 2 = no, 9 = missing
Units	Rupees? months? per week?
Universe / who answers	Only women 15–49? only workers?
Reference period	Last 7 days? last 30? last year?
Skip patterns	Why a block is blank for some rows

The 'universe' field explains most 'missing' data: a question about pregnancy is blank for men by design, not by error.

Codebook field	Question it prevents
Universe	"Why is this 40% missing?"
Reference period	"Last week or last year?"
Units	"Rupees or hundreds of rupees?"
Value codes	"Is 2 higher or lower than 1?"

Reference period is the field most often skipped and most often decisive. Employment measured over the last week and over the last year give genuinely different rates, and neither is wrong.

Where a survey offers both, say which you used in the sentence that reports the number, not in a methods annex.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Levels

Four levels of measurement

Level	Meaning	Survey example	Valid maths
Nominal	Labels, no order	Religion, state, ration-card type	Counts, mode
Ordinal	Ordered, unequal gaps	Education level, wealth quintile	Median, rank
Interval	Equal gaps, no true zero	Year of birth	Mean, difference
Ratio	Equal gaps, true zero	Age, income, consumption	All, ratios

The level decides which statistics are legal. You cannot average religion codes, and a wealth quintile is a rank (1–5), not a rupee amount.

Level	Common mistake
Nominal	Taking a mean of state codes
Ordinal	Treating quintile gaps as equal
Interval	Saying 20 degrees is twice 10
Ratio	None inherent — but watch zeros

The ordinal case is the one that survives review. Wealth quintiles and education levels are routinely averaged, and the resulting "mean wealth quintile of 3.2" assumes the gaps between quintiles are equal, which they are not.

Ordinal data has a legitimate summary: the median, the mode, and the full distribution. Report those rather than an average that quietly asserts a measurement scale.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Categorical vs Continuous

The first sort: categories or measures

Categorical

A fixed set of labels — sex, caste category, district, employment status. You count and tabulate these.

Continuous

A measured quantity — age, income, MPCE, height. You take means, medians and histograms of these.

Watch the trap of coded categoricals: education stored as 1–8 looks numeric, but its mean is meaningless. Check the codebook, not the column type.

Looks continuous but is not	Why
State code	Nominal label stored as a number
Wealth quintile	Ordinal ranking, unequal gaps
Religion code	Nominal
Number of children	Genuinely a count — this one is fine
Education years vs level	One is continuous, one is ordinal

Education is the sharpest example because both forms exist in Indian surveys. Years of schooling is continuous; the coded level variable is ordinal, and they are not interchangeable in a mean.

The test is simple: would the halfway point between two values mean anything? Between 4 and 6 years of schooling, yes. Between "primary" and "higher secondary", no.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Coded Numbers

Numbers that are really labels

Survey files store almost everything as numbers to save space. State = 9 means Uttar Pradesh, not the quantity nine. Treating such codes as measurements is one of the commonest EDA errors.

Before computing any mean, ask: is this a quantity or a code? The dictionary, not the data type, decides.

Column	What a mean of it would be
State code	A number between two states
Caste category code	Meaningless
Sex (1/2)	A rescaled sex ratio, by accident
District code	Nonsense with three decimal places

Nothing in the software objects. A mean state code of 14.7 prints exactly like a mean age, and if it reaches a table nobody downstream can tell it apart from a real statistic.

Convert coded columns to labelled factors as the first cleaning step. After that the software will refuse to average them, which is the behaviour you want.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Special Codes

Missing codes hide in plain sight

Code	Often means	Risk if treated as a value
99 / 999	Not known / not stated	Inflates the mean enormously
97 / 98	Refused / not applicable	Phantom category in tables
0	Sometimes a real zero, sometimes 'none'	Ambiguous — check codebook
Blank	Skip pattern or true missing	Silent loss of cases

Recode special codes to explicit missing before any summary. A mean income of ₹1,400 lakh usually means a 9999999 'not stated' code slipped through.

Code	Typical meaning	What to do
99 / 999	Not known / not stated	Recode to missing
97 / 98	Refused / not applicable	Distinguish the two
0	Sometimes real, sometimes none	Check the codebook
Blank	Skipped or genuinely absent	Check the skip logic

"Not applicable" and "not stated" must not collapse into one missing category. The first is a design feature and should shrink your denominator; the second is non-response and may bias your estimate.

A histogram of the raw variable finds these fastest: a lone spike at 99 in a variable that otherwise stops at 60 is unmistakable.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Derived Variables

Variables you build yourself

Much of EDA involves deriving variables: monthly per-capita consumption from total expenditure and household size; age groups from age; a binary 'has toilet' from a coded sanitation variable.

Derive in a script, label the new variable clearly, and sanity-check its distribution. A derived variable inherits every error in its inputs.

Derived variable	Decision hiding inside it
MPCE	Which expenditure items count
Age group	Where the cut points fall
"Has toilet"	Whether shared facilities count
Poverty status	Which line, which year, which deflator
Household size	Whether visitors and absentees are members

Every derived variable embeds a judgement, and the judgement usually disappears into a variable name. Six months later "poor" is a column, not a decision anyone remembers making.

Write the definition as a comment beside the code that creates it, in words a colleague could apply independently. That comment is the difference between a derived variable and a black box.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Profiling

Build a variable profile

Name and meaning (from the codebook)
Type: categorical or continuous; level of measurement
Range or set of levels; special missing codes
Universe: who is supposed to have a value
% missing, and whether the missingness looks patterned

A one-line profile per variable, written as you go, is the backbone of trustworthy analysis.

Profile line	Example entry
Name and meaning	mpce_mrp — monthly per-capita consumption
Type and level	Continuous, ratio
Range / levels	210 to 42,000 rupees
Missing codes	None; blanks are genuine
Universe	All sampled households
% missing	0.4%

A profile of thirty variables fits on two pages and answers most of the questions a reviewer will ask before they ask them.

Build it as a table in the script rather than a separate document, so it regenerates when the data changes. A profile that has to be updated by hand will be wrong within a month.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Five

Univariate Exploration

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

One at a Time

Start with one variable

Univariate exploration looks at each variable on its own — its frequencies, its distribution, its centre and spread. It is where most data problems first surface.

CATEGORICAL: frequency table, bar chart

→

CONTINUOUS: histogram, density, summary stats

→

BOTH: count and inspect the missing

Variable type	First look	What you are hunting
Categorical	Frequency table	Unexpected levels, tiny cells
Binary	Percentage + denominator	A third category hiding as missing
Continuous	Histogram + five-number summary	Skew, spikes, impossible values
Date	Min and max	Dates outside fieldwork
ID	Count of distinct values	Duplicates

Univariate work is unglamorous and finds most of the problems. Nearly every error that reaches a published table would have been visible in a plain frequency table of one column.

Do it for all variables before any bivariate work. A relationship between two badly-coded variables is not worth exploring.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Frequency Tables

Count every level

Sanitation facility	Households	%
Improved, not shared	6,420	64.2
Improved, shared	1,180	11.8
Unimproved	910	9.1
Open defecation	1,390	13.9
Not stated	100	1.0

Illustrative, patterned on NFHS-style categories. A frequency table is the first thing to run on any categorical variable — it reveals tiny categories, typos and stray codes at a glance.

Read the table for	What it reveals
A level with a tiny count	Too small to report separately
A level that should not exist	A coding or entry error
"Not stated" size	How much of the denominator is unusable
A missing level	A category with zero cases — say so

Percentages should say what they are out of. Here 1.0% is "not stated", so a coverage figure computed over all rows differs slightly from one computed over rows with a stated answer — small, but the reader cannot tell which you did.

Report both n and % in every frequency table. A percentage without its count cannot be checked and cannot be pooled.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Histograms

The histogram of a skewed variable

Monthly per-capita consumption expenditure (MPCE), illustrative

Illustrative, patterned on NSS consumption distributions

Note the long right tail. Consumption, like income and landholding, is right-skewed — a few households spend many times the typical amount.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Reading Shape

What a histogram reveals

Centre: where the bulk of households sit
Spread: how wide the distribution is
Skew: a long tail on one side
Modes: one peak, or two (a hidden subgroup?)
Gaps & spikes: heaping at round numbers, or a wall at a top-code

The bin width matters: too wide hides structure, too narrow shows noise. Try a few widths before you conclude.

Feature	Likely cause
Two peaks	Two populations pooled (rural / urban)
Spike at a round number	Heaping — respondents rounding ages or amounts
Wall at the maximum	Top-coding
Spike at zero	"None" coded as 0 alongside real zeros
Gap in the middle	A filter or a recode error

Age heaping at multiples of five is well documented in South Asian surveys and is a measure of data quality in its own right — demographers compute indices from it.

Bin width changes what you see. Look at two or three widths before concluding a distribution is bimodal; a single histogram can manufacture or hide a second peak.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Density

Density: the smoothed histogram

A density plot smooths the histogram into a curve, making it easier to compare shapes — for example, the consumption distribution for rural versus urban households on one set of axes.

Smoothing is a choice: too much smoothing erases real peaks; too little invents them. Treat the curve as one view, not the truth — and keep the histogram alongside.

Density plot	Histogram
Smooth, easy to overlay groups	Honest about the actual counts
Bandwidth is a choice	Bin width is a choice
Can show mass below zero	Cannot invent impossible values
Good for comparison	Good for spotting spikes and gaps

The bounded-variable problem is worth knowing: a density estimate of a variable that cannot be negative will often draw a tail below zero, because the smoother does not know the boundary exists.

For EDA, plot both. The histogram tells you what is there; the density tells you how two groups compare.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Summary Stats

The five-number summary

Min & Max

the extremes — sanity-check both

Q1, Median, Q3

the middle and the quartiles

IQR

Q3 − Q1: the robust spread of the middle 50%

Tukey's five-number summary (min, Q1, median, Q3, max) describes a distribution without assuming any shape — the heart of EDA.

Statistic	Robust to outliers?	Use when
Mean	No	Roughly symmetric, no extreme tail
Median	Yes	Skewed money variables
SD	No	Alongside a mean, symmetric data
IQR	Yes	Alongside a median
Min / max	They are the outliers	Always, as a sanity check

Report the pair that belongs together: mean with SD, median with IQR. A median reported with a standard deviation is a common tell that the summary was assembled rather than chosen.

Min and max deserve to be looked at every single time even though they are never reported. They are where impossible values live.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Mean vs Median

For skewed money, trust the median

Mean is pulled above the median by the right tail (illustrative)

Illustrative

The mean sits well above the median because the long tail of high spenders drags it up. For consumption, income and land, report the median.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Skew

Naming the skew

Right-skewed

Long tail to the right. Mean > median. Income, MPCE, landholding, firm size. The common case in development data.

Left-skewed

Long tail to the left. Mean < median. Rarer — e.g. age at death in a high-survival population.

Skew is a signal, not a defect. It tells you which centre to report and warns you off methods that assume symmetry.

Shape	Mean vs median	Report
Right-skewed	Mean > median	Median, and say it is skewed
Left-skewed	Mean < median	Median
Symmetric	Roughly equal	Either; mean is fine
Bimodal	Both mislead	Split the groups and describe each

The bimodal row is the one that matters most and is least often acted on. If a distribution has two peaks, no single number describes it, and reporting either the mean or the median is a way of not saying so.

A quick diagnostic: compute mean minus median as a fraction of the median. For Indian consumption data a gap of 15-25% is ordinary; much more usually means a tail worth investigating.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Transforms

The log scale tames the tail

For strongly right-skewed money variables, plotting on a logarithmic scale spreads the squashed low values and pulls in the tail, often revealing a near-symmetric shape that is easier to read.

A transform is an exploratory lens, not a fact about the world. Always label the axis as logged, and remember to back-transform before reporting rupee figures.

Log scale helps	Log scale hurts
Money spread over orders of magnitude	Variables with true zeros
Seeing the low end of a squashed range	Audiences who read the axis as linear
Making a multiplicative pattern look linear	When absolute differences are the point

Zeros are the practical obstacle: log(0) is undefined, and the usual fixes — adding one, dropping the zeros — both change what the plot shows and both need to be stated.

In exploration a log axis is a lens you pick up and put down. In a published chart it is a claim about the reader, so label it unmistakably or they will misread the gaps.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Six

Spotting Trouble

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Watchlist

What can be wrong with a column

Problem	Survey example	How EDA spots it
Missing values	Income blank for some	Count of NA per variable
Impossible values	Age = 230, −3 children	Min/max range check
Special codes as data	99 = 'not stated'	Spike at 99 in histogram
Top-coding	Income capped at a max	Wall of cases at the ceiling
Heaping	Ages bunched at 0, 5, 10	Comb pattern in histogram
Duplicates	Same HH twice	Duplicate household IDs

Problem	Cheapest detector
Missing values	Count of NA by column
Impossible values	Min / max table
Codes read as data	Histogram — look for a lone far spike
Duplicates	Count of rows vs count of distinct keys
Heaping	Frequency of the last digit
Top-coding	Frequency of the maximum value

Every detector in this column is one line of code and runs in seconds on a full survey file. There is no version of this work that is too expensive to do.

Run all six on every column, automatically, as part of loading. Making it automatic is what stops it from being skipped on the day you are in a hurry.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Missingness

Map where the gaps are

Share of records missing, by variable (illustrative)

Illustrative

A missingness bar chart across variables is one of the most useful EDA plots. Income and 'not applicable' fields are usually the worst — and that is rarely random.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Why Missing

Why a value is missing matters most

By design (skip): the question did not apply — not a problem
Missing at random: gaps unrelated to the value — least harmful
Missing not at random: the richest refuse to state income — this biases results

Dropping rows with gaps can quietly delete the very households you care about. Before deleting or filling, ask why the value is absent — the pattern of absence is itself data.

Mechanism	Effect on your estimate	What helps
Skip / not applicable	None — adjust the denominator	Read the skip logic
Missing completely at random	Loss of precision only	Nothing much needed
Missing at random	Bias unless you condition	Condition on the observed cause
Missing not at random	Bias you cannot remove from the data	Bound it; say so

The names are unhelpfully similar and the distinction is real: "at random" does not mean haphazard, it means the gap is explained by variables you can see.

Income refused by the richest is the standard MNAR example in household surveys, and it biases inequality measures in a predictable direction — downward. Complete-case analysis does not fix it, it hides it.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Missingness Patterns

Look for structure in the gaps

Cross-tabulate missingness against other variables: is income missing more for richer households? is a health variable missing more in one state? Patterned missingness is a finding, not just a nuisance.

Create a 'missing flag' variable and explore it like any other — who is missing, and how do they differ from who is present?

Cross-tabulate missingness against	Question it answers
Sector (rural / urban)	Is one setting harder to survey?
Sex of respondent	Is a module reaching women less well?
State	Is it a fieldwork problem in one place?
Wealth quintile	Is refusal concentrated at the top?
Interviewer or PSU	Is it one team rather than the population?

The last row is the one people forget. Missingness clustered in a handful of primary sampling units is a fieldwork artefact, not a behavioural finding, and it is visible in about four lines of code.

Create the missingness indicator as a real variable and treat it as an outcome. Patterned gaps are often the most interesting thing in a survey file.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Outliers

Outliers: error, or the story?

Could be an error

Age 230, a household of 60, income of ₹0 with five earners — check for a data-entry slip or a stray code before analysing.

Could be the story

The one district with triple the stunting rate may be exactly where the programme is needed. Do not delete it — investigate it.

Never silently drop outliers. Flag them, explain them, and decide transparently — and record it in the EDA log.

Before deciding	Check
Is it possible at all?	Range and logic rules
Is it a code?	The codebook, again
Is it one record or many?	A cluster of extremes is a pattern
Does it change the conclusion?	Run it with and without

The honest procedure for a genuine extreme value is to report the estimate both ways. If the conclusion survives, say so; if it does not, that is the finding.

Never delete an outlier because it is inconvenient, and never keep one because it is interesting. Both are decisions about the answer rather than the data.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Finding Outliers

Rules of thumb for flagging extremes

Range checks: can this value exist at all? (% in 0–100)
IQR rule: flag points beyond Q1 − 1.5×IQR or Q3 + 1.5×IQR
Logic checks: a 6-year-old cannot be married with children
Visual: the point detached from the cloud in a scatter

These rules flag candidates for human review. They do not decide — judgement, not a threshold, removes a value.

Rule	Good at	Fails when
Range check	Impossible values	The value is possible but wrong
IQR rule	Symmetric-ish data	Skewed money — flags the whole tail
SD rule (±3)	Normal data	Skew or heavy tails
Logic check	Internal contradictions	Needs you to write the rules
Visual	Anything unexpected	Large n, where the eye gives out

The IQR rule applied to consumption will flag several percent of households as outliers, which is not a data-quality finding — it is the definition of a right-skewed distribution.

Rules flag candidates. A human decides. Automating the decision is how a real district gets deleted for being poor.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Impossible Values

Logic the data must obey

Rule	Violation to flag
Age between 0 and ~110	Age = 230, age = −1
Percentages in 0–100	Vaccination = 140%
Members ≥ earners	8 earners in a 4-person household
Mother older than child	Mother 14, child 10
Consumption > 0	MPCE = 0 with members present

Codify these checks once and re-run them on every extract. Built-in validation beats hoping you will notice.

Rule family	Example
Range	0 <= age <= 110
Sum	Members >= earners
Relational	Mother at least 12 years older than child
Conditional	Pregnancy variables only for women
Cross-file	Every member row has a household row

Write these as a list of assertions the cleaning script runs and reports on, rather than as things you check by eye. The list then grows every time someone finds a new way for the data to be wrong.

The cross-file rule catches orphaned records — member rows whose household was dropped somewhere upstream — which otherwise show up as a household count that will not reconcile.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Top-Coding

Top-coding: the artificial ceiling

Top-coding

Capping a variable at a maximum value to protect privacy or limit outliers — e.g. all incomes above ₹10 lakh recorded as exactly ₹10 lakh. It creates a spike of identical values at the ceiling.

Top-coding makes the mean and the upper tail unreliable. Spot it as a wall of cases at one value, and note that you cannot study the top accurately from such data.

Top-coding does	So you cannot
Caps the highest values	Compute a true mean
Creates a spike at the ceiling	Trust the SD or the range
Protects respondent privacy	Recover the real tail
Varies between surveys and rounds	Compare tails across sources naively

Because top-coding truncates only the top, it biases the mean downwards and inequality measures downwards too. The median is usually unaffected, which is another reason to prefer it here.

Check for it by tabulating the maximum value: a legitimate distribution rarely has hundreds of households at exactly the same rupee figure.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Before / After

Cleaning changes the picture

Age distribution before and after removing impossible values (illustrative)

Illustrative

The impossible '>110' sliver vanishes after recoding it to missing. Small in count, but it would have distorted any mean age — and revealed a data-entry problem worth reporting.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Seven

Bivariate & Multivariate Exploration

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Two at a Time

Now look at variables together

Bivariate exploration asks how two variables move together; multivariate brings in a third. This is where relationships, gaps and confounders start to appear.

CAT × CAT: cross-tabulation

→

CAT × CONTINUOUS: grouped summaries

→

CONTINUOUS × CONTINUOUS: scatter plot

→

MANY: correlation matrix, faceting

Pair	Tool	Summary
Categorical x categorical	Cross-tab	Row or column %
Categorical x continuous	Grouped medians, box plots	Median by group
Continuous x continuous	Scatter	Correlation, with caution
Any pair x a third	Faceting, colour	Same plot, split

Add the third variable early rather than at the end. Most apparent bivariate relationships in survey data change when you split by sector, state or wealth, and finding that out late means redoing the work.

Bivariate exploration is also where composition effects appear — a difference between two groups that is really a difference in their age or urban mix.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Cross-Tabs

Cross-tabulation: two categoricals

Sector	Has toilet	No toilet	% with toilet
Rural	5,180	2,020	71.9
Urban	2,610	190	93.2
All	7,790	2,210	77.9

Illustrative. A cross-tab is the bivariate workhorse. The key choice is the direction of percentages: row percents answer 'of rural households, what share have a toilet?' — usually what you want.

Cross-tab decision	Consequence
Which variable in rows	Determines the natural comparison
Row or column %	Two different questions
Include "not stated"?	Changes every percentage
Weighted or not	Sample vs population
Cell counts shown?	Whether the reader can judge precision

Always show the counts alongside the percentages. A cell of 93.2% built on 190 households reads identically to one built on 19,000 unless the n is printed.

Suppress or flag cells below a threshold — 25 unweighted cases is a common convention. A percentage from seven households is noise presented as a finding.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Row vs Column

Percentage in the right direction

Row %

Within each row group: 'of rural households, X% have a toilet.' Compares the outcome across groups.

Column %

Within each column: 'of households with a toilet, X% are rural.' A different question entirely.

Choosing the wrong direction silently answers a different question. State the comparison in words before you tabulate.

You want to say	Percentage direction
"Of rural households, 72% have a toilet"	Row % (rows = sector)
"Of households with toilets, 33% are urban"	Column %
"Rural households are 72% of the sample"	Neither — marginal %

The direction encodes the causal reading, whether or not you intend it. Row percentages by sector invite "sector affects sanitation"; column percentages invite the reverse, which is nonsense here.

Rule of thumb: put the variable you think of as the cause or the grouping in the rows, and percentage across the row. Then read your sentence aloud and check it matches.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Grouped Summaries

A continuous variable, split by groups

Median MPCE by social group (illustrative, patterned on NSS)

Illustrative, patterned on NSS consumption by social group

Grouping a continuous variable by a category — here median consumption by social group — is the single most common development EDA move. Use the median for skewed money.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Quartiles by Group

Compare spread, not just centre

MPCE quartiles by sector — the spread differs (illustrative)

Illustrative

A true box plot is awkward in this toolkit, so we plot the quartiles directly. Urban consumption is both higher and more spread out — a fact the medians alone would hide.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Scatter

Two continuous variables: the scatter

Female literacy (%) vs total fertility rate, major states

Illustrative, patterned on Census 2011 & NFHS-5

A clear negative pattern — but remember it is a state-level picture. The ecological fallacy (next) warns against reading it as an individual truth.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Correlation

Correlation, and its limits

Correlation (r)

A number from −1 to +1 summarising how strongly two continuous variables move together linearly. It captures direction and strength — but only of a straight-line relationship.

Correlation is not causation — a confounder may drive both
r misses curves — a strong U-shape can give r near 0
Outliers move r a lot — one point can fake a relationship

r does not tell you	Why
The slope	r is scale-free; the slope is not
Whether it is causal	It never did
Whether the relation is curved	It measures straight lines only
Whether one point drives it	A single outlier can create or destroy r
Whether it holds in subgroups	It is a pooled number

Two datasets can share an r of 0.8 and look nothing alike. Always plot the scatter alongside the coefficient — the number is a summary of the picture, not a substitute for it.

For skewed money variables, Spearman rank correlation is often the more honest summary, because it does not let a handful of very large values set the answer.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Correlation Heatmap

The correlation matrix idea

With many continuous variables, compute every pairwise correlation and lay them out as a colour-shaded heatmap — dark for strong, pale for weak. It is a fast scan for which variables travel together.

Treat the heatmap as a map of questions, not answers. A strong cell says 'look here', then you plot the pair to see whether the relationship is real, curved or driven by an outlier.

Heatmap use	Heatmap misuse
Spotting variable clusters fast	Reading it as a causal structure
Finding near-duplicate variables	Picking predictors by highest r
Sanity-checking derived variables	Reporting the strongest cell as a finding

With 20 variables a heatmap contains 190 correlations, so a few will be large by chance alone. It is a scan, and everything it surfaces is a candidate for a look, not a result.

Its most reliable use is unglamorous: a correlation of 0.99 between two columns usually means one was derived from the other, and you have found a duplicate rather than a relationship.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Disaggregate

Always split before you conclude

A relationship in the whole sample can hide, or even reverse, within subgroups — Simpson's paradox. A scheme can look worse overall yet be better in every state if the states differ in size and baseline.

Disaggregate by sex, sector, caste, state as part of routine EDA. The aggregate can point the opposite way to the truth.

Split by	Because
Rural / urban	The largest single divide in most Indian data
State	Policy, prices and baselines differ enormously
Wealth quintile	Most outcomes gradient sharply
Sex	Many outcomes and many gaps are gendered
Social group	Persistent structural differences

Simpson's paradox is not a curiosity. It happens whenever group sizes differ and the outcome varies by group, which describes most district- and state-level comparisons in India.

The discipline: never report an aggregate difference without having looked at it within the obvious subgroups. If it holds throughout, say so — that is a stronger claim than the aggregate alone.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Eight

Survey Weights & Design Effects in EDA

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Catch

Unweighted exploration misleads

Household surveys are not simple random samples. Some groups are deliberately over-sampled and response rates vary, so the raw sample does not mirror the population. Exploring it unweighted gives the wrong picture.

A raw mean from NFHS or PLFS unit data is an estimate for the sample, not the population. To speak about India, you must apply the survey weights.

Unweighted gives you	Weighted gives you
A description of the sample	An estimate for the population
Useful for finding data errors	Useful for any published number
No design assumptions	Correct selection and non-response adjustment

Both are legitimate, for different questions. The failure is not using unweighted figures — it is publishing them without saying so, so a reader takes a sample description for a national estimate.

Label every table "weighted" or "unweighted". It costs one word and it is the difference between a caveat and a mistake.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

What Weights Are

Weights: how many people a row represents

Survey weight

A number attached to each record saying how many people or households in the population that one respondent stands for. It corrects for unequal selection probabilities and non-response.

If poor districts were over-sampled, their households carry smaller weights so they do not dominate the national figure; under-sampled groups carry larger weights.

Weight type	Applies to	Use for
Household weight	Household records	Household-level estimates
Person / member weight	Member records	Population-level rates
Woman weight	Women's questionnaire	Fertility, anthropometry
Child weight	Child records	Stunting, immunisation

Using the wrong weight is silent and produces a plausible number. A person-level rate computed with household weights understates large households, which in Indian data means understating the poorest.

The codebook names which weight goes with which file. Where it is ambiguous, the survey report's own published figures are the check: reproduce one of them before trusting your own.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Why Over-Sample

Why surveys over-sample on purpose

To report reliably on a small group — a small state, a minority, an urban slum — the survey must interview enough of them. So it deliberately over-samples, then uses weights to restore the correct national balance.

This is a feature, not a flaw. But it means the unweighted sample over-represents those groups — which is exactly why weights exist.

Over-sampled because	Consequence
Small states need usable estimates	They are over-represented in the raw file
Urban slums are policy-relevant	Raw urban share exceeds the true one
A minority group is of interest	Their raw share is inflated
Response rates differ	Weights carry that adjustment too

Over-sampling is good design, not a flaw. It is the only way to get a reliable figure for a small group without a nationally enormous sample.

It does mean the unweighted file is not a miniature India, and any raw share computed from it — of states, of sectors, of groups — is wrong in a knowable direction.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

See the Gap

Weighted vs unweighted: a worked gap

Estimated open-defecation rate, weighted vs unweighted (illustrative)

Illustrative

State estimates match, but the national figure differs: an over-sampled high-rate state inflates the unweighted national average. Weighting fixes it — a four-point swing in the headline.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Design Effect

The design effect (DEFF)

Design effect (DEFF)

How much the clustered, weighted design inflates the variance of an estimate compared with a simple random sample of the same size. DEFF = 2 means your effective sample is half the nominal one.

Because households in the same village are similar, clustering means each extra interview adds less new information than a fresh random draw would. The DEFF quantifies that loss.

DEFF value	Reading
1.0	As good as a simple random sample
1.5-2.5	Typical for a clustered household survey
> 4	Strong clustering — check the estimate carefully
Varies by variable	Normal: DEFF is per estimate, not per survey

DEFF is not a property of the survey but of each estimate. Variables that cluster geographically — water source, sanitation, local prices — have much higher design effects than individual attributes like age.

Where you need one number and cannot compute DEFF, published survey reports usually give indicative values. Using an assumed DEFF and saying so beats assuming 1 silently.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Effective Sample

Your sample is smaller than it looks

n = 10,000

nominal sample size

DEFF = 2

typical for a clustered survey

≈ 5,000

effective sample size (n ÷ DEFF)

Ignore the design effect and your confidence intervals will be too narrow — you will claim more precision than the data supports.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

In Practice

Handling weights and design in EDA

Find the weight, cluster (PSU) and stratum variables in the codebook
Apply weights to every estimate meant to describe the population
Use survey-aware tools so standard errors account for the design
Report whether each figure is weighted — and at what level it is valid

Rule of thumb for EDA: explore shapes unweighted to find problems, but report any population number weighted.

Task	Tool that handles design
R	`survey`, `srvyr`
Python	`samplics`, `statsmodels` (partial)
Stata	`svyset` then `svy:` prefix
SPSS	Complex Samples module

Applying weights to the point estimate but not declaring the design gets the number right and the uncertainty wrong — the estimate is unbiased, the confidence interval is too narrow.

Declare stratum, PSU and weight once at the top of the script, then let the tool handle every subsequent estimate. Doing it per-table is how one table ends up unweighted.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Level of Validity

Respect the design's resolution

A survey is only reliable down to the level it was designed for — usually state or district. Using a state-level estimate to claim something about one block asks the data a question it cannot answer.

Common error: drilling NFHS or PLFS to a tiny subgroup until the cell has 11 households, then reporting a precise percentage. Check the unweighted count behind every estimate.

Survey designed for	Do not claim
State-level estimates	A district figure
District-level (NFHS)	A block or village figure
National totals	A small-group rate
Rural and urban separately	A single combined rate without weighting

The temptation is strongest when a district has an interesting value and a small sample. The interest and the small sample are usually the same fact: small samples produce extreme estimates.

Small-area estimation is a real technique for going below the design level, and it is a modelling exercise with its own assumptions — not something EDA can do by slicing further.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Nine

Visual EDA Done Well

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Why Plot

Plotting is the heart of EDA

EDA leans on display because the eye catches what tables hide — skew, clusters, gaps, outliers, curved relationships. The goal in exploration is speed and honesty, not polish.

There is no excuse for failing to plot and look.

— J. W. Tukey & F. Mosteller

The eye catches	A table hides
Skew and long tails	Behind a mean
Two populations	Behind a single average
A curved relation	Behind a correlation of 0.1
A spike at a code	Behind a plausible range
Clustering of gaps	Behind a total missing count

Exploratory plots have one audience — you — and one requirement: that they are honest about the data. Titles, colours and fonts are for later.

Plot every continuous variable once, unconditionally, before deciding which matter. The variables you were not interested in are where the coding errors are.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Small Multiples

Small multiples: compare many at once

Rather than cramming ten states into one tangled chart, draw ten tiny identical charts in a grid — small multiples. Shared scales let the eye compare shapes effortlessly.

Ideal for EDA across states, sectors or social groups: one consumption histogram per state, same axes, side by side. Patterns and exceptions jump out.

Small multiples work when	They fail when
The scales are shared	Each panel is auto-scaled
Panels are ordered meaningfully	Panels are alphabetical by accident
There are 4-30 panels	There are 200
The shape is the message	A precise value is the message

Shared scales are what make the grid comparable; independent scales turn it into thirty separate charts that happen to be adjacent, and the eye will compare them anyway.

Order the panels by something — the median, the outcome, geography — rather than by name. Ordering is free and does much of the analytical work.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Faceting

Faceting: one plot, split by a variable

Faceting

Splitting a single plot into a grid of panels, one per category of a variable — the same scatter or histogram drawn separately for rural and urban, or for each social group.

Faceting is small multiples generated automatically from your data. It is the fastest way to disaggregate visually and catch a Simpson's-paradox reversal.

Facet by	Reveals
Sector	Whether a pattern is urban only
State	Where the aggregate comes from
Sex	Gendered differences in shape, not just level
Wealth quintile	Gradients and their direction
Survey round	Change over time, and comparability breaks

Faceting is the cheapest defence against Simpson's paradox. If the relationship reverses in one panel, you will see it immediately rather than discovering it in review.

Watch the sample size per panel. Facet a 10,000-household survey by district and most panels hold too few cases to show anything but noise.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Box Plots

Box plots: distributions side by side

A box plot draws Tukey's five-number summary as a box (Q1 to Q3) with a median line and whiskers, marking outliers as points. Lined up by group, it compares whole distributions at a glance.

Box plots show centre, spread and outliers together — perfect for comparing MPCE across states. Where a tool lacks them, plotting the quartiles (as we did earlier) is a fair substitute.

Box plot shows	Box plot hides
Median, quartiles, spread	Bimodality
Outliers as individual points	The number of observations
Group comparisons at a glance	The shape between the quartiles

The bimodality blind spot is real: two distributions, one with a single peak and one with two, can produce identical box plots. Where shape matters, overlay the points or use a violin plot.

Print the group n under each box. Comparing a box built on 4,000 households with one built on 90 is otherwise invisible, and the small one will look like a finding.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Honest Axes

Honest axes — usually start at zero

Y starts at 90 (misleading)

Illustrative

Y starts at 0 (honest)

Illustrative

Same data. Truncating the y-axis makes a 4-point rise look like a leap. Bar charts should start at zero.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Overplotting

When the scatter becomes a blob

With tens of thousands of households, a scatter turns into a solid blob and hides its own density. EDA fixes this with transparency, smaller points, sampling, or binning into a 2-D histogram.

Large household surveys almost always overplot. If your scatter is a black cloud, you are seeing the count, not the pattern — thin it out.

Fix	Trade-off
Transparency (alpha)	Still saturates at very high n
Smaller points	Harder to see isolated cases
Random sample of rows	Rare combinations may vanish
2-D binning / hexbin	Individual outliers disappear
Contours	Reads as a model, not the data

Each fix trades density information against outlier information, and the two are usually both of interest — which is why the honest answer is often two plots rather than one.

With survey weights, plain point density is misleading anyway: an over-sampled state looks like a dense cloud. Weight the binning or say the plot is unweighted.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Checklist

An exploratory-plot checklist

Is the variable type matched to the chart (histogram for continuous, bar for categorical)?
Is the baseline honest, and are axes and units labelled?
Have you tried a log scale for skewed money?
Is the plot weighted if it is meant to describe the population?
Did you disaggregate to check for hidden subgroups?

Check	Failing looks like
Chart matches variable type	A bar chart of a continuous variable
Baseline honest	A 4-point rise that looks like a doubling
Axes labelled with units	"Amount" on the y-axis
Log scale considered for money	Ninety per cent of points in one corner
Weighted if it describes people	A sample picture read as a national one
n stated	A percentage with no denominator

Run the list on exploratory plots too, not only published ones. A misleading axis fools its author first, and the finding it suggests is what you go on to test.

Where a chart is not weighted, put "unweighted sample" in the subtitle rather than the methods note. Subtitles travel with screenshots; methods notes do not.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Ten

From EDA to Questions & Hypotheses

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Hand-off

EDA ends where testing begins

Exploration is generative: it surfaces patterns and hunches. But a pattern found by exploring is a hypothesis, not a result. The next, separate step is to test it — ideally on fresh data.

EDA: notice a pattern (median consumption lower in one social group)

→

QUESTION: is the gap real, or sampling noise?

→

HYPOTHESIS: state it precisely, in advance

→

TEST: confirm with appropriate, weighted methods

EDA produces	Testing requires
A pattern noticed in this data	A hypothesis fixed in advance
Many comparisons, informally	A stated comparison, and correction
A plausible mechanism	A design that can distinguish mechanisms
A shortlist worth funding	Ideally fresh or held-out data

The clean version is to split the data at the outset: explore one half, test on the other. It halves your precision and removes the largest single source of false findings, and it is rarely done.

Where a split is impossible, the fallback is honesty about the search: say how many comparisons were looked at before this one was chosen.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Can Do

What EDA can legitimately conclude

Describe the sample: distributions, gaps, structure
Reveal data-quality problems and their extent
Compare groups descriptively (with weights, with caution)
Suggest relationships worth testing
Generate hypotheses and sharpen questions

Legitimate EDA claim	Phrase it as
A distribution	"Median MPCE in the sample was..."
A data-quality problem	"Income is missing for 18%, concentrated in..."
A group comparison	"Weighted median differs by X between..."
A suggested relationship	"This pattern would be worth testing"

Notice that every one of these is descriptive or conditional. None asserts a cause, and none reports significance — those are the two things exploration cannot supply.

Descriptive findings are not lesser findings. Most of what a practitioner needs from a household survey is descriptive, and describing it correctly is the harder half of the work.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Cannot Do

What EDA cannot conclude

Prove causation — a pattern is not a cause
Confirm a hypothesis it was used to find
Establish significance by eyeballing — that needs a test
Generalise below the survey's design level

The cardinal sin: exploring until something 'looks significant', then reporting it as a confirmed finding. That is p-hacking by another name.

Cannot conclude	Because
X causes Y	Nothing here rules out confounding or reverse causation
This effect is significant	The hypothesis came from the data
This holds in village Z	Below the design resolution
The relation is linear	Only if you plotted it and it was

The cardinal sin named on this slide has a literature and a name — p-hacking, or the garden of forking paths — and it does not require dishonesty. Ordinary curiosity produces it.

The defence is procedural rather than moral: pre-register what you will test, keep a log of what you looked at, and report the search alongside the finding.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Forking Paths

Explore enough, and noise looks real

Slice a dataset twenty ways and roughly one slice will show a striking pattern by chance alone. Exploration is meant to roam — which is exactly why its findings must be confirmed elsewhere.

Be honest about how many things you looked at. A surprising subgroup result found on the twentieth cut deserves scepticism, not a headline.

Slices tried	Chance of one striking result
1	About 5% at the usual threshold
5	About 23%
20	About 64%
50	About 92%

The arithmetic assumes independent tests and no real effect, so it is a rough guide rather than a calculation — but the direction is not in doubt, and twenty slices is a modest afternoon.

This is not an argument against exploring. It is an argument for saying how much you explored, so a reader can discount accordingly.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Documenting

Document findings so they travel

State each finding with its denominator and unit
Say whether it is weighted, and at what level it holds
Note the data-quality caveats behind it
Separate what EDA suggested from what was tested

A finding without its caveats is a liability. The caveats are what make it usable by someone else — and by future-you.

Record with every finding	Example
Unit and denominator	Women 15-49 who were tested
Weighted?	Yes, person weights
Level it holds at	State, not district
Data-quality caveats	12% missing, concentrated in two states
Explored or tested	Exploratory — not yet tested

These five lines travel with the number into a slide deck, a press note and someone else's report. Without them the number arrives stripped of everything that qualifies it.

Keep them in the same file that produces the number, so they cannot drift apart. A caveat in a separate document is a caveat that will be lost.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Telling the Story

From exploration to a clear claim

When EDA is done, translate it into plain language a decision-maker can act on: what is typical, where the gaps are, who is missing, and which questions still need a formal test.

Far better an approximate answer to the right question than an exact answer to the wrong one.

— John W. Tukey

Instead of	Say
"Mean MPCE is 3,240"	"Half of households spend less than 2,700 per head"
"18% missing"	"Nearly one in five gave no income figure, mostly the better off"
"r = 0.62"	"States with higher female literacy tend to have lower fertility"
"DEFF = 2"	"This sample is about as informative as 5,000 independent households"

Translation is not simplification. Each right-hand phrasing carries the same content and is harder to misread, which is the test worth applying.

Tukey's line on the slide is often quoted to excuse imprecision. It says the opposite: an approximate answer to the right question, not an approximate answer.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Takeaways

The judgement EDA builds

Look before you model — always plot and summarise first
Know your unit, codes and universe before any statistic
Why a value is missing matters more than how many
Weight any number meant to describe the population
A found pattern is a question, not yet an answer

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Section Eleven

Tools & Reproducibility

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

The Toolkit

Tools for household-survey EDA

Tool	Good for	Note
R + tidyverse	Cleaning, plotting, reproducible analysis	Free; survey & srvyr packages handle weights
Python + pandas	Cleaning, large data, automation	Free; samplics / statsmodels for survey design
Stata	Standard for official microdata	svyset built-in for weights & design; widely used
Spreadsheets	Quick first look, small tables	Fine to start; not for weighted survey estimates

Whichever you pick, choose a tool that understands survey weights and clustering — spreadsheets do not.

Tool	Weights	Learning cost
R + tidyverse + srvyr	Full survey support	Moderate
Python + pandas	Partial; samplics for design	Moderate
Stata	Excellent (svyset)	Licence cost
SPSS	Complex Samples module	Licence cost
Spreadsheet	None	Low — and unreproducible

The spreadsheet row is not a joke entry. A great deal of real analysis happens there, and its failure mode is specific: no record of what was changed, so no way to re-run it.

Choose for what your team can maintain, not what is most powerful. A script nobody else can run is a single point of failure with a person attached to it.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Reproducibility

Scripts, not manual edits

Fragile

Hand-edits in a spreadsheet, no record of what changed. Next month nobody can reproduce the number — including you.

Robust

A documented script from raw extract to result. Re-run it any time, audit every step, hand it to a colleague.

Golden rule, restated: never edit the raw file. Every clean, recode and exclusion lives in code.

Question a reviewer asks	Script answers it
"Where did this number come from?"	A line in a file
"What did you exclude?"	The filter, with its reason
"Can you redo it for 2019?"	Change one path
"Why did this change?"	The diff

Reproducibility is usually argued for on grounds of scientific integrity. In practice the argument that wins is simpler: the same analysis will be asked for again, with one thing changed.

Set a seed for anything random, and record package versions. Both are one line, and both are what make "re-run it" a real instruction rather than an approximate one.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Project Hygiene

A tidy, reproducible project

Keep the raw extract read-only and dated
One script does the cleaning, another the analysis
Name files with versions and dates, not 'final_FINAL_v3'
Record the source, download date and any filters applied
Keep the codebook beside the data

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Where to Get Data

Open microdata you can explore today

microdata.gov.in — MoSPI's NSS & PLFS unit-level data
dhsprogram.com — NFHS / DHS datasets (on request)
data.gov.in — India's open government data portal
censusindia.gov.in — Census tables and maps
World Bank Microdata Library — regional comparisons

Always download the codebook and survey report alongside the data — the file is unreadable without them.

Source	You get	Note
microdata.gov.in	NSS, PLFS unit-level data	Registration; codebooks alongside
dhsprogram.com	NFHS / DHS files	Free on request, per-survey approval
data.gov.in	Aggregated open data	Rarely unit level
censusindia.gov.in	Census tables	Tables, not microdata

Download the documentation in the same session as the data. Portals reorganise, and a datafile whose codebook has moved is substantially less useful than it was.

Record the download date and the exact file name in your project notes. Survey files are revised, and "NFHS-5 household file" is not enough to identify which version you used.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Keep Learning

A short reading list

Exploratory Data Analysis — John W. Tukey (the founding text)
R for Data Science — Wickham & Grolemund (free online)
The Visual Display of Quantitative Information — Edward Tufte
Analysis of Health Surveys — Korn & Graubard (survey design)
Your survey's own report and methodology note — read it first

Pair this deck with ImpactMojo's Data Literacy, Quantitative Methods and Research Ethics 101 courses.

ImpactMojoExploratory Data Analysis 101www.impactmojo.in

Exploratory Data Analysis 101 · Complete

Now go open
the data — and look.

Before the model, before the finding, before the headline: inspect, clean, describe, plot and question. That habit, applied to every household survey you touch, is what makes your numbers trustworthy. Explore the rest of the ImpactMojo 101 Series, free forever.

More 101 Courses Explore ImpactMojo Dataverse

Continue your learning

Sampling Design Studio Data Visualization Flagship

CC BY-NC-ND 4.0·Free Forever·ImpactMojo 101 Series