pandas for Development Data

Back to Code Studio

How the live code works. Your first Run downloads Pyodide (Python for the browser, about 10 MB) and pandas; numpy and matplotlib follow when a cell needs them. Give the first click up to a minute on a slow connection; after that it is quick. Two files are already in the working folder: households.csv and districts.csv. Both are illustrative data, invented for teaching. The district names are real places, but no number here describes them.

Module 1 of 9

Reading data into a DataFrame

pandas is Python's library for tables. A table is a DataFrame: rows with an index (the row labels on the left) and named columns, each column a Series of one type. This course also uses numpy for weighted means and matplotlib for charts.

The data is a household survey of 240 households across 10 districts in four states. Illustrative data, invented for teaching pd.read_csv() reads it.

import pandas as pd
hh = pd.read_csv("households.csv")
print(hh.head())

.head() shows the first five rows. The ... in the middle means pandas hid some columns to fit the width; the last line says the table has 13 columns. The numbers 0 to 4 on the left are the index, which pandas adds when the file has none.

import pandas as pd
hh = pd.read_csv("households.csv")
hh.info()

.info() lists every column with its count of non-missing values and its type. All 240 rows are complete. int64 and float64 are numbers; object is text, which includes the four Yes/No indicators.

import pandas as pd
hh = pd.read_csv("households.csv")
print(hh.shape)
print(type(hh))
print(hh["district"].unique())

.shape is (rows, columns): (240, 13). .unique() lists the 10 districts.

Try it. Change "households.csv" to "districts.csv" in the third cell. What is its shape?

Module 2 of 9

Selecting, filtering and new columns

Square brackets with one name return one column as a Series; with a list of names they return a DataFrame. .loc selects by label and .iloc by position.

import pandas as pd
hh = pd.read_csv("households.csv")
print(hh["monthly_pc_exp"].head(3))
print(hh[["hh_id", "district", "monthly_pc_exp"]].head(3))
print(hh.loc[0:2, ["district", "caste"]])
print(hh.iloc[0:3, 0:4])

Note the difference: hh.loc[0:2, ...] includes row label 2 and returns three rows, while hh.iloc[0:3, 0:4] stops before position 3, the usual Python rule. Use .loc with names in analysis code; positions break when someone adds a column.

Filtering rows

A condition on a column gives a True/False Series. Put it inside square brackets to keep the True rows. Combine conditions with & (and) or | (or), and wrap each in parentheses.

import pandas as pd
hh = pd.read_csv("households.csv")
rural_f = hh[(hh["area"] == "Rural") & (hh["head_gender"] == "Female")]
print(rural_f[["hh_id", "district", "caste", "hh_size", "monthly_pc_exp"]]
      .sort_values("monthly_pc_exp"))

26 rural households are headed by a woman, and the poorest, in Gaya, spends ₹880 per person per month. The numbers on the left are the original row labels, kept through the filter and the sort.

import pandas as pd
hh = pd.read_csv("households.csv")
out = (hh.query("district in ['Gaya', 'Purnia', 'Patna'] and head_edu_years <= 5")
         [["hh_id", "district", "head_edu_years", "monthly_pc_exp"]]
         .sort_values("monthly_pc_exp", ascending=False))
print(len(out), "households")
print(out.head(10))

.query() takes the condition as a string, which reads closer to plain language. The three Bihar districts have 41 households whose head had five years of schooling or fewer; ascending=False sorts from largest down.

Try it. Select urban households in Kozhikode and Wayanad whose head had more than 10 years of schooling, and show only hh_id, district and land_acres.

Adding columns

.assign() returns a copy of the table with new columns, computed from existing ones row by row. You can also write hh["new"] = ..., which changes hh in place.

import pandas as pd
hh = pd.read_csv("households.csv")
out = hh.assign(annual_pc_exp=hh["monthly_pc_exp"] * 12,
                hh_monthly_exp=hh["monthly_pc_exp"] * hh["hh_size"],
                below_2000=hh["monthly_pc_exp"] < 2000)
print(out[["hh_id", "district", "monthly_pc_exp", "annual_pc_exp",
           "hh_monthly_exp", "below_2000"]].head(8))

below_2000 is a True/False flag, the way to mark households under a threshold such as a poverty line. Household 7 has one member, so its household total equals its per-person figure.

Counting

import pandas as pd
hh = pd.read_csv("households.csv")
print(hh["caste"].value_counts())
print(hh.value_counts(["area", "caste"]).sort_index())

.value_counts() counts each value, largest first. Given a list of columns it counts each combination, sorted here with .sort_index(). OBC households are the largest group (108 of 240), and 20 are ST, of whom only 5 are urban.

Try it. Add a column big_hh that is True when hh_size >= 6, then count it by area.

Module 3 of 9

Group and aggregate

.groupby() splits the table by one or more columns; .agg() reduces each group to one row. Named aggregation, new_name=("column", "function"), sets the output column names in the same line. A share is the mean of a True/False Series, so 100 * (s == "Yes").mean() is a percentage.

import pandas as pd
hh = pd.read_csv("households.csv")
out = (hh.groupby("district")
         .agg(n=("hh_id", "size"),
              mean_exp=("monthly_pc_exp", "mean"),
              median_exp=("monthly_pc_exp", "median"),
              pct_toilet=("has_toilet", lambda s: 100 * (s == "Yes").mean()))
         .sort_values("median_exp", ascending=False)
         .round(1))
print(out)

Look at Udaipur: a mean of ₹3,610.8 and a median of ₹2,795. A few high-spending households pull the mean up, which is normal for consumption and income data. Report the median for a typical household. Kozhikode has the highest median, ₹6,080, and Rewa the lowest toilet coverage, 50 per cent.

import pandas as pd
hh = pd.read_csv("households.csv")
out = (hh.groupby(["caste", "head_gender"], as_index=False)
         .agg(n=("hh_id", "size"), median_exp=("monthly_pc_exp", "median")))
print(out)

as_index=False keeps the group columns as ordinary columns, which is easier to merge or save later. Check n before reading any median: the SC female-headed group holds 2 households, so its median of ₹4,240 says almost nothing. In a real report you would suppress or flag cells that small.

Try it. Group by ["district", "area"] and add pct_bank for bank accounts. Which district has the widest rural-urban gap in median spending?

Module 4 of 9

Merging households with districts

The household file has the district name; a separate district file has the state, region, programme phase and field team. .merge() matches rows on a shared key.

import pandas as pd
hh = pd.read_csv("households.csv")
districts = pd.read_csv("districts.csv")
print(districts)
hh_d = hh.merge(districts, on="district", how="left", validate="many_to_one")
print(hh_d.value_counts(["state", "district"]).sort_index())

how="left" keeps every household. validate="many_to_one" makes pandas raise an error if a district appears twice in the district file, which would duplicate households. Every district has 24 households after the merge, so nothing was lost or doubled.

Merges fail silently when keys are spelt differently. Purnia is often written Purnea. Watch what happens when the district file uses the other spelling:

import pandas as pd
hh = pd.read_csv("households.csv")
districts = pd.read_csv("districts.csv")
# a misspelt district name, the kind you meet in real files
bad = districts.replace({"district": {"Purnia": "Purnea"}})
m = hh.merge(bad, on="district", how="left", indicator=True)
print(m["_merge"].value_counts())
print(m.loc[m["_merge"] == "left_only", "district"].value_counts())

indicator=True adds a _merge column saying where each row came from. 216 rows matched and 24, all from Purnia, are left_only: still in the table but with missing state, so every state total would be quietly short. Check _merge after every merge.

import pandas as pd
hh = pd.read_csv("households.csv")
districts = pd.read_csv("districts.csv")
out = (hh.merge(districts, on="district", how="left")
         .groupby(["state", "programme_phase"], as_index=False)
         .agg(households=("hh_id", "size"),
              pct_transfer=("received_transfer", lambda s: round(100 * (s == "Yes").mean(), 1))))
print(out)

Kerala has rows for phases 1 and 3 only, because none of its districts is in phase 2. A missing row is a fact about the design, worth a footnote in a report.

Try it. Group by field_team instead and compute the share of SHG members for each team.

Module 5 of 9

melt and pivot for indicator tables

The four Yes/No indicators sit in four columns: wide form. .melt() makes it long: one row per household per indicator, so one groupby computes all four shares.

import pandas as pd
hh = pd.read_csv("households.csv")
yn = ["has_toilet", "has_bank_account", "shg_member", "received_transfer"]
long = hh.melt(id_vars=["hh_id", "district"], value_vars=yn,
               var_name="indicator", value_name="answer")
print(len(long))
print(long.head(8))
summary = (long.assign(yes=long["answer"] == "Yes")
               .groupby(["district", "indicator"], as_index=False)["yes"].mean())
summary["pct_yes"] = (100 * summary["yes"]).round()
print(summary.head(8))

240 households times 4 indicators gives 960 rows. The grouped result has one row per district per indicator, 40 in all; the first eight are shown.

.pivot() goes the other way, spreading one column's values into new columns. That gives the table a report needs: districts down the side, indicators across the top.

import pandas as pd
hh = pd.read_csv("households.csv")
yn = ["has_toilet", "has_bank_account", "shg_member", "received_transfer"]
pd.set_option("display.width", 120)
long = hh.melt(id_vars=["hh_id", "district"], value_vars=yn,
               var_name="indicator", value_name="answer")
summary = (long.assign(yes=long["answer"] == "Yes")
               .groupby(["district", "indicator"], as_index=False)["yes"].mean())
summary["pct_yes"] = (100 * summary["yes"]).round()
wide = summary.pivot(index="district", columns="indicator", values="pct_yes")
print(wide)

Rewa has 92 per cent of households with a bank account and 50 per cent with a toilet, and the lowest SHG membership, 4 per cent. Long form is for computing and charting, wide form is for reading.

Try it. Add "head_gender" to id_vars and to the groupby list, then pivot with index=["district", "head_gender"].

Module 6 of 9

Categoricals, bands and labels

Text sorts alphabetically, which puts General before SC and ST. A categorical column stores a fixed set of categories in the order you choose.

import pandas as pd
hh = pd.read_csv("households.csv")
print(hh["caste"].value_counts().sort_index())
hh["caste"] = pd.Categorical(hh["caste"], categories=["SC", "ST", "OBC", "General"])
print(hh["caste"].value_counts().sort_index())
print(hh["caste"].cat.categories)

After the conversion, .sort_index() follows the category order instead of the alphabet. Categoricals also use less memory on large files, which helps on a modest laptop.

import pandas as pd
hh = pd.read_csv("households.csv")
hh["edu_band"] = pd.cut(hh["head_edu_years"],
                        bins=[-1, 0, 5, 10, 100],
                        labels=["No schooling", "1-5 years", "6-10 years", "Over 10 years"])
print(hh["edu_band"].value_counts().sort_index())

pd.cut() turns a number into bands. Each bin includes its right edge, so (-1, 0] is no schooling and (0, 5] is one to five years. 36 heads had no schooling and 25 more than 10 years. A value outside every bin becomes missing, so check the counts add up to 240.

import pandas as pd
hh = pd.read_csv("households.csv")
yn = ["has_toilet", "has_bank_account", "shg_member", "received_transfer"]
hh[yn] = hh[yn].apply(lambda s: s.map({"Yes": True, "No": False}))
hh["area"] = hh["area"].map({"Rural": "Rural household", "Urban": "Urban household"})
print((100 * hh.groupby("area")[yn].mean()).round())

.map() replaces values using a dictionary: Yes/No become True/False, and short codes become readable labels. Here 63 per cent of rural and 92 per cent of urban households have a toilet.

Try it. Band land_acres into landless, under 1 acre, 1 to 2.5 acres and over 2.5 acres, then count each band.

Module 7 of 9

Weighted means and survey weights

Most large surveys do not give every household the same chance of selection. A household drawn with probability 1 in 500 stands for 500 households, so its design weight is 500. Unweighted means describe the sample; weighted means estimate the population. pandas has no weighted mean of its own, so we use np.average(values, weights=...).

Our file has no weight column, so we invent a design for teaching: rural households drawn 1 in 500 and urban households 1 in 1,000.

import pandas as pd, numpy as np
hh = pd.read_csv("households.csv")
# Illustrative design: rural households drawn 1 in 500, urban 1 in 1,000
hh["prob"] = np.where(hh["area"] == "Rural", 1/500, 1/1000)
hh["weight"] = 1 / hh["prob"]
print(hh.value_counts(["area", "weight"]))
toilet = (hh["has_toilet"] == "Yes").astype(float)
print("unweighted mean:", round(hh["monthly_pc_exp"].mean(), 1))
print("weighted mean:  ", round(np.average(hh["monthly_pc_exp"], weights=hh["weight"]), 1))
print("toilet % unweighted:", round(100 * toilet.mean(), 1))
print("toilet % weighted:  ", round(100 * np.average(toilet, weights=hh["weight"]), 1))

Weighting raises mean spending from ₹3,400.9 to ₹3,856.9 and toilet coverage from 72.5 to 77.2 per cent, because urban households, which spend more and more often have toilets, now count for twice as much.

import pandas as pd, numpy as np
hh = pd.read_csv("households.csv")
hh["weight"] = np.where(hh["area"] == "Rural", 500, 1000)
def wmean(g):
    return np.average(g["monthly_pc_exp"], weights=g["weight"])
out = pd.DataFrame({
    "unweighted": hh.groupby("caste")["monthly_pc_exp"].mean(),
    "weighted": hh.groupby("caste")[["monthly_pc_exp", "weight"]].apply(wmean),
}).round(0)
print(out)

For weighted means by group, write a small function and apply it to each group. Weighting moves every caste group upward here, because each contains urban households.

NFHS weights

The DHS Program lists NFHS-5 (2019-21) as India's Standard DHS survey. Its Guide to DHS Statistics (checked October 2026) explains that the women's weight v005 is stored without its decimal point and must be divided by 1,000,000 before use. Three made-up rows show the step:

import pandas as pd, numpy as np
# Three made-up rows shaped like an NFHS women's file
women = pd.DataFrame({"caseid": ["A1", "A2", "A3"],
                      "v005": [1234567, 876543, 2045110],
                      "anaemic": [1, 0, 1]})
women["wt"] = women["v005"] / 1_000_000
print(women)
print("unweighted:", round(women["anaemic"].mean(), 3))
print("weighted:  ", round(np.average(women["anaemic"], weights=women["wt"]), 3))

Weights fix the estimate; they do not fix the standard error. np.average() gives the right point estimate but knows nothing about clusters and strata, so a confidence interval built from it will be too narrow. For design-based standard errors in Python, look at samplics (0.6.1 on PyPI as of October 2026). The most widely used tool is R's survey package, covered in the tidyverse course.

Try it. Set both weights to 500 and run the first weights cell again. Weighted and unweighted results should now agree.

Module 8 of 9

Charts with matplotlib

matplotlib draws on a figure holding one or more axes (plot areas). plt.subplots() makes both; you draw on the axes and call show(fig), a helper this page provides, to display the chart under the code. The first run downloads matplotlib.

import pandas as pd
import matplotlib
matplotlib.use("AGG")
import matplotlib.pyplot as plt
hh = pd.read_csv("households.csv")
m = hh.groupby("caste")["monthly_pc_exp"].median()
fig, ax = plt.subplots(figsize=(6, 3.5))
ax.bar(m.index, m.values, color="#0369A1")
ax.set_title("Median monthly per-capita expenditure by caste")
ax.set_ylabel("Rupees per person per month")
ax.set_ylim(bottom=0)
print(m)
show(fig)

Medians range from ₹2,050 for SC households to ₹3,170 for General. set_ylim(bottom=0) fixes the axis at zero, because a bar's length is its value.

import pandas as pd
import matplotlib
matplotlib.use("AGG")
import matplotlib.pyplot as plt
hh = pd.read_csv("households.csv")
fig, ax = plt.subplots(figsize=(6, 4))
for area, colour in [("Rural", "#047857"), ("Urban", "#0369A1")]:
    d = hh[hh["area"] == area]
    ax.scatter(d["head_edu_years"], d["monthly_pc_exp"], s=14, alpha=0.7,
               color=colour, label=area)
ax.set_yscale("log")
ax.set_xlabel("Years of schooling, household head")
ax.set_ylabel("Rupees per person per month (log scale)")
ax.legend()
show(fig)

Each point is a household. Spending is skewed, so a log scale spreads out the crowded lower end; say so in the axis label, because a reader will otherwise read the gaps as equal.

import pandas as pd
import matplotlib
matplotlib.use("AGG")
import matplotlib.pyplot as plt
hh = pd.read_csv("households.csv")
t = (hh.assign(toilet=hh["has_toilet"] == "Yes")
       .groupby(["district", "area"])["toilet"].mean() * 100)
districts = sorted(hh["district"].unique())
fig, axes = plt.subplots(2, 5, figsize=(10, 4.5), sharey=True)
for ax, d in zip(axes.flat, districts):
    s = t.loc[d]
    ax.bar(s.index, s.values, color="#047857")
    ax.set_title(d, fontsize=9)
    ax.set_ylim(0, 100)
axes[0, 0].set_ylabel("% with toilet")
axes[1, 0].set_ylabel("% with toilet")
fig.suptitle("Households with a toilet, by district and area")
fig.tight_layout()
show(fig)

Small multiples: one panel per district, all on a shared 0 to 100 scale (sharey=True), so panels can be compared at a glance. Giving each panel its own scale would let a short bar in one panel stand for more than a tall bar in another. Kozhikode shows no rural bar because the value is zero: none of its 3 rural households has a toilet. Three households are far too few to report a percentage for.

Honest axes

import pandas as pd
import matplotlib
matplotlib.use("AGG")
import matplotlib.pyplot as plt
hh = pd.read_csv("households.csv")
m = hh.groupby("area")["monthly_pc_exp"].median()
print(m)
fig, (a1, a2) = plt.subplots(1, 2, figsize=(8, 3.2))
a1.bar(m.index, m.values, color="#0369A1"); a1.set_ylim(2000, 5000)
a1.set_title("Axis starts at 2,000")
a2.bar(m.index, m.values, color="#0369A1"); a2.set_ylim(bottom=0)
a2.set_title("Axis starts at zero")
fig.tight_layout()
show(fig)

The median is ₹2,380 for rural and ₹4,515 for urban households, about 1.9 times. In the left chart the axis starts at 2,000 and the urban bar looks more than six times as tall; the right chart, from zero, shows the real ratio. See Data Visualization 101.

Try it. In the bar chart, group by district instead of caste and set figsize=(9, 3.5) so the ten names fit.

Module 9 of 9

A reproducible script

Put the whole analysis in one script that runs from the raw files to the final table with no hand edits. Anyone with the files can then reproduce every number.

import pandas as pd, numpy as np

# 1. Read
hh = pd.read_csv("households.csv")
districts = pd.read_csv("districts.csv")

# 2. Join, and stop if any household fails to match
df = hh.merge(districts, on="district", how="left",
              validate="many_to_one", indicator=True)
assert (df["_merge"] == "both").all(), "unmatched districts"

# 3. Derive: illustrative weights and True/False indicators
yn = ["has_toilet", "has_bank_account", "shg_member", "received_transfer"]
df["weight"] = np.where(df["area"] == "Rural", 500, 1000)
df[yn] = df[yn] == "Yes"

# 4. Summarise: one row per state, weighted per cent
def wpct(g):
    return pd.Series({c: round(100 * np.average(g[c], weights=g["weight"]), 1) for c in yn})
table_out = df.groupby("state")[yn + ["weight"]].apply(wpct)
print(table_out)

# 5. Save, then read back to prove the file is right
table_out.to_csv("state_indicators.csv")
print(len(pd.read_csv("state_indicators.csv")), "rows written")

The script stops with an error if any household fails to match a district, builds the weights and indicators, and writes a four-row state table: Bihar 80.9 per cent toilet coverage, Kerala 82.2, Madhya Pradesh 72.9 and Rajasthan 71.7 (weighted, on illustrative data). It then reads the file back and confirms 4 rows.

assert turns a silent problem into a loud one. Add one after every merge and filter you depend on.
Keep raw files read-only and write outputs to new files.
Keep names, phone numbers and other identifiers out of files you share. Data Protection & the DPDP Act 101 covers what India's Digital Personal Data Protection Act, 2023 asks of survey teams.
On your own laptop, save it as a .py file and run it with python analysis.py, or use a Jupyter notebook and restart and run all cells before sharing.

Try it. Group by ["state", "region"] in step 4 and change the output file name. Run it and check the row count.

Where next

→

pandas for development data