Linking Caste Lists

Back to Code Studio

How the live code works. Your first Run downloads Pyodide (Python for the browser, about 10 MB) and pandas, so give it up to a minute on a slow connection. Three files are in the working folder: mandal_bihar.csv (168 entries), obc_bihar.csv (132) and sc_bihar.csv (23). This is real data, taken from Jolad and Kalyani's Database of castes (Harvard Dataverse, doi:10.7910/DVN/WT5VIY, CC0), which digitises the Mandal Commission report and the National Commission for Backward Classes' Central List.

Module 1 of 5

Two lists, forty years apart

In 1980 the Second Backward Classes Commission, chaired by B. P. Mandal, listed the backward classes of each state. The Central List of OBCs that governs central government jobs and admissions today was built after 1993 and has been amended by resolution ever since. A researcher who wants to know which communities Mandal named and the Central List kept has to link the two lists name by name, because neither carries the other's identifiers.

Linking two files that share no key is called record linkage. The same problem turns up when you match beneficiary lists to a survey, villages across census rounds, or schools across two years of UDISE. Caste lists make the difficulty unusually visible: one community is spelt several ways, one entry lists several communities, and two different communities can have names a letter apart.

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")
print(len(mandal), "entries on the Mandal list for Bihar (1980)")
print(len(obc), "active entries on the Central List for Bihar")
print(mandal.sample(5, random_state=1).to_string(index=False))

Bihar has 168 entries on the Mandal list and 132 active entries on the Central List. Look at the sample: one entry can carry several names (Dhunia, Dhumian), and brackets hold either another spelling (Sauta (Sota)) or another name.

Try it. Print the entries whose name contains a bracket with obc[obc["name"].str.contains(r"\(")]. How many are there, and what do the brackets say?

Module 2 of 5

Cleaning names before you compare them

Two spellings that a reader sees as the same, Kewat and kewat , are different strings to a computer. So every name is reduced to a key: lower case, letters and spaces only, single spaces, and one key per name the entry lists.

The brackets need a decision. In the Bihar Central List they hold three kinds of thing: a religion ((Muslim), on 23 entries), a limit on area ((only in the districts of Sivan & Rohtas)) and another name for the same community (Chandrabanshi (Kahar)). The function keeps the third and drops the first two.

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")

def names(entry):
    """Every name an entry lists, cleaned for comparison."""
    extra = []
    for inside in re.findall(r"\((.*?)\)", entry):
        low = inside.lower().strip()
        if low == "muslim" or low.startswith(("only", "in ", "this")):
            continue  # a religion or a limit on area, not another name
        extra.append(inside)
    entry = re.sub(r"\(.*?\)", " ", entry) + "," + ",".join(extra)
    out = []
    for part in re.split(r"[,/;\u2014]| or | and | & ", entry):
        part = part.lower().replace("-", " ")
        part = re.sub(r"[^a-z ]", "", part)
        part = re.sub(r"\s+", " ", part).strip()
        if part:
            out.append(part)
    return out

for e in ["Kasab(Kasai) (Muslim)", "Chandrabanshi (Kahar)", "Kewat, Keot",
          "Aguri\u2014Vaishya, Sudi, Halwai"]:
    print(e, "->", names(e))

Kasab(Kasai) (Muslim) becomes two keys, kasab and kasai, and the religion goes. The Mandal entry for Aguri is one long entry in which a dash separates the first name from the rest, so the function splits on it too.

Exact matches

.explode() turns each list of keys into one row per key, and a merge on the key finds every pair of entries that share a cleaned name.

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")

def names(entry):
    """Every name an entry lists, cleaned for comparison."""
    extra = []
    for inside in re.findall(r"\((.*?)\)", entry):
        low = inside.lower().strip()
        if low == "muslim" or low.startswith(("only", "in ", "this")):
            continue  # a religion or a limit on area, not another name
        extra.append(inside)
    entry = re.sub(r"\(.*?\)", " ", entry) + "," + ",".join(extra)
    out = []
    for part in re.split(r"[,/;\u2014]| or | and | & ", entry):
        part = part.lower().replace("-", " ")
        part = re.sub(r"[^a-z ]", "", part)
        part = re.sub(r"\s+", " ", part).strip()
        if part:
            out.append(part)
    return out

m = mandal.assign(key=mandal["name"].map(names)).explode("key")
o = obc.assign(key=obc["name"].map(names)).explode("key")
print(len(m), "Mandal names;", len(o), "Central List names")
pairs = m.merge(o, on="key", suffixes=("_m", "_o"))
print(pairs["mandal_id"].nunique(), "of", len(mandal), "Mandal entries share at least one exact name")
print(pairs[["mandal_id", "obc_id", "key"]].head(6).to_string(index=False))

The 168 Mandal entries carry 272 names and the 132 Central List entries carry 201. 114 Mandal entries share at least one exact name with the Central List. Mandal entry 432 matches Central List entry 325 on seven names at once, because both lists group the trading castes of the Vaishya cluster into a single entry.

Try it. In names(), delete the line extra.append(inside) so that every bracket is thrown away, and run the exact match again. How many Mandal entries do you lose, and which?

Module 3 of 5

Near matches, and why a score cannot decide

The 54 entries left over may be missing from today's list, or spelt differently. difflib.SequenceMatcher from Python's standard library scores how alike two strings are, from 0 to 1. For each unmatched Mandal name the cell finds the most similar Central List name.

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")

def names(entry):
    """Every name an entry lists, cleaned for comparison."""
    extra = []
    for inside in re.findall(r"\((.*?)\)", entry):
        low = inside.lower().strip()
        if low == "muslim" or low.startswith(("only", "in ", "this")):
            continue  # a religion or a limit on area, not another name
        extra.append(inside)
    entry = re.sub(r"\(.*?\)", " ", entry) + "," + ",".join(extra)
    out = []
    for part in re.split(r"[,/;\u2014]| or | and | & ", entry):
        part = part.lower().replace("-", " ")
        part = re.sub(r"[^a-z ]", "", part)
        part = re.sub(r"\s+", " ", part).strip()
        if part:
            out.append(part)
    return out

m = mandal.assign(key=mandal["name"].map(names)).explode("key")
o = obc.assign(key=obc["name"].map(names)).explode("key")
from difflib import SequenceMatcher
pairs = m.merge(o, on="key")
left = m[~m["mandal_id"].isin(pairs["mandal_id"])]
print(left["mandal_id"].nunique(), "Mandal entries with no exact name in common")
keys = o["key"].unique()
rows = []
for mid, key in zip(left["mandal_id"], left["key"]):
    score, best = max((SequenceMatcher(None, key, k).ratio(), k) for k in keys)
    rows.append((mid, key, best, round(score, 2)))
cand = pd.DataFrame(rows, columns=["mandal_id", "mandal_name", "obc_name", "score"])
cand = cand.sort_values("score", ascending=False)
print(cand.head(25).to_string(index=False))

Read the list from the top. churihara and churihar differ by a final vowel, and kaghzi and kagzi by one letter: these look like one name written two ways. Further down, sunri and sunar score 0.80, and so do lodha and lohar. Those are different communities whose names happen to share most of their letters. The score measures spelling, and caste names are short, so a single letter is a large share of the string.

Choosing a cut-off

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")

def names(entry):
    """Every name an entry lists, cleaned for comparison."""
    extra = []
    for inside in re.findall(r"\((.*?)\)", entry):
        low = inside.lower().strip()
        if low == "muslim" or low.startswith(("only", "in ", "this")):
            continue  # a religion or a limit on area, not another name
        extra.append(inside)
    entry = re.sub(r"\(.*?\)", " ", entry) + "," + ",".join(extra)
    out = []
    for part in re.split(r"[,/;\u2014]| or | and | & ", entry):
        part = part.lower().replace("-", " ")
        part = re.sub(r"[^a-z ]", "", part)
        part = re.sub(r"\s+", " ", part).strip()
        if part:
            out.append(part)
    return out

m = mandal.assign(key=mandal["name"].map(names)).explode("key")
o = obc.assign(key=obc["name"].map(names)).explode("key")
from difflib import SequenceMatcher
pairs = m.merge(o, on="key")
left = m[~m["mandal_id"].isin(pairs["mandal_id"])]
keys = o["key"].unique()
rows = []
for mid, key in zip(left["mandal_id"], left["key"]):
    score, best = max((SequenceMatcher(None, key, k).ratio(), k) for k in keys)
    rows.append((mid, key, best, round(score, 2)))
cand = pd.DataFrame(rows, columns=["mandal_id", "mandal_name", "obc_name", "score"])
for cut in (0.95, 0.9, 0.85, 0.8):
    print(cut, (cand["score"] >= cut).sum(), "names would be accepted")
print(cand[cand["score"] >= 0.85].sort_values("score", ascending=False).to_string(index=False))

No cut-off separates the true from the false. At 0.95 nothing is accepted. At 0.85 seven names are accepted, and each needs a reason before you keep it: godhi and godi are plausibly one community, because the Mandal entry also gives Chhavo and the Central List entry Chhava, while kawar and kalwar score 0.91 on spelling alone and need a source, such as the state's own list or an ethnographic survey, before anyone treats them as the same. Lower the cut-off to 0.80 and 26 names come in, including the pairs above that are plainly different.

A score ranks candidates for a person to check. It does not make the decision. Record every accepted near match with the reason you accepted it, so that someone else can disagree with you.

Try it. Add a column decision to cand and fill it by hand for the seven pairs at 0.85 or above, with "same", "different" or "check". Then count each.

Module 4 of 5

One name on two lists

The same cleaning can link the Central List of OBCs to Bihar's list of Scheduled Castes.

import re
import pandas as pd
mandal = pd.read_csv("mandal_bihar.csv")
obc = pd.read_csv("obc_bihar.csv")

def names(entry):
    """Every name an entry lists, cleaned for comparison."""
    extra = []
    for inside in re.findall(r"\((.*?)\)", entry):
        low = inside.lower().strip()
        if low == "muslim" or low.startswith(("only", "in ", "this")):
            continue  # a religion or a limit on area, not another name
        extra.append(inside)
    entry = re.sub(r"\(.*?\)", " ", entry) + "," + ",".join(extra)
    out = []
    for part in re.split(r"[,/;\u2014]| or | and | & ", entry):
        part = part.lower().replace("-", " ")
        part = re.sub(r"[^a-z ]", "", part)
        part = re.sub(r"\s+", " ", part).strip()
        if part:
            out.append(part)
    return out

m = mandal.assign(key=mandal["name"].map(names)).explode("key")
o = obc.assign(key=obc["name"].map(names)).explode("key")
sc = pd.read_csv("sc_bihar.csv")
s = sc.assign(key=sc["name"].map(names)).explode("key")
both = o.merge(s, on="key", suffixes=("_obc", "_sc"))
print(both[["name_obc", "name_sc", "key"]].to_string(index=False))

Six names sit on both lists: Dhobi, Nat, Mehtar, Lalbegi, Halalkhor and Bhangi. They come from three Central List entries, and each is marked (Muslim), which the cleaning function threw away. That qualifier is the whole difference. Paragraph 3 of the Constitution (Scheduled Castes) Order, 1950 allows only a person who professes Hinduism, Sikhism or Buddhism to be a member of a Scheduled Caste, and the Muslim members of these communities appear on the backward classes list instead.

The general point: cleaning decides what a match means. A rule that is right for linking Mandal to the Central List, where religion is not the question, is wrong for linking the OBC and SC lists, where it is the question. Write the rule down next to the result.

Try it. Change names() so that it keeps muslim as part of the key, and run the cell again. What is left?

Module 5 of 5

What a match can and cannot tell you

Every match here is a match of spellings. A Mandal entry with no match has not been shown to be dropped from the Central List: it may be listed under a name neither list prints, or folded into another entry. A match has not been shown to be the same people either: two communities in different parts of a state can share a name.

Report the count of exact matches and the count of near matches separately, and say how the near ones were decided.
Keep the original text of both entries in the output, beside the keys you matched on.
Treat the cut-off as a choice you made and report what happens to the result if it moves.
Check a sample of matches and non-matches against a third source before you publish a rate.

Try it. Repeat the exact match for the whole country with the full Database of castes, and compare the share of Mandal entries matched across states. Which states fall well below Bihar's 114 of 168, and is that a difference in lists or in spelling?

Where next

→

Linking caste lists

Two lists, forty years apart

Cleaning names before you compare them

Exact matches

Near matches, and why a score cannot decide

Choosing a cut-off

One name on two lists

What a match can and cannot tell you

Where next

Caste Lists Explorer

Caste Studies 101

pandas for Development Data

Database of castes