ImpactMojo ImpactMojo
Membership

Bivariate Analysis 101

Survey design effects, assumptions testing & advanced methods
ImpactMojo Workshop Series • Robust Analysis for Complex Survey Data
75-90 Minutes

Workshop 2: Survey data complexities & robust methods

Target Audience: Researchers working with complex survey data who need to account for design effects and assumption violations

Prerequisites: Workshop 1 or equivalent knowledge of basic bivariate analysis

Materials Needed: Statistical software with survey analysis capabilities, complex survey datasets

Learning objectives

By the end of this workshop, participants will be able to:

Part 1: Survey design effects in bivariate analysis

25 minutes

The sampling design matters: NFHS clustering effects

The Problem: Analyzing child immunization rates by state using NFHS-style survey data (illustrative)

Naive Approach:

  • Treat all 230,000 children as independent observations
  • Standard t-test for state comparisons
  • Result: Nearly all state differences "highly significant" (p < 0.001)

Survey-Aware Approach:

  • Account for two-stage clustering (PSU → household → child)
  • Apply sampling weights for population representation
  • Calculate design-based standard errors
  • Result: Much larger confidence intervals, fewer "significant" differences

The Impact: Design-corrected standard errors are larger, often by a wide margin. What seemed like strong evidence became much more uncertain.

Ignoring survey design doesn't just affect statistical precision - it can lead to completely wrong policy conclusions.

Understanding design effects

Design Effect (DEFF) = Design-based variance / Simple random sample variance

  • DEFF = 1: Survey design is as efficient as simple random sampling
  • DEFF > 1: Survey design reduces precision (most common)
  • DEFF < 1: Survey design improves precision (rare, due to stratification)
  • Typical values: 1.5-3.0 for household surveys, higher for rare outcomes
Clustering effects

Cause: Observations within clusters are more similar than random

Impact: Reduces effective sample size, inflates standard errors

Solution: Design-based standard errors, finite population correction

Sampling weights

Cause: Unequal selection probabilities across subgroups

Impact: Biased estimates if weights ignored

Solution: Weighted analysis for all population estimates

Stratification

Cause: Systematic sampling within homogeneous strata

Impact: Usually improves precision (DEFF < 1)

Solution: Account for stratification in variance estimation

Multiple stages

Cause: Hierarchical sampling (PSU → household → individual)

Impact: Complex correlation structure

Solution: Multi-level variance estimation

Problem set: Survey design impact assessment (15 minutes)

Scenario: Comparing child stunting rates between urban and rural areas using NFHS-style survey data (illustrative figures, not NFHS-5 estimates)

Sample Data: Child Stunting Analysis • Total children (12-59 months): 185,000 • Urban sample: 65,000 children, 35% stunted • Rural sample: 120,000 children, 42% stunted • Design: 2-stage clustering, 707 PSUs • Average cluster size: 261 children • Intracluster correlation (ICC): 0.15 for stunting
Problem 1: Naive vs. design-based analysis (8 minutes)

Calculate both approaches:

Approach Urban % (SE) Rural % (SE) Difference (SE) 95% CI p-value
Naive (ignore design) 35% (±____) 42% (±____) 7% (±____) _____ to _____ _____
Design-based 35% (±____) 42% (±____) 7% (±____) _____ to _____ _____
# Standard error calculations: # Naive SE for proportion = sqrt(p*(1-p)/n) # Design-based SE ≈ Naive SE × sqrt(DEFF) # DEFF ≈ 1 + (cluster_size - 1) × ICC # DEFF = 1 + (261 - 1) × 0.15 = 40 # For difference in proportions: # SE_diff = sqrt(SE_urban² + SE_rural²)

Interpretation Questions:

  • How much larger are the design-based standard errors?
  • Does the urban-rural difference remain statistically significant?
  • What is the design effect (DEFF) for this analysis?
  • How would this affect sample size planning for future surveys?
Problem 2: Subgroup analysis planning (7 minutes)

Research Question: How does the urban-rural stunting gap vary by wealth quintile?

Wealth Quintile Effective Sample Size Expected SE (%) Minimum Detectable Difference
Poorest (Q1) _____ (37,000 ÷ DEFF) _____ _____ % points
Richest (Q5) _____ (37,000 ÷ DEFF) _____ _____ % points

Design Questions:

  • Which quintiles have adequate power for detecting 5% point differences?
  • How would you adjust for multiple comparisons across 5 quintiles?
  • What alternative analysis strategies would you consider?

Part 2: Testing and handling assumption violations

25 minutes

Common assumption violations and robust alternatives

When assumptions fail

Non-normality: Highly skewed income data

Unequal variances: Different spread across groups

Outliers: Extreme values affecting means

Small samples: Central limit theorem doesn't apply

Clustered data: Non-independent observations

Robust alternatives

Non-parametric tests: Mann-Whitney, Kruskal-Wallis

Bootstrap methods: Empirical confidence intervals

Welch's t-test: Unequal variances

Trimmed means: Reduce outlier influence

Permutation tests: No distributional assumptions

Diagnostic framework for choosing methods

4-Step diagnostic protocol

Step 1: Visual Inspection

  • Histograms and Q-Q plots for normality
  • Box plots for outliers and group differences
  • Scatter plots for linearity (correlations)

Step 2: Formal Tests

  • Shapiro-Wilk test for normality (small samples)
  • Levene's test for equal variances
  • Durbin-Watson test for independence

Step 3: Effect Assessment

  • How severe are the violations?
  • Do they affect conclusions substantially?
  • Is the analysis robust to violations?

Step 4: Method Selection

  • Choose most appropriate alternative
  • Document assumptions and decisions
  • Report sensitivity analyses

Problem set: Robust analysis methods (15 minutes)

Dataset: Household expenditure data with known distributional challenges

Household Monthly Expenditure by Education Level: • No Education: n=2,500, median=₹8,500, mean=₹12,400, SD=₹18,200 • Primary: n=1,800, median=₹11,200, mean=₹15,800, SD=₹22,100 • Secondary: n=2,200, median=₹15,600, mean=₹21,300, SD=₹28,900 • Higher: n=1,500, median=₹28,400, mean=₹42,600, SD=₹45,300 Distribution characteristics: • Highly right-skewed (many poor households, few very rich) • Increasing variance with education level • 5% extreme outliers (>3 SD from group mean)
Problem 1: Assumption checking (5 minutes)

Identify violations:

Assumption Evidence Violated? Severity
Normality Mean >> Median in all groups Y/N Mild/Moderate/Severe
Equal variances SD ranges from ₹18K to ₹45K Y/N Mild/Moderate/Severe
No outliers 5% extreme values identified Y/N Mild/Moderate/Severe
Problem 2: Method comparison (10 minutes)

Compare four analytical approaches:

Method Test Statistic p-value Effect Size Recommendation
Standard ANOVA (means) F = 375.7 < 0.001 η² = 0.12 Use/Don't use
Welch ANOVA (unequal var) F = 228.9 < 0.001 η² = 0.12 Use/Don't use
Kruskal-Wallis (medians) H = 1,847 < 0.001 η² = 0.23 Use/Don't use
Bootstrap ANOVA F = 375.7 < 0.001 Bootstrap CI Use/Don't use

Decision Framework:

  • Research question: Are you interested in means or medians?
  • Robustness: How sensitive are conclusions to method choice?
  • Interpretability: Which approach is easiest to communicate?
  • Precedent: What do similar studies use?

Your Recommendation: Which method would you choose and why? Consider both statistical validity and practical interpretation.

Part 3: Advanced effect size interpretation

15 minutes

Moving beyond statistical significance

Effect size guidelines for development research

Analysis Type Effect Size Measure Small Medium Large Policy Relevant
Group differences Cohen's d 0.2 0.5 0.8 Context-dependent
Correlations Pearson's r 0.1 0.3 0.5 r > 0.2 often meaningful
Variance explained R² or η² 0.01 0.06 0.14 Even 1% can matter
Categorical associations Cramer's V 0.1 0.3 0.5 V > 0.15 often meaningful

Development-Specific Considerations:

  • Small effects matter: 5% improvement in child mortality is huge
  • Cost considerations: Expensive interventions need larger effects
  • Equity focus: Effects on marginalized groups may be more important
  • Scalability: Small effects across millions of people = large impact

Effect size interpretation exercise (10 minutes)

Scenario: Evaluating the relationship between mother's education and child vaccination

Analysis Results: • Sample: 45,000 children from NFHS-5 • Mother's education: 0-20 years of schooling • Child vaccination: Complete schedule by age 2 (Yes/No) Statistical Results: • Point-biserial correlation: r = 0.31 (p < 0.001) • Logistic regression: OR = 1.18 per year of education • Effect size (Cohen's d): 0.67 • Variance explained: R² = 0.096 (9.6%) Practical Results: • No education: 58% fully vaccinated • Primary (5 years): 71% fully vaccinated • Secondary (10 years): 81% fully vaccinated • Higher (15 years): 89% fully vaccinated

Multi-Dimensional Interpretation:

Perspective Metric Value Interpretation
Statistical Correlation (r) 0.31 _____ (Small/Medium/Large)
Clinical Rate difference 31 percentage points _____ (Meaningful/Not meaningful)
Policy Number needed to treat _____ (1/0.31) _____ mothers need education for 1 additional vaccination
Explanatory Variance explained 9.6% _____ % of vaccination differences explained

Integration Questions:

  1. Is a 31 percentage point difference between no education and higher education practically significant for public health?
  2. How would you communicate the 9.6% variance explained to policymakers?
  3. What other factors might explain the remaining 90.4% of variance?
  4. Would you recommend education interventions based on these results?

Part 4: Multiple comparisons and subgroup analysis

15 minutes

The multiple testing problem

The Problem: When testing multiple hypotheses simultaneously, the probability of finding at least one "significant" result by chance increases rapidly.

Example: Testing 20 independent comparisons at α = 0.05

  • Expected false positives: 20 × 0.05 = 1 spurious finding
  • Family-wise error rate: 1 - (0.95)²⁰ = 64% chance of ≥1 false positive
  • Solution needed: Adjust significance levels or interpret more conservatively
Adjustment Method When to Use Adjustment Pros/Cons
Bonferroni Independent tests, conservative approach α/m (m = number of tests) Simple but overly conservative
Holm-Bonferroni Sequential testing, more power Step-down procedure Less conservative than Bonferroni
False Discovery Rate Exploratory analysis, many tests Control proportion of false discoveries Higher power, acceptable error rate
No Adjustment Pre-specified hypothesis, single test α = 0.05 Use only with strong justification

Multiple comparisons challenge (10 minutes)

Scenario: Testing immunization rate differences across 28 Indian states

Research Question: Which states have significantly different immunization rates? • Number of pairwise comparisons: 28 × 27 ÷ 2 = 378 tests • Significance level: α = 0.05 • Results: 89 comparisons showed p < 0.05 Without adjustment: 89 "significant" differences With Bonferroni: α = 0.05/378 = 0.00013, only 12 remain significant With FDR (q = 0.05): 34 remain significant

Strategic Decisions:

Approach Significant Differences Interpretation Recommendation
No adjustment 89 About one in four state pairs differ significantly Use/Don't use
Bonferroni 12 Only extreme differences are significant Use/Don't use
FDR control 34 Balance between discovery and false positives Use/Don't use
Effect size focus Focus on magnitude, not p-values Emphasize practical significance Use/Don't use

Alternative Strategies:

  • Planned contrasts: Test only theoretically motivated comparisons
  • Clustering approach: Group similar states, then test group differences
  • Continuous analysis: Model immunization rate as function of state characteristics
  • Hierarchical analysis: Multi-level modeling with state random effects

Your Recommendation: Given this is exploratory analysis for policy prioritization, which approach would you choose? Consider both statistical validity and practical utility.

Integration & best practices

5 minutes

Advanced Bivariate Analysis Checklist:

Before Analysis:

  • Understand survey design (clustering, stratification, weights)
  • Check assumptions using visual and formal diagnostics
  • Plan for multiple comparisons if relevant
  • Define practical significance thresholds

During Analysis:

  • Use appropriate survey analysis procedures
  • Apply robust methods when assumptions violated
  • Calculate both statistical and practical effect sizes
  • Document all decisions and alternatives considered

Reporting Results:

  • Report design effects and effective sample sizes
  • Include confidence intervals alongside p-values
  • Discuss practical significance alongside statistical significance
  • Acknowledge limitations and assumption violations

Summary

Choose the method that fits the data and the question. Survey design effects, violated assumptions and multiple tests come with real data, so report how you handled each one.

Advanced analysis resources

Survey Analysis Software:

Robust Methods References:

Effect Size Calculators:

Next Steps in ImpactMojo:

This handout is part of the ImpactMojo 101 Knowledge Series
Licensed under CC BY-NC-ND 4.0 • Free to use with attribution • www.impactmojo.in

For survey analysis code templates, robust method tutorials, and effect size interpretation guides, visit the ImpactMojo platform.