Reproducibility package for The Effect Of Survey Mode On Data Quality: Experimental Evidence From Nigeria

In a large-scale experiment in rural Nigeria, we study the role of survey mode — in-person vs. over the phone — for survey measurement and data quality. Our experimental design isolates mode effects from other common sources of errors in surveys and covers twenty outcome measures across topics such as health, labor, shocks, wellbeing, and food security. We consistently find mode effects across outcomes, with phone responses differing from in-person responses by up to 63%, and a median difference of 17–18%. We document that these effects are large relative to other errors in phone surveys such as undercoverage of households without phones. Using a within-respondent design, we measure the full, respondent-level distribution of mode effects and find them to vary much more than averages let on. Respondents with higher education levels are less prone to mode effects whereas mode effects sharply increase in prevalence as respondents face more answer options. We present evidence that mode effects are driven by lower cognitive engagement of respondents over the phone. Given the growing reliance on phone surveys in LMICs, our findings provide novel guidance for survey design and advance the research frontier on how survey mode influences measurement.