{"type":"script","doc_desc":{"producers":[{"name":"Reproducibility WBG","abbr":"DECDI","affiliation":"World Bank - Development Impact Department","role":"Verification and preparation of metadata"}],"prod_date":"2026-07-29","version":"1"},"project_desc":{"authoring_entity":[{"name":"Carolina Lopez","affiliation":"World Bank","email":"carolina_lopez@worldbank.org"},{"name":"Ezequiel Molina","affiliation":"World Bank","email":"molina@worldbank.org"},{"name":"Matthew D. Krasnow","affiliation":"Harvard University","email":"matt@swiftscore.org"},{"name":"Jostin Kitmang","affiliation":"Harvard University","email":"jkitmang@g.harvard.edu"},{"name":"Carolina Moreira Vasquez","affiliation":"World Bank","email":"cmm9mn@virginia.edu"},{"name":"William J. Krasnow","affiliation":"Swiftscore","email":"will@swiftscore.org"}],"title_statement":{"title":"Reproducibility package for Scaling Up Classroom Observation With Large Language Models: A Reliability Study Of The World Bank's Teach Framework In Peru","idno":"RR_PER_2026_700"},"data_statement":"Some data is restricted and has not been included in the reproducibility package. For more details, please refer to the README file. Intermediate data is forthcoming in the World Bank Microdata Library under a restricted license and can be used to fully reproduce the results in the paper.\n","software":[{"name":"Python","version":"3.11.15"}],"scripts":[{"title":"Reproducibility package for Scaling Up Classroom Observation With Large Language Models: A Reliability Study Of The World Bank's Teach Framework In Peru","date":"2026-07","notes":"Computational reproducibility verified by Development Impact (DECDI) Analytics team, World Bank.","instructions":"See README in reproducibility package.","file_name":"RR_PER_2026_700","zip_package":"RR_PER_2026_700.zip","dependencies":"Python dependencies are listed in the \"requirements.txt\" file."}],"repository_uri":[{"name":"Reproducible Research Repository (World Bank)","uri":"https:\/\/reproducibility.worldbank.org"}],"production_date":"2026-07-29","abstract":"Classroom observation is widely used to measure teaching practices but is costly at scale, limiting its use as a formative tool for teacher improvement in low- and middle-income countries (LMICs). We test whether a large language model (LLM) can agree with a certified human observer as closely as the World Bank's Teach reliability protocol requires of trained raters. Across 199 Peruvian primary-classroom transcripts, we evaluate eight LLM configurations (Gemini 2.0 and 2.5 Flash) varying rubric quality, prompt format, and reasoning depth, and apply the Teach certification rule to the LLM-observer agreement via Monte Carlo simulation. Our best configuration meets the within-one-point certification criterion against the certified observer in 92.9% of simulated exams (95% CI [91.3, 94.5]), with element-level agreement of 81-99% across Teach elements\u2014in the range trained human raters reach, though our gold standard is a single certified field observer rather than the program's master coders. A regression that isolates each design lever shows that rubric content and distributional priors are the two dominant drivers of agreement, while chain-of-thought reasoning and snapshot tagging are not; a minimal-rubric baseline almost never clears the criterion. A reliable LLM scorer can help ministries deliver formative feedback within teacher-support programs more frequently and at lower marginal cost.\n","geographic_units":[{"name":"Peru","code":"PER"}],"keywords":[{"name":"Automated Scoring"},{"name":"Classroom Observation"},{"name":"Large Language Models"},{"name":"Teacher Professional Development"},{"name":"Rubric Engineering"},{"name":"Teach"},{"name":"Peru"}],"topics":[{"id":"I21","uri":"https:\/\/www.aeaweb.org\/econlit\/jelCodes.php?view=jel","vocabulary":"Journal of Economic Literature (JEL)","name":"Analysis of Education","parent_id":"I2"},{"id":" I25","uri":"https:\/\/www.aeaweb.org\/econlit\/jelCodes.php?view=jel","vocabulary":"Journal of Economic Literature (JEL)","name":"Education and Economic Development","parent_id":"I2"},{"id":" C45","uri":"https:\/\/www.aeaweb.org\/econlit\/jelCodes.php?view=jel","vocabulary":"Journal of Economic Literature (JEL)","name":"Neural Networks and Related Topics","parent_id":"C4"},{"id":" C81","uri":"https:\/\/www.aeaweb.org\/econlit\/jelCodes.php?view=jel","vocabulary":"Journal of Economic Literature (JEL)","name":"Methodology for Collecting, Estimating, and Organizing Microeconomic Data \u2022 Data Access","parent_id":"C8"}],"output":[{"type":"Working Paper","description":"Policy Research Working Papers (PRWP)","title":"Scaling Up Classroom Observation With Large Language Models: A Reliability Study Of The World Bank's Teach Framework In Peru","authors":"Matthew D. Krasnow, Carolina Lopez, Jostin Kitmang, William J. Krasnow, Carolina Moreira Vasquez, Ezequiel Molina"}],"language":[{"name":"English","code":"EN"}],"technology_requirements":"Run time: ~5 minutes","disclaimer":"The materials in the reproducibility packages are distributed as they were prepared by the staff of the International Bank for Reconstruction and Development\/The World Bank. The findings, interpretations, and conclusions expressed in this event do not necessarily reflect the views of the World Bank, the Executive Directors of the World Bank, or the governments they represent. The World Bank does not guarantee the accuracy of the materials included in the reproducibility package.","license":[{"name":"MIT License","uri":"https:\/\/opensource.org\/license\/mit"},{"name":"World Bank IGO Rider","uri":"https:\/\/github.com\/worldbank\/metadata-editor\/blob\/main\/WB-IGO-RIDER.md"}],"contacts":[{"name":"Carolina Lopez","affiliation":"World Bank","email":"carolina_lopez@worldbank.org"},{"name":"Reproducibility WBG","affiliation":"World Bank","email":"reproducibility@worldbank.org"}],"technology_environment":"Paper exhibits were reproduced on a computer with the following specifications:\n\u2022 OS: Windows 11 Enterprise\n\u2022 Processor: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz (2.30 GHz) (2 processors)\n\u2022 Memory available: 16.0 GB\n\u2022 Software version: Python 3.11.1","reproduction_instructions":"To reproduce the findings in this paper, a replicator must:\n1. **Secure Access to Data:** Access the datasets not included in the package. See the Datasets section for more details.\n2. **Run the Package:** Restore the environment using `requirements.txt`, set the environment variable AITEACH_USE_CACHED_LLM=1 to run the cached version of the pipeline using the shipped de-identified intermediate data, and run `00_master.py`.\n\nSince not all the data is included, the package includes the results produced by replicators. These files can be used to review the results presented in the paper.","datasets":[{"access_type":"Data access was granted directly to the study authors by the data owners. Intermediate data is forthcoming in the World Bank Microdata Library under a restricted license.","note":"Source: World Bank Global Education Policy Dashboard. This paper uses the classroom observation component of a dataset collected in 2019 from a nationally representative sample of public primary schools in Peru by the World Bank\u2019s Global Education Policy Dashboard team (https:\/\/www.educationpolicydashboard.org\/). The raw classroom video recordings contain personally identifiable information (PII) and are not distributed. The recordings and all data-collection information may be requested from the World Bank\u2019s Global Education Policy Dashboard team at educationdashboard@worldbank.org. De-identified intermediate data is forthcoming in the World Bank Microdata Library under a restricted license. ","license":"Custom license","name":"Global Education Policy Dashboard Survey: Peru 2019","citation":"World Bank. 2019. \"Global Education Policy Dashboard Survey: Peru 2019\" [dataset]. Accessed April 2025. Forthcoming at https:\/\/microdatalib.worldbank.org\/","uri":"Forthcoming at https:\/\/microdatalib.worldbank.org\/"}]},"datacite":{"creators":[{"givenName":"Carolina","familyName":"Lopez","nameType":"Personal","affiliation":[{"name":"World Bank","affiliationIdentifier":"https:\/\/ror.org\/00ae7jd04","affiliationIdentifierScheme":"ROR","schemeUri":"https:\/\/ror.org"}]},{"givenName":"Ezequiel","familyName":"Molina","nameType":"Personal","affiliation":[{"name":"World Bank","affiliationIdentifier":"https:\/\/ror.org\/00ae7jd04","affiliationIdentifierScheme":"ROR","schemeUri":"https:\/\/ror.org"}]},{"givenName":"Matthew D.","familyName":"Krasnow","nameType":"Personal","affiliation":[{"name":"Harvard University","affiliationIdentifier":"https:\/\/ror.org\/03vek6s52","affiliationIdentifierScheme":"ROR","schemeUri":"https:\/\/ror.org"}]},{"givenName":"Jostin","familyName":"Kitmang","nameType":"Personal","affiliation":[{"name":"Harvard University","affiliationIdentifier":"https:\/\/ror.org\/03vek6s52","affiliationIdentifierScheme":"ROR","schemeUri":"https:\/\/ror.org"}]},{"givenName":"Carolina Moreira","familyName":"Vasquez","nameType":"Personal","affiliation":[{"name":"World Bank","affiliationIdentifier":"https:\/\/ror.org\/00ae7jd04","affiliationIdentifierScheme":"ROR","schemeUri":"https:\/\/ror.org"}]},{"givenName":"William J.","familyName":"Krasnow","nameType":"Personal","affiliation":[{"name":"Swiftscore"}]}],"titles":[{"lang":"en","title":"Reproducibility package for Scaling Up Classroom Observation With Large Language Models: A Reliability Study Of The World Bank's Teach Framework In Peru"},{"title":"RR_PER_2026_700","titleType":"Other"}],"publisher":"World Bank","publicationYear":"2026","types":{"resourceType":"Reproducibility package","resourceTypeGeneral":"Other"},"url":"https:\/\/reproducibility.worldbank.org\/index.php\/catalog\/study\/RR_PER_2026_700","language":"en"},"tags":[{"tag":"DOI"},{"tag":"Open Code"},{"tag":"Restricted Data"}],"schematype":"script"}