{"type":"script","doc_desc":{"producers":[{"name":"Reproducibility WBG","abbr":"DECDI","affiliation":"World Bank - Development Impact Department","role":"Verification and preparation of metadata"}],"prod_date":"2025-12-17","version":"1"},"project_desc":{"authoring_entity":[{"name":"Anja Sautmann","affiliation":"World Bank","email":"asautmann@worldbank.org"},{"name":"Jason Abaluck","affiliation":"Yale School of Management, Yale University and the National Bureau of Economic Research","email":"jason.abaluck@yale.edu"},{"name":"Robert Pless","affiliation":"Department of Computer Science, George Washington University","email":"pless@gwu.edu"},{"name":"Nirmal Ravi","affiliation":"eHealth Africa EHA Clinics Nigeria and Department of Emergency Medicine, George Washington University","email":"nirmal.ravi@ehealthafrica.org"},{"name":"Aaron Schwartz","affiliation":"Department of Medical Ethics and Health Policy and  Division of General Internal Medicine, University of Pennsylvania Perelman School of Medicine, and Crescenz VA Medical Center","email":"aaron.schwartz@pennmedicine.upenn.edu"}],"title_statement":{"title":"Reproducibility package for Can LLMs Improve Healthcare Delivery? Evidence From Physician Review And Objective Testing","idno":"RR_NGA_2025_481"},"data_statement":"All data is limited-access and has not been included in the reproducibility package. For more details, please refer to the README file","software":[{"name":"Stata","version":"18 MP"}],"scripts":[{"title":"Reproducibility package for Can LLMs Improve Healthcare Delivery? Evidence From Physician Review And Objective Testing","date":"2025-12","notes":"Computational reproducibility verified by Development Impact (DECDI) Analytics team, World Bank.","instructions":"See README in reproducibility package.","file_name":"RR_NGA_2025_481","zip_package":"RR_NGA_2025_481.zip","dependencies":"Stata dependencies are listed in the ado folder."}],"repository_uri":[{"name":"Reproducible Research Repository (World Bank)","uri":"https:\/\/reproducibility.worldbank.org"}],"production_date":"2025-12-17","abstract":"We deployed large language model (LLM) decision support using GPT4 for health workers at two outpatient clinics in Nigeria. For each patient, health workers drafted care plans that were optionally revised after LLM feedback. We compared unassisted and assisted plans using (i) blinded randomized assessments by on-site physicians who assessed and treated the same patients and (ii) results from laboratory tests for common conditions. Academic physicians performed blinded retrospective reviews of a subset of notes. Providers reported high satisfaction with LLM feedback, and retrospective academic reviewers rated LLM-assisted plans more favorably.  However, on-site physicians observed little to no improvement in diagnostic alignment or treatment decisions. Objective testing showed mixed effects of LLM-assistance, with reduced over testing for malaria but increased over testing for urinary tract infection and anemia. This highlights a gap between chart-based reviews and real-world clinical relevance that may be especially important in evaluating the effectiveness of LLM based interventions.","geographic_units":[{"name":"Nigeria","code":"NGA"}],"output":[{"type":"Working Paper","description":"Policy Research Working Papers (PRWP) WPS11298","title":"Can LLMs Improve Healthcare Delivery? Evidence From Physician Review And Objective Testing","uri":"https:\/\/documents.worldbank.org\/en\/publication\/documents-reports\/documentdetail\/099814501152625141"}],"language":[{"name":"English","code":"EN"}],"technology_requirements":"Run time: ~ 5 minutes","disclaimer":"The materials in the reproducibility packages are distributed as they were prepared by the staff of the International Bank for Reconstruction and Development\/The World Bank. The findings, interpretations, and conclusions expressed in this event do not necessarily reflect the views of the World Bank, the Executive Directors of the World Bank, or the governments they represent. The World Bank does not guarantee the accuracy of the materials included in the reproducibility package.","license":[{"name":"Modified BSD3","uri":"https:\/\/opensource.org\/license\/bsd-3-clause\/"}],"contacts":[{"name":"Anja Sautmann","affiliation":"World Bank","email":"asautmann@worldbank.org"},{"name":"Reproducibility WBG","affiliation":"World Bank","email":"reproducibility@worldbank.org"}],"technology_environment":"Paper exhibits were reproduced on a computer with the following specifications:\n\u2022 OS: Windows 11 Enterprise\n\u2022 Processor: INTEL(R) XEON(R) PLATINUM 8562Y+ 2.80 GHz (4 processors)\n\u2022 Memory available: 32.0 GB","reproduction_instructions":"1. **Secure Access to Data:** Access the datasets not included in the package. See subsection Datasets and the README for more details.\n2. **Download and Place Data:** Once the data is accessed, users should place it in the appropriate folder.\n3. **Run the Package:** After placing the data in the folder, run the files in the order:\n      - Update the global in line 7 of the do-file \"main\" to your folder's location and run the do-file.\n\nSince all the data is not included, the package includes the results produced by replicators. These files can be used to review the results presented in the paper.  \n","datasets":[{"name":"Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests 2025","citation":"Abaluck, J., Pless, R., Ravi, N., Sautmann, A., & Schwartz, A. (2025). Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests 2025 [Data set].","uri":"https:\/\/microdatalib.worldbank.org\/index.php\/catalog\/17051\/","access_type":"Data access requires purchase or human approval and is not included in the reproducibility package.","license_uri":"https:\/\/microdatalib.worldbank.org\/index.php\/data-access\/#research_license","license":"Research microdata with license","note":"The analysis uses de-identified survey data, electronic medical records (EMR), patient flow data, clinical reference files, and LLM-generated evaluation outputs. Data were collected between January and October 2025 and include quality-of-care assessments, EMR records with identifiers removed, study flow information, and reference materials compiled by the research team. Raw data files should be placed in build\/data\/. The data can be accessed by World Bank staff via the internal World Bank Microdata Library. A detailed list of datasets is provided in data_has_report.csv.\n"}]},"tags":[{"tag":"DOI"},{"tag":"Limited-access Data"},{"tag":"Open Code"}],"schematype":"script"}