Uncovering individualised treatment effects for educational trials

Xiao, ZhiMin; Hauser, Oliver; Kirkwood, Charlie; Li, Daniel Z.; Ford, Tamsin; Higgins, Steven

Abstract

Large-scale Randomised Controlled Trials (RCTs) are widely regarded as “the gold standard” for testing the causal effects of school-based interventions. RCTs typically present the statistical significance of the average treatment effect (ATE), which captures the effect an intervention has had on average for a given population. However, key decisions in child health and education are often about individuals who may be very different from those averages. One way to identify heterogeneous treatment effects across different individuals, not captured by the ATE, is to conduct subgroup analyses. For example, free school meal (FSM) pupils as required for projects funded by the Education Endowment Foundation (EEF) in England. These subgroup analyses, as we demonstrate in 48 EEF-funded RCTs involving over 200,000 students, are usually not standardised across studies and offer flexible degrees of freedom to researchers, potentially leading to mixed, if not misleading, results. Here, we develop and deploy an alternative to ATE and subgroup analysis, a machine-learning and regression-based framework to predict individualised treatment effects (ITEs). ITEs could show where an intervention worked, for which individuals, and to what extent. Our findings have implications for decision-makers in fields like education, healthcare, law, and clinical practices concerning children and adolescents.

Citation

Xiao, Z., Hauser, O., Kirkwood, C., Li, D. Z., Ford, T., & Higgins, S. (2024). Uncovering individualised treatment effects for educational trials. Scientific Reports, 14(1), Article 22606. https://doi.org/10.1038/s41598-024-73714-z

• Journal Article Type: Article
• Acceptance Date: Sep 20, 2024
• Online Publication Date: Sep 30, 2024
• Publication Date: Sep 30, 2024
• Deposit Date: Sep 20, 2024
• Publicly Available Date: Oct 9, 2024
• Journal: Scientific Reports
• Electronic ISSN: 2045-2322
• Publisher: Nature Research
• Peer Reviewed: Peer Reviewed
• Volume: 14
• Issue: 1
• Article Number: 22606
• DOI: https://doi.org/10.1038/s41598-024-73714-z
• Keywords: Causal inference, Subgroup analysis, Evaluation, Free school meal pupils, RCT, Data science
• Public URL: https://durham-repository.worktribe.com/output/2870294