Improving power calculations in educational trials

Singh, Akansha; Uwimpuhwe, Germaine; Vallis, Dimitrios; Akhter, Nasima; Coolen-Maturi, Tahani; Einbeck, Jochen; Higgins, Steve; Culliney, Martin; Demack, Sean

Abstract

The aim of this study was to investigate and empirically derive parameters commonly used for statistical power and sample size calculations to better inform future trial design. Towards achieving this aim, the research project leveraged the richness of the National Pupil Database (NPD) and the Education Endowment Foundation (EEF) Archive to:
I) Estimate unconditional and conditional school-level intra-cluster correlation coefficients (ICCs) for English and maths attainment outcomes at four educational Key Stages – Early Years Foundation Stage (EYFS), Key Stage 1 (KS1), Key Stage 2 (KS2) and Key Stage 4 (KS4).
II) Estimate correlation coefficients between test scores at pupil and school level for English and maths for three subsequent Key Stages – EYFS to KS1, KS1 to KS2, and KS2 to KS4, along with explanatory power of three subsequent Key Stage pre-test scores at school and pupil level.
III) Draw from NPD derived estimates from objectives I and II to provide examples of MDES calculations for each Key Stage.
IV) Build on Allen et al.’s (2018) work which analysed the test properties of the major commercial assessments that have been used by the EEF. The aim was to assess the value of using a commercial pre-test in explaining variation at post-test, and to examine the absolute reduction in the minimum detectable effect size (MDES) achieved by commercial tests relative to NPD data.
The empirical estimation of ICCs and correlations was also conducted for pupils eligible for free school meals (FSM).

Citation

Singh, A., Uwimpuhwe, G., Vallis, D., Akhter, N., Coolen-Maturi, T., Einbeck, J., Higgins, S., Culliney, M., & Demack, S. (2023). Improving power calculations in educational trials. Education Endowment Foundation

• Report Type: Project Report
• Online Publication Date: Sep 15, 2023
• Publication Date: 2023-09
• Deposit Date: Aug 21, 2024
• Publicly Available Date: Aug 23, 2024
• Public URL: https://durham-repository.worktribe.com/output/2763068
• Publisher URL: https://educationendowmentfoundation.org.uk/projects-and-evaluation/evaluation/eef-evaluation-reports-and-research-papers/methodological-research-and-innovations/improving-power-calculations-in-educational-trials