Skip to main content

Research Repository

Advanced Search

A recipe for disappointment: policy, effect size and the winner’s curse

Simpson, Adrian

A recipe for disappointment: policy, effect size and the winner’s curse Thumbnail


Authors



Abstract

Evidence-based education aims to support policy makers choosing between potential interventions. This rarely involves considering each in isolation; instead, sets of evidence regarding many potential policy interventions are considered. Filtering a set on any quantity measured with error risks the ‘winner’s curse’: conditional on selecting higher valued measures, the measurement likely overestimates the latent value. This paper explains the winner’s curse, illustrates it for one constrained and complete set of educational trials – the UK’s Education Endowment Foundation’s projects, where evidence is summarized with standardized effect size – and shows the results of adjusting for the curse on this set. This analysis suggests selecting policies for higher effect size can result in substantial effect size inflation and in some cases order reversals: one intervention ranking above another on estimated effect size but below it when adjusted. The issue has implications for evaluation programs, power analyses and policy decisions. For example, even in the absence of other problems with interpreting effect size, it can help explain why policies tend to deliver less than promised.

Journal Article Type Article
Acceptance Date Mar 31, 2022
Online Publication Date Jun 7, 2022
Publication Date 2023-12
Deposit Date Apr 7, 2022
Publicly Available Date Jul 20, 2022
Journal Journal of Research on Educational Effectiveness
Print ISSN 1934-5747
Electronic ISSN 1934-5739
Publisher Taylor and Francis Group
Peer Reviewed Peer Reviewed
Volume 16
Issue 4
Pages 643-662
DOI https://doi.org/10.1080/19345747.2022.2066588
Public URL https://durham-repository.worktribe.com/output/1210434

Files

Published Journal Article (2.2 Mb)
PDF

Publisher Licence URL
http://creativecommons.org/licenses/by-nc-nd/4.0/

Copyright Statement
© 2022 The Author(s). Published with license by Taylor & Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.





You might also like



Downloadable Citations