An Empirical Unravelling of Lord’s Paradox

Xiao, Z.; Higgins, S.; Kasim, A.

Abstract

Lord's Paradox occurs when a continuous covariate is statistically controlled for and the relationship between a continuous outcome and group status indicator changes in both magnitude and direction. This phenomenon poses a challenge to the notion of evidence-based policy, where data are supposed to be self-evident. We examined 50 effect size estimates from 34 large-scale educational interventions and found that impact estimates are affected in magnitude, with or without reversal in sign, when there is substantial baseline imbalance. We also demonstrated that multilevel modeling can ameliorate the divergence in sign and/or magnitude of effect estimation, which, together with project specific knowledge, promises to help those who are presented with conflicting or confusing evidence in decision-making.

Citation

Xiao, Z., Higgins, S., & Kasim, A. (2019). An Empirical Unravelling of Lord’s Paradox. The Journal of Experimental Education, 87(1), 17-32. https://doi.org/10.1080/00220973.2017.1380591

• Journal Article Type: Article
• Acceptance Date: Sep 6, 2017
• Online Publication Date: Nov 7, 2017
• Publication Date: 2019
• Deposit Date: Jul 17, 2017
• Publicly Available Date: May 7, 2019
• Journal: Journal of Experimental Education
• Print ISSN: 0022-0973
• Electronic ISSN: 1940-0683
• Publisher: Taylor and Francis Group
• Peer Reviewed: Peer Reviewed
• Volume: 87
• Issue: 1
• Pages: 17-32
• DOI: https://doi.org/10.1080/00220973.2017.1380591
• Keywords: Lord's Paradox, RCT, Multilevel Modelling, Evidence-Based Policy, Evaluation.
• Public URL: https://durham-repository.worktribe.com/output/1354938
• Related Public URLs: https://osf.io/auq24/