Do we really need Confidence Intervals in the new statistics?

Gorard, S.

Abstract

This paper compares the use of confidence intervals (CIs) and a sensitivity analysis called the number needed to disturb (NNTD), in the analysis of research findings expressed as ‘effect’ sizes. Using 1,000 simulations of randomised trials with up to 1,000 cases in each, the paper shows that both approaches are very similar in outcomes, and each one is highly predictable from the other. CIs are supposed to be a measure of likelihood or uncertainty in the results, showing a range of possible effect sizes that could have been produced by random sampling variation alone. NNTD is supposed to be a measure of the robustness of the effect size to any variation, including that produced by missing data. Given that they are largely equivalent and interchangeable under the conditions tested here, the paper suggests that both are really measures of robustness. It concludes that NNTD is to be preferred because it requires many fewer assumptions, is more tolerant of missing data, is easier to explain, and directly addresses the key question of whether the underlying effect size is zero or not.

Citation

Gorard, S. (2019). Do we really need Confidence Intervals in the new statistics?. International Journal of Social Research Methodology, 22(3), 281-291. https://doi.org/10.1080/13645579.2018.1525064

• Journal Article Type: Article
• Acceptance Date: Sep 13, 2018
• Online Publication Date: Sep 26, 2018
• Publication Date: 2019
• Deposit Date: Sep 10, 2018
• Publicly Available Date: Mar 26, 2020
• Journal: International Journal of Social Research Methodology
• Print ISSN: 1364-5579
• Electronic ISSN: 1464-5300
• Publisher: Taylor and Francis Group
• Peer Reviewed: Peer Reviewed
• Volume: 22
• Issue: 3
• Pages: 281-291
• DOI: https://doi.org/10.1080/13645579.2018.1525064
• Public URL: https://durham-repository.worktribe.com/output/1320285