A sensitivity test does everything that a significance test does, and better

Gorard, S.

Abstract

This paper reminds readers of some of the problems in using significance testing, and of using “effect” sizes instead. It looks at a simple sensitivity test for effect sizes (the number of counterfactuals needed to disturb a finding or NNTD). Using 1,000 simulations of two sets of 100 random numbers each, the paper shows that the p-values from significance tests and the results from an NNTD analysis are equivalent and interchangeable. Both are really a scaled “effect” size, based on a difference between means, their variance, and the number of cases in the comparison. A similar point could be made for all effect sizes, including R2 from correlation or regression, and odds ratios from tables of categorical variables. As a measure of sensitivity NNTD should be preferred to p-values for several key reasons. NNTD requires fewer, if any, assumptions about the data, permits missing data and measurement error, assesses the robustness of findings in the face of missing data, directly addresses the key question of whether the underlying effect size is zero or not, and is much easier to explain and understand. It has an everyday meaning. Perhaps more importantly as an implication for research methods, a significance test is meant to provide a measure of the probabilistic uncertainty in a research finding, that could have been produced by random sampling variation alone. As used in practice, and illustrated in this paper, it is really nothing of the sort.

Citation

Gorard, S. (2023). A sensitivity test does everything that a significance test does, and better. IOSR journal of research & method in education, 13(2), 50-56. https://doi.org/10.9790/7388-1302045056

• Journal Article Type: Article
• Acceptance Date: Apr 25, 2023
• Online Publication Date: Apr 27, 2023
• Publication Date: 2023-04
• Deposit Date: Apr 26, 2023
• Publicly Available Date: Apr 28, 2023
• Journal: Journal of Research and Method in Education
• Print ISSN: 2320-737X
• Electronic ISSN: 2320-7388
• Peer Reviewed: Peer Reviewed
• Volume: 13
• Issue: 2
• Pages: 50-56
• DOI: https://doi.org/10.9790/7388-1302045056
• Public URL: https://durham-repository.worktribe.com/output/1176519
• Publisher URL: http://www.iosrjournals.org/iosr-jrme/pages/vol13-issue2-Series-4.html