What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’

Gorard, S.; Gorard, J.

Abstract

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The ‘number needed to disturb’ a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or ‘effect’ size disappears, minus the number of cases missing key values. This way of presenting the security of findings has several advantages over the use of significance tests, effect sizes and confidence intervals. It is not predicated on random sampling, full response or any specific distribution of data. It bundles together the sample size, magnitude of the finding and the level of attrition in a way that is standardised and therefore comparable between studies.

Citation

Gorard, S., & Gorard, J. (2016). What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’. International Journal of Social Research Methodology, 19(4), 481-490. https://doi.org/10.1080/13645579.2015.1091235

• Journal Article Type: Article
• Acceptance Date: Aug 23, 2015
• Online Publication Date: Oct 6, 2015
• Publication Date: Jul 1, 2016
• Deposit Date: Apr 23, 2015
• Publicly Available Date: Apr 30, 2015
• Journal: International Journal of Social Research Methodology
• Print ISSN: 1364-5579
• Electronic ISSN: 1464-5300
• Publisher: Taylor and Francis Group
• Peer Reviewed: Peer Reviewed
• Volume: 19
• Issue: 4
• Pages: 481-490
• DOI: https://doi.org/10.1080/13645579.2015.1091235
• Keywords: Significance testing, Sampling, Attrition, Evaluation, Reliability.
• Public URL: https://durham-repository.worktribe.com/output/1440035