Director of Postgraduate Taught Programmes Mark Rubin mark.rubin@durham.ac.uk
Professor
The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.
Rubin, M. (2023). Type I Error Rates Are Not Usually Inflated
Working Paper Type | Working Paper |
---|---|
Publication Date | Nov 6, 2023 |
Deposit Date | Dec 11, 2023 |
Publicly Available Date | Jan 4, 2024 |
Keywords | exploratory analyses; false positives; forking paths; HARKing; model misspecification; multiple comparisons; multiple testing; optional stopping; p-hacking; questionable research practices; replication crisis; selective inference; significance testing; st |
Public URL | https://durham-repository.worktribe.com/output/2022429 |
Publisher URL | https://doi.org/10.31222/osf.io/3kv2b |
Published Working Paper
(360 Kb)
PDF
Licence
http://creativecommons.org/licenses/by-nd/4.0/
Publisher Licence URL
http://creativecommons.org/licenses/by-nd/4.0/
Questionable Metascience Practices
(2023)
Journal Article
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
Apache License Version 2.0 (http://www.apache.org/licenses/)
Apache License Version 2.0 (http://www.apache.org/licenses/)
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search