Type I Error Rates Are Not Usually Inflated

Rubin, Mark

Abstract

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

Citation

Rubin, M. (2024). Type I Error Rates Are Not Usually Inflated. Journal of trial and error, 4(2), https://doi.org/10.36850/4d35-44bd

• Journal Article Type: Article
• Acceptance Date: May 11, 2024
• Online Publication Date: Nov 16, 2024
• Publication Date: Nov 16, 2024
• Deposit Date: Dec 11, 2023
• Publicly Available Date: Jan 4, 2024
• Journal: Journal of trial and error
• Electronic ISSN: 2667-1204
• Peer Reviewed: Peer Reviewed
• Volume: 4
• Issue: 2
• DOI: https://doi.org/10.36850/4d35-44bd
• Keywords: exploratory analyses; false positives; forking paths; HARKing; model misspecification; multiple comparisons; multiple testing; optional stopping; p-hacking; questionable research practices; replication crisis; selective inference; significance testing; statistical inference; Type I error inflation; Type I error rate inflation; Type I error rates
• Public URL: https://durham-repository.worktribe.com/output/2022429