Skip to main content

Research Repository

Advanced Search

Sophisticated statistics cannot compensate for method effects if quantifiable structure is compromised

Birney, D.P.; Beckmann, J.F.; Beckmann, N.; Stemler, S.E.

Sophisticated statistics cannot compensate for method effects if quantifiable structure is compromised Thumbnail


D.P. Birney

S.E. Stemler


Researchers rely on psychometric principles when trying to gain understanding of unobservable psychological phenomena disconfounded from the methods used. Psychometric models provide us with tools to support this endeavour, but they are agnostic to the meaning researchers intend to attribute to the data. We define method effects as resulting from actions which weaken the psychometric structure of measurement, and argue that solution to this confounding will ultimately rest on testing whether data collected fit a psychometric model based on a substantive theory, rather than a search for a model that best fits the data. We highlight the importance of taking the notions of fundamental measurement seriously by reviewing distinctions between the Rasch measurement model and more generalised 2PL and 3PL IRT models. We then present two lines of research that highlight considerations of making method effects explicit in experimental designs. First, we contrast the use of experimental manipulations to study measurement reactivity during the assessment of metacognitive processes with factor-analytic research of the same. The former suggests differential performance-facilitating and -inhibiting reactivity as a function of other individual differences, whereas factor-analytic research suggests a ubiquitous monotonically predictive confidence factor. Second, we evaluate differential effects of context and source on within-individual variability indices of personality derived from multiple observations, highlighting again the importance of a structured and theoretically grounded observational framework. We conclude by arguing that substantive variables can act as method effects and should be considered at the time of design rather than after the fact, and without compromising measurement ideals.


Birney, D., Beckmann, J., Beckmann, N., & Stemler, S. (2022). Sophisticated statistics cannot compensate for method effects if quantifiable structure is compromised. Frontiers in Psychology, 13, Article 812963.

Journal Article Type Article
Acceptance Date Jan 24, 2022
Online Publication Date Feb 16, 2022
Publication Date 2022
Deposit Date Feb 21, 2022
Publicly Available Date Feb 21, 2022
Journal Frontiers in Psychology
Print ISSN 1664-1078
Publisher Frontiers Media
Peer Reviewed Peer Reviewed
Volume 13
Article Number 812963


Published Journal Article (5.7 Mb)

Publisher Licence URL

Copyright Statement
Copyright © 2022 Birney, Beckmann, Beckmann and Stemler. This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

You might also like

Downloadable Citations