Hidden Wording Effects Distort Psychology Scores, But a New Statistical Fix Emerges

Every year, millions of people fill out psychological questionnaires designed to measure everything from grit and self-esteem to anxiety and burnout. These instruments shape hiring decisions, clinical diagnoses, and entire research literatures. Yet a subtle flaw has haunted survey design for decades: when questionnaires mix positively and negatively worded items, respondents often produce answers that reflect the phrasing of the questions rather than the trait being measured. A new study published in Behavior Research Methods by Palmira Faraci and Giuliana Nasonte of the Psychometrics Laboratory at the University Kore of Enna puts a sophisticated statistical tool, random intercept item factor analysis, to the test, and the results suggest it can cleanly separate genuine personality signals from the noise created by question wording.

The problem, known in psychometrics as method variance or the wording effect, arises because scales are frequently built with a deliberate balance of positively and negatively keyed items. The idea sounds sensible: reversing the direction of some questions should discourage automatic agreement and force respondents to read carefully. But the strategy backfires in a measurable way. Items that share the same wording direction tend to correlate with one another for reasons that have nothing to do with the underlying construct. When researchers run a factor analysis on such data, the statistical machinery often obligingly splits the scale into two clusters, one of positively worded items and one of negatively worded items, rather than the single dimension the scale was designed to capture.

This artificial splitting, sometimes called spurious bidimensionality, has real consequences. A researcher who concludes that a questionnaire measures two distinct traits may publish misleading findings, build theories on phantom subfactors, or compute reliability estimates that are inflated or deflated for the wrong reasons. The grit scale, a wildly popular measure of perseverance and passion for long-term goals, has been a particular battleground for this debate. Since its development by Angela Duckworth and colleagues, researchers have argued endlessly about whether grit truly has two facets, perseverance of effort and consistency of interest, or whether the apparent two-factor structure is simply an artifact of the negatively worded items embedded in the short version of the scale.

Faraci and Nasonte tackled this question with random intercept item factor analysis, or RIIFA, a modeling approach originally introduced by Albert Maydeu-Olivares and Donna Coffman in 2006. The core insight of RIIFA is elegant: it adds a random intercept factor to the standard factor model, a latent variable on which every item in the scale loads regardless of its wording direction. This intercept factor absorbs the shared response tendency that cuts across all items, including acquiescence, the general willingness to agree with statements, and other response styles. Once that pervasive method variance is siphoned off into its own latent dimension, the remaining substantive factor can be interpreted as the true trait, purified of the contamination introduced by how the questions happen to be phrased.

To evaluate whether RIIFA delivers on this promise, the researchers conducted two studies using two independent samples from the United Kingdom. The first study analyzed data from 977 participants, and the second from 496. Their test case was the Short Grit Scale, known as Grit-S, an eight-item instrument that mixes positively and negatively worded questions. The choice was strategic: the Grit-S is short, widely used, and famously prone to the wording-effect problem, making it an ideal proving ground for a method intended to restore structural validity to mixed-worded scales.

The first study focused on dimensionality assessment, the task of determining how many latent factors actually underlie a set of items. The researchers compared traditional factor retention techniques, including parallel analysis, a classic method that compares observed eigenvalues against those from random data, with RIIFA-based counterparts that incorporate the random intercept factor. They also employed exploratory graph analysis, a newer network psychometrics approach that treats items as nodes in a network and uses community detection algorithms, borrowed originally from network science methods like the fast unfolding of communities, to identify clusters of tightly connected items. The results were striking. Traditional retention methods consistently overestimated the number of factors, suggesting the Grit-S contained two dimensions when the theoretical expectation was one. In contrast, the RIIFA-based techniques produced unidimensional solutions, and bootstrap analyses, which resample the data thousands of times to gauge stability, showed that these solutions were considerably more stable than those obtained without controlling for the wording effect.

The second study shifted from exploration to confirmation, using confirmatory factor analysis to formally test competing models of the scale’s structure. The researchers fit models with and without a random intercept factor and compared them on both fit and parsimony, the principle that simpler explanations should be preferred unless complexity earns its keep. The model incorporating the random intercept factor achieved the best balance between the two. Its fit statistics were excellent by conventional standards: a root mean square error of approximation of .048, with a confidence interval spanning .022 to .072, a comparative fit index of .984, a Tucker-Lewis index of .974, and a standardized root mean square residual of .027. Values close to .95 or above on the fit indices and below about .06 or .08 on the error measures are typically considered indicative of a well-fitting model, and this model cleared every bar comfortably.

Reliability told a similarly encouraging story. The RIIFA model yielded a hierarchical omega of .84, a coefficient derived from bifactor-style measurement models that estimates how well the total scale score reflects a single dominant common factor after accounting for the variance absorbed by subsidiary dimensions. In practical terms, this means that once the method variance was reallocated to the random intercept factor, the substantive grit factor explained enough common variance to support meaningful interpretation of total scores. The researchers interpret this as evidence that RIIFA does not merely hide the problem; it actively redistributes explained variance between the substantive factor and the method factor, mitigating the artificial bidimensionality that plagues conventional analyses and enhancing the interpretability of the latent structure.

The implications reach well beyond the grit scale. Mixed-worded instruments are ubiquitous across psychology, appearing in measures of self-esteem, core self-evaluations, need for cognition, perceived stress, learning burnout, and countless clinical and organizational questionnaires. For each of these, the same dilemma recurs: researchers must decide whether an emerging second factor represents a genuine substantive distinction or a methodological artifact. Historically, that judgment has been made with tools, such as standard parallel analysis and conventional confirmatory factor analysis, that have no built-in mechanism for separating content from phrasing. The findings of this study suggest that RIIFA offers a principled alternative, and the authors explicitly recommend its application in cases where wording effects threaten the validity of psychometric measurement.

The study also connects to a broader movement in quantitative psychology toward modeling response styles and careless responding rather than simply hoping they wash out in large samples. Recent work has documented how even a few inconsistent respondents can confound the structure of personality survey data, how acquiescence distorts exploratory factor analysis, and how attention checks and response-style detection methods carry their own complications. Faraci and Nasonte’s contribution fits squarely into this research program, demonstrating on real data that a model explicitly designed to partition substantive and method variance can outperform traditional approaches. Importantly, the authors have made their data openly available through the Open Science Framework, and all of the R and Mplus code used in the analyses is accessible as well, lowering the barrier for other researchers to adopt the technique. For a field that depends so heavily on self-report questionnaires, a validated method for ensuring that scores reflect traits rather than question phrasing is not a technical nicety. It is a foundation for the credibility of the measurements themselves, and this study provides compelling empirical grounds for adding random intercept item factor analysis to the standard psychometric toolkit.

Subject of Research: Using random intercept item factor analysis to separate substantive variance from wording-effect method variance in mixed-worded psychological scales

Article Title: Disentangling substantive and method variance in mixed-worded scales: An empirical application of the random intercept item factor analysis (RIIFA)

Article References: Faraci, P., & Nasonte, G. (2026). Disentangling substantive and method variance in mixed-worded scales: An empirical application of the random intercept item factor analysis (RIIFA). Behavior Research Methods, 58(11), Article 299. https://doi.org/10.3758/s13428-026-03163-1

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03163-1

Keywords: psychometrics, random intercept item factor analysis, wording effects, method variance, Short Grit Scale, factor analysis, exploratory graph analysis, parallel analysis, dimensionality, structural validity, confirmatory factor analysis, survey methodology