At the heart of the debate lies a conceptual tension that will be familiar to anyone who studies mental health measurement. The Impostor Phenomenon, the persistent feeling that one’s accomplishments are undeserved and that one is a fraud despite evidence of competence, is not a diagnosis in the way that, say, major depressive disorder is. It is conceptually a continuous construct, distributed along a spectrum across the population. Critics argue that applying the epidemiological language of prevalence to such a construct is conceptually incorrect, because prevalence traditionally refers to the proportion of a population that has a categorical condition. You either have an infection or you do not; you cannot, in the same strict sense, have a fraction of impostorism.

Salari and colleagues do not dispute this premise. In their response, they explicitly agree that the Impostor Phenomenon is continuous and that referring to its prevalence in the strict epidemiological sense may be conceptually wrong. But they argue that this does not rule out threshold-based prevalence estimates, and they point to well-established precedents. Depression and anxiety are also conceptualized as continua, yet researchers routinely estimate their prevalence using instruments such as the PHQ-9 for depression and the GAD-7 for anxiety, which employ empirically derived thresholds to classify individuals above a cutoff. In those fields, decades of validation work have established which scores correspond to clinically meaningful impairment, giving the thresholds legitimacy. The question the exchange raises is whether the Impostor Phenomenon field has earned the same privilege.

The instrument in question is the Clance Impostor Phenomenon Scale, or CIPS, a questionnaire that yields a continuous score of impostor feelings. Primary studies included in the meta-analysis reported the percentage of health service providers scoring above various cutoffs on this scale, effectively equating the proportion of high scorers with prevalence. Salari’s team then pooled those reported proportions into a single estimate. Here the authors make a striking concession: they fully acknowledge that the current cutoffs for the CIPS are inconsistently applied across the existing literature and lack systematic validation. Different studies have used different thresholds, sometimes without empirical justification, meaning that two studies of identical populations could report wildly different prevalence figures depending solely on where they drew the line.

This is precisely the point Brauer and Proyer drove home with an empirical demonstration. By applying six different cutoffs to the same datasets, they showed that the resulting proportions of cases diverged dramatically. They also argued that the extremely high heterogeneity in the meta-analysis, quantified by an I-squared statistic of 98.6 percent, is evidence that pooling prevalence rates in the first place is invalid. In meta-analysis, I-squared describes the percentage of variation across studies that is due to real differences rather than chance; a value near 100 percent means the studies are barely measuring the same thing. To the critics, pooling estimates from studies that use incompatible thresholds produces a number that means little.

Salari and colleagues offer a different reading of that statistic. In their view, the high heterogeneity is itself an important empirical finding, a quantitative portrait of a literature that has operationalized its central construct inconsistently. Meta-analysis, they argue, is precisely the tool that allows researchers to measure and expose this problem, highlighting how different thresholds, samples, and instruments contribute to variance. Without such a synthesis, the field would lack any systematic appreciation of how widely prevalence estimates diverge. They also note that some of the cutoffs applied in the critics’ demonstration were arbitrary and drawn from two convenience datasets that are not population-representative, which they contend artificially inflates the apparent instability and should not be generalized to broader populations. Demonstrating variability across arbitrary thresholds, they write, does not invalidate prevalence research; it underscores the need for consensus and validation studies.

The exchange also touches on a provocative interpretive question. Brauer and Proyer suggested that if 62 to 70 percent of individuals report impostor feelings, as some estimates imply, then the experience should perhaps be regarded as normative rather than pathological. The Salari team counters that high prevalence does not imply triviality. They draw an analogy to physician burnout, which large systematic reviews have shown to be reported by a strikingly high proportion of doctors, yet which has clear and consequential implications for health care delivery. Similarly, they argue, impostor feelings, even if common, are associated with impaired well-being, reduced job satisfaction, and elevated burnout risk. Prevalence estimates, on this view, remain useful for scoping the potential consequences of a widespread experience, regardless of whether it is labeled a disorder.

Notably, the authors point out that their original review contained findings that survive the critique largely intact. Beyond the contested prevalence figures, the meta-analysis summarized correlations between impostor feelings and constructs such as self-esteem, anxiety, depression, stress, and burnout. Because those analyses used continuous scores rather than categorical cutoffs, they are conceptually reliable and less dependent on the threshold decisions that lie at the center of the dispute. This distinction matters for the field’s future direction: the relational findings, which describe how impostorism moves alongside other psychological variables, rest on firmer measurement ground than the headline prevalence numbers.

The response also ventures into a broader reflection on the nature of psychological measurement itself. The authors observe that screening tools for mental health conditions inevitably involve trade-offs, with all instruments producing false positives and false negatives, and that different thresholds can be deployed to compensate for these limitations and optimize screening performance. A screening tool, they emphasize, is not a diagnosis; it flags individuals for further clinical attention. Their stated goals in examining the Impostor Phenomenon were twofold: to draw attention to a condition that has received comparatively little scrutiny, and to provide clinicians with usable information. On that view, the very act of publishing a contested prevalence estimate, and engaging publicly with its critics, serves the scientific process by forcing the field to confront how much its conclusions depend on arbitrary measurement decisions.

Where does this leave the science of feeling like a fraud? The two sides converge more than the heated framing might suggest. Salari’s team agrees that any prevalence estimates should be presented with caution and explicit caveats until robust validation studies are conducted, and they endorse the critics’ implicit agenda: large-scale validation of CIPS thresholds against external criteria, studies using population-representative samples rather than convenience datasets, and community-wide consensus on cutoff standards, modeled on the practices that stabilized depression and anxiety research. The meta-analysis, they argue, does not endorse any single cutoff but documents the consequences of their inconsistency, functioning as a warning, while the commentary supplies the conceptual framework for understanding that warning. Both papers, read together, point toward the same next steps. For the millions of health care workers who quietly doubt their own competence, the practical takeaway is unchanged: the feelings are real, common, and linked to burnout. But for researchers hoping to put a single trustworthy number on how many people experience them, the message from this exchange is clear: the field first needs to agree on where the line should be drawn, and then prove that the line means something.

Subject of Research: Methodological debate over estimating prevalence rates of the Impostor Phenomenon using the Clance Impostor Phenomenon Scale

Article Title: Response to: Matters Arising from Salari et al. (2025): Why it is (currently) impossible to estimate accurate prevalence rates of the Impostor Phenomenon

Article References: Salari, N., Hashemian, S. H., Hosseinian-Far, A., Fallahi, A., Heidarian, P., Rasoulpoor, S., & Mohammadi, M. (2026). Response to: Matters Arising from Salari et al. (2025): Why it is (currently) impossible to estimate accurate prevalence rates of the Impostor Phenomenon. BMC Psychology, 14(1), Article 1400. https://doi.org/10.1186/s40359-026-05555-6

Image Credits: AI Generated

DOI: 10.1186/s40359-026-05555-6

Keywords: Impostor Phenomenon, prevalence, meta-analysis, Clance Impostor Phenomenon Scale, CIPS cutoffs, heterogeneity, psychological measurement, health service providers, burnout, screening thresholds, BMC Psychology, validation studies

Garston Liang of the University of Technology Sydney, together with Mackenzie Glover and Guy E. Hawkins of the University of Newcastle, compared two widely used methods of preference elicitation: ranking and best–worst scaling, often abbreviated as BWS. In a ranking task, respondents order a full list of items from most to least preferred, a format familiar from hiring panels, consumer surveys, and even Australia’s preferential voting system. In best–worst scaling, by contrast, respondents see only small subsets of items at a time and simply pick the one they favor most and the one they favor least. Across three experiments involving hundreds of participants, the team found that both methods largely agree on the ordering of preferences, but that BWS delivers markedly better test–retest reliability—meaning people’s answers stay more consistent when measured repeatedly, even over intervals as short as a few minutes.

The theoretical stakes of this comparison are considerable. One long-standing view in psychology and behavioral economics assumes that people hold stable internal representations of their preferences, which measurement merely reveals, with any variability attributable to the tool being used. A competing perspective, known as the constructed preferences view, holds that preferences are assembled on the spot, packaged with the conditions of the moment—the available options, the framing of the question, even the mood of the respondent. Under that view, elicited preferences need not be coherent over time at all. Both perspectives agree, however, that the choice of elicitation method matters enormously, which makes the question of whether ranking and BWS actually measure the same thing a critical one for anyone designing surveys.

Each method carries distinct strengths and weaknesses. Ranking is quick, intuitive, and computationally cheap, and it works well when only a handful of options are involved. But its demands grow brutally with set size: ordering a top-ten list requires the equivalent of 45 pairwise comparisons, and marketing researchers routinely confront sets of dozens of competing products. Rankings also force respondents to discriminate between options they may genuinely feel indifferent about, manufacturing distinctions where none exist. BWS sidesteps both problems. Because people are naturally good at identifying what they love and what they loathe, asking for best and worst choices within small subsets keeps cognitive load low regardless of how many items are in play. Over repeated, carefully balanced choice sets, preferences for every item accumulate incrementally, and each choice constrains the possible orderings of items not even shown in that round.

To compare the two methods fairly, the researchers needed a domain where true preferences are unlikely to shift during the experiment. They settled on life values, using a validated set of ten values drawn from Schwartz’s Value Theory—concepts such as benevolence, hedonism, power, security, and self-direction. Fundamental changes in personal values unfold over years, dwarfing the minutes required for a survey, so any inconsistency observed between two measurements could be attributed to the measurement process itself rather than to genuine change in what people believe. The team also manipulated how the values were labeled: some participants saw broad dimension labels like “benevolence,” while others saw example item labels like “helpful, honest, forgiving.” Because both label types target the same underlying concept, switching labels between measurements provided an additional stress test of each method’s stability.

The experimental design was elegant in its simplicity. In Experiment 1a, 101 participants completed the ranking task twice; in Experiment 1b, 100 participants completed the BWS task twice; and in Experiment 2, 148 participants completed both tasks in randomized order. Participants were recruited through the online platform Prolific, restricted to native English speakers with high approval ratings, and paid at a rate commensurate with £9.00 per hour. The BWS task used a balanced pairwise design in which each of the ten life values appeared exactly six times across eleven subsets of five or six options, with any pair of values co-occurring exactly three times. Agreement between measurement occasions was quantified using Kendall’s Tau, a rank-correlation statistic bounded between −1 and 1, where a value of 1 indicates perfect agreement between two ordered lists.

The headline result was unambiguous. Average individual-level agreement between repeated measurements was substantially higher for BWS than for ranking, with mean Kendall’s Tau values of 0.69 versus 0.47—a difference supported by Bayes factors exceeding 1000, indicating decisive statistical evidence. The pattern of errors in the ranking task revealed exactly why. The most unstable position in the ranked list was rank six, smack in the middle: 81 percent of rank-six assignments changed between measurements. Consistency improved monotonically toward the extremes, with rank one changing for only 54 percent of responses. BWS, by capitalizing on people’s reliable judgments at the extremes, effectively avoids forcing respondents to manufacture precision in the muddled middle—precisely where rankings are weakest.

Importantly, the two methods were not measuring different things. The aggregate preference orderings agreed almost perfectly: benevolence emerged as the most important value and power as the least important in both tasks, and the top five values were ordered identically across methods, with only a single pair of middling values trading positions. The BWS scores, however, carried extra information that rankings cannot provide. Because normalized best–worst scores range continuously from −1 to 1, they quantify the strength of preferences, not just their order. In Experiment 2, the score gaps between first and second place and between ninth and tenth place were roughly 0.40 and 0.38, dwarfing the average gap of about 0.11 between adjacent middle ranks—confirming that middle-ranked options were nearly indistinguishable in preference strength. This added resolution has real-world value: in a prior oncology study cited by the authors, BWS revealed that surgeon training was twice as important to patients as surgeon experience and more than three times as important as hospital reputation.

The label manipulation produced a subtle but instructive finding. When the labels changed between measurement occasions—say, from “benevolence” to “helpful, honest, forgiving”—agreement dropped sharply for both methods, indicating that the two label sets evoke genuinely different interpretations of the same value. Yet the direction and even the relative magnitude of those interpretive shifts were consistent across both elicitation methods: benevolence was preferred as an item label and security as a dimension label, to similar degrees, whether measured by ranking or by BWS. This convergence suggests that the reliability advantage of BWS stems from the method itself rather than from any accidental alignment between the method and the particular stimulus set used.

The practical implications extend well beyond survey methodology. BWS is not free—completion times in the study ran about 8.6 minutes for the BWS task versus 4.6 minutes for ranking, because larger item sets require more choice rounds even though each round stays cognitively light. But the authors argue the added reliability is worth the cost, particularly in applied settings such as health care priority-setting, marketing research, and personality assessment, where item sets of 16 to 57 options make full rankings unwieldy. BWS also naturally captures ties, revealing when respondents genuinely hold two options equal, something a forced ranking can never show. For researchers who have long treated the humble ranked list as the default instrument for measuring what people want, the message of this study is clear: if you care whether your measurements would come out the same way twice, asking people only for their best and their worst may be the most honest question you can ask.

Subject of Research: Comparing the test–retest reliability of best–worst scaling and ranking methods for eliciting personal preferences

Article Title: Time after time, the best–worst scaling method is more reliable than ranking

Article References: Liang, G., Glover, M., & Hawkins, G. E. (2026). Time after time, the best–worst scaling method is more reliable than ranking. Behavior Research Methods, 58(10), Article 284. https://doi.org/10.3758/s13428-026-03167-x

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03167-x

Keywords: best–worst scaling, ranking, preference elicitation, test–retest reliability, survey methods, life values, Schwartz Value Theory, Kendall's Tau, decision making, behavioral economics, psychological measurement, Behavior Research Methods

The problem, known in psychometrics as method variance or the wording effect, arises because scales are frequently built with a deliberate balance of positively and negatively keyed items. The idea sounds sensible: reversing the direction of some questions should discourage automatic agreement and force respondents to read carefully. But the strategy backfires in a measurable way. Items that share the same wording direction tend to correlate with one another for reasons that have nothing to do with the underlying construct. When researchers run a factor analysis on such data, the statistical machinery often obligingly splits the scale into two clusters, one of positively worded items and one of negatively worded items, rather than the single dimension the scale was designed to capture.

This artificial splitting, sometimes called spurious bidimensionality, has real consequences. A researcher who concludes that a questionnaire measures two distinct traits may publish misleading findings, build theories on phantom subfactors, or compute reliability estimates that are inflated or deflated for the wrong reasons. The grit scale, a wildly popular measure of perseverance and passion for long-term goals, has been a particular battleground for this debate. Since its development by Angela Duckworth and colleagues, researchers have argued endlessly about whether grit truly has two facets, perseverance of effort and consistency of interest, or whether the apparent two-factor structure is simply an artifact of the negatively worded items embedded in the short version of the scale.

Faraci and Nasonte tackled this question with random intercept item factor analysis, or RIIFA, a modeling approach originally introduced by Albert Maydeu-Olivares and Donna Coffman in 2006. The core insight of RIIFA is elegant: it adds a random intercept factor to the standard factor model, a latent variable on which every item in the scale loads regardless of its wording direction. This intercept factor absorbs the shared response tendency that cuts across all items, including acquiescence, the general willingness to agree with statements, and other response styles. Once that pervasive method variance is siphoned off into its own latent dimension, the remaining substantive factor can be interpreted as the true trait, purified of the contamination introduced by how the questions happen to be phrased.

To evaluate whether RIIFA delivers on this promise, the researchers conducted two studies using two independent samples from the United Kingdom. The first study analyzed data from 977 participants, and the second from 496. Their test case was the Short Grit Scale, known as Grit-S, an eight-item instrument that mixes positively and negatively worded questions. The choice was strategic: the Grit-S is short, widely used, and famously prone to the wording-effect problem, making it an ideal proving ground for a method intended to restore structural validity to mixed-worded scales.

The first study focused on dimensionality assessment, the task of determining how many latent factors actually underlie a set of items. The researchers compared traditional factor retention techniques, including parallel analysis, a classic method that compares observed eigenvalues against those from random data, with RIIFA-based counterparts that incorporate the random intercept factor. They also employed exploratory graph analysis, a newer network psychometrics approach that treats items as nodes in a network and uses community detection algorithms, borrowed originally from network science methods like the fast unfolding of communities, to identify clusters of tightly connected items. The results were striking. Traditional retention methods consistently overestimated the number of factors, suggesting the Grit-S contained two dimensions when the theoretical expectation was one. In contrast, the RIIFA-based techniques produced unidimensional solutions, and bootstrap analyses, which resample the data thousands of times to gauge stability, showed that these solutions were considerably more stable than those obtained without controlling for the wording effect.

The second study shifted from exploration to confirmation, using confirmatory factor analysis to formally test competing models of the scale’s structure. The researchers fit models with and without a random intercept factor and compared them on both fit and parsimony, the principle that simpler explanations should be preferred unless complexity earns its keep. The model incorporating the random intercept factor achieved the best balance between the two. Its fit statistics were excellent by conventional standards: a root mean square error of approximation of .048, with a confidence interval spanning .022 to .072, a comparative fit index of .984, a Tucker-Lewis index of .974, and a standardized root mean square residual of .027. Values close to .95 or above on the fit indices and below about .06 or .08 on the error measures are typically considered indicative of a well-fitting model, and this model cleared every bar comfortably.

Reliability told a similarly encouraging story. The RIIFA model yielded a hierarchical omega of .84, a coefficient derived from bifactor-style measurement models that estimates how well the total scale score reflects a single dominant common factor after accounting for the variance absorbed by subsidiary dimensions. In practical terms, this means that once the method variance was reallocated to the random intercept factor, the substantive grit factor explained enough common variance to support meaningful interpretation of total scores. The researchers interpret this as evidence that RIIFA does not merely hide the problem; it actively redistributes explained variance between the substantive factor and the method factor, mitigating the artificial bidimensionality that plagues conventional analyses and enhancing the interpretability of the latent structure.

The implications reach well beyond the grit scale. Mixed-worded instruments are ubiquitous across psychology, appearing in measures of self-esteem, core self-evaluations, need for cognition, perceived stress, learning burnout, and countless clinical and organizational questionnaires. For each of these, the same dilemma recurs: researchers must decide whether an emerging second factor represents a genuine substantive distinction or a methodological artifact. Historically, that judgment has been made with tools, such as standard parallel analysis and conventional confirmatory factor analysis, that have no built-in mechanism for separating content from phrasing. The findings of this study suggest that RIIFA offers a principled alternative, and the authors explicitly recommend its application in cases where wording effects threaten the validity of psychometric measurement.

The study also connects to a broader movement in quantitative psychology toward modeling response styles and careless responding rather than simply hoping they wash out in large samples. Recent work has documented how even a few inconsistent respondents can confound the structure of personality survey data, how acquiescence distorts exploratory factor analysis, and how attention checks and response-style detection methods carry their own complications. Faraci and Nasonte’s contribution fits squarely into this research program, demonstrating on real data that a model explicitly designed to partition substantive and method variance can outperform traditional approaches. Importantly, the authors have made their data openly available through the Open Science Framework, and all of the R and Mplus code used in the analyses is accessible as well, lowering the barrier for other researchers to adopt the technique. For a field that depends so heavily on self-report questionnaires, a validated method for ensuring that scores reflect traits rather than question phrasing is not a technical nicety. It is a foundation for the credibility of the measurements themselves, and this study provides compelling empirical grounds for adding random intercept item factor analysis to the standard psychometric toolkit.

Subject of Research: Using random intercept item factor analysis to separate substantive variance from wording-effect method variance in mixed-worded psychological scales

Article Title: Disentangling substantive and method variance in mixed-worded scales: An empirical application of the random intercept item factor analysis (RIIFA)

Article References: Faraci, P., & Nasonte, G. (2026). Disentangling substantive and method variance in mixed-worded scales: An empirical application of the random intercept item factor analysis (RIIFA). Behavior Research Methods, 58(11), Article 299. https://doi.org/10.3758/s13428-026-03163-1

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03163-1

Keywords: psychometrics, random intercept item factor analysis, wording effects, method variance, Short Grit Scale, factor analysis, exploratory graph analysis, parallel analysis, dimensionality, structural validity, confirmatory factor analysis, survey methodology