Accounting for response styles in rating data using IRT and IRTree models: A tutorial in R with applications to self-reported metacognitive competence

Every self-report questionnaire in psychology carries a hidden passenger. When respondents rate statements about their personality, attitudes, or abilities on a Likert scale, their answers reflect not only the construct being measured but also idiosyncratic habits in how they use the response format itself. Some people gravitate toward the midpoint of the scale regardless of content, a tendency known as midscale response style, while others cluster at the extremes, a pattern called extreme response style. These response styles are not random noise; they are stable individual differences that can bias test scores, inflate statistical uncertainty, distort correlations between measures, and corrupt group comparisons. A new open-access tutorial in Behavior Research Methods by Thorsten Meiser of the University of Mannheim, Esther Ulitzsch of the University of Oslo, and Mirka Henninger of the University of Basel now offers researchers a practical, step-by-step guide to disentangling these response habits from the substantive traits they contaminate, using nothing more exotic than standard R software.

The tutorial, published as part of the journal’s Tutorial Special Collection, addresses a problem that is far more pervasive than most applied researchers realize. Self-report measures dominate cognitive and social psychology, personality research, and educational assessment, and they overwhelmingly rely on Likert-type rating formats in which individuals indicate the degree to which a statement applies to them. But rating responses are never pure reflections of the underlying construct. Alongside response styles, the authors note, answers can be shaped by social desirability and careless responding. What makes response styles particularly insidious, however, is that they are intricately bound up with the rating scale format itself. Unlike faking, which varies with high-stakes versus low-stakes contexts, or careless responding, which tracks cognitive burden, midscale and extreme responding are properties of how individuals interpret and use response categories, irrespective of what the items ask.

The statistical machinery at the heart of the tutorial is item response theory, or IRT, a framework that specifies the relationship between presumed underlying response processes and observed item answers in a theory-driven way. The authors begin with the Partial Credit Model, a prominent psychometric model for ordinal responses, and its generalized extension, the generalized Partial Credit Model, which adds item-specific discrimination parameters. In these divide-by-total models, the probability of each response category is a function of a person’s trait value and a vector of item-specific threshold parameters. The thresholds mark the points on the latent continuum where adjacent response categories are equally likely, and their spread reveals how respondents interpret the response format: the interval between two adjacent thresholds defines the range of trait values for which a given category is the most probable answer. Category characteristic curves, which plot the probability of each response category as a function of the trait, make this geometry visible, with the extreme categories showing monotonically increasing or decreasing probabilities and the non-extreme categories showing single-peaked, non-monotonic curves.

Within this framework, response styles can be conceptualized as systematic increases or decreases in the probabilities of specific response categories above what the trait alone would predict. A midscale response style enlarges the probabilities of the middle categories, shifting the flanking thresholds outward so that the midscale categories attain modal probabilities over larger sections of the trait continuum. An extreme response style does the reverse, heightening the probabilities of the two extreme categories and pulling the outer thresholds inward. The tutorial then shows how extended IRT models accommodate these tendencies by introducing additional latent person dimensions, weighted by a priori determined scoring weights for each response category. A joint midscale dimension with scoring weight vector (0, 0, 1, 1, 0, 0) captures individual differences in the tendency to favor categories two and three on a six-point scale, while a joint extreme dimension with scoring weight vector (1, 0, 0, 0, 0, 1) captures the tendency to favor categories zero and five. Because changes in category probabilities and shifts in threshold locations are formally intertwined, the authors explain, specifications in terms of additional dimensions or random effects on thresholds can be considered formally equivalent techniques within an integrative framework.

To make the workflow concrete, the authors illustrate the models with an empirical data set on self-reported metacognitive competence drawn from research by Double. The data comprise eight items with a six-point rating format, divided into two subscales of four items each: one labeled online monitoring, with items such as I deliberately assess my performance on all tasks, and one labeled metacognitive awareness, with items such as I’m realistic about my capabilities. After merging responses from two studies recruited from the same population and applying identical attention screening criteria, the effective sample size was 477 respondents. A first screening step recommended in the tutorial is to inspect item response frequencies for skewness and zero-category frequencies. For the metacognitive items, responses skewed toward the higher categories, reflecting that respondents ascribe high levels of metacognitive competence to themselves and that the items are relatively easy in a technical sense. One item showed a zero frequency for the lowest category, a data peculiarity that must be handled explicitly when specifying scoring weight vectors and interpreting threshold parameters.

The authors first estimated baseline models without response styles, comparing the Partial Credit Model with the generalized Partial Credit Model through a likelihood-ratio test. The significant result indicated that the more restrictive model assuming equal discrimination across items fit the empirical responses more poorly, so the generalized version was selected as the baseline. The estimated threshold parameters were monotonically ordered for all but the last item, in line with the interpretation of category characteristic curves as yielding sections of modal category probabilities. The missing threshold for the item with the zero frequency was handled by estimating only four thresholds for that item, since the lowest observed category is equated with the second category in this situation. The tutorial walks through these practical challenges, including parameter fixations, equality constraints, and the handling of zero frequencies and missing responses, with concise R code using the mirt package.

The extended model with response style dimensions then demonstrated a superior fit over the baseline, confirming that individual differences in midscale and extreme responding affect the observed response behavior over and above the two substantive traits. The estimated factor loadings showed that extreme response style has a stronger effect than midscale response style, replicating previous findings. Of particular interest was the estimated variance-covariance matrix of the four latent dimensions: a strong positive correlation between the two metacognitive traits, a strong negative correlation between the midscale and extreme response style dimensions, but noticeably weaker relations between the substantive traits and the response styles. This pattern supports both the convergent validity of the trait dimensions and the discriminant validity between traits and response styles, indicating that the model successfully separates the two kinds of individual differences. The strong but far-from-perfect correlation between the two response style factors further shows that midscale and extreme responding represent distinct tendencies rather than a single bipolar dimension.

A major payoff of the extended models is the adjustment of person estimates on the substantive traits for potential biases due to confounding response styles. For respondents with a high level of midscale responding, trait values are biased toward the center of the distribution when the response style is not controlled for, so low trait values are overestimated and large trait values are underestimated. The tutorial shows that the extended model counteracts these biases by amplifying trait estimates in either direction for high levels of midscale responding and by shrinking estimates toward the mean for low levels. Given the skewed response frequencies toward the higher categories, the adjustment was especially evident for the underestimation of positive trait values in combination with high midscale responding. For extreme responding, the pattern is reversed: high levels of extremity bias lead to underestimation of lower trait values and overestimation of larger ones, and the extended model shrinks estimates toward the mean for such respondents while amplifying them for those with anti-extreme tendencies. These model-implied adjustment effects, illustrated graphically in the tutorial, demonstrate concretely how accounting for response styles changes the numbers researchers report.

The second approach presented in the tutorial rests on a different psychometric tradition: multiprocess item response tree models, or IRTree models. These models decompose observed rating responses into a series of logically contingent judgments, or nodes, that jointly lead to the manifest outcome. For a six-point rating scale, the tree distinguishes five decision nodes: a first node separating general disagreement from agreement with the item content, two subsequent nodes representing the choice of midscale versus non-midscale categories conditional on disagreement or agreement, and two final nodes separating extreme from non-extreme responses conditional on non-midscale decisions. Each node is parameterized through its own IRT model, and the observed responses are transformed into binary pseudo-items that can be estimated as a multidimensional IRT model in standard software. Crucially, the tutorial departs from most earlier IRTree applications, which were limited to unidimensional node models in which midscale and extreme responses are driven solely by response styles. Such unidimensional parameterizations, the authors warn, are prone to conflating trait effects and response style effects, particularly when items are overall too easy or too difficult for the sample. The multidimensional node models adopted here instead allow traits and response styles to jointly drive the midscale and extreme decisions, disentangling the two kinds of influence.

The IRTree analysis of the metacognitive data replicated the findings from the extended IRT approach with striking fidelity. The estimated discrimination parameters for the substantive traits decreased from the agreement node to the midscale and extremity nodes, supporting the notion that the traits have a stronger influence on the overall agreement decision than on the fine-grained choices among gradual response options. The latent correlations again showed a pronounced positive relation between the two metacognitive traits, a strong negative relation between the two response style dimensions, and only minor relations between traits and response styles. A direct comparison of the model-based estimates revealed correlations of .997 for online monitoring and .993 for metacognitive awareness between the trait estimates from the two approaches, and correlations of .958 for midscale and .990 for extreme responding between the response style estimates. Because psychometric modeling primarily aims to measure substantive traits while response style factors statistically control potential biases, the nearly perfect correlations suggest that the two approaches are largely interchangeable for the given data. The authors caution, however, that the two model classes differ conceptually: in extended IRT models, all latent dimensions enter the model equation for every response category through the common denominator, whereas in IRTree models the dimensions are selectively assigned to individual nodes. Model choice therefore entails assuming that the model captures the true data-generating process, and misspecification can bias parameters and lead to under- or overadjustment of trait estimates.

The tutorial closes with a discussion of recent developments that push the flexibility of these models further. Simulation studies have shown that IRTree models with multidimensional node parameterizations are more robust to mismatches between item difficulty and the person sample than their unidimensional counterparts, and the metacognitive application confirmed low correlations between traits and response styles despite skewed response frequencies, indicating successful separation of response processes. Newer extensions allow systematic variability of trait and response style effects as a function of item position, releasing the homogeneity assumption that these effects influence responses to the same degree for all items, and mixture distributions and score-based partitioning methods relax the homogeneity assumption across respondents, with empirical analyses corroborating the existence of subpopulations with different weightings of trait and response style dimensions. The models can also be generalized to non-symmetrical response styles such as acquiescence, the tendency to agree irrespective of content, provided that reversed items are available in the item set. For applied researchers, the practical guidance is pragmatic: if the primary goal is to adjust trait estimates for potential biases, extended IRT models may be the natural choice because they do not require the definition of pseudo-items and can be implemented with concise R code; if the main objective is a detailed analysis of the latent processes driving categorical responses, IRTree models may be preferred because they allow researchers to explicitly specify and test assumptions about contingent response processes. Either way, the authors conclude, choosing a model that accommodates response styles is generally better than not accommodating them at all when individual differences in response styles are present in the data, a situation that their own application to self-reported metacognitive competence suggests is far from rare.

Subject of Research: Accounting for response styles in rating data using IRT and IRTree models: A tutorial in R with applications to self-reported metacognitive competence

Article Title: Accounting for response styles in rating data using IRT and IRTree models: A tutorial in R with applications to self-reported metacognitive competence

Article References: Accounting for response styles in rating data using IRT and IRTree models: A tutorial in R with applications to self-reported metacognitive competence. (n.d.). https://doi.org/10.3758/s13428-026-03181-z

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03181-z

Keywords: Accounting, response, styles, rating, data, IRTree, models, tutorial, applications, self-reported, metacognitive, competence