The stakes are higher than many researchers realize. Previous work cited in the study shows that a careless responding rate of roughly 14 percent inflated correlations in published psychological studies by about .07, while simulations have demonstrated that even 5 percent careless responding can produce sign reversals in estimated correlations. Other research has found that removing as little as 4 to 10 percent of careless responders improved model fit in empirical datasets, and that just 10 percent contamination can substantially bias estimates of item discrimination parameters. Because researchers never know in advance how contaminated their data are, the authors argue, evaluating results with and without adjustments for presumed carelessness is essential for trustworthy science.

Traditionally, researchers have hunted for careless responders using screening indices: the long-string index that counts how many identical answers appear in a row, the intraindividual response variability index, the even–odd consistency index that correlates halves of a scale, Mahalanobis distance for multivariate outliers, and attention check items planted in the questionnaire. The trouble is that each index catches only one flavor of carelessness, so researchers typically stack several of them into a multihurdle approach. Worse, every hurdle requires a threshold—a cutoff above which a respondent is declared careless—and there are no firmly grounded rules for setting those thresholds. Small changes in cutoff choices have repeatedly been shown to produce substantially different classification rates, handing researchers enormous and often invisible degrees of freedom.

Mixture models offer an alternative that sidesteps the threshold problem entirely. The idea is elegant: assume that the sample contains two latent groups, one of attentive respondents whose answers reflect the traits being measured, and one of careless respondents whose answers do not. A standard measurement model—in this study, a multidimensional graded response model appropriate for categorical Likert items—links responses to latent traits in the attentive class. In the careless class, the model assumes responses do not depend on item content at all: threshold parameters are constrained to be equal across items, and a single latent variable representing each respondent’s preference for certain response categories replaces the substantive traits. The model then estimates, for every person, a posterior probability of belonging to the attentive class, conveying not just a verdict but a degree of certainty. A posterior probability of .60 signals that the response pattern says little either way; a .90 signals strong evidence of attentiveness.

The exemplar model used in the study was deliberately designed to capture the core features shared by most existing mixture approaches for C/IER while avoiding their more restrictive assumptions. Unlike many predecessors that treat item responses as continuous, it acknowledges their categorical nature, allowing the shape of response distributions to differ between classes—for instance, roughly normal for attentive respondents versus uniform for random responders. It also accommodates dependencies among careless responses arising from individual differences in category preferences. Because the specification retains the two-class structure common to the wider family of mixture models, the authors expect their findings to generalize to closely related approaches.

The heart of the study is a simulation comprising 240 crossed conditions, each replicated 200 times, with samples of 500 respondents and 10 percent careless responding—a realistic baseline given reported rates of roughly 5 to 12 percent in offline studies and 6 to 10 percent in large-scale assessments, but also an extreme class imbalance that makes detection genuinely hard. The researchers varied the number of five-point Likert items (5, 10, or 20), the proportion of negatively worded items (0 or 40 percent), the heterogeneity of item thresholds, the strength of item discriminations, and the type of careless behavior: straightlining, respondent-specific category preference, random responding, or a blend of the last two. Three criteria were evaluated: whether model comparisons correctly detected the presence of C/IER using the Bayesian information criterion, whether the estimated proportion of careless respondents was accurate, and whether individuals were correctly classified.

The results reveal a clear recipe for success. When the data contained no careless responders at all, the simpler model was always correctly preferred—no false alarms. When 10 percent careless responding was present, straightlining was detected consistently under every condition tested. For the subtler patterns, however, questionnaire design mattered enormously. With only five items, reliable detection generally required scales mixing positively and negatively worded items together with high item heterogeneity, meaning items differ substantially in how easy they are to endorse. With ten items, either mixed wording or high heterogeneity sufficed. With twenty items, the mixture model was preferred in nearly every condition regardless of wording or heterogeneity. The logic is intuitive: on a short, uniformly worded scale of easy-to-endorse items, an attentive respondent who consistently picks the category that best describes them produces a pattern nearly indistinguishable from a straightliner. Paradoxically, stronger item discriminations can make this problem worse, sharpening the tendency of attentive respondents to answer uniformly, while under random responding the effect reverses and weak discriminations become the hazard.

Individual-level classification told a similar story. Specificity—the probability of correctly assigning simulated attentive respondents to the attentive class—exceeded .90 in every single condition, partly because with only 10 percent contamination, a classifier defaulting to attentive already achieves 90 percent specificity. Sensitivity, the ability to catch true careless responders, depended heavily on scale length, item heterogeneity, and wording, rising with all three for most careless response types. For estimating the overall proportion of careless respondents, mixed-worded scales with high heterogeneity kept bias within acceptable bounds even at five items, while unidirectional, low-heterogeneity, short scales inflated estimates toward 14 percent. Straightlining proved uniquely stubborn: the proportion of careless respondents was systematically underestimated, dropping as low as 6 percent, and longer scales actually made the underestimation worse rather than better.

To show that the latent classes really mean what researchers hope they mean, the team reanalyzed a publicly available Big Five dataset collected across four online crowdsourcing platforms—MTurk, CloudResearch, Prolific, and Qualtrics—with roughly 500 respondents each, using the 50-item NEO-PI-R inventory in which each trait is measured by ten items, half negatively worded. By the simulation’s own criteria, these scales offered favorable conditions for detection. The model classified between 76.5 and 83.3 percent of respondents as attentive depending on the scale, and classifications were strikingly consistent: 62.4 percent of respondents were classified as attentive on all five scales, while only 8.4 percent failed to reach the attentive class on any scale, with posterior probabilities across scales correlating between .509 and .666. This cross-scale stability suggests the model captures a stable disposition toward careless responding rather than scale-specific quirks.

Crucially, the model-based classifications converged with evidence that does not rely on response patterns at all. Respondents who passed more of the five embedded attention check items received higher posterior probabilities of being attentive, with virtually all respondents passing four or five checks classified as attentive with near-certainty across all scales. Even more striking, the latent class variable tracked the data-collection platform more strongly than the attention check index did: averaged across scales, Nagelkerke’s pseudo-R-squared was 0.224 for the model-based classification versus 0.071 for the index. Both methods identified MTurk as by far the worst platform, but the model painted a starker picture—only 47 to 59 percent of MTurk respondents were classified as attentive on some scales, compared with 85.6 percent flagged by attention checks, while Prolific fared best at 88 to 95 percent attentive. The authors conclude that with scales of at least ten items, mixed wording, and sufficient item heterogeneity, mixture modeling is a viable, comparatively easy-to-implement tool for addressing careless responding—available in standard software such as the R package mirt—while cautioning that short, uniformly worded, low-heterogeneity scales remain its blind spot, and that questions about trait-score accuracy and misclassification consequences await future research.

Subject of Research: Detection of careless and insufficient effort responding in self-report questionnaires using latent variable mixture models

Article Title: Understanding when mixture models effectively identify careless responding: Simulation and validity evidence

Article References: Uglanova, I., Nagy, G., & Ulitzsch, E. (2026). Understanding when mixture models effectively identify careless responding: Simulation and validity evidence. Behavior Research Methods, 58(11), Article 313. https://doi.org/10.3758/s13428-026-03165-z

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03165-z

Keywords: careless responding, mixture models, item response theory, survey methodology, psychometrics, latent class analysis, attention checks, straightlining, Big Five, data quality, online platforms, simulation study

The study, published in the journal Behavior Research Methods, was led by Milla Pihlajamäki together with Ginette Lafit, Olivia J. Kirtley, Inez Myin-Germeys, and Gudrun Eisele. The team recruited 192 students and randomly assigned 64 of them to each of three incentive conditions. The first group received a fixed payment for taking part, the standard arrangement in most experience sampling research. The second group received the same fixed payment plus personalized feedback, a summary of the data they had contributed, on the theory that seeing one’s own psychological patterns might motivate more careful responding. The third group was paid per beep, earning money incrementally with every completed prompt, a structure that mirrors the piece-rate logic of behavioral economics rather than a flat salary model.

The protocol itself was demanding by design. For fourteen consecutive days, participants received nine prompts — or beeps — per day on their smartphones, producing up to 126 measurement moments per person and roughly 24,000 potential data points across the sample. This intensity is precisely what makes experience sampling so scientifically valuable and so operationally fragile. Unlike a one-off questionnaire, an intensive longitudinal design asks people to interrupt whatever they are doing — in a lecture, at dinner, with friends — to report on their inner states. Compliance, in this context, is not a given; it is an achievement that must be engineered through careful protocol design, and incentives are one of the most powerful levers available.

What makes the study methodologically notable is that the researchers did not stop at counting completed surveys. They systematically distinguished between data quantity and data quality, treating the two as separable outcomes that incentives might influence in different directions. Quantity was straightforward: compliance rates, or the proportion of beeps answered, and how that proportion changed over the two-week study period. Quality required more nuance. The team screened for careless responding — the phenomenon where participants click through items without genuine attention — and examined whether the temporal dynamics of such responses shifted across the study, alongside retrospective measures of participant burden and experience.

Careless responding matters because its consequences are surprisingly severe. As the methodological literature has repeatedly shown, even a modest proportion of inattentive respondents can distort correlations, inflate or deflate reliability estimates, and lead analysts astray in ways that are hard to detect after the fact. In experience sampling data, the problem is compounded by the sheer number of measurement moments and the fatigue that accumulates as the days wear on. A participant might respond attentively on day one and slide into patterned, mechanical answers by day ten. The Leuven team, drawing on their own prior work on the temporal dynamics of careless responding, built this dimension directly into the analysis, using analyses of variance and multilevel linear and logistic regression models to test whether incentive structure affected not only average behavior but its trajectory over time.

The headline finding is refreshingly clean: payment per beep increased compliance rates compared with the other two conditions, and that was essentially where the differences ended. The incremental payment scheme did not make participants answer more carelessly, did not change their reported burden, and did not alter their retrospective experience of the study in measurable ways. Personalized feedback, meanwhile — despite its intuitive appeal and growing popularity in mobile health and self-tracking applications — produced no measurable advantage over a plain fixed payment on any of the outcomes examined. The feedback condition neither boosted compliance nor improved the quality of responses, a null result that carries real practical weight for researchers weighing the added complexity of generating individualized reports against any assumed motivational payoff.

The finding that pay-per-beep worked without degrading quality is worth unpacking, because incentives in psychology have a complicated reputation. Classic motivation research has documented cases where external rewards crowd out intrinsic motivation or encourage gaming of the reward structure. In an experience sampling context, one might have worried that paying per beep would incentivize rapid, low-effort completion — quantity at the expense of quality. Recent related work on game-based rewards in experience sampling found exactly that pattern: gamification increased data quantity but reduced data quality. The Leuven results suggest that a straightforward monetary increment per completed beep does not carry the same risk, at least not in a motivated student population completing a two-week protocol.

The authors’ recommendation follows directly from the evidence: for motivated samples such as students, use payment per beep to boost data quantity. This is not a trivial prescription. Compliance in intensive longitudinal research varies enormously across populations and study designs, with meta-analyses reporting widely different completion rates depending on the sample, the burden of the protocol, and the population under study. Clinical populations, adolescents, and people experiencing acute distress often show lower compliance, and every percentage point of missing data constrains the statistical models that researchers use to extract meaning from these datasets. Multilevel time-series analyses, network models of psychological dynamics, and idiographic prediction approaches all benefit directly from denser, more complete measurement streams.

There are, of course, boundaries to the conclusion. The study was conducted in a student sample, a population that is generally well-motivated, digitally fluent, and accustomed to participating in research. Whether payment per beep would deliver the same benefit — and remain free of quality costs — in harder-to-reach populations, longer protocols, or studies measuring stigmatized behaviors remains an open question. The reward magnitudes involved were also specific to this context; the psychology of incremental incentives plausibly depends on the size of each increment relative to the total payment. And burden, while measured, was assessed retrospectively and through momentary reports within a demanding 14-day design; incentive effects on the lived experience of participation might differ in studies with different rhythms and demands.

Still, the study exemplifies a broader and welcome shift in psychological methods research: treating study design choices — payment schemes, feedback provision, sampling frequency, questionnaire length — as empirical questions rather than conventions inherited from previous studies. The team post-registered their study, documented every deviation transparently, and made all materials and analysis code openly available on the Open Science Framework, with data accessible through a secure checkout system. In a field increasingly aware that methodological decisions ripple through every downstream finding, knowing that a simple per-beep payment can fill in more of the data grid without compromising what fills those cells is exactly the kind of actionable, evidence-based guidance that turns methodological debate into better science.

Subject of Research: Effects of incentive type on data quantity and quality in experience sampling research

Article Title: The effect of incentive type on data quality and quantity in an experience sampling study in a student population

Article References: Pihlajamäki, M., Lafit, G., Kirtley, O. J., Myin-Germeys, I., & Eisele, G. (2026). The effect of incentive type on data quality and quantity in an experience sampling study in a student population. Behavior Research Methods, 58(11), Article 300. https://doi.org/10.3758/s13428-026-03164-0

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03164-0

Keywords: experience sampling, ecological momentary assessment, incentives, compliance, data quality, careless responding, survey methodology, participant payment, personalized feedback, intensive longitudinal data, burden, psychological methods