The core problem the authors identify is deceptively simple. Classical test theory, the statistical backbone of most reliability work in psychology, assumes that an observed score is the sum of a stable true score and random error. Four assumptions underpin the resulting reliability coefficients: true score and error must be additive, independent, and random, and the true score must not change between repeated measurements. In practice, these assumptions are routinely violated by practice and habituation effects — the well-documented tendency of participants to improve on cognitive and coordinative tasks simply by repeating them, without any intervention in between. When the retest score is systematically better than the test score, the parallel-readings assumption collapses, and with it the interpretability of the reliability coefficient.

The scale of the problem is striking. Habituation effects have been documented across a wide range of standard instruments, including the Stroop test, the Eriksen flanker task, the Simon task, the Digit Symbol Substitution Test, the Wechsler Memory Scale, and the Halstead-Reitan neuropsychological battery. The largest gains typically appear between the first two testing sessions, with progressively flatter improvement curves as participants grow familiar with the task. Long-term learning can even persist for years: performance on the Wechsler Adult Intelligence Scale and the Wechsler Memory Scale has been shown to be affected by task exposure that occurred months or years earlier. In one series of experiments cited in the paper, trial-to-trial random errors in neurocognitive tests started as high as 6 to 43 percent and only stabilized below 5 percent in reaction tasks after several days of repeated testing.

What makes this especially dangerous is that the field’s favorite reliability statistics can look excellent while hiding serious problems. The intraclass correlation coefficient (ICC) and Cronbach’s alpha summarize the ratio of true-score variance to observed variance, but they cannot distinguish between different sources of measurement error. The authors point to published examples where an ICC of 0.9 or higher — conventionally rated as excellent — coexisted with a mean absolute test–retest error of 20 percent and maximum individual errors approaching 90 percent. A high correlation between two measurements says nothing about how close those measurements actually are to each other on the original scale, which is what clinicians and experimenters usually need to know.

The classical remedy for this blind spot is the Bland–Altman analysis, introduced in 1986, which plots the difference between two measurements against their mean. The average difference reveals systematic bias, while the 95 percent limits of agreement — the mean difference plus or minus 1.96 standard deviations — describe the expected range of random scatter. But the authors argue that Bland–Altman analyses, as commonly used, suffer from a critical limitation: the limits of agreement are purely descriptive. They describe where the data fell, but they say nothing about whether that amount of error is acceptable for the scientific or clinical question at hand. Bland and Altman themselves noted in the original 1986 paper that acceptable limits should ideally be defined in advance to aid interpretation — advice that has largely been ignored in practice.

There is a second, subtler flaw. Standard limits of agreement are drawn as parallel horizontal lines on the Bland–Altman plot, which implicitly assumes that the absolute size of measurement error is constant across the whole range of measured values. In many empirical domains, however, the scatter grows as the measured value grows — a phenomenon known as heteroscedasticity. Parallel limits therefore tolerate a disproportionately large percentage error for small measurement values while being overly strict for large ones. A limit of ±1 degree might be trivial for a mean of 180 degrees but catastrophic for a mean of 0.5 degrees, as an example from visual-vertical perception research in stroke patients illustrates.

The new R application addresses both problems with a concept the authors call limits of acceptance, or LoAcc. Unlike limits of agreement, which are computed from the data after the fact, limits of acceptance are defined a priori by the researcher as a proportional threshold — for example, 5 percent of each participant’s mean test–retest value. Because the acceptable band scales with the magnitude of the measurement, the resulting boundaries fan out across the Bland–Altman plot rather than running parallel, correctly accommodating heteroscedastic data. A test–retest difference is classified as acceptable only if it falls within the proportional band for that individual. The app then color-codes the results: green if the systematic bias is non-significant and at least 95 percent of observations fall within the acceptance limits, red if either criterion fails.

The software itself is a Shiny app that automates the entire test–retest evaluation workflow. Users upload an Excel file in wide format, with a participant identifier in the first column and measurement pairs in consecutive columns; the app can also be used for validity studies against a gold standard or for inter-rater objectivity analyses. It then computes the full battery of reliability metrics — all six ICC variants with 95 percent confidence intervals, Cronbach’s alpha, the standard error of measurement, the minimal detectable change, the coefficient of variation, the mean absolute error, and the mean absolute percentage error — alongside a paired-samples t-test for systematic bias and publication-ready Bland–Altman plots. Users can set the acceptance threshold per variable pair, specify measurement units, and export results tables and figures automatically. The code is openly available via the Open Science Framework.

To demonstrate the tool, the authors simulated Stroop test data across four consecutive testing days with 100 virtual participants, using a 5 percent acceptance threshold calibrated to the error level observed after four days of habituation in their earlier empirical work. The progression is instructive. Between days 1 and 2, mean performance rose significantly and 86 of 100 observations fell within the acceptance limits — both criteria failed, and the plot lit up red. Between days 2 and 3, the systematic bias had largely vanished, but random scatter remained too high, with 88 of 100 observations inside the limits. Only between days 3 and 4 did the data satisfy the criterion: no significant bias, 97 of 100 observations within the limits, and the mean absolute error shrinking from 0.828 seconds to 0.479 seconds. Applied to a real dataset from a choice reaction task tested twice daily over five days, the app showed the same convergence, and also revealed a near-threshold case at a 10 percent limit where a single data point sitting exactly on the boundary tipped the classification — a reminder, the authors note, that such judgments should consider the magnitude of the deviation rather than being treated as purely binary.

The broader message is a call for transparency that resonates far beyond cognitive psychology. The authors argue that expected intervention-induced changes from previous studies can serve as a rational anchor for setting acceptance limits: a 10 percent measurement error might be tolerable if an intervention is expected to change the outcome by 100 percent, but it is unacceptable if the expected effect is only 10 percent. They acknowledge that no universal guideline exists for what counts as acceptable, and that the 5 and 10 percent thresholds used in their examples are illustrative rather than prescriptive. Still, by forcing researchers to state their tolerance for error in advance and by separating systematic from random error — since a valid ICC interpretation requires the absence of systematic bias — the tool turns habituation from an invisible confound into a measurable, plottable, and ultimately controllable quantity. For a field increasingly worried about the reproducibility of its measurements, knowing exactly how much habituation is enough may prove to be one of the most practical questions psychology has finally learned to answer.

Subject of Research: Test–retest reliability, habituation effects, and agreement analysis in behavioral and neurocognitive research

Article Title: How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research

Article References: Warneke, K., Siegel, S. D., Afonso, J., Herbsleb, M., & Wallot, S. (2026). How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research. Behavior Research Methods, 58(11), Article 307. https://doi.org/10.3758/s13428-026-03170-2

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03170-2

Keywords: habituation, practice effects, test–retest reliability, Bland–Altman analysis, limits of agreement, limits of acceptance, intraclass correlation coefficient, Cronbach's alpha, measurement error, R software, psychometrics, neurocognitive testing