The core problem the authors identify is deceptively simple. Classical test theory, the statistical backbone of most reliability work in psychology, assumes that an observed score is the sum of a stable true score and random error. Four assumptions underpin the resulting reliability coefficients: true score and error must be additive, independent, and random, and the true score must not change between repeated measurements. In practice, these assumptions are routinely violated by practice and habituation effects — the well-documented tendency of participants to improve on cognitive and coordinative tasks simply by repeating them, without any intervention in between. When the retest score is systematically better than the test score, the parallel-readings assumption collapses, and with it the interpretability of the reliability coefficient.

The scale of the problem is striking. Habituation effects have been documented across a wide range of standard instruments, including the Stroop test, the Eriksen flanker task, the Simon task, the Digit Symbol Substitution Test, the Wechsler Memory Scale, and the Halstead-Reitan neuropsychological battery. The largest gains typically appear between the first two testing sessions, with progressively flatter improvement curves as participants grow familiar with the task. Long-term learning can even persist for years: performance on the Wechsler Adult Intelligence Scale and the Wechsler Memory Scale has been shown to be affected by task exposure that occurred months or years earlier. In one series of experiments cited in the paper, trial-to-trial random errors in neurocognitive tests started as high as 6 to 43 percent and only stabilized below 5 percent in reaction tasks after several days of repeated testing.

What makes this especially dangerous is that the field’s favorite reliability statistics can look excellent while hiding serious problems. The intraclass correlation coefficient (ICC) and Cronbach’s alpha summarize the ratio of true-score variance to observed variance, but they cannot distinguish between different sources of measurement error. The authors point to published examples where an ICC of 0.9 or higher — conventionally rated as excellent — coexisted with a mean absolute test–retest error of 20 percent and maximum individual errors approaching 90 percent. A high correlation between two measurements says nothing about how close those measurements actually are to each other on the original scale, which is what clinicians and experimenters usually need to know.

The classical remedy for this blind spot is the Bland–Altman analysis, introduced in 1986, which plots the difference between two measurements against their mean. The average difference reveals systematic bias, while the 95 percent limits of agreement — the mean difference plus or minus 1.96 standard deviations — describe the expected range of random scatter. But the authors argue that Bland–Altman analyses, as commonly used, suffer from a critical limitation: the limits of agreement are purely descriptive. They describe where the data fell, but they say nothing about whether that amount of error is acceptable for the scientific or clinical question at hand. Bland and Altman themselves noted in the original 1986 paper that acceptable limits should ideally be defined in advance to aid interpretation — advice that has largely been ignored in practice.

There is a second, subtler flaw. Standard limits of agreement are drawn as parallel horizontal lines on the Bland–Altman plot, which implicitly assumes that the absolute size of measurement error is constant across the whole range of measured values. In many empirical domains, however, the scatter grows as the measured value grows — a phenomenon known as heteroscedasticity. Parallel limits therefore tolerate a disproportionately large percentage error for small measurement values while being overly strict for large ones. A limit of ±1 degree might be trivial for a mean of 180 degrees but catastrophic for a mean of 0.5 degrees, as an example from visual-vertical perception research in stroke patients illustrates.

The new R application addresses both problems with a concept the authors call limits of acceptance, or LoAcc. Unlike limits of agreement, which are computed from the data after the fact, limits of acceptance are defined a priori by the researcher as a proportional threshold — for example, 5 percent of each participant’s mean test–retest value. Because the acceptable band scales with the magnitude of the measurement, the resulting boundaries fan out across the Bland–Altman plot rather than running parallel, correctly accommodating heteroscedastic data. A test–retest difference is classified as acceptable only if it falls within the proportional band for that individual. The app then color-codes the results: green if the systematic bias is non-significant and at least 95 percent of observations fall within the acceptance limits, red if either criterion fails.

The software itself is a Shiny app that automates the entire test–retest evaluation workflow. Users upload an Excel file in wide format, with a participant identifier in the first column and measurement pairs in consecutive columns; the app can also be used for validity studies against a gold standard or for inter-rater objectivity analyses. It then computes the full battery of reliability metrics — all six ICC variants with 95 percent confidence intervals, Cronbach’s alpha, the standard error of measurement, the minimal detectable change, the coefficient of variation, the mean absolute error, and the mean absolute percentage error — alongside a paired-samples t-test for systematic bias and publication-ready Bland–Altman plots. Users can set the acceptance threshold per variable pair, specify measurement units, and export results tables and figures automatically. The code is openly available via the Open Science Framework.

To demonstrate the tool, the authors simulated Stroop test data across four consecutive testing days with 100 virtual participants, using a 5 percent acceptance threshold calibrated to the error level observed after four days of habituation in their earlier empirical work. The progression is instructive. Between days 1 and 2, mean performance rose significantly and 86 of 100 observations fell within the acceptance limits — both criteria failed, and the plot lit up red. Between days 2 and 3, the systematic bias had largely vanished, but random scatter remained too high, with 88 of 100 observations inside the limits. Only between days 3 and 4 did the data satisfy the criterion: no significant bias, 97 of 100 observations within the limits, and the mean absolute error shrinking from 0.828 seconds to 0.479 seconds. Applied to a real dataset from a choice reaction task tested twice daily over five days, the app showed the same convergence, and also revealed a near-threshold case at a 10 percent limit where a single data point sitting exactly on the boundary tipped the classification — a reminder, the authors note, that such judgments should consider the magnitude of the deviation rather than being treated as purely binary.

The broader message is a call for transparency that resonates far beyond cognitive psychology. The authors argue that expected intervention-induced changes from previous studies can serve as a rational anchor for setting acceptance limits: a 10 percent measurement error might be tolerable if an intervention is expected to change the outcome by 100 percent, but it is unacceptable if the expected effect is only 10 percent. They acknowledge that no universal guideline exists for what counts as acceptable, and that the 5 and 10 percent thresholds used in their examples are illustrative rather than prescriptive. Still, by forcing researchers to state their tolerance for error in advance and by separating systematic from random error — since a valid ICC interpretation requires the absence of systematic bias — the tool turns habituation from an invisible confound into a measurable, plottable, and ultimately controllable quantity. For a field increasingly worried about the reproducibility of its measurements, knowing exactly how much habituation is enough may prove to be one of the most practical questions psychology has finally learned to answer.

Subject of Research: Test–retest reliability, habituation effects, and agreement analysis in behavioral and neurocognitive research

Article Title: How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research

Article References: Warneke, K., Siegel, S. D., Afonso, J., Herbsleb, M., & Wallot, S. (2026). How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research. Behavior Research Methods, 58(11), Article 307. https://doi.org/10.3758/s13428-026-03170-2

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03170-2

Keywords: habituation, practice effects, test–retest reliability, Bland–Altman analysis, limits of agreement, limits of acceptance, intraclass correlation coefficient, Cronbach's alpha, measurement error, R software, psychometrics, neurocognitive testing

The problem the developers set out to solve is one that any practicing metabolomics researcher will recognise. Instruments from SCIEX, Agilent, Waters, Thermo, and Bruker each produce data in proprietary formats that lock laboratories into specific software ecosystems, complicating cross-laboratory harmonisation and restricting long-term accessibility and reuse under FAIR data principles. Converting files to the community-standard open format mzML is a necessary first step, but it is only the beginning. Metabolic phenotyping experiments are also vulnerable to systematic variation arising from sample preparation, instrument performance drift, and batch effects, and even the choice of processing tool and its parameterisation introduces additional variance. Robust, standardised quality-control procedures are therefore essential, yet until now no single open tool combined vendor file conversion, automated peak integration, full auditability, and a documented quality-control layer in one scriptable environment.

MStargetR, developed by Harrison Szemray, Luke Whiley, and colleagues at Murdoch University and collaborating institutions in Australia and Colombia, takes a pragmatic architectural approach: rather than reinventing established tools, it builds on them. The package is organised into three core modules that map directly onto the sequential stages of a targeted multiple reaction monitoring (MRM) or selected reaction monitoring (SRM) experiment. The first, msConvertR, interfaces with ProteoWizard’s msConvert tool to convert proprietary vendor files into mzML format, accepting raw data from SCIEX, Thermo, Agilent, Bruker, and Waters instruments as well as files already in open formats. The second, PeakForgeR, performs retention time alignment, peak boundary optimisation, and automated integration. The third, qcCheckR, handles missing value imputation, internal standard normalisation, concentration calculation, signal drift and batch correction, filtering, and reporting.

The technical heart of the pipeline lies in how PeakForgeR handles peak integration. For each analytical batch, a lightweight algorithm identifies the peak apex and boundaries for each transition window, using the median apex retention time localisation across quality-control samples to define a plate-wide reference. These pre-defined boundaries are then pushed to the Skyline command-line interface for peak area integration, ensuring that consistent integration windows are applied across all samples rather than executing independent per-sample peak detection that could introduce variable integrals. Crucially, this approach produces a fully populated .sky document for every analytical plate. That document preserves every chromatogram, retention time annotation, and peak boundary used to generate the result table, and it can be reopened directly in the free Skyline graphical interface, where analysts can visually inspect individual peaks, manually adjust integration boundaries where automated detection falls short, and re-export corrected results that qcCheckR reimports without rerunning any upstream steps.

This handoff between automation and human oversight is arguably the package’s most distinctive design choice. Automated peak picking becomes a transparent first pass that the analyst remains in control of, while the .sky document itself becomes a shareable, auditable record of every integration decision behind a reported concentration matrix. It is a deliberate attempt to bridge the gap between proprietary vendor platforms, which offer rich visual inspection but are platform-locked and licence-dependent, and open-source R packages, which offer scriptability but have traditionally left the audit trail behind. The developers also provide a standalone batch correction module, batchCorrectR, that exposes the signal drift and batch correction routines for use on any tabular metabolomics dataset, including data processed through other software entirely.

On the quality-control front, qcCheckR offers three well-established correction methods that laboratories can select according to their own practices: quality-control-based random forest signal correction (QC-RFSC), which captures non-linear drift that linear models cannot; quality-control-based robust LOESS signal correction (QC-RLSC); and ComBat, an empirical Bayes method suitable for datasets that lack QC samples altogether. Missing values below a user-specified instrumental limit of detection are handled by imputing half the minimum observed value for each feature, a strategy designed to maintain data integrity. The module generates comprehensive outputs in both HTML and Excel formats, with the HTML report including summary tables of key quality metrics and interactive visualisations such as principal components analysis plots and control charts that allow dynamic exploration of data quality at the level of individual metabolites.

Reproducibility is engineered into the package at the infrastructure level. All third-party tools, including msConvert and Skyline, are bundled within version-pinned Docker images, so that the exact software environment travels with the analysis and eliminates complex dependency management. For high-performance computing environments that prohibit Docker, MStargetR can be configured to use Apptainer instead. The pipeline runs natively on AMD64 architecture across Windows and Linux, with macOS and ARM64 users currently needing a virtual machine, a limitation the team is actively working to address. For analysts who prefer not to write code at all, a Shiny-based graphical user interface provides code-free access to the full pipeline, and a Windows installer delivers the application directly from the Start menu, requiring only a container runtime alongside it.

The performance evaluation is striking. Applied to a published targeted lipidomics dataset of 128 human plasma samples targeting 1,161 lipid species with 73 stable isotope-labelled internal standards, MStargetR processed all samples end-to-end, recovering all 1,161 targeted lipid features. Of these, 949 features (81.7 percent) were detected and returned a relative standard deviation below 30 percent across replicated long-term reference quality-control samples, and 823 features met the more stringent RSD threshold of 10 percent, a figure the authors emphasise reflects combined effects of biological absence, method sensitivity, and deliberately stringent quality criteria rather than detection failure. Because per-transition precision metrics and the corresponding chromatograms are both retained, the reason any individual target falls below threshold can be inspected directly rather than inferred.

Scalability testing on a standard workstation showed that total runtime grew linearly from 14.9 minutes at 84 samples to 302.5 minutes at 4,200 samples, with per-sample processing time falling to just 4.1 seconds at larger batch sizes as fixed overhead was distributed across more injections. CPU utilisation plateaued around 62 to 67 percent once enough plates were available to keep the parallel workers supplied, and memory consumption was dominated by per-plate parallelism overhead rather than per-sample data accumulation. Predictable, linear scaling means laboratories can reliably estimate processing time for large cohort studies from the reported per-sample rate, a practical consideration as metabolomics studies grow to thousands of specimens.

The developers are careful to position MStargetR’s contribution as integration rather than algorithmic superiority. Quantification currently relies on single-point internal standard calculation rather than multi-point calibration curves, and dedicated MRM processors remain more complete in that respect, while the browser-based MRMPro remains stronger for rapid multi-user manual curation of very large cohorts. But no existing open tool, the authors argue, combines vendor conversion, automated integration, a re-importable audit artefact, and a documented quality-control and batch correction layer within one version-pinned, R-native environment. Future work will expand supported input formats, add multi-point calibration for absolute quantification across wider dynamic ranges, and tighten integration with public repository deposition. For a field whose credibility increasingly depends on whether results can be independently reproduced, MStargetR represents a meaningful step toward making rigorous, auditable targeted metabolomics the default rather than the exception.

Subject of Research: A reproducible, containerised R workflow for end-to-end targeted MRM/SRM mass spectrometry data processing

Article Title: MStargetR: a reproducible, containerised workflow for end-to-end targeted (MRM/SRM) mass spectrometry data processing in R

Article References: Szemray, H., Nambiar, V., Hicks, D., Lodge, S., Wist, J., Lawler, N. G., & Whiley, L. (2026). MStargetR: a reproducible, containerised workflow for end-to-end targeted (MRM/SRM) mass spectrometry data processing in R. Metabolomics, 22(5), Article 160. https://doi.org/10.1007/s11306-026-02541-2

Image Credits: AI Generated

DOI: 10.1007/s11306-026-02541-2

Keywords: metabolomics, mass spectrometry, MRM, SRM, lipidomics, reproducibility, R software, quality control, batch correction, FAIR data, open-source software, Docker