Hierarchical estimation untangles individual and group correlations in cognitive models

Cognitive models that translate raw reaction times and choices into psychologically meaningful quantities have become central tools for studying individual differences in psychology and neuroscience. But a team of researchers at the University of Amsterdam and the University of Newcastle now warns that a widely used statistical shortcut can turn phantom correlations into seemingly solid scientific findings. In a new study published in Behavior Research Methods, Michelle Donzallaz, Niek Stevenson, Andrew Heathcote, and Dora Matzke demonstrate how mathematical dependencies built into popular decision-making models can masquerade as genuine differences between people, and they offer a practical remedy grounded in hierarchical Bayesian estimation.

The models in question, known as evidence-accumulation models, describe decision making as a gradual, noisy accumulation of evidence toward one of two response boundaries. The most prominent example is the diffusion decision model, or DDM, which decomposes behavior in rapid two-choice tasks such as random-dot motion experiments into distinct parameters. Drift rate captures the speed and quality of information processing, boundary separation quantifies response caution, the starting point reflects response bias, and non-decision time accounts for processes outside the decision itself, such as stimulus encoding and motor execution. Because these parameters map onto interpretable cognitive processes, researchers often correlate them across participants to ask whether, for example, more cautious people also take longer to encode stimuli.

The pitfall identified by Donzallaz and colleagues concerns how these parameters are estimated. In what the authors call the two-step approach, researchers fit the model separately to each participant’s data, extract point estimates, and only then compute correlations across participants. The problem is that cognitive models are “sloppy”: their parameters trade off against one another within an individual because of the mathematical structure of the model’s likelihood function. In the DDM, for instance, there is a pronounced negative within-subject trade-off between non-decision time and boundary separation, and a positive trade-off between drift rate and boundary separation, particularly when error rates are low. These within-subject correlations are properties of the model itself, not facts about the people being studied, yet they leave fingerprints on the parameter estimates.

The new paper shows analytically and through simulation that a correlation computed between parameter estimates using the two-step approach is actually a weighted sum of the true between-subject correlation and the within-subject trade-off correlation. The weights depend on how much of the total variance in each parameter can be attributed to stable individual differences, quantified by intraclass correlations. When there is genuine between-subject variability in both parameters, the observed correlation falls somewhere between the true correlation and the trade-off correlation. But when between-subject variability is absent, the observed correlation becomes a pure echo of the within-subject dependency, even if the true relationship between people is exactly zero.

The study revisits a striking case reported by Grange and Schuch in 2023, who found strong negative correlations, reaching around negative 0.70 and beyond, between difference scores for boundary separation and non-decision time in evidence-accumulation models. Those authors concluded that such difference scores should not be used in individual-differences research at all. The Amsterdam-led team replicated the original simulations with the EZ diffusion model and confirmed the spurious correlation of roughly negative 0.71 to negative 0.76 when between-subject variability was set to zero. Crucially, however, they showed that as true individual differences in the difference scores were gradually introduced, the observed correlation drifted back toward its true value of zero, regardless of whether the experimental conditions differed on average. More trials per condition accelerated this return to the truth, because more data reduce estimation noise, but only an infinite number of trials would eliminate the contamination entirely.

The authors also identify a subtle but important technical flaw in the earlier simulations. A Pearson correlation coefficient assumes that both variables have finite, positive standard deviations. When all participants are simulated with identical parameter values, the true between-subject variability is zero, making the intended population correlation mathematically undefined rather than zero. The observed correlation between the estimates is then driven entirely by sampling noise, which itself is structured by the model’s internal trade-offs. By regenerating parameters from a multivariate normal distribution with explicit, adjustable variability in the difference scores, Donzallaz and colleagues were able to trace precisely how the mixture of within- and between-subject correlations unfolds, with a closed-form weighted-sum expression that accurately predicted their simulation results.

The proposed solution is hierarchical, or multilevel, modeling. Instead of estimating each participant separately and correlating the results afterward, hierarchical models assume that participant-level parameters are drawn from a population-level multivariate distribution whose mean vector and variance-covariance matrix are estimated simultaneously with the individual parameters. This single-model approach explicitly separates the two sources of covariation: within-subject trade-offs are absorbed by the likelihood at the individual level, while between-subject correlations are captured in the population-level covariance matrix. Inference is then drawn directly from the posterior distribution of the between-subject correlation, properly propagating estimation uncertainty from the participant level to the population level and yielding well-calibrated results.

When the team refit subsets of the simulated data with a Bayesian hierarchical implementation of the DDM using the EMC2 software package, the estimated between-subject correlations between the difference scores hovered credibly around zero across all levels of true between-subject variability. The spurious correlation of approximately negative 0.70 was confined to the within-subject correlations, which the authors extracted from each participant’s joint posterior distribution. Notably, an earlier attempt by Grange and Schuch to apply hierarchical modeling had still produced spurious results, because two independent hierarchical models were fit to the two conditions separately and difference scores were computed afterward from posterior means. As the new paper makes clear, explicitly modeling the covariance structure of all participant-level parameters within a single hierarchical model is essential; fitting separate models and correlating point estimates afterward inherits all the problems of the two-step approach.

The empirical reanalysis drove the point home. Grange and Schuch had randomly split trials from a single condition of a random-dot motion experiment, drawn from a many-analysts project, into two artificial conditions and found a staggering spurious correlation of negative 0.964 between difference scores. Donzallaz and colleagues repeated this random-split procedure ten times and fit both a simple three-parameter hierarchical DDM and a full seven-parameter version that also modeled starting-point bias and between-trial variability parameters. The full model correctly returned between-subject correlations indistinguishable from zero in every repetition, while the simple model showed a persistent downward bias. Posterior predictive checks revealed why: the simple model failed to describe the data, overestimating slow responses and underestimating fast ones. Descriptive adequacy, the authors conclude, is a prerequisite for trustworthy inference from parameter correlations, even under hierarchical estimation.

The implications extend well beyond the DDM. Sloppiness, or strong parameter trade-offs, arises in any multi-parameter model estimated with substantial uncertainty, including the ex-Gaussian distribution used for reaction times, simple and polynomial regression, and models throughout systems biology. The issue also parallels a long-standing concern in time-series research, famously illustrated by the correlation between typing speed and typos: across people, faster typists make fewer errors, but within a person, typing faster produces more mistakes. Cross-sectional correlations blend these levels according to intraclass correlations, a principle the new paper imports directly into cognitive modeling. Analogous multilevel techniques, such as multilevel vector autoregression and dynamic structural equation models, have been recommended for intensive longitudinal data for the same reason.

The authors distill their findings into a recommended workflow for anyone using cognitive models in individual-differences research. First, specify theoretically or empirically informed prior distributions where possible. Second, estimate the model hierarchically, including the full variance-covariance matrix of the parameters of interest, parameterized in terms of difference scores when condition effects are relevant. Third, check convergence of the sampling algorithm and evaluate the model’s descriptive adequacy against the observed data. Fourth, conduct model selection with tools such as Bayes factors or information criteria to remove unnecessary complexity; in many real applications, a simpler model with parameters constrained across conditions would be preferred, and difference scores would never be estimated in the first place. Fifth, perform prior sensitivity analyses when prior choices are arbitrary. When hierarchical modeling is genuinely infeasible, the fallback is to collect many trials per participant and condition, which shrinks within-subject variability and limits, though never eliminates, its contaminating influence.

The study also carries a constructive message for the field: contrary to the earlier recommendation to abandon difference scores altogether, correlations among evidence-accumulation model parameters, and among their differences, can indeed be used safely in individual-differences research, provided the estimation strategy respects the layered structure of the data. Recent advances in Bayesian sampling methods have made hierarchical estimation of the full DDM computationally practical, removing what had long been the main obstacle to adopting the approach routinely. As cognitive models continue to serve as measurement instruments linking psychological processes to neural and behavioral variation, the paper argues, the two-step habit of estimating first and correlating later should be retired in favor of approaches that disentangle, rather than conflate, the variation that lives within people and the variation that lives between them.