New model untangles agreement from accuracy in teamwork measurement

Every team leader knows the frustration of watching two colleagues nod along in perfect harmony while marching in the wrong direction. Agreement, in other words, is not the same thing as being right. Yet most existing tools for measuring collaborative problem-solving quietly blur that distinction, scoring a team by its final answer and leaving researchers blind to the interaction processes that produced it. A new study published in Behavior Research Methods aims to fix exactly that, introducing a statistical framework that pulls apart the two threads of dyadic teamwork: whether partners converge on a joint response, and whether that response is actually correct.

The model, called the Teamwork Response Tree, or TRTree, was developed by Peida Zhan and Gaohong Chu of the School of Psychology at Zhejiang Normal University in China. It belongs to a family of techniques known as item response tree models, which have proven powerful in individual assessment for disentangling the psychological processes that lie behind a single observable answer. What Zhan and Chu have done is extend this machinery to pairs of people working together, modeling the dyad rather than the lone test-taker as the unit of collaboration.

The core idea is elegant. When two team members respond to a collaborative item, their joint outcome can be represented as a branching tree of latent decisions. At the first branch sits the agreement process: did the partners, whatever their individual views, manage to settle on a common answer? At the second branch sits the accuracy process: given that agreement was reached, was the agreed-upon answer correct? Each branch is governed by its own latent variables. Individual ability captures how skilled each member is, team-level ability reflects the joint competence of the pair, agreement propensity describes a team’s tendency to converge rather than disagree, and item-level agreement difficulty captures how hard it is for any team to reach consensus on a particular task.

This decomposition matters because conventional scoring methods typically conflate the two dimensions. A team that agrees quickly but is wrong receives the same low score, roughly speaking, as a team that disagrees endlessly on what was actually a solvable problem. Conversely, a team lucky enough to guess identically and correctly looks indistinguishable from one that reasoned its way to the truth together. By formally separating convergence from correctness, the TRTree model allows researchers to ask subtler questions: Is this team good at problem-solving but poor at communication? Does agreement on easy items come more readily than agreement on hard ones? Does one member’s ability dominate the joint outcome?

The authors formulated the model within the item response tree framework and estimated its parameters using Bayesian methods, an approach that has become standard for complex psychometric models because it handles the intricate dependency structures that arise in dyadic data. Bayesian estimation also provides full uncertainty quantification for every parameter, which is essential when the quantities of interest, such as a team’s latent agreement propensity, are never directly observed.

To demonstrate the model in action, Zhan and Chu applied it to empirical data from dyadic collaborative matrix reasoning tasks, the kind of visual puzzles familiar from intelligence assessments but adapted for two people to solve jointly. Their earlier work in this area drew on a dataset of 67 teams, and the data used in the present study expanded the total to 160 teams. The empirical illustration showed that the model yields interpretable estimates across all four key dimensions: individual ability, team-level ability, team-level agreement propensity, and item-level agreement difficulty. In other words, the outputs are not just statistically well-behaved but substantively meaningful for anyone trying to understand how a particular pair works together.

Interpretability alone is not enough for a measurement model, however; it must also recover the truth from limited data. To test that, the researchers conducted a Monte Carlo simulation study, generating synthetic datasets under a range of conditions and checking whether the estimation procedure could recover the known parameter values. The results indicated satisfactory parameter recovery, with estimation accuracy improving in predictable ways as the data became more informative. That predictability is valuable for practitioners: it means researchers designing collaborative assessments can anticipate how much data they need, and how informative their items must be, to obtain trustworthy measurements of agreement and accuracy separately.

The broader context makes clear why this contribution arrives at a pressing moment. Collaborative problem-solving has been elevated to the status of a core twenty-first-century competency, appearing in international assessments such as PISA 2015 and shaping hiring and training in sectors from healthcare to aviation. Yet measurement has lagged behind ambition. Assessments of teamwork have often relied on self-report questionnaires, observer ratings, or final-outcome scores, none of which fully capture the dynamic interplay between team members. Prior psychometric work has begun to model interaction patterns and process-stream data, and Zhan and Chu’s own earlier teamwork cognitive diagnostic model contributed to that movement, but the TRTree adds a distinct process-level lens by treating agreement itself as a measurable latent tendency rather than a byproduct of correctness.

The implications run in several directions. For assessment designers, the model offers a principled scoring approach that preserves information about the interaction process, potentially enabling feedback that tells a team not just that they underperformed but why, whether through weak individual skills or through a failure to converge. For researchers studying team effectiveness, separating agreement from accuracy opens new questions: teams with high agreement propensity but modest ability may be fast but fragile, while teams with high ability but low agreement propensity may contain the right raw material undone by coordination failure. Items themselves can now be characterized by agreement difficulty, allowing test developers to build assessments that deliberately probe consensus-building alongside problem-solving.

The model code and data for estimating the TRTree have been made openly available on the Open Science Framework, lowering the barrier for other laboratories to adopt and extend the approach. Like any new measurement framework, it will face the test of application across different task types, team sizes beyond dyads, and real-world settings where collaboration is messier than a matrix reasoning puzzle. But by giving researchers a way to formally distinguish the science of agreeing from the science of being right, the Teamwork Response Tree model marks a genuine step toward assessments that measure how teams work, not merely what they produce.

Subject of Research: A psychometric model separating agreement and accuracy processes in dyadic collaborative problem-solving assessment

Article Title: A teamwork response tree model for dyadic agreement-accuracy responses

Article References: Zhan, P., & Chu, G. (2026). A teamwork response tree model for dyadic agreement-accuracy responses. Behavior Research Methods, 58(11), Article 303. https://doi.org/10.3758/s13428-026-03178-8

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03178-8

Keywords: teamwork measurement, collaborative problem solving, item response tree model, item response theory, agreement-accuracy responses, dyadic data, Bayesian estimation, psychometrics, matrix reasoning, team ability, behavior research methods, teamwork