The study, conducted by independent researcher Vera Wilde, introduces what she calls a capability-gap test. The logic is deceptively simple. Certain problems, such as calculating a positive predictive value from base-rate information, have been studied so extensively in humans that scientists know, with meta-analytic precision, exactly how well people can perform. When participants in an online study dramatically exceed those well-established human ceilings, the most plausible explanation is not superhuman cognition but machine assistance — a canary in the data coalmine, signaling that the sample may be contaminated by large language models.

The empirical basis for the proposal comes from two preregistered pilot studies of a Bayesian reasoning training tool, with a combined sample of 148 participants recruited through the Prolific platform. The tool was designed to teach people how to solve Bayesian inference problems, the kind of task exemplified by medical diagnosis questions: given a disease with a certain prevalence, a test with a certain sensitivity and false-positive rate, what is the probability that a person who tests positive actually has the disease? Decades of research, dating back to classic work by Daniel Kahneman and Amos Tversky and extended by Gerd Gigerenzer and Ulrich Hoffrage, have shown that most people fail such problems, even when the numbers are presented in natural frequency formats that make the underlying logic easier to grasp.

That failure is precisely what makes the task useful as a detector. A meta-analysis by McDowell and Jacobs found that only about 24 percent of people can solve a single Bayesian reasoning problem presented in natural frequency format — and that figure represents a ceiling, a level at which achieving a perfect score on a battery of five such problems is effectively unattainable for genuine human respondents. Yet in Wilde’s pilots, participants’ accuracy on positive predictive value calculation problems reached roughly three times the established human performance ceiling. In the second pilot, 57 percent of participants achieved perfect 5-for-5 scores, a result that should be extraordinarily rare in an uncontaminated human sample.

The technical heart of the approach lies in distinguishing two outcome measures: accuracy and algorithm use. Accuracy refers simply to whether the participant produced the correct numerical answer. Algorithm use, by contrast, refers to evidence in the participant’s response that they actually followed the Bayesian reasoning process — for example, constructing a frequency tree, counting cases, or showing the intermediate steps of the calculation. This distinction matters because a training intervention designed to improve Bayesian reasoning should, if it works, change both measures in tandem. A large language model, however, can produce correct answers without any visible reasoning process, or with a reasoning process that does not respond to the training manipulation in the way human learning does.

By tracking both measures simultaneously, researchers can separate two rival explanations for suspiciously high performance. If accuracy spikes but algorithm use does not, contamination is the likelier culprit, because the model supplies answers without the participant acquiring the underlying skill. If both accuracy and algorithm use rise together, the pattern is consistent with authentic learning effects, and the treatment signal can be preserved rather than discarded. In this way, the capability-gap test does not merely flag bad data; it helps researchers decide which parts of their dataset reflect genuine psychological phenomena and which parts reflect machine-generated noise.

Wilde frames the detection problem itself as a signal detection problem, structurally analogous to the mass screenings for low-prevalence conditions — such as disease screening — around which the Bayesian reasoning literature was originally developed. Just as a medical test must balance hits against false alarms, a contamination detector must catch bot-driven responses without wrongly excluding honest participants who happen to be statistically savvy. The known reference distributions from the Bayesian reasoning literature make this calibration possible in a way that ad hoc attention checks cannot. Rather than relying on generic screening questions, researchers can compare observed performance against quantified human benchmarks and estimate the probability that a given response pattern arose from machine assistance.

The approach offers four practical advantages over existing data-quality tools. First, the human performance ceilings are grounded in meta-analyses rather than informal intuition, giving researchers a defensible threshold for suspicion. Second, the human–large language model performance gap on these problems is large, which increases the sensitivity of the test. Third, the known reference distributions allow nuanced assessment rather than crude pass–fail judgments. Fourth, Bayesian reasoning problems are easy to embed in existing surveys, requiring no special software or platform cooperation. Together, these properties make the method deployable at scale across the many fields — psychology, marketing, political science, epidemiology — that increasingly depend on online samples.

The stakes are considerable. Prior research on crowd work has documented substantial and growing use of large language models by online workers, and studies of data contamination in machine learning itself show how memorized content can masquerade as genuine capability. If a meaningful fraction of respondents in an online study are completing tasks with chatbot help, effect sizes may be distorted, replication attempts may fail for reasons that have nothing to do with the underlying science, and the credibility of entire literatures built on crowdsourced data could be undermined. The problem echoes an older statistical concern: John Tukey’s foundational work on sampling from contaminated distributions warned that even small amounts of contamination can seriously mislead inference drawn from nominally clean data.

Wilde is careful to note the provenance of the idea: neither pilot study was originally designed to validate a contamination detection method, and the proposal emerged from post hoc analysis of unexpectedly strong results. That origin makes the capability-gap test a promising hypothesis rather than a fully validated diagnostic, and the author provides practical recommendations for researchers who wish to use these problems as data-quality diagnostics while the validation literature matures. Both studies were preregistered on the Open Science Framework, and all data, materials, and analysis code are publicly available, allowing other teams to scrutinize and extend the approach. If the method holds up under broader testing, Bayesian reasoning problems — long a symbol of human statistical frailty — may find a second career as guardians of scientific integrity, ensuring that the data feeding behavioral science come from human minds rather than the machines trained on those minds’ collective output.

Subject of Research: Detecting large language model contamination in online behavioral research samples using Bayesian reasoning problems as capability-gap tests

Article Title: An LLM canary in the online data coalmine: Bayesian reasoning problems as a capability-gap test for LLM contamination in online samples

Article References: Wilde, V. (2026). An LLM canary in the online data coalmine: Bayesian reasoning problems as a capability-gap test for LLM contamination in online samples. Behavior Research Methods, 58(11), Article 301. https://doi.org/10.3758/s13428-026-03184-w

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03184-w

Keywords: large language models, Bayesian reasoning, data contamination, online research, data quality, signal detection, natural frequencies, positive predictive value, crowdsourcing, behavioral research methods, capability-gap test, Prolific

The problem the researchers set out to solve is one of infrastructure. Studying how people and autonomous agents make decisions together requires an unusual combination of capabilities: a simulation environment flexible enough to model social dilemmas or cooperative tasks, reinforcement learning machinery to train AI partners, networking code to synchronize multiple participants in real time, and the mundane apparatus of online experiments, including consent forms, waiting rooms, matchmaking, surveys, and data collection. In most labs, each of these pieces has been rebuilt from scratch for every study, a burden that pushes researchers toward simpler proxies such as surveys, rule-based expert systems, or Wizard-of-Oz setups in which a human secretly plays the role of the AI. The result is a literature in which state-of-the-art learning agents are rarely tested against real people.

CoGrid attacks the first half of the problem. It is a multi-agent, grid-based simulation library inspired by Minigrid, a widely used single-agent toolkit, but redesigned so that any number of agents—human or artificial—can inhabit the same grid. Environments are formalized as partially observable Markov decision processes, with modular components for observations, actions, and rewards that researchers can mix, extend, and customize through configuration files and small Python classes. Crucially, CoGrid adopts the PettingZoo API, the emerging standard for multi-agent reinforcement learning, so any environment built with it plugs directly into existing training libraries. Agents are implemented as first-class grid objects rather than hidden state tracked outside the world, which is precisely the design choice that made the original Minigrid single-agent only.

The most distinctive feature of CoGrid is its dual backend. All simulation code runs through a backend-agnostic array namespace that dispatches to either NumPy or JAX at runtime. With the JAX backend active, environment functions are just-in-time compiled and can be vectorized across thousands of parallel environment instances on a GPU, enabling the high-throughput training that modern reinforcement learning demands. With the NumPy backend, the identical environment code runs without any JAX dependency, which makes it compatible with Pyodide, a compilation of Python to WebAssembly that allows code to execute directly inside a web browser. This single-implementation approach means a researcher can train an agent at scale on hardware and then deploy the exact same environment to participants’ browsers, something JAX-only libraries such as JaxMARL and NAVIX cannot do.

The empirical payoff of that architecture is striking. In a demonstration recreating the Overcooked cooperative cooking environment, CoGrid’s JAX backend scaled from roughly 4,500 simulation steps per second with a single instance to 5.6 million steps per second with 1,024 parallel instances running on a single NVIDIA GeForce RTX 3090, a throughput improvement of about 1.9 times over JaxMARL’s implementation of the same task. The NumPy backend is far slower, at around 450 steps per second, but it is the only variant of the three compared that can run natively in a browser. The authors argue this trade-off is the point: rapid prototyping, accelerated training, and web deployment from one codebase.

MUG handles the second half of the pipeline: turning a simulation into an actual experiment. It takes any Python environment that follows the Gymnasium or PettingZoo APIs and makes it playable in the browser, without game engines or a video-game development cycle. Experiments are composed as sequences of scenes—landing pages, instructions, waiting rooms, gameplay, surveys—stitched together by a Stager that can randomize or counterbalance conditions at the participant level. Data collection is handled through callbacks that fire at key events, logging actions, states, and custom variables to disk or external databases, and the platform generates completion codes for recruitment platforms such as Prolific and Amazon Mechanical Turk. Participants can even be filtered in advance by browser, device, or network latency.

The genuinely novel engineering lies in how MUG deals with latency. Most existing platforms run the environment on a server and stream frames to participants, which means every action requires a network round trip. At the 30 to 60 frames per second typical of real-time tasks, even moderate latency renders such interactions unusable, and the problem compounds in multiplayer settings where all players’ inputs must reach the server before the world can advance. MUG instead offers client-side execution, running the Python environment inside the participant’s browser via Pyodide, and for multiplayer it implements GGPO-style rollback netcode, a technique borrowed from latency-sensitive fighting games. Each client runs a deterministic copy of the environment with a shared random seed, applies its own input immediately, predicts the remote players’ inputs, and, when a prediction proves wrong, rolls back to the last confirmed state and silently re-simulates the intervening frames with the correct actions.

To demonstrate the framework end to end, the team ran proof-of-concept studies with human participants recruited through Amazon Mechanical Turk, approved by Carnegie Mellon’s Institutional Review Board. In the Overcooked studies, 29 participants played 20 episodes alongside a fixed reinforcement learning partner trained with Proximal Policy Optimization, while 40 participants played in human–human pairs; after exclusions, 23 and 36 participants respectively remained. The results echoed earlier findings from the original Overcooked-AI work: AI–AI pairs delivered roughly 7.5 dishes per episode, far outperforming human–AI pairs, whose performance stayed flat over time even as human–human pairs steadily improved by developing shared strategies from scratch.

A subtler behavioral pattern emerged in the contribution data. Although total team performance in human–AI pairs barely changed across episodes, humans gradually shifted toward delivering finished dishes rather than placing onions in the pot, quietly ceding the cooking role to the AI even though this produced no measurable gain. That dissociation—stable performance but evolving division of labor—is exactly the kind of psychological phenomenon the authors say the platform is built to expose, touching on questions of trust, role adoption, and complementarity in mixed human–AI teams. A second case study using the fast-paced physics game Slime Volleyball, which is not grid-based, showed that MUG works with external Gymnasium environments too, with human–AI pairs sustaining longer rallies than human–human pairs thanks to the trained agent’s higher baseline skill.

The authors are candid about limitations. CoGrid is confined to grid-based worlds and cannot model continuous control or physics-heavy dynamics, and MUG currently supports no explicit communication—no text chat or structured messaging—so all interaction is mediated through actions and observations. Communication features are planned for future releases. Even so, the framework fills a conspicuous gap. Existing environment libraries either require non-Python languages, lack hardware acceleration, or cannot run in browsers, and existing experiment platforms do not jointly support standard APIs, real-time multi-human play, and client-side execution. By lowering the barrier to entry on both fronts, CoGrid and MUG are intended to let researchers ask novel questions about how humans and machines coordinate, compete, and adapt to one another—questions that will only grow more urgent as autonomous agents enter everyday social life. Both tools, with documentation and source code, are freely available to the research community.

Subject of Research: An open-source framework for multi-agent simulation and web-based human–AI interaction experiments

Article Title: CoGrid & the Multi-User Gymnasium: A framework for multi-agent experimentation

Article References: McDonald, C., & Gonzalez, C. (2026). CoGrid & the Multi-User Gymnasium: A framework for multi-agent experimentation. Behavior Research Methods, 58(11), Article 295. https://doi.org/10.3758/s13428-026-03078-x

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03078-x

Keywords: human-AI interaction, multi-agent systems, reinforcement learning, CoGrid, Multi-User Gymnasium, open-source software, web-based experiments, PettingZoo, JAX, rollback netcode, Overcooked, behavioral research methods