Round-robin tests quantify catalyst activity and deactivation in CO2 hydrogenation modelling

In the race to harness artificial intelligence for discovering better catalysts, a sobering new study has emerged that quantifies just how fragile the experimental foundations of machine learning in chemistry can be. An international consortium of four laboratories, working in a coordinated round-robin testing campaign, set out to measure how much uncertainty creeps into catalyst performance data even under the most controlled conditions imaginable. Their findings, published in Nature Catalysis, reveal that variability between laboratories—driven largely by something as mundane as heat management—can render seemingly robust structure–performance relationships statistically meaningless when data from different labs are pooled together. For a field increasingly reliant on curated datasets to train predictive models, the implications are far-reaching.

The subject of the study was a well-defined catalytic system: rhodium nanoparticles supported on titanium dioxide (Rh/TiO₂), a material of intense interest for the thermocatalytic hydrogenation of carbon dioxide. This reaction, which converts CO₂ and hydrogen into carbon monoxide and methane, sits at the heart of efforts to close the carbon cycle and produce sustainable fuels and chemical feedstocks. Rh/TiO₂ is a particularly instructive test case because the balance between the two products—CO via the reverse water–gas shift pathway and CH₄ via methanation—is sensitive to subtle differences in catalyst structure and reaction conditions. A machine learning model trained on data from this system should, in principle, be able to learn which input variables—reaction temperature, rhodium loading, and synthesis method—govern conversion, selectivity, and the production rates of CO and CH₄.

What the team did was deceptively simple in design but logistically demanding in execution. A single batch of catalyst, prepared under identical conditions using multiple synthesis methods and rhodium loadings, was distributed to four independent laboratories. Each lab followed the same testing protocol, applying the same temperatures, feed compositions, and analytical procedures to measure conversion, selectivity, and deactivation behavior over time. This round-robin approach is a standard in fields like metrology and clinical chemistry, where interlaboratory comparisons are used to validate measurement methods, but it is strikingly rare in catalysis research. The result is one of the first quantitative assessments of both intralaboratory and interlaboratory variability for a heterogeneous catalyst under reaction conditions.

Within any single laboratory, the data told a familiar and encouraging story. The relationships between the input variables and the outputs were clear and reproducible: increasing reaction temperature drove conversion upward in predictable ways, rhodium loading influenced activity, and synthesis method left measurable fingerprints on selectivity. Statistical analysis of the intralaboratory datasets yielded strong correlations, the kind of clean structure–performance relationships that researchers routinely publish and that machine learning models are designed to discover. Had each laboratory worked in isolation, each would have concluded that its dataset was of high quality, suitable for training reliable predictive models.

The picture changed dramatically once data from all four laboratories were combined. When interlaboratory variability was included in the statistical analysis, many of the relationships that had been significant within individual labs became statistically insignificant across the pooled dataset. The input–output correlations that machine learning algorithms depend on were, in effect, drowned out by noise of a magnitude comparable to—or larger than—the effects themselves. In other words, a model trained on the combined data would struggle to distinguish genuine chemical trends from artifacts introduced by differences in how the identical catalysts were tested in different places.

Digging into the sources of this variability, the researchers identified heat management as a key contributor. CO₂ hydrogenation is not thermally neutral; depending on the product distribution, it can release substantial heat, and methane formation in particular is strongly exothermic. Small differences in how reactors dissipate heat—reactor geometry, catalyst bed configuration, heat transfer characteristics, and the placement of temperature sensors—can create local temperature gradients and hot spots that alter both activity and selectivity. Since temperature is itself one of the most important input variables for the reaction, uncontrolled deviations between the nominal setpoint and the actual catalyst temperature introduce systematic errors that no amount of replicate testing within a single lab can reveal. This makes heat management a hidden confounder: the reported temperature may be identical across labs, while the effective temperature experienced by the catalyst is not.

The study also examined catalyst deactivation, a critical yet frequently underreported aspect of catalytic performance. Deactivation rates, which determine how long a catalyst remains useful, are notoriously sensitive to operating conditions, trace impurities in feed gases, and reactor materials. The round-robin results showed that deactivation behavior, like initial activity, carried significant uncertainty when compared across laboratories, compounding the challenge for any modeling effort that seeks to predict catalyst lifetime from experimental data.

The lessons for the machine learning community in catalysis are direct and actionable. First, uncertainty analysis must be built into the selection of performance metrics and input features for data-driven models, not treated as an afterthought. A feature that appears important in a single-lab dataset may lose its significance entirely when interlaboratory variability is accounted for, meaning that models trained on single-source data risk learning lab-specific artifacts rather than generalizable chemistry. Second, the field needs benchmarks that capture realistic variability. Datasets assembled from literature, where different reactors, protocols, and analytical methods abound, almost certainly carry interlaboratory uncertainties of the same order as those measured here—yet these uncertainties are almost never quantified or reported. Without them, the error bars on model predictions are systematically underestimated.

There is a constructive message embedded in the findings as well. By identifying heat management as a dominant source of variability, the study points toward concrete mitigation strategies: standardized reactor designs and testing protocols, better thermal characterization of catalyst beds, explicit reporting of temperature measurement uncertainty, and the routine incorporation of round-robin or interlaboratory validation into dataset curation efforts. The authors argue that rigor and reproducibility in catalysis research can be materially improved by acknowledging and quantifying these variability sources rather than assuming they are negligible.

As machine learning continues to be championed as an accelerator of catalyst discovery, this work serves as a reminder that the quality of the data is the limiting reagent. No algorithm, however sophisticated, can extract reliable chemistry from measurements whose uncertainties have not been characterized. The four-laboratory round-robin on Rh/TiO₂ provides both a warning and a template: a demonstration of how large the hidden variability can be, and a methodological framework for measuring it. For a field aspiring to predictive, data-driven catalysis, quantifying uncertainty is not optional bookkeeping—it is the foundation on which trustworthy models must be built.