New Benchmark Tests Whether AI Materials Models Get Physics Right

A machine learning model can predict how well a material conducts heat and still be wrong about the physics that makes it happen. That unsettling finding sits at the center of a new benchmark developed by Michele Simoncelli, assistant professor of applied physics at Columbia Engineering, together with Balázs Póta, Paramvir Ahlawat, and Gábor Csányi at the University of Cambridge. Published in Nature Communications, the benchmark evaluates machine learning models that aim to translate quantum-level atomic interactions into macroscopic thermal and mechanical properties, and it exposes cases where apparently sensible predictions arise for the wrong reasons.

The work arrives at a moment when such models are no longer laboratory curiosities: they are being adopted by major technology companies and startups to screen candidate materials, making the question of whether their answers are trustworthy a practical one rather than an academic exercise. The physical problem the benchmark addresses begins at an almost unimaginably small scale. Properties such as sound insulation, resistance to extreme heat, and thermal expansion originate from how the microscopic building blocks of a material—its nuclei and electrons—interact at equilibrium and respond to perturbations.

Atoms are typically about one ten-billionth of a metre across, so even a modest chunk of material contains an enormous number of interacting parts to keep track of. The task is further complicated by quantum mechanics, where particles behave neither as classical points nor as simple waves, and their states are described probabilistically until observed. Connecting this strange microscopic world to the tangible properties engineers care about has long been one of the central challenges of computational materials science.

In principle, the quantum behavior of a solid follows from a solution of the Schrödinger equation, the foundational equation of quantum mechanics. That equation describes the quantum behavior of particles such as electrons and nuclei, and its solutions give the energy levels and the spatial probability distribution—the "shape"—of a quantum particle. From these solutions, researchers can calculate the microscopic forces between atoms in a material.

In practice, however, the Schrödinger equation is too complex to solve analytically for realistic materials, so scientists rely on computational approximations. And those approximations become increasingly expensive as the number of particles grows: modeling a molecule or material can mean accounting for hundreds or even thousands of electrons, a burden that scales punishingly with system size. This is where machine learning has entered the picture.

A family of models known as machine-learning interatomic potentials attempts to sidestep the full quantum-mechanical calculation by learning from datasets of atomic positions, energies, forces, and stresses obtained from quantum-mechanical calculations. Once trained, these models predict atomic interactions without explicitly solving the Schrödinger equation each time, and they can run roughly 1000 times faster than traditional approaches. That speedup has fueled an explosion of interest, with models being proposed not only to characterize known materials but to discover new ones, predict their properties, and flag the best candidates for synthesis—increasingly with robotic laboratories closing the loop by automatically making and testing what the algorithms suggest.

Speed, however, is not the same as correctness. Póta, the graduate student who is first author of the article, points out that these models had been compared to each other mainly on their performance in predicting energy, which is indeed the important quantity for most material properties. But judging models on energy alone can miss potential errors in forces, which determine the dynamics of atoms—a quantity that is particularly relevant when predicting how materials respond to temperature or mechanical perturbations.

In other words, a model can ace the standard tests and still carry hidden defects that only surface when the question shifts from static stability to dynamic behavior. The missing ingredient, the team argues, is more physics. Atoms in materials vibrate at the quantum mechanical level, and these thermal vibrations—extraordinarily small, fast, and short-lived—are critical to properties like thermal conductivity and thermal expansion.

Models that appear well-suited to energy-based predictions have tended to distort the small effects of vibrations, missing important details about a material's underlying physics even when they happen to get the bigger picture mostly right. Simoncelli offers a crisp definition of the standard the field should aim for: a model can be called physics-aware when it predicts the macroscopic properties of materials as a consequence of correctly describing the materials' atomistic physics, namely their atomic vibrations. There are cases, he notes, in which ML models give apparently sensible predictions but for the wrong reasons.

To quantify that gap, the team began their benchmarking efforts about two years ago, when machine learning models were attracting increasing attention for their out-of-the-box performance and the researchers wanted to see how well they could predict their property of interest: thermal conductivity. They compared the predictions of several models available at the time against traditional quantum-mechanical calculations for a set of more than 100 different crystalline materials. The results were revealing.

Among models with very similar accuracy in predicting formation energies and material stability, some accurately predicted both atomic vibrational properties and macroscopic thermal conductivity. Others were significantly off on atomic vibrations but still predicted thermal conductivity accurately because microscopic errors cancelled out. And others were off on both.

The benchmark is not merely diagnostic; it also points toward a remedy. With a small amount of additional, material-specific training, the team achieved agreement within a few percent of reference calculations in selected materials and, for lithium bromide, agreement with experiments. "Our benchmark provides a 'physics-aware' signal to ML developers to optimize their models," said Póta.

By separating models that genuinely capture vibrational physics from those that merely land on the right number, the benchmark gives developers a concrete target to optimize against—one that reflects the underlying dynamics rather than an aggregate outcome that can mask compensating errors. Adoption has been rapid. In addition to being used by companies such as Meta and Microsoft, and by startups including Radical AI and Orbital Materials, the benchmark has been incorporated into Matbench Discovery, an interactive leaderboard that ranks machine learning models on their ability to predict crystal stability, structure, and thermal conductivity.

Janosh Riebesell, who created and maintains Matbench Discovery, welcomed the contribution: he was very happy when Simoncelli's group offered to contribute their test set and metric design to diversify the leaderboard, noting that it revealed large performance gaps between models that looked nearly identical on crystal stability prediction, exposing weaknesses in models lacking strong physical constraints. He added that the thermo-mechanical properties that must be modeled accurately to master heat conductivity are highly technologically relevant, so the benchmark gives an early signal on how useful machine learning models are becoming for advancing materials-driven technologies. The stakes extend well beyond leaderboard rankings.

Thermal properties are critical to thermal management across many areas of active research, from microchips, batteries, and quantum computers to jet engines, spacecraft heat shields, and nuclear reactors. Thermal conductivity is especially sensitive to the microscopic vibrational physics of a material, Simoncelli notes, though he adds that the general idea is relevant to other physical properties as well, a topic his group is currently exploring. In an era when AI models are beginning to drive discovery itself—proposing materials, predicting properties, and selecting synthesis candidates with little human oversight—a model that quietly gets aspects of a material's underlying physics wrong does not just score poorly on a benchmark.

It can discard good options and promote bad ones, with no one checking each answer. Efforts like those of Simoncelli and his colleagues aim to ensure that machine learning models for atomic interactions get their answers right, for the right reasons.

Subject of Research: A physics-aware benchmark for evaluating machine learning interatomic potentials on thermal and mechanical properties of materials

Article Title: AI for materials needs to be more physics-aware

Article References: AI for materials needs to be more physics-aware. Original publication (Bibliographic details unavailable.)

Image Credits: AI Generated

DOI: Not provided

Keywords: machine learning, interatomic potentials, thermal conductivity, atomic vibrations, materials science, benchmark, quantum mechanics, Schrödinger equation, Matbench Discovery, Columbia Engineering, Nature Communications, physics-aware AI