The team’s central insight begins with a deceptively simple observation about the data that scientific machine learning models consume. Most property prediction tasks in chemistry and materials science involve what the authors call coordinate–category data: continuous positional vectors, such as the three-dimensional coordinates of atoms in a crystal, paired with discrete categorical labels, such as which chemical elements sit at those positions. Predicting a formation energy or an elastic modulus requires the model to reason jointly about the ‘what’ and the ‘where’. The researchers asked a question that, surprisingly, had not been answered systematically: can transformer-based language models actually learn from the coordinate half of that pairing, or do they quietly fall back on the categorical patterns they find so easy to digest?

Answering that question with real materials data is treacherous, because real datasets are riddled with correlations. Certain elemental compositions almost always crystallize in certain structures, so a model can appear to understand geometry while merely memorizing composition. To cut through this confounding, the authors engineered synthetic datasets built on a physics-inspired hypothetical potential, an energy function that blends two contributions with a tunable mixing parameter alpha. At one extreme, the target energy depends entirely on a category term, a weighted sum over the counts of each entity type, mapped to force-field-like parameters. At the other extreme, it depends entirely on a coordinate term, a sum of pairwise interaction potentials, akin to Lennard-Jones interactions, over the distances between neighboring points. By sweeping alpha from zero to one, the researchers could dial the difficulty of learning geometry up and down with surgical precision, and measure exactly where each architecture breaks down.

The results were unambiguous. Fine-tuned BERT models, whether trained on purely coordinate-dominated or category-dominated versions of the synthetic tasks, consistently showed larger errors when coordinate information mattered most. The team quantified this with two aggregate metrics, the coordinate contribution and the category contribution, each measuring performance in the relevant regime relative to a balanced baseline. The difference between them, dubbed the coordinate–category cliff, or CC-cliff, was positive for language models across every dataset tested. In plain terms: the models excelled at learning which types of entities were present and how many, but systematically failed to extract the spatial relationships that determine the energy. Graph neural networks showed the mirror-image behavior, handling coordinate-dominated tasks with ease, while a descriptor-based model, MODNet, sat near the balanced middle ground. The cliff, the authors conclude, is not a property of the task itself but a consequence of architectural inductive bias.

Perhaps the most revealing comparison came from an unlikely source: n-gram models, the crude statistical text models of a previous era. Because prior work has shown that some behaviors of large language models can be approximated by n-gram statistics, the researchers trained simple n-gram baselines on the same tasks. The match was striking. Across all datasets, the n-gram models traced essentially the same performance curves as the fine-tuned transformers, suggesting that for coordinate–category data, the large models behave like superficial pattern matchers rather than learners of physical law. The arithmetic explains why. Encoding atomic positions at a modest resolution of 0.1 angstrom across a 20-angstrom cell inflates the vocabulary of possible location tokens to over a billion, and learning even a simple pairwise interaction would require sampling from a space roughly 6.4 times ten to the thirteenth times larger than the compositional-only case. Learning geometry from text tokens this way is astronomically sample-inefficient.

To test whether these synthetic findings carried over to real materials science, the team built MatText, an open-source benchmarking framework that converts crystal structures into nine different text representations, each encoding different combinations of information and inductive bias. Some representations, like SLICES, contain only composition and bonding information. Others, like CIF Symmetrized, encode the full positions of all atoms along with unit cell and symmetry details. The framework also supports multiple tokenization schemes and both causal and masked language models, allowing the researchers to disentangle the effects of representation, tokenizer, architecture and dataset. The verdict from more than 2,000 language model training jobs was blunt: adding explicit geometric information to the text frequently did not help at all, and sometimes hurt. Representations lacking coordinates, such as SLICES and local-environment descriptions rendered in natural language, matched or beat geometry-rich formats on predicting shear modulus, bulk modulus and perovskite formation energy.

Then came the scaling experiments, and with them the study’s most provocative result. In language modeling, scale has been the reliable lever of progress, formalized in empirical scaling laws that have driven the field’s meteoric rise. The researchers pretrained BERT models from scratch on datasets ranging from 30,000 to 2 million crystal structures drawn from the NOMAD database, and separately applied parameter-efficient fine-tuning to Llama-2 models of 7 billion, 13 billion and 70 billion parameters. For shear and bulk modulus, neither axis of scaling moved the needle meaningfully, with changes in root-mean-square error staying within the noise band of the experimental design. Formation energy showed somewhat larger gains, up to roughly 50 percent for some geometric representations under model scaling, but even accounting for these improvements, the absolute performance of the language models remained far below graph neural network approaches. Bigger was not better; it was barely different.

The final piece of evidence came from the widely used MatBench leaderboard, where the authors plotted the normalized errors of graph-based approaches, including COGN, SchNet and DimeNet, against language-based approaches, including LLM-Prop, CrabNet and Robocrystallographer-derived models. The segregation was stark enough to draw a line through: graph neural networks clustered at low error, language models at high error, with an almost empty band between them. The authors call this segregation the GNN-LM wall, and its message is that mechanical and other structure-dominated properties simply cannot be modeled efficiently by current language architectures, independent of representation, model size or dataset size. Closing the gap, they argue, would require fundamental changes to how language models handle coordinates, such as coordinate-aware tokenization in which atomic positions become learned continuous embeddings rather than discretized text tokens.

The study is candid about its limits. The experiments covered three properties from a restricted set of benchmark datasets, only encoder-based BERT and decoder-based Llama-2 transformers were evaluated, and the scaling study relied on parameter-efficient fine-tuning rather than full fine-tuning or pretraining from scratch on coordinate-rich corpora. Multimodal models that fuse text with explicit coordinate embeddings were not investigated. The authors also lay out a roadmap for what comes next: extending the CC-cliff framework to molecular quantum chemical properties and biological sequence–structure tasks, testing coordinate-aware tokenization to determine whether the limitation is intrinsic to transformers or merely to their text-based interface, and using causal intervention techniques such as activation patching to pinpoint which layers suppress geometric information.

Still, the practical guidance is already clear, and it cuts against the prevailing enthusiasm for one-model-fits-all artificial intelligence. Language models are excellent at learning from categorical patterns, the element types and compositions that dominate many datasets, and they can be deployed productively for tasks where those patterns carry most of the signal. But when the property in question hinges on how atoms are arranged in space, purpose-built geometric architectures remain the right tool, and no amount of text-wrapping or brute-force scaling currently changes that. In an era rushing toward universal models, the study is a reminder that understanding the inductive biases of our tools matters as much as ever. The future of scientific machine learning, the authors suggest, lies not in a single model to rule them all, but in matching the right architecture to the right problem—and knowing, precisely, where the wall stands.

Subject of Research: Limitations of large language models in learning coordinate information for materials property prediction

Article Title: Less can be more for predicting properties with large language models

Article References: Less can be more for predicting properties with large language models. (n.d.). https://doi.org/10.1038/s43588-026-01045-2

Image Credits: AI Generated

DOI: 10.1038/s43588-026-01045-2

Keywords: large language models, materials science, graph neural networks, property prediction, coordinate-category data, inductive bias, scaling laws, MatText, crystal structures, machine learning benchmarks, CC-cliff, transformers