The research, led by Alan Victor de Souza Pinho and Rosalvo Ferreira de Oliveira Neto of the Federal University of San Francisco Valley, together with Edilson Beserra de Alencar Filho, tackles a stubborn bottleneck in environmental risk assessment. Experimental toxicity testing is expensive, slow, and ethically fraught, and for pollinators the available data are especially sparse. Quantitative structure-activity relationship, or QSAR, modeling offers a computational alternative, predicting a compound’s biological effects directly from its molecular structure. But QSAR models are only as good as the molecular representations fed into them, and building those representations traditionally requires laborious feature engineering with specialized descriptor libraries.

The team drew its data from ApisTox, a recently released open-access benchmark containing curated toxicity information for 1,035 compounds, of which 296 are classified as toxic to honey bees and 739 as non-toxic. The classification follows the United States Environmental Protection Agency’s regulatory convention: a compound is deemed toxic if its acute LD50 value, whether by oral or contact exposure, is at or below 11 micrograms per bee. The dataset aggregates information from authoritative sources including the EPA’s ECOTOX knowledgebase and the Pesticide Properties DataBase, and for each chemical it retains the most toxic exposure route, a conservative strategy designed to capture the highest potential risk to pollinators.

Into this benchmark the researchers introduced three competing ways of describing molecules. The first used PaDEL, a widely adopted open-source software that calculates hundreds of handcrafted molecular descriptors. The second used Morgan fingerprints computed with RDKit, circular substructure patterns that encode the atomic neighborhoods of each molecule at fixed radii. The third and most novel approach extracted embeddings from MolFormer, a BERT-style chemical language model developed by IBM that treats SMILES strings, the text-based notation for molecular structures, as a language. MolFormer was pretrained on millions of molecules from the PubChem and ZINC databases using a masked language modeling objective, learning to predict hidden tokens from their context and thereby absorbing structural, semantic, and physicochemical information without any labeled toxicity data.

Crucially, the researchers did not fine-tune the massive model. Instead, they adopted a feature-based transfer learning strategy, using the pretrained network as a frozen feature extractor that converts each SMILES string into a dense 768-dimensional vector. These embeddings then served as input to three classical machine learning classifiers: Random Forest, Support Vector Machine, and a Multilayer Perceptron. All models were evaluated with five-fold cross-validation using the area under the receiver operating characteristic curve, or ROC-AUC, a threshold-independent metric well suited to the dataset’s class imbalance. Notably, the embeddings came from a publicly available reduced-scale MolFormer checkpoint trained on only about 100 million molecules, roughly ten percent of the combined ZINC and PubChem corpora, since the full-scale checkpoints trained on the reported 1.1 billion molecules are not publicly available.

The results delivered a clear verdict. Random Forest paired with Morgan fingerprints achieved the best overall discrimination, with a mean ROC-AUC of 0.866, confirming that substructure-based representations remain formidable baselines. Yet Support Vector Machine combined with MolFormer embeddings came astonishingly close, reaching a ROC-AUC of 0.859, a gap of just 0.007. Considering that the embeddings originated from a comparatively lightweight pretraining regime, the near-parity is striking, and it raises an tantalizing question the authors pose explicitly: had the full-scale model trained on the complete 1.1-billion-molecule corpus been accessible, the remaining gap might have narrowed further or even reversed.

Against the older PaDEL descriptors, the verdict was unambiguous. MolFormer embeddings outperformed PaDEL across all three classifiers, with ROC-AUC gains ranging from 0.005 to 0.021, the largest improvement appearing for the Support Vector Machine. The authors attribute this to the alignment between MolFormer’s learned representations and margin-based decision boundaries. More fundamentally, the advantage reflects a difference in representational philosophy. Morgan fingerprints encode the presence of local substructures without capturing the global molecular context in which a functional group sits, and their binary hashing is vulnerable to bit collisions. MolFormer’s bidirectional self-attention, by contrast, encodes every atomic position in relation to all other tokens in the SMILES sequence simultaneously, producing a holistic representation in which toxicophoric groups are described within the context of the complete molecular architecture.

The qualitative analysis of individual predictions illuminated this distinction vividly. Examining cases where the embedding-based model succeeded while the fingerprint-based model failed, the researchers found compounds rich in phosphate esters and carbamate groups, or polyhalogenated aromatic systems, structural motifs associated with acetylcholinesterase inhibition and modulation of neuronal ion channels, the classic mechanisms of insecticidal action. Conversely, when the fingerprint model won, the compounds tended to belong to well-defined agrochemical classes with established low bee toxicity, such as triazine herbicides and triazole fungicides, where simple class-defining substructures are the dominant signal and local encoding suffices. The two models, in other words, see different things in the same molecules.

That complementarity was quantified as well. Although the two best models achieved similar average performance, the Pearson correlation between their predicted probabilities was only 0.7483, indicating partially overlapping but non-identical predictive patterns. In some cross-validation folds the correlation dropped as low as 0.6417 without any degradation in predictive accuracy, evidence that each representation captures distinct aspects of chemical space. The authors argue that this makes the embedding-based model a strong candidate for ensemble frameworks, in which majority voting, probability averaging, or stacking with a meta-learner could combine the strengths of both representations, potentially alongside graph neural network architectures, to build unified predictive platforms accessible through online tools and mobile applications.

The practical implications extend well beyond the leaderboard. By eliminating the need to integrate multiple descriptor calculation libraries and perform complex variable selection, the transfer learning approach substantially simplifies the modeling pipeline, lowering the barrier for ecotoxicological screening under limited labeled data, precisely the conditions that prevail in pollinator protection. The authors caution that their models were developed and evaluated exclusively within the chemical space of the ApisTox dataset, and they recommend formal applicability domain analysis, using tools such as leverage statistics or nearest-neighbor distances in embedding space, before any regulatory deployment. Future work, they note, should explore full-scale pretrained models and incorporate chronic toxicity endpoints. As honey bee populations continue to decline worldwide under the combined pressures of pesticide exposure, habitat loss, and climate change, tools that can rapidly and reliably flag dangerous chemicals before they reach the field may prove not just convenient but essential.

Subject of Research: Transfer learning with chemical language models for predicting honey bee toxicity

Article Title: Transfer learning for honey bee toxicity prediction: MolFormer versus classical QSAR representations

Article References: de Souza Pinho, A. V., de Alencar Filho, E. B., & de Oliveira Neto, R. F. (2026). Transfer learning for honey bee toxicity prediction: MolFormer versus classical QSAR representations. Ecotoxicology, 35(7), Article 166. https://doi.org/10.1007/s10646-026-03149-x

Image Credits: AI Generated

DOI: 10.1007/s10646-026-03149-x

Keywords: transfer learning, MolFormer, QSAR, honey bee toxicity, Apis mellifera, ApisTox, chemical language models, Morgan fingerprints, PaDEL descriptors, ecotoxicology, pesticide risk assessment, machine learning

The study revolves around transformer-based chemical language models, an AI architecture that has revolutionized natural language processing and is now being adapted to the natural sciences. Transformative models like ChatGPT, Google Gemini, and others operate by training on vast corpora of text, enabling them to generate coherent and contextually appropriate sentences. Chemical language models, however, operate on fundamentally different data: molecular representations coded as sequences such as SMILES strings, which translate the structure and elements of molecules into a sequence of characters comprehensible to the model. Despite the inherent differences in data type and volume—CLMs are generally trained on far less data than their linguistic counterparts—the question arises whether these models acquire genuine biochemical insights or make predictions based primarily on superficial correlations extracted from the training set.

To explore this question, the Bonn team, led by Prof. Dr. Jürgen Bajorath and doctoral student Jannik P. Roth, conducted a well-designed set of experiments involving systematic manipulation of the training data. Their model was trained on pairs consisting of amino acid sequences of enzymes or target proteins and compounds known to inhibit these proteins’ functions. In pharmaceutical research, finding molecules that can inhibit specific enzymes is a critical step in drug discovery, often guided by the functional relationship between the enzyme’s biochemical properties and potential drug candidates. The team’s approach aimed at understanding how a CLM would generate new compound suggestions when exposed to enzymes either similar to or distinct from those in the training set.

Initially, the researchers limited training to enzymes within specific families alongside their corresponding inhibitors. When the model was later tested with new enzymes from these same families, it successfully proposed plausible inhibitors, suggesting some internalization of patterns within that group. However, when challenged with enzymes from entirely different families whose biochemical functions diverged significantly, the model failed to produce meaningful inhibitor predictions. This outcome strongly suggests that the model’s “knowledge” resides more in recognizing statistical similarities rather than in mastering underlying biochemical mechanisms.

Delving deeper, it emerged that the models gauged similarity between enzymes based primarily on amino acid sequence homology, requiring only about 50–60% sequence alignment to make a positive match. This approach overlooks the critical detail that biochemically, only specific regions or active sites within an enzyme dictate its function, and minor variations — even a single amino acid substitution — can crucially impact activity. By placing equal importance on all portions of the sequence, the model failed to discriminate between functionally relevant and irrelevant segments. Such indiscriminate analysis leads to predictions driven by bulk sequence similarity rather than nuanced chemical or biological understanding.

Crucially, the manipulation experiments revealed that models could tolerate extensive scrambling or randomization of amino acid sequences without severely affecting outcomes, as long as the overall sequence retained some original residues. This further underscored the models’ reliance on superficial features and statistical correlation in their predictions rather than any deep, mechanistic insight into enzyme inhibition.

The study thereby challenges the perception that CLMs have achieved a substantive chemical understanding comparable to human experts. Rather, the transformer architectures appear predominantly to reflect patterns ingrained in their training datasets, effectively “echoing” known biochemical relationships in slightly modified forms. While this might suggest a limitation in their scope, it does not diminish their practical utility. The models can still generate viable suggestions for active compounds, which could serve as valuable starting points in drug discovery pipelines. Their ability to identify statistically similar enzymes and compounds holds potential for repurposing known drugs or guiding targeted molecular design.

These findings carry significant implications for how researchers and practitioners interpret CLM output. It cautions against overinterpreting the models’ predictions as evidence of biochemical comprehension. Instead, it frames them as powerful heuristic tools that sift through complex data patterns quickly and, importantly, generate hypotheses to be validated experimentally. The distinction between model “understanding” and pattern matching is not merely academic but has real consequences for the direction of AI-driven research in chemical and pharmaceutical sciences.

Despite these limits, CLMs remain impactful players in the drug discovery arena. By efficiently suggesting compounds that share characteristics with known inhibitors, they save time and resources in early research phases. The University of Bonn team’s work encourages the development of improved models that might incorporate biochemical rules more explicitly or integrate structural information so as to refine predictions beyond sequence-level similarity. This fusion of statistical learning with domain-specific chemical knowledge could be the next milestone in transforming AI’s role in molecular design.

The study also underscores the ongoing challenge of interpretability in AI models — often referred to as the “black box” problem. As Prof. Bajorath eloquently points out, peering inside these computational constructs to discern the causal dynamics behind their output remains difficult. Techniques for model explainability and an emphasis on transparent AI might therefore be key in advancing trustworthy applications of such technology in sensitive areas like drug development.

Financially supported by the German Academic Scholarship Foundation, this research has been formally published in the journal Patterns on October 14, 2025, under the title “Unraveling learning characteristics of transformer models for molecular design.” The detailed insights contribute significantly to the broader discourse about AI in life sciences, encouraging the scientific community to critically assess the capabilities and boundaries of current transformer-based CLMs.

For further inquiries, Prof. Dr. Jürgen Bajorath, Chair for Life Science Informatics at the University of Bonn, remains available for contact. This work collectively moves the field toward more sophisticated, chemically aware AI systems, setting a thoughtful agenda for future study that harmonizes empirical data with molecular biochemistry.

Subject of Research: Not applicable

Article Title: Unraveling learning characteristics of transformer models for molecular design

News Publication Date: 14-Oct-2025

Web References:
10.1016/j.patter.2025.101392

References:
Roth, J.P., Bajorath, J. Unraveling learning characteristics of transformer models for molecular design, Patterns, 2025.

Image Credits:
Photo: Gregor Hübl/University of Bonn

Keywords

Chemical language models, transformer models, AI in drug discovery, molecular design, SMILES strings, enzyme inhibition, sequence-based molecular design, machine learning interpretability, biochemical understanding, pharmaceutical research, computational modeling, artificial intelligence