DNA replication begins at specific genomic locations known as replication origins. From each origin, molecular machines assemble and copy the surrounding chromosome in both directions. Bacteria often use a single origin, but eukaryotic chromosomes are much larger and typically require many starting points. The researchers focused on Saccharomyces cerevisiae, or budding yeast, whose replication origins are unusually well characterized. In this organism, origins occur within autonomous replication sequences, or ARS regions, generally about 100 to 200 base pairs long. These regions contain several functional elements, including the A element and B1, which together help form the principal binding site for the origin recognition complex, or ORC. The B2 element may provide an additional ORC-binding site or a platform for components of the replicative helicase.

A central sequence signal in yeast origins is the 11-base-pair ARS consensus sequence, or ACS. It is commonly represented as WTTTAYRTTTW, where W means either adenine or thymine and Y means cytosine or thymine. An extended version spans 17 base pairs and is written WWW-WTTTAYRTTTW-GTT. Yet the ACS alone cannot explain which origins actually function. The yeast genome contains roughly 12,000 matches to ACS-like motifs, but only about 500 are functional under normal conditions. Origin activity is therefore shaped by additional factors, including chromatin accessibility, transcription that can interfere with initiation, and secondary DNA features. This mismatch between abundant sequence motifs and comparatively rare active origins makes replication-origin prediction a demanding test for machine learning.

Earlier computational approaches often depended on labor-intensive feature engineering. Researchers had to convert sequences into numerical descriptions of nucleotide composition, DNA shape, physical properties, or sequence order before training support-vector machines, random forests, or other classifiers. Deep-learning methods reduced some of that manual work but frequently required models to be trained from scratch, and their learned features could be difficult to connect to recognizable biological mechanisms. The new study instead used pretrained transformer models. Transformers process sequences through self-attention, a mechanism that allows each input segment to weigh information from other segments while constructing a contextual representation. In principle, this enables a model to detect both short motifs and relationships between separated regions without being told in advance which biological features to search for.

DNABERT and DNABERT-2 share the basic BERT architecture, but they read DNA differently. DNABERT was pretrained on the human genome and converts sequences into overlapping four-base units, or 4-mers. Because adjacent tokens overlap, a DNA sequence is represented through a dense series of partially shared local windows. DNABERT-2 was pretrained on genomes from multiple species, including yeast, and uses byte-pair encoding, or BPE. BPE builds a vocabulary by repeatedly merging frequently occurring sequence segments, producing tokens of variable length. The newer model has about 117 million parameters and a vocabulary of 4,096 tokens, compared with approximately 86 million parameters for DNABERT. The difference in size comes mainly from the larger embedding matrix needed for BPE, not from deeper or wider transformer layers.

To train and test the models, the team assembled balanced datasets using 325 experimentally confirmed yeast origins from the curated OriDB resource. Each origin sequence was placed into a standardized 500-base-pair window. Shorter origins were extended with genuine neighboring genomic DNA, with the extra sequence distributed randomly between the two sides. This prevented the model from learning a trivial rule based on where the origin appeared within the window. The investigators created two contrasting negative datasets. In the Random-Neg set, non-origin sequences were sampled from genomic regions that did not overlap known origins. In the more difficult ACS-Neg set, negative sequences were selected from approximately 12,000 ACS matches that do not function as origins. Positive and negative sequences in the latter set therefore contained ACS-like signals of comparable strength, forcing the models to search for additional distinctions.

Both models were fine-tuned for binary classification: origin-containing sequence versus non-origin sequence. The DNA entered each model with special beginning and end tokens, while DNABERT-2 sequences were padded when its variable-length tokenization required equal input lengths. A classification layer attached to the representation of the initial classification token produced the probability that a sequence belonged to the origin class. The researchers used seven independent 70/10/20 training, validation, and test splits, selecting checkpoints based on validation accuracy and area under the receiver operating characteristic curve rather than simply taking the final training epoch. This design was intended to test robustness rather than maximize a single headline score. They also performed chromosome-based splitting to check that random partitioning had not artificially inflated performance.

DNABERT achieved an average test accuracy of 0.83 and an area under the curve of 0.90 on the easier Random-Neg dataset. DNABERT-2 reached 0.81 accuracy and 0.82 area under the curve. When ACS-rich non-origins were used as negatives, both models achieved approximately 0.72 accuracy, demonstrating that the task became substantially harder once the most obvious motif-based distinction was removed. The results indicate that the models were not merely detecting the presence of an ACS match. They retained some ability to distinguish functional origins from nonfunctional ACS sites, although the moderate accuracy also shows that sequence alone does not fully determine origin activity. Chromatin state, transcriptional context, and other cellular features remain outside the information available in the sequence windows.

The most revealing differences appeared when the researchers examined how the models arrived at their decisions. For DNABERT, they extracted attention scores associated with the classification token and projected those token-level values back onto individual nucleotides. Sharp attention peaks repeatedly fell within experimentally annotated origin regions rather than being distributed randomly across the 500-base-pair windows. The team collected short, 20-base-pair fragments around the strongest peaks and analyzed them with MEME, a probabilistic motif-discovery program that uses expectation-maximization to align variable motif instances. In test sequences from the Random-Neg dataset, the resulting motif was TTTTTWTTTATRTTT, with an E-value of 2.6 × 10−6, closely matching the known ACS pattern. A related training-set motif, TATATTTATRTWTWT, had an E-value of 2.3 × 10−32. In the ACS-Neg condition, where ACS motifs appeared in both classes, no comparably significant test-set motif emerged, consistent with the greater difficulty of the classification problem.

Attention maps from DNABERT-2 were less straightforward to interpret, so the researchers used perturbation experiments and Shapley additive explanations, or SHAP. Perturbation analysis asks what happens when a portion of the input is removed, altered, or rearranged. Eliminating ACS motifs from origin sequences changed DNABERT-2’s predictions, confirming that the motif contributed to classification. Randomly shuffling the model’s BPE tokens produced another important result: token identity appeared to matter strongly, while the precise order of some tokens mattered less than expected. SHAP analysis, repeated 30 times to reduce the randomness of its estimates, identified tokens that consistently supported origin or non-origin predictions. The team combined these findings into an AT-index incorporating overall adenine-thymine content, frequencies of AT-rich motifs, the longest uninterrupted AT run, and alternating AT runs. The index provided a compact way to quantify the AT-rich character of sequences associated with the model’s decisions, but the authors caution that SHAP values describe model behavior relative to its training background, not universal causal rules governing DNA replication.

The study also used shuffled sequences to test whether the models depended mainly on nucleotide composition or on the arrangement of bases. In one control, nucleotides within each positive sequence were randomly rearranged, preserving overall composition but destroying sequence order. In another, the sequence was divided into five-base-pair blocks and the blocks were shuffled, preserving local patterns while disrupting larger-scale organization. These experiments were designed to reveal whether the classifiers had learned broad compositional differences between origins and non-origins or more structured sequence information. The models’ near-random performance on shuffled data before task-specific fine-tuning showed that pretraining alone did not automatically confer the ability to distinguish intact from rearranged origin sequences. That capability emerged during adaptation to the replication-origin task.

Training from scratch produced weaker results than fine-tuning the pretrained systems. On the Random-Neg dataset, DNABERT trained from scratch averaged 0.71 accuracy, while DNABERT-2 averaged 0.60. The pretrained models used without fine-tuning performed at 0.37 and 0.55 accuracy, respectively, showing that general DNA representations alone were insufficient. Fine-tuning was therefore essential, but pretraining still supplied a useful starting point. The researchers emphasize that their goal was not to declare one architecture the universal winner. Instead, they wanted to determine whether genomic language models could reduce manual feature engineering while producing biologically interpretable signals. On that measure, the two systems behaved differently: overlapping k-mers gave DNABERT attention maps that more visibly reflected known motifs, whereas BPE tokenization in DNABERT-2 produced less transparent attention but supported alternative attribution analyses.

The findings matter because replication origins are not simply strings containing one magic sequence. In yeast, the ACS is an important anchor for ORC recognition, yet thousands of similar matches fail to initiate replication. A useful predictive model must therefore identify combinations of sequence properties while avoiding shortcuts created by biased negative examples. By deliberately challenging the models with ACS-containing non-origins, the researchers showed that transformer classifiers can recover signals beyond the canonical motif, though not perfectly. The work also illustrates a broader principle for biological AI: predictive accuracy is only part of the result. Tokenization, dataset construction, and explanation method can substantially influence what a model appears to have learned. As genomic language models are applied to human and other eukaryotic genomes—where replication origins are less sharply defined—the ability to distinguish meaningful biological signals from statistical artifacts will be crucial.

The study revolves around transformer-based chemical language models, an AI architecture that has revolutionized natural language processing and is now being adapted to the natural sciences. Transformative models like ChatGPT, Google Gemini, and others operate by training on vast corpora of text, enabling them to generate coherent and contextually appropriate sentences. Chemical language models, however, operate on fundamentally different data: molecular representations coded as sequences such as SMILES strings, which translate the structure and elements of molecules into a sequence of characters comprehensible to the model. Despite the inherent differences in data type and volume—CLMs are generally trained on far less data than their linguistic counterparts—the question arises whether these models acquire genuine biochemical insights or make predictions based primarily on superficial correlations extracted from the training set.

To explore this question, the Bonn team, led by Prof. Dr. Jürgen Bajorath and doctoral student Jannik P. Roth, conducted a well-designed set of experiments involving systematic manipulation of the training data. Their model was trained on pairs consisting of amino acid sequences of enzymes or target proteins and compounds known to inhibit these proteins’ functions. In pharmaceutical research, finding molecules that can inhibit specific enzymes is a critical step in drug discovery, often guided by the functional relationship between the enzyme’s biochemical properties and potential drug candidates. The team’s approach aimed at understanding how a CLM would generate new compound suggestions when exposed to enzymes either similar to or distinct from those in the training set.

Initially, the researchers limited training to enzymes within specific families alongside their corresponding inhibitors. When the model was later tested with new enzymes from these same families, it successfully proposed plausible inhibitors, suggesting some internalization of patterns within that group. However, when challenged with enzymes from entirely different families whose biochemical functions diverged significantly, the model failed to produce meaningful inhibitor predictions. This outcome strongly suggests that the model’s “knowledge” resides more in recognizing statistical similarities rather than in mastering underlying biochemical mechanisms.

Delving deeper, it emerged that the models gauged similarity between enzymes based primarily on amino acid sequence homology, requiring only about 50–60% sequence alignment to make a positive match. This approach overlooks the critical detail that biochemically, only specific regions or active sites within an enzyme dictate its function, and minor variations — even a single amino acid substitution — can crucially impact activity. By placing equal importance on all portions of the sequence, the model failed to discriminate between functionally relevant and irrelevant segments. Such indiscriminate analysis leads to predictions driven by bulk sequence similarity rather than nuanced chemical or biological understanding.

Crucially, the manipulation experiments revealed that models could tolerate extensive scrambling or randomization of amino acid sequences without severely affecting outcomes, as long as the overall sequence retained some original residues. This further underscored the models’ reliance on superficial features and statistical correlation in their predictions rather than any deep, mechanistic insight into enzyme inhibition.

The study thereby challenges the perception that CLMs have achieved a substantive chemical understanding comparable to human experts. Rather, the transformer architectures appear predominantly to reflect patterns ingrained in their training datasets, effectively “echoing” known biochemical relationships in slightly modified forms. While this might suggest a limitation in their scope, it does not diminish their practical utility. The models can still generate viable suggestions for active compounds, which could serve as valuable starting points in drug discovery pipelines. Their ability to identify statistically similar enzymes and compounds holds potential for repurposing known drugs or guiding targeted molecular design.

These findings carry significant implications for how researchers and practitioners interpret CLM output. It cautions against overinterpreting the models’ predictions as evidence of biochemical comprehension. Instead, it frames them as powerful heuristic tools that sift through complex data patterns quickly and, importantly, generate hypotheses to be validated experimentally. The distinction between model “understanding” and pattern matching is not merely academic but has real consequences for the direction of AI-driven research in chemical and pharmaceutical sciences.

Despite these limits, CLMs remain impactful players in the drug discovery arena. By efficiently suggesting compounds that share characteristics with known inhibitors, they save time and resources in early research phases. The University of Bonn team’s work encourages the development of improved models that might incorporate biochemical rules more explicitly or integrate structural information so as to refine predictions beyond sequence-level similarity. This fusion of statistical learning with domain-specific chemical knowledge could be the next milestone in transforming AI’s role in molecular design.

The study also underscores the ongoing challenge of interpretability in AI models — often referred to as the “black box” problem. As Prof. Bajorath eloquently points out, peering inside these computational constructs to discern the causal dynamics behind their output remains difficult. Techniques for model explainability and an emphasis on transparent AI might therefore be key in advancing trustworthy applications of such technology in sensitive areas like drug development.

Financially supported by the German Academic Scholarship Foundation, this research has been formally published in the journal Patterns on October 14, 2025, under the title “Unraveling learning characteristics of transformer models for molecular design.” The detailed insights contribute significantly to the broader discourse about AI in life sciences, encouraging the scientific community to critically assess the capabilities and boundaries of current transformer-based CLMs.

For further inquiries, Prof. Dr. Jürgen Bajorath, Chair for Life Science Informatics at the University of Bonn, remains available for contact. This work collectively moves the field toward more sophisticated, chemically aware AI systems, setting a thoughtful agenda for future study that harmonizes empirical data with molecular biochemistry.

Subject of Research: Not applicable

Article Title: Unraveling learning characteristics of transformer models for molecular design

News Publication Date: 14-Oct-2025

Web References:
10.1016/j.patter.2025.101392

References:
Roth, J.P., Bajorath, J. Unraveling learning characteristics of transformer models for molecular design, Patterns, 2025.

Image Credits:
Photo: Gregor Hübl/University of Bonn

Keywords

Chemical language models, transformer models, AI in drug discovery, molecular design, SMILES strings, enzyme inhibition, sequence-based molecular design, machine learning interpretability, biochemical understanding, pharmaceutical research, computational modeling, artificial intelligence