DNA replication begins at specific genomic locations known as replication origins. From each origin, molecular machines assemble and copy the surrounding chromosome in both directions. Bacteria often use a single origin, but eukaryotic chromosomes are much larger and typically require many starting points. The researchers focused on Saccharomyces cerevisiae, or budding yeast, whose replication origins are unusually well characterized. In this organism, origins occur within autonomous replication sequences, or ARS regions, generally about 100 to 200 base pairs long. These regions contain several functional elements, including the A element and B1, which together help form the principal binding site for the origin recognition complex, or ORC. The B2 element may provide an additional ORC-binding site or a platform for components of the replicative helicase.

A central sequence signal in yeast origins is the 11-base-pair ARS consensus sequence, or ACS. It is commonly represented as WTTTAYRTTTW, where W means either adenine or thymine and Y means cytosine or thymine. An extended version spans 17 base pairs and is written WWW-WTTTAYRTTTW-GTT. Yet the ACS alone cannot explain which origins actually function. The yeast genome contains roughly 12,000 matches to ACS-like motifs, but only about 500 are functional under normal conditions. Origin activity is therefore shaped by additional factors, including chromatin accessibility, transcription that can interfere with initiation, and secondary DNA features. This mismatch between abundant sequence motifs and comparatively rare active origins makes replication-origin prediction a demanding test for machine learning.

Earlier computational approaches often depended on labor-intensive feature engineering. Researchers had to convert sequences into numerical descriptions of nucleotide composition, DNA shape, physical properties, or sequence order before training support-vector machines, random forests, or other classifiers. Deep-learning methods reduced some of that manual work but frequently required models to be trained from scratch, and their learned features could be difficult to connect to recognizable biological mechanisms. The new study instead used pretrained transformer models. Transformers process sequences through self-attention, a mechanism that allows each input segment to weigh information from other segments while constructing a contextual representation. In principle, this enables a model to detect both short motifs and relationships between separated regions without being told in advance which biological features to search for.

DNABERT and DNABERT-2 share the basic BERT architecture, but they read DNA differently. DNABERT was pretrained on the human genome and converts sequences into overlapping four-base units, or 4-mers. Because adjacent tokens overlap, a DNA sequence is represented through a dense series of partially shared local windows. DNABERT-2 was pretrained on genomes from multiple species, including yeast, and uses byte-pair encoding, or BPE. BPE builds a vocabulary by repeatedly merging frequently occurring sequence segments, producing tokens of variable length. The newer model has about 117 million parameters and a vocabulary of 4,096 tokens, compared with approximately 86 million parameters for DNABERT. The difference in size comes mainly from the larger embedding matrix needed for BPE, not from deeper or wider transformer layers.

To train and test the models, the team assembled balanced datasets using 325 experimentally confirmed yeast origins from the curated OriDB resource. Each origin sequence was placed into a standardized 500-base-pair window. Shorter origins were extended with genuine neighboring genomic DNA, with the extra sequence distributed randomly between the two sides. This prevented the model from learning a trivial rule based on where the origin appeared within the window. The investigators created two contrasting negative datasets. In the Random-Neg set, non-origin sequences were sampled from genomic regions that did not overlap known origins. In the more difficult ACS-Neg set, negative sequences were selected from approximately 12,000 ACS matches that do not function as origins. Positive and negative sequences in the latter set therefore contained ACS-like signals of comparable strength, forcing the models to search for additional distinctions.

Both models were fine-tuned for binary classification: origin-containing sequence versus non-origin sequence. The DNA entered each model with special beginning and end tokens, while DNABERT-2 sequences were padded when its variable-length tokenization required equal input lengths. A classification layer attached to the representation of the initial classification token produced the probability that a sequence belonged to the origin class. The researchers used seven independent 70/10/20 training, validation, and test splits, selecting checkpoints based on validation accuracy and area under the receiver operating characteristic curve rather than simply taking the final training epoch. This design was intended to test robustness rather than maximize a single headline score. They also performed chromosome-based splitting to check that random partitioning had not artificially inflated performance.

DNABERT achieved an average test accuracy of 0.83 and an area under the curve of 0.90 on the easier Random-Neg dataset. DNABERT-2 reached 0.81 accuracy and 0.82 area under the curve. When ACS-rich non-origins were used as negatives, both models achieved approximately 0.72 accuracy, demonstrating that the task became substantially harder once the most obvious motif-based distinction was removed. The results indicate that the models were not merely detecting the presence of an ACS match. They retained some ability to distinguish functional origins from nonfunctional ACS sites, although the moderate accuracy also shows that sequence alone does not fully determine origin activity. Chromatin state, transcriptional context, and other cellular features remain outside the information available in the sequence windows.

The most revealing differences appeared when the researchers examined how the models arrived at their decisions. For DNABERT, they extracted attention scores associated with the classification token and projected those token-level values back onto individual nucleotides. Sharp attention peaks repeatedly fell within experimentally annotated origin regions rather than being distributed randomly across the 500-base-pair windows. The team collected short, 20-base-pair fragments around the strongest peaks and analyzed them with MEME, a probabilistic motif-discovery program that uses expectation-maximization to align variable motif instances. In test sequences from the Random-Neg dataset, the resulting motif was TTTTTWTTTATRTTT, with an E-value of 2.6 × 10−6, closely matching the known ACS pattern. A related training-set motif, TATATTTATRTWTWT, had an E-value of 2.3 × 10−32. In the ACS-Neg condition, where ACS motifs appeared in both classes, no comparably significant test-set motif emerged, consistent with the greater difficulty of the classification problem.

Attention maps from DNABERT-2 were less straightforward to interpret, so the researchers used perturbation experiments and Shapley additive explanations, or SHAP. Perturbation analysis asks what happens when a portion of the input is removed, altered, or rearranged. Eliminating ACS motifs from origin sequences changed DNABERT-2’s predictions, confirming that the motif contributed to classification. Randomly shuffling the model’s BPE tokens produced another important result: token identity appeared to matter strongly, while the precise order of some tokens mattered less than expected. SHAP analysis, repeated 30 times to reduce the randomness of its estimates, identified tokens that consistently supported origin or non-origin predictions. The team combined these findings into an AT-index incorporating overall adenine-thymine content, frequencies of AT-rich motifs, the longest uninterrupted AT run, and alternating AT runs. The index provided a compact way to quantify the AT-rich character of sequences associated with the model’s decisions, but the authors caution that SHAP values describe model behavior relative to its training background, not universal causal rules governing DNA replication.

The study also used shuffled sequences to test whether the models depended mainly on nucleotide composition or on the arrangement of bases. In one control, nucleotides within each positive sequence were randomly rearranged, preserving overall composition but destroying sequence order. In another, the sequence was divided into five-base-pair blocks and the blocks were shuffled, preserving local patterns while disrupting larger-scale organization. These experiments were designed to reveal whether the classifiers had learned broad compositional differences between origins and non-origins or more structured sequence information. The models’ near-random performance on shuffled data before task-specific fine-tuning showed that pretraining alone did not automatically confer the ability to distinguish intact from rearranged origin sequences. That capability emerged during adaptation to the replication-origin task.

Training from scratch produced weaker results than fine-tuning the pretrained systems. On the Random-Neg dataset, DNABERT trained from scratch averaged 0.71 accuracy, while DNABERT-2 averaged 0.60. The pretrained models used without fine-tuning performed at 0.37 and 0.55 accuracy, respectively, showing that general DNA representations alone were insufficient. Fine-tuning was therefore essential, but pretraining still supplied a useful starting point. The researchers emphasize that their goal was not to declare one architecture the universal winner. Instead, they wanted to determine whether genomic language models could reduce manual feature engineering while producing biologically interpretable signals. On that measure, the two systems behaved differently: overlapping k-mers gave DNABERT attention maps that more visibly reflected known motifs, whereas BPE tokenization in DNABERT-2 produced less transparent attention but supported alternative attribution analyses.

The findings matter because replication origins are not simply strings containing one magic sequence. In yeast, the ACS is an important anchor for ORC recognition, yet thousands of similar matches fail to initiate replication. A useful predictive model must therefore identify combinations of sequence properties while avoiding shortcuts created by biased negative examples. By deliberately challenging the models with ACS-containing non-origins, the researchers showed that transformer classifiers can recover signals beyond the canonical motif, though not perfectly. The work also illustrates a broader principle for biological AI: predictive accuracy is only part of the result. Tokenization, dataset construction, and explanation method can substantially influence what a model appears to have learned. As genomic language models are applied to human and other eukaryotic genomes—where replication origins are less sharply defined—the ability to distinguish meaningful biological signals from statistical artifacts will be crucial.