RNA Foundation Models Improve Non-Coding RNA Annotation

Researchers published a comprehensive review in BMC Artificial Intelligence examining how RNA sequence foundation models are advancing the annotation of non-coding RNAs. The study, authored by Naima Vahab and Sonika Tyagi, analyzes the current landscape of these large-scale pre-trained deep learning models, which are designed to learn generalizable representations from vast collections of biological sequence data. By leveraging widely available unannotated sequence data, these models aim to improve upon the generalisability and accuracy of earlier thermodynamic and deep learning techniques used for RNA structure and function prediction.

Non-coding RNAs constitute the majority of the expressed transcriptome, with only less than 2% of the human genome encoding proteins. These molecules, including microRNAs, small interfering RNAs, and long non-coding RNAs, play critical roles in regulating gene expression and contributing to disease mechanisms. Their functions are primarily driven by complex secondary and tertiary structures, making accurate prediction of these structural features essential for understanding biological processes and designing new therapies. However, experimental determination of RNA structures remains costly and labour-intensive, prompting the development of computational approaches.

Traditional computational methods for RNA structure prediction have relied on thermodynamic models, alignment-based techniques, and deep learning architectures. While deep learning models have significantly improved accuracy, they often face generalisation issues when applied to unseen RNA families. For instance, models trained on common RNA types like tRNAs may fail when applied to unfamiliar sequences. The core limitation of these traditional approaches is their reliance on small labelled datasets, which largely ignores the vast pool of unlabelled RNA data available in biological repositories.

Foundation models address this limitation by using self-supervised learning to generate labels from unannotated data, allowing for powerful pretraining that can be fine-tuned on limited high-quality labelled data. This approach reduces reliance on task-specific models by facilitating the modification of the final prediction module depending on the specific downstream task. The review highlights that these models are increasingly capable of capturing both structural and functional aspects of RNA, offering a potential “one-model-for-all” solution for diverse bioinformatics applications.

The review details several key RNA foundation models, including RNA-FM, Uni-RNA, RiNALMo, and AIDO.RNA. RNA-FM, one of the first specialised foundation models for RNA, was built upon 23 million non-coding RNA sequences from the RNACentral database. It consists of 12 transformer-based bidirectional encoder blocks and has been shown to improve accuracy in downstream tasks, even on low-redundancy datasets. Uni-RNA, developed by Wang et al., comprises a set of four foundation models with parameters ranging from 25 million to 400 million, trained on one billion RNA sequences. Despite achieving an 18.3% improvement in F1-score on downstream tasks, its performance plateaus once the model size reaches 400 million parameters.

RiNALMo, a model with 650 million parameters, is primarily trained on 36 million non-coding RNA sequences from the RNACentral database, along with data from Rfam, nt, and Ensembl databases. It has demonstrated better generalising capability in 2D structure prediction tasks on unseen families, a critical factor for robust model performance. AIDO.RNA stands out as the largest RNA model to date, with 1.6 billion parameters. It is an encoder-only transformer model trained on 42 million non-coding RNA sequences at single-nucleotide resolution using masked language modelling as the pre-training objective. AIDO.RNA incorporates advanced architectural features such as rotary position embedding and the SwiGLU activation function to enhance positional modelling and overall performance.

Architectural diversity is a key theme in the development of RNA foundation models. While most models utilize transformer architectures, newer approaches are exploring alternative techniques to address computational challenges. LoRNASH, for example, is built on the StripedHyena model, which integrates data-controlled convolutional operators with multi-head attention layers. This architecture enables LoRNASH to process sequences of up to 64,000 nucleotides, facilitating comprehensive zero-shot predictions of transcriptome architecture. DGRNA, another recent model, is built on the Mamba-2 architecture and was pre-trained on approximately 100 million RNA sequences derived from the MARS database, making it one of the largest models specifically designed for RNA data.

Functional annotation is another critical application of RNA foundation models. These models have expanded their scope to predict various RNA functional properties beyond 2D structure, including cross-species splice site prediction, protein-RNA interaction prediction, and mean ribosome load estimation. The use of large-scale embeddings has enabled these models to capture richer and more diverse features of RNA, thereby enhancing their generalisability and improving overall predictive performance. For instance, features learned in foundation models have been shown to perform comparably to experimentally derived 2D structure features in predicting protein-RNA interactions, despite being trained solely on unannotated sequences.

Despite these advancements, the review identifies several significant challenges facing the field. Data constraints remain a major hurdle, as the Protein Data Bank contains only a few thousand RNA structures, with protein structures outnumbering RNA structures by a factor of 25. Moreover, available RNA data is heavily skewed, with tRNA and rRNA making up over 50% of entries in databases like bpRNA. This bias undermines the generalisability of machine learning models for other sub-categories of non-coding RNA. Functional annotations are also sparse, and widely used resources like Rfam are known to contain structural errors and systematic biases.

Pretraining objectives and architectural innovation also present ongoing challenges. Most RNA foundation models use BERT-style transformer encoders with masked language modelling, relying on standard pretraining methods without leveraging recent advances in dynamic masking or span masking. Architectural innovation remains limited, with most progress focused on variations in pretraining datasets and fine-tuning of transformer hyperparameters. Furthermore, transformer-based architectures pose significant computational challenges due to their quadratic time complexity, leading to high memory consumption and hardware demands. For example, models like RiNALMo and RNA-FM required several high-end GPUs with 80GB memory each and long training times.

Model generalisation and explainability are critical areas for future development. Ensuring robust generalisation beyond the pretraining and fine-tuning datasets is essential, particularly in biological domains where data may be limited or highly structured within specific families. Evaluating models on unseen RNA families is crucial for assessing true generalisability. Additionally, the interpretability and explainability of foundation models remain significant hurdles. Understanding the basis for predictions is critical in biological applications, and more advanced interpretability techniques are needed to move beyond the perception of foundation models as black boxes.

The practical utility of RNA foundation models in bioinformatics workflows is also underdeveloped. Many models lack intuitive application programming interfaces, user-friendly web interfaces, or sufficient documentation, which hinders adoption among researchers who may not have a background in machine learning. Bridging this usability gap will be crucial for democratising RNA foundation model technologies and enabling their broader application across the life sciences. The review concludes that while challenges remain, ongoing improvements in foundation model-based RNA modelling are expected to lead to important breakthroughs in biomedical research and clinical applications.

Subject of Research: Application of RNA sequence foundation models for non-coding RNA structure and function annotation

Article Title: Advancing non-coding RNA annotation with RNA sequence foundation models: structure and function perspectives

Article References: Vahab, N., & Tyagi, S. (2025). Advancing non-coding RNA annotation with RNA sequence foundation models: structure and function perspectives. BMC Artificial Intelligence, 1(1), Article 12. https://doi.org/10.1186/s44398-025-00012-7

Image Credits: AI Generated

DOI: 10.1186/s44398-025-00012-7

Keywords: RNA foundation models, non-coding RNA, structure prediction, functional annotation, deep learning, bioinformatics, machine learning, transcriptome, computational biology, RNA sequencing, Advancing, non-coding