Source-linked AI summary
Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, Irwin King, Yu Li
TL;DR
RNA-FM addresses the limited annotated data and 3D structures available for RNA structure and function prediction by pre-training on 23 million unannotated ncRNA sequences. Its self-supervised representations encode sequential and evolutionary information and improve multiple structural and functional downstream applications. The authors report stronger gains for structural tasks than for functional tasks.
Problem
Existing RNA computational approaches rely heavily on limited labeled data, while experimentally determined RNA 3D structures are scarce, restricting generalization across RNA types and tasks.
Method
RNA-FM is a self-supervised RNA foundation model pre-trained on 23 million ncRNA sequences to generate transferable representations for structural and functional downstream applications.
Results
RNA-FM representations substantially improve multiple RNA structure and function prediction applications and generalize to mRNA UTRs and SARS-CoV-2-related tasks.
Takeaways & Limitations
RNA-FM provides a reusable representation model for RNA applications while implicitly encoding sequential and evolutionary information from unannotated sequences.
Takeaways & Limitations
Improvements on functional tasks are more slight than gains on structural tasks, possibly because functional applications differ in sequence distribution and structure–function relations are complex.
Abstract
from arXiv · showhide
Non-coding RNA structure and function are essential to understanding various biological processes, such as cell signaling, gene expression, and post-transcriptional regulations. These are all among the core problems in the RNA field. With the rapid growth of sequencing technology, we have accumulated a massive amount of unannotated RNA sequences. On the other hand, expensive experimental observatory results in only limited numbers of annotated data and 3D structures. Hence, it is still challenging to design computational methods for predicting their structures and functions. The lack of annotated data and systematic study causes inferior performance. To resolve the issue, we propose a novel RNA foundation model (RNA-FM) to take advantage of all the 23 million non-coding RNA sequences through self-supervised learning. Within this approach, we discover that the pre-trained RNA-FM could infer sequential and evolutionary information of non-coding RNAs without using any labels. Furthermore, we demonstrate RNA-FM's effectiveness by applying it to the downstream secondary/3D structure prediction, SARS-CoV-2 genome structure and evolution prediction, protein-RNA binding preference modeling, and gene expression regulation modeling. The comprehensive experiments show that the proposed method improves the RNA structural and functional modelling results significantly and consistently. Despite only being trained with unlabelled data, RNA-FM can serve as the foundational model for the field.
Introduction
RNA-FM addresses the limited annotation, structure data, and cross-task generalization of existing RNA models by learning representations from 23 million unannotated ncRNA sequences. The resulting model is presented as a reusable foundation for structural and functional RNA applications.
- Motivation and limitations: RNA 3D structure modeling remains under-explored because experimentally determined 3D structures are scarce.Existing approaches optimize structures or predict intermediate 2D-related information, but no end-to-end deep-learning method directly generates RNA 3D structures.
- Motivation and limitations: Existing RNA methods rely heavily on labels, often use at most 30K annotated sequences, and generalize poorly across unknown RNA types or related tasks.E2Efold performs well on dominant RNA types but degrades on unknown types, while RNA–protein interaction models do not accurately predict UTR regulation.
- RNA-FM approach: RNA-FM is proposed as a self-supervised RNA foundation model trained on 23 million ncRNA sequences to learn transferable sequence representations.Its pre-training is task-agnostic; downstream tasks use generated embeddings or lightweight task-specific fine-tuning.
- RNA-FM approach: RNA-FM produces interpretable representations containing evolutionary information that can support inference of evolutionary trends in lncRNAs and SARS-CoV-2 variants.The model is intended to provide rich representations for the ncRNA universe, with a web server and released code and weights.
Results
RNA-FM embeddings encode structural, functional, and evolutionary information from RNA sequences and improve downstream structure-prediction performance. Across secondary-structure and 3D distance tasks, RNA-FM features consistently outperform conventional sequence, secondary-structure, and alignment-derived inputs.
- Interpretability: RNA-FM embeddings organize RNAs according to structural and functional similarity rather than sequence length, while also recovering a roughly accurate lncRNA evolutionary trend.The evolutionary trend emerges despite training on pure RNA sequences without explicit evolutionary features.
- Secondary structure: RNA-FM outperforms competing approaches on nearly all secondary-structure metrics across ArchiveII600 and TS0, exceeding SPOT-RNA by 22.8 and 7.5 points, respectively.It also exceeds UFold by 3.4 and 4.0 points on those datasets, respectively.
- Secondary structure: 85.5% of ArchiveII600 instances favor or match RNA-FM over UFold by F1 score, with especially strong performance on sequences longer than 150 nucleotides.RNA-FM probability maps are described as more robust, less noisy, and closer to ground truth than UFold maps.
- 3D structure: RNA-FM transfer learning matches or exceeds MSA covariance plus secondary-structure features on 77.5% of RNAcontact TE80 instances for long-range Top-L precision.The embedding is derived from pure sequences, avoiding the time-consuming multiple-sequence-alignment step.
- 3D structure: RNA-FM+Seq achieves lower distance-map MSE than SS+Seq on 94.2% of TE80 instances and outperforms it across all sequence lengths.The embedding enables downstream models to capture distance-map details that sequence and predicted secondary structure alone do not capture sufficiently.
Discussion
RNA-FM uses 23 million unannotated RNA sequences to support structural and functional downstream applications. Its gains are stronger in structural tasks than functional tasks, where sequence-distribution differences and complex structure–function relationships remain challenges.
- RNA-FM is trained self-supervised on 23 million RNA sequences and can support structural and functional downstream applications.
- RNA-FM embeddings improve performance across structural tasks ranging from simple to complex.
- Fine-tuning RNA-FM with downstream modules is more effective for large complex tasks, whereas transfer learning helps avoid over-fitting on small simple datasets.
- RNA-FM's functional-task improvement is more slight than its structural-task gain.
- Sequence-distribution differences and the complex relationship between RNA structure and function may limit functional-task improvements.
Methods
The methods build RNA-FM from large-scale unannotated ncRNA data, then apply its representations through feature-based training or fine-tuning across structure and function tasks.
- Model overview: RNA-FM provides a unified foundation model for downstream RNA applications by replacing task-specific prediction layers.The approach is designed as “ONE-FOR-ALL” and targets both structure-related and function-related applications.
- Pre-training data: 23.7 million ncRNA sequences from RNAcentral are collected into RNAcentral100 for self-supervised pre-training.The dataset is produced after preprocessing the large RNAcentral collection.
- Model architecture: The model uses 12 BERT-style transformer encoder blocks with 640-dimensional hidden representations and 20-head self-attention.For a sequence of length L, RNA-FM produces an L × 640 embedding matrix.
- Self-supervised training: Masked language modeling randomly masks about 15% of nucleotide tokens and trains RNA-FM to predict the original masked nucleotides.The objective minimizes the negative log-likelihood of each true nucleotide given the masked sequence context.
- Representation analysis: RNA-FM embeddings are analyzed with RNA Atlas using 640-dimensional sequence-level vectors, UMAP projections, and VIA trajectory inference.Sequence-level vectors are obtained by averaging position-wise embeddings across each RNA sequence.
- Downstream modeling: Downstream tasks use either frozen RNA-FM features or joint fine-tuning, with a shared 2D ResNet32 module for several structure-prediction tasks.The structure applications include secondary structure, contact-map, and distance-map prediction, while other tasks include RNA 3D and function modeling.
Appendix
The appendix provides additional comparisons showing RNA-FM’s performance and prediction quality against alternative approaches across secondary-structure and 3D-closeness tasks.
- Secondary structure: RNA-FM beats UFold on nearly all ArchiveII secondary-structure instances in cross-dataset validation without retraining.The comparison covers nine RNA types using scatter plots of F1 scores.
- Secondary structure: RNA-FM produces less noisy probability maps and secondary structures closer to ground truth than UFold in two examples.The appendix compares both methods against ground-truth probability maps and graph views.
- 3D closeness: The RNA 3D-closeness appendix compares MCC scatter plots for sequence, covariance, predicted-secondary-structure, RNA-FM, and combined features.ResNet is used to reproduce RNAcontact results for equal comparison across feature types.
GGACGCGGAGUAGCCCGUGUGAGCCCCGUUAGGGGCUUGAAGAGGCUGC
The appendix illustrates RNA 3D reconstruction using probability and binary maps, graph views, and 3dRNA models for representative structures.
- 3D reconstruction: 3D structures are modeled with 3dRNA from predictor-generated probability and binary maps.The figure includes examples from PDB entry 5m73-1-A and the Zika Virus DCS-PK.