RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

Rafael Josip Penić; Tin Vlašić; Roland G. Huber; Yue Wan; Mile Šikić

Source-linked AI summary

RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

Rafael Josip Penić, Tin Vlašić, Roland G. Huber, Yue Wan, Mile Šikić

arXiv:2403.00043v2Feb 29, 2024q-bio.BMcs.LG

TL;DR

RNA structure and function are important for biology and drug targeting, yet RNA language modeling is constrained by limited data and difficult structure-related problems. The paper introduces RiNALMo, a 650M-parameter model pretrained on 36M ncRNA sequences, and reports strong downstream performance, including generalization to unseen RNA families in secondary-structure prediction.

Problem
RNA structure and function matter for biological processes and drug targeting, but RNA language modeling has less data and structures available and faces more difficult related problems.
Method
RiNALMo is a 650M-parameter BERT-style Transformer pretrained with masked language modeling on 36M curated ncRNA sequences from multiple databases.
Results
RiNALMo improves performance across several structural and functional downstream tasks and generalizes to unseen RNA families in secondary-structure prediction, unlike other deep-learning methods.
Takeaways & Limitations
RiNALMo provides a general-purpose RNA sequence representation whose reported value is especially notable for inter-family secondary-structure prediction.
Takeaways & Limitations
The authors identify future expansion to coding RNAs, multimodal data, tertiary-structure prediction, and larger models as open directions.

Abstract

from arXiv · show

While RNA has recently been recognized as an interesting small-molecule drug target, many challenges remain to be addressed before we take full advantage of it. This emphasizes the necessity to improve our understanding of its structures and functions. Over the years, sequencing technologies have produced an enormous amount of unlabeled RNA data, which hides a huge potential. Motivated by the successes of protein language models, we introduce RiboNucleic Acid Language Model (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA language model to date, with 650M parameters pre-trained on 36M non-coding RNA sequences from several databases. It can extract hidden knowledge and capture the underlying structure information implicitly embedded within the RNA sequences. RiNALMo achieves state-of-the-art results on several downstream tasks. Notably, we show that its generalization capabilities overcome the inability of other deep learning methods for secondary structure prediction to generalize on unseen RNA families.

1 Introduction

RNA language modeling remains less developed than protein language modeling because RNA data and structures are scarcer and related problems are more difficult. RiNALMo addresses this gap with a large general-purpose model trained on curated unlabeled ncRNA sequences and designed to improve downstream structural and functional prediction, especially across unseen RNA families.

RNA structure and function are important for biological processes and drug targeting, but RNA language modeling has received less attention because relevant data and structures are scarcer and problems are more difficult.
Existing general-purpose RNA foundation models include RNA-FM and Uni-RNA, while other models target narrower applications such as secondary structure prediction, splicing, or cell-type annotation.
RiNALMo is a 650M-parameter RNA language model pre-trained on 36M curated ncRNA sequences from RNAcentral and other RNA databases.It uses a BERT-style Transformer with RoPE, SwiGLU, and FlashAttention-2.
RiNALMo’s representations improve performance on several structural and functional downstream tasks compared with other RNA language models and state-of-the-art methods.The paper reports that extensive experiments show improvements on most datasets.
RiNALMo generalizes to secondary-structure prediction for RNA families absent from training, addressing a limitation reported for other deep-learning methods.

2 Results

RiNALMo is a large RNA language model pretrained with masked language modeling and adapted through embeddings for structural and functional prediction tasks. Its representations cluster RNAs by family, while fine-tuned models achieve strong results on intra-family structure, inter-family structure generalization, splice-site, and MRL prediction.

General-purpose RNA language model: RiNALMo is pretrained by reconstructing 15% randomly masked nucleotide tokens in unlabeled RNA sequences, then provides embeddings for downstream prediction tasks.The model uses a Transformer encoder and masked language modeling, with embeddings reused by task-specific prediction heads.
General-purpose RNA language model: RiNALMo embeddings cluster RNAs by family in t-SNE space, consistent with structural information being represented in the embedding space.The paper links family-level clustering to the observation that RNAs from the same families fold similarly.
Intra-family secondary structure prediction: RiNALMo outperforms other state-of-the-art deep-learning approaches in precision, recall, and F1 on the TS0 intra-family secondary-structure evaluation.The structure pipeline classifies nucleotide pairs as paired or unpaired using RiNALMo-derived pair representations and a convolutional ResNet.
Inter-family secondary structure prediction: RiNALMo outperforms RNAstructure and CONTRAfold in eight of nine held-out RNA families, but struggles on telomerase RNAs.The inter-family evaluation holds out each of nine families in turn; RiNALMo achieves the highest F1 on the other families, while the authors do not determine why telomerase performance is weaker.
Splice-site prediction: Fine-tuned RiNALMo outperforms other models on splice-site prediction, including SpliceBERT despite SpliceBERT’s pretraining exclusively on pre-mRNA sequences.The model is separately fine-tuned for donor and acceptor splice-site prediction and compared with RNA language models and established splice-site methods.
Mean ribosome loading prediction: Fine-tuned RiNALMo outperforms other compared models on MRL prediction and generalizes to human UTRs after fine-tuning only on sequences of random origin.MRL prediction uses 5′ UTR sequences and R2 as the evaluation metric.

3 Discussion

RiNALMo is a general-purpose RNA language model whose representations capture structural and functional information from unlabeled sequences. Its strongest result is generalization to RNA families unseen during training, where it outperforms thermodynamics-based and deep learning methods.

3 Discussion: RiNALMo’s pre-trained representations capture RNA family and structural information, supporting strong downstream prediction performance.The authors assessed representation expressiveness through downstream tasks and family-oriented embedding analysis.
3 Discussion: RiNALMo outperformed thermodynamics-based and deep learning methods on inter-family secondary structure prediction, generalizing to unseen RNA families.This addresses the reported difficulty of deep learning methods generalizing across RNA families.
3 Discussion: RiNALMo also generalized on function-related tasks involving mRNA examples absent from pre-training, capturing functional information from previously unseen RNA types.
3 Discussion: Future work will test coding-RNA augmentation, larger or multimodal models, tertiary structure prediction, and RNA design applications.The authors specifically identify coding RNAs, chemical mapping data, tertiary structure, and sequence conditioning as directions.
3 Discussion: The study concludes that RiNALMo captures hidden structural knowledge and generalizes across unseen RNA families unlike other deep learning methods.

4 Methods

RiNALMo is a 650M-parameter encoder-only Transformer trained with masked language modeling on curated RNA sequences. Its methods combine sequence clustering, modern Transformer components, size comparisons, and task-specific fine-tuning pipelines.

4 Methods: RiNALMo is an encoder-only Transformer with 33 blocks, RoPE, 20-head attention, FlashAttention-2, SwiGLU, residual connections, and 650M parameters.The model uses 1280-dimensional embeddings and feed-forward layers with hidden size d_ff = 3413.
4 Methods: The pre-training corpus contains 36M unique ncRNA sequences clustered into 17M clusters to promote sequence diversity during training.Sequences were collected from RNAcentral, nt, Rfam, and Ensembl, filtered by length, deduplicated, and clustered.
4 Methods: Masked language modeling corrupts 15% of sequence tokens, then trains RiNALMo to reconstruct the original nucleotides.The supplied training description specifies masked-token reconstruction and optimization over sampled masked positions.
4 Methods: Model-size experiments compared RiNALMo-650M, RiNALMo-150M, and RiNALMo-33M using validation perplexity on a random holdout.Increasing model size decreased perplexity, indicating better masked-token reconstruction; the authors also link model size to downstream performance.
4 Methods: For secondary structure prediction, RiNALMo representations feed a residual prediction head that classifies nucleotide pairs and outputs pairing-probability logits.A greedy post-processing procedure selects high-probability non-conflicting canonical pairs, while training uses symmetry to score one matrix triangle.
4 Methods: Structure evaluation uses F1 scores that count pairings offset by one nucleotide as correct, reflecting tolerance for structurally close predictions.The fine-tuning schedule gradually unfreezes RiNALMo layers over 15 epochs.

5 Data availability

The paper provides downloadable datasets for pre-training-related RNA sequences and multiple downstream evaluation tasks. These resources cover sequence collections, secondary structure splits, splice-site data, and ribosome-loading measurements.

5 Data availability: RNAcentral, nt, Rfam, and Ensembl sequence datasets are identified as sources of unannotated RNA sequences.
5 Data availability: Intra-family and inter-family secondary structure datasets, including nine inter-family splits, are available for download.
5 Data availability: The multi-species splice-site dataset is available through the listed GitLab and Zenodo resources.
5 Data availability: The mean ribosome loading dataset is available through the cited NCBI GEO download.

6 Code availability

The authors release RiNALMo’s code, pre-trained and fine-tuned weights, and scripts for downstream-task training and data preparation.

6 Code availability: The code repository provides scripts to download and preprocess downstream-task data and fine-tune RiNALMo.
6 Code availability: Pre-trained and fine-tuned model weights are available for automatic download from the cited Zenodo resource.
6 Code availability: The implementation is hosted in the authors’ GitHub repository.

Additional information

The paper provides extended data and supplementary information alongside the main text.

Extended data are available in the appendix.
This section directs readers to additional materials beyond the main paper.
The paper contains supplementary material.

Appendix

The appendix contains supplementary materials for RiNALMo, including secondary-structure examples and language-model training diagnostics.

Appendix: Extended Data Fig. 1 shows secondary-structure prediction examples for the inter-family dataset.It displays target structures and predictions by RNAstructure and RiNALMo, with F1 scores shown for each predicted structure.
Appendix: Extended Data Fig. 2 presents validation perplexity curves for RiNALMo models ranging from 7.5M to 650M parameters.The models were trained for six epochs, corresponding to approximately 77,000 steps.
Appendix: The supplementary materials include the RiNALMo paper and associated author and affiliation information.The materials identify the work as arXiv:2403.00043v2, dated 12 November 2024.

S1 RNA Language Model

RiNALMo uses self-attention and multiple model configurations, with rotary positional embedding replacing traditional absolute positional encoding.

S1 RNA Language Model: RiNALMo configurations include models with 33.5M, 148M, and 650M parameters.These configurations are listed in Table S1.
S1 RNA Language Model: The model uses self-attention to identify and weigh the importance of different parts of an input RNA sequence.
S1 RNA Language Model: RiNALMo replaces traditional absolute positional encoding with rotary positional embedding for more robust and scalable positional information.

S2 Secondary Structure Prediction

The supplementary evaluation examines sequence-length-weighted secondary-structure performance and documents the unavailable Uni-RNA comparison.

S2 Secondary Structure Prediction: RiNALMo’s performance drops only on longer SRP RNAs while holding up on longer sequences overall.The comparison uses weighted and non-weighted F1 scores for each RNA family during inter-family generalization evaluation.
S2 Secondary Structure Prediction: Table S2 reports sequence-length-weighted F1 scores for inter-family secondary-structure generalization.
S2 Secondary Structure Prediction: The study omits direct Uni-RNA comparison because the model is unavailable and its reported datasets raise unresolved data-leakage concerns.The authors note that 79 test sequences, or 6%, may overlap with training data if no filtering was applied.

S3 Ablation Study

Ablations show that RiNALMo’s larger scale and architectural choices improve downstream performance, while pre-training alone captures substantial structural information and fine-tuning remains beneficial for several tasks.

Model size: RiNALMo outperforms RiNALMo-150M, RNA-FM, and RiNALMo-33M across all evaluated downstream tasks, supporting the benefit of the larger model.Evaluations cover secondary structure, multi-species splice-site, and mean ribosome load prediction tasks.
Model size: RiNALMo-33M matches or slightly exceeds RNA-FM despite being three times smaller, indicating that architectural improvements contribute beyond model scale.RiNALMo uses a more sophisticated Transformer architecture with RoPE and SwiGLU components absent from RNA-FM.
Fine-tuning: Frozen RiNALMo is often comparable to fine-tuned RiNALMo for inter-family secondary structure prediction but performs worse on 23S rRNA, 16S rRNA, and function prediction tasks.Fine-tuning improves inter-family performance further and is particularly important for intra-family structure and function prediction.
Architectural choices: Replacing sinusoidal positional embeddings and GELU with RoPE and SwiGLU progressively boosts secondary structure prediction performance.The architectural ablation used RiNALMo-33M for efficiency and computational-cost reasons.