Source-linked AI summary
Accurate RNA 3D structure prediction using a language model-based deep learning approach
Tao Shen, Zhihang Hu, Siqi Sun, Di Liu, Felix Wong, Jiuming Wang, Jiayang Chen, Yixuan Wang, Liang Hong, Jin Xiao, Liangzhen Zheng, Tejas Krishnamoorthi, Irwin King, Sheng Wang, Peng Yin, James J. Collins, Yu Li
TL;DR
RNA 3D structure prediction is limited by structural flexibility and scarce experimental data, despite its relevance to RNA biology and applications. RhoFold+ uses an RNA language model-based, fully automated deep learning pipeline for single-chain RNAs, and it outperforms existing approaches on RNA-Puzzles and CASP15 natural RNA targets while also predicting secondary structures and inter-helical angles.
Problem
RNA 3D structure prediction remains difficult because RNA flexibility contributes to scarce experimental structures, limiting understanding of RNA function and applications.
Method
RhoFold+ is a fully automated and differentiable deep learning method that predicts single-chain RNA 3D structures from sequence using an RNA language model and data-augmentation strategies.
Results
RhoFold+ outperforms existing RNA structure prediction approaches on RNA-Puzzles and CASP15 natural RNA targets, achieving a sub-4 Å mean RMSD on non-overlapping, non-redundant RNA-Puzzles structures.
Takeaways & Limitations
RhoFold+ provides a fast, efficient approach for single-chain RNA structure prediction and additionally predicts secondary structures and inter-helical angles.
Takeaways & Limitations
Prediction remains difficult for large and complex RNAs, especially sequences longer than 500 nucleotides, and for RNAs adopting multiple conformations or involving ligands or proteins.
Abstract
from arXiv · showhide
Accurate prediction of RNA three-dimensional (3D) structure remains an unsolved challenge. Determining RNA 3D structures is crucial for understanding their functions and informing RNA-targeting drug development and synthetic biology design. The structural flexibility of RNA, which leads to scarcity of experimentally determined data, complicates computational prediction efforts. Here, we present RhoFold+, an RNA language model-based deep learning method that accurately predicts 3D structures of single-chain RNAs from sequences. By integrating an RNA language model pre-trained on ~23.7 million RNA sequences and leveraging techniques to address data scarcity, RhoFold+ offers a fully automated end-to-end pipeline for RNA 3D structure prediction. Retrospective evaluations on RNA-Puzzles and CASP15 natural RNA targets demonstrate RhoFold+'s superiority over existing methods, including human expert groups. Its efficacy and generalizability are further validated through cross-family and cross-type assessments, as well as time-censored benchmarks. Additionally, RhoFold+ predicts RNA secondary structures and inter-helical angles, providing empirically verifiable features that broaden its applicability to RNA structure and function studies.
Introduction
RNA 3D structures are important for understanding RNA function and applications, but structural flexibility and scarce experimental data make prediction difficult. RhoFold+ addresses this challenge with a language model-based deep learning approach for fast, automated single-chain RNA structure prediction.
- Motivation: RNA structures inform understanding of gene regulation and function, drug development, and synthetic biology design.High-resolution structural information can support more predictive understanding of RNAs with unknown functions and structures.
- Motivation: Over 85% of the human genome is transcribed, while only 3% encodes proteins, leaving many RNAs with unknown functions and structures.
- The prediction challenge: RNA conformational flexibility makes experimental 3D structure determination difficult, and RNA-only structures comprise less than 1.0% of approximately 214,000 PDB structures.RNA-containing complexes account for only 2.1% of PDB structures as of December 2023.
- The prediction challenge: Existing computational approaches include template-based modeling constrained by limited template libraries and de novo methods using sampling or related strategies.
- RhoFold+: RhoFold+ is a language model-based deep learning method designed for accurate and fast de novo prediction of single-chain RNA 3D structures.The method is presented as a fully automated and differentiable improvement over RhoFold, integrating MSAs and other features.
Results
RhoFold+ outperformed existing methods across RNA-Puzzles, CASP15 natural targets, newly determined structures, and secondary-structure benchmarks, while generalizing across unseen sequences, RNA types, and families. Its main constraints were reduced accuracy for flexible junctions and complex RNA classes, and sensitivity to MSA quality and sequence length.
- RNA-Puzzles: 4.02 Å average RMSD and 0.57 average TM-score established RhoFold+ as the strongest RNA-Puzzles performer, surpassing the second-best model.It achieved RMSD < 5 Å on 17 of 24 targets and exceeded the best single training template by 0.05 TM-score on average.
- RNA-Puzzles: RhoFold+ showed little evidence of sequence-similarity-driven overfitting and improved on structurally dissimilar RNA-Puzzles targets.For PZ38, its prediction had 8.92 Å RMSD versus 16.46 Å for the most sequence-similar training structure.
- CASP15: RhoFold+ outperformed RhoFold by approximately 1 Å RMSD on CASP15 natural targets and surpassed the other methods evaluated across all six targets.Its performance was on par with AIchemy RNA2 by Z-score and exceeded other participating or published methods.
- Generalization: Cross-type and cross-family validation showed broad generalization, including TM-scores up to 0.73 for tRNAs and miRNAs, but weaker performance on introns, riboswitches, and complex families.The average cross-family RMSD was 6.69 Å; protein or enzyme interactions may limit accuracy for some complex RNA elements.
Discussion
RhoFold+ provides automated RNA 3D structure prediction with strong benchmark performance, but its applicability remains constrained by RNA conformational diversity, complex structures, molecular interactions, and MSA availability.
- Discussion: RhoFold+ achieves sub-4 Å mean RMSD on non-overlapping, non-redundant RNA-Puzzles structures while generalizing across training sets and newly determined structures.The model is fully automated and differentiable, uses an RNA language model trained on approximately 23.7 million sequences, and avoids structural information leakage.
- Discussion: Dynamic RNA conformations, especially at junctions, make it difficult for RhoFold+ and related models to represent multiple structures of the same molecule.RNA junctions are described as dynamic ensembles because they can adopt multiple conformations and interact with other molecules.
- Discussion: Large and complex RNAs, including molecules with multiple helices or pseudoknots and sequences longer than 500 nucleotides, remain difficult to predict because of insufficient data.
- Discussion: RNA complexes involving ligands or proteins remain challenging because current methods may not adequately account for these interactions, reducing prediction accuracy.The authors identify integration with tools such as RoseTTAFoldNA or AlphaFold3 as a possible way to improve these capabilities.
- Discussion: Training on specific environmental conditions may limit generalization to the diverse solution conditions encountered in vivo, including changing ion concentrations and ligand presence.
- Discussion: MSA-dependent methods struggle with artificially designed or orphan RNAs that lack corresponding alignments, limiting accurate prediction for these sequences.The paper proposes improving MSA extraction and combining RhoFold+ with probing, molecular dynamics, and energy-function techniques as possible mitigations.
Methods
RhoFold+ uses RNA language-model representations derived from large-scale sequence pre-training and incorporates multiple sequence alignments as evolutionary features. Its RNA-FM encoder is trained with masked language modeling to capture sequence dependencies.
- Multiple sequence alignment feature generation: RhoFold+ uses Infernal- and rMSA-constructed MSAs to capture co-evolutionary information as an additional model input.The MSAs draw on Rfam and RNAcentral sequence databases.
- Multiple sequence alignment feature generation: During inference, RhoFold+ uses 256 MSAs selected by ranking, random sampling, or clustering to generate standard or multiple predictions.Clustering can use conserved secondary structure or RNA-FM sequence embeddings.
- RNA-FM language model: RNA-FM applies a bidirectional transformer architecture followed by unsupervised training based on BERT.The framework provides sequence-derived representations for downstream tasks with insufficient annotated data.
- RNA-FM language model: RNA-FM was pre-trained on approximately 27 million ncRNA sequences collected from RNAcentral and spanning 47 databases.The dataset represents ncRNA types from a broad range of organisms.
- Masked language modeling: During pre-training, 15% of nucleotide tokens are randomly selected and replaced with mask tokens for masked language modeling.The objective predicts each masked nucleotide from the remaining sequence context using negative log-likelihood.
Efficient development of a self-distillation dataset
RhoFold+ addresses limited RNA structural data by constructing a non-redundant self-distillation dataset with experimentally derived secondary-structure labels. The filtered dataset contains 27,732 sequences of lengths between 16 and 256 nucleotides.
- Dataset construction: 27,732 sequences remained after filtering RNAStralign and bpRNA-1M data to lengths from 16 to 256 nucleotides.The dataset contains ground-truth secondary structures for self-distillation.
- Motivation: The self-distillation dataset was constructed because RNA structural data remain scarcer than protein structural data.RNA-FM was used to alleviate sequence-data scarcity, but structural-data scarcity persisted.
- Training procedure: RhoFold+ was initially trained using PDB data before incorporating the self-distillation dataset.
A structure prediction module
The structure module reconstructs RNA 3D coordinates from Rhoformer sequence and pair representations using nucleotide-centered frames, torsion angles, and invariant point attention. Iterative recycling refines predictions until the model reaches its stopping criterion.
- Structure module inputs: The structure module predicts an RNA 3D structure from sequence and pair representations extracted by Rhoformer.
- Coordinate representation: RNA full-atom coordinates are reconstructed using a C4′, N1/N9 frame and four torsion angles: α, β, γ, and ω.The method uses this representation instead of directly predicting nucleotide base frames.
- Geometric prediction: Invariant point attention predicts rotation and translation matrices for each frame from Rhoformer outputs and pair representations.
- Iterative refinement: A recycling strategy iteratively refines the predicted structure by feeding the previous prediction back into Rhoformer.The process ends when the predicted pLDDT criterion is reached.
Feature processing with Rhoformer
Rhoformer uses gated transformer modules to learn evolutionary information while jointly updating pairwise sequence embeddings and MSA representations. Transition blocks expand embedding dimensions to increase model capacity.
- Rhoformer architecture: Rhoformer comprises transformer modules with gated self-attention layers that learn evolutionary information.
- Joint feature processing: The modules simultaneously update pairwise sequence embeddings and multiple sequence alignment representations.
- Embedding expansion: Transition blocks use two linear layers to increase the embedding dimension fourfold for pair and MSA representations.The expansion is intended to increase model capacity.
The structure prediction loss
RhoFold+ trains with losses at 1D, 2D, and 3D levels to extract sequence information, learn residue-pair relationships, and impose structural constraints.
- 1D loss: At the 1D level, masked language modeling reconstructs 5% of randomly masked nucleotides to extract co-evolutionary information from MSAs.A linear projection layer reconstructs the masked nucleotides without curated correlation features.
- 2D loss: At the 2D level, distance and secondary-structure losses supervise pairwise positional correlations between RNA residues.Distances between P, C4, and N atoms are classified into 40 bins, while secondary structure is represented as an L×L binary base-pair matrix.
- 3D loss: At the 3D level, FAPE, secondary-structure constraint, and clash losses provide gradients for structure prediction and penalize geometrically implausible models.FAPE compares predicted and ground-truth atom coordinates under local frames and is independent of rigid motions.
- 3D secondary-structure constraint: Lss3d constrains distances between four fixed pseudo-atoms in base-paired residues using standard pairwise distances and a tolerance threshold.For self-distilled samples, statistical distances from PDB base-pair structures are used to reduce pseudo-label overfitting and exploit secondary-structure information.
- Overall loss: The loss function combines masked language modeling, distance, secondary-structure, clash, FAPE, 3D secondary-structure, and lDDT-evaluator losses.The stated overall loss weights these terms as L = Lmlm + 0.3 ∗Ldis + 0.1 ∗Lss + 0.03 ∗Lclash + 2 ∗LFAPE + 0.1 ∗Lss3d + 0.01 ∗LpLDDT.
Structure relaxation by force fields
RhoFold+ can apply restrained force-field minimization after prediction to resolve remaining clashes and geometric violations while staying close to the input structure.
- Post-prediction relaxation: Restrained AMBER or BRiQ minimization can relax RhoFold+ predictions to resolve remaining structural clashes and violations.The procedure uses harmonic restraints to maintain a close relationship with the input structure.
- Post-prediction relaxation: Post-prediction relaxation also enforces phosphodiester-bond geometric features.
Implementation details and running time
RhoFold+ was trained with specified optimization and hardware settings, while benchmarking compared it with several published prediction systems and CASP15 natural RNA targets.
- Training configuration: RhoFold+ used Adam at a 0.0003 learning rate for 300,000 iterations, with 10,000 warm-up steps, batch size 16, and dropout ratio 0.1.Training used 1,600 epochs and a cluster with eight NVIDIA A100 GPUs.
- Benchmarking: Benchmarking evaluated DeepFoldRNA, DRfold, RoseTTAFold2NA, FARFAR2, trRosettaRNA, and AlphaFold3 using official repositories or the AlphaFold3 server.For FARFAR2, the authors followed default settings and trained 100 models.
Data Availability
The study used public RNA structure and sequence resources for training, pre-training, multiple-sequence alignment construction, and secondary-structure self-distillation.
- RNA structure data: RNA 3D structures came from the BGSU RNA Representative Sets and were downloaded from the Protein Data Bank.
- Sequence and MSA data: RNA-FM pre-training used unannotated RNAcentral sequences, while MSA construction combined NCBI nucleotide data, Rfam, and RNAcentral with rMSA.Secondary-structure information was used for self-distillation.
Code Availability
RhoFold+ provides open-source trained weights and inference scripts, alongside an academic server. The associated RNA-FM pipeline and supporting software dependencies are also specified.
- Code Availability: RhoFold+ trained weights and inference scripts are available under an open-source license.The project provides source access through its GitHub repository.
- Code Availability: RhoFold+ is freely available as an academic server.The server is offered for academic purposes through the project website.
- Code Availability: The pre-trained RNA-FM model and inference pipeline are available through GitHub.The implementation also documents RNA MSA search using Infernal, Blastn, HMMER, and rMSA, plus OpenMM 7.7 for AMBER forcefield relaxation.
- Code Availability: The source code is written in Python 3.7 and uses specified tools for data collection, analysis, and visualization.Listed dependencies include Infernal 1.1.3, CD-HIT 4.8.1, HMMER 3.3, HH-suite 2.0.15, NumPy 1.21.2, PyTorch 1.10.2, pandas, and related packages.