RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Ziyuan Wang; Bohao Tang; Fei Zhang; Shuo Han; Pengfei Liu

Source-linked AI summary

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Ziyuan Wang, Bohao Tang, Fei Zhang, Shuo Han, Pengfei Liu

arXiv:2608.22849v1Aug 24, 2026cs.LGq-bio.GN

TL;DR

Existing RNA foundation models often cannot model complete long transcripts at single-nucleotide resolution. RIBOSPAN addresses this with a 1.61B-parameter bidirectional model natively pretrained to 10,240 nt, achieving strong long-context representation results and supporting full-length mRNA generation and redesign, while its broader biological design and reinforcement-learning evaluation remain future work.

Problem
Existing RNA foundation models use context lengths that can limit complete-transcript modeling at single-nucleotide resolution, especially for long mRNAs.
Method
RIBOSPAN combines single-nucleotide tokenization, dense bidirectional self-attention, native 10,240-nt pretraining, and a conditional discrete-diffusion framework for full-length mRNA design.
Results
Native 10K pretraining provides clear advantages beyond short-context extrapolation, while 40% masking improves recovery under heavy corruption and preserves representation quality.
Takeaways & Limitations
RIBOSPAN provides a long-context foundation for transferable RNA representation learning and full-transcript mRNA design.
Takeaways & Limitations
The complete mRNA design and reinforcement-learning post-training framework, biological evaluation, and downstream applications are deferred to a forthcoming publication.

Abstract

from arXiv · show

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.

1 Introduction

RIBOSPAN addresses the unresolved challenge of modeling complete, long mRNAs at single-nucleotide resolution while preserving bidirectional context. The paper also introduces a benchmark for long-context representations and a full-transcript mRNA design framework.

1 Introduction: mRNA regions have coupled effects on translation, folding, stability, localization, and molecular interactions, making transcript-wide design subject to multiple simultaneous constraints.
1 Introduction: Existing methods typically constrain either the sequence region or optimization objective, leaving complete-transcript representation learning less systematically explored.
1 Introduction: Existing long-RNA models have not simultaneously combined single-nucleotide tokenization, dense bidirectional self-attention, and billion-scale capacity at transcript-representative context lengths.
1 Introduction: RIBOSPAN is introduced as a billion-scale dense bidirectional Transformer with single-nucleotide tokenization and native pretraining up to 10,240 nt.
1 Introduction: The paper contributes a benchmark that evaluates long-range information integration and contextual representation quality as sequence length extends beyond pretraining context.
1 Introduction: A full-stack framework jointly generates and redesigns full-length mRNAs across the 5′ UTR, CDS, and 3′ UTR using multidimensional conditioning and synonymous-codon diffusion.

2 Native Long-Context Pretraining

RIBOSPAN uses a 1.61B-parameter, nucleotide-level bidirectional Transformer trained natively on 10,240-token contexts. Its corpus spans diverse RNA sequences, and its schedule includes continued high-masking pretraining for reconstruction-oriented adaptation.

2.1 Model Architecture and Tokenization: RIBOSPAN uses a 1.61B-parameter bidirectional Transformer encoder with single-nucleotide tokenization and native 10,240-token pretraining.
2.1 Model Architecture and Tokenization: Each nucleotide occupies one token position, preserving nucleotide-level resolution for long-RNA modeling.
2.2 Pretraining Corpus: The pretraining corpus contains 67.6M RNA sequences and 85.7B nucleotide tokens assembled from RNAcentral, Ensembl, and Ensembl Genomes.
2.2 Pretraining Corpus: The corpus provides broad RNA-class coverage, while Ensembl contributes protein-coding transcripts with complete CDS and UTR annotations across multiple organism groups.
2.3 Long-Context Pretraining Schedule: Pretraining begins with 15% masking and continues with 40% masking for reconstruction-oriented adaptation.
2.3 Long-Context Pretraining Schedule: The study trains 1,024-token baseline models with the same corpus, architecture, and masking schedule to assess native-context effects.
2.3 Long-Context Pretraining Schedule: Sequence packing retains independent RNA boundaries, restricts attention within each RNA, resets positional indices, and prevents cross-sequence information leakage.

3 Experiments

Experiments show that native 10K pretraining preserves reconstruction and representation quality on long RNAs, while 40% masking improves robustness to corruption. Long-context tests further distinguish native pretraining from extrapolation and efficient architectures by their trade-offs between contextual differentiation and distal propagation.

3.1 mRNA Reconstruction Evaluation: Native 10K variants retain strong reconstruction at 10,240 tokens, whereas direct extrapolation of short-context models degrades substantially.The evaluation uses mRNA sequences under 15% and 40% masking; native 10K models remain within their pretrained context.
3.1 mRNA Reconstruction Evaluation: 40% masking continuation improves recovery under heavy corruption while changing performance little under 15% masking.Global reconstruction accuracy shows this improvement at both evaluated context lengths, with RIBOSPAN-10K-40 closely matching 10K-15 under 15% masking.
3.2.3 Long-Context Representation Analysis: The benchmark measures contextual responsiveness, region-specific representation organization, and the spatial extent of perturbation-induced changes across complete mRNAs.It compares localized composition-preserving rearrangements with surrounding and distant unchanged regions across transcript lengths from 1,024 to 10,240 nt.
3.2.3 Long-Context Representation Analysis: Direct extrapolation reduces contextual organization, while YaRN restores ∆CS and Ccross but markedly increases Ddistal.The result indicates that positional extension recovers context-dependent interactions without calibrating their propagation over transcript-scale distances.
3.2.3 Long-Context Representation Analysis: HydraRNA shows very small Ddistal but substantially lower ∆CS and higher Ccross than native 10K RIBOSPAN, revealing a propagation–integration trade-off.Its bidirectional state-space-dominated architecture constrains propagation while potentially limiting flexible long-range contextual integration.
3.2.3 Long-Context Representation Analysis: Native 10K RIBOSPAN combines interaction flexibility with long-range calibration, and 10K-40 preserves the representation profile of native 10K pretraining.The 10K-15 and 10K-40 checkpoints remain closely aligned in ∆CS and Ccross and maintain very low Ddistal.
3.3.3 RNA Type Separability: RIBOSPAN-10K-15 achieves the highest Overall Biotype accuracy and neighborhood purity, while 10K-40 remains closely matched.The close agreement indicates that reconstruction-oriented 40% masking continuation preserves overall RNA-type representation quality.
3.3.3 RNA Type Separability: RIBOSPAN-10K achieves the strongest frozen-representation performance on long RNAs, with the highest accuracy and neighborhood purity for sequences longer than 1,024 nt.The checkpoints show strongest neighborhood purity in six of eight long-RNA classes, including mRNA, lncRNA, miscRNA, and sRNA.

4 Full-Length mRNA Generation Framework

The framework extends RIBOSPAN into conditional discrete diffusion for full-length mRNA generation and redesign, jointly modeling transcript regions under multidimensional conditions. It supports bidirectional denoising across complete transcripts and synonymous-codon optimization for protein-preserving CDS redesign.

Framework overview: The framework uses conditional discrete diffusion to jointly generate and redesign full-length mRNAs across the 5′ UTR, CDS, and 3′ UTR.RIBOSPAN serves as the generative backbone, with multidimensional conditioning and synonymous-codon diffusion for cross-region and protein-preserving optimization.
Denoising design: Unlike autoregressive decoding, diffusion jointly updates multiple nucleotide positions so each denoising step can integrate bidirectional context across the complete transcript.This design extends the bidirectional reconstruction capabilities learned during pretraining to generation.
Framework overview: At each diffusion step, RIBOSPAN encodes the corrupted sequence, while timestep and design conditions enter diffusion blocks through AdaLN-Zero conditioning.The modulation network produces layer-specific scale, shift, and residual-gating parameters.
Denoising design: A diffusion timestep determines corruption severity, and the corresponding position set is replaced by mask tokens to create the corrupted sequence.Training uses a conditional masked-diffusion objective with stratified timestep sampling across corruption levels.
Design constraints: Synonymous-codon diffusion restricts CDS substitutions to synonymous alternatives, while unified bidirectional context supports coordinated changes across distant mRNA regions.The broader design system also describes reinforcement-learning post-training for closed-loop optimization, but its biological evaluation is deferred to a forthcoming publication.

5 Conclusion

RIBOSPAN is a 1.61B-parameter bidirectional RNA foundation model pretrained natively to 10,240 nt with single-nucleotide tokenization and dense self-attention. Native long-context pretraining supports reconstruction, contextual representation quality, and transferable RNA modeling, while the same backbone enables full-transcript mRNA design.

Model and training: RIBOSPAN combines 1.61B parameters, single-nucleotide tokenization, dense self-attention, and native pretraining at context lengths up to 10,240 nt.The model was trained on 67.6 million RNA sequences containing 85.7 billion nucleotide tokens.
Evaluation findings: Native 10K variants retain strong reconstruction at 10,240 tokens, while 40% masking continuation improves recovery under heavy corruption and preserves representation quality.These findings span reconstruction and representation evaluations.
Evaluation findings: Native 10K pretraining balances contextual differentiation with controlled distal propagation more effectively than direct extrapolation, positional scaling, and the hybrid state-space/attention architecture.The conclusion summarizes this pattern across the long-context benchmark.
Evaluation findings: Frozen-representation evaluations show state-of-the-art RNA representation quality across diverse RNA types, with a clear advantage on long RNAs.The conclusion presents this as a transferable representation-learning outcome.
Design extension: The RIBOSPAN backbone also supports multidimensionally conditioned discrete diffusion for full-length mRNA generation and redesign, including protein-preserving CDS optimization.This extends the foundation model toward full-transcript mRNA design.

A.1 Data Curation and Splits

The training corpus was assembled from RNAcentral, Ensembl, and Ensembl Genomes resources, with source-specific filtering, normalization, and deduplication. The supplied passages identify appendix tables for corpus statistics and Ensembl filtering criteria.

Source curation: RNAcentral v26.0, Ensembl release 115, and Ensembl Genomes release 62 were curated independently before merging.The sources were normalized and filtered before construction of the final corpus.
Normalization and deduplication: Sequences were uppercased, uracil was mapped to thymine, unsupported symbols were mapped to N, and RNAcentral sequences were exact-deduplicated.The passage also describes combining active and inactive RNAcentral sequence sets.
Dataset documentation: Table 8 reports pretraining corpus statistics by split and source, while Table 9 reports filtering criteria for Ensembl protein-coding transcripts.The supplied passages provide the table captions but not their contents.

A.2 Tokenization and Training Sample Construction

RIBOSPAN uses single-nucleotide sequences with independent document boundaries, packed attention, and context-specific evaluation procedures. The appendix also defines long-sequence positional-scaling settings, paired interventions, representation endpoints, and statistical comparisons.

A.2 Tokenization and sample construction: Each RNA is tokenized nucleotide by nucleotide, with [CLS] and [SEP] boundary tokens counted toward the context length.RNAs exceeding the native context length are truncated to fit the corresponding window.
A.2 Tokenization and sample construction: Masked-language-model pretraining replaces selected positions with [MASK], random tokens, or unchanged tokens using an 80%/10%/10% allocation.Selected positions may form spans of up to three nucleotides, and the same procedure is used for 15% and 40% masking stages.
A.2 Tokenization and sample construction: Variable-length RNAs are greedily packed into context-bounded bins while packed attention preserves independent sequence boundaries and resets positional indices.This prevents attention leakage across packed sequences and reduces padding overhead.
A.2 Tokenization and sample construction: The 40% continuation resumes from the corresponding 15% checkpoint with a newly initialized optimizer, and Table 10 summarizes both training stages.The appendix identifies Table 10 as the pretraining hyperparameter table.
B.1 Long-sequence evaluation settings: YaRN extends short-context RoPE at inference through frequency-dependent interpolation and attention-score rescaling without additional training.Direct RoPE extrapolation instead retains the original rotary frequencies.
B.2 Benchmark construction and paired intervention: The paired intervention constructs at most 24 candidates by permuting nonempty nucleotide blocks, then selects a candidate using transition entropy, Hamming distance, and short-period repetition criteria.The selected interval replacement preserves transcript length, interval composition, and surrounding sequence context.
B.3 Evaluation measures: Representation endpoints use final-layer hidden states with boundary tokens excluded, while distal diffusion samples unchanged positions across absolute-distance ranges from [0, 8) to [4096, 10240).Sampling coordinates are reused across model configurations, and confidence intervals use bootstrap resampling.

B.5 Cross-Length Representation Results

The benchmark compares representation behavior across sequence lengths, using fixed held-out inputs and mean-pooled frozen representations. Native 10K models remain localized under distal perturbations across thresholds, whereas YaRN produces broader distal changes.

Cross-length evaluation: Table 11 compares three representation metrics from 1,024 to 8,192 nt, while 10,240-nt results are reported separately.The benchmark therefore provides both a cross-length view and a dedicated maximum-length evaluation.
Distal propagation: YaRN produces substantially broader distal representation changes, whereas HydraRNA and native 10K models remain strongly localized across normalized-distance thresholds.This pattern persists for the primary r_i ≥ 0.75 endpoint and additional thresholds of r_i ≥ 0.25 and r_i ≥ 0.50.
Statistical comparison: At 10,240 nt, paired comparisons report model differences for three primary representation endpoints using Model B minus Model A.The comparisons also report Cohen’s d_z and Benjamini-Hochberg-adjusted q-values, with transcript-level means aggregated across 10 transcript pairs.
Benchmark design: The benchmark uses 90,000 held-out sequences capped at 10,240 nt, with shorter-context models receiving corresponding prefixes.Sequence order is fixed across models, and 417 sequences exceeding the cap are represented by their first 10,240 nucleotides.
Representation extraction: Frozen sequence representations are produced by mean-pooling valid final-layer nucleotide states without feature standardization, PCA, or learned projection.Representations are evaluated in FP32 with boundary and padding tokens excluded.
Evaluation spaces: RNA-type evaluations cover overall biotypes, functional groups, regulatory labels, long RNAs, and Rfam families with separately defined sequence subsets.The Long RNA evaluation contains 17,130 sequences longer than 1,024 nt, while the Rfam evaluation retains 33,000 sequences across 20 families.

C.5 Functional Class-Level Analysis

The functional class analysis finds strong organization across coding, housekeeping, and regulatory RNAs. RIBOSPAN-10K checkpoints lead regulatory neighborhood purity and minimize coding-regulatory confusion, while HydraRNA leads coding purity.

Neighborhood purity: RIBOSPAN-10K checkpoints achieve the highest neighborhood purity for regulatory RNAs, while HydraRNA achieves the highest coding-RNA purity.Housekeeping-RNA purity remains closely matched across the strongest models.
Confusion analysis: Both RIBOSPAN-10K checkpoints yield the lowest total coding-regulatory confusion among evaluated models.Most remaining cross-class errors occur between coding and regulatory RNAs.

C.6 Long RNA Class-Level Analysis

The long-RNA class analysis shows that RIBOSPAN-10K representations separate diverse long-RNA types especially well. They lead neighborhood purity in most classes and have the lowest aggregate confusion across reported directions.

Neighborhood purity: RIBOSPAN-10K checkpoints achieve the highest neighborhood purity in six of eight long-RNA classes, including lncRNA, mRNA, miscRNA, and sRNA.The evaluation covers long sequences retained across eight RNA types.
Confusion analysis: Both RIBOSPAN-10K checkpoints yield the lowest total confusion across the four reported long-RNA error directions.HydraRNA also performs strongly in two individual directions, but RIBOSPAN shows the strongest aggregate separation.