PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling RNA Foundation Models through Iterative Refinement

Pengan Chen, Liang Hong, Xinyuan Liu, Han Huang, Ziqian Lin, Jiayang Chen, Yimin Fan, Jiyue Jiang, Yu Li

arXiv:2610.13106v1cs.LG

TL;DR

RNA foundation models use fixed inference computation graphs, despite RNA function depending on information distributed across sequence scales. Eco-RNA reuses an input-conditioned Transformer core across adjustable inference iterations and trains it across depths. Its refinement improves RNAGym fitness prediction and supports gains across reconstruction and downstream RNA prediction tasks.

  • Problem

    Conventional RNA foundation models apply a fixed inference computation graph, limiting flexibility to trade additional computation for better predictions despite RNA function depending on information across sequence scales.

    Source: §1 · p. 1

  • Method

    Eco-RNA is an 80M-parameter sequence model that reuses an input-conditioned Transformer core across inference-selected iterations and trains across depths with masked reconstruction, state regularization, and one-sided reconstruction anchoring.

    Source: §1 · p. 2

  • Results

    Across reconstruction and downstream RNA prediction, added depth yields gains; on RNAGym, aggregate absolute Spearman correlation rises from 0.242 to 0.286 from one to four iterations, and four-iteration Eco-RNA exceeds Evo 2 40B on all three aggregate fitness metrics.

    Source: §1 · p. 2, §4 · pp. 9–10

  • Takeaways & Limitations

    A single recurrent checkpoint provides several compute–performance operating points, making execution depth a scaling dimension users can control after training.

    Source: §4 · p. 9

Abstract

from arXiv · show

Biological function in RNA is encoded across scales, from local sequence motifs to long-range structural dependencies. Conventional RNA foundation models couple parameter capacity and inference depth within a fixed architecture, prescribing a fixed computation graph for each inference. Here we present Eco-RNA, an 80M-parameter RNA foundation model that makes inference depth adjustable through learned iterative refinement. Its shared, input-conditioned Transformer core refines sequence representations through repeated application. The iteration count is selected after training and controls effective depth at fixed model parameters. We train across recurrent depths with masked reconstruction, state regularization, and a one-sided loss that anchors deeper reconstruction to a shallow reference. On the 70-assay RNAGym fitness benchmark, increasing refinement from one to four iterations raises aggregate absolute Spearman correlation from 0.242 to 0.286, with the gain retained through eight iterations. At four iterations, Eco-RNA surpasses the current SOTA Evo 2 40B on all three aggregate fitness metrics despite 500x fewer parameters. Its iteration configurations advance the compute-performance Pareto frontier among the evaluated models. It also outperforms the evaluated baselines in non-coding RNA family classification, RNA secondary-structure prediction, and chemical probing profile prediction. These results establish internal inference computation as an effective scaling dimension for RNA foundation models.

1 INTRODUCTION

RNA foundation models face a fixed inference computation graph despite RNA function depending on information across sequence scales. Eco-RNA uses iterative refinement to make inference depth adjustable, improving RNAGym fitness prediction and supporting gains across downstream RNA tasks.

1 INTRODUCTION

  • RNA function depends on information ranging from local recognition motifs to interactions between distant nucleotides.Foundation models learn transferable representations from unannotated sequences and support structure and function prediction.Source: §1 · p. 1
  • Conventional RNA foundation-model backbones apply a fixed stack of transformations, limiting flexibility to trade additional inference computation for better predictions.Recurrent depth offers an internal scaling approach, but the shared update must remain predictive across budgets while retaining sequence information and controlling state scale.Source: §1 · p. 1
  • 0.242 to 0.286 aggregate absolute Spearman correlation as Eco-RNA depth increases from one to four iterations, with gains retained through eight and four-iteration Eco-RNA exceeding Evo 2 40B on all three aggregate fitness metrics.The four-iteration configuration uses approximately 500× fewer parameters; one through four iterations are Pareto-optimal among evaluated models.Source: Figure 1 · p. 2, §1 · p. 2
  • Eco-RNA reuses an input-conditioned Transformer core for an inference-selected number of iterations, with persistent input injection and bounded, normalized state updates.Pretraining combines masked reconstruction at sampled terminal depths, state regularization, and a one-sided anchor that adds supervision when deeper reconstruction loss exceeds a shallow reference.Source: §1 · p. 2, Figure 2 · p. 3
  • Eco-RNA task-specific models achieve the highest compared accuracy, secondary-structure F1, and probe-mean chemical-reactivity AUC, while tasks show different saturation depths.Depth-dependent gains persist after supervised adaptation, with classification saturating earlier than structure and reactivity.Source: §1 · p. 2

2 METHOD

Eco-RNA makes recurrent depth an inference budget by repeatedly applying a shared Transformer core to an input-conditioned state. It trains the shared update across sampled depths with reconstruction supervision and trajectory regularization.

2.1 RECURRENT DEPTH AS AN INFERENCE BUDGET

  • Recurrent depth K controls how many times the shared Transformer core is applied, adding computation to the same input without changing parameters or the prediction interface.A prelude encodes the sequence once, the core updates the latent state, and a coda maps the state at depth K to a prediction.Source: §2.1 · p. 3

2.2 AN INPUT-CONDITIONED SHARED UPDATE

  • Each update combines the preceding state with a persistent input-conditioned prelude representation, using learned channelwise retention and input scales.The direct retention coefficient lies below one; the nonlinear core and normalization determine the dynamics of the full update.Source: §2.2 · p. 4

2.3 TRAINING OBJECTIVE

  • Masked reconstruction supervises each sampled terminal depth, while relative displacement regularization penalizes the final recurrent update relative to the preceding state’s scale.The displacement penalty covers attention-valid positions, including those without reconstruction targets.Source: §2.3 · pp. 4–5
  • The one-sided anchor reinforces deeper endpoints only when reconstruction loss exceeds a shallow reference by more than a tolerance; improvements beyond the reference remain unpenalized.The reference and terminal losses use the same corrupted input, initial state, supervised positions, and model parameters.Source: §2.3 · p. 5

2.4 DEPTH SAMPLING AND TRUNCATED BACKPROPAGATION

  • Terminal depth is sampled independently for each sequence, concentrating probability on shorter trajectories while retaining a tail of larger compute budgets.All sampled iterations execute forward, while gradients are retained through only the final min(Ki, Wbwd) iterations.Source: §2.4 · p. 6

2.5 PRETRAINING AND INFERENCE

  • Pretraining proceeds from random initialization, then continues on the same data sources with sampling shifted toward RNA; downstream tasks use the Stage-B checkpoint.At inference, K is explicit, and task-adapted checkpoints and heads remain fixed across each depth sweep.Source: §2.5 · p. 6

3 EXPERIMENTS

The experiments test whether repeated inference updates improve reconstruction and downstream RNA prediction at adjustable compute, then examine how gains vary across assays and task granularities. Deeper refinement improves fitness and fine-grained predictions, while the preferred saturation depth differs across tasks.

3.1 PROTOCOL

  • Depth sweeps vary K from 1 to 8 with weights and inputs fixed; fitness uses pretrained Eco-RNA, while family, reactivity, and structure tasks use separately fine-tuned copies.Fitness metrics are averaged within RNA type and equally across five types.Source: §3.1 · p. 6

3.2 PROGRESSIVE LATENT REFINEMENT EMERGES FROM PRETRAINING

  • Across four checkpoints from 50k to 300k updates, K=1 to K=4 reduces held-out reconstruction loss by 0.040–0.042, while K=4 to K=8 changes it by less than 0.001.Later checkpoints have lower losses, and each measured checkpoint benefits from additional iterations.Source: §3.2 · p. 7, Table 1 · p. 7

3.3 RECURRENT DEPTH ADVANCES THE COMPUTE-PERFORMANCE FRONTIER

  • At K=4, aggregate absolute Spearman correlation rises from 0.242 to 0.286; Eco-RNA exceeds Evo 2 40B on all three fitness metrics with 500× fewer parameters and approximately 200× fewer forward FLOPs.The gain is retained through eight iterations, and K=1–4 configurations lie on the estimated compute-performance Pareto frontier.Source: §3.3 · p. 7

3.4 ASSAY-LEVEL ANALYSIS OF DEPTH GAINS

  • Selected assays differ: splicing and aptamer gain most from weak K=1 baselines; coding improves gradually, while tRNA and ribozyme start stronger and gain modestly.Splicing Spearman correlation and AUC peak at K=3, but absolute MCC continues improving at K=4; validation can guide depth without retraining.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

3.5 OPTIMAL EXECUTION DEPTH SCALES WITH PREDICTION GRANULARITY

  • At K=4, nRC family-classification accuracy reaches 98.69%, and Eco-RNA leads the compared models before the observed early plateau.Additional iterations provide little further benefit for family assignment once this performance level is reached.Source: §3.5 · p. 8, Table 2 · p. 8
  • At K=4, Eco-RNA leads all three probe-mean metrics; all three means reach their best observed values at K=7 on held-out RNAGym sequences.The evaluation follows adaptation on Ribonanza and includes DMS and 2A3 predictions.Source: §3.5 · p. 8, Table 3 · p. 9
  • From K=1 to K=4, secondary-structure recall rises 3.3 percentage points while precision stays around 83–84%, with most F1 gain appearing early.Evaluation uses the bpRNA training split for fine-tuning and TS0 for evaluation.Source: §3.5 · p. 8, Table 2 · p. 8
  • Across evaluated tasks, coarse family labels plateau earlier than nucleotide-level and pairwise outputs, which retain smaller opportunities for correction at greater depth.The paper presents this as a pattern consistent with the greater prediction granularity of dense outputs.Source: §3.5 · p. 9

4 DISCUSSION AND CONCLUSION

Eco-RNA uses adjustable inference depth as a scaling dimension, with depth-dependent gains in reconstruction and downstream prediction that vary by task.

4 DISCUSSION AND CONCLUSION

  • One recurrent checkpoint provides several compute–performance operating points and advances the RNAGym Pareto frontier.Users can control execution depth after training, complementing capacity and data choices made during pretraining.Source: §4 · p. 9
  • Additional depth reduces held-out reconstruction loss at multiple pretraining stages, and gains persist after adaptation to sequence labels, nucleotide pairs, and chemical reactivity.Source: §4 · p. 9
  • Classification saturates early, while structure and reactivity show smaller gains at higher budgets; for structure, increased recall at sustained precision accounts for most improvement.Source: §4 · p. 10
  • Relative terminal displacement regularizes state movement, while a one-sided reconstruction anchor reinforces deeper endpoints that underperform a shallower reference.The anchor retains the observed nucleotide as its target, so improvements beyond the reference remain unpenalized.Source: §4 · p. 10

REPRODUCIBILITY STATEMENT

The statement identifies the paper’s methodological specifications, data resources, and planned release materials.

REPRODUCIBILITY STATEMENT

  • The paper directs readers to Section 2 and appendices for model, training, evaluation, and compute details, and says weights, inference code, and evaluation configurations will be released later.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 ARCHITECTURE AND NUMERICAL SPECIFICATION

The appendix specifies recurrent initialization and numerical controls alongside the Transformer block, while Table A1 presents backbone settings and values.

A.1 ARCHITECTURE AND NUMERICAL SPECIFICATION

  • The recurrent state uses a Gaussian initialization sampled independently during training and a fixed, position-indexed Gaussian table reused across evaluation depths.The evaluation initialization remains invariant as iteration count changes.Source: §A.1 · p. 14
  • The retention coefficient caps its exponent, and transition vectors and boundary maps are shared across iterations.Source: §A.1 · p. 14
  • Each Transformer block combines bidirectional grouped-query attention, QK normalization, RoPE, SwiGLU, and sandwich-normalized residual updates.The block applies attention and then an MLP, each within a normalized residual update.Source: §A.1 · p. 14
  • The backbone has 6+3K executed blocks, and Table A1 presents its configuration as settings and values.Source: §A.1 · p. 14, Table A1 · p. 14

A.2 VARIABLE-DEPTH TRAINING AND OBJECTIVES

Variable-depth training samples terminal iteration counts and combines masked reconstruction with trajectory regularization and a one-sided deeper-to-shallow anchor.

A.2 VARIABLE-DEPTH TRAINING AND OBJECTIVES

  • Right-aligned wavefront execution lets each row complete its sampled iteration count at a shared endpoint; long trajectories detach early states and backpropagate only through the final configured window.Shorter trajectories are differentiated in full.Source: §A.2 · p. 14
  • Each update corrupts inputs, runs the recurrent wavefront, reads terminal and eligible detached anchor states, then accumulates token-normalized gradients from masked-token and auxiliary objectives.Source: §A.2 · p. 14
  • Span corruption selects eligible non-special positions in boundary-clipped random spans, and every selected position contributes to reconstruction.Source: §A.2 · p. 15
  • The displacement penalty applies to sequences with at least two iterations and covers all attention-valid positions; anchor eligibility requires depth beyond Ka and at least one masked position.The captured anchor state is detached and evaluated without gradients.Source: §A.2 · p. 15
  • Dynamic token batches are normalized over supervised tokens across ranks and accumulation microbatches, with auxiliary means weighted to match each microbatch’s supervised-token contribution.Source: §A.2 · p. 15

A.3 PRETRAINING DATA AND CONFIGURATION

Pretraining uses genomic and RNA sequence sources, with RNA exposure increasing in Stage B. The appendix reports configuration tables for both stages.

A.3 PRETRAINING DATA AND CONFIGURATION

  • Stage A trains the backbone from random initialization on length-partitioned OpenGenome2 genomic sources, its non-coding-RNA partition, and filtered RNAcentral sequences.The schedule progressively increases RNA sampling while retaining genomic exposure; sequences beyond the context limit are randomly cropped.Source: §A.3 · p. 16
  • Stage B continues self-supervised pretraining with greater RNA exposure, using the architecture and objective established in Stage A.Its configuration is reported in Table A3.Source: §A.3 · p. 16, Table A3 · p. 16
  • Stage-A and Stage-B configuration tables report the settings for the two pretraining stages.Table A2 covers Stage A; Table A3 covers Stage B.Source: Table A2 · p. 16, Table A3 · p. 16

A.4 COMPUTE ACCOUNTING

This section accounts for inference compute and specifies evaluation procedures for classification, secondary-structure prediction, and zero-shot fitness scoring across recurrent depths.

A.4 COMPUTE ACCOUNTING

  • The accounting combines dense-block matrix-multiplication cost with per-iteration input projections and one-time output and vocabulary-head costs.The block formula includes attention and feed-forward terms; fused multiply–adds count as two operations.Source: §A.4 · p. 16
  • At L=512, Eco-RNA uses 90.011, 180.608, and 301.404 GFLOPs at K=1, 4, and 8, respectively.The count covers dense matrix multiplications in one forward pass, which dominate cost at this scale; compared models are counted the same way.Source: §A.4 · p. 16

A.5 NON-CODING RNA CLASSIFICATION AND SECONDARY-STRUCTURE PREDICTION

  • Family classification uses the 13-class nRC split, while secondary-structure prediction uses bpRNA training and TS0 testing with strict global full-matrix micro-F1.Both tasks adapt the full backbone with task-specific heads and hold the adapted checkpoint and head fixed across K=1,...,8.Source: §A.5 · p. 17

A.6 RNAGYM FITNESS PREDICTION

  • Zero-shot fitness evaluation covers 70 RNAGym assays, scoring each variant’s substitutions as mutant-versus-wild-type log-odds under the uncorrupted wild-type context.The headline metrics orient assay scores invariantly and average assays within each of five RNA types before weighting types equally.Source: §A.6 · p. 17
  • The fitness depth sweep holds the checkpoint, alphabet convention, initial-state table, and scoring rule fixed; its control uses nine distinct blocks once per sequence.The fixed-depth control matches the recurrent model’s parameter count, block count, data, and training schedule.Source: §A.6 · p. 17

A.7 CHEMICAL-REACTIVITY PREDICTION

Chemical-reactivity evaluation uses held-out sequences and a fixed-head depth sweep; related work situates Eco-RNA among adjustable-depth and recurrent biological models.

A.7 CHEMICAL-REACTIVITY PREDICTION

  • Chemical-reactivity evaluation fine-tunes learnable models on Ribonanza and tests on 114,836 held-out sequences, with matched heads for RNA-FM, UFold, and MXfold2.Thermodynamic predictors supply reactivity scores from their structure-derived unpaired probabilities.Source: §A.7 · p. 17
  • Eco-RNA uses a four-block convolutional head over terminal representations and their difference from the first-iteration representation, with checkpoint and head fixed across K.The difference is zero at K=1; mean AUC, macro-F1, and MAE weight DMS and 2A3 equally.Source: §A.7 · p. 17
  • Figure A1 compares reference and predicted base pairs across recurrent depths; Table A6 reports chemical-reactivity depth results with the best value marked in each column.Figure edges encode correct, false, and missed base pairs; the table’s mean weights DMS and 2A3 equally.Source: Figure A1 · p. 18, Table A6 · p. 19

B RELATED WORK

  • Unlike nucleotide backbones with prescribed transformation counts, Eco-RNA lets one checkpoint select inference depth while preserving its input and prediction interface.The passage frames inference depth as a separate scaling axis.Source: B RELATED WORK · p. 20
  • Recurrent-depth designs vary in where computation is repeated or allocated; Eco-RNA applies a full-resolution sequence update with one sequence-wide inference-time iteration count.The cited comparison includes shared transitions, fixed layer sharing, adapters, token-level allocation, and resolution changes.Source: B RELATED WORK · p. 20
  • Prior shared-depth methods vary depth sampling and backward windows, while other work studies recurrent dynamics through scaling, Jacobian regularization, or latent anchoring.The passage cites Huginn (Geiping et al., 2025), Parcae (Prairie et al., 2026), DeepLoop (Li et al., 2026), STARS (Yang et al., 2026), and Think Shallow, Solve Deep (Viakhirev et al., 2026).Source: B RELATED WORK · p. 20
  • Eco-RNA regulates recurrent-state motion and scale, then uses a one-sided reconstruction anchor to gate deeper supervision while leaving observed nucleotides as targets.Deeper predictions remain free to improve beyond the shallow reference.Source: B RELATED WORK · p. 20
  • Biological-model iteration varies by recurrent object and learning objective; Eco-RNA instead places recurrence in its self-supervised sequence backbone across RNA prediction tasks.The comparisons include RNA pairing representations, predicted protein structures, and memory tokens across genomic segments.Source: B RELATED WORK · p. 20
Loading 2610.13106v1…