Source-linked AI summary
Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction
Zhiyuan Chen, Zhenfeng Deng, Pan Deng, Yue Liao, Xiu Su, Peng Ye, Xihui Liu
TL;DR
RNA secondary-structure predictors can perform strongly on familiar held-out data, but it remains unclear whether they generalize across families and structural regimes. The paper introduces CHANRG, a structure-aware benchmark and padding-free evaluation framework, and finds that foundation-model leaders lose most of their held-out advantage out of distribution while structured and direct neural predictors are more robust.
Problem
It remains unclear whether recent predictor gains reflect transferable structure learning, because current benchmarks may not adequately test generalization across unseen RNA families, structural regimes, and reference genomes.
Method
CHANRG uses structure-aware deduplication, biologically distinct genome- and architecture-aware splits, multiscale evaluation, and a padding-free, symmetry-aware implementation to benchmark RNA structure predictors.
Results
Foundation-model predictors achieved the highest held-out accuracy but lost most of that advantage out of distribution, whereas structured decoders and direct neural predictors remained markedly more robust.
Takeaways & Limitations
Held-out leadership is not sufficient evidence of robust RNA structure prediction; predictor choice should depend on whether targets are familiar or structurally novel and family-distant.
Takeaways & Limitations
CHANRG is an operational stress test rather than an exhaustive taxonomy of RNA novelty, with strongest control for non-crossing topology while pseudoknot-involved interactions remain difficult to evaluate.
Abstract
from arXiv · showhide
Accurate prediction of RNA secondary structure underpins transcriptome annotation, mechanistic analysis of non-coding RNAs, and RNA therapeutic design. Recent gains from deep learning and RNA foundation models are difficult to interpret because current benchmarks may overestimate generalization across RNA families. We present the Comprehensive Hierarchical Annotation of Non-coding RNA Groups (CHANRG), a benchmark of 170{,}083 structurally non-redundant RNAs curated from more than 10 million sequences in Rfam~15.0 using structure-aware deduplication, genome-aware split design and multiscale structural evaluation. Across 29 predictors, foundation-model methods achieved the highest held-out accuracy but lost most of that advantage out of distribution, whereas structured decoders and direct neural predictors remained markedly more robust. This gap persisted after controlling for sequence length and reflected both loss of structural coverage and incorrect higher-order wiring. Together, CHANRG and a padding-free, symmetry-aware evaluation stack provide a stricter and batch-invariant framework for developing RNA structure predictors with demonstrable out-of-distribution robustness.
1 Introduction
RNA secondary-structure prediction supports biological annotation and design, but existing benchmarks may overestimate generalization across RNA families. CHANRG addresses this gap with structure-aware, biologically motivated evaluation and a padding-free implementation, revealing that held-out leaders do not retain their advantage out of distribution.
- Motivation: RNA secondary structure shapes folding, function, three-dimensional conformation, and applications including transcriptome annotation and RNA therapeutic design.Experimental assays are condition-dependent and incomplete, so computational prediction complements them at transcriptome scale.
- Motivation: Recent predictor gains may reflect fitting to permissive benchmark settings rather than transferable structure learning.The central unresolved question is whether models generalize across unseen families, structural regimes, and reference genomes.
- Benchmark gap: Existing benchmarks can overestimate generalization because sequence-only deduplication leaves structurally similar RNAs across evaluation boundaries and pair-level scores mask higher-order errors.Older source collections and incorrect junction wiring or topology further limit evaluation fidelity.
- Contribution: CHANRG combines structure-aware deduplication, architecture-aware splits, held-out and biologically distinct out-of-distribution regimes, and standardized evaluation across 29 predictors.The benchmark is curated from Rfam 15.0, while its reference implementation removes padded positions and redundant computation on symmetric contact maps.
- Contribution: CHANRG shows that foundation-model predictors lead on held-out accuracy but lose most of that advantage out of distribution, while structured and direct neural predictors remain more robust.Its overview explicitly contrasts held-out leadership with performance across out-of-distribution regimes.
2 Results
CHANRG reverses conventional held-out rankings: foundation models lead on Test but lose most of their advantage across biologically distinct OOD regimes, while structured decoders and direct neural predictors remain more robust. The gap persists after length matching and model scaling, and hierarchical metrics reveal failures in both interaction coverage and higher-order organization.
- 0.6731 Test base-pair F1 made foundation-model predictors the held-out leaders, ahead of direct neural predictors at 0.3495 and structured decoders at 0.3015.
- 26.7% retention left foundation-model predictors at 0.1796 OODmean, versus 82.5% for direct neural predictors and 92.3% for structured decoders.Foundation-model performance fell by 0.4935 from Test to OODmean, while direct neural predictors decreased from 0.3495 to 0.2883 and structured decoders from 0.3015 to 0.2784.
- 0.200 Spearman correlation between Test and OODmean ranks showed weak foundation-model rank stability, whereas structured decoders preserved rankings with ρ = 0.905.The foundation-model correlation was not significant (P = 0.704), while the structured-decoder correlation had P = 2.0 × 10−3.
- 0.2074 OODmean base-pair F1 remained for foundation models after restricting all splits to 50–200 nt, compared with 0.3156 for structured decoders and 0.3299 for direct neural predictors.Length matching did not rescue foundation-model performance out of distribution, despite their mean Test base-pair F1 of 0.7016.
- 0.1357 Test base-pair F1 was gained by scaling RiNALMo-U-Net from micro to giga, but OODmean improved by only 0.0680.Test F1 increased from 0.6222 to 0.7579, whereas OODmean increased from 0.1460 to 0.2140; topology metrics showed the same asymmetry.
- 17.2% topology-F1 retention for foundation models was substantially below their 31.1% stem-F1 retention, indicating deeper degradation in higher-order organization than in helix recovery.Foundation-model mean stem F1 fell from 0.6087 to 0.1890, while topology F1 fell from 0.4240 to 0.0729; structured decoders retained 89.8% and 88.6%, respectively.
3 Discussion
CHANRG shows that held-out accuracy does not reliably predict robustness to biologically distinct RNA regimes. Foundation-model predictors led on familiar held-out data but declined sharply out of distribution, while structured decoders and direct neural predictors retained more performance.
- 3 Discussion: Foundation-model predictors achieved the strongest held-out Test performance but declined sharply across GenA, GenC, and GenF.Their larger deterioration in topology F1 and topology GED than in stem F1 indicates weaker recovery of higher-order structure outside familiar families.
- 3 Discussion: Structured decoders and the strongest direct neural baselines retained substantially more performance across CHANRG’s out-of-distribution regimes.The comparison suggests that explicit structural constraints and task-aligned inductive bias remain important for transfer.
- 3 Discussion: Increasing foundation-model capacity improved held-out accuracy much more than out-of-distribution robustness.This scaling result supports a gap between high-capacity representation learning and transferable structural bias.
- 3 Discussion: CHANRG separates held-out interpolation from transfer to architectural, clan-level, and genome-sparse family regimes using a multilevel metric ladder.Its metrics distinguish local contact recovery, helix recovery, topology recovery, and higher-order structural damage.
- 3 Discussion: CHANRG is a stringent stress test of structural generalization rather than an exhaustive taxonomy of RNA novelty.Its strongest control concerns non-crossing secondary-structure topology, while pseudoknot-involved interactions remain difficult to recover.