PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentiable RNA Secondary Structure Extraction for Deep Learning

Tyler Illman, Max Ward, Marcell Szikszai, Ryan K. Krueger

arXiv:2609.30752v1cs.LG

TL;DR

Deep-learning RNA predictors commonly output weight matrices that need a separate extraction step to become interpretable, while the relationship between training and extraction has received relatively little attention. The paper compares extraction methods across pretrained RiNALMo and three proof-of-concept models, finding that performance depends on training and that SDSM outperforms BCE under all four extraction methods while producing outputs closest to ground truth before extraction.

  • Problem

    Deep-learning RNA predictors often produce abstract weight matrices requiring a separate extraction step, and the congruence between training and extraction has received relatively little attention.

    Source: §1 · p. 2

  • Method

    The study evaluates four extraction algorithms on pretrained RiNALMo outputs and three proof-of-concept models, including one trained with differentiable SDSM normalization to output probability matrices directly.

    Source: §1 · p. 2

  • Results

    SDSM normalization outperformed the BCE baseline under all four extraction methods and produced the lowest pre-extraction error, with mean squared error 0.000496 versus 0.00196 for the baseline and 0.1459 for the Nussinov model.

    Source: §5 · p. 13, §4 · p. 13

  • Takeaways & Limitations

    The results suggest aligning the training objective with the intended extractor or training the model to produce the required representation directly.

    Source: §5 · p. 13

    The trained models were intentionally small proof-of-concept models; applying SDSM to larger architectures and testing inter-family generalization remain future directions.

    Source: §5 · p. 13

Abstract

from arXiv · show

Many deep learning approaches to RNA secondary structure prediction have recently been proposed. They typically output a weight matrix $W$ where $W_{ij}$ is an arbitrary weight for base $i$ pairing with base $j$. Converting this matrix to a predicted secondary structure or base-pairing probability matrix typically involves ad hoc and problematic downstream algorithms. Despite the importance of this conversion step, which we refer to as structure extraction, it has received relatively little attention in the literature. In this work, we analyze how the congruence between training and extraction methods affects prediction performance. To do this, we compare four extraction algorithms: a Nussinov-like dynamic programming method, maximum-weight graph matching and the greedy extraction algorithms used by SPOT-RNA and RiNALMo. These are evaluated on outputs from the pretrained RiNALMo model and three toy models trained in this paper: a differentiable Nussinov-like model, a binary cross-entropy (BCE) baseline, and a model that incorporates a novel symmetric doubly stochastic matrix (SDSM) normalization algorithm during training which allows it to output base-pairing probability matrices directly, without a separate extraction step. This SDSM normalization algorithm is differentiable and can be added inline to any deep learning model during training and evaluation. We find that the performance of each extraction method depends strongly on how the corresponding model was trained. Considering the toy models themselves, the SDSM model showed the strongest overall performance: it outperformed the BCE baseline under all four extraction algorithms and produced pre-extraction outputs closest to the ground truth. These results suggest that SDSM normalization is a tractable alternative to traditional structure extraction.

1 Introduction

Deep-learning RNA predictors often produce weight matrices that require a separate, non-differentiable step to yield interpretable structures. This work examines how aligning training objectives with extraction methods affects prediction and evaluates approaches that produce interpretable representations directly.

  • RNA secondary structure is important because ncRNA function is largely governed by its pattern of base-pairing interactions.The paper frames accurate structure prediction from nucleotide sequence as a central computational biology challenge.Source: §1 · p. 2
  • Thermodynamic prediction methods established dynamic-programming and probabilistic foundations, including maximizing base pairs, finding minimum-free-energy structures, and calculating pairing probabilities.The paper traces these approaches to Nussinov, Zuker and Stiegler, and McCaskill.Source: §1 · p. 2
  • Deep-learning models often output unconstrained weight matrices that require separate, non-differentiable extraction into interpretable structures or probability matrices.These representations are often needed by downstream tasks, but extraction is frequently independent of model training objectives.Source: §1 · p. 2
  • Training-extraction congruence describes the alignment between a model’s training objective and its extraction algorithm, which the authors identify as central to their study.A model can minimize numerical loss yet still produce matrices that extract poorly into interpretable representations.Source: §1 · p. 2
  • The study evaluates four extraction methods on pretrained RiNALMo outputs and three proof-of-concept models, including a differentiable Nussinov-like model and an SDSM-normalized model.The third model is a binary cross-entropy baseline; SDSM normalization enables direct probability-matrix output.Source: §1 · p. 2

2 Materials and Methodology

The materials and methodology describe how unconstrained RNA pairing scores are converted into interpretable structures, using differently trained models and four extraction methods. They also specify pseudoknot constraints and how matrix outputs, extracted structures, and sequence-level performance differences are evaluated.

2.1 Dataset and Preprocessing

  • Experiments use ArchiveII, but training and testing exclude sequences of 200 or more nucleotides and RNA families with fewer than 15 sequences.The 3,975-sequence dataset is randomly split 80/20, so homologous sequences appear in both sets; the authors state that the experiment addresses training-extraction congruence, not generalization. Sequences and structures are encoded as one-hot and adjacency matrices, respectively, with only the upper triangle used for training.Source: §2.1 · pp. 2–3

2.2 Interpretable Secondary Structure Representations

  • Weight matrices can assign conflicting pairing scores and generally are not interpretable, whereas probability matrices and discrete structures are symmetric and doubly stochastic.In probability matrices, off-diagonal entries represent pairing probabilities and diagonal entries represent unpaired probabilities. SDSM normalization enforces these properties differentiably, allowing direct probability-matrix outputs instead of a separate extraction procedure.Source: §2.2 · p. 4

2.3.1 RiNALMo’s Structure Extraction Algorithm

  • RiNALMo [11] and SPOT-RNA use thresholded greedy extraction, but differ in masking, thresholds, and when they resolve conflicting pairings.RiNALMo masks non-canonical pairs, applies a loop filter and typically a 0.5 confidence threshold, then greedily selects pairs while eliminating conflicts. SPOT-RNA retains its default allowance for non-canonical pairs, uses a 0.335 threshold, and initially selects multiplets before pruning conflicts by weight.Source: §2.3.1 · p. 4, §2.3.2 · pp. 4–5, §2.3 · p. 4

2.3.3 Maximum Weight Matching Extraction Algorithm

  • Maximum weight matching selects the highest-total-weight valid pairing, while Nussinov-like dynamic programming finds the highest-scoring nested structure.Matching uses canonical or wobble pairs and a minimum loop length, with an unpaired node and Edmonds’ Blossom algorithm [22]. Nussinov-like extraction uses traceback to enforce non-crossing pairs, minimum loop length, and no pseudoknots.Source: §2.3.3 · p. 5, §2.3.4 · p. 5

2.5 Model Architectures and Implementation

  • The toy models share a 200 × 200-output MLP, and all four extractors were evaluated by F1 on pretrained RiNALMo [11] outputs from ArchiveII.The pretrained-model results are not directly comparable to those of the paper’s small proof-of-concept models, which use an identical fully connected architecture and sequences padded to 200 nucleotides.Source: §2.4 · p. 5, §2.5 · pp. 5–6

2.5.1 Nussinov-like Model

  • The Nussinov-like model predicts bounded pseudo-free energies, differentiably sums Boltzmann weights over valid structures, and trains by ground-truth structure negative log-likelihood.The partition-function recurrence sums over nested structures; scaling, a 200-nucleotide length limit, and gradient clipping address numerical stability. Its energy and Boltzmann-weight outputs still require downstream extraction.Source: §2.5.1 · pp. 6–7

2.5.2 Binary Cross-Entropy Baseline Model

  • The binary cross-entropy baseline predicts independent sigmoid scores, uses class-balanced loss for sparse pairing labels, and requires downstream extraction.Because entries are treated independently, its weight matrix is not constrained to be symmetric or doubly stochastic and is not directly interpretable as a structure.Source: §2.5.2 · pp. 7–8

2.5.3 Symmetric Doubly Stochastic Matrix (SDSM) Model

  • SDSM normalization makes the model output a symmetric doubly stochastic base-pairing probability matrix, removing the need for a separate extraction step.Raw scores are symmetrized, made positive with softplus, then normalized with a modified Sinkhorn-Knopp procedure that preserves symmetry. The output is trained with the class-balanced binary cross-entropy loss; the four extractors are still applied for consistent comparisons.Source: §2.5.3 · pp. 8–9

2.6 Treatment of Pseudoknots

  • Nussinov training and extraction enforce nesting; BCE/SDSM and RiNALMo outputs, plus Graph Matching, RiNALMo, and SPOT-RNA extraction, can represent or retain pseudoknots, whose prediction was not assessed.Pseudoknots account for 1.4% of base-pairing interactions and complicate recursive dynamic programming.Source: §2.6 · p. 9

2.7 Evaluation Metrics

  • MSE evaluates output matrices before extraction, while relaxed F1 measures extracted structures and counts exact pairs or pairs shifted by one position in a cardinal direction as correct.Source: §2.7 · p. 9
  • A win-based permutation test compares sequence-level F1 wins using 10,000 random swaps of method assignments, testing consistency rather than the magnitude of mean F1 differences.Source: §2.7 · p. 9
  • Differences were deemed statistically significant when p < 0.05.Source: §2.7 · p. 9

3 Results

Extraction performance varied with the model outputs: pretrained RiNALMo results were relatively close across methods, while the three proof-of-concept models showed different extraction-method rankings. RiNALMo results are contextual and not directly comparable to those models.

3.1 Extraction Method Performance on RiNALMo Outputs

  • RiNALMo extraction achieved the highest pretrained RiNALMo mean F1 score (0.8713), followed by SPOT-RNA (0.8675), Nussinov (0.8630), and Graph Matching (0.8626).RiNALMo extraction significantly outperformed Graph Matching and SPOT-RNA, with insufficient evidence to distinguish it from Nussinov; Nussinov significantly outperformed Graph Matching and SPOT-RNA.Source: Table 3 · p. 10, §3.1 · p. 10
  • Pretrained RiNALMo results are not directly comparable to the proof-of-concept models because RiNALMo is larger and uses more diverse training data, a more sophisticated architecture, and a different optimization procedure.These results provide contextual background for the training-extraction congruence analysis.Source: §3.1 · p. 10

3.2 Nussinov-like Model Results

  • Graph Matching achieved the highest differentiable Nussinov-like model mean F1 score (0.9163), followed by Nussinov extraction (0.8451), SPOT-RNA (0.6774), and RiNALMo (0.6524).Graph Matching significantly outperformed all other methods; Nussinov significantly outperformed both greedy methods, and SPOT-RNA significantly outperformed RiNALMo.Source: Table 4 · p. 10, §3.2 · p. 10

3.3 Binary Cross-Entropy Baseline Model Results

  • SPOT-RNA achieved the highest class-balanced BCE-model mean F1 score (0.8707), followed by Nussinov (0.8478), RiNALMo (0.7147), and Graph Matching (0.6008).SPOT-RNA significantly outperformed all other methods; Nussinov significantly outperformed Graph Matching and RiNALMo, and RiNALMo significantly outperformed Graph Matching.Source: Table 5 · p. 11, §3.3 · p. 11

3.4 SDSM Model Results

  • SPOT-RNA achieved the highest SDSM-model mean F1 score (0.9004), followed by Graph Matching (0.8935), Nussinov (0.8494), and RiNALMo (0.7171).SPOT-RNA significantly outperformed all other extraction methods; Graph Matching significantly outperformed Nussinov and RiNALMo, and Nussinov significantly outperformed RiNALMo.Source: Table 6 · p. 11, §3.4 · p. 11

4 Discussion

Extraction performance depends on the model’s training and output representation: the SDSM model consistently outperformed the BCE baseline, while the pretrained RiNALMo outputs were relatively extractor-agnostic. Results across the trained models show that no extraction method was best for every model.

SDSM model

  • SDSM outperformed the BCE baseline under all four extraction methods, with statistically significant gains and its largest increase for Graph Matching, from 0.6008 to 0.8935.SPOT-RNA scored highest on SDSM at 0.9004; RiNALMo scored lowest at 0.7171, likely because its 0.5 threshold discards probability mass spread below the cutoff.Source: Table 7 · p. 11, §4 · p. 12

Pretrained RiNALMo model

  • All four extractors achieved similar performance on fixed pretrained RiNALMo outputs, but those scores are not directly comparable to the trained toy models’ scores.The RiNALMo model and matrices were held constant while the extractor varied; the trained models differ in architecture, training data, and optimization.Source: §4 · pp. 11–12

Nussinov-like model

  • Graph Matching scored 0.9163 on the Nussinov-like model, ahead of the Nussinov extractor at 0.8451 and the SPOT-RNA and RiNALMo extractors at 0.6774 and 0.6524.The passage attributes Graph Matching’s advantage to additive pseudo-free-energy weights and its ability to select non-nested structures when pseudoknots occur.Source: §4 · p. 12

BCE baseline

  • On BCE, SPOT-RNA scored 0.8707, while Nussinov and RiNALMo scored 0.8478 and 0.7147, and Graph Matching scored 0.6008.The passage links Graph Matching’s poor result to BCE’s independently computed sigmoid confidences, which lack an additive interpretation and comparable pairing and unpaired scores.Source: §4 · p. 12

SDSM model

  • SDSM had the lowest pre-extraction mean squared error, 0.000496, versus 0.00196 for BCE and 0.1459 for the Nussinov-like model.Figure 1 shows SDSM concentrating probability mass on true base pairs with little off-target signal.Source: Figure 1 · p. 12, §4 · p. 13

5 Conclusion

The conclusion is that extractor effectiveness depends on how its assumptions align with model training, rather than one extractor being universally optimal. SDSM normalization directly produces an interpretable probability matrix, but its evaluation here used intentionally small proof-of-concept models.

5 Conclusion

  • Extractor rankings reshuffled between the Nussinov-like and BCE models, showing that effectiveness depends on alignment between extraction procedures and training assumptions.Model training affects both output-representation quality and how effectively outputs can be extracted.Source: §5 · p. 13
  • SDSM normalization produces a non-negative, symmetric, doubly stochastic base-pairing probability matrix directly, achieved the lowest pre-extraction error, and outperformed BCE under every evaluated extractor.The direct probability matrix removes the need for additional extraction for almost all downstream tasks.Source: §5 · p. 13
  • The models were intentionally small proof-of-concept models; applying SDSM to larger architectures and testing inter-family generalization remain future directions.The conclusion presents aligning training with the intended extractor and directly producing the required representation as two approaches.Source: §5 · p. 13

A Model Architectures

The fully connected MLP models share a three-hidden-layer base architecture, while their final-layer activations differ by model.

A Model Architectures

  • All models use three ReLU-activated hidden layers followed by a dense output layer of size 200 × 200.Source: Table 8 · p. 15
  • The Nussinov-like model uses tanh as its final-layer activation, while the CB BCE baseline uses sigmoid.Source: A Model Architectures · p. 15
  • The SDSM model applies softplus to raw scores, followed by SDSM normalization.Source: A Model Architectures · p. 15

B.1 Pre-trained RiNALMo Model: Pairwise Extraction Method Comparisons

Table 9 reports pairwise permutation-test p-values for extraction methods on pretrained RiNALMo outputs across the full ArchiveII dataset.

B.1 Pre-trained RiNALMo Model: Pairwise Extraction Method Comparisons

  • Each table cell gives the p-value for the hypothesis that the row method outperforms the column method.Source: Table 9 · p. 15
  • The comparisons use pretrained RiNALMo outputs on the full ArchiveII dataset, with n = 3,975 and 10,000 permutations.Source: Table 9 · p. 15
  • The table marks significant results in bold when p < 0.05.Source: Table 9 · p. 15

B.2 Nussinov-like Model: Pairwise Extraction Method Comparisons

Table 10 reports pairwise permutation-test p-values for extraction methods applied to the Nussinov-like model, with significant results highlighted.

  • 10,000 permutations were used to calculate the pairwise tests for the Nussinov-like model.Source: Table 10 · p. 15
  • Each table cell gives the p-value for the hypothesis that its row method outperforms its column method.Source: Table 10 · p. 15
  • Bold formatting marks significant results at p < 0.05.Source: Table 10 · p. 15

B.3 CB BCE Baseline Model: Pairwise Extraction Method Comparisons

Table 11 reports pairwise permutation-test p-values for extraction methods applied to the CB BCE baseline, with significant results highlighted.

  • 10,000 permutations were used to calculate the pairwise tests for the CB BCE baseline model.Source: Table 11 · p. 16
  • Each table cell gives the p-value for the hypothesis that its row method outperforms its column method.Source: Table 11 · p. 16
  • Bold formatting marks significant results at p < 0.05.Source: Table 11 · p. 16

B.4 SDSM Model: Pairwise Extraction Method Comparisons

Table 12 reports pairwise permutation-test p-values for extraction methods applied to the SDSM model, with significant results highlighted.

  • 10,000 permutations were used to calculate the pairwise tests for the SDSM model.Source: Table 12 · p. 16
  • Each table cell gives the p-value for the hypothesis that its row method outperforms its column method.Source: Table 12 · p. 16
  • Bold formatting marks significant results at p < 0.05.Source: Table 12 · p. 16

B.5 Nussinov-like Model vs CB BCE Baseline

Tables 13 and 14 report model-vs-model permutation tests comparing each trained model with the CB BCE baseline across extraction methods.

B.5 Nussinov-like Model vs CB BCE Baseline

  • Table 13 reports per-extraction-method permutation-test p-values comparing the Nussinov-like model with the CB BCE baseline.Source: Table 13 · p. 16

B.6 SDSM Model vs CB BCE Baseline

  • Table 14 reports per-extraction-method permutation-test p-values comparing the SDSM model with the CB BCE baseline.Source: Table 14 · p. 16

C SDSM Normalization Implementation

The section identifies a modified Sinkhorn–Knopp normalization algorithm and gives fragments of its implementation, including symmetry preservation and a non-convergence branch.

C SDSM Normalization Implementation

Related papers

Loading 2609.30752v1…