PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Design RNA

Frederic Runge, Danny Stoll, Stefan Falkner, Frank Hutter

arXiv:1812.11951v2cs.LGq-bio.QMstat.ML

TL;DR

RNA Design asks for sequences that satisfy target structural constraints, a problem important because RNA structure supports diverse regulatory and engineering functions. The paper introduces LEARNA’s sequential deep-reinforcement-learning approach and Meta-LEARNA’s cross-task policy, with joint architecture and hyperparameter search. The resulting system reaches state-of-the-art performance on two established benchmarks and reaches prior state-of-the-art results much faster.

  • Problem

    RNA Design seeks an RNA sequence that folds into a desired structure, supporting engineering applications motivated by RNA’s regulatory and functional roles.

    Source: §1 · pp. 1–2

  • Method

    LEARNA uses deep reinforcement learning to sequentially generate complete RNA sequences, while Meta-LEARNA learns a transferable policy across tasks and jointly optimizes architecture, hyperparameters, and state representation.

    Source: §1 · p. 2

  • Results

    Meta-LEARNA achieves state-of-the-art performance on Rfam-Taneda and Eterna100 and reaches previous state-of-the-art results 63× and 1765× faster, respectively.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    A learned generative policy with local improvement and extensive optimization can solve RNA Design efficiently across established benchmarks.

    Source: §7 · p. 9

    The reported application uses the Zuker folding algorithm, although the authors state that the approach is not limited to it.

    Source: §2 · p. 3

Abstract

from arXiv · show

Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation. Since an RNA's function depends on its structural properties, the RNA Design problem is to find an RNA sequence which satisfies given structural constraints. Here, we propose a new algorithm for the RNA Design problem, dubbed LEARNA. LEARNA uses deep reinforcement learning to train a policy network to sequentially design an entire RNA sequence given a specified target structure. By meta-learning across 65000 different RNA Design tasks for one hour on 20 CPU cores, our extension Meta-LEARNA constructs an RNA Design policy that can be applied out of the box to solve novel RNA Design tasks. Methodologically, for what we believe to be the first time, we jointly optimize over a rich space of architectures for the policy network, the hyperparameters of the training procedure and the formulation of the decision process. Comprehensive empirical results on two widely-used RNA Design benchmarks, as well as a third one that we introduce, show that our approach achieves new state-of-the-art performance on the former while also being orders of magnitudes faster in reaching the previous state-of-the-art performance. In an ablation study, we analyze the importance of our method's different components.

1 INTRODUCTION

RNA Design seeks sequences that fold into desired structures, motivated by RNA’s regulatory roles and engineering applications. LEARNA and Meta-LEARNA use deep reinforcement learning and meta-learning to design sequences, with architecture search and benchmark results supporting their effectiveness.

Motivation

  • RNA’s regulatory functions and links to disease motivate engineering methods spanning medicine, biotechnology, synthetic biology, and related applications.The introduction connects functional non-protein-coding RNAs to transcription, epigenetic regulation, translation, and other cellular processes.Source: §1 · p. 1

LEARNA

  • LEARNA sequentially predicts an entire RNA sequence from a target secondary structure and uses folded-structure distance as the reinforcement-learning error signal.The generated sequence is folded and locally adapted before its distance to the target guides the agent.Source: §1 · p. 2

Meta-LEARNA

  • Meta-LEARNA learns one policy across many RNA Design tasks and can apply its learned generative model directly to novel target structures.The paper reports that many problems can be solved with the first sampled sequence.Source: §1 · p. 2

Benchmarks

  • The authors introduce a benchmark with explicit training, validation, and test splits because prior validation often used undisclosed data or lacked training splits.This benchmark addresses a stated evaluation limitation in earlier RNA Design literature.Source: §1 · p. 2

Architecture and optimization

  • The method jointly optimizes policy architecture, training hyperparameters, and state representation, which the authors identify as essential for best results.They describe this as the first application of architecture search to reinforcement learning and meta-learning, and to selecting convolutional-recurrent combinations.Source: §1 · p. 2

Results

  • 63× and 1765× faster than previous state-of-the-art approaches, respectively, while achieving new state-of-the-art performance on Rfam-Taneda and Eterna100.These are the two most commonly used RNA Design benchmarks cited in the paper.Source: §1 · p. 2

2 THE RNA DESIGN PROBLEM

RNA Design is the inverse problem of finding a sequence that folds into a specified secondary structure under a folding algorithm. The paper formulates this through structural loss and commonly uses Hamming distance, while noting that the Zuker folding algorithm is its main setting.

Definition

  • RNA Design finds an RNA sequence φ whose folding under algorithm F matches a target secondary structure ω.The target structure is represented using dot-bracket notation, with dots for unbound sites and brackets for hydrogen-bonded nucleotides.Source: §2 · pp. 2–3

Folding algorithm

  • The paper employs the Zuker algorithm, which uses a thermodynamic model to minimize free energy and identify the most stable secondary-structure conformation.The authors state that their approach also applies to other folding algorithms.Source: §2 · p. 3

Objective

  • The design objective minimizes a structural loss L_ω(F(φ)) measuring the difference between the target and the folded candidate structure.A minimizer of this loss is a solution for the specified target structure.Source: §2 · p. 3

Loss and solutions

  • Hamming distance is the common loss used in this work, although additional constraints such as stability or GC content can also be incorporated.Multiple RNA sequences may fold into the same secondary structure, so solutions are generally not unique.Source: §2 · p. 3

3 LEARNING TO DESIGN RNA

The paper models RNA Design as a generative reinforcement-learning decision process that builds candidate sequences from scratch. It introduces LEARNA for task-specific learning and Meta-LEARNA for transferring knowledge across target structures.

3.1 MODELLING RNA DESIGN AS A DECISION PROCESS

  • At paired sites, actions place Watson-Crick nucleotide pairs simultaneously, whereas unpaired sites receive one nucleotide and closing brackets require no action.This action design lets the agent construct the candidate sequence while traversing the target dot-bracket structure.Source: §3.1 · p. 3, Figure 2 · p. 4
  • The state is a padded local (2κ + 1)-gram centered on the current target-structure site, and deterministic transitions expose this local context step by step.Padding characters allow centered n-grams to be formed at every site.Source: §3.1 · pp. 3–4
  • The terminal reward is based on normalized Hamming distance between the folded candidate and target, with a local exhaustive improvement step for near-solutions.When the distance is below ξ, neighboring sequences are searched; the experiments use ξ = 5, requiring at most 256 neighboring sequences.Source: §3.1 · p. 4

3.2 OBTAINING POLICIES FOR RNA DESIGN

  • The policy networks use state embeddings and deep neural architectures whose convolutional, recurrent, and fully connected components are optimized jointly with training choices.LEARNA instead learns online from scratch for each target using PPO.Source: §3.2 · p. 4
  • Meta-LEARNA trains one policy across target structures and applies the fixed policy to new tasks, while Meta-LEARNA-Adapt further fine-tunes that initialization per task.Meta-LEARNA uses asynchronous parallel PPO updates across training tasks.Source: §3.2 · pp. 4–5

4 JOINT ARCHITECTURE AND HYPERPARAMETER SEARCH

The paper jointly searches policy architectures, environment representations, rewards, and training hyperparameters for RNA Design. BOHB evaluates configurations using bounded budgets and validation data created with explicit splits.

4 JOINT ARCHITECTURE AND HYPERPARAMETER SEARCH

  • The search jointly optimizes policy architecture, training hyperparameters, and decision-process formulation because deep-RL performance is sensitive to these choices.The architecture space combines convolutional and recurrent components selected by Bayesian optimization.Source: §4 · p. 5

4.1 SEARCH SPACE

  • The search space includes architecture, state and reward parameters, and PPO settings, yielding a 14-dimensional space.Optimized PPO parameters include learning rate, batch size, and entropy-regularization strength.Source: §4.1 · p. 5

4.2 SEARCH PROCEDURE

  • BOHB searches mixed discrete and continuous configurations using parallel resources and low-fidelity evaluations based on limited training or evaluation time.The procedure limits wall-clock time for Meta-LEARNA training or LEARNA evaluation.Source: §4.2 · pp. 5–6
  • The newly introduced Rfam-Learn benchmark supplies training and validation splits used to optimize the approaches without relying on undisclosed data sources.The dataset is based on Rfam version 13.0, and the test set is kept separate from optimization.Source: §4.2 · p. 6
  • The selected LEARNA configuration uses a 30-minute optimization variant with periodic PPO restarts and outperformed the 10-minute alternative.Evaluations later use the same restart idea under benchmark-specific time limits.Source: §4.2 · p. 6
  • Because single RL evaluations can be noisy, optimization uses summed minimum distances during search and selects the final configuration by unsolved-sequence count among the top five configurations.All configuration evaluations are performed on the validation set.Source: §4.2 · p. 6

5 RELATED WORK

Related work spans local and global RNA Design methods, reinforcement learning for combinatorial construction, and approaches incorporating human solutions. The paper positions its method as generative sequence construction rather than local sequence modification.

5 RELATED WORK

  • The paper describes joint architecture search for reinforcement learning as a reversal of earlier work that used reinforcement learning to search architectures.It presents this as the first application of architecture search for RL to the authors' knowledge.Source: §5 · p. 6
  • The paper contributes evidence that generative machine-learning methods are competitive in RNA Design and related molecular-design settings.This connection is framed alongside prior generative models for protein design and other matter-engineering problems.Source: §5 · p. 6
  • RNA Design methods include local sequence-editing algorithms and global methods that manipulate many candidates or sample from a learned distribution.Examples include RNAInverse, RNA-SSD, INFO-RNA, NUPACK, ERD, MODENA, antaRNA, and MCTS-RNA.Source: §5 · p. 7
  • Human-solution methods perform well on Eterna100 but cannot be compared across the paper's other benchmarks because human solutions and strategies are unavailable there.The paper also excludes that approach from comparison because of computational costs.Source: §5 · p. 7
  • The paper is influenced by reinforcement learning for combinatorial problems, adapting complete-solution generation to predict RNA nucleotides end to end.Unlike local modification, the agent constructs each candidate from scratch using episodic feedback.Source: §5 · p. 7
  • Eastman et al. (2018) use reinforcement learning for local search from a random sequence, with rewards only when a correct sequence is found.Their policy is a convolutional network pretrained on fixed-length random sequences, contrasting with the paper's generative approach.Source: §5 · p. 7

6 EXPERIMENTS

Across three benchmarks, the proposed methods achieve strong RNA Design performance, with Meta-LEARNA reaching prior state-of-the-art results substantially faster on established benchmarks. Ablations indicate that local improvement and joint search over architecture, environment, and training parameters materially affect performance.

6.1 COMPARATIVE STUDY

  • Meta-LEARNA and Meta-LEARNA-Adapt solve up to 68% of Eterna100 target structures, establishing new state-of-the-art results for the benchmark.Source: §6.1 · p. 8
  • Meta-LEARNA reaches previous state-of-the-art performance in less than 5 seconds on Rfam-Taneda and about 25 seconds on Eterna100, approximately 63× and 1765× faster, respectively.Source: §6.1 · p. 8
  • On Rfam-Taneda, Meta-LEARNA and Meta-LEARNA-Adapt achieve new state-of-the-art results after 400 and 90 seconds, respectively, solving 83% of target structures.Source: §6.1 · p. 8
  • On Rfam-Learn, Meta-LEARNA and antaRNA are the only methods solving all target structures, requiring 29 and 20 minutes, respectively.Source: §6.1 · p. 8

6.2 ABLATION STUDY AND PARAMETER IMPORTANCE

  • Removing the local improvement step consistently reduces performance, whereas restart mainly helps on Eterna100 and continued adaptation does not improve results.Source: §6.2 · p. 8
  • fANOVA identifies important parameters across policy architecture, environment, and training hyperparameters, supporting their joint optimization.Source: §6.2 · p. 9

7 CONCLUSION

The paper concludes that LEARNA formulates RNA Design as end-to-end sequential construction using deep reinforcement learning. Meta-LEARNA combines this approach with pre-training, local improvement, and extensive optimization to achieve state-of-the-art benchmark performance.

7 CONCLUSION

  • Meta-LEARNA achieves state-of-the-art results on the Eterna100 and Rfam-Taneda benchmarks through pre-training, local improvement, and architecture and hyperparameter optimization.Source: §7 · p. 9
  • LEARNA sequentially constructs candidate RNA solutions end to end using deep reinforcement learning.Source: §7 · p. 9

A PSEUDOCODE FOR COMPUTING THE REWARD

B CREATING THE RFAM-LEARN DATASETS

The Rfam-Learn dataset was constructed from Rfam 13.0 structures, filtered toward difficult instances and split into training, validation, and test sets. The experiments used specified ViennaRNA and reinforcement-learning software on controlled CPU hardware.

B CREATING THE RFAM-LEARN DATASETS

C SOFTWARE AND HARDWARE DETAILS

D BENCHMARKS

The study uses three RNA Design benchmarks and records their timeouts, evaluation counts, dataset sizes, and sequence-length ranges.

D BENCHMARKS

  • The benchmark overview covers Eterna100, Rfam-Taneda, and Rfam-Learn, reporting timeout, evaluations per target, sequence counts, and sequence-length ranges.Source: Table 2 · p. 15

E JOINT ARCHITECTURE AND HYPERPARAMETER SEARCH

The study jointly searches policy architectures, environment choices, and training hyperparameters for LEARNA and Meta-LEARNA, using BOHB budgets to prioritize promising configurations.

E JOINT ARCHITECTURE AND HYPERPARAMETER SEARCH

  • Joint optimization spans policy architecture, environment parameters, and training hyperparameters for the RNA Design agents.The search spaces for LEARNA and Meta-LEARNA are described in Tables 3 and 4.Source: E JOINT ARCHITECTURE AND… · p. 15, Table 3 · p. 15, Table 4 · p. 15
  • BOHB uses geometrically distributed budgets to eliminate poor configurations quickly while allocating more resources to promising ones.LEARNA varies evaluation timeouts, whereas Meta-LEARNA varies training time and fixes validation evaluation at 60 seconds.Source: E JOINT ARCHITECTURE AND… · pp. 15–16
  • Validation loss measures the sum of minimal normalized Hamming distances, while its relationship with solved-sequence fraction is monitored during optimization.For Meta-LEARNA, validation uses 100 sequences evaluated for 60 seconds each.Source: Figure 4 · p. 16, Figure 5 · p. 16
  • The selected LEARNA and Meta-LEARNA configurations differ mainly in architecture, with LEARNA using a larger CNN and an LSTM while Meta-LEARNA uses a smaller CNN without LSTM layers.Both use modest feed-forward components, and their learning rates and entropy regularization also differ.Source: E JOINT ARCHITECTURE AND… · p. 16, Table 5 · p. 17
  • The best configurations vary in key parameters, supporting joint optimization of multiple reinforcement-learning design choices for each scenario.The authors avoid concluding that CNNs are inherently better than LSTMs because longer training or further optimization could change Meta-LEARNA’s configuration.Source: E JOINT ARCHITECTURE AND… · pp. 16–17

F FUNCTIONAL ANALYSIS OF VARIANCE FOR META-LEARNA

Functional ANOVA identifies training, reward, and architectural parameters associated with performance across Meta-LEARNA’s search space, while warning that global effects may not represent peak-performing configurations.

F FUNCTIONAL ANALYSIS OF VARIANCE FOR META-LEARNA

  • The fANOVA analysis uses random-forest estimates of global variance explained, based on a limited and highly biased set of optimized evaluations.The analysis is designed to quantify parameter importance across the search space rather than only among the best configurations.Source: F FUNCTIONAL ANALYSIS OF VARIANCE FOR… · p. 18, Figure 6 · p. 18
  • Learning rate, entropy regularization, reward exponent, and fully connected-layer width are among Meta-LEARNA’s four most important individual parameters.These parameters span training, reward representation, and architecture, motivating their inclusion in the optimization.Source: F FUNCTIONAL ANALYSIS OF VARIANCE FOR… · p. 18, Figure 6 · p. 18
  • The plotted fraction of solved validation sequences is at most about 35%, although the best configurations achieved almost 90% in Figure 5.Consequently, global fANOVA behavior need not represent the narrow region containing peak performance.Source: F FUNCTIONAL ANALYSIS OF VARIANCE FOR… · p. 18
  • Parameter interactions also matter, so architecture and state-space choices may be more important for reaching peak performance than global fANOVA values indicate.The pairwise interaction analysis is shown in Figure 7.Source: F FUNCTIONAL ANALYSIS OF VARIANCE FOR… · p. 18, Figure 7 · p. 19
  • The reward exponent should be set relatively high together with a relatively large learning rate, while limited entropy regularization appears advantageous on average.The interpretation comes from marginal prediction plots over the evaluated search space.Source: F FUNCTIONAL ANALYSIS OF VARIANCE FOR… · pp. 18–19, Figure 7 · p. 19

G ABLATION STUDY

Ablations identify the model-based agent and local improvement as the largest contributors to performance, while the comparisons examine robustness across lengths, repeated runs, times, and target subsets.

G ABLATION STUDY

  • The model-based agent has the largest performance impact relative to random actions, followed by the local improvement step activated below 5 mismatches.Restarts affect Eterna100, while continued training in Meta-LEARNA-Adapt appears negligible in the evaluated setting.Source: G ABLATION STUDY · p. 20, Figure 8 · p. 20, Figure 9 · p. 21

H COMPARISON: PERFORMANCE ACROSS SEQUENCE LENGTHS

  • Figure 10 compares minimum solution times across sequence lengths on Rfam-Learn-Test using a 1-hour evaluation timeout.Points above the timeout line were not solved.Source: Figure 10 · p. 22

I COMPARISON: NUMBER OF SOLUTIONS PER K RUNS

  • The benchmark comparisons report solved-target percentages across repeated evaluation runs for Eterna100, Rfam-Taneda, and Rfam-Learn-Test.Tables 6–8 summarize the number of structures solved in specified numbers of runs.Source: Table 6 · p. 23, Table 7 · p. 23, Table 8 · p. 23

J COMPARISON: NUMBER OF SOLUTIONS AT DIFFERENT TIMES

  • The time-based comparisons report the percentage of solved target structures at different time points on Eterna100, Rfam-Taneda, and Rfam-Learn-Test.Tables 9–11 provide the corresponding comparisons across methods.Source: Table 9 · p. 24, Table 10 · p. 24, Table 11 · p. 24

K COMPARISON: SPECIFIC TARGET STRUCTURES

  • Specific-target analyses use five independent attempts on halves of Rfam-Learn-Test and Eterna100, and 50 attempts on each Rfam-Taneda target structure.Tables 12–15 cover the two halves of Rfam-Learn-Test and Eterna100, while Table 16 covers Rfam-Taneda.Source: Table 12 · p. 25, Table 13 · p. 26, Table 14 · p. 27

Related papers

Loading 1812.11951v2…