Source-linked AI summary
CocoRNA: Collective RNA Design with Cooperative Multi-agent Reinforcement Learning
Tianmeng Hu, Biao Luo, Ke Li
TL;DR
RNA inverse design seeks sequences that fold into specified secondary structures, but the task is ill-posed and difficult to search. CocoRNA uses cooperative multi-agent reinforcement learning to decompose the design task; on Rfam, it reports higher success and faster design than the best baseline.
Problem
RNA inverse design seeks sequences that fold into a specified secondary structure, but multiple sequences can yield the same structure and the sequence space grows exponentially.
Source: §1 · pp. 1–2Method
CocoRNA applies cooperative multi-agent reinforcement learning to RNA design, decomposing the task into subproblems and using search-augmented exploration.
Source: §1 · p. 3Results
On Rfam, CocoRNA achieves a 97.75% success rate under identical time constraints, 27.25 percentage points above the best baseline, and averages 3.24 seconds per solution, over 70 times faster.
Source: §1 · p. 3Takeaways & Limitations
The conclusion reports that decomposition reduces the complexity faced by each agent, enhances policy exploration, and improves learning efficiency and generalizability.
Source: §6 · p. 11The authors note that applying AI to biological sequence design may raise interpretability and reliability risks.
Source: Impact Statement · p. 12
Abstract
from arXiv · showhide
Designing RNA sequences that reliably fold into specific secondary structures is essential for understanding their biological functions but remains a challenging computational problem. We propose CocoRNA, a cooperative multi-agent reinforcement learning framework for RNA inverse design. CocoRNA simplifies the design task by decomposing it into smaller sub-problems, each solved collaboratively by multiple agents. This approach reduces the complexity of the problem and improves the exploration of design policies. During training, a centralized critic uses global structural information to guide the agents, enabling them to jointly optimize their design strategies. As a result, CocoRNA learns high-quality RNA design policies that generalize effectively to unseen structures without additional training. Experiments on the Rfam dataset demonstrate that CocoRNA substantially outperforms state-of-the-art methods in both success rate and design speed. Further experiments on other biological sequence design tasks highlight the effectiveness and broad potential of CocoRNA for complex design tasks.
1 Introduction
RNA inverse design seeks sequences that fold into a target structure, but the search is large and may be non-unique. CocoRNA applies cooperative multi-agent reinforcement learning to this task and reports higher success and faster design than baseline methods.
1 Introduction
- RNA inverse design seeks sequences that fold into a specified secondary structure, but multiple distinct sequences can share that structure and the search space grows exponentially.A 74-nucleotide tRNA example has an approximately 10^36-sequence search space, making brute-force approaches computationally infeasible.Source: §1 · pp. 1–2
- RL design is difficult because exact matches are rare, rewards are delayed, and local optima can hinder policy improvement.Auxiliary rewards may overshadow the primary objective in long tasks, while limited data makes value-function estimation difficult.Source: §1 · p. 2
- The authors frame biological sequence design explicitly as a multi-agent decision-making problem and apply MARL to the domain for the first time, to their knowledge.They evaluate CocoRNA on Rfam and conduct ablation studies.Source: §1 · p. 3
- CocoRNA formulates RNA design as a cooperative multi-agent task, decomposing sequence generation into subproblems and using a search-augmented exploration mechanism.The framework uses a Dec-POMDP formulation over joint sequence–structure states.Source: §1 · p. 3
- 97.75% success under identical time constraints, a 27.25-percentage-point lead over the best baseline; CocoRNA also solves designs in 3.24 seconds on average, over 70 times faster.With the strongest baseline given 20 times more computational time, CocoRNA retains an 8.21-percentage-point lead.Source: §1 · p. 3
2 Preliminaries
RNA inverse design starts from a target secondary structure and seeks sequences predicted to fold into it. The preliminaries also define a cooperative partially observable multi-agent model and centralized training with decentralized execution.
RNA secondary structure representation
- Dot-bracket notation marks unpaired nucleotides with dots and base pairs with matching parentheses.The example ((. . .)) represents a hairpin loop, with paired bases at the ends and unpaired bases in the middle.Source: §2 · p. 3
Problem formulation
- RNA inverse design uses a target secondary structure as input and seeks sequences whose predicted folds match it.Multiple sequences can realize the same fold, allowing additional properties to be optimized while preserving the target.Source: §2 · p. 3
Decentralized partially observable Markov decision process
- The cooperative task is modeled as a decentralized partially observable Markov decision process with shared rewards and agent-specific observations and actions.The model includes agents, global states, local observation and action spaces, transition probabilities, a shared reward, and a discount factor.Source: §2 · pp. 3–4
Centralized training with decentralized execution
- Under centralized training with decentralized execution, agents use global information during training but act from local observations during execution.The centralized Critic uses global information to guide decentralized Actors.Source: §2 · p. 4
3 Proposed Method
CocoRNA decomposes RNA inverse design into cooperative agent tasks, then trains decentralized policies with centralized value guidance. Its reward combines incremental structural improvement with a strong success signal, while early search augments training experience.
3.1 Problem decomposition
- Agents divide RNA design by sequence position or structural type, reducing each agent’s state and policy spaces to support more focused learning.Each agent can work on a segment or a structural class such as stems or loops, using a local sequence-and-structure observation; the framework can also use alternative encodings, including matrix-based representations for pseudoknots.Source: §3.1 · p. 5
3.2 Policy optimization
- The environment hard-masks nucleotide actions that violate Watson-Crick or wobble compatibility with an already assigned paired position.The Actor outputs probabilities over four nucleotides, but invalid choices are excluded before execution.Source: §3.2 · p. 6
- Each agent’s Actor maps a local observation to nucleotide actions, while a centralized Critic evaluates the global sequence–structure state to guide policy updates.Agents maintain independent network parameters; the Actor is an MLP and the Critic uses a CNN-based architecture.Source: §3.2 · pp. 5–6
3.3 Reward function
- CocoRNA rewards normalized reductions in Hamming distance at intermediate steps and gives a substantial final reward when the predicted structure exactly matches the target.The distance is computed between the ViennaRNA-predicted structure and the target; a zero distance ends the episode.Source: §3.3 · pp. 6–7
- Auxiliary GC-content or minimum-free-energy rewards activate only after structural success, preserving the target fold as the primary condition.The GC term penalizes deviation from a target GC content, while the MFE term is based on predicted free energy.Source: §3.3 · p. 7
3.4 Search-augmented exploration
- Search-augmented exploration replaces an early policy experience with a higher-reward alternative found by brief local search, improving the quality of training experiences.CocoRNA applies SAE during the first 30% of training without changing the original optimization objective.Source: §3.4 · p. 7
4 Empirical Study
CocoRNA performs strongly on Rfam and related design evaluations, with high success and speed, and the ablations support benefits from its multi-agent architecture and SAE heuristic.
4.1 Experiments on RNA inverse design
- 97.75% success under a 30-second limit exceeds SAMFEO-30s by 27.25 percentage points; with 20× longer baseline budgets, CocoRNA leads Meta-LEARNA-Adapt-600s by 8.21 points.The evaluation uses Rfam structures split into training and test sets, with additional cluster-based test splits for robustness.Source: §4.1 · pp. 7–8, Table 1 · p. 8
- 3.24 seconds is CocoRNA’s average solution time, over 70× faster than Meta-LEARNA-Adapt at 234.2 seconds; its average pairwise normalized Hamming distance is 0.405.Figure 3 presents CPU-time distributions over successfully solved cases.Source: §4.1 · p. 8, Figure 3 · p. 9
- Without retraining, CocoRNA achieves 92.4% success on 500 Rfam sequences of 500–971 nucleotides using a sliding-window approach.These sequences are significantly longer than training examples, reaching approximately twice their length.Source: §4.1 · pp. 8–9
- On a pseudoknotted dataset, matrix-based CocoRNA reaches 92.67% mean success across three held-out test sets, outperforming antaRNA by over 28 percentage points.The variant uses IPknot as oracle and is fine-tuned with 10% of the original training budget.Source: §4.1 · p. 9
- With ViennaRNA training and RibonanzaNet evaluation, CocoRNA achieves 94.90% success, indicating generalization across folding oracles.Source: §4.1 · p. 9
- CocoRNA exceeds 99% success on three RNA families; ribozyme success is 94.0%, rising to 98.8% with a 600-second limit.The authors associate the longer search requirement with a more rugged optimization landscape for some ribozymes.Source: §4.1 · p. 9
- Across TF Bind 8, TF Bind 10, ChEMBL, and UTR design tasks, experiments report strong generalization.Source: §4.1 · pp. 9–10
4.2 Ablation studies
- The Single-agent variant underperforms CocoRNA during training, while removing SAE slows convergence and increases variance; broader decomposition, reward, and training-setting analyses support robustness.The authors attribute improved exploration and convergence to multi-agent decomposition and early high-quality experiences to SAE.Source: §4.2 · p. 10
5 Related Works
Prior RNA design methods span heuristic search and learning-based approaches. CocoRNA differs from related decomposition methods through cooperative policies guided by global structural context.
5 Related Works
- Earlier RNA design methods use stochastic optimization, including adaptive random walks, local search, evolutionary algorithms, Monte Carlo tree search, and simulated annealing.These approaches often depend on handcrafted heuristics and can become computationally expensive as sequence length grows, with limited generalization to new problems.Source: §5 · pp. 10–11
- Learning-based RNA design includes pre-trained RL policies, RL with neural architecture search and instance-specific fine-tuning, and generative models such as RNAinformer and OmniGenome.The text notes that RNAinformer is computationally intensive, limited in practical applicability to sequences under 100 nucleotides, and not open-source.Source: §5 · p. 11
- Unlike RNA-SSD, CocoRNA uses a centralized Critic with complete global context to guide cooperative agents toward globally optimal solutions.RNA-SSD divides sequences into independently optimized subsequences but does not effectively incorporate global structural information.Source: §5 · p. 11
- Unlike GameOpt, which searches for a fixed solution over predefined variables, CocoRNA trains dynamic policies for cooperative adaptation.Source: §5 · p. 11
6 Conclusion
CocoRNA applies cooperative multi-agent reinforcement learning to RNA secondary-structure design by dividing the task among agents. The authors report effectiveness and generalizability, while identifying future directions for larger biological design problems.
6 Conclusion
- CocoRNA decomposes RNA secondary-structure design into agent-assigned subtasks, reducing each agent’s complexity and improving policy exploration and learning efficiency.Source: §6 · p. 11
- The authors report that experiments demonstrate CocoRNA’s effectiveness and generalizability.Source: §6 · p. 11
- Future work could investigate more accurate reward models, adaptive decomposition, and scalable credit assignment for larger multi-agent biological design problems.Source: §6 · p. 11
Impact Statement
The impact statement notes potential applications and risks for AI in biological sequence design. The pseudocode describes CocoRNA’s MAPPO-based training, with local actors and a centralized critic.
Impact Statement
- The authors suggest potential applications in synthetic biology, RNA-based drug development, and gene-regulatory research, while noting possible interpretability and reliability risks.Source: Impact Statement · p. 12
A Pseudocode for CocoRNA
- CocoRNA uses MAPPO, with each actor selecting actions from local observations and a centralized Critic evaluating global state for policy updates.Source: A Pseudocode for CocoRNA · p. 18
- The pseudocode initializes a target structure and sequence, decomposes the task among agents, and uses assigned paired partners to form local observations and validity masks.Source: A Pseudocode for CocoRNA · p. 18
B.1 Convergence Analysis
CocoRNA’s multi-agent actor-critic converges to a local maximum of expected joint return under the stated conditions. Its discussion links practical progress beyond local optima to exploration and coordinated action across sequence regions.
B.1 Convergence Analysis
- Under the stated conditions, CocoRNA’s multi-agent actor-critic converges to a local maximum of expected joint return.The derivation reduces its policy gradient to the single-agent actor-critic gradient for a joint “super-agent.”Source: §B.1 · pp. 18–19
B.2 Limitation and Discussion
- Gradient-based deep learning and reinforcement learning generally guarantee local, not global, optima in non-convex landscapes.In practice, progress beyond local optima depends on balancing exploration and exploitation.Source: §B.2 · p. 20
- CocoRNA’s PPO entropy bonus encourages continued exploration by preventing policies from becoming too deterministic too quickly.Its multi-agent framework also lets agents act simultaneously on different sequence regions, increasing state-space exploration opportunities.Source: §B.2 · p. 20
C.1 Dataset Splitting
Additional experiments test CocoRNA’s generalization and design choices, finding strong results across data splits, decomposition schemes, reward settings, and biological design tasks. They also identify sensitivity to agent count, policy sharing, and learning rate.
C.1 Dataset Splitting
- 97.44% average success across six alternative train-test splits supports CocoRNA’s generalization beyond the main split.The splits use distinct test clusters; Table 3 reports comparisons with two strong baselines.Source: §C.1 · p. 20, Table 3 · p. 20
C.2 Using Different Decomposition Schemes
- Position-based and structure-type-based decomposition both significantly outperform the single-agent approach, while random decomposition performs worse.The authors suggest random position assignments make cooperation harder; despite this, performance does not collapse.Source: Table 4 · p. 20, §C.2 · p. 21
C.3 Ablation Study on SAE
- Removing SAE causes a slight performance decrease, while the single-agent ablation performs significantly worse.The SAE-ablated model continues improving late in training, suggesting further training may help.Source: §C.3 · p. 21, Table 5 · p. 21
C.4 Ablation Study on Reward Signals
- With delayed rewards, CocoRNA has almost no performance loss and uses one-tenth as many folding calls; terminal-only rewards slightly reduce performance.CocoRNA-ablated performs worse under both settings, and overall CocoRNA remains robust across these reward signals.Source: Table 6 · p. 21, §C.4 · p. 22, Figure 6 · p. 22
C.5 Effect of Varying the Number of Agents
- Four or six agents achieve the best performance; too few or too many agents tend to reduce it.The authors attribute the trade-off to greater per-agent task complexity with fewer agents and harder cooperation in a more non-stationary environment with more agents.Source: §C.5 · pp. 22–23, Table 7 · p. 23, Figure 7 · p. 23
C.6 Effect of Sharing Policy Parameters
- Shared policy parameters slightly decrease performance compared with independent policies.The authors suggest independent policies may better adapt to agents’ different subtasks, whereas sharing constrains them to the same policy function.Source: §C.6 · pp. 23–24, Table 8 · p. 23, Figure 8 · p. 24
C.7 Training Parameters
- Excessively high learning rates lead to unstable training and poor convergence; the main experiments use 1 × 10^-5.The authors report greater sensitivity to learning rate than to other hyperparameters.Source: §C.7 · p. 24, Figure 9 · p. 24
C.8 Other Results
- CocoRNA’s median solving time is 1.47 s, and it solves 95.54% of sequences within one episode.Meta-LEARNA-Adapt solves 3.38% in a single attempt, while both baselines require numerous iterative optimizations.Source: §C.8 · p. 25, Figure 10 · p. 25
D Other biological sequence design tasks
CocoRNA is evaluated beyond RNA design on DNA functional optimization and drug-molecule design tasks, where it consistently outperforms its single-agent ablation. The appendix also describes the Rfam data and the actor-critic network architecture.
D Other biological sequence design tasks
- Across TF Bind 8, TF Bind 10, UTR, and ChEMBL, CocoRNA consistently outperforms its single-agent counterpart and reaches the global optimum on TF Bind 8 and TF Bind 10.The tasks are adapted as DesignBench-based RL environments, with only minor state-space, action-space, and hyperparameter changes.Source: D Other biological sequence design tasks · p. 25
E Datasets Details
- The Rfam dataset contains RNA secondary structures without explicit sequence labels, and its structures are computationally predicted using RNA-folding tools.The appendix describes structure-length distributions and dataset statistics, but the supplied passages give no values.Source: E Datasets Details · p. 26, Figure 11 · p. 26
F.1 Network Architectures
- CocoRNA uses separate Actor and Critic networks per agent, with the Critic processing global sequence and structure information to estimate cumulative system reward.The Actor uses local observations to choose a nucleotide action.Source: §F.1 · pp. 26–27
F.2 Training and Optimization
CocoRNA uses on-policy PPO with GAE for training and reports efficient inference. The appendix lists training settings and discusses graph neural networks as a possible alternative with computational trade-offs.
F.2 Training and Optimization
- CocoRNA collects on-policy experience over fixed horizons and uses PPO with GAE to optimize policies and compute lower-variance advantages.PPO’s clipped surrogate objective is used to stabilize training.Source: §F.2 · p. 27
- Training takes approximately 8 to 10 hours on one NVIDIA 4090 GPU, or 3 to 4 hours with delayed rewards; inference averages 3.24 seconds per successful design.The delayed-reward reduction is reported with negligible performance degradation.Source: §F.2 · p. 27
F.4 Other Settings
- The specified settings include 350 maximum training steps per episode, 400 testing steps, 25 million training steps, and a local observation window of 61.Global state is standardized to dimension 900, and ViennaRNA 2.6.4 computes folded structures during design.Source: §F.4 · p. 28
G.1 Graph Neural Networks
- GNNs may capture nucleotide relationships, but their higher computational and memory overhead could reduce training efficiency.CocoRNA instead uses CNNs in a centralized Critic to extract global structural information and address partial observations.Source: §G.1 · p. 28
G.2 Limitations and Future Work
CocoRNA faces limitations in reward modeling and task decomposition, while increasing agent counts make coordination and credit assignment more challenging. The authors propose model-based RL, multi-objective design, adaptive decomposition, and more scalable coordination mechanisms as future directions.
G.2 Limitations and Future Work
- Designing an accurate, reliable reward model is non-trivial given biological complexity, while large pretrained models may be computationally expensive.The authors suggest model-based RL as a potential way to predict future states and rewards more efficiently.Source: §G.2 · p. 28
- The framework could be extended to multi-objective settings that jointly consider structural fidelity, free energy, and GC content.Source: §G.2 · p. 29
- The current decomposition may be limited for the high-dimensional nucleotide design space; adaptive decomposition with hierarchical policies is proposed as a promising direction.In the proposed hierarchy, high-level agents decide task decomposition and low-level agents handle specific sub-tasks.Source: §G.2 · p. 29
- As agent counts grow, non-stationarity makes coordination and credit assignment more challenging, and performance degrades moderately when the number of agents becomes too large.Agents share a global structural-quality reward, with credit assignment handled implicitly through centralized value estimation.Source: §G.2 · p. 29