PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-aware Reinforcement Learning for Protein Directed Evolution

Zikun Nie, Suyuan Zhao, Yizhen Luo, Siqi Fan, Zaiqing Nie

arXiv:2609.39048v1cs.LGcs.AI

TL;DR

Sequence-only MLDE methods can miss structural context, while reliable mutant structures are difficult to obtain. StructEvo uses delta-based structural features and hierarchical mutation decisions; it reports gains on two hard benchmarks and an experimentally validated GFP epistasis pattern.

  • Problem

    Many MLDE approaches rely on sequence-only features and can overlook spatial constraints and co-evolutionary interactions represented by protein structures.

    Source: §1 · p. 1

  • Method

    StructEvo is a structure-aware RL framework that approximates mutant structure features with delta representations and decomposes mutation choices through a hierarchical action network.

    Source: §1 · p. 2

  • Results

    StructEvo achieves relative fitness gains of up to 9.2% on AAV-hard and 16.3% on GFP-hard, and an independently discovered GFP triple-mutation epistasis pattern was validated by existing wet-lab experiments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The validated GFP epistasis pattern highlights the potential of StructEvo for real-world applications.

    Source: §1 · p. 2

    The conclusion identifies extension to protein complexes, multi-objective properties, and wet-lab feedback as remaining limitations.

    Source: §6 · p. 10

Abstract

from arXiv · show

Protein optimization remains a longstanding goal in life sciences. Existing machine learning-assisted directed evolution (MLDE) methods primarily rely on sequence-only features, overlooking the critical spatial constraints and co-evolutionary interactions encoded in protein structures. However, directly integrating structural information remains challenging due to the scarcity of reliable mutant structures. To address these issues, we propose StructEvo, a novel structure-aware reinforcement learning framework for protein directed evolution. StructEvo employs a delta-structure fusion encoder to approximate mutant structure features via feature differences, enabling dynamic incorporation of spatial knowledge. The vast mutation space is then decomposed into manageable subspaces through a structure-aligned hierarchical action network, while a geometric constraint further stabilizes delta feature learning. Our approach outperforms prior state-of-the-art methods by 9.2% and 16.3% on two challenging optimization benchmarks, and further identifies an experimentally validated epistasis pattern in GFP, highlighting the importance of structural guidance for effective protein directed evolution.

1 Introduction

StructEvo addresses the limits of sequence-only MLDE by adding dynamic structural guidance to mutation proposals. It uses delta features and hierarchical decisions, with reported gains on hard benchmarks and a validated GFP epistasis pattern.

1 Introduction

  • Sequence-only MLDE methods can overlook spatial context and functionally relevant interactions encoded by protein structures.Structure provides signals including spatial proximity and functional motifs.Source: §1 · p. 1
  • 9.2% on AAV-hard and 16.3% on GFP-hard are the relative fitness gains over prior strongest baselines; StructEvo also achieves the highest GFP-hard fitness with comparable diversity.The experiments cover two combinatorial and two full-length benchmarks, and the GFP triple-mutation epistasis pattern was validated by existing wet-lab experiments.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • Reliable mutant structures are scarce, and joint selection of mutation positions and amino-acid substitutions creates a large action space.The paper notes that structure prediction is costly and can be insensitive to point mutations; it therefore models structural changes at the feature level and separates position selection from amino-acid refinement.Source: §1 · p. 2
  • StructEvo combines a delta-structure fusion encoder, a structure-aligned hierarchical action network, and a geometric constraint to incorporate dynamic structural knowledge into mutation proposals.The encoder approximates mutant structure features via delta representations and integrates sequence and structure information.Source: §1 · p. 2, Figure 2 · p. 3

2 Related work

Prior MLDE methods include evolutionary search, energy-based approaches, and generative models, while related work also studies RL and multimodal fitness prediction. StructEvo distinguishes itself by incorporating structural knowledge into mutation proposals for dynamic evolution.

2 Related work

  • Earlier MLDE methods use evolutionary search, energy models, or generative models, while StructEvo adds structural guidance to high-fitness mutant proposals.The cited approaches include distance-penalized exploration, gradient ascent, variational autoencoders, generative flow networks, and controllable generation.Source: §2 · p. 3
  • Prior protein-optimization RL methods use self-play with Monte Carlo tree search, amino-acid knowledge-graph priors, or continuous latent-space perturbations; StructEvo uses delta representations for structural variation.The passage cites Wang et al., Wang et al., and Lee et al. [13] for these respective approaches.Source: §2 · p. 3
  • Multimodal fitness-prediction work uses wild-type structures, sequence-structure fusion, surface topology, or MSA, whereas StructEvo targets dynamic protein evolution in active learning.Prior representations include graph neural networks, inverse-folding models, joint vocabularies, and structure quantization.Source: §2 · p. 3

3 Task formulations

The task formulates directed evolution as iterative candidate evaluation and pool refinement under limited oracle queries, with a proxy guiding mutation proposals. Its RL formulation selects single-point mutations sequentially and trains a proposal policy using proxy rewards.

3.1 Directed evolution in an active learning setting

  • The oracle maps a protein sequence to scalar fitness, but costly evaluations restrict queries to N per round, with N much smaller than the sequence space.In MLDE, oracle evaluations correspond to wet-lab experiments.Source: §3.1 · p. 4
  • Newly proposed mutants are evaluated and incorporated into the candidate pool at each round.The initial pool contains only a small set of low-fitness candidates.Source: §3.1 · p. 4
  • A trainable proxy is iteratively fine-tuned on the updated pool to approximate the oracle-defined fitness landscape.The proxy is trained by minimizing mean squared error.Source: §3.1 · p. 4
  • The proposal policy uses the current pool and fine-tuned proxy to generate N novel mutants per round for oracle evaluation.The proposal policy is the component the work aims to improve.Source: §3.1 · p. 4, §3.2 · p. 4
  • The objective is to identify K high-fitness protein mutants within a fixed budget of R rounds.The process begins with an initial pool and iterates over rounds of evaluation and refinement.Source: §3.1 · pp. 3–4

3.2 Reinforcement learning-based directed evolution

  • RL models multi-point mutations as sequential single-point decisions, with each action specifying a mutation position and substituted amino acid.The task is formulated as a Markov decision process with deterministic state updates.Source: §3.2 · p. 4
  • The proxy supplies sparse rewards only at trajectory termination, while the oracle is reserved for final candidate evaluation.A trajectory ends when a mutant beats its starting sequence or reaches the predefined maximum number of steps.Source: §3.2 · p. 4
  • The policy-training objective maximizes expected return over mutation trajectories, and PPO is used for optimization.A trajectory is a sequence of states and actions terminating at step T; γ is the discount factor.Source: §3.2 · p. 4

4 Methodology

StructEvo incorporates structural knowledge into protein mutation policies through delta-feature fusion, hierarchical mutation sampling, and a geometric constraint.

4 Methodology

  • StructEvo combines sequence–structure fusion, structure-aligned hierarchical sampling, and a geometric constraint on delta features in its mutation policy.The framework maps protein states to mutation actions.Source: §4 · p. 5

4.1 Delta-structure fusion encoder

  • A delta-structure fusion encoder projects sequence-feature differences into structural space and combines approximate mutant structure features with sequence features through cross-attention.It uses a reference structure as a scaffold because reliable mutant structures are generally unavailable.Source: §4.1 · p. 5

4.2 Structure-aligned hierarchical action network

  • The hierarchical action network first samples a mutation position from the fused feature, then samples the substituted amino acid conditioned on that position.The two decisions decompose mutation selection into position and amino-acid-type subspaces.Source: §4.2 · pp. 5–6

4.3 Geometric constraint for delta features

  • A per-residue geometric constraint regularizes sequence- and structure-delta features to encourage stable, faithful mapping, jointly with the mutation-policy objective.The objective combines this consistency constraint with standard PPO loss terms.Source: §4.3 · p. 6

5 Experiments

StructEvo is evaluated on combinatorial and full-length benchmarks, then analyzed through component ablations and a GFP epistasis case study.

5 Experiments

  • Experiments span two four-site combinatorial datasets and two full-length benchmarks, with five independent runs per setting.The combinatorial evaluation reports mean and maximum fitness and, because ground-truth fitness is available, proxy-ranking quality using NDCG.Source: §5 · p. 6, §5.1 · pp. 6–7, §5.2 · pp. 7–8

5.1 Optimization on combinatorial benchmarks

  • At round 3, StructEvo exceeds KnowRLM by 5.7% NDCG on GB1 and gains 3.7% NDCG plus 18.8% mean fitness on PhoQ.PhoQ uses an AlphaFold2-predicted reference structure; the authors report comparable performance with KnowRLM and note the importance of accurate structural information for landscape exploration.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2 Optimization on full-length benchmarks

  • StructEvo achieves state-of-the-art fitness across all four full-length scenarios, with relative mean-fitness gains of 9.2% on AAV-hard and 16.3% on GFP-hard.LatProtRL converges earlier, but StructEvo continues improving and ultimately surpasses it. Hard-setting improvements exceed medium-setting improvements; higher diversity or novelty alone does not imply higher fitness.Source: Table 2 · p. 8, §5.2 · pp. 8–9, Figure 3 · p. 8

5.3 Ablation studies

  • Removing structure causes larger drops—9.6% mean fitness on AAV-hard and 16.0% on GFP-hard—while flat joint sampling reduces fitness and increases runtime.Removing the delta module also degrades performance; substituting reference sequence for structure yields no improvement.Source: Table 3 · p. 9, §5.3 · p. 9

5.4 Case study: a triple-mutation epistasis in GFP

  • In GFP, 271 F64L/V163A/I171V mutants averaged 0.91 fitness, versus 0.07 for 1,834 single mutants and 0.23 for 427 double mutants.Wet-lab experiments validated and explained the pattern: smaller side-chains at residues 64, 163, and 171 facilitate tighter packing and photon-emission energy dissipation.Source: §5.4 · p. 9, Figure 4 · p. 9

6 Conclusions

StructEvo combines structure-aware mutation proposals with an active-learning and reinforcement-learning pipeline. The conclusion reports consistent gains across combinatorial and full-length benchmarks while identifying scope limits and safety requirements.

6 Conclusions

  • StructEvo uses delta representations and a hierarchical action network to incorporate structural knowledge into mutation proposals, with consistent improvements across combinatorial and full-length benchmarks.The conclusion also calls for extending the framework to protein complexes, multi-objective properties, and wet-lab feedback.Source: §6 · p. 10
  • The authors limit the stated scope to monomers and single-objective fitness, leave wet-lab feedback for future work, and require experimental validation and human inspection before practical use.These are stated limitations and safety precautions, not reported failures of the method.Source: §6 · p. 10

A.1 Symbol definitions

  • The symbol table defines notation used throughout the paper.The supplied table caption does not specify individual symbols.Source: Table 4 · p. 15

A.2 Algorithm for StructEvo framework pipeline

  • The pipeline alternates proxy fine-tuning and candidate selection in an active-learning loop with trajectory collection, reward assignment, and policy updates inside each round.The outer loop evaluates selected mutants with an oracle; the inner loop performs reinforcement learning.Source: §A.2 · p. 15

A.3 The detach operator

  • Detaching the localized feature blocks amino-acid-choice gradients from the fusion module, giving the position network a cleaner, more stable learning signal.The authors state this decomposition can reduce learning complexity and improve data efficiency under limited experimental feedback.Source: §A.3 · pp. 15, 17

A.4 The geometric constraint

This appendix explains the geometric constraint and outlines related modeling approaches and benchmark settings. It describes combinatorial datasets with broad four-site coverage and full-length tasks divided by initial-pool difficulty.

A.4 The geometric constraint

  • The per-residue geometric loss constrains the mapping from delta sequence features to delta structure features; the authors say it enforces a stronger condition than Lipschitz continuity and stabilizes RL learning signals.The mapping is defined between the delta sequence and delta structure manifolds, and the loss encourages consistent sensitivity.Source: §A.4 · pp. 17–18

B Protein fitness landscape modeling

  • Protein fitness models include sequence-based approaches, structure encoders such as graph neural networks, inverse-folding models, and hybrid sequence-structure methods.The cited hybrid examples use a structure-aware vocabulary or a structure-quantization module with disentangled attention.Source: B Protein fitness landscape modeling · p. 18

C.1 Combinatorial mutation benchmarks

  • GB1 and PhoQ measure distinct functions, and their datasets nearly exhaustively cover the 160,000-variant four-site combinatorial space.GB1 measures binding affinity to IgG-Fc; PhoQ measures kinase and phosphatase responsiveness in the PhoP/PhoQ system.Source: §C.1 · p. 19

C.2 Full-length mutation benchmarks

  • AAV and GFP benchmarks are split into medium and hard tasks by initial-pool fitness percentile and minimum pairwise Hamming distance.Medium pools use the 20th–40th percentiles and distance 6; hard pools use the bottom 30th percentile and distance 7.Source: §C.2 · p. 19

C.3 Baselines for combinatorial benchmarks

This appendix inventories baselines and evaluation measures for combinatorial and full-length benchmarks, then documents the reference structures used. The listed methods span evolutionary search, proxy-guided optimization, and latent-space approaches.

C.3 Baselines for combinatorial benchmarks

  • Combinatorial baselines include MCTS, PPO with an amino-acid knowledge graph, continuous-relaxation optimization, Bayesian optimization, greedy search, proxy-model methods, hierarchical clustering, and proximal exploration.The passage names EvoPlay, KnowRLM, CMA-ES, BO, AdaLead, MLDE, ftMLDE, CLADE, CLADE 2.0, and PEX.Source: §C.3 · pp. 19–20

C.4 Baselines for full-length benchmarks

  • Full-length baselines include graph-smoothed sampling, latent-space PPO, flow-matching optimization, and controllable GFlowNets; latent-space methods were excluded from combinatorial comparisons for compatibility and fairness.The named methods are GGS, LatProtRL, VLGPO, and GFN-δ-CS.Source: §C.4 · pp. 20–21

C.5 Metrics for combinatorial benchmarks

  • Combinatorial mean and maximum fitness use a 480-candidate set, while NDCG compares proxy predictions with ground-truth fitness across the full 160,000-variant space.The set combines 96 initial mutants, 288 proposed over three rounds, and 96 final proxy-ranked mutants.Source: §C.5 · p. 21

C.6 Metrics for full-length benchmarks

  • Full-length evaluation reports mean and maximum fitness among the top 128 proposed sequences, plus diversity and distance from the initial set measured using Hamming distances.The passages define diversity as mean pairwise distance and also specify mean minimum distance to the initial set.Source: §C.6 · p. 21

D.1 Structure process

  • GB1, AAV, and GFP use experimental structures, whereas PhoQ uses a high-confidence AlphaFold prediction with average pLDDT 82.94.The PhoQ crystal structure was unavailable in the PDB.Source: §D.1 · p. 21

D.2 Training details

The appendix specifies sequence and structure encoders, proxy-model training, and PPO settings. It points to three-round GB1 and PhoQ results tables but provides no result values in the supplied passages.

D.2 Training details

  • Training uses ESM-2 for sequence encoding and ESM-IF for structure encoding, with benchmark-specific proxy models fine-tuned for 30 epochs after each round.The combinatorial proxy is ProSST; full-length tasks use provided checkpoints of a smoothed CNN proxy.Source: §D.2 · p. 22
  • PPO training uses eight parallel environments, a 0.3 clipping range, 0.99 discount factor, 3,000 or 15,000 steps by benchmark type, and geometric-constraint weight 0.05.The scaling factor c is set to 1.Source: §D.2 · p. 22

E.1 Full results on 4-site benchmarks

E.2 Full ablation results on full-length benchmarks

The full-length benchmark ablations compare four component-removal or replacement settings, while the additional analyses cover structure prediction models, proxy robustness, and geometric-constraint weights.

E.2 Full ablation results on full-length benchmarks

  • The ablation study tests removing the delta module, removing structure, using mutant sequence alone, and replacing the hierarchical action network with a flat network.The full results are presented in Table 11.Source: §E.2 · p. 23, Table 11 · p. 23
  • Results on the two medium settings show trends consistent with those on the hard setting.The text refers readers to Sec. 5.3 for analysis of those trends.Source: §E.2 · p. 23

F Additional in-depth analysis

F.1 Analysis on structure prediction models

The analysis finds weak relationships between predicted structural differences and fitness, and between structure-confidence metrics and fitness, on sampled GFP mutants.

F.1 Analysis on structure prediction models

  • The GFP sample comprises 100 mutants selected across 10 fitness bins, with an average Hamming distance of 7.5.Only one predicted structure was generated per sequence; the 100 predictions took approximately 13 hours, making full-scale prediction prohibitive inside the RL loop.Source: §F.1 · p. 23
  • All three confidence metrics correlate weakly with fitness; the highest reported correlation is 0.29 for pLDDT-Pearson.Figure 6 presents the confidence-score comparisons.Source: Figure 6 · p. 24, §F.1 · p. 24
  • The analysis compares reference-based and pairwise structural differences with fitness using Cα-RMSD and Pearson and Spearman correlations.Pairwise analysis covers 4,950 sequence pairs and uses absolute fitness gaps.Source: §F.1 · p. 24
  • Reference-based structural differences correlate weakly negatively with fitness (Pearson -0.19, Spearman -0.12), while pairwise differences show near-zero correlations (Pearson 0.03, Spearman 0.02).Figure 7 presents the reference-based and pairwise comparisons.Source: Figure 7 · p. 24, §F.1 · pp. 24–25
  • The authors suggest periodically updating the reference structure, but say update timing requires careful evaluation and leave it to future work.Examples include updating every T steps or after the accumulated mutation distance passes a threshold.Source: §F.1 · p. 25

F.2 Robustness analysis of the proxy model

On GFP-hard, the proxy-model analysis compares the default proxy-reward setup with unconstrained and query-limited oracle alternatives.

F.2 Robustness analysis of the proxy model

  • In the default setting, the proxy supplies RL rewards, selects top-N candidates for oracle evaluation, and is iteratively refined with those evaluations.The oracle-full and oracle-limited settings replace the proxy with the oracle, with the latter retaining the default per-round query limit.Source: §F.2 · p. 25
  • Oracle-full performs best as an idealized upper bound, whereas oracle-limited shows a significant performance drop.Table 12 reports the proxy robustness analysis on GFP-hard.Source: Table 12 · p. 25, §F.2 · p. 25
  • Oracle-full is impractical because oracle queries consume costly wet-lab experimental budgets.Source: §F.2 · p. 25
  • The authors conclude that frequent proxy rewards are crucial for effective evolution under limited oracle signals.They attribute the oracle-limited drop to insufficient trajectories collected by the RL agent.Source: §F.2 · p. 25

F.3 Sensitivity analysis of the geometric constraint weight

The geometric-constraint weight analysis on GFP-hard favors λgeo = 0.05, while the paper checklist reports experimental reproducibility and disclosure details.

F.3 Sensitivity analysis of the geometric constraint weight

  • Increasing λgeo generally improves training stability, reflected in reduced standard deviations across mean and maximum fitness metrics.Source: §F.3 · p. 25
  • The authors suggest larger weights overly restrict policy updates, while smaller weights provide insufficient regularization; novelty also decreases as λgeo grows.They associate stronger constraints with more proximal candidates.Source: §F.3 · pp. 25–26
  • λgeo = 0.05 achieves the best performance on fitness metrics; both larger and smaller weights degrade performance.The experiments vary λgeo on GFP-hard over five independent runs; Table 13 reports the analysis.Source: §F.3 · p. 25, Table 13 · p. 26

NeurIPS Paper Checklist

Related papers

Loading 2609.39048v1…