PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design

Mingquan Liu, Jiangyu Chen, Hanqun Cao, Xujun Zhang, Pengsen Ma, Xiangru Tang, Shuting Jin, Zhuo Yang, Tianfan Fu, Fang Wu, Xiangxiang Zeng

arXiv:2608.26747v1cs.AI

TL;DR

AgentFold asks whether LLM agents can improve tightly coupled scientific ML systems through executable changes and expensive validation, using protein folding as the testbed. It performs closed-loop multi-agent search over executable ESMFold variants, and at matched budget achieves 7.5% higher best lDDT than independent Codex proposals while revealing recurring intervention patterns.

  • Problem

    It remains unclear whether LLM agents can autonomously improve tightly coupled scientific ML systems that require executable implementation, failure recovery, expensive validation, and domain-aware interpretation.

  • Method

    AgentFold uses a multi-agent closed-loop search that proposes, implements, debugs, evaluates, analyzes, and stores executable folding-model interventions under an MCTS-style controller.

  • Results

    7.5% higher best lDDT than an independent Codex-proposal baseline was achieved at matched evaluation budget, while AgentFold also outperformed random search.

  • Takeaways & Limitations

    Intervention traces suggest stable gains frequently co-occur with early soft learnable priors and gated refinement, whereas direct geometric perturbations and geometry-conditioned feedback often destabilize training.

  • Takeaways & Limitations

    Evidence is limited to a compact one-block ESMFold-derived codebase, a 1,000-chain training subset, and CAMEO2022 development evaluation; transfer to stronger systems and broader biological settings remains unverified.

Abstract

from arXiv · show

Scientific LLM agents have shown promise in literature reasoning, tool use, and experiment planning, but it remains unclear whether they can autonomously improve large, tightly coupled scientific machine-learning systems through executable code changes and computationally expensive validation. We study this question in protein folding, where progress requires coordinated architectural modifications, multi-objective evaluation, and domain-aware interpretation. We present AgentFold, a multi-agent framework that formulates folding-model development as a closed-loop search over executable code variants. Starting from ESMFold, AgentFold proposes hypotheses, implements and debugs code-level modifications, evaluates model variants, analyzes experimental outcomes, and stores both successful and failed interventions in structured memory. An MCTS-style policy allocates computational resources across high-scoring search branches. On an engineering-scale protein-folding codebase comprising more than 2,000 lines of code, AgentFold explores approximately 80 model variants using approximately 5,000 GPU-hours and 170 million LLM tokens. Under a matched computational budget, AgentFold improves the best lDDT by 7.5% over independent Codex proposals and outperforms a random-search control. Beyond model improvement, the resulting intervention traces reveal recurring empirical design patterns: stable gains tend to arise from early, soft, learnable priors and gated refinement, whereas direct geometric perturbations and geometry-conditioned feedback often destabilize training. The code and experimental resources are publicly available at https://github.com/lmqfly/AgentFold.

1 Introduction

AgentFold addresses whether LLM agents can autonomously improve tightly coupled scientific ML systems through executable changes and costly, multi-objective validation. In protein folding, it performs closed-loop code search and identifies both performance gains and recurring intervention patterns.

  • Execution-grounded scientific ML requires agents to implement changes, recover from failures, and compare expensive, noisy, multi-objective experiments.
  • Protein folding tests closed-loop development because models couple sequence and pair representations, geometric refinement, recycling, structure losses, and local and global metrics.
  • AgentFold searches executable folding-model variants through propose–implement–debug–evaluate cycles, retaining successful and failed interventions in structured memory.
  • 7.5% higher best lDDT than an independent Codex-proposal baseline was achieved at matched evaluation budget, while also outperforming random search.The system explored roughly 80 variants on an engineering-scale codebase.
  • Stable gains were associated with early soft learnable priors and gated refinement, whereas direct geometric perturbations and geometry-conditioned feedback often destabilized training.

2 Related Work

Related work spans LLM-enabled scientific workflows, execution-grounded experimentation, program and architecture search, and protein-structure prediction platforms. AgentFold combines these ideas in a domain-specific protein-folding setting requiring executable, validated interventions.

  • LLM systems support literature synthesis, hypothesis generation, tool use, scientific workflows, and biomedical research planning.
  • Execution-grounded benchmarks show that long-horizon machine-learning experimentation and scientific-code development remain difficult despite automatically checkable outcomes.
  • Program-evolution, heuristic-design, and neural-architecture-search methods provide general components that AgentFold instantiates for protein-folding model development.
  • Protein-structure research includes MSA-based and unified complex predictors, open trainable platforms, and MSA-free language-model-based prediction.

3 Method

AgentFold conducts autonomous folding-model development as MCTS-style search over executable code variants, combining iterative experimentation, memory, debugging, analysis, and periodic policy updates. Its design records intervention outcomes to support attribution and recurring-pattern mining.

  • Framework: AgentFold searches executable code-level interventions while simultaneously summarizing recurring empirical design patterns from repeated outcomes.
  • Framework: Each intervention trace records the parent variant, typed edit, code diff, stability signals, and metric deltas for cross-variant attribution.
  • Search control: The dual-loop design combines continuous sampling, evolution, experimentation, and analysis with batched periodic refinement of the search tree and candidate sets.
  • Memory and analysis: Database-backed memory stores code snapshots, diffs, configurations, logs, attributions, analyses, literature, and failure modes for subsequent proposals.
  • Search control: An MCTS-style controller organizes executable variants as tree nodes and allocates exploration using high-scoring nodes, diverse references, folding metrics, and normalized search utility.
  • Implementation loop: A specialized evolution chain deduplicates proposals, unifies architectural planning with coding, and interactively debugs syntax or runtime errors until training launches successfully.
  • Memory and analysis: Postprocessing streams training logs to an Analyst agent, which attributes metric and stability changes and updates candidate patterns with support, refutation, or qualification.
  • Search control: Every 10 iterations, batched metric-parser and Critic-Agent scores update tree values, using critic scores to prioritize expensive experiments rather than claim final improvements.

4 Results

AgentFold’s closed-loop search produced consistent local-structure gains across executable folding variants, while targeted analyses linked success to early soft priors and gated refinement. Global-fold improvements were less uniform, and follow-up tests examined robustness, component dependence, and loop behavior.

  • Search and evaluation: AgentFold explores folding-model variants through benchmark-guided executable-code search and analyzes both successful and failed interventions.The evaluation uses a compact ESMFold-derived baseline, CAMEO2022 scoring, and multiple structural metrics.
  • Search-controller comparison: 36 evaluations per controller support AgentFold over random selection and independent Codex proposals under matched search conditions.The controllers share the edit space, models, prompts, checks, training, and evaluator, while AgentFold additionally uses tree search and intervention history.
  • Quantitative results: Representative variants improve NWRS by +0.007 to +0.026 and mean lDDT by +0.016 to +0.053 relative to ESMFold.esmfold_struct_enhanced_v4 has the largest composite gain (+0.026) and largest mean/median lDDT gains (+0.053/+0.059).
  • Targeted metrics: AgentFold’s gains concentrate on local and medium-range structure rather than broad global-fold improvement.Targeted results include a largest loop-lDDT gain of +0.063, a MolProbity reduction of -0.157, and selective contact improvements.
  • Evolutionary analysis: High-performing variants place soft, learnable priors before coordinates are instantiated, whereas direct geometric perturbations and geometry-conditioned feedback are associated with collapse.The analysis attributes the contrast to steering attention or update magnitudes versus imposing direct geometric changes in an already coupled refinement process.
  • Robustness and ablation: The strongest variant remains robust in repeated runs and deeper trunks, while removing either IPA bias or BackboneUpdate gating reduces mean lDDT.Mean lDDT rises from 0.238 to 0.274 with one trunk layer and from 0.321 to 0.355 with eight; ablations lower mean lDDT by 0.017 and 0.012.

5 Conclusion

AgentFold uses closed-loop search over executable folding-model code variants and identifies stable empirical design patterns associated with local-accuracy gains and preserved global fold quality.

  • AgentFold formulates folding-model development as closed-loop search over executable code variants.The framework searches through propose–implement–debug–evaluate cycles rather than limiting exploration to textual hypotheses.
  • AgentFold identifies parameter-efficient variants with consistent gains, primarily in local structural accuracy, while largely preserving global fold quality.
  • Early soft learnable priors and gated refinement are associated with more stable gains in the search.
  • Direct geometric perturbations and geometry-conditioned feedback often destabilize training.

Limitations

The evidence is bounded to a compact ESMFold-derived setting and does not establish transfer to stronger folding systems or broader biological applications.

  • The evaluation uses a one-block, compact ESMFold-derived codebase, a 1,000-chain training subset, and CAMEO2022 development-benchmark evaluation.
  • Transfer to stronger folding systems and broader biological settings remains unverified.
  • Extending the interventions to larger and multi-chain systems requires model-specific interfaces, chain-aware representations, interface-sensitive objectives, retraining, and evaluation.
  • Cross-domain use requires a domain-specific codebase, evaluator, reward, and failure-analysis loop.

Ethical Considerations

AgentFold may support biological and medical research through improved structure prediction, while increased AI-for-biology capability introduces dual-use risks requiring responsible safeguards.

  • Better structure prediction can support biological and medical research.
  • Increased AI-for-biology capability may introduce dual-use risks.
  • Responsible release, careful evaluation, and human oversight are therefore important.

A Experiment Details

Experiments use a one-layer ESMFold Folding Trunk and fixed training setup to enable broad variant exploration while attributing differences primarily to architectural interventions.

  • A.1 Model details: All experiments use a single ESMFold Folding Trunk block unless otherwise noted, reducing per-variant training and evaluation cost.The trunk operators, including triangular multiplicative updates and triangle attention, remain unchanged.
  • A.1 Model details: The experiments modify trunk depth while leaving all trunk operators unchanged.
  • A.1 Model details: The ESMFold codebase was consolidated into one implementation file without changing architecture, parameterization, or numerical behavior.The refactoring covers the Structure Module and trunk components used in the experiments.
  • Training setup: Variants are trained for 150 epochs with Adam, batch size 8, and a peak learning rate of 1 × 10−3.Training hyperparameters are kept fixed across variants so performance differences primarily reflect architectural interventions.

A.2 Mini-data curation

The mini-data curation procedure samples nonredundant single protein chains while controlling protein-family representation and sequence-length distribution.

  • Training data are derived from single protein chains in the Protein Data Bank.
  • Chains are clustered by sequence identity using a 0.4 minimum identity threshold with MMseqs2.
  • Sampling down-weights over-represented families using an inverse family-size factor and favors moderate-length sequences.
  • The resulting procedure produces a more diverse training set while controlling redundancy and sequence-length distribution.
  • The study uses publicly available research artifacts under their respective licenses and terms of use.Artifacts include ESMFold/ESM code and weights, OpenStructure, MMseqs2, PDB-derived structures, and CAMEO2022 data.

A.4 Metric definitions

The paper evaluates protein structures with local, global, oligomeric, loop, contact, and physical-plausibility metrics, then combines ten aggregated measures into NWRS.

  • Local and global structure metrics: lDDT measures local distance agreement without structural superposition, while backbone lDDT restricts the calculation to backbone atoms.The lDDT thresholds are 0.5, 1.0, 2.0, and 4.0 Å, and higher scores are better.
  • Local and global structure metrics: GDT-TS averages GDT scores at 1, 2, 4, and 8 Å cutoffs after superposition.
  • Local and global structure metrics: Oligomeric GDT-TS applies the analogous multi-cutoff calculation to the full multi-chain complex after all-chain superposition.
  • Local and global structure metrics: RMSD measures average aligned-atom deviation after optimal rigid-body alignment, whereas TM-score is a length-normalized global similarity metric.RMSD is lower-is-better; TM-score uses target length, aligned residues, pair distances, and a length-dependent normalization constant.
  • Targeted metrics: Targeted evaluation covers loop-region accuracy, contact precision/recall/F1 across sequence-separation bins, and physical-plausibility measures.
  • Aggregate scoring: NWRS baseline-normalizes and weights ten mean-and-median metrics, maps an exact baseline match to 0.5, and uses uniform weights of 0.1.The ablation variant instead maps the setting-matched ESMFold baseline to 0.500.
  • Aggregate scoring: All metrics are oriented so larger values indicate better performance, with RMSD treated as the exception because lower values are better.
  • Aggregate scoring: The reported TM-score distribution has mean 0.648 and median 0.693.

B.3 Targeted Evaluation Details

The targeted-evaluation section reports complete metric summaries for loop regions, physical plausibility, and contacts, alongside a variant index and parameter-count listing.

  • Loop-region evaluation: Table 6 provides complete loop-region metrics, with absolute ESMFold means and other rows reporting changes relative to ESMFold.
  • Physical plausibility: Table 7 provides complete physical-plausibility metrics using the same absolute-versus-relative reporting convention.
  • Contact evaluation: Table 8 provides complete contact metrics using the same absolute-versus-relative reporting convention.
  • Variant inventory: Table 9 lists each variant with its corresponding index and parameter count.

B.4 Architecture Comparison

The architecture comparison preserves ESMFold’s iterative structure-refinement stack while adding learned biases conditioned on residue position, sequence separation, and chunk boundaries, plus gated backbone updates.

  • Structure module: ESMFold uses 8 IPA blocks over single and pair representations, followed by Backbone Update, Angle ResNet, and Frame geometric heads.These shared heads iteratively refine backbone frames and torsions.
  • Structure module: The variant prepends a residue-index-conditioned bias MLP to each IPA block while preserving the existing geometric refinement heads.The MLP injects a learned, position-aware bias based on residue indices.
  • Invariant Point Attention: IPA attention logits combine content similarity, static pairwise bias, an SE(3)-invariant point term, and masking.The variant retains the same IPA core while adding learned bias terms conditioned on current state and sequence separation.
  • Trunk chunk-boundary bias: Chunk-boundary bias strengthens cross-chunk communication at axial-attention interfaces, while a low-magnitude bias remains active without chunking.The bias is kept active when chunking is disabled so its parameter continues to train.
  • BackboneUpdate gating: BackboneUpdate gating applies a learned gate to the raw 6-dimensional rigid-body update to stabilize iterative refinement.The raw update is denoted ∆∈R6, and the learned parameter is denoted g∈R6.
Loading 2608.26747v1…