PaperlayerView exact revision on arXiv

Source-linked AI summary

PROTEAN: Beyond De Novo Generation - Can LLM Agents Perform Text-Guided Protein Structure Editing?

Encheng Su, Xinping Liu, Yizhou Wang, Bozitao Zhong, Pengze Li, Jianyu Wu, Mingxiao Li, Xinzhu Ma, Aoran Wang, Liang Hong, Wanli Ouyang, Chen Tang, Houqiang Li, Shixiang Tang

arXiv:2610.17765v1cs.CE

TL;DR

Protein engineering often requires targeted edits to existing proteins, but current generators do not directly interpret natural-language goals and no standardized text-guided 3D editing benchmark exists. PROTEAN evaluates an LLM-to-backend editing pipeline across 380 tasks; even the top agent reaches only 46.0% joint geometric compliance, with biophysical refolding posing a steeper barrier.

  • Problem

    No standardized benchmark formalizes and evaluates natural-language-guided 3D editing of existing proteins under strict preservation constraints.

    Source: §1 · p. 2

  • Method

    PROTEAN benchmarks 380 text-guided editing tasks using an LLM to translate instructions into backend-specific constraints and a generator to remodel the backbone.

    Source: §1 · pp. 2–3

  • Results

    46.0% joint geometric compliance (G1 ∧G3) for Claude-Opus-5 versus below 28% for competitor baselines, while end-to-end biophysical refolding presents a steeper sampling barrier.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PROTEAN provides a diagnostic testbed for developing autonomous agents that perform multi-backend macromolecular design under strict physical constraints.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Generative AI has achieved remarkable success in de novo protein design, yet protein engineering typically begins with existing, characterized proteins. The primary demand is targeted local editing, e.g., engineering a specific region to optimize affinity, alter immunogenicity, or improve stability, while strictly preserving core catalytic motifs and overall scaffold integrity. However, no standardized framework currently exists for carrying out and evaluating such edits. In current generative-design workflows, researchers often need to manually specify complex low-level parameters and rely on repetitive, trial-and-error sampling and evaluation across generative backends to search for viable conformations. While large language models (LLMs) offer an intuitive interface to express these goals using natural language, prevailing structure-generation models cannot directly interpret high-level human instructions. To bridge this gap, we introduce PROTEAN, a standardized benchmark for text-guided protein structural editing. PROTEAN comprises 380 sub-tasks spanning six intervention families and evaluates a two-stage pipeline in which an LLM translates natural-language goals into backend-specific geometric constraints and a generative model remodels the protein backbone. A two-tier evaluation separates geometric compliance, including target attainment and off-target scope fidelity, from biophysical realizability assessed by inverse folding and refolding consistency. Across five frontier LLMs paired with three backends, strict end-to-end biophysical success starts at just 7.0%. Ablation studies show that LLM constraint compilation is already highly accurate, whereas the main bottlenecks are backend-specific scope leakage and refolding failure. PROTEAN provides an open testbed for developing autonomous, multi-backend macromolecular design agents.

1 INTRODUCTION

Protein engineering often needs localized edits to existing proteins, but current generators require low-level constraints and do not directly interpret natural-language instructions. PROTEAN formalizes this task and benchmarks an LLM-plus-generator workflow, finding that geometric and biophysical success remain substantially bottlenecked.

1 INTRODUCTION

  • The main demand is editing existing, evolutionarily vetted proteins while preserving catalytic motifs and scaffold integrity, rather than generating proteins from scratch.Current workflows require manual low-level parameter configuration and trial-and-error sampling across fragmented backends.Source: §1 · p. 2
  • 46.0% joint geometric compliance (G1 ∧G3) for Claude-Opus-5 contrasts with below 28% for competitor baselines, while end-to-end refolding remains a steeper sampling barrier.The reported benchmark pairs five frontier LLMs with three generative backends.Source: §1 · p. 2
  • The reported bottlenecks include generator-specific non-local contact strain, scope leakage, and persistent Pareto deadlocks during complex remodeling.The authors identify backend coordinate leakage and refolding collapse as primary bottlenecks for autonomous macromolecular design agents.Source: §1 · pp. 2–3
  • PROTEAN formalizes text-guided editing under hard conservation boundaries, connecting natural-language goals to low-level generative backends across eight structural intervention families.Its two-tier evaluation separates geometric compliance from biophysical realizability through inverse folding and refolding consistency.Source: §1 · pp. 2–3
  • The benchmark comprises 380 verified editing sub-tasks across six intervention families and eight concrete tasks, with evaluation separating geometric compliance from biophysical realizability.The geometric tier checks target attainment and off-target scope fidelity; the biophysical tier tests whether edited backbones can stably self-assemble after sequence redesign and refolding.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work provides de novo structure generators, language-driven protein-design methods, and benchmarks for isolated design stages. PROTEAN addresses the lack of a benchmark for text-guided 3D editing that distinguishes geometric fidelity from biophysical plausibility.

2.1 STRUCTURE GENERATION AND HETEROGENEOUS DESIGN BACKENDS

  • Generative backends have specialized capabilities and require expert-built, backend-specific inputs; PROTEAN evaluates whether LLMs can select these tools and compile their incompatible specifications.BoltzGen handles cyclization, ODesign supports partial diffusion and distance modulation, and RFdiffusion3 provides generalized backbone generation under spatial bounds.Source: §2.1 · p. 3

2.2 LANGUAGE-DRIVEN PROTEIN ENGINEERING

  • Existing language-driven methods edit protein sequences, while text-guided 3D structural editing remains distinct because sequence-only approaches lack explicit geometric constraints and spatial preservation.Related scientific-agent benchmarks largely target general coding or de novo binder-generation workflows.Source: §2.2 · p. 4

2.3 PROTEIN DESIGN BENCHMARKS & DECOUPLED EVALUATION

  • Existing protein-design benchmarks focus on isolated stages and can conflate geometric non-compliance with biophysical instability, whereas PROTEAN separates structural fidelity from downstream plausibility.This decoupled protocol is intended as a fine-grained diagnostic benchmark for autonomous protein engineering.Source: §2.3 · p. 4

3 THE PROTEAN BENCHMARK

PROTEAN benchmarks natural-language-directed structural interventions on existing proteins under explicit preservation contracts. Its task taxonomy, curated examples, and multi-gated protocol assess both whether edits meet geometric objectives and whether they remain biophysically plausible.

3 THE PROTEAN BENCHMARK

  • PROTEAN comprises eight task categories and 380 sub-tasks, including seven length-preserving editing tasks and one variable-length motif-scaffolding task.The benchmark evaluates targeted intervention on an existing protein while preserving its unedited scaffold.Source: §3 · p. 4

3.1 TASK FORMULATION

  • Each task pairs a source sequence and backbone with a natural-language instruction and a residue mask partitioning editable, relaxation-allowed, and strictly protected regions.Length-preserving tasks retain source length, topology, and residue ordering; protected residues retain identity and backbone geometry within 0.1 Å after anchored superposition.Source: §3.1 · p. 4

3.2 TASK TAXONOMY AND CRITERIA

  • Six intervention families span secondary-structure enrichment, inter-domain distance modulation, fragment mimicry, core burial, cyclization, and motif scaffolding under task-specific geometric criteria.Together, the tasks probe local structure and packing, domain motion, chain topology, and functional motif scaffolding.Source: §3.2 · p. 5

3.3 BENCHMARK CURATION PIPELINE

  • The curation funnel collects and deduplicates structures, assigns candidate edit types, then retains only source–task pairs that pass feasibility checks.The resulting benchmark contains 380 cases, including 350 length-preserving tasks and 30 motif-scaffolding cases; protein blinding is used to reduce potential pretraining leakage.Source: §3.3 · pp. 5–6

3.4 MULTI-GATED EVALUATION PROTOCOL

  • The evaluation gates test target attainment, refolding self-consistency, and scope compliance; overall success requires passing all applicable gates.Refolding uses a fixed redesign-and-refold loop and is a finite-budget proxy, so failure indicates a sampling miss rather than definitive biological incompatibility; execution failures are reported separately.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate five LLMs across three generation backends on structural editing and motif scaffolding, using task-specific geometric and biophysical validity criteria. Results expose substantial bottlenecks in joint geometric compliance, refolding success, and coordination of constraints.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five LLMs and three backends across 380 tasks, assessing editing by joint target attainment and scope compliance and motif scaffolding by joint geometric and refolding validity.Motif designs require a ProteinMPNN sequence that ESMFold refolds with whole-chain Cα-RMSD < 2.0 Å and motif Cα-RMSD < 1.0 Å.Source: §4.1 · p. 7

4.2 STRUCTURAL GEOMETRIC COMPLIANCE ACROSS TASK FAMILIES

  • Helix enrichment outperforms sheet enrichment by 20–30 percentage points across models, as non-local strand pairing and registry deform conserved flanking residues.Helices rely on local contiguous hydrogen bonds, whereas sheet design must coordinate strand geometry with the preserved framework.Source: §4.2 · p. 7
  • Fragment mimicry spans a four-fold performance gap, from 56.0% for Claude-Opus-5 to 14.0% for Kimi-K3, amid residue-mapping and rigid-body alignment demands.The task requires RMSD ≤2.0 Å over ≥80% coverage while preserving invariant boundary bounds; weaker models often misalign residue mappings or propose incompatible grafts.Source: §4.2 · p. 8
  • Inter-domain modulation and core burial expose tension between metric optimization and coordinate invariance, with rigid translations stressing protected cores and compaction compressing backbones.Core burial targets ∆RSA ≥0.10 to ≤0.40; these mechanical effects cap joint compliance across architectures.Source: §4.2 · p. 8
  • Head-to-tail cyclization reaches 100% on marginal G1 and G3 separately, but 0.0% joint compliance because closure relaxes the backbone beyond preservation bounds.The authors describe these objectives as mutually exclusive under rigid Euclidean bounds and conclude that cyclization requires concerted multi-scale deformation planning.Source: §4.2 · p. 8

4.3 FUNCTIONAL MOTIF SCAFFOLDING AND BIOPHYSICAL REALIZABILITY

  • At pass@16, Claude-Opus-5 and GPT-5.6-Sol achieve 43.0–47.0% self-consistent motif-scaffolding success, versus 23.0–33.0% for competitor baselines.Frontier models infer linker lengths that accommodate canonical secondary structures; over-compacted or excessively extended baseline linkers induce conformational tension and refolding collapse.Source: §4.3 · p. 9

4.4 MECHANISTIC FAILURE MODE PROFILING VIA LLM-AS-JUDGE

  • Failure profiling finds spatial planning failure at approximately 60% across models, while multi-constraint coordination failure ranges from 76.3% to 92.3%; categories may overlap.The analysis also identifies ring-closure coordinate displacement, boundary conflation, and hallucinated scaffold changes as failure cases.Source: Figure 3 · p. 9, §4.4 · p. 10

5 CONCLUSION

The conclusion presents PROTEAN as a benchmark for evaluating autonomous macromolecular editing under strict physical constraints. It formalizes boundary preservation, source-isolated task splits, and multi-candidate evaluation.

5 CONCLUSION

  • PROTEAN formalizes macromolecular editing evaluation through explicit boundary-preservation contracts, source-isolated task splits, and multi-candidate evaluation.The authors position it as an operational benchmark to guide development of next-generation biomolecular AI agents.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement specifies standardized task materials and deterministic or fixed-seed procedures for alignment, sequence redesign, and evaluation.

REPRODUCIBILITY STATEMENT

  • The benchmark standardizes task definitions, source coordinates, and frozen thresholds, with deterministic Kabsch alignment, fixed-seed ProteinMPNN redesign, and ESMFold refolding.Standard pass@k metrics evaluate candidate budgets k ∈ {1, 2, 4, 8, 16}.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement describes use of publicly available materials under their licenses and terms, with limits on personal-data collection and released information.

ETHICS STATEMENT

  • The work uses public datasets and benchmark materials under their licenses, does not intentionally collect personally identifiable information, and limits released records to what reproducibility requires.Residual identifiers should be removed where applicable.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The appendix documents PROTEAN’s construction and model-facing protocol, from source grouping and task selection through frozen evaluation and logged LLM requests. It also states that AI models were accessed with fixed prompts and deterministic decoding, while authors implemented and verified the experiments.

AI USE STATEMENT

  • Language models were accessed through standard APIs with fixed system prompts and deterministic decoding; generative AI also assisted with manuscript text refinement and formatting.The authors state that they implemented and verified all experimental protocols, structural scoring pipelines, and analytical evaluations.Source: AI USE STATEMENT · p. 10

B.1.1 SOURCES.

  • The source pool combines filtered experimental structures with predicted human Swiss-Prot models, and the release mixes 1,142 experimental and 258 predicted tasks.The experimental leg inherits interface-harvest quality filters; predicted entries require coverage and confidence thresholds.Source: §B.1.1 · pp. 15–16

B.1 TASK CONSTRUCTION PIPELINE

  • The task pipeline groups sources into isolation components before constructing seven equal-length task types, then quota-samples, canonicalizes, audits, and freezes the evaluation set.Motif scaffolding enters separately from MotifBench and joins only at the frozen set.Source: §B.1 · p. 15, Figure 4 · p. 16

B.1.2 PER-FAMILY CONSTRUCTION.

  • Editable windows are selected using family-specific feasibility rules, including distance ranges, paired structures, fragment similarity, shape witnesses, and opened cyclic peptides.For cyclization, nine witness chains of 13–40 residues are opened by rotating termini until their N–C distance is 2.0–4.0 Å.Source: §B.1.2 · p. 16

B.1.3 MOTIF IMPORT.

  • The 30 motif-scaffolding cases come from MotifBench, with canonical scaffold gaps used for construction bookkeeping rather than fixed model instructions.The instruction leaves scaffold lengths open and requires arithmetic consistency with the public total length.Source: §B.1.3 · p. 17

B.1.4 CANONICAL VIEW, QUOTAS, ISOLATION, FREEZE.

  • Sources are canonicalized and related structures grouped; quota sampling caps tasks at eight per source group, and the audited manifest is frozen before evaluation.The final set is the complete suite, with no held-out selection.Source: §B.1.4 · p. 17

B.2 TASK CARDS, ONE PER EDIT TYPE

  • Each task card provides the instruction, residue-range role partition, sequence policy, and public target criteria; the model receives the same card without routing metadata.Move-gate evaluation uses the frozen ±1.0 Å threshold, not the card-level value.Source: §B.2 · p. 17

B.3 WHAT THE MODEL SEES

  • Each attempt is a single API call, and the model must return a JSON backend specification using the capability card, scoring criteria, and task card.Sampling is deterministic at temperature 0.2, with at most one structured retry for malformed output; failure feedback may appear in later rounds.Source: §B.3 · pp. 17–18

B.4 ROLLOUTS: PROMPT, SPECIFICATION, AND OUTCOME

The worked rollouts show how models translate task cards into backend specifications and how generated structures fare against target and scope gates. The cyclization example reaches its target but fails scope, unlike the successful approach and helix examples.

B.4 ROLLOUTS: PROMPT, SPECIFICATION, AND OUTCOME

  • The inter-domain-approach rollout is labeled successful, and its specification derives the keep/redesign split, diffusion window, and hotspot conditioning from the task card.The full fixed chain is retained; the redesign segment maps to B:99–110 and the hotspots match the listed reference patch.Source: §B.4 · p. 19
  • The helix-enrichment rollout places an anonymous 48-residue redesign segment exactly over A:97–144, where the candidate gains helices over 56% of the window.The source region is coil throughout; the model’s diffusion region names the same span as the redesign contig.Source: §B.4 · p. 19
  • For cyclization, the returned specification covers the terminal edit core and declares a head-to-tail amide bond between A22:C and A1:N.The two design intervals cover the terminal core exactly.Source: §B.4 · p. 19
  • The cyclization candidate forms exactly one new bond at dN-C = 1.32 Å, passing the 1.20–1.50 Å target band while failing the 0.10 Å scope tolerance.Protected-backbone relaxation reaches 0.21 Å, despite both substitutions remaining inside the edit core.Source: §B.4 · p. 19
  • No backend candidate passes all three gates for this task type in the frozen run, because ring closure cannot leave fourteen protected residues rigid.The passage characterizes this failure as mechanical rather than a constraint-compilation error.Source: §B.4 · p. 19

B.5 GENERATED STRUCTURES

  • Figures 5–7 render the source and candidate structures with editable regions in red and protected context in slate; the cyclization view marks the terminal distance.Structures are aligned using protected Cα anchors; the cyclization example has post-alignment anchor RMSD of 0.075 Å.Source: §B.5 · p. 19

B.6 CYCLIZATION DECOMPOSITION: SHARED SCOPE CANDIDATES AND CLOSURE

Cyclization exposes a conflict between closing the peptide and preserving the protected backbone under the frozen scope contract. The decomposition distinguishes shared scope-preserving candidates from model-dependent closure geometry.

B.6 CYCLIZATION DECOMPOSITION: SHARED SCOPE CANDIDATES AND CLOSURE

  • The 50-task head-to-tail family is reported on the BoltzGen lane because ODesign and RFdiffusion3 lack an explicit covalent-bond primitive.The available audit uses five model–BoltzGen score files; the Gemini score file was unavailable at freeze.Source: §B.6 · pp. 19–20, Table 5 · p. 21
  • The successful approach example shortens its closest patch contact from 10.7 Å to 6.0 Å, while the helix example remodels a source window with helix fraction 0.00 to 0.56.The cyclization illustration reaches dN-C = 1.32 Å but exceeds the protected-backbone tolerance.Source: Figure 5 · p. 20, Figure 6 · p. 20, Figure 7 · p. 21
  • All 50 scope-pass rows are rollout 1 and atomically identical across five models after coordinate and topology canonicalization, so they are not independent model-specific solutions.Forty-nine CIF files are byte-identical; the remaining file differs only in occupancy metadata.Source: §B.6 · pp. 20–21
  • Qwen3.8Max has the highest geometry-pass rate (779/850 = 91.6%), while Kimi K3 has the lowest (766/850 = 90.1%); the 1.5-point spread is not a scope-preservation difference.The passage says these differences arise only after the explicit bond constraint is compiled successfully.Source: Table 5 · p. 21, §B.6 · p. 21
  • 0/50 strict joint successes reflect disjoint candidate sets: scope-pass candidates preserve context but miss cyclization distance, while closed candidates relax the protected backbone beyond tolerance.Closed candidates have median protected-backbone displacement of approximately 0.43–0.45 Å versus the frozen 0.10 Å tolerance.Source: §B.6 · p. 21
  • The failure examples distinguish a merged editable/protected partition, a genuinely unclosed chain, and an apparent target failure caused by residue mapping.Mapping-artifact rows should be remapped and rescored before attribution to a model.Source: Figure 8 · p. 22

B.7 CYCLIZATION FAILURE CASES AND ACCOUNTING OF FALSE FAILURES

Cyclization failures must be separated by pipeline layer: strict success was absent across the compared BoltzGen models, while visual and scoring artifacts require distinct accounting. The following evaluation and case-study sections describe frozen gates and show how specification differences relate to outcomes.

B.7 CYCLIZATION FAILURE CASES AND ACCOUNTING OF FALSE FAILURES

  • 0/50 task-level joint successes occurred for every available BoltzGen model, and shared scope-pass candidates provide no evidence of a model-specific protected-context advantage.Gemini was excluded because its BoltzGen score file was unavailable at the appendix freeze.Source: §B.7 · p. 23
  • Open-chain target misses, region or scope violations, and evaluation-map artifacts are distinct failure terminals; only the open-chain panel is itself a target failure.A representative diagnostic without a task or model identifier should not be attributed to a particular model without its task-level log.Source: §B.7 · pp. 22–23, Table 6 · p. 23

B.8 GATE EVALUATION, IN PSEUDOCODE

  • The imported motif-scaffolding run had grounding, placement assembly, and motif preservation pass 30/30 across configurations, while sidecar-refold scoring remained pending and no task counted as successful without it.The RFdiffusion3 adapter used fixed motif sequences, and Claude Opus 5 requests omitted a temperature parameter rejected by its provider.Source: §B.8 · p. 23

B.9 QUALITATIVE CASE STUDIES: SPEC-SPACE PLANNING DISPARITY

  • The qualitative comparison links differences in compiled specifications to macroscopic task outcomes.The section introduces two ODesign compilation episodes as examples of how biological reasoning is translated into backend parameterizations.Source: §B.9 · pp. 23–24

B.10 EXECUTION BOTTLENECKS AND BIOPHYSICAL ATTRITION ANALYSIS

Execution analysis identifies scope leakage and refolding as important barriers despite target attainment, and the motif-scaffolding setting shows a strong budget effect. Two ODesign cases illustrate how hotspot selection and boundary auditing shape compiled specifications.

B.10 EXECUTION BOTTLENECKS AND BIOPHYSICAL ATTRITION ANALYSIS

  • 96.0 tasks (27.4%) met RFdiffusion3 target attainment, but only 51.0 tasks (14.6%) maintained scope fidelity and 26.0 tasks (7.4%) achieved joint geometric compliance.BoltzGen and ODesign retained 94.5% and 80.6% scope fidelity, respectively; strict coordinate-level boundary auditing is identified as necessary to prevent global conformational drift.Source: §B.10 · pp. 24–25

B.11 SPECIAL SETTING IN MOTIF SCAFFOLDING

  • Scaling the motif-scaffolding candidate budget from s@1 to s@16 substantially amplified performance across models, including a greater than 3× gain for Claude-Opus-5.The passage attributes the benefit to increased structural diversity for navigating rugged, non-convex landscapes and avoiding kinetically trapped local minima.Source: §B.11 · p. 25, (B) CASE 2 — APPROACH: BOUNDARY FIELD… · p. 26

(A) CASE 1 — SEPARATION: HOTSPOT CURATION

  • Claude-Opus-5 selected three coherent hotspot residues to create a directional conditioning vector, whereas transcribing all 12 residues could cancel directional guidance and selecting one could under-specify it.All three models satisfied the syntax format, but the passage characterizes only Claude-Opus-5’s output as reflecting directional comprehension of interface mechanics.Source: (A) CASE 1 — SEPARATION: HOTSPOT… · p. 26

(B) CASE 2 — APPROACH: BOUNDARY FIELD AUDITING

  • Failure diagnostics attribute 58.7–61.6% of failures to spatial planning and 73.0–82.1% to co-optimizing local deformation with contextual preservation; boundary auditing can expose scope failure.The figure caption describes boundary auditing as the discriminating factor in Case 2, where confusing the editable core with the relaxation shell causes catastrophic G3 scope failure.Source: (B) CASE 2 — APPROACH: BOUNDARY FIELD… · pp. 26–27, Figure 9 · p. 26
Loading 2610.17765v1…