Source-linked AI summary
Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi, Jake Yukich, Seth Donoughe
TL;DR
ABLE addresses limited evidence about how LLM agents use BAIMs in multi-step protein-design workflows. It benchmarks 15 models across planning and execution tasks, compares selected results with human experts, and finds strong but inconsistent performance, with important scope limitations.
Problem
Existing risk assessments largely evaluate LLMs or BAIMs in isolation rather than agentic LLM use of BAIMs in protein-design workflows.
Method
ABLE decomposes a protein-design workflow into one planning task and eight scored execution tasks, providing agents with prompts, tools, and computational environments.
Results
Claude Sonnet 4 and Gemini 3 Pro performed best overall, while models remained inconsistent in planning, strategy generation, environment navigation, and integrating design theory with tool use.
Takeaways & Limitations
The results suggest that current frontier models can reduce barriers to protein-design workflows, particularly by supporting information retrieval, tool identification, and some direct tool use.
Takeaways & Limitations
ABLE tests decomposed computational tasks in pre-provisioned environments and excludes wet-lab validation, so it is not a true end-to-end or experimental evaluation.
Abstract
from arXiv · showhide
We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use protein design workflows. ABLE assesses agent performance through a set of tasks spanning structure retrieval, sequence generation, and design validation. We evaluate 15 frontier models and find that seven refuse all tasks, while the remaining models exhibit substantial performance differences. Claude Sonnet 4 and Gemini 3 Pro achieve the highest scores across information retrieval, tool selection, and tool use. We further compare model performance on a subset of tasks against an expert human baseline. Our results suggest that current LLMs can substantially lower barriers to protein design, but remain inconsistent in planning, strategy generation, and integrating biological knowledge with tool use.
1. Introduction
ABLE addresses the gap between evaluating LLM knowledge or BAIMs in isolation and evaluating agentic BAIM use in protein-engineering workflows. It benchmarks agents on a redacted dual-use workflow involving human-infecting viruses while limiting potentially hazardous details.
- Prior risk assessments mainly examined LLMs or BAIMs separately, with LLM evaluations emphasizing biosecurity knowledge rather than agentic tool use.
- ABLE evaluates LLM agents’ ability to use BAIMs for protein engineering on human-infecting viruses.
- The benchmark decomposes a multi-step workflow into individually scored tasks spanning protein design and computational validation.
- The evaluation deliberately redacts or abstracts the pathogen, targeted properties, and scoring mechanism to mitigate dual-use risks.
2. Methods
ABLE operationalizes protein design as a scored benchmark of planning and eight execution tasks, providing agents with prompts, tools, and isolated environments. It also compares model performance with expert human baselines under task-specific constraints.
- ABLE measures agents’ generation and selection of viral-protein variants while maintaining sufficient structural stability.
- The benchmark combines one planning task, ABLE0, with eight sequential execution tasks scored from 0 to 1, allowing partial credit.
- ABLE assesses task performance through supplied prompts, software environments, computational tools, and web search, with ten independent runs per agent-task pair.
- Tasks test combinations of protein-design theory, computational biology, BAIM use, programming, structural biology, and literature review.
- Submissions are scored algorithmically except for ABLE0, which uses a model-graded rubric developed with biology experts.
- Human baselines included 11 researchers, who completed ABLE0, ABLE6, and ABLE8 under allocated time limits.
3. Results
Across nine tasks, models varied substantially: Claude Sonnet 4 and Gemini 3 Pro led overall, while several models refused all tasks and others struggled with tool interaction, planning, and theory–tool integration. Human experts were stronger at hands-on execution but weaker on planning and interpretation.
- Seven of 15 frontier models refused every task, while the remaining models showed substantial performance differences.
- Claude Sonnet 4 and Gemini 3 Pro achieved the strongest overall performance, including perfect scores across retrieval and tool-selection tasks.
- Human experts scored 0.86 ± 0.09 on ABLE6 but 0.55 ± 0.07 on ABLE0 and 0.70 ± 0.08 on ABLE8, with top models exceeding them on planning and interpretation.
- Researchers with hands-on BAIM experience outperformed minimally qualified participants on ABLE6, scoring 0.94 versus 0.72.
- Models generally identified appropriate BAIMs and understood structural-stability metrics, but performance worsened when tasks required biological theory to be synthesized with tool use.
- Several models frequently stopped early, failed to invoke tools, or hallucinated tool outputs; only Claude Sonnet 4 and Gemini 3 Pro entirely avoided this failure mode.
4. Discussion
ABLE finds substantial variation in frontier-model performance: leading models handle retrieval, selection, and some tool use, but complex planning and reliable workflow execution remain inconsistent. The benchmark also exposes important scope and governance considerations.
- Claude Sonnet 4 and Gemini 3 Pro consistently outperformed the other tested models.
- Gemini 3 Pro was the only model to achieve workflow success, yet ABLE3 and ABLE4 each had a 30% success rate.Workflow success required at least one perfect score on every execution task.
- Most models handled information retrieval and tool identification, while planning, strategy generation, environment navigation, and theory integration remained inconsistent.
- Human baseliners scored 0.86 on hands-on tool execution, whereas Claude Sonnet 4 scored 0.88 on planning compared with 0.55 for humans.Top models also scored 0.90–0.92 on ABLE8 interpretation, compared with 0.70 for human baseliners.
- ABLE assesses independently scored subtasks rather than a fully autonomous end-to-end workflow, and pre-provisioned environments leave setup and configuration untested.
- The benchmark excludes wet-lab validation, leaving the relationship between computational predictions and experimental functionality unresolved.
- ABLE provides task-level metrics that can inform capability thresholds, deployment decisions, and mitigation evaluation.The paper discusses managed deployment, tiered access, and testing refusals or unlearning as governance approaches.
Impact Statement
The paper frames ABLE as an AI-safety and biosecurity evaluation of LLM–BAIM integration. Its findings have implications for both beneficial protein-design applications and potential misuse, while the benchmark deliberately withholds hazardous details.
- The work aims to advance AI safety and biosecurity through systematic evaluation of how LLMs interface with biological AI models.
- ABLE evaluates dual-use protein-design capabilities with direct biosecurity implications while redacting hazardous task details.The redactions cover the specific pathogen, targeted properties, and scoring mechanisms.
- The findings connect frontier-LLM assistance in retrieval, tool identification, and direct tool use to beneficial applications and potential misuse.
A.1. ABLE Prompts
The ABLE prompts operationalize a computational protein-design workflow through planning, structure retrieval, sequence redesign, structural validation, and functionality assessment, with strict tool-use and output-format requirements.
- A.1. ABLE Prompts: The appendix withholds full prompts and the ABLE0 rubric while redacting hazardous viral-protein details to reduce attention-hazard risk.
- A.1.1. ABLE0 PROMPT: ABLE0 asks agents to produce a concrete, step-by-step plan specifying models, computational tools, workflow steps, and validation metrics.
- A.1.1. ABLE0 PROMPT: The planning prompt provides access to a computing instance with a dedicated GPU and lists web_search() as an available tool.
- A.1.1. ABLE0 PROMPT: Agents must explain their plan before each tool call and summarize the response and remaining questions afterward.
- A.1.2. ABLE1 PROMPT SAMPLE: The sample redesign task requires generating an alternative sequence that improves the target function while retaining structural stability.
- A.1.2. ABLE1 PROMPT SAMPLE: The workflow starts from a high-quality 3D structure, uses sequence design, validates the redesigned structure, and computationally assesses functionality.
- A.1.2. ABLE1 PROMPT SAMPLE: ABLE1 asks agents to identify the best high-resolution structure, defined here as below 3.0 Å, and provide its RCSB PDB code after comparing candidates.
- A.1.2. ABLE1 PROMPT SAMPLE: Prompt outputs are machine-parsed: responses must use specified XML blocks, contain only the required content, and place the final PDB ID inside a pdb_id tag.
A.2.1. ABLE0 DESIGN
ABLE0 evaluates whether agents can independently plan a protein-design workflow from a high-level goal. Its expert-reviewed rubric separates core completeness from useful technical details, while model grading standardizes open-ended responses.
- ABLE0 prompts models to generate a workflow from a high-level, detail-sparse query.
- The expert-verified rubric contains 6 completeness criteria and 7 details criteria, with binary scoring and partial credit of 0.5 where applicable.
- Reported ABLE0 scores average only the completeness criteria, while details-criteria scores are reported separately.
- Claude Sonnet 4 graded all open-ended model responses because the task required consistent model-based evaluation.
A.2.2. ABLE0 RESULTS
ABLE0 shows that models generally captured the core steps of the planning workflow, while performance on technical details remained substantially weaker. Frontier open-weight models matched the overall performance of non-refusing closed-weight models.
- Seven models generated responses on ABLE0, while Grok 4 refused every ABLE0 run despite attempting execution tasks.
- All non-refusing models captured most core workflow steps, as reflected by high completeness-criteria scores.
- Performance was substantially lower on details criteria, with no model exceeding 0.40 on D1, D4, or D5.
- Frontier open-weight models matched the overall performance of frontier non-refusing closed-weight models.
A.3. Transcripts of Model Planning
The transcript illustrates a model-generated, multi-phase protein design procedure that moves from structure preparation through sequence design, refinement, scoring, and final selection. It includes concrete tools, evaluation criteria, estimated timelines, and computational requirements.
- The example procedure is presented as a streamlined computational workflow for protein binding optimization.
- Structure Preparation: Structure preparation includes cleaning a target structure, identifying binding sites, detecting cavities, and mapping interactions.
- Sequence Design: Sequence design specifies designable positions, allowed amino acids, design trajectories, and interface-quality metrics.
- Refinement & Scoring: Refinement and scoring combine stability predictions, molecular-dynamics snapshots, electrostatics, language-model scores, affinity predictors, and developability metrics.
- Final Selection: Final selection uses visual inspection, sequence diversity, expression and stability predictions, and ranking for experimental testing.
- The transcript lists Rosetta, PyMOL or ChimeraX, FoldX, and ESMFold or ColabFold as supporting tools, with 5–10 variants as a typical output.
A.4. Refusal Rates
The refusal-rate and human-baseline materials define how model refusals, early stopping, and human experience were measured across ABLE tasks. They show that hands-on BAIM experience strongly affects direct tool execution, while models can outperform humans on planning and interpretation tasks.
- Refusal Rates: Refusal rate is the fraction of 10 runs in which a model declined to attempt a task.
- Early Stopping: Early stopping is the fraction of 10 runs terminated before the model completed the specified workflow.
- Human Baseliner Analysis: The human baseline included 11 researchers performing ABLE0, ABLE6, and ABLE8, with seven preferred-qualification and four minimal-qualification participants.
- Survey Metrics: Survey metrics covered effort, mental demand, time pressure, and self-rated success on a 1–5 scale, reported by task and qualification level.
- ABLE6: Preferred-qualification participants scored 0.94 ± 0.03 on ABLE6 versus 0.72 ± 0.24 for minimal-qualification participants.
- Cross-task comparison: Preferred-qualification participants exceeded Claude Sonnet 4 and Gemini 3 Pro on ABLE6, whereas both models outperformed humans on ABLE8.