Source-linked AI summary
Federation Is Nearly Free, Reasoning Is Not: Tradeoffs for AI Co-Scientists in Protein Characterization Workflows
Maia Kapur, Timothy Boe, Abby Jerger, Paul Rigor
TL;DR
Scientific-agent workflows must balance flexible reasoning with reproducibility and cost, especially when agents communicate across institutional boundaries. This study compares federation, harness, model, and prompting choices for protein-function characterization and finds model choice dominates accuracy, while deterministic PPO offers near-frontier performance with perfect consistency and zero token cost.
Problem
Scientific-agent workflows must balance flexible reasoning with determinism, reproducibility, observability, cost, and latency when agents communicate across institutional boundaries.
Method
A controlled ablation evaluates federation topology, RL versus LLM harnesses, language models, and prompting strategies on verifiable protein-function characterization tasks.
Results
Opus achieved ~92–94% accuracy versus ~40–50% for o4-mini, while federation had negligible performance impact and PPO delivered 1.000 consistency at zero token cost.
Takeaways & Limitations
For routine, verifiable recovery workflows, inexpensive deterministic policies are appropriate, while flexible LLM harnesses should be reserved for tasks requiring open-ended reasoning.
Takeaways & Limitations
The study evaluates recovery of known protein annotations, so its findings may not generalize to discovery-focused work involving unverifiable answers.
Abstract
from arXiv · showhide
Natural language driven autonomous co-scientist workflows involve a fundamental trade-off between flexibility and reasoning at the expense of determinism, reproducibility, and observability. Such agents increasingly must communicate across institutional boundaries, where federation topology can shape latency and cost. We systematically evaluated these tradeoffs using a controlled ablation on a production agentic platform for science. We use a verifiable task: given a protein sequence, we ask an agent to confidently characterize its function by routing across common tools. We compare federation topology, classic RL vs LLM-driven harnesses, language model, and prompt expertise. We also stratify results by protein novelty. We find that the choice of LLM dominated prediction quality far more than topology or prompting (Opus ~92%-94% vs o4-mini ~40%-50%). The PPO policy was nearly as accurate as the best LLM (88%) at zero token cost, fastest latency, and perfect consistency, but yields no reasoning trace. Expert prompted LLMs reached the highest accuracy but were high-cost and less consistent; prompt dependence was largest when the task was hardest. Federation imposed a negligible penalty on performance. These results offer actionable guidance for deploying agents for scientific workflows: for routine, verifiable tasks, a cheap deterministic policy delivers near-frontier accuracy with complete reproducibility, while flexible LLM reasoning is best reserved for open-ended discovery.
Methods
The study used a verifiable protein-function benchmark to evaluate autonomous workflows across federation topology, agent harness, and decision-making conditions. Its dataset and experiments varied protein difficulty, tool availability, and orchestration while instrumenting cost, latency, and delegation behavior.
- Benchmark: Agents predicted protein function from accession IDs and/or genetic sequences using numeric Gene Ontology annotations in a verifiable benchmark.Target proteins were sourced from UniProt, with BLAST against Swiss-Prot and AlphaFold servers used in the workflow.
- Dataset: 314 unique proteins were split 80%/20% into training and test groups across species and annotation-confidence regimes.The dataset targeted varied BLAST confidence and evolutionary divergence, including under-characterized proteins where structure prediction could add value.
- Ablation Experiments: Ten ablation experiments compared monolithic versus federated topology and classic RL-PPO versus prompt-driven LLM harnesses.The experiments represented a broad, but not exhaustive, set of approaches for driving an autonomous workflow.
- Topology handling: Three EC2 instances implemented federation with separate ADEPT stacks, identity providers, MCP tool servers, and orchestration services.In the federated topology, Lab A hosted BLAST, Lab B hosted AlphaFold, and Lab C served as the orchestrating gateway.
- PPO RL specifications: The PPO agent used a 9-dimensional continuous observation vector and chose among three discrete actions: call BLAST, call AlphaFold, or STOP.Episodes terminated at STOP or after a maximum of 5 steps, with observations encoding sequence, confidence, cost, and tool-use information.
- PPO RL specifications: PPO received structured numeric tool features without natural-language input or interpretation, learning tool sequences from reward signals under cost-accuracy and latency tradeoffs.Returned features included BLAST identity percentage, e-value, hit count, and AlphaFold pLDDT.
Reward and cost structure
The workflow’s reward structure combines GO-term accuracy from BLAST-based annotation transfer with tool-cost, time, structural-confidence, and BLAST-confidence components. Evaluation used isolated temperature-0 LLM calls and multiple metrics across five test-data rollouts.
- Reward components: Tool penalties summed pseudodollar costs ($5 BLAST, $10 AlphaFold) and wall-clock time.Structural bonus used weighted pLDDT when AlphaFold was called, while confidence bonus used weighted, normalized BLAST percent identity.
- Reward components: Accuracy matched predicted and ground-truth Gene Ontology terms for Molecular Function and Biological Process, and only successful BLAST use contributed.AlphaFold shaped reward through structural confidence but could not independently predict GO terms or transfer functional annotations.
- Evaluation setup: LLM experiments used temperature 0.0, with one isolated inference call per protein and response variation arising from greedy decoding.Each episode represented one characterization task without cross-episode memory.
- Evaluation setup: Metrics were calculated across 5 rollouts of the test data and summarized for each of ten experiments.The suite extended the PPO-RL cost-function metrics to compare harnesses and federation topologies.
- Performance metrics: Consistency ranges from 0.5 to 1.0, where 0.5 is maximally inconsistent and 1.0 is perfectly consistent across trials.The metric uses the larger of accurate or inaccurate responses divided by the number of trials.
- Performance metrics: Token use included ingress and egress tokens, with approximately 11,000 system-prompt tokens subtracted from Langfuse trace totals.The evaluation also defined confidence from BLAST confidence and AlphaFold PLDDT, plus a tool-cost ratio.
Learned vs. Emergent Routing Strategies:
Routing strategies differed sharply: PPO and most LLM harnesses generally escalated to structural tools, while expert-prompted Opus rarely called AlphaFold. PPO matched strong accuracy with perfect consistency, zero token cost, and substantially lower latency, whereas LLM performance varied by model, prompt, and topology.
- Learned vs. Emergent Routing Strategies:: PPO and most LLM harnesses frequently escalated to AlphaFold across BLAST-confidence and sequence-length bins, whereas expert-prompted Opus rarely called it.The PPO policy learned a near-universal escalation strategy; most LLM harnesses behaved similarly, with expert-prompted Opus as the notable exception.
- Learned vs. Emergent Routing Strategies:: 1.000 was the RL harness consistency in both topologies, with a 90% CI of 1.000-1.000; the most-consistent Opus experiment reached 0.974.The RL harness produced no differing answers for any test protein across five rollout trials.
- Learned vs. Emergent Routing Strategies:: 15-16 seconds per protein was the RL latency, compared with 36 to 58 seconds for LLM harnesses.Opus latency was more stable at approximately 39-42 seconds, while o4-mini latency varied more across topologies.
- Learned vs. Emergent Routing Strategies:: Zero LLM tokens and zero inference cost characterized the RL experiments, while o4-mini expert prompting consumed 28,554 or 28,297 tokens per protein.The cited o4-mini values correspond to E3 and E6, respectively.
Discussion
For routine, verifiable protein-characterization workflows, model capability dominated performance while federation imposed negligible overhead. Deterministic PPO offered 88% accuracy with perfect consistency and zero token cost, whereas LLM harnesses traded observability and flexibility for higher cost and lower consistency, especially in discovery-oriented work.
- Performance and federation: LLM choice dominated accuracy: Opus reached ~92-94% versus o4-mini at ~40-50%, while topology and prompting produced much smaller gaps.Federation, including A2A communication, authentication, and cross-instance trace logging, imposed no perceived performance tax on any harness.
- Shared routing strategy: Both PPO and expert-prompted Opus converged on confidence-conditional routing: trust BLAST at high confidence and escalate to AlphaFold at low confidence.The diversity bonus for supplementary AlphaFold exploration was less influential than this established routing logic.
- LLM harness tradeoffs: LLM harnesses supplied reasoning and flexibility but introduced stochastic inconsistency, token costs, and latency, with inconsistency especially pronounced in low-confidence cases.These tradeoffs were strongly model-dependent, and o4-mini’s failures illustrate the risk.
- Deterministic policy tradeoffs: The RL-PPO harness achieved perfect consistency (1.000), consumed zero tokens, and remained fast and topology-robust, but provided no reasoning trace.Its rigidity also required a curated training set and a verifiable reward.
- Scope and deployment guidance: The study evaluated recovery of known protein annotations, whereas discovery requires conjecture and inference; even well-designed proteins exposed failures in weaker models such as o4-mini.The authors therefore distinguish verifiable recovery from novel discovery when selecting agent architectures.