PaperlayerView exact revision on arXiv

Source-linked AI summary

Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows

Yuhang Fu, Ruishan Fang, Jiaqi Shao, Huiyu Zheng, Zhengtao Zhu, Bing Luo, Tao Lin

arXiv:2606.05670v1cs.AI

TL;DR

The paper asks whether adding agents improves LLM workflows once benchmark loading, tools, answer contracts, accounting, and logging are aligned. BenchAgent enables controlled substrate-internal comparisons and a separate protocol-aligned external GAIA study. Across broad benchmarks, MAS does not consistently improve over the matched single-agent anchor, while the Claude-Code-style runtime workflow reaches 66.72% overall GAIA accuracy and leads on harder levels.

  • Problem

    Accuracy-only comparisons can confound workflow organization with unequal inputs, answer contracts, tools, accounting, or trajectory logging.

  • Method

    BenchAgent places single-agent, fixed MAS, and evolving MAS workflows under shared execution, tool, evaluator, accounting, and logging interfaces, alongside a PAE runtime-workflow comparison.

  • Results

    Across broad benchmarks, fixed and evolving MAS do not consistently beat the matched single-agent anchor; the PAE Claude-Code-style workflow reaches 66.72% overall GAIA accuracy and leads on harder levels.

  • Takeaways & Limitations

    More agents alone do not guarantee workflow lift, while runtime-generated workflows can occupy a different accuracy–cost profile on long-horizon tool-use tasks.

  • Takeaways & Limitations

    The comparisons are deployed-configuration results rather than isolated mechanism estimates, with partially visible runtime internals and one-run pass@1 evidence.

Abstract

from arXiv · show

Does adding more agents help an LLM workflow once compared systems share the same benchmark loader, tool access, answer contract, usage accounting, and trajectory logging? We introduce BenchAgent, an evaluation framework that places single-agent, fixed multi-agent (MAS), and evolving MAS workflows under one normalized execution and logging protocol. BenchAgent evaluates these substrate-internal workflows across ten reasoning, coding, and tool-use benchmarks with GPT-4.1, and separately reports a Protocol-Aligned External (PAE) GAIA study of a runtime-generated workflow. Under SI conditions, at most one of six tested MAS exceeds the matched single-agent anchor on benchmark-balanced average accuracy: EvoAgent lies within the Wilson one-run guidance, while the remaining five trail by 2.56-11.29 points and occupy more expensive accuracy-cost trade-offs. On the PAE GAIA snapshot, a Claude-Code-style runtime workflow reaches 66.72% overall and 69.23% on Level 3, more than 20 points above the strongest non-Claude baseline, Jarvis, a fixed MAS.

1 INTRODUCTION

The paper asks whether workflow organization itself improves LLM-agent performance when evaluation protocols are aligned, and introduces BenchAgent to separate workflow lift from protocol differences. It combines a controlled substrate-internal comparison with a protocol-aligned external GAIA case study.

  • Motivation: The paper situates BenchAgent within research spanning single-agent loops, role-specialized MAS, and workflows that configure, evolve, or generate execution procedures.The related paradigms include reasoning-and-acting systems, role-based coordination, and runtime workflow generation.
  • Contribution: BenchAgent normalizes benchmark loading, tool access, answer contracts, usage accounting, and trajectory logging so workflow paradigms can be compared under shared interfaces.The framework covers single-agent, fixed MAS, evolving MAS, and runtime-generated workflows.
  • Main findings: Under SI conditions, adding agents or explicit coordination does not guarantee positive workflow lift, with fixed and evolving MAS varying in accuracy, token usage, and latency.Several MAS workflows trail the matched single-agent anchor.
  • Main findings: At most one of six tested MAS exceeds the matched single-agent anchor on average: EvoAgent gains 1.44 points within Wilson one-run guidance, while five trail by 2.56–11.29 points.The comparison uses a GPT-4.1 backend with shared tools, evaluator, and logger.
  • Main findings: The PAE GAIA study reports 66.72% overall accuracy for the Claude-Code-style workflow, with higher performance on harder levels and fewer recorded tokens than the strongest non-Claude baseline.This is a deployed-configuration comparison rather than evidence for a single mechanism.

2 RELATED WORK

Prior work ranges from single instrumented controllers to fixed, debated, optimized, and dynamically generated multi-agent workflows. BenchAgent differs by aligning execution and instrumentation for workflow comparison rather than introducing a new benchmark or treating engineering documentation as empirical evidence.

  • Single-agent workflows: Single-agent systems such as ReAct, Toolformer, WebGPT, SWE-agent, and HuggingGPT/JARVIS establish that one instrumented controller can be a competitive workflow.The paper therefore uses a matched single-agent workflow as the comparison anchor, not a bare model call.
  • Multi-agent workflows: Fixed MAS make division of labor explicit through roles, communication procedures, conversable-agent substrates, debate aggregation, or sampling-and-voting.Examples include CAMEL, MetaGPT, AutoGen, debate systems, and related voting pipelines.
  • Dynamic workflows: Dynamic and evolving systems search over or mutate workflow structure, extending multi-agent design beyond predefined roles and communication patterns.This line of work overlaps with systems that optimize or compile language-model pipelines.
  • Evaluation infrastructure: Agent benchmarks broaden evaluation to interaction, tool use, web and software tasks, analytical traces, and distributed coordination, while Harbor focuses on sandboxed agents and models.BenchAgent occupies a different infrastructure scope from these datasets and evaluation platforms.
  • Runtime-generated workflows: LangGraph, CrewAI, and Claude Code document graph orchestration, role-based crews, subagent creation, context isolation, and permission-scoped delegation, but these sources are treated as engineering descriptions rather than peer-reviewed empirical evidence.The GAIA experiment compares the CC-workflow with BenchAgent MAS under a documented PAE protocol.

3 EVALUATION PROTOCOL

BenchAgent defines workflows by their active agents, communication topology, and tool scopes, then evaluates them through a shared execution and reporting substrate. The protocol separates substrate-internal workflow-lift estimates from an external protocol-aligned comparison and uses Wilson guidance for interpreting one-run gaps.

  • Workflow representation: A workflow maps each benchmark instance x to a final answer and, when available, an execution trace containing model calls, tools, messages, artifacts, and termination events.Task success is evaluated by E(x, ŷ), while cost summaries record tokens, latency, tool calls, and delegation structure.
  • Workflow taxonomy: Workflow categories differ by how active agents, communication or delegation topology, and tool scopes are specified over time.Single-agent workflows keep one active agent; fixed MAS use predefined structures; evolving MAS mutate topology; runtime-generated workflows change these elements during execution.
  • Workflow lift: Workflow lift is the change in accuracy and cost when replacing a single-agent workflow with fixed or evolving MAS while holding the model, loader, tools, answer contract, evaluator, and accounting substrate constant.The single-agent anchor need not be the strongest possible implementation; the shared control substrate is the comparison requirement.
  • Shared substrate: BenchAgent normalizes benchmark loading, input and answer formatting, runtime control, tool access, usage and trajectory logging, and evaluator calls for SI comparisons.Representative fixed and evolving MAS are implemented as workflow instances over this common substrate.
  • PAE comparison: The CC-workflow is evaluated through PAE because reimplementing it inside BenchAgent would alter its runtime semantics, while backend family, inputs, output schema, evaluator, and relevant tool classes are aligned.Its internal controller details remain only partially visible through retained traces.
  • Reporting and uncertainty: Reporting records success, token usage, wall-clock latency, and full trajectories; broad-benchmark accuracy is the unweighted mean across ten benchmark accuracies, while GAIA overall is weighted by level size.All reported numbers are pass@1 single-run results, and Wilson intervals provide a conservative scale for interpreting observed gaps.

4 EXPERIMENTS

The experiments compare matched single-agent, fixed MAS, and evolving MAS workflows across ten benchmarks under SI conditions, alongside a PAE GAIA study of a runtime workflow. Results show that multi-agent coordination does not reliably improve broad-suite accuracy, while the CC-workflow leads on longer-horizon GAIA tasks with lower recorded resource use than the strongest non-Claude baseline.

  • 4 EXPERIMENTS: The studies use matched SI protocols for the broad comparison but a PAE GAIA comparison because the external Claude-Code runtime cannot be reimplemented without changing its semantics.BenchAgent normalizes benchmark loading, tools, answer contracts, usage accounting, and trajectory logging; the GAIA controller remains partially external.
  • 4.2 Broad Benchmark Results: 74.12% benchmark-balanced accuracy for the single-agent anchor versus 75.56% for EvoAgent makes EvoAgent the only numerically positive MAS, but its +1.44-point gain is within one-run uncertainty.The remaining MAS fall below the anchor at 62.83%–71.56%; all figures are descriptive benchmark-balanced means from one end-to-end run.
  • 4.2 Broad Benchmark Results: Similar accuracy can entail sharply different resource use: EvoAgent costs more tokens for its small gain, ChatEval is token-intensive while trailing, and Camel and Jarvis are lighter but less accurate.Figure 2 plots benchmark-balanced average accuracy against instance-level token usage and execution time, with a descriptive empirical Pareto front.
  • 4.2 Broad Benchmark Results: MAS effectiveness follows task-protocol fit rather than agent count, with gains tied to scaffold search, verifiable proposals, or multi-judge instruction checking and losses elsewhere.EvoAgent gains on BBH, LLM-Debate on HumanEval and MATH, and ChatEval on IFEval but collapses on AIME.
  • 4.3 GAIA Runtime Workflow Results: 66.72% overall GAIA accuracy puts the CC-workflow 20.06 points above Jarvis, with gaps of 26.60 points on Level 2 and 42.31 points on Level 3.Jarvis leads on Level 1, while the CC-workflow uses fewer recorded tokens and less wall-clock time than the strongest non-Claude baseline.
  • 4.3 GAIA Runtime Workflow Results: The CC-workflow retains task-specific subagents, reusable evidence artifacts, and verifier-stage events before finalization, whereas EvoAgent can lose a task-critical constraint during linear decomposition.Its traces average 6.25 subagents with maximum delegation depth 2.5, but runtime delegation, artifacts, verification, and context management remain confounded.

5 DISCUSSION

The discussion finds no uniform multi-agent advantage: workflow organization helps selectively, while runtime-generated workflows require evaluation beyond final accuracy. SI and PAE are kept separate because their comparisons answer different questions.

  • 5 DISCUSSION: Task-specific evidence favors different coordination patterns: debate helps on checkable proposals, evolutionary search helps on BBH, while handoffs can hurt exact instruction following and evidence control.The paper frames Table 1 as a workflow-lift comparison around a shared single-agent anchor, not a general leaderboard.
  • 5 DISCUSSION: Runtime-generated workflows expose separate context windows, permission scopes, persistent artifacts, and runtime delegation as candidate mechanisms that final scores alone cannot distinguish.The discussion recommends reporting topology, context boundaries, tool scopes, and repair gates alongside scores.
  • 5 DISCUSSION: SI isolates workflow wrappers over a common core, whereas PAE reports a mature runtime workflow under a documented but less internally controllable protocol.Combining them would mix workflow lift with protocol advantage.

6 CONCLUSION

BenchAgent evaluates LLM-agent workflows with aligned execution and instrumentation, finding no consistent multi-agent advantage across broad benchmarks. A Claude-Code-style workflow performs better on harder GAIA levels at lower retained-token and wall-clock cost, but remains a deployed-configuration comparison.

  • 6 CONCLUSION: Fixed and evolving MAS do not consistently beat the matched single-agent anchor and occupy different accuracy–cost trade-offs across broad benchmarks.The paper releases BenchAgent and trajectory tooling to study workflow generation, context management, tool scoping, and verification separately from agent count.
  • 6 CONCLUSION: On GAIA, the CC-workflow performs better on harder levels with lower retained-token and wall-clock cost under PAE.This result remains a deployed-configuration comparison rather than a mechanism estimate.

7 LIMITATIONS

The comparison is controlled but not fully causal: MAS are re-instantiations, and the external CC-workflow bundles several workflow and provider-side factors. Mechanism evidence is also limited by one-run results and partial trace visibility.

  • 7 LIMITATIONS: The study compares deployed configurations rather than isolated mechanisms because the CC-workflow may combine generation, context compaction, file and shell tooling, permission scoping, and provider-side bookkeeping.The broad-benchmark MAS are re-instantiations rather than exact reproductions.
  • 7 LIMITATIONS: Mechanism evidence is partial because the study reports one-run pass@1 results and retained traces expose only part of Claude Code’s internals.The paper calls for repeated trials, controller-sensitivity checks, stricter tool-surface-matched ablations, runtime-workflow ablations, and fuller instrumentation.

A EVALUATION PROTOCOL AND SUPPLEMENTARY EVIDENCE

The appendix supplies protocol checks, baseline-transfer evidence, runtime-workflow traces, and supplementary broad-benchmark analyses. Its sections organize sampling, tool regimes, calibration, alignment tables, trace excerpts, process signals, and runtime statistics.

  • A EVALUATION PROTOCOL AND SUPPLEMENTARY EVIDENCE: The appendix reports protocol checks, auxiliary baseline-transfer evidence, runtime-generated workflow traces, and supplementary broad-benchmark analyses.These materials extend the main evaluation with supporting protocol and trace evidence.
  • A EVALUATION PROTOCOL AND SUPPLEMENTARY EVIDENCE: Appendix A.1 covers dataset sampling, tool regimes, Wilson guidance, framework comparison, ReAct calibration, backend checks, and protocol-alignment tables.It concentrates the evaluation and calibration details supporting the main comparisons.
  • A EVALUATION PROTOCOL AND SUPPLEMENTARY EVIDENCE: Appendix A.2–A.4 provide retained runtime-workflow excerpts, process signals, mechanism-versus-trace distinctions, runtime statistics, and same-task visual evidence.The listed sections separate observed trace evidence from mechanisms requiring ablation.

A.1 PROTOCOL AND FIDELITY CHECKS

The appendix documents sampling, uncertainty guidance, benchmark infrastructure comparisons, controller calibration, backend checks, and retained protocol evidence for the evaluation.

  • Sampling: 400-instance caps and full evaluation of smaller splits define the dataset sampling protocol, with benchmark-balanced averages computed over benchmark-level scores.AIME, HumanEval, MBPP, and GAIA validation are evaluated in full below 400 instances.
  • Uncertainty: Wilson half-widths are used only to interpret one-run pass@1 gaps, not as estimates of rerun variance.The guidance treats gaps smaller than the benchmark-specific half-width as descriptive rather than stable ordering evidence.
  • Controller calibration: Table 7 calibrates controller strength against a vanilla ReAct controller without replacing the matched-substrate workflow-lift comparison.The calibration uses aligned tool regimes, GPT-4.1, and the same evaluator pipeline as the corresponding BenchAgent anchor.
  • Backend checks: Backend checks test whether the workflow pattern is specific to GPT-4.1; under Qwen3-32B, evaluated MAS rows trail the BenchAgent anchor overall.The checks are scoped evidence for the same interpretation rather than a separate conclusion.
  • Protocol fidelity: The GAIA external comparison retains explicit protocol evidence and reports visible token usage as retained-log accounting rather than provider-independent billing.The retained setup includes role configuration and restrictions, while uncertifiable fields are marked rather than silently normalized.

A.2 PROMPT AND TRAJECTORY EXCERPTS

The appendix records prompt contracts, reproduced workflow wrappers, trajectory excerpts, and transfer checks to distinguish controlled BenchAgent evidence from auxiliary plausibility checks.

  • Prompt contracts: Prompt-contract tables make fixed MAS roles and topologies explicit before execution, while EvoAgent records seeded evolutionary operations.The fixed systems assign roles and communication procedures before task input; EvoAgent documents crossover and mutation operations.
  • Trajectory excerpts: Trajectory excerpts connect debate behavior to the reported token and model-call overhead and document runtime subagent perspectives within a Claude Code run.The records extend coverage across fixed and runtime-generated workflow paradigms.
  • Transfer checks: Independent transfers fall within a small-to-moderate range of official reference numbers, but they serve only as a sanity check.The controlled evidence remains the main-paper Table 1 comparison under the shared BenchAgent substrate.

A.3 ILLUSTRATIVE PROCESS SIGNALS AND MECHANISM EVIDENCE

The appendix uses retained process traces to contextualize broad-benchmark patterns while separating descriptive observations from mechanisms that require controlled ablation.

  • Mechanism evidence: Illustrative GAIA trace counts are used to interpret plausible workflow behavior, not to claim that process counts causally explain broad-benchmark scores.A separate table distinguishes observed trace signals from causal claims requiring ablation.
  • Failure-step analysis: Most analyzed errors occur at the first agent turn, while ChatEval later-round correction accounts for 16.8% of failures without yielding a net GAIA gain over Jarvis.The added debate coordination expands token usage without a proportionate observed error-correction benefit in the retained runs.

A.4 RUNTIME-GENERATED WORKFLOW EVIDENCE

The runtime-workflow evidence shows dynamically instantiated roles, explicit handoffs, verifier gating, and preserved routing state, while limiting causal and generality claims.

  • Observed runtime structure: Runtime traces repeatedly instantiate explicit subagents, reach delegation depth up to 4, and expose verifier-stage events in two structured runs.These statistics support a bounded process claim rather than an exhaustive count of recovery loops.
  • Dynamic construction: The team lead constructs a four-role planner–solver–verifier–writer pipeline live, assigns prompts at spawn time, and conditions the structure on the task batch.The workflow is not hard-coded as a static template.
  • Execution behavior: Typed SendMessage handoffs make boundary state auditable, the verifier applies a format gate with possible solver rerouting, and the Writer prevents duplicate writes.These behaviors are logged as runtime state transitions in the retained execution trace.
  • Behavioral contracts: Spawn-time prompts form each role’s complete behavioral contract, so role specialization persists through mailbox traffic without repeated central dispatch.The Solver is initialized to route its output specifically to the Verifier.
  • Scope and limitations: The retained traces come from one GAIA batch partition and partial visibility, so they do not establish universal role structures or isolate the accuracy advantage’s mechanism.Controlled ablations would be required to separate verifier gating, context packaging, prompt specificity, and other runtime behaviors.
  • Same-task evidence: A controlled same-task example shows that partial intermediate forwarding can lose instruction constraints, whereas retained state and verifier evidence preserve the runtime workflow’s context.The supplementary matched-success case likewise highlights more explicit state externalization and verification for Claude Code.
Loading 2606.05670v1…