PaperlayerView exact revision on arXiv

Source-linked AI summary

General Agent Evaluation

Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, Michal Shmueli-Scheuer

arXiv:2602.22953v2cs.AI

TL;DR

General-purpose agents lack systematic evaluation across heterogeneous protocols and unfamiliar environments without domain-specific customization. This paper introduces a unified protocol, evaluation harness, and leaderboard to compare agent architectures and backbone models across six benchmarks, finding meaningful cross-domain performance and parity with customized specialists on four of six benchmarks.

  • Problem

    Existing evaluations restrict agents to single protocols or require costly per-benchmark integration, limiting systematic evidence about general-agent performance across unfamiliar environments.

    Source: §1 · pp. 1–2

  • Method

    The study uses a Unified Protocol and Exgentic harness to evaluate unmodified agents and benchmarks across five architectures, five backbone LLMs, and six heterogeneous benchmarks.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    On 4 of 6 tested benchmarks, the best general-agent configurations are statistically indistinguishable from heavily-customized domain specialists.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    General agents are a viable research direction, achieving meaningful performance across every evaluated domain without per-domain customization.

    Source: §1 · p. 2, §6 · p. 10

    The evaluation is bounded by five LLMs, five agent implementations, and six benchmarks at approximately $20K, with per-benchmark confidence-interval half-widths of ±7–9.5pp.

    Source: §6 · p. 10

Abstract

from arXiv · show

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures x 5 backbone LLMs (three closed-source, two open-weight) x 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks; (v) a behavioral failure analysis reveals architecture-distinctive error signatures that aggregate scoring cannot discriminate. Code, harness, leaderboard, and traces are at https://www.exgentic.ai.

1 Introduction

This paper introduces a unified evaluation framework for testing unmodified general-purpose agents across heterogeneous benchmarks and protocols. Its factorial leaderboard shows broad adaptation, model-dominated performance differences, architecture-specific effects and failures, and competitiveness with specialized agents.

Motivation and evaluation gap

  • Existing benchmarks and evaluation frameworks constrain agents through bespoke protocols, benchmark-specific knowledge, or fixed interfaces, preventing faithful general-agent assessment.These limitations include domain-specific assumptions, single integration interfaces, and manual per-benchmark wiring of tools, sandboxes, or solvers.Source: §1 · pp. 1–2

Framework and contributions

  • The paper contributes the Unified Protocol, Exgentic harness, and first public Open General Agent Leaderboard for plug-and-play evaluation across heterogeneous agents, models, and benchmarks.The study tests each agent unmodified in unfamiliar environments and evaluates 25 agent-architecture × backbone-model configurations at a total cost of $20K.Source: §1 · p. 2, Table 2 · p. 3

Leaderboard findings

  • General agents achieved non-trivial performance across software engineering, customer service, technical support, deep research, and personal assistance without per-domain engineering.The evaluation spans six benchmarks and measures cross-benchmark success for agents not customized to the tested benchmarks.Source: §1 · p. 2
  • 27.8% of variance was explained by model choice versus 0.5% by architecture, while architecture still changed results by up to 12pp within a single model.Across closed-source configurations, model choice produced a 58× larger aggregate variance contribution than architecture.Source: §1 · p. 2
  • On 4 of 6 benchmarks, the best general-agent configuration was statistically indistinguishable from the top published domain-specific score, while BrowseComp+ and τ 2-Bench-Telecom remained specialist-led.The Open General Agent Leaderboard ranks all 25 configurations using standardized benchmark success rates and benchmark-weighted averages.Source: §1 · p. 2, Table 2 · p. 3

Failure analysis

  • Open-weight models exhibited generality sinks, including a τ 2-Bench score swing from 0.83 to 0.00 across architectures, while architectures showed distinctive failure modes hidden by aggregate scoring.On AppWorld every architecture collapsed; Claude Code and OpenAI Solo tended to stop too early, whereas ReAct and ReAct Short tended to skip evidence gathering.Source: §1 · pp. 2–3

2 Unified Protocol Methodology

The Unified Protocol mediates heterogeneous agent–benchmark interactions through a minimal, semantically faithful interface, reducing integration complexity. Exgentic applies this interface through wrappers and isolated sessions across diverse protocols.

2 Unified Protocol Methodology

  • The Unified Protocol is a faithful mediation layer and narrow waist that lets agents and benchmarks interoperate without costly pairwise integration.It was derived from common structures in existing protocols rather than imposed as a standard, preserving their expressible semantics during translation.Source: §2 · pp. 3–4

2.1 Agent Benchmark Unified Protocol

  • Each protocol instance minimally specifies task, context, and actions, with optional message and final-answer actions for common environment interactions.Actions expose typed parameters and arbitrary observations, while the three-field representation accommodates protocols decomposable into discrete actions.Source: §2.1 · p. 4

2.2 Running the Evaluation

  • Exgentic manually adapted six heterogeneous benchmarks and five agents using Unified Protocol wrappers, then executed the originals as black boxes in isolated, reproducible sessions.The harness preserves benchmark semantics by deriving each interface from a reference agent implementation and executes 150 agent × model × benchmark configurations.Source: §2.2 · p. 4

3 Experimental Setup

The study compares five agent architectures and five LLMs across six benchmarks using a unified evaluation setup. It standardizes benchmark interactions, defines shared architectural and performance components, and applies common metrics and statistical procedures.

3 Experimental Setup

  • The evaluation covers 150 configurations: 5 agent architectures, 5 LLMs, and 6 benchmarks, with 100 tasks per benchmark except 50 for τ 2-Bench Airline and a 100-turn cap.The LLMs comprise three frontier closed-source models—GPT 5.2, Claude Opus 4.5, and Gemini 3 Pro —and two open-weight models, DeepSeek-V3.2 [14] and Kimi-K2.5 [21].Source: §3 · p. 5

3.1 Benchmarks

  • The benchmark suite spans deep research, customer service, software engineering, and user-assistance tasks, with protocols adapted into the Unified Protocol while preserving native evaluation conditions where applicable.BrowseComp+ [11] fixes the retriever and reports dense-retrieval results; τ 2-Bench maps simulated-user messages and predefined tools; SWE-Bench Verified uses sandboxed bash interaction and hidden tests; AppWorld retains its native interpreter protocol and official harness.Source: §3.1 · p. 5

3.2 Agents

  • The five evaluated architectures are ReAct, ReAct Short, Smolagent, OpenAI Solo, and Claude Code, distinguished by their interaction protocols and substantive implementation components.ReAct Short adds tool shortlisting; Smolagent generates Python tool calls; OpenAI Solo uses MCP tool calls; and Claude Code is evaluated in an isolated Docker container without modifying its internal logic.Source: §3.2 · pp. 5–6

3.2.1 Agent Architectural Components

  • The component-level analysis tracks execution runtime, tool shortlisting, schema guards, communication protocols, memory, and planning across otherwise different agent implementations.Appendix B.3 documents which components are present in each evaluated agent.Source: §3.2.1 · p. 6

3.3 Metrics

  • Performance is measured with success rate, cost per task, average steps, bench-weighted mean, and cost-efficiency, alongside preregistered-style statistical tests and multiplicity corrections.Step-based analyses exclude 7.0% zero-step sessions and cap remaining step counts at 50; variance decomposition, paired comparisons, McNemar tests, z-tests, Spearman correlations, and Benjamini–Hochberg/Benjamini–Yekutieli corrections are specified.Source: §3.3 · p. 6

4 Results

Across six benchmarks, backbone model choice dominates aggregate performance, while architecture effects emerge through model–architecture interactions, benchmark-specific sensitivity, cost, and distinctive failure modes. General agents match customized specialists on four benchmarks, but open-weight models exhibit architecture and benchmark sinks absent from frontier closed-source models.

4.1 The Open General Agent Leaderboard

  • OpenAI Solo+Claude Opus 4.5 leads the leaderboard, while all top-ten configurations are closed-source and the top three cluster within 6pp.The gap from this top cluster to other backbone models is significant (p < 0.001).Source: §4.1 · p. 6

4.3 Model Choice Dominates Aggregate Variance; Architecture Decides Within Cells

  • Claude Opus 4.5 averages 0.66 success across architectures versus 0.59 for Gemini 3, 0.41 for GPT 5.2 and DeepSeek-V3.2, and 0.37 for Kimi-K2.5.Model choice explains 27.8% of variance versus 0.5% for architecture among closed-source configurations, while model–architecture interaction explains 5.4%.Source: §4.1 · p. 6, §4.3 · p. 7

4.2 Generalists Match Heavily-Customized Specialists

  • Across 4 of 6 benchmarks, the top general-agent configuration is statistically indistinguishable from the top reported domain-specific agent, with the optimal architecture varying by benchmark.Generalists match or favor specialists on SWE-Bench Verified, τ 2-Bench-Airline, τ 2-Bench-Retail, and AppWorld, but trail on BrowseComp+ and τ 2-Bench-Telecom.Source: §4.2 · p. 7, Table 3 · p. 7

4.4 Architectural Sensitivity Scales with Model Tier

  • Architectural spread is 7–12pp for closed-source backbones and 14–18pp for open-weight backbones, making architecture–model co-design more important for open-weight systems.Within-model architectural spread reaches 12pp for closed-source backbones and 14–18pp for open-weight backbones.Source: §4.4 · p. 7, §4.1 · p. 7

4.5 Cross-Benchmark Consistency

  • Closed-source benchmark scores are predominantly positively correlated, with median pairwise Spearman correlation +0.67 and range [+0.44, +0.81].BrowseComp+ has the lowest pairwise correlations, suggesting it captures somewhat distinct capabilities.Source: §4.5 · p. 8

4.6 Schema Guards and Tool Shortlisting Yield Cross-Model Gains

  • Schema guards characterize three top-performing architectures, while tool shortlisting improves ReAct performance for four of five backbones, including a 5.5-point gain for ReAct+GPT 5.2.ReAct+Claude Opus 4.5 also reduces cost by $1.97 per task, while DeepSeek-V3.2 is the lone exception.Source: §4.6 · p. 8

4.7 Generality Sinks

  • Open-weight models show architecture sinks and benchmark sinks: τ 2-Bench failures arise under autonomous architectures, while AppWorld collapses across architectures.On τ 2-Bench, 94% of Kimi-K2.5 autonomous sessions take zero steps versus 1.7% for closed-source models; on AppWorld, the best DeepSeek-V3.2+Smolagent reaches 0.13 versus 0.70 for Claude Opus 4.5+Smolagent.Source: §4.7 · p. 8

4.8 Cost-Efficiency Spans 30× Without a Universal Best

  • Cost-efficiency varies by approximately 30×: ReAct+GPT 5.2 and OpenAI Solo+DeepSeek-V3.2 tie at 2.43 score/$, while OpenAI Solo+Claude Opus 4.5 scores 0.09 score/$.The highest-scoring configuration is not the most efficient, and Claude Code+Claude Opus 4.5 is least efficient at 0.08 score/$.Source: §4.8 · p. 8

5 Related Work

Prior agent benchmarks target individual domains and define domain-specific protocols, while consolidation efforts remain limited by benchmark-specific adaptation or fixed transports. The Unified Protocol instead mediates between native agent and benchmark protocols through a canonical task/context/actions representation, preserving heterogeneous evaluation without per-benchmark agent adaptation.

Domain-Specific Agent Benchmarks

  • Existing benchmarks span software engineering, customer service, and deep scientific research, but each defines domain-specific protocols and task specifications [46] [15] [39] [23] [27].The surveyed benchmark landscape includes software engineering [18] [40] [26], customer service [42], and deep scientific research [10].Source: §5 · p. 9

Attempts at Consolidation

  • Prior consolidation systems either require per-benchmark agent adaptation or restrict evaluation to fixed interaction protocols such as web or CLI [12].Inspect consolidates infrastructure but still requires manual benchmark-specific selection of tools, sandboxes, and solvers; AgentBeats4 and CUBE [22] bind evaluation to specified transport subsets or schemas.Source: §5 · p. 9
  • The Unified Protocol wraps agents and benchmarks in their native protocols and mediates them through a canonical task/context/actions representation, enabling evaluation across heterogeneous benchmarks without per-benchmark agent adaptation.Exgentic implements this integration-layer protocol and can be layered on infrastructure harnesses such as Inspect.Source: §5 · pp. 9–10

6 Discussion … A.1 Adapting Existing Benchmarks

Exgentic and the Unified Protocol enable systematic evaluation of unmodified general-purpose agents across unmodified benchmarks, showing broad performance without domain-specific customization while exposing architecture, model, and cost tradeoffs. The framework adapts benchmark interfaces by making agent-visible assumptions explicit and protocol-independent.

6 Discussion

  • Exgentic and the Unified Protocol provide infrastructure for running unmodified agents on unmodified benchmarks and release the first Open General Agent Leaderboard.The leaderboard is intended as a public reference for measuring progress.Source: §6 · p. 10
  • Models must work consistently across agent architectures because the same open-weight model can range from strong performance to total failure depending on its paired architecture.Schema guards, tool shortlisting, and resistance to early termination identify concrete opportunities, while widely varying cost per task leaves cost-efficiency as another open axis.Source: §6 · p. 10
  • The evaluation is bounded to five LLMs, five agent implementations, and six benchmarks at approximately $20K, with per-benchmark Wilson CI half-widths of ±7–9.5pp.Aggregated comparisons remain highly significant, while multimodal and continuous-action extensions are deferred to Appendix I.Source: §6 · p. 10

A Framework and Adaptation Details

  • The framework positions evaluation along five axes: multi-protocol benchmarks, multi-protocol agents, agent plug-and-play, general-purpose tasks, and systematic factorial study.These axes capture support for heterogeneous native protocols, benchmark-independent integration, cross-domain tasks, and the full agent-architecture × backbone-model × benchmark comparison.Source: A Framework and Adaptation Details · p. 15

A.1 Adapting Existing Benchmarks

  • Existing benchmarks commonly couple tasks to fixed interaction protocols or rely on prior benchmark knowledge and human integration, so Exgentic decouples benchmark semantics from those assumptions.The adaptation process inspects reference agents, exposes required interactions and observations, and omits implementation-specific details and redundant signals.Source: §A.1 · pp. 15–16
  • For SWE-BENCH VERIFIED5, the reference MINI-SWE AGENT6 interface places the agent in a pre-cloned bash environment and automatically creates and submits a patch after completion.Exgentic therefore exposes explicit bash-command execution and patch-submission actions rather than treating repository cloning or patch creation as evaluation targets.Source: §A.1 · pp. 15–16
  • Task definitions retain essential domain instructions, such as τ 2-Bench’s customer-service policy, while excluding directions that assume a particular tool-calling protocol.This preserves intended task semantics while allowing any general agent implementation to interact through its native communication protocol.Source: §A.1 · p. 16

A.1.1 Per-Benchmark Adaptations … B Agent Adaptation

The paper maps heterogeneous benchmark interfaces and existing agents onto a benchmark-agnostic Unified Protocol, then runs them through an isolated, reproducible orchestrator. External adaptors preserve native agent and benchmark behavior while enabling interchangeable evaluation across supported tasks, agents, and LLMs.

A.1.1 Per-Benchmark Adaptations

  • AppWorld exposes approximately 468 APIs as Unified Protocol actions, while its supervisor completion becomes the finish action and implicit environment information moves into context.Any agent able to call the exposed actions can solve AppWorld; GPT 5.2 configurations without tool shortlisting score 0.00 because of the 128-tool request ceiling.Source: §A.1.1 · p. 17
  • BrowseComp+ is adapted with search, optional document retrieval, and structured submission actions, while the task includes the scoring rubric and requires submit to finish.The fixed corpus remains server-side, so the context field is empty and access occurs only through search or document retrieval.Source: §A.1.1 · p. 17
  • τ 2-Bench maps user replies to message actions, benchmark tools to Unified Protocol actions, and airline, retail, or telecom policies to context.Protocol-specific turn constraints are omitted because they constrain interaction shape rather than task semantics.Source: §A.1.1 · p. 17

A.2 Adapting Existing Agents

  • Existing-agent adaptation maps task and context fields to agent APIs, remains benchmark agnostic, and generally maps each action to one Python, OpenAI, or MCP tool.Context is typically concatenated with task instructions, though MCP agents could instead store it as MCP resources; user interactions may become message actions.Source: §A.2 · p. 17

A.3 Exgentic Orchestrator Design

  • Exgentic runs any supported agent on any supported benchmark task with any LLM through a few lines of Python or a dedicated GUI.The framework also supports parallelism, caching, isolated execution, reproducibility, and standardized benchmark, trajectory, and cost outputs.Source: §A.3 · p. 18
  • Each benchmark task becomes a session whose initialized task, context, and available actions drive an observation-action loop until completion, termination, or a safety threshold.The threshold prevents deadlocks or excessive costs.Source: §A.3 · p. 18

A.3.1 Solving the Integration Problem

  • External adaptors translate benchmark tasks and agent interfaces into the Unified Protocol while allowing both components to run natively in separate processes.The orchestrator mediates communication, preserving original benchmark and agent behavior without intrusive code changes.Source: §A.3.1 · p. 18

B Agent Adaptation

  • The five leaderboard agent integrations document pinned versions, prompts, first-turn instructions, integration approaches, and released adaptor code.These integrations support the Open General Agent Leaderboard’s agent configurations.Source: B Agent Adaptation · p. 18

B.1 Pinned Agent Versions · B.2 First-Turn Instructions per Agent · B.3 Architectural Components

The evaluation fixes exact agent versions and first-turn instructions across leaderboard cells, while cataloging the architectural components and protocol mechanisms that distinguish the evaluated agents. Exgentic standardizes agent–benchmark interaction through a unified protocol and adaptors.

B.1 Pinned Agent Versions

  • Exgentic’s orchestrator passes task definitions and mediates observations and actions between agents and benchmarks through protocol adaptors.The benchmark supplies quality metrics, while the agent supplies runtime cost.Source: Figure 4 · p. 19
  • Exact agent version strings are pinned for every leaderboard cell, with released results.csv using the same identifiers and Claude Code fixed at CLI version 2.1.7.Table 7 also specifies the first-party adaptor releases and the LiteLLM dependency pinned for ReAct Short.Source: §B.1 · p. 18, Table 7 · p. 19

B.2 First-Turn Instructions per Agent

  • The Unified Protocol exposes task, context, and actions, while adaptors add short, stable instructions that clarify each agent’s interaction requirements.These instructions are reproduced verbatim and adapt the shared fields to each underlying agent interface.Source: §B.2 · p. 19
  • ReAct exposes XML-tagged task and context fields with standard LiteLLM tool calls, whereas ReAct Short shortlists at most 30 relevant tools before each main call when needed.Shortlisting is bypassed when 30 or fewer actions are available.Source: §B.2 · pp. 19–20
  • Smolagent exposes Python functions and requires code-based action execution, while OpenAI Solo and Claude Code expose MCP tools and require environment-tool interaction to avoid failure.Claude Code runs unmodified in a Docker container, and OpenAI Solo uses documented solo mode.Source: §B.2 · pp. 20–21
  • Action translation maps protocol actions to Python functions, OpenAI tool specifications, or MCP tools, with special message and finish actions handled separately.This preserves the benchmark-defined termination methodology across agent interfaces.Source: §B.2 · p. 21

B.3 Architectural Components

  • The evaluated architectures vary in execution runtime, tool shortlisting, schema validation, communication protocol, memory, and planning, as summarized in Table 8.Execution environments support dynamic code execution; memory stores state beyond conversation history, while planning decomposes tasks into structured subgoals.Source: §B.3 · p. 21

C Benchmark Adaptation … Actions

The appendix adapts six heterogeneous benchmarks to a unified agent-facing protocol, preserving each benchmark’s task context and available actions across software engineering, deep research, personal assistance, and customer service.

C Benchmark Adaptation

  • Six benchmarks span software engineering, customer service, deep research, and personal assistance in the Open General Agent Leaderboard.The appendix presents the benchmark set and shows concrete task, context, and actions examples for each benchmark.Source: Figure 5 · p. 22, C Benchmark Adaptation · p. 22

Task

  • SWE-Bench Verified tasks require agents to resolve repository issues by directly editing source files on a remote machine.The example issue concerns a missing make_hashable call on through_fields in ManyToManyRel, producing an unhashable-list error; the prescribed solution adds the missing call.Source: Task · pp. 22–23

Actions

  • SWE-Bench actions expose bash for repository interaction and finish for submitting the completed patch.The benchmark requires filesystem edits and execution through bash, with the final patch captured from repository changes.Source: Actions · p. 24

Task

  • BrowseComp+ tasks require assembling answers from information discovered through multiple searches, with correctness and search efficiency determining performance.The task instructs agents not to message users and to submit a structured final answer through the required submission action.Source: Task · p. 25

Actions

  • BrowseComp+ provides search, optional full-document retrieval, and structured submission actions for research tasks.The available actions are search(query), get_document(docid), and submit(exact_answer, explanation, confidence).Source: Actions · p. 25

Context

  • AppWorld tasks operate across fixed applications and APIs, with credentials and authenticated access obtained through the supervisor and application interfaces.The environment includes services such as amazon, phone, file_system, spotify, venmo, gmail, splitwise, simple_note, and todoist.Source: Context · p. 26

Actions (∼468 total)

  • AppWorld exposes roughly 468 unified actions, generated from benchmark APIs with preserved names, signatures, and descriptions, plus a benchmark-level finish action.Examples include supervisor.show_profile, amazon.search_sellers, phone.send_text_message, file_system.create_directory, and simple_note.create_note.Source: Actions (∼468 total) · pp. 26–27

Context

  • τ 2-Bench airline tasks constrain agents to booking, modification, cancellation, refunds, and compensation while requiring explicit confirmation before booking-related updates.The airline subset maps the message action and registered tools including reservation lookup, human transfer, baggage updates, flight updates, and passenger updates.Source: Context · p. 27, Actions · p. 28

D Detailed Benchmark Agent Interaction Example … E.3 Generalist vs. Specialist Wilson CI Comparison

The appendix details how Exgentic mediates agent–benchmark interaction and provides full configuration, cost-efficiency, and generalist–specialist comparison results. Generalist parity holds on four of six benchmarks, while specialists lead beyond Wilson uncertainty on BrowseComp+ and τ 2-Bench-Telecom.

D Detailed Benchmark Agent Interaction Example

  • A SmolAgents code-generation agent interacts with τ 2-Bench through Exgentic wrapper functions that translate agent actions into queued protocol messages and return observations.The adaptor initializes a standard SmolAgent with the task definition and wrapper functions; calls block until corresponding observations arrive.Source: D Detailed Benchmark Agent Interaction… · p. 28
  • The orchestrator repeatedly passes actions to the benchmark and observations back to the agent until either side produces no further interaction, ending the session.The benchmark wrapper translates τ 2-Bench tool specifications into Exgentic actions and uses a proxy agent compatible with τ 2-Bench’s internal API.Source: D Detailed Benchmark Agent Interaction… · pp. 28–29

E Detailed Results

  • The appendix reports complete per-agent, per-model, and per-benchmark leaderboard results covering all 25 configurations across six benchmarks.Table 9 contains binary success rates deduped from runs.csv for five agents and five models.Source: E Detailed Results · p. 29, Table 9 · p. 29

E.1 Cost-Efficiency per Configuration

  • Table 10 reports cost-efficiency for all 25 architecture–model configurations, calculated as bench-weighted success divided by bench-weighted cost per task in dollars and sorted from most to least efficient.Source: §E.1 · p. 29, Table 10 · p. 30
  • Adding tool shortlisting reduces Claude Opus 4.5’s ReAct cost from $5.75 to $3.78 per task, while GPT 5.2 remains approximately flat at $0.17→$0.26.The reported Claude reduction is −$1.97 per task.Source: §E.1 · p. 29

E.3 Generalist vs. Specialist Wilson CI Comparison

  • Four of six benchmarks are indistinguishable between the best generalist configuration and top reported specialists; specialists lead on BrowseComp+ by 19pp and τ 2-Bench-Telecom by 10pp.The indistinguishable verdict is based on Wilson uncertainty or a generalist advantage; positive gaps favor specialists.Source: §E.3 · p. 30, Table 11 · p. 30

E.4 Tool Shortlisting Ablation … F.5 Step-Zero Rates

Across ablations and statistical analyses, backbone model effects generally exceed architecture effects for closed-source models, while open-weight models show architecture-sensitive failure patterns and benchmark-specific generality sinks. Tool shortlisting, failure-step dynamics, and step-zero rates further reveal protocol- and architecture-dependent behavior that aggregate scores alone can obscure.

E.4 Tool Shortlisting Ablation

  • Tool shortlisting broadly helps on AppWorld, led by GPT 5.2 at +22pp on AppWorld and +5pp aggregate, but DeepSeek-V3.2 regresses by −5pp and −1pp, respectively.Other benchmarks are unchanged because their action sets fit within all five models’ tool limits; AppWorld’s roughly 468 actions exceed the relevant tool limits.Source: §E.4 · pp. 30–31, Table 12 · p. 31

E.6 Per-Model Failure-Step Dynamics

  • Failures can terminate earlier than successes for open-weight autonomous agents, with DeepSeek-V3.2+OpenAI Solo reaching −27% in per-model bench-weighted step difference.The pattern is attributed to protocol-violation early termination; some Kimi-K2.5 cells lack sufficient τ 2-Bench data after filtering.Source: §E.6 · pp. 31–32, Table 15 · p. 32

F Statistical Significance

  • Individual benchmark differences smaller than approximately 8–10pp are uncertain, whereas aggregate scores use 550 instances per configuration and typically have 4–5pp confidence-interval half-widths.The evaluation contains 550 instances per agent-model configuration; backbone contrasts aggregate 2,750 observations per model with approximately ±1.8pp half-width.Source: F Statistical Significance · pp. 32–33

F.2 Pairwise Scaffold Comparisons

  • Pairwise architecture comparisons use paired t-tests on shared benchmark-task outcomes, while multiple-testing corrections preserve the reported qualitative conclusions and model-axis separations.All model pairs differ at p < 0.0001, whereas no architecture pair reaches p < 0.1; Benjamini–Hochberg and Benjamini–Yekutieli corrections are applied at α = 0.05.Source: F Statistical Significance · p. 33, §F.2 · p. 33, Table 17 · p. 33

F.1 Within-Model Scaffold Sensitivity

  • Closed-source architectures swing 7–12pp within model, compared with 14–18pp for open-weight models, showing greater scaffold sensitivity in the open-weight systems.The comparison uses each model’s best and worst architecture by bench-weighted mean success with paired tests on shared benchmark-task pairs.Source: §F.1 · p. 33, Table 16 · p. 33

F.3 Variance Decomposition (Closed-Source)

  • Model effects dominate closed-source variance, with model main effect F(2, 78) = 34.95 (p < 10^-10) versus agent main effect F(4, 78) = 0.16 (p = 0.96).Benchmark also has a significant main effect, F(5, 78) = 25.20 (p < 10^-14); including open-weight models makes the agent effect significant, F(4, 136) = 3.82 (p = 0.006).Source: §F.3 · p. 34, Table 22 · p. 35

F.4 Generality Sinks Decomposition

  • Generality sinks are concentrated in AppWorld and τ 2-Bench: open-weight capability gaps reach +57.0pp on AppWorld, while architectural-sensitivity gaps reach +54.6pp on τ 2-Bench-Retail.AppWorld is labeled a benchmark sink, whereas the three τ 2-Bench domains are labeled architecture sinks; SWE-Bench Verified and BrowseComp+ have no sink.Source: §F.4 · p. 35

F.5 Step-Zero Rates

  • Open-weight autonomous sessions show sharply elevated zero-step rates: Kimi-K2.5 reaches 94% and DeepSeek-V3.2 31%, versus 1.7% for closed-source models.The comparison covers Claude Code and OpenAI Solo sessions and is statistically significant for every pairwise comparison (p < 10^-15).Source: §F.5 · p. 35

F.6 Cross-Benchmark Spearman Correlations … G.3 Code Release and Audit Scripts

The appendices extend benchmark analysis to full configuration coverage, data provenance, and reproducibility infrastructure. They report cross-benchmark consistency, document evaluation settings and data quality, and release code and audits for recomputing every numeric claim.

F.6 Cross-Benchmark Spearman Correlations

  • Cross-benchmark correlations are predominantly positive across the 15 closed-model configurations, indicating that model identity drives consistency across benchmarks.Table 23 reports the Spearman rank correlations used for this analysis.Source: Table 23 · p. 35, §F.6 · p. 35

F.7 Full Leaderboard with Open-Weight Models

  • The full leaderboard covers all 25 combinations of five agent architectures and five backbone models, reporting per-benchmark success rates and a bench-weighted mean.The weighting uses 1/4 for each non-tau2 benchmark and 1/12 for each tau2 subdomain.Source: §F.7 · p. 35, Table 24 · p. 36

F.8 Data Quality and Run Provenance

  • Data-quality reporting records canonical completed/planned task counts and the number of raw run rows for every configuration cell.Duplicate or partial-run artifacts are collapsed by the deduplication step into the canonical run.Source: Table 25 · p. 36, §F.8 · p. 36

G Reproducibility Details

  • Appendix G releases the framework, benchmark adaptors, session-level results, aggregated runs, shared configuration, methodology conventions, and evaluated task_id pools needed to reproduce the analyses.It also identifies provider settings and audit scripts that recompute the paper’s numeric claims.Source: G Reproducibility Details · p. 38

G.1 Shared Run Configuration

  • All runs share a documented configuration, while full per-run settings, agent versions, prompts, tool templates, and benchmark adaptations are released in the framework and appendices.Per-benchmark prompts and tool-description templates follow the protocol in Appendix C.Source: §G.1 · p. 38, Table 26 · p. 39

G.2 Inference Arguments

  • Provider-documented default inference settings are used unchanged in every cell, avoiding hyperparameter-search confounding but potentially underestimating tuned absolute scores.The defaults are reported verbatim in Table 27; undisclosed or unsupported parameters remain so where providers do not specify or expose them.Source: §G.2 · p. 39, Table 27 · p. 39

G.3 Code Release and Audit Scripts

  • Released audit scripts recompute every numeric claim from the canonical filtered and normalized data, compare manuscript and computed values, and flag OK, DRIFT, or CONTRADICTED outcomes.Table 28 maps scripts to reproduced claims and includes a generator for all automatically generated appendix tables.Source: §G.3 · p. 39, Table 28 · p. 39

G.4 Data and Environment … H.3 Per-Model Profiles

The appendices document reproducible evaluation data and task selection, then characterize agent failures across 27 behavioral categories and model-specific error profiles. Failure prevalence is benchmark-dependent, while backbone models show distinct deviations in category shares.

G.4 Data and Environment

  • Primary configurations use three closed models across five architectures and six benchmarks, while the open-weight extension adds DeepSeek-V3.2 and Kimi-K2.5.The closed-only subset contains 90 cells; including open models expands the analyzed configuration set.Source: §G.4 · p. 40
  • ∼$20K covered 150 agent–model–benchmark cells, with roughly 30× per-task cost variation from ∼$2.43 score/$ to ∼$0.08 score/$.The efficiency frontier includes ReAct+GPT 5.2 and OpenAI Solo+DeepSeek-V3.2, while Claude Code+Claude Opus 4.5 is lowest-efficiency.Source: §G.4 · p. 40

G.5 Task IDs Used per Benchmark

  • The evaluated task set is pinned by 550 exact task_id values, including all 50 τ 2-Bench-Airline tasks, independently of sampling seeds or benchmark revisions.These identifiers match the released per-session results file and uniquely specify each benchmark subset.Source: §G.5 · pp. 40, 43

H Behavioral Error Analysis on Agentic Trajectories

  • The released analysis retains 27 behavioral failure categories over failed sessions with full trajectories, but is descriptive and not powered for causal claims.The taxonomy adapts ERRORMAP to long-horizon agent failures and was validated on a 100-record sample.Source: H Behavioral Error Analysis on Agentic… · p. 45

H.1 Setup

  • The trajectory analysis uses five architectures, five backbones, and six benchmarks, excludes zero-step orchestrator failures, and finds peer-success gold trajectories for 76.7% of retained sessions.Zero-step sessions lack trajectories; the gold trajectory comes from another architecture or model on the same task.Source: §H.1 · p. 45
  • The ERRORMAP port preserved its clustering pipeline, schema, and output format; deterministic synonym linking reduced Other from 25.8% to 2.1% across 27 categories.The end-to-end run used gpt-5.5 as judge, took 42 minutes, and cost $246.Source: §H.1 · pp. 45–46

H.2 Distribution of Failure Categories

  • Premature Termination is the largest failure category at 8.1% of categorized records (n = 2,868), while the top 10 cover 57.5%; category prevalence reflects benchmark mix.Diagnostic Protocol Error is 97% τ 2-Bench-Telecom, whereas Search Recovery & Adaptation is 92% BrowseComp+.Source: §H.2 · p. 46, Figure 6 · p. 46

H.3 Per-Model Profiles

  • Model failure profiles diverge: GPT 5.2 over-represents Escalation Handling by +4.6pp and has the lowest Unsupported Fabrication share at −4.3pp from mean.DeepSeek-V3.2, Kimi-K2.5, Claude Opus 4.5, and Gemini 3 Pro each over-represent different failure categories.Source: §H.3 · p. 46, Figure 7 · p. 46

H.4 Per-Architecture Profiles … H.7 Category Definitions

Failure profiles vary substantially by benchmark and meaningfully by architecture, despite architecture explaining little success-rate variance. The taxonomy is validated with 85% overall judge agreement and defines 27 top-level categories for interpreting these errors.

H.4 Per-Architecture Profiles

  • Architecture profiles diverge in failure composition: autonomous agents over-represent Premature Termination, tool-calling agents over-represent Evidence Retrieval Omission, and Smolagent elevates Search Recovery & Adaptation.The architecture axis produces non-trivial within-axis spread in failure categories despite contributing only η2 = 0.5% of success-rate variance.Source: Figure 8 · p. 47, §H.4 · p. 47

H.5 Per-Benchmark Profiles

  • BrowseComp+ has the largest single category deviation (+17.3pp Search Recovery & Adaptation), while τ 2-Bench-Telecom reaches +19.0pp Diagnostic Protocol Error and AppWorld shows the broadest spread.Benchmark profiles are highly distinct across all six benchmarks, with unequal sample sizes from n = 125 to n = 788; SWE-Bench Verified carries the largest uncertainty.Source: Figure 9 · p. 47, §H.5 · p. 47

H.6 Validation

  • The judge prefers the assigned failure label in 85/100 validation cases, with 90% agreement for named categories versus 70% for the residual Other bucket.The validation sample includes 27/100 Other cases to stress-test the residual bucket, where misclassification concentrates.Source: §H.6 · pp. 47–48

H.7 Category Definitions

  • The taxonomy contains 27 top-level failure categories generated by the LLM judge during recursive categorization and used as the classification reference.Definitions cover failures such as diagnostic protocol, entity resolution, escalation handling, evidence interpretation, evidence retrieval, and finalization.Source: Table 30 · p. 48, §H.7 · pp. 48–49
  • The definitions distinguish failures in numeric reasoning, query formulation, remediation, and search recovery, including incorrect calculations, poor retrieval queries, omitted fixes, and ineffective search adaptation.Search Recovery & Adaptation includes failure to retry, repair, pivot terms, decompose clues, change scope, or switch sources after retrieval problems.Source: §H.7 · p. 49
  • The taxonomy also captures state and goal tracking, tool invocation, unauthorized actions, and unsupported fabrication as distinct execution failures.These categories cover losing objectives or environment state, violating tool constraints, taking unapproved actions, and relying on unsupported facts or explanations.Source: §H.7 · p. 49
  • User Communication Error covers unclear or incomplete communication and proceeding without essential input, while Validation Recovery Failure covers ineffective responses to tool validation feedback.The latter includes repeating invalid calls, making ineffective repairs, or failing to choose a valid alternative path.Source: §H.7 · p. 50

I Limitations

The evaluation is limited by integration effort, benchmark scope, evaluation cost, and incomplete coverage of agents, models, stochastic reruns, and human judgment. Future work should extend the protocol and reduce evaluation costs through intelligent sampling and early stopping.

Integration limitations

  • Integrating new agents or benchmarks still requires familiarity with Exgentic’s capabilities and additional development work.Exgentic provides reusable adaptation building blocks, but integration is not fully turnkey.Source: I Limitations · p. 50

Benchmark and protocol scope

  • The study stresses cross-domain capability rather than multimodal processing, leaving continuous-action environments such as pixel-level computer use and robotics for future protocol extensions.The evaluated protocols accept multimodal inputs, but the Unified Protocol’s typed-action model does not yet support these continuous action spaces.Source: I Limitations · p. 50

Evaluation scope and cost

  • Cost constraints limit the evaluated agents and models, and the two-model open-weight tier cannot support claims about open-weight checkpoints generally.The study evaluates DeepSeek-V3.2 and Kimi-K2.5, runs each cell once with default sampling, and does not measure rerun variability or human judgments of partial-credit outputs.Source: I Limitations · p. 50

Future directions

  • Future evaluations should use intelligent sampling and early stopping when agent–model combinations clearly underperform, while addressing stochastic scores and benchmark-native evaluation limits.Native automated evaluators may diverge from human judgment, particularly on BrowseComp+ and SWE-Bench Verified.Source: I Limitations · p. 50

Related papers

Loading 2602.22953v2…