PaperlayerView exact revision on arXiv

Source-linked AI summary

General Agent Evaluation

Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, Michal Shmueli-Scheuer

arXiv:2602.22953v2cs.AI

TL;DR

General-purpose agents lack systematic evaluation across heterogeneous protocols and unfamiliar environments without domain-specific customization. This paper introduces a unified evaluation framework and leaderboard, finding broad adaptability, strong model effects, architecture-specific failures, and bounded scope.

  • Problem

    The central gap is systematic evidence about how general-agent architectures perform across heterogeneous protocols and unfamiliar environments without manual domain customization.

  • Method

    The paper introduces the Unified Protocol, Exgentic harness, and Open General Agent Leaderboard to evaluate unmodified agents and benchmarks across common interfaces.

  • Results

    General agents achieve meaningful performance across every tested domain, while model choice dominates aggregate variance and architecture can still swing within-model results by up to 12pp.

  • Takeaways & Limitations

    The findings support general agents as a viable research direction and show that behavioral failure analysis can reveal architecture-specific targets beyond aggregate scores.

  • Takeaways & Limitations

    The evaluation is limited to five LLMs, five agent implementations, and six benchmarks, with multimodal continuous-action settings left for future protocol extensions.

Abstract

from arXiv · show

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures x 5 backbone LLMs (three closed-source, two open-weight) x 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks; (v) a behavioral failure analysis reveals architecture-distinctive error signatures that aggregate scoring cannot discriminate. Code, harness, leaderboard, and traces are at https://www.exgentic.ai.

1 Introduction

The paper targets evaluation of general-purpose agents across unfamiliar, heterogeneous environments without domain-specific customization. It introduces a unified study and finds that model choice dominates aggregate performance, while architectures show distinctive failures and generalists match specialists on most benchmarks.

  • Motivation: Existing benchmarks and harnesses restrict general-agent evaluation through bespoke protocols, fixed interfaces, or benchmark-specific integration requirements.BrowserGym and Harbor exemplify fixed web- or CLI-based interfaces, while other systems require manual per-benchmark wiring.
  • Contributions: The study introduces a Unified Protocol, Exgentic harness, and public Open General Agent Leaderboard for systematic cross-agent, cross-model, and cross-benchmark comparison.The leaderboard evaluates agent configurations across heterogeneous domains and protocols.
  • Findings: 27.8% versus 0.5%: backbone model choice explains far more closed-source success-rate variance than architecture, although architecture can swing results by up to 12pp within one model.The aggregate model-versus-architecture gap is 58×, while architecture remains conditionally decisive for individual backbones.
  • Findings: On 4 of 6 benchmarks, the best general-agent configuration is statistically indistinguishable from the top published domain-specific score.BrowseComp+ and τ 2-Bench-Telecom remain specialist-led.
  • Findings: Open-weight models exhibit generality sinks, while behavioral analysis distinguishes architecture-specific failures that aggregate success rates obscure.Observed signatures include early stopping for Claude Code and OpenAI Solo and skipped evidence gathering for ReAct variants.

2 Unified Protocol Methodology

The Unified Protocol mediates between heterogeneous agent and benchmark interfaces through a shared task, context, and actions representation. Exgentic wraps original systems as black boxes so agents and benchmarks can be combined without per-benchmark adaptation.

  • Unified Protocol: The Unified Protocol acts as a narrow waist between agents and benchmarks that communicate through different native protocols.Adding an agent or benchmark requires adherence to the Unified Protocol rather than integration with every counterpart.
  • Design rationale: The protocol was derived from existing communication patterns and is intended to preserve their expressible semantics during translation.The paper presents it as a mediation layer rather than an imposed standard.
  • Representation: Each protocol instance contains three fields: task, context, and actions.Task describes the objective, context supplies information needed to complete it, and actions expose available environment interactions.
  • Representation: Implementers may designate message and final-answer actions to represent user communication and task completion.These optional action types support common interaction patterns across agent and benchmark protocols.
  • Exgentic harness: Exgentic adapts original agents and benchmarks through wrappers, then executes them as black boxes in isolated, reproducible sessions.The harness supports heterogeneous protocols including tool-calling APIs, MCP, Python code generation, bash/CLI, and conversational messaging.

3 Experimental Setup

The evaluation spans five agent architectures, five LLM backbones, and six benchmarks covering heterogeneous agentic domains and interaction protocols. It measures success, cost, steps, and statistically tested effects under standardized execution conditions.

  • Study design: 5 architectures × 5 LLMs × 6 benchmarks: the study evaluates 150 configurations with 100 tasks per benchmark, except 50 for τ 2-Bench Airline.The five architectures are ReAct, ReAct Short, Smolagent, OpenAI Solo, and Claude Code; the backbones include three closed-source and two open-weight models.
  • Benchmarks: The benchmark suite covers deep research, customer service, software engineering, personal assistance, and related heterogeneous environments.BrowseComp+ isolates agent reasoning with a fixed retriever, while τ 2-Bench evaluates policy-compliant customer-service completion and violation rejection.
  • Agents: The architectures use distinct interaction mechanisms, including tool calling, tool shortlisting, Python code generation, MCP, and CLI-based execution.ReAct Short is treated as a distinct architecture because tool shortlisting changes the substantive agent bundle.
  • Metrics: The evaluation reports success rate, cost per task, and average steps while preserving benchmark-specific success definitions and pricing-based cost estimates.Step analyses exclude zero-step orchestrator failures and cap remaining step counts at 50.
  • Analysis: Variance decomposition and paired comparisons use shared benchmark-task outcomes with multiplicity control across the reported statistical analyses.The paper applies η2-based variance decomposition, paired t-tests, McNemar tests, z-tests, and rank correlations.

4 Results

Results show that backbone models dominate aggregate performance, while architecture choices matter conditionally through model pairings, benchmark-specific sinks, cost trade-offs, and distinctive failure modes.

  • 4.1 The Open General Agent Leaderboard: Claude Opus 4.5 leads backbone models at 0.66 mean success, followed by Gemini 3 at 0.59, GPT 5.2 and DeepSeek-V3.2 at 0.41, and Kimi-K2.5 at 0.37.These rankings are consistent and significant across the five architectures.
  • 4.3 Model Choice Dominates Aggregate Variance; Architecture Decides Within Cells: Architecture rankings are not significant in aggregate, but architecture-model pairings matter: OpenAI Solo reaches 0.73 with Claude Opus 4.5 versus 0.39 with GPT 5.2.ReAct Short configurations are more consistent across backbone models.
  • 4.3 Model Choice Dominates Aggregate Variance; Architecture Decides Within Cells: 27.8% of closed-source success-rate variance is attributed to model choice versus 0.5% to architecture, while model-architecture interaction contributes 5.4%.Within a single backbone, architecture spread reaches up to 12 percentage points; including open-weight models makes the architecture main effect detectable.
  • 4.7 Generality Sinks: Open-weight models show larger architectural spread of 14–18pp than closed-source models’ 7–12pp, including architecture sinks and benchmark sinks.On τ 2-Bench, the same open-weight model can score 0.83 with one architecture and 0.00 with another; on AppWorld, every open-weight configuration collapses.
  • 4.6 Schema Guards and Tool Shortlisting Yield Cross-Model Gains: Tool shortlisting improves ReAct performance for four of five backbone models, with ReAct+GPT 5.2 gaining 5.5 percentage points overall.ReAct+Claude Opus 4.5 instead shows a smaller gain alongside a $1.97 per-task cost reduction; DeepSeek-V3.2 is the exception.
  • 4.10 Behavioral Failure Analysis: Architecture explains distinctive failure profiles despite only 0.5% success-rate variance, with premature termination, evidence-retrieval omission, and search-recovery errors concentrated by architecture.Across 2,868 failed sessions and 27 categories, the architecture-by-category association is significant: χ2(196) = 328.5, p < 10^-10, Cramér’s V = 0.17.

5 Related Work

Prior evaluation systems either require benchmark-specific agent adaptation or consolidate benchmarks behind a fixed protocol, limiting heterogeneous general-agent evaluation.

  • Domain-specific benchmarks target software engineering, customer service, and deep scientific research with domain-specific protocols and task specifications.
  • HAL unifies infrastructure but requires per-benchmark agent adaptation.
  • BrowserGym and Harbor standardize interaction through fixed web or CLI protocols, restricting evaluation to single environment classes.
  • Exgentic layers a Unified Protocol over infrastructure harnesses such as Inspect, preserving native evaluation across heterogeneous benchmarks without per-benchmark agent adaptation.

6 Discussion

The paper positions Exgentic and its leaderboard as infrastructure for studying unmodified general-purpose agents, while finding broad capability alongside model-dependent failures and bounded evaluation scope.

  • Exgentic enables unmodified agents to run on unmodified benchmarks and releases the first Open General Agent Leaderboard.
  • General agents achieve meaningful performance across every evaluated domain without per-domain customization, and top configurations are indistinguishable from domain specialists on most benchmarks.
  • The leaderboard identifies model consistency across architectures as a progress direction because open-weight models can shift from strong performance to total failure across architectures.
  • The evaluation covers five LLMs, five agent implementations, and six benchmarks at approximately $20K, with per-benchmark Wilson CI half-widths of ±7–9.5pp.
  • The Unified Protocol makes benchmark assumptions explicit by defining task, context, and action interfaces from benchmark and reference-agent behavior.

A.1.1 Per-Benchmark Adaptations

Per-benchmark adaptors translate native benchmark interfaces into the Unified Protocol while preserving task semantics, native agent behavior, and reusable cross-agent mappings.

  • Benchmark adaptations preserve native interfaces by mapping benchmark-specific actions, messages, and completion signals into Unified Protocol actions.
  • Agent adaptors are benchmark agnostic and map task, context, and actions to Python functions, OpenAI tools, or MCP tools.
  • Exgentic supports any currently supported agent on any supported benchmark task with any LLM through standard Python code or a dedicated GUI.
  • The framework executes isolated, reproducible runs and standardizes benchmark results, trajectories, and cost reports.
  • External adaptors translate protocols without intrusive changes, allowing independently running agents and benchmarks to retain their original behavior.

B.1 Pinned Agent Versions

The evaluation pins agent versions and specifies how each architecture exposes actions, handles termination, and supplies components such as runtimes, shortlisting, schema guards, memory, and planning.

  • Table 7 pins exact agent versions for every leaderboard cell, including the LiteLLM and Claude Code versions used by specific configurations.
  • The Unified Protocol exposes task, context, and actions, while each adaptor determines their presentation to the underlying agent.
  • ReAct Short queries the same backbone LLM to shortlist the 30 most relevant tools whenever more than 30 actions are available.
  • Smolagent exposes actions as Python functions, while ReAct variants use LiteLLM tool calls and OpenAI Solo and Claude Code use MCP tools.
  • Evaluated agents differ in execution runtimes, tool shortlisting, schema guards, communication protocols, memory, and planning.

C Benchmark Adaptation

The paper adapts heterogeneous benchmarks and agent environments into a unified evaluation setting, while preserving task semantics and protocol-specific execution. The adaptation is motivated by a Django hashing issue example and benchmark-specific interaction requirements.

  • Six benchmarks span software engineering, customer service, deep research, and personal assistance.
  • Benchmark adaptors translate native tasks, actions, and execution flows into the shared harness.The released adaptor code is organized per benchmark, with complete adaptation examples documented in the appendix.
  • The motivating Django issue arises because ManyToManyRel can pass list-valued through_fields into an identity used for hashing without the missing make_hashable call.The resulting failure is an unhashable-list TypeError during model checks, especially when checking proxy models.
  • Benchmark interactions expose native actions such as search, document retrieval, structured submission, application APIs, and finish operations.AppWorld exposes more than 460 additional tools beyond representative application actions.

D Detailed Benchmark Agent Interaction Example

The interaction example connects a code-generation agent to τ 2-Bench through queued actions and observations, while the benchmark wrapper translates its native interface into shared protocol actions. The surrounding appendix reports leaderboard, efficiency, and generalist–specialist comparisons.

  • Detailed Benchmark Agent Interaction Example: SmolAgent adaptors convert Exgentic actions into Python wrapper functions used by a standard agent instance.The wrappers coordinate agent-side function calls with orchestrator-side action handling.
  • Detailed Benchmark Agent Interaction Example: Agent function calls enter an action queue and block until the orchestrator returns observations through an observation queue.The orchestrator repeatedly invokes react(observation), forwarding actions to the benchmark and returning resulting observations to the agent.
  • Detailed Benchmark Agent Interaction Example: The interaction cycle ends when the agent produces no further actions or the benchmark produces no further observations.
  • Detailed Benchmark Agent Interaction Example: The τ 2-Bench wrapper translates native tool specifications into Exgentic actions and mediates simulated user messages through the same queues.
  • Detailed Benchmark Agent Interaction Example: The appendix provides full per-agent, per-model, and per-benchmark results, including cost-efficiency and generalist–specialist comparisons.The cost-efficiency tables cover all 25 architecture–model configurations, while the comparison table reports Wilson half-widths and benchmark gaps.

E.4 Tool Shortlisting Ablation

Tool shortlisting primarily addresses AppWorld’s unusually large action set and the resulting API-limit failures. Its effect is model-dependent, improving GPT 5.2 substantially while slightly reducing DeepSeek-V3.2 performance.

  • E.4 Tool Shortlisting Ablation: Shortlisting changes the tool-rich AppWorld benchmark, while the other benchmarks fit within all five models’ tool limits.
  • E.4 Tool Shortlisting Ablation: +22pp on AppWorld is the largest tool-shortlisting gain, achieved by GPT 5.2, whose 128-tool API limit otherwise drives the cell to 0.00.The aggregate GPT 5.2 gain is +5pp because AppWorld contributes one quarter of the bench-weighted score.
  • E.4 Tool Shortlisting Ablation: The evaluation uses paired comparisons on shared benchmark-task outcomes, confidence intervals, and multiple-testing corrections.The reported procedures include paired t-tests, McNemar tests, Wilson intervals, and Benjamini–Hochberg and Benjamini–Yekutieli corrections.
  • E.4 Tool Shortlisting Ablation: Open-weight models show larger architecture sensitivity than closed-source models, with 14–18pp versus 7–12pp best-to-worst swings.The agent main effect becomes evident when open-weight models are included in the variance analysis.
  • E.4 Tool Shortlisting Ablation: Architecture sinks on τ 2-Bench are especially concentrated among open-weight models violating autonomous tool-call instructions on the first turn.Kimi-K2.5 has 94% zero-step autonomous sessions, compared with 1.7% for closed-source models.

F.6 Cross-Benchmark Spearman Correlations

The appendix documents cross-benchmark correlations, complete configuration leaderboards, data-quality records, shared settings, and reproducibility scripts. These materials support auditing and reproduction of the paper’s comparative analyses.

  • F.6 Cross-Benchmark Spearman Correlations: Cross-benchmark Spearman correlations are reported across the 15 closed-model configurations.Predominantly positive correlations indicate consistency associated with model identity across benchmarks.
  • F.6 Cross-Benchmark Spearman Correlations: The full leaderboard contains success rates for all 25 agent configurations across six benchmarks.The rightmost leaderboard column is the bench-weighted mean across the six benchmarks.
  • F.6 Cross-Benchmark Spearman Correlations: Per-cell data-quality records distinguish completed and planned tasks and count raw run rows before deduplication.Values above one indicate reruns or partial-run artifacts collapsed to a single canonical row.
  • F.6 Cross-Benchmark Spearman Correlations: The released framework includes shared run configuration, provider-default inference settings, audit scripts, data-layout conventions, and evaluated task pools.Per-run agent versions, prompts, and benchmark adaptations are also documented and released.
  • F.6 Cross-Benchmark Spearman Correlations: Provider defaults are used without retuning across agents, models, or benchmarks.The authors state that this avoids confounding comparisons with hyperparameter search, while absolute scores may under-estimate tuned performance.

G.4 Data and Environment

The appendix fixes the evaluated data, scoring conventions, task subsets, and failure-analysis corpus. It also characterizes failure-category distributions across models and benchmarks.

  • Scoring conventions: Each task exposes continuous score and binary success fields, but SWE-Bench uses score ≥1 as success because session completion is near 99% while patch passing is near 37%.Other analyses use the binary success field.
  • Data and configurations: The primary configurations use three closed-source backbones, with two open-weight backbones added through an extension flag.The closed-only subset contains 3 × 5 × 6 = 90 cells.
  • Task subsets: The evaluation reports 550 task IDs per agent–model cell, including 100 tasks per benchmark and 50 τ 2-Bench-Airline tasks.The released IDs identify the evaluated subset independently of sampling seed or upstream benchmark revision.
  • Failure analysis: Failure analysis covers 27 behavioral categories over failed sessions with full trajectories and is explicitly descriptive rather than causal.The analysis adapts ERRORMAP to the leaderboard’s agent architectures, models, and benchmarks.
  • Failure distributions: Premature Termination is the largest category at 8.1% of 2,868 categorized records, while the top 10 categories cover 57.5%.Several categories are concentrated in particular benchmarks, so prevalence reflects the corpus benchmark mix.

H.4 Per-Architecture Profiles

Failure profiles differ substantially by architecture and benchmark, even though aggregate architecture effects on success-rate variance are small. The analysis uses category-share comparisons and judge validation to expose these distinctive error patterns.

  • Per-Architecture Profiles: Claude Code and OpenAI Solo over-represent Premature Termination, while ReAct and ReAct Short over-represent Evidence Retrieval Omission.Relative to the five-architecture mean, Premature Termination is +4.0pp for Claude Code and +7.3pp for OpenAI Solo, while Evidence Retrieval Omission is +2.1pp and +1.5pp for ReAct and ReAct Short.
  • Per-Architecture Profiles: Smolagent has a distinct profile with elevated Search Recovery & Adaptation, Authentication Handling Error, and Evidence Interpretation Error.Its deviations are +3.5pp, +1.9pp, and +1.7pp, respectively.
  • Per-Architecture Profiles: Architecture produces non-trivial failure-category spread despite explaining only η2 = 0.5% of success-rate variance.The behavioral dependent variable therefore reveals differences that aggregate success-rate scoring understates.
  • Per-Benchmark Profiles: BrowseComp+ has the largest benchmark-level deviation, over-representing Search Recovery & Adaptation by +17.3pp.It also over-represents Candidate Verification Failure by +8.4pp and Evidence Retrieval Omission by +6.1pp.
  • Per-Benchmark Profiles: SWE-Bench Verified over-represents Action Execution Error by +14.9pp and Premature Termination by +14.3pp.These benchmark-specific shares differ from the architecture-specific patterns shown in Figure 8.
  • Validation: The judge prefers assigned failure labels in 85/100 sampled cases, with higher agreement for named categories than for Other.Agreement is 66/73 (90%) for named categories and 19/27 (70%) for Other, where misclassification concentrates.

I Limitations

The evaluation is bounded by integration effort, protocol coverage, benchmark selection, and costly experimental scope. Extending it to multimodal continuous-action environments or broader model and agent populations requires additional work and evaluation.

  • Integration Scope: Integrating new agents or benchmarks still requires familiarity with Exgentic’s capabilities and additional development work.The reusable methodology reduces integration complexity but does not eliminate adaptation effort.
  • Protocol and Benchmark Scope: The study emphasizes cross-domain capability rather than multimodal processing, despite evaluating protocols that accept multimodal inputs.Continuous-action environments such as pixel-level computer use or robotic control require additions to the Unified Protocol’s typed-action model.
  • Evaluation Scope: Cost constraints limit the evaluation to five LLMs, five agent implementations, and six benchmarks, excluding the full range of open-source models and general-purpose agents.The open-weight tier contains only two checkpoints, so claims about open-weight generality sinks apply to those tested models.
Loading 2602.22953v2…