PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmark Test-Time Scaling of General LLM Agents

Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, Chenyan Xiong

arXiv:2602.18998v1cs.AIcs.CL

TL;DR

General-purpose agents need realistic evaluation across open-ended, multi-domain requests, beyond the constrained environments used by existing benchmarks. The paper introduces General AgentBench and studies sequential and parallel test-time scaling, finding substantial robustness degradation and practical limits from context ceilings and verification gaps.

  • Problem

    Existing benchmarks provide limited evidence about whether agents can infer intent, select tools, and scale performance across diverse domains within one framework.

  • Method

    General AgentBench evaluates general-purpose agents across diverse scenarios using a unified interface, shared tool pool, and end-to-end interaction process, while studying sequential and parallel test-time scaling.

  • Results

    Across ten leading LLMs, the general-agent setting produces a substantial performance drop relative to domain-specific evaluations, while sequential and parallel scaling face distinct practical limits.

  • Takeaways & Limitations

    The results indicate that robust general agents require better context management, reasoning control, and verification mechanisms rather than simply more interaction or sampled trajectories.

  • Takeaways & Limitations

    Sequential scaling rarely yields meaningful gains when raw interaction histories are extended, because long-horizon tasks expose context-utilization and reasoning-stability challenges.

Abstract

from arXiv · show

LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench.

1. Introduction

General-purpose agents must handle open-ended requests across diverse skills and tools, but existing domain-specific benchmarks do not capture this setting. General AgentBench addresses the gap and exposes two limits on test-time scaling: context ceilings in sequential interaction and verification gaps in parallel selection.

  • Real-world requests require agents to compose planning, reasoning, search, coding, computation, and tool use under uncertain conditions.
  • Existing benchmarks are usually domain-specific, whereas real interactions are multi-turn, open-ended, and span disparate domains.
  • General AgentBench evaluates general-purpose agents across diverse scenarios through a unified framework that hides domain-specific implementations while exposing a shared tool pool.
  • Across ten leading LLMs, moving from domain-specific configurations to the general-agent setting produces a substantial performance drop, with most models declining approximately 30%.
  • Sequential scaling improves performance only within a modest interaction range; beyond an effective context ceiling, longer histories often cause instability or degradation.
  • Parallel scaling raises the theoretical upper bound past@K, but practical gains are limited because agents often fail to select correct generated solutions.

2. General AgentBench

General AgentBench combines four task domains and their tools behind a shared interface designed to simulate diverse, evolving user interactions. Agents operate across the unified environment, selecting tools and interacting with active benchmark servers before final answers are evaluated.

  • 2.1. Domains and Sources: General AgentBench spans Coding, Search, Tool-use, and Reason, covering software engineering, information seeking, service workflows, and analytical reasoning.
  • 2.1. Domains and Sources: Coding tasks test production-level issue analysis and iterative execution, while Search tasks require identifying missing information and navigating evolving web contexts.
  • 2.1. Domains and Sources: Tool-use tasks require selecting, invoking, and coordinating multiple tools, whereas Reason tasks test sustained reasoning over noisy long-context mathematical inputs.
  • 2.2. Unified Realistic Evaluation Framework: The benchmark exposes all tasks and tools through one interaction interface, reflecting cross-domain diversity, comprehensive skill requirements, and evolving multi-turn interactions.
  • 2.2. Unified Realistic Evaluation Framework: A centralized Host maintains a global tool registry and routes agent calls from the shared tool space to the corresponding benchmark servers.
  • 2.2. Unified Realistic Evaluation Framework: Unified tool descriptions, user queries, and accumulated histories can grow into the long-context regime, requiring reasoning over instructions, documentation, feedback, and prior decisions.
  • 2.2. Unified Realistic Evaluation Framework: All benchmark servers are instantiated simultaneously, and even irrelevant tool calls return valid outputs, preserving the possibility of incorrect or irrelevant tool usage.
  • The evaluation covers ten frontier language models, including open-source and proprietary systems with different efficiency and attention designs.

3. Main Results

General AgentBench compares agents in unified, multi-domain settings and exposes substantial degradation relative to specialized evaluations, while cross-domain tool use can improve some search tasks.

  • Overall performance: Claude Sonnet 4.5 achieves the strongest overall General AgentBench performance, while GPT-5 leads Search and Reason.DeepSeek-V3.2 outperforms both Gemini variants among open-source models.
  • Overall performance: Most agents experience average relative performance drops of 10% to 30% in the general-agent setting.Gemini 2.5-Pro drops over 60% in Reason, while Claude Sonnet 4.5 has only 0.2% average degradation.
  • Overall performance: Figure 3 reports relative performance changes by domain, with negative values indicating degradation from specialized to general-agent settings.The comparison uses baseline specialized agents and general agents with unified context and tools.
  • Cross-domain tool usage: Several models improve on Search by repurposing specialized tools for cross-domain reasoning and information retrieval.Claude used specialized domain tools in 26% of 189 analyzed search tasks, including Maps, paper-search, and Hugging Face APIs.
  • Cross-domain tool usage: A case study shows that structured Hugging Face APIs return richer model metadata than iterative plain web search.The specialized API directly retrieves filters, download counts, tags, model IDs, and comprehensive model information.
  • Scaling overview: Figure 5 summarizes parallel sampling and sequential interaction-history scaling across five models and four domains.Parallel scaling expands sampled solutions, whereas sequential scaling shows unstable or diminishing returns.

4. Test-Time Scaling Evaluation

The evaluation studies sequential scaling through longer interaction histories and parallel scaling through multiple sampled trajectories. Sequential scaling encounters a context ceiling, while parallel scaling encounters a verification gap that limits practical gains.

  • Parallel scaling: Increasing K expands the reachable action space and raises the chance that at least one sampled trajectory is correct.Parallel scaling samples K trajectories independently relative to a single-shot baseline.
  • Parallel scaling: Self-choice evaluates and selects among sampled trajectories through point-wise or pair-wise strategies, whereas pass@K represents an idealized upper bound.Pair-wise choice performs K −1 comparisons before selecting one trajectory.
  • Scaling methodology: Sequential scaling extends interaction histories with environment feedback to encourage reflection and alternative solution paths.The agent receives another feedback round when attempting to terminate an episode.
  • Sequential scaling: Most models show little or no consistent sequential-scaling improvement despite additional iterative reasoning and reflection.Some exceptions include Qwen3-235B on Search and Deepseek-v3.2 on Reason.
  • Sequential scaling: Sequential behavior is either stagnant and fluctuating or initially improving before saturation and degradation.Reasoning commonly oscillates within a narrow range, while coding can degrade after a turning point.
  • Sequential scaling: A context ceiling emerges when accumulated history passes a model- and domain-dependent threshold.In Search, performance plateaus or degrades at approximately 112K tokens for Qwen3-235B and 96K for Gemini 2.5-Flash.
  • Parallel scaling: Parallel pass@K improves monotonically, with K increasing from 1 to 4 producing roughly 50% average improvement.DeepSeek-V3.2 approaches a twofold gain in Coding and Reasoning.
  • Parallel scaling: Self-choice consistently trails pass@K and can degrade as K increases, revealing a persistent verification gap.Even GPT-5 sometimes misclassifies correct trajectories, and self-choice gains often saturate quickly.

5. Related Work

Prior work covers specialized agent benchmarks, general-purpose agent frameworks, and test-time scaling methods. These directions span domain-specific evaluation, tool-using agents, reasoning allocation, search, refinement, and verification.

  • Agent benchmarks: Most agentic benchmarks evaluate specialized agents with task-specific environments, interaction protocols, and tool access.Representative domains include software engineering and web navigation.
  • General-purpose agents: General-purpose agent research develops systems for planning, acting, reflection, and tool invocation across heterogeneous tasks.Examples include ReAct, Reflexion, Toolformer, and HuggingGPT.
  • Test-time scaling: Test-time scaling allocates additional inference computation through deeper reasoning and multi-sample decoding.Related approaches include chain-of-thought prompting, self-consistency, search, and solution-path refinement.
  • Test-time scaling: Verifier-based inference augments sampling or search with learned ranking or rejection to improve reliability.Process-level supervision has improved reliability in mathematical reasoning.

6. Conclusions

General AgentBench evaluates general LLM agents in realistic multi-domain interactions and identifies robustness and scaling limits. The benchmark highlights context ceilings in sequential scaling and verification gaps in parallel scaling.

  • Conclusions: General AgentBench requires agents to infer intent, select tools from a shared pool, and act end-to-end across realistic multi-domain interactions.The benchmark is designed to assess general-purpose agents rather than only domain-specialized systems.
  • Conclusions: Across ten leading models, general-agent evaluation reveals a substantial robustness gap relative to domain-specific evaluation.The magnitude of degradation varies across models.
  • Conclusions: Sequential scaling is bounded by a context ceiling beyond which longer interactions become unstable.This limits the practical value of allocating computation through extended histories.
  • Conclusions: Parallel scaling provides limited practical gains because generation capacity exceeds reliable self-choice capability.The resulting verification gap prevents sampled solution potential from translating consistently into selected outputs.

Impact Statement

General AgentBench is intended for research and evaluation of general-purpose agents under unified, realistic interaction settings. The paper highlights both evaluation benefits and risks involving computational cost, self-verification, reliability, and user trust.

  • General AgentBench evaluates agents across diverse tasks and tools within a unified framework for interpreting requests, selecting tools, and scaling inference-time computation.The framework targets general-purpose systems rather than isolated domain capabilities.
  • Unified evaluation and test-time scaling may favor computationally intensive or proprietary models.
  • Unreliable self-choice under parallel scaling may produce unstable or misleading behavior if applied naively in deployment.
  • Failures in long-horizon reasoning, tool use, or self-verification may affect reliability and user trust as general-purpose agents enter real-world applications.
  • The benchmark spans search, coding, reasoning, and tool-use settings that require interaction with complex environments and, in several domains, multi-turn behavior.The supplied passages identify coding, reasoning, search, and tool-use benchmarks, while noting that some tasks are nearly unsolvable within short context windows.

A.3. Comparison with related long-context benchmarks

Agentic long-context settings differ from conventional long-context benchmarks in both context composition and output demands. These differences limit direct transfer from static benchmark performance to agentic interaction.

  • Unified toolsets can approach 64K tokens, while queries and multi-turn histories can push agent contexts toward nearly 128K tokens.
  • Context composition: Static long-context benchmarks typically use single-turn question answering, whereas agentic contexts combine documents, environment feedback, and prior model decisions.
  • Long-output reasoning: Agentic tasks require sustained reasoning, planning, reflection, and structured tool-call outputs over extended contexts, unlike benchmarks with relatively short outputs.
  • Performance on prior non-agentic long-context benchmarks does not directly reflect model behavior under agentic interaction.

A.4. Transferability of Long-Context Abilities from Static to Agentic Benchmarks

The study tests whether static long-context abilities transfer to agentic benchmarks and finds weak overall correspondence. It also examines attention behavior as a possible architectural explanation for differences in sequential scaling.

  • The transfer study evaluates LongBench, HELMET, and MRCR against General AgentBench results.
  • LongBench v2 primarily measures static long-context comprehension and retrieval rather than dynamic decision-making, planning, and self-conditioning in agentic multi-turn settings.
  • HELMET emphasizes robustness, faithfulness, information utilization, and attribution over extended contexts.
  • MRCR tests multi-round coreference, entity tracking, memory persistence, and consistency across long contexts.
  • Static long-context performance shows consistently weak correlation with agentic performance across ten models and four agentic domains.A moderate correlation appears between MRCR and the reasoning domain, which the passage describes as expected.
  • Qwen3-Next shows weaker sequential scaling potential than Qwen3-235B-A22B, motivating attention-focused analysis of their architectural difference.
  • The attention analysis samples 25 trajectories per domain and extracts reasoning-behavior sentences from inference trajectories.
  • Mean Attention Distance and Top-K Overlap measure effective contextual view and similarity of attended tokens across heads and layers.

B.2. Results and findings

Full-attention models use longer-range context and show more structured differentiation across heads and layers than linear-attention models. The section also describes how evaluation costs are estimated from token usage and provider prices.

  • Attention behavior: Full-attention models exhibit larger mean attention distances, with most heads attending to long-range context across layers.Only minor exceptions occur in early layers and around layer 70.
  • Attention behavior: Full attention shows a V-shaped intra-layer overlap pattern, whereas linear attention has higher average overlap and reduced head specialization.
  • Attention behavior: Full attention displays increasing inter-layer overlap across depth, while DeltaNet layers show high overlap across distant layers and homogeneous behavior.
  • Findings: Overall, linear attention shows weaker functional differentiation and reduced long-context utilization in agentic reasoning than full attention.
  • Pricing and accounting: The cost analysis compares default evaluation, parallel scaling with multiple trajectories, and sequential scaling over multiple interaction steps.
  • Pricing and accounting: Costs use provider-reported input and output token counts, with prices normalized to USD per 1M tokens.The estimate uses input unit prices because provider caching mechanisms and evaluation intervals are uncertain.

C.2. Model cost

The benchmark estimates evaluation costs using unit API prices and aggregates costs across complete benchmark splits under parallel, sequential, and default-context settings.

  • Unit prices for input, cached input, and output tokens are reported in USD per 1M tokens.
  • Tables 7–9 aggregate evaluation costs separately for parallel scaling, sequential scaling, and general default-context settings.Each entry covers the full benchmark split for a dataset or domain and sums costs across all queries.

D. General AgentBench Implementation Details

General AgentBench integrates diverse tools and benchmark environments through MCP-based tool registration, containerized execution, native evaluators, and self-choice procedures for judging trajectories.

  • Tool registration and schemas: The benchmark uses MCP’s Host-Client-Server architecture to establish transport connections, initialize sessions, discover tools, and route registered tools globally.Connections use STDIO or HTTP, and tools are discovered through list tools().
  • Tool registration and schemas: The toolset contains 301 tools across 35 servers, using OpenAI function-calling schemas with names, descriptions, and JSON-Schema parameters.Tool names follow Bedrock-compatible naming requirements, with format differences between MCP-Bench and Tau2.
  • Tool registration and schemas: Description compression truncates tool descriptions and removes parameter defaults, reducing token use by 18.6%, while minimal-tools converts JSON schema to plain text.The compression procedure preserves the first sentence when truncation would cut it midway.
  • Environment interaction and evaluation: Tau2Bench interactions are mediated by an adapter that converts agent responses, generates subsequent simulated-user messages, and maintains conversation state across turns.Terminal Bench and SWEBench use persistent MCP servers with Docker Bridge Mode for isolated task execution.
  • Environment interaction and evaluation: Evaluators delegate directly to each benchmark’s original evaluation code rather than reimplementing evaluation logic.Tau2Bench combines environment-state matching, action-sequence validation, and communication checks; containerized benchmarks run pytest harnesses through docker exec.
  • Self-choice evaluation: Point-wise self-choice prioritizes final-answer correctness, while pair-wise self-choice compares trajectories and uses bump-sort to select the best response.Point-wise judgments use Correct/Wrong tags extracted by regex, whereas pair-wise selection performs n−1 comparisons across n passes.
Loading 2602.18998v1…