Source-linked AI summary
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, Pengfei Liu
TL;DR
Existing benchmarks provide limited evidence about autonomous agents’ performance on long-horizon, real-world tasks and do not scale easily when evaluation depends on human feedback. AgencyBench addresses these gaps with a large, diverse benchmark and automated rollout evaluation, finding that closed-source models outperform open-source models on average.
Problem
Existing benchmarks often lack long-horizon, diverse tasks, while human-in-the-loop feedback makes realistic rollout collection and evaluation difficult to scale.
Method
AgencyBench evaluates 6 agentic capabilities across 32 real-world scenarios and 138 tasks using simulated user feedback, Docker sandboxes, and automated rubric-based assessment.
Results
Closed-source models average 48.4%, outperforming open-source models’ 32.1% average score across AGENCYBENCH.
Takeaways & Limitations
The benchmark supports studying resource efficiency, feedback-driven self-correction, tool-use behavior, and model–scaffold compatibility in extended agent execution.
Takeaways & Limitations
The benchmark evaluates software-based agents in digital environments and does not cover embodied agents or physical-world interaction.
Abstract
from arXiv · showhide
Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.
1. Introduction
AGENCYBENCH addresses the limited realism and scalability of existing agent benchmarks with long-horizon tasks and an automated evaluation pipeline. It evaluates diverse real-world scenarios and finds a substantial performance gap between closed- and open-source models.
- Existing benchmarks often focus on single capabilities or lack long-horizon tasks, while realistic human feedback limits scalable rollout collection and evaluation.
- AGENCYBENCH evaluates 6 core agentic capabilities across 32 real-world scenarios containing 138 tasks with queries, deliverables, and rubrics.A typical scenario requires approximately 90 tool calls, 1 million tokens, and hours of execution.
- The benchmark combines multi-turn agent interactions, simulated user feedback, Docker-based operation execution, and automated rubric-based artifact assessment.Deliverables move from an isolated workspace to a Docker sandbox and then to eval-space for scoring.
- Closed-source models average 48.4%, compared with 32.1% for open-source models on AGENCYBENCH.The reported results also identify differences in self-correction, token efficiency, tool preferences, and scaffold performance.
- The study reports distinct model behaviors and a scaffold ‘home-field advantage,’ with peak performance occurring in native or specifically optimized frameworks.
- The authors provide a challenging benchmark, a unified automated evaluation framework, and comparative analysis of frontier models.
2 Related Work
Prior work has produced agent benchmarks for specific domains and capabilities, but many remain too shallow to test frontier agents on complex, extended interactions. AGENCYBENCH responds by increasing task authenticity, interaction depth, and context demands.
- Existing benchmarks span tool use, software development, open-ended research, and potential economic utility, often targeting specific vertical domains.
3. AGENCYBENCH
AGENCYBENCH organizes authentic tasks into long-horizon scenarios and evaluates them through isolated execution, simulated feedback, and automated rubric-based scoring. Its rollout formalization captures sequential tasks and iterative agent responses.
- 3. AGENCYBENCH: AGENCYBENCH targets a benchmark gap by using 138 authentic, high-fidelity tasks across 6 capabilities, with average rollouts exceeding 1M tokens and requiring over 90 tool calls.
- 3. AGENCYBENCH: The benchmark is hierarchically structured across game development, frontend and back-end development, code generation, research, and MCP tool use in 32 scenarios.
- Workspace, Eval-space and Scaffold: Agents work in isolated workspaces with tool scaffolds, then deliverables are transferred to eval-space for automated executable-script assessment.
- 3. AGENCYBENCH: Each task is collected with a query, expected deliverables, and rubrics that define evaluation criteria.
- Rollout Collection: A scenario rollout concatenates five sequential task trajectories, each containing queries, agent reasoning and tool calls, and feedback-triggered actions.
- Rollout Collection: The user simulation agent returns failed rubrics and reasons when deliverables miss the prescribed threshold, enabling targeted iterative improvement.Human verification over 50 sampled rollouts produced an average alignment score of 4.69.
- Evaluation: Evaluation maps deliverables to scores from 0 to 10 using rule-based assertions for objective tasks or text and vision LLM judges for subjective and visual outputs.For visual tasks, the final score averages text- and vision-based ratings; human comparison yielded a Kappa score of 0.93.
4. Experiments
Experiments compare models across performance, feedback-based correction, resource efficiency, tool-use behavior, and scaffold compatibility. Results show substantial differences in capability, cost, strategies, and framework sensitivity.
- Overall Performance: 56.5% was GPT-5.2’s highest overall average score, while GLM-4.6 led open-source models at 38.6% and Qwen-3-235B-A22B-Thinking scored 27.0%.GPT-5.2 and Gemini-3-Pro required the fewest average attempts at 1.46, compared with 1.79 for Qwen-3-235B-A22B-Thinking and Kimi-K2-Thinking.
- Agentic Capabilities: 60.7% and 81.0% were Gemini-3-Pro’s scores in game and front-end tasks, while Claude-4.5-Sonnet achieved 71.4% in research.Performance varies across capabilities, with GPT-5.2 excelling in back-end and code and Qwen-3-235B-A22B-Thinking showing relative strength in research.
- Feedback-driven Self-correction: 300% was Kimi-K2-Thinking’s improvement after feedback, compared with 0.0% Rise for DeepSeek-V3.2.GPT-5.2 achieved an 88.9% relative increase, while Gemini-3-Pro’s improvement was 33.3% Rise despite matching GPT-5.2’s initial Pass@1.
- Resource Use: 3.4 million tokens and 89 turns were GPT-5.2’s average resource use, whereas Grok-4.1-Fast used 1.2M tokens and 0.3h.GLM-4.6 used 105 turns with moderate resource usage, illustrating different performance-cost profiles.
- Attempt and Token Efficiency: 38.7% was GPT-5.2’s Attempt Efficiency, while Grok-4.1-Fast led Token Efficiency at 37.2% and Claude-4.5-Sonnet ranked lowest at 11.4%.Claude-4.5-Sonnet’s 4.1M tokens did not yield proportional performance gains.
- Tool Invocation Patterns: 77.6% of Qwen-3-235B-A22B-Thinking’s tool use involved file operations, while Claude-4.5-Opus and GPT-5.2 preferred shell execution at 45.5% and 43.5%.Grok-4.1-Fast and GLM-4.6 relied heavily on web search at 9.5% and 8.6%, respectively.
5 Conclusion
AGENCYBENCH evaluates autonomous agents in long-horizon, real-world contexts using authentic tasks and an automated evaluation pipeline. Results show persistent performance, efficiency, and scaffold-dependence challenges.
- AGENCYBENCH combines 138 authentic tasks across 32 scenarios, averaging 1 million tokens and 90 tool calls per scenario.The benchmark is intended to bridge toy evaluations and real-world economic-production complexity.
- The unified pipeline uses user simulation agents and Docker-based remote sandboxes to support scalable and reproducible automated evaluation.
- Proprietary models lead in complex reasoning and self-correction, but the gap between closed-source and open-source models remains significant.
- Advanced models still struggle with long-horizon autonomy without substantial resource consumption, highlighting the need for improved efficiency.
- Performance is highly sensitive to the interaction environment, with models often exhibiting a native advantage within proprietary ecosystems.
- AGENCYBENCH is positioned as a diagnostic tool for developing more resource-efficient, self-correcting, and scaffold-agnostic agents.
Limitations
The evaluation covers representative models rather than the full and rapidly changing model landscape, and it focuses on software-based agents in digital environments. These boundaries limit how broadly its findings can be generalized.
- The benchmark cannot exhaustively cover every emerging model variant, intermediate checkpoint, or specialized fine-tune because of computational and budget constraints.
- The benchmark evaluates high-complexity tasks in digital environments, including game development, software engineering, and web research.
- The framework does not extend to embodied agents or tasks requiring physical-world interaction, such as robotics.
Ethical Considerations
AGENCYBENCH reports ethical safeguards for its human contributors and operational safeguards for autonomous code and shell execution. Human data were anonymized and contributors were compensated fairly, while evaluations ran in isolated containers.
- Human contributors were informed about the project, their data were anonymized, and they were compensated above the local hourly minimum wage.
- The project states that it followed the ACL Code of Ethics for surveys and expert data validation.
- Evaluations involving executable code and shell operations were confined to isolated Docker containers with controlled network access.
- The containment measures were intended to prevent autonomous execution from harming host systems.
A Appendix
The appendix supplements the main benchmark description with dataset composition, tool-use analysis, evaluation prompts, and a concrete long-horizon scenario example. It also summarizes how scenarios and tasks are distributed across six agentic capabilities.
- The appendix presents the distribution of scenarios and tasks across AGENCYBENCH’s six core agentic capabilities.
- It provides detailed statistics describing the composition of the benchmark dataset.
- It analyzes tool-use frequency across different models.
- It documents prompts for the Text-based Judge, Vision-based Judge, and User Simulation Agent.
- A game-development scenario illustrates multi-turn, long-horizon interaction behavior.
A.1 Dataset and Tool Statistics
AGENCYBENCH combines broad scenario coverage with model-specific analyses of tool-use behavior. Its statistics expose distinct navigation, editing, memory, and information-retrieval strategies across models.
- Dataset Composition and Domain Diversity: 32 scenarios and 138 tasks span six core agentic capabilities, with Game Development comprising 36.2% of tasks.The Game Development domain includes 50 tasks across 10 scenarios.
- Tool Usage and Behavioral Fingerprints: Tool-usage analysis identifies three behavioral archetypes across models.The reported categories include navigation versus execution, editing styles, and memory utilization, with information retrieval also analyzed.
- Tool Usage and Behavioral Fingerprints: 158 list directory invocations make GLM-4.6 a navigator, while GPT-5.2 and Claude-4.5-Sonnet prioritize run shell command with 425 and 362 invocations.These patterns contrast environmental inspection with empirical code and script execution.
- Tool Usage and Behavioral Fingerprints: GPT-5.2 uses replace 146 times, whereas GLM-4.6 uses write file 381 times, reflecting localized editing versus whole-file rewriting.The rewrite strategy is described as safer but significantly less token-efficient.
- Tool Usage and Behavioral Fingerprints: Gemini-3-Pro alone records substantial memory-bank use, with 22 update memory bank and 7 initialize memory bank invocations.The passage contrasts this external state persistence with other models’ reliance on their context windows.
- Tool Usage and Behavioral Fingerprints: GLM-4.6 uses web fetch 96 times, while Claude-4.5-Opus and GPT-5.2 rely more on internal knowledge or search file content.This indicates distinct external-information retrieval profiles.
A.2 Evaluation Prompts SII-GAIR
The evaluation table organizes tool-invocation frequencies across models and highlights distinct usage patterns, including shell-heavy behavior and Gemini’s memory-tool use.
- Tool Usage and Behavioral Fingerprints: Table 6 reports tool-invocation frequencies across different models.The caption identifies high shell usage by Claude/GPT and specific memory-tool usage by Gemini as distinct patterns.
- Tool Usage and Behavioral Fingerprints: The table abbreviates Claude-4.5-Opus, Claude-4.5-Sonnet, Gemini-3-Pro, Grok-4.1-Fast, Qwen3-235B-A22B-Thinking, and Kimi-K2-Thinking.These abbreviations are used for Claude-4.5-O, Claude-4.5-S, Gemini-3, Grok-4.1, Qwen3, and Kimi-K2, respectively.
A.2 Evaluation Prompts
The appendix specifies role-based prompts for static code, visual, and feedback evaluation, then illustrates their use through a progressively specified Gomoku scenario. Together, these materials define inputs, scoring outputs, and concrete functional, visual, and behavioral checks.
- A.2 Evaluation Prompts: The evaluation framework uses Text-based Judge, Vision-based Judge, and User Simulation Agent roles to enforce rubrics and reduce subjective variance.The prompts are designed for rigorous, reproducible, and automated evaluation.
- A.2 Evaluation Prompts: Code scores range from 0–2 for critical failure to 8–9 for high compliance, with confidence and comments documenting evidential support.The prompt also specifies intermediate marginal-acceptance and substantial-deficiency bands.
- A.2 Evaluation Prompts SII-GAIR: The visual verifier compares screenshots or video frames against visual, functional, and aesthetic rubrics, marking unseen features as Not Demonstrated.Its output includes a score, confidence, and justification based strictly on visual evidence.
- A.2 Evaluation Prompts SII-GAIR: Visual scores range from 0–2 for no evidence to 10 for pixel-aligned compliance, with confidence reflecting evidence clarity and completeness.The scale distinguishes significant deviation, partial compliance, and high fidelity.
- A.2 Evaluation Prompts SII-GAIR: The feedback generator performs gap analysis between deliverables and rubrics when evaluation results fall below a threshold.It reports rejection, unmet rubrics, and root causes using the evaluation result, threshold, rubrics, deliverables, and artifacts.
- A.3 Scenario Example: The Gomoku example organizes tasks around explicit queries, deliverables, and rubrics, progressing from launch-view construction to gameplay extensions.The scenario demonstrates hierarchical task specification for a 15×15 board, assets, APIs, and interaction behavior.
- A.3 Scenario Example: The Gomoku rubrics test board layout, DOM and API contracts, turn order, duplicate-move rejection, status updates, win detection, replay, and layout regression.They require concrete checks such as 225 intersections, idempotent initialization, alternating colors, and a Black win after the specified sequence.