Source-linked AI summary
CocoaBench: Evaluating Unified Digital Agents in the Wild
CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang, Hexi Jin, Boyuan Zheng, Kun Zhou, Yu Wang, Feng Yao, Licheng Liu, Yijiang Li, Zhifei Li, Zhengtao Han, Pracha Promthaw, Tommaso Cerruti, Xiaohan Fu, Ziqiao Ma, Jingbo Shang, Lianhui Qin, Julian McAuley, Eric P. Xing, Zhengzhong Liu, Rupesh Kumar Srivastava, Zhiting Hu
TL;DR
Existing evaluations largely test digital-agent capabilities in isolation, leaving limited evidence about systems that must flexibly combine vision, search, and coding. The paper introduces COCOABENCH and the COCOA-AGENT scaffold for infrastructure-agnostic, automatically evaluated tasks and controlled backbone comparisons. Current agents remain unreliable, with the best evaluated system reaching only 45.1% success and weaknesses in planning, tool execution, and visual grounding.
Problem
Existing benchmarks typically assess only subsets of the capabilities required by general-purpose digital agents, limiting systematic evaluation of unified systems.
Method
COCOABENCH uses human-designed tasks requiring vision, search, and coding, specified by instructions and automatic final-output evaluation, while COCOA-AGENT enables controlled backbone comparisons.
Results
45.1% success rate is achieved by the best evaluated system, and current systems remain unreliable in reasoning and planning, tool execution, and visual grounding.
Takeaways & Limitations
COCOABENCH and COCOA-AGENT provide foundations for research on more capable general-purpose digital agents, while coding appears important for strong performance.
Takeaways & Limitations
Tasks are required to integrate multiple capabilities and pose nontrivial challenges for humans in realistic settings.
Abstract
from arXiv · showhide
LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.
1 Introduction
COCOABENCH addresses the gap between increasingly unified digital agents and benchmarks that still test capabilities in isolation. It evaluates flexible composition of vision, search, and coding through infrastructure-agnostic tasks with automatic outcome evaluation, finding current agents remain unreliable.
- Existing benchmarks largely focus on a single domain or interaction mode, despite agents increasingly combining software engineering, GUI automation, and deep research capabilities.
- COCOABENCH evaluates general-purpose agents on complex tasks requiring composition of coding, search, and vision.Coding supports code-based problem solving and structured tools, search supports information seeking and synthesis, and vision supports visual interpretation and GUI interaction.
- Tasks are specified by an instruction and final-output evaluation function, without dependence on a particular runtime, interface, or tool ecosystem.Outcome-based proxy evaluators support action-centric tasks while preserving open-ended workflows and enabling reproducible, scalable assessment.
- 45.1% success rate is achieved by the best evaluated agent, while Kimi-k2.5 and Qwen3.5 reach 11.8% and 9.8%, respectively.The benchmark therefore exposes substantial room for improvement in current agent capabilities.
- Current systems remain unreliable in reasoning and planning, tool interaction and execution, and visual grounding.
2 Related Work
Prior agent benchmarks capture useful but limited slices of digital-agent behavior, often restricting domains, interaction modalities, or infrastructure. COCOABENCH instead targets general-purpose agents by explicitly requiring composition of vision, search, and coding with reproducible evaluation.
- Representative benchmarks compare application focus, infrastructure coupling, reward verifiability, and coverage of vision, search, and coding.
- OSWorld studies computer use in VM-based desktops, while SWE-bench Pro and TerminalBench-2 focus largely on software-engineering and CLI execution domains.
- Existing systems such as SWE-Agent, OpenHands, Codex, Claude Code, and visual-interface agents typically operate within a single interaction modality.
- COCOABENCH and COCOA-AGENT address this gap with tasks requiring vision, search, and coding alongside a lightweight framework and integrated sandbox support.
3 CocoaBench
COCOABENCH is a human-authored benchmark of realistic, diverse tasks designed around multi-capability problem solving. Its infrastructure-agnostic instructions, automatic evaluators, quality controls, and stable resources support reproducible assessment.
- Task construction: 153 human-authored tasks span research, entertainment, shopping, business, and other everyday scenarios.Authors instantiated 3 to 5 concrete tasks for each scenario.
- Task construction: Each task must integrate multiple capabilities and pose a nontrivial challenge for humans in realistic settings.
- Task construction: External-resource dependencies are required to remain stable over time so task validity is not compromised by third-party changes.
- Inclusive task settings: Tasks are minimally specified by instructions and evaluation functions rather than fixed environments or tool ecosystems.Required multimodal assets and resources are hosted online and linked in task instructions.
- Automatic evaluation functions: Automatic evaluators check structured outputs or proxy outcomes that make correct results unlikely without successful multi-step execution.
- Quality control: Peer review verifies unambiguous instructions, defined output formats, correct reference answers, no trivial shortcuts, and accessible external resources.
- Task diversity and composition: 98% of tasks require multiple capabilities, labeled as Vision, Search, or Coding according to human annotations.The benchmark covers nine domains and uses diverse webpages, videos, images, and documents.
4 Experiment settings
The experiments evaluate diverse existing agent systems and model backbones, including unified, coding-oriented, research-oriented, and shared-scaffold configurations. COCOA-AGENT provides a lightweight, modular basis for controlled backbone comparisons.
- Existing agentic systems: Representative systems include ChatGPT Agent Mode, OpenClaw, Codex, Claude Code, and OpenAI Deep Research.
- Existing agentic systems: The evaluated systems cover browsing, coding, visual interaction, terminal execution, and long-horizon web information seeking.
- Shared scaffold: COCOA-AGENT is a lightweight, modular scaffold designed to control agentic components and make backbone comparisons more analytically interpretable.It is built on the AIO Sandbox runtime.
- Backbone models: The backbone comparison includes Claude Sonnet 4.6, GPT-5.4, Gemini-3.1-pro, Gemini-Flash-3.0, Kimi-k2.5, and Qwen3.5-397B-A13B.
5 Results and analysis
COCOABENCH results show that backbone and scaffold choices materially affect performance, while higher cost or completion time does not ensure better accuracy. Tool-use profiles and failure analyses reveal substantial differences in capability composition and persistent reasoning, execution, and visual-grounding challenges.
- Overall performance: 45.1% is GPT 5.4’s success rate under both Codex and OpenClaw, while it reaches 36.6% under COCOA-AGENT.GPT 5.4 is the most consistently strong backbone across scaffolds.
- Overall performance: 11.8% and 9.8% are the success rates of Kimi k2.5 and Qwen3.5 397B A13B, respectively, behind the leading proprietary models.Claude Sonnet 4.6 reaches 34.0% under OpenClaw but drops to 25.5% in Claude Code and 15.7% in COCOA-AGENT.
- Scaffolds: Coding-oriented scaffolds such as Codex and Claude Code can act as general problem solvers, while OpenClaw performs strongly with GPT 5.4 and Claude Sonnet 4.6.COCOA-AGENT is not the strongest-performing scaffold but supports meaningful backbone comparisons.
- Accuracy, cost, and time: Codex provides the best balance between cost efficiency and performance and lies on the Pareto frontier.Average cost ranges from $0.5 to $2.5 per task, and average completion time ranges from 380s to 3400s.
- Tool statistics: GPT-5.4 and Gemini 3.1 Pro allocate over 60% of tool calls to coding, whereas Kimi-k2.5 assigns 51.7% to vision tools.Gemini-Flash-3.0 directs 34.0% of calls toward DOM-level search operations, showing substantial variation in tool-use profiles.
- Error analysis: 712 of 918 task attempts were failure trajectories, and the analysis organizes failures into reasoning and planning, tool and execution, and visual-grounding classes.Compared with GPT 5.4, Kimi K2.5 shows more incorrect-reasoning and format errors over long interaction horizons.
6 Conclusion
COCOABENCH evaluates unified digital agents by requiring flexible composition of vision, search, and coding rather than isolated capability tests. Across end-to-end systems and controlled scaffold evaluations, current agents remain unreliable, while the benchmark and scaffold provide foundations for future research.
- Conclusion: COCOABENCH tests whether unified agents can flexibly compose vision, search, and coding to solve complex digital tasks.This extends evaluation beyond isolated capability tests.
- Conclusion: Current systems still struggle to solve COCOABENCH reliably across both end-to-end agent systems and controlled COCOA-AGENT evaluations.The analysis identifies weaknesses in planning and reasoning, tool use and execution, and visual grounding.
- Future research: COCOABENCH and COCOA-AGENT are presented as foundations for research on more capable general-purpose digital agents.COCOA-AGENT is a shared scaffold with an integrated sandbox and supports future work including data engineering and reinforcement-learning-based training.
C Failure Mode Taxonomy
The failure taxonomy categorizes COCOABENCH errors by whether their root cause lies in adaptive planning, execution, or visual perceptual grounding. COCOA-AGENT’s tools are grouped into vision, search, and coding capabilities, with a separate completion-control category.
- Failure taxonomy: Three hierarchical failure layers are defined: adaptive planning (E1), execution loop (E2), and visual perceptual grounding (E3).The taxonomy is based on trajectory analysis across different agentic systems on 153 tasks.
- Tool organization: COCOA-AGENT groups tools into Vision, Search, and Coding, while Control contains only the task completion signal.Vision covers GUI-level browser interaction; Search covers DOM-level access and navigation; Coding covers execution, shell commands, and file operations.
C.1 Type 1 Reasoning & Planning
Reasoning and planning failures include invalid strategies, imprecision, scope mistakes, and format errors, even when agents identify the right task or compute correct intermediate results. The examples show that execution details and output compliance can determine evaluation success.
- E1.1 Incorrect Reasoning: Incorrect reasoning occurs when an agent pursues a flawed logical path, displaces the goal, or omits a satisfiability check.Examples include submitting a path-dependent puzzle result after exploratory actions and producing a schedule despite unsatisfiable constraints.
- E1.2 Imprecision: Imprecision causes incorrect outputs when a sound procedure is applied with premature rounding or to the wrong data boundary.The gas-receipt example differs by 0.001 after intermediate rounding, while the thesis example counts appendix citations that should be excluded.
- E1.3 Format Error: Format errors arise when agents omit required tags or submit only part of a multi-part answer despite deriving correct information.In the Tableau example, the agent computed both requested insights but submitted only the highest-margin month.
C.2 Type 2 Tool & Execution
Type 2 Tool & Execution failures arise when agents cannot execute the right steps, recover from disruptions, or recognize that an interaction strategy has stalled. The examples span exhaustion loops, anti-bot misinterpretation, and context-truncation-driven repetition.
- Type 2 Tool & Execution: Poor execution can halt a task despite a sound high-level plan through tool misuse, missing recovery, or behavioral stagnation.The taxonomy treats these as breakdowns in the active interaction loop rather than failures of task logic.
- E2.1 Infinite Loop: Exhaustion loops consume the interaction budget through repeated low-level adjustments without progressing toward the task’s actual solution.In the nonogram task, the agent spent all 51 turns cropping and reading tiny regions instead of using a higher-level OCR-and-solving strategy.
- E2.2 Anti-Bot Barriers: Anti-bot failures occur when agents treat security interstitials as valid target content rather than recognizing the access barrier.The citation-analysis agent interpreted a Cloudflare challenge as an empty database result and returned zero citations.
- E2.3 Tool Result Hallucination: Tool-result hallucination includes fabricated outputs and continuing execution from corrupted memory after context truncation.In the GLP-1 analysis, evicted findings caused repeated retrieval and restarting until the run halted at turn 71.
C.3 Type 3 Visual Grounding
Type 3 Visual Grounding failures occur when agents misread visual details, lack the knowledge needed to interpret what they see, or fail to inspect rendered visual state. These failures can propagate into incorrect retrieval and final answers.
- E3.1 Visual Detail: Visual Detail failures involve missing small targets, misreading text, or misjudging thin objects and boundaries.A mountain-video task failed because one digit in an elevation label was read incorrectly despite successful frame isolation and OCR attempts.
- E3.2 Visual Knowledge: Visual Knowledge failures occur when agents perceive scene features correctly but cannot map them to the corresponding textual concepts.The color task extracted RGB values precisely but labeled a purplish palette as gray.
- E3.2 Visual Knowledge: Visual misidentification can poison later multi-hop reasoning and retrieval even when the initial image is successfully inspected.Misidentifying Lana Del Rey as Taylor Swift led to an award total of 3 instead of 7.
- E3.2 Visual Knowledge: Contextual visual knowledge can reverse an otherwise correct geometric interpretation, as with applying the wrong traffic convention at a roundabout.The agent perceived the layout and statue orientation but produced northwest instead of the expected northeast.
- E3.3 Missing Visual Perception: Missing Visual Perception arises when agents rely on DOM or raw data instead of inspecting rendered pixels, especially for canvas, SVG, or client-side visualizations.The Tableau example returned May / 26.0% from raw fields, whereas the rendered dashboard required March / 15.3%.
C.4 More Error Analysis
Across the six evaluated models, failure mentions are dominated by Reasoning & Planning, followed by Visual Grounding and Tool & Execution. The aggregate and per-model breakdowns show broadly similar high-level distributions.
- Aggregate breakdown: 54% of aggregate failure-mode mentions are Reasoning & Planning, compared with 17% Tool & Execution and 29% Visual Grounding.The aggregate covers 722 failed runs and all 9 active leaf subcategories.
- Per-model breakdown: 50% of Claude Sonnet 4.6 failure-mode mentions are Reasoning & Planning, with 17% Tool & Execution and 33% Visual Grounding.The chart summarizes 114 failed runs across 9 active subcategories.
- Per-model breakdown: 57% of GPT-5.4 failure-mode mentions are Reasoning & Planning, compared with 11% Tool & Execution and 32% Visual Grounding.The GPT-5.4 breakdown covers 96 failed runs and 8 active subcategories.
- Per-model breakdown: Reasoning & Planning remains the largest category for Kimi-k2.5, Qwen3.5-397B, Gemini 3.1 Pro Thinking, and Gemini 3 Flash.Their reported shares are respectively 55%, 56%, 55%, and 53%.
- Per-model breakdown: 59% of OpenAI Codex failure-mode mentions are Reasoning & Planning, while Tool & Execution accounts for 5% and Visual Grounding 36%.The distribution is based on 86 failed runs and includes 8 active subcategories.
C.5 LLM-as-Judge: Error Classification Prompt
The paper uses an LLM judge to assign taxonomy codes to failed trajectories from structured run summaries and compact action traces. The prompt permits multiple applicable codes and restricts outputs to parsable category lists.
- Judge inputs: The LLM judge receives the task description, expected answer, run summary, and a compact execution trace for each failed run.Screenshots and base-64 blobs are removed, while observations are truncated before the trace is assembled.
- Classification rules: The judge is instructed to assign every category that partially or primarily matches the failure, allowing multiple codes per trajectory.A free-text OTHER label is reserved for failures not captured by the taxonomy.
- Error taxonomy: The taxonomy groups failures into Reasoning & Planning, Tool & Execution, and Visual Grounding, with three leaf categories under each type.The leaf codes are E1.1–E1.3, E2.1–E2.3, and E3.1–E3.3.
- Output format: The classification prompt requires only a comma-separated list of Ex.x codes, making the judge output unambiguously parsable.The prompt also permits an optional OTHER:<short-phrase> escape.
- Trace construction: Observation truncation preserves opening structure for each action type before applying a total trace cap.The stated purpose is to retain diagnostic signal while keeping the judge prompt within the model’s effective context.