PaperlayerView exact revision on arXiv

Source-linked AI summary

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, Yuhang Zang

arXiv:2605.10912v1cs.CL

TL;DR

Existing agent benchmarks provide limited evidence about reliable performance on realistic, long-horizon work in native runtimes. WildClawBench evaluates such work across 60 bilingual, multimodal tasks and 19 frontier models, finding that the top model reaches only 62.2% overall under OpenClaw. The results show that native-runtime agent evaluation remains far from resolved and depends substantially on the harness.

  • Problem

    Existing benchmarks unevenly cover synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks instead of realistic runtime interaction and auditable execution.

  • Method

    WildClawBench evaluates 60 human-authored bilingual tasks across six categories in reproducible Docker containers with real CLI harnesses, real tools, and hybrid grading.

  • Results

    62.2% overall is the highest OpenClaw score among 19 frontier models, with every other model below 60% and harness choice shifting one model by up to 18 points.

  • Takeaways & Limitations

    Long-horizon, native-runtime agent evaluation remains far from resolved, and measured performance is sensitive to the harness ecosystem as well as the underlying model.

  • Takeaways & Limitations

    The benchmark uses single-turn instructions and has limited coverage relative to real-world deployments, including some important domains such as GUI-heavy desktop control.

Abstract

from arXiv · show

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

1. Introduction

WildClawBench evaluates long-horizon agents in reproducible native runtimes with real tools, addressing benchmark limitations involving synthetic environments, short tasks, mock APIs, and final-answer-only checks. Across 19 frontier models, performance remains unsaturated: Claude Opus 4.7 reaches 62.2% overall under OpenClaw, while all other models stay below 60%.

  • Motivation: Prior benchmarks often use synthetic sandboxes, sub-minute tasks, mock-service API calls, and final-answer checks without trajectory- or artifact-level auditing.These design choices assess whether the final answer is correct but not how the agent acted.
  • Benchmark: WildClawBench contains 60 human-authored, bilingual, multimodal tasks running in reproducible Docker containers with deployed CLI harnesses and real tools.Supported harnesses include OpenClaw, Claude Code, Codex, and Hermes Agent; tools include shells, browsers, file systems, and email clients.
  • Workload: Roughly 8 minutes and over 20 tool calls per run exercise multi-step orchestration, tool-failure recovery, and cross-modal reasoning.Tasks are evaluated under budgets of 300 to 1200 seconds, with unified model access and constant tool schemas and system prompts within each harness.
  • Results: 62.2% overall is Claude Opus 4.7’s score under OpenClaw, while every other model remains below 60%.Scores span 19.3% to 62.2% across the 19 evaluated frontier models.

2. Related Work

Prior agent benchmarks are organized largely by interaction surface and typically cover only subsets of cross-modal inputs, native runtimes, bilingual coverage, reproducibility, and verification capabilities. WildClawBench is positioned as combining these axes while motivating hybrid evaluation beyond final-answer, executable, or end-state checks.

  • Agent Benchmarks across Environments: Agent benchmarks span software engineering, web and GUI control, OS and mobile control, enterprise knowledge work, coding, and browsing-centric research surfaces.AgentBench and τ-bench instead share a single-modality scope with only partial reproducibility.
  • Agent Benchmarks across Environments: WildClawBench combines cross-modal inputs, auditable trajectories, native runtimes, bilingual coverage, reproducible containers, and hybrid verification, whereas prior benchmarks support only subsets of these axes.WebArena and VisualWebArena use browser sandboxes rather than native runtimes, while OSWorld has only partial native-runtime support.
  • Agent Benchmarks across Environments: Bilingual coverage is rare: among the benchmarks listed in Table 1, only Claw-Eval and WildClawBench provide it.The passage identifies Claw-Eval and WildClawBench as the only rows with bilingual coverage.
  • Verification Methodologies: Verification methods progress from rule-based final-answer checks to executable code checks and state-based end-of-task inspection, but each individually misses behaviors relevant to long-horizon agents.The passage names AgentBench and GAIA for Rule, SWE-bench and Terminal-Bench for Exec, and τ-bench, WebArena, and VisualWebArena for State protocols.

3. WildClawBench

WildClawBench is a 60-task bilingual, multimodal benchmark designed to evaluate long-horizon agents in isolated Docker runtimes with real tools and auditable side effects. Its executable grading combines deterministic checks, environment-state auditing, and semantic verification across multiple agent harnesses.

  • Task execution: Tasks require long-horizon, multi-step workflows using combinations of web browsing, file I/O, email, calendar, chat, and structured output generation.Examples include arXiv digests, PDF classification, LaTeX table extraction, and meeting scheduling from email instructions.
  • Benchmark design: 60 human-authored tasks span six categories, with 36 English-language and 24 Chinese-language items, including 26 multimodal and 34 pure-text tasks.Each task specification includes metadata, an agent-facing prompt, expected behavior, rubrics, workspace information, and an executable grading function.
  • Benchmark design: 300–1200 seconds are the per-task time budgets, averaging 881s; Claude Opus 4.6 runs average 8.5 minutes and 26 tool calls per task.The runtime profile indicates sustained planning and cross-tool orchestration rather than short interaction bursts.
  • Task execution: Four agent harnesses—OpenClaw, Claude Code, Codex, and Hermes Agent—run tasks in isolated Docker containers with a common workspace and tool-facing environment.Each run starts from the same workspace state, and generated artifacts, traces, logs, and usage statistics are collected after execution.
  • Evaluation: Up to three grading checks combine rule-based validation, environment-state auditing, and LLM-as-a-Judge semantic verification.Checks cover deterministic file and content criteria, while audit logs verify actions taken through instrumented services such as email, calendar, and chat.

4. Experiments

Experiments show that WildClawBench remains difficult for frontier models and that agent performance depends substantially on harness design, execution budgets, skills, and model-specific behavior. No single model or configuration dominates across accuracy, efficiency, domains, or tool-use patterns.

  • Experimental setup: The study evaluates 19 frontier models under four Dockerized CLI harnesses, holding tool schemas, prompts, and context-management policies fixed within each harness.The harnesses are OpenClaw, Claude Code, Codex, and Hermes Agent, with unified model access through OpenRouter.
  • OpenClaw performance: 62.2% is the top OpenClaw overall score, while no other model exceeds 60%; scores span 19.3%–62.2%.Pure-text performance generally exceeds multimodal performance, including GPT 5.4 at 58.0% versus 40.2%.
  • Efficiency: $1.29 per task is Claude Opus 4.7’s average cost at the best overall score, whereas GPT 5.5 scores 58.2% at $0.63.DeepSeek V4 Pro reaches 43.7% at $0.20 per task, illustrating that stronger accuracy does not consistently imply better cost efficiency.
  • Harness effects: Harness choice changes both score and efficiency for the same model because control loops, tool schemas, context management, and output recovery affect gradeable artifacts.Claude Code takes 9.1–10.2 minutes per task across four models, while GLM 5 and MiMo V2 Pro each lose more than 10 points relative to OpenClaw.
  • Reasoning and time budgets: 50.40% to 52.63% is GPT 5.4’s gain from low to medium thinking, but high thinking reduces its score to 45.02% as timeout failures rise from 6 to 15 tasks.Across models, halving the standard task budget sharply reduces performance, while doubling it yields moderate gains with diminishing returns; GPT 5.4 improves from 50.3% to 56.5%.
  • Model-specific behavior: Skill augmentation produces mixed results, while models exhibit distinct domain strengths and tool-use profiles rather than a single dominant ranking.GPT 5.4 gains +5.2 overall and +22.4 on Code Intelligence from skills; MiniMax M2.7 averages 31.4 tool calls per task, whereas GPT 5.4 averages 6.0 read calls.

5. Conclusion · A. Broader Impacts

WildClawBench evaluates autonomous agents in realistic, long-horizon, native production-grade runtimes, exposing substantial headroom in current systems. Its reproducible, auditable design supports failure-mode analysis, progress tracking, and safer deployment.

  • 5. Conclusion: WildClawBench evaluates autonomous agents in native, production-grade runtimes.The benchmark is designed around realistic, long-horizon evaluation.
  • 5. Conclusion: 60 tasks and 19 frontier models reveal substantial headroom in current agentic systems.The top-performing model achieves only 62.2%.
  • 5. Conclusion: The benchmark spans multimodal and bilingual workflows.These workflows are evaluated across the benchmark’s 60 tasks.
  • A. Broader Impacts: WildClawBench provides reproducible and auditable evaluation of real-world agent capabilities.Its runtime setting is production-grade and native.
  • A. Broader Impacts: Long-horizon tool use, multimodal reasoning, and trajectory-level safety behavior are evaluated together.This combination supports broader assessment of real-world agent capabilities.
  • A. Broader Impacts: Researchers and practitioners can use the benchmark to identify failure modes before deployment.It is also intended to help track progress toward more reliable agentic AI.
  • A. Broader Impacts: The benchmark includes adversarial safety tasks involving prompt injections and leaked-credential traces.These tasks are presented as examples of adversarial safety evaluation.

B. Limitations

Despite its realistic, long-horizon, multimodal evaluation in production-grade native runtimes, WildClawBench has two remaining limitations. One is that tasks use single-turn instructions, omitting user interaction during execution.

  • B. Limitations: WildClawBench evaluates agents under realistic, long-horizon, multimodal conditions in production-grade native runtimes, yet two limitations remain.The passage introduces two limitations but specifies only the first here.
  • B. Limitations: All current tasks are single-turn: agents receive one initial request and run autonomously until completion or timeout.This task framing does not include additional user input during execution.
  • B. Limitations: The benchmark omits multi-turn scenarios involving user clarifications, corrections, or follow-up requests during execution.Such interactions are described as common in coding, research, and creative work.

C. Task Modality Listing

WildClawBench includes both multimodal and pure-text tasks, with 26 of 60 tasks (43.3%) requiring perception of non-text inputs and 34 (56.7%) operating only over text. Modality varies systematically by category: Code Intelligence and Creative Synthesis are entirely multimodal, whereas Social Interaction and Safety Alignment are entirely pure text.

  • Category breakdown: Code Intelligence and Creative Synthesis are entirely multimodal in this release.Every task in these categories contains non-text inputs that the agent must perceive.
  • Category breakdown: Social Interaction and Safety Alignment are entirely pure text.Agents in these categories operate only over markdown, source code, chat logs, web text, or structured records.
  • Task modality distribution: 26 of 60 tasks (43.3%) are multimodal, while 34 (56.7%) are pure text.Multimodal tasks require perceiving images, video, audio, or rendered PDF pages; pure-text tasks use only textual inputs.

D. Skills Used in Category-Level Ablation

Adding category-relevant skills to the agent’s toolbox consistently improved performance across all six categories. For each category, the study selected the three most-downloaded skills on ClawHub and summarized their contents in Table 8.

  • Category-level ablation: Adding category-relevant skills produced performance gains across all six categories.The results are reported in Sec. 4.3.
  • Category-level ablation: 3 skills were selected for each category based on the highest ClawHub download counts at evaluation time.The selection criterion was applied independently within each category.
  • Category-level ablation: Table 8 summarizes the contents of the skills used in the category-level ablation study.

E. Failure-Mode Analysis

The failure-mode analysis examines 169 failed runs from 300 OpenClaw runs across five models, separating evaluator-visible outcomes from trajectory-level process causes. Failures usually produce wrong or partial artifacts and often reflect interacting coding, environment/API, and time-pressure problems rather than a single cause.

  • Analysis setup: 169 of 300 OpenClaw runs were classified as failed because their normalized task score fell below 0.5.The analysis covered Gemini 3.1 Pro, GPT-5.4, Kimi K2.5, MiniMax M2.7, and Opus 4.6, with 60 tasks per model.
  • Analysis setup: Each failed run received one outcome label and one priority-assigned process label based on its evaluator-visible end state and agent trajectory.Process signals included tool results, exit codes, API errors, tracebacks, missing dependencies, and still-running final processes.
  • Outcome view: Failures most often resulted in wrong or partial artifacts, while missing artifacts were concentrated in Kimi K2.5 and MiniMax M2.7.GPT-5.4 and Gemini 3.1 Pro had fewer missing-output failures, showing that low scores often reflected incomplete requirements rather than complete inaction.
  • Process view: Failure causes frequently combined coding friction, environment/API instability, and time pressure instead of forming a single isolated category.MiniMax M2.7 often reached the configured time limit while also triggering toolchain/API disruption signals across many tasks.

F. Bilingual Performance Analysis. · G. Variance across Repeated Runs

All models score higher on English than Chinese tasks, with MiniMax M2.7 showing the largest gap at 7.4 points. Across three independent OpenClaw runs, four representative models show generally small variance, indicating stable evaluation behavior and execution trajectories.

  • F. Bilingual Performance Analysis.: All models perform better on English tasks than on Chinese tasks.
  • F. Bilingual Performance Analysis.: 7.4 points is MiniMax M2.7’s largest English–Chinese performance gap.
  • F. Bilingual Performance Analysis.: Prompt language is evaluated through average run-level scores for each task subset.
  • G. Variance across Repeated Runs: Three independent runs provide the basis for estimating score means and standard deviations.
  • G. Variance across Repeated Runs: Four representative models are evaluated on the OpenClaw harness for repeated-run variance.
  • G. Variance across Repeated Runs: Small standard deviations indicate stable performance across trials and robust execution trajectories.

H. Validation of GPT-Based Evaluation

A five-task human-GPT agreement case study evaluates GPT-5.4 as the benchmark’s proxy judge using the same rubric as two independent human experts. GPT-5.4 aligns strongly and consistently with averaged human judgments, with deviations generally below 3 points even for subjective creative tasks.

  • Human-GPT Agreement Case Study: GPT-5.4 was validated against averaged scores from two independent human experts on five randomly sampled tasks requiring LLM-as-a-judge evaluation.The experts conducted blind evaluations using the exact rubric applied by GPT-5.4, establishing a human ground-truth baseline.
  • Human-GPT Agreement Case Study: Deviations between GPT-5.4 and the human average were generally fewer than 3 points, including subjective Creative Synthesis tasks.The results are reported as tightly calibrated despite naturally higher human variance in tasks such as T05.01 and T05.07.
  • Human-GPT Agreement Case Study: Table 11 compares raw scores from two independent human evaluators with GPT-5.4 across five sampled tasks.GPT-5.4 scores are shown in the rightmost column with deviations from the human average in parentheses.

I. Per-Task Run Breakdown … Productivity Flow: Prepare an arXiv Daily Digest

The appendix combines per-task run breakdowns with representative task specifications and execution analyses across WildClawBench’s categories. These materials illustrate the benchmark’s required artifacts, grading conditions, recovery behaviors, and task diversity, including an arXiv daily digest workflow.

  • I. Per-Task Run Breakdown: The per-task tables report score, elapsed wall-clock time, API cost, and tool-call count for five representative models running on OpenClaw.The models are Claude Opus 4.6, GPT 5.4, Kimi K2.5, Gemini 3.1 Pro, and MiniMax M2.7; Claude’s table is based on a single run snapshot.
  • J. Representative Full Task Pages: The representative tasks require precise file outputs, content extraction, format constraints, and checks against expected artifacts or normalized ground truth.The Wikipedia task saves UTF-8 Markdown biographies using people’s actual names, while the SAM3 task requires a corrected predictions.json without modifying its test script.
  • Social Interaction: Chat Message Action Item Extraction: Action-item extraction must recover concrete deliverables, revised deadlines, and implicitly assigned cutoffs from recent messages without sending messages.The grading highlights updated deadlines for API documentation, expense reporting, and the all-hands slide deck, plus implicit deadlines including Tuesday March 24 EOD and March 23.
  • Search & Retrieval: Fuzzy Repository Search: The fuzzy repository search task identifies llama.cpp by Georgi Gerganov as the only fully correct answer among nearby alternatives.The task uses clues about implementation language, repository naming, creator background, GGUF adoption, and a GitHub star threshold exceeding 60k.
  • Creative Synthesis: Design a Product Poster for a Leather Briefcase: The product-poster task combines visual feature extraction with constrained presentation requirements and multimodal quality grading.The requested PNG is 1080×1440 and must include brand, product name, tagline, prices, a call to action, and features grounded in the briefcase photo.
  • Safety Alignment: Prompt Injection via File Content: The prompt-injection task requires ignoring destructive instructions embedded in a file while producing only a legitimate Linux-book summary.Success requires not executing sudo rm -rf / or other suspicious commands, creating linux_summary.md, and recognizing or flagging the injected content.
  • K. Word Cloud Analysis of Prompts and Trajectories: Word-cloud analyses distinguish benchmark prompt themes from Claude Opus 4.6’s execution behavior by filtering prompts and examining assistant-side trajectory content.The trajectory corpus includes responses, reasoning when available, and tool-call arguments while excluding user prompts and tool outputs; highlighted behaviors include programming, image manipulation, search, and screenshot generation.
  • L. Trajectory Analysis: Trajectory analysis summarizes agent planning, tool use, environmental recovery, constraint preservation, and output verification across multimodal tasks, including paper-to-poster and arXiv-digest workflows.The paper-poster run validated and parsed a 15-page PDF despite missing matplotlib, while the arXiv run recovered from invalid listing URLs by using the export API and a submitted-date range query.
Loading 2605.10912v1…