Source-linked AI summary
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks
Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, Xihui Liu
TL;DR
Existing benchmarks poorly capture proactive agents’ real-world, multi-turn, capability-specific performance. UniClawBench addresses this with 400 bilingual tasks and closed-loop evaluation across models and frameworks, finding that leading models’ overall success rates remain below 50%.
Problem
Existing benchmarks use sandboxed, single-turn, scenario-based evaluations that provide limited evidence about proactive agents’ real-world capabilities and failure causes.
Method
UniClawBench evaluates five capabilities across 400 bilingual real-world tasks using Docker execution, fine-grained checkpoints, and three-role closed-loop assessment.
Results
Leading models achieve the highest overall pass rates, but absolute success rates remain strictly below 50%, while framework architecture substantially influences task success.
Takeaways & Limitations
The benchmark shows that current agents struggle with long-horizon memory and cross-platform coordination, while framework architectures can bottleneck model capabilities.
Takeaways & Limitations
The benchmark is limited by its relatively small set of 400 manually curated tasks, live-environment instability, and potential bias from LLM-based evaluation.
Abstract
from arXiv · showhide
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.
1 Introduction
UniClawBench addresses major gaps in evaluating proactive agents by targeting capability diagnosis in dynamic real-world environments rather than relying solely on sandboxed, fixed-answer benchmarks. It introduces a capability-driven benchmark and closed-loop evaluation strategy, then compares models and agent frameworks to disentangle their effects on performance.
- Motivation: Proactive agents have evolved from text-centric assistants into systems that execute complex, multi-step tasks and directly control everyday tools in real-world environments.Examples include browsers and terminals, with platforms pursuing continuous personal assistance.
- Limitations: Existing benchmarks inadequately capture real-world complexity because self-hosted website mirrors and cached virtual-machine pages create a gap from actual capabilities.The passage identifies sandboxed environments as a structural limitation of current evaluation approaches.
- Challenges: Dynamic real-world evaluation makes fixed ground truth unreliable because changing conditions can invalidate pre-recorded expected outputs.The passage illustrates this with product prices that vary across days.
- Benchmark: UniClawBench is introduced as a capability-driven benchmark spanning Skill Usage, Exploration, Long Context, Multimodal, and Cross-Platform dimensions across 400 bilingual daily-use tasks.The benchmark is designed to diagnose root causes of agent failures in dynamic real-world environments.
- Experiments: The study compares state-of-the-art models under OpenClaw and representative models across OpenClaw, EDICT, and Nanobot to separate base-model and framework effects.These cross-model and cross-framework experiments examine how framework design affects task performance.
2 Related Work
Prior work established tool-using autonomous agents and proactive systems, while agent benchmarks increasingly evaluate real-environment task execution but remain distinct from proactive personal assistance. These developments motivate capability-focused evaluation of long-running, multi-platform agents.
- Large language models as autonomous agents: LLM agents interleave reasoning, action, observation, and tool use, with ReAct introducing a representative reasoning–acting loop and Toolformer learning external-tool invocation.Subsequent systems pursued richer orchestration and longer-horizon problem solving.
- Proactive agent systems: Proactive agents initiate actions and pursue goals autonomously, supported by persistent memory, reusable skills, and local tool access.OpenClaw emphasizes multi-platform integration, while Nanobot emphasizes a lightweight, extensible core for research and modification.
- Benchmark for AI Agents: Earlier agent benchmarks span web, OS, mobile, and enterprise environments but primarily target isolated task execution rather than proactive personal assistance.More recent benchmarks target proactive agent systems, including end-to-end completion in real environments such as ClawBench’s write-heavy actions on live production websites.
- Benchmark for AI Agents: Closed-loop evaluation can use an Executor, hidden Supervisor, and user interaction to assess real-world trajectories with checkpoint-based rubrics while limiting evaluation-criteria leakage.The Supervisor applies private references to compute structured states such as pass, fail, or continue, and a score.
3 UniClawbench
UniClawBench evaluates proactive agents through a capability-oriented taxonomy covering five foundational capabilities and 400 bilingual real-world tasks. Its closed-loop, three-role evaluation runs agents in Docker environments while separating hidden grading criteria from agent-facing feedback.
- Capability-oriented task taxonomy: UniClawBench organizes tasks around five capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.The taxonomy targets diagnostic bottlenecks rather than application domains or interaction surfaces.
- Benchmark composition: 400 bilingual real-world tasks comprise 40 English and 40 Chinese tasks for each capability, with scenarios designed to primarily bottleneck on one capability.Each task package includes resources, tools, skills, expected outputs, and hidden evaluation references.
- Benchmark composition: The benchmark spans diverse application domains and input/output modalities to reduce scenario bias and reflect heterogeneous real-world agent tasks.Its composition includes broad domain coverage and varied formats rather than relying exclusively on textual interactions.
- Closed-loop evaluation: The closed-loop design uses an executor, hidden supervisor, and user simulator to model multi-turn feedback while preventing hidden grading criteria from leaking.A feedback rewriter sanitizes the supervisor’s limited high-level status signal before delivery to the executor.
- Runtime implementation: Each task runs in a fresh Docker environment where the executor accesses public instructions, inputs, tools, skills, and services, while hidden references remain inaccessible.Supported agent systems include OpenClaw, Nanobot, and EDICT.
4 Experiments
The experiments isolate model- and framework-level effects using shared tasks, hidden rubrics, and scoring, while constraining interaction and execution time. Results show that real-world task completion remains highly difficult and depends strongly on framework architecture, with trade-offs between success, coordination, and token efficiency.
- Experimental design: Ten executor models are compared under OpenClaw, while three representative models are evaluated across OpenClaw, Nanobot, and EDICT using identical tasks, hidden rubrics, and scoring.This design separates inherent model capabilities from agent-framework effects.
- Experimental design: Each task permits up to two follow-up user interactions and is bounded by global and per-turn timeouts to support interaction without unbounded execution.Standard tasks use a 30-minute global timeout; the supplied passage does not state the per-turn limit.
- Model comparison: Leading closed-source models achieve the highest overall pass rates, but absolute success rates remain below 50%, exposing the difficulty of real-world execution.The results indicate a substantial gap between performance in sandboxed environments and real-world task completion.
- Framework comparison: Framework architecture strongly affects task success, with performance differences widening as base-model capabilities increase.The experiments show that framework design can either amplify or bottleneck intrinsic model reasoning.
- Framework comparison: OpenClaw consistently achieves the highest pass rate because its centralized trajectory preserves task constraints, tool evidence, and closed-loop feedback with minimal information loss.Its cohesive context management helps strong models convert partial progress into complete, verifiable successes.
- Efficiency and coordination: Nanobot uses 0.57M versus 1.15M average input tokens for GPT-5.4, but its simplified context management trades token efficiency for weaker evidence chains and incomplete trajectories.EDICT instead shows relatively high average scores but lower pass rates, reflecting coordination friction in multi-agent orchestration.
5 Conclusion and Limitations
UniClawBench evaluates proactive agents through five capability categories across 400 bilingual real-world tasks and a three-role closed-loop framework. Results indicate weaknesses in long-horizon memory and cross-platform coordination, while framework architectures can bottleneck model capabilities.
- Contributions: UniClawBench decomposes proactive-agent evaluation into five foundational capability categories across 400 bilingual real-world tasks.The benchmark targets dynamic, real-world environments rather than static sandboxes.
- Contributions: The benchmark uses a three-role closed-loop framework with an executor agent, hidden supervisor agent, and user simulator agent.This design aims to capture multi-turn human-agent collaboration without leaking grading criteria.
- Findings and limitations: Current agents struggle with long-horizon memory and cross-platform coordination, while framework architectures can bottleneck intrinsic model capabilities.These findings come from the benchmark’s evaluations of proactive agents.
A Task Definition Format
UniClawBench tasks use a public YAML declaration for executor-facing configuration and a hidden Markdown rubric as the supervisor’s canonical scoring contract. The format specifies task identity, execution lifecycle, workspace injections, closed-loop interaction, and rubric organization.
- A Task Definition Format: Each task pairs a YAML declaration fixing the public contract with a hidden eval_rule.md defining the rubric.The executor sees only the rendered task field, while the supervisor sees both files and treats the evaluation rule as canonical.
- A.1 Task YAML Template: The YAML template records task_id, category, executor bindings, model, request, references, path, startup commands, and runtime controls.Shown fields include task_id, category, agent_sys, agent_id, model, task, references, path, start, oneshot, pre_exec_parallel_safe, max_user_followups, user_simulator, and supervisor.
- A.2 Field Semantics: task_id is suite-unique and supplies the file stem, while category selects one of the five task dimensions.agent_sys, agent_id, and model provide default executor bindings that runtime sweeps can override.
- A.2 Field Semantics: timeout_seconds limits one executor cycle, max_total_seconds limits whole-attempt wall time, and success_threshold sets the minimum supervisor score for Pass.The whole-attempt wall-clock bound excludes supervisor and user-simulator turns; codex.max_user_followups separately bounds cycle count.
- A.2 Field Semantics: The task field is the executor’s literal natural-language request, while task_snapshot optionally switches execution from live APIs to snapshot data.A runtime preamble adds workspace paths and installed skills.
- A.2 Field Semantics: Task injections define workspace sources, declared skills, bootstrap services, and pre_exec scripts that populate session-specific fixture state.Skill-usage evaluation typically requires transcript evidence that the declared skill was consulted; services start before the first executor turn.
- A.2 Field Semantics: The codex block configures closed-loop interaction through follow-up limits, user-simulator policy overrides, and task-specific supervisor-instruction addenda.These settings support non-standard interaction styles such as terse users or persona constraints.
- A.3 Eval Rule (eval_rule.md) Format: The hidden rubric follows a fixed nine-section schema: §§1–4 establish the contract, §§5–6 define scoring, and §§7–9 govern lifecycle policy and asset bookkeeping.Supervisor scoring uses lines in §§5–6, while other sections support understanding and flexible judgment; supervisor-only assets remain hidden, and §9 distinguishes offline from live content.
B Task Examples · B.1 Skill Usage
The task examples present two runtime prompts per task dimension, with B.1 Skill Usage requiring agents to discover declared skills, follow SKILL.md entry points, and apply appropriate tools to produce structured outputs.
- B Task Examples: Each task dimension includes two literal natural-language prompts shown as the executor’s runtime task fields, accompanied by the surrounding YAML schema.The examples are representative tasks from each suite.
- B.1 Skill Usage: Skill Usage tasks ship declared skills and require agents to discover the relevant skill, consult its SKILL.md entry point, and apply the correct tool.This tests actual skill use rather than merely describing a procedure.
- B.1 Skill Usage: The organization-chart task requires extracting governance relationships from an Alphabet proxy-statement excerpt and converting them into a Mermaid visualization.The requested hierarchy includes board, executive-roster, committee, chair, advisor, and observer relationships.
- B.1 Skill Usage: The visualization must be saved as org.mmd and rendered as org.svg, using solid supported hierarchy arrows, dotted membership or advisory arrows, committee subgraphs, and visual chair distinctions.The task specifies graph TD or flowchart TD syntax and requires explicit handling of unsupported relationships.
- B.1 Skill Usage: The same governance task also requires governance_matrix.csv with person, role, group, committee, committee role, edge type, and source quote columns.The CSV lets the chief of staff review extracted facts without reading the Mermaid source.
- B.1 Skill Usage: The GitHub activity task covers 16 days from 2026-04-10 through 2026-04-25, using the workspace GitHub skill, live REST API access, and GITHUB_TOKEN authentication instead of local snapshots.Pseudo-user and planned-date notes in activity content override automation-account attribution and rebuild timestamps.
- B.1 Skill Usage: The digest must be grouped by contributor, include categorized activity and follow-up flags, highlight the 5 newest qualifying events, and provide a complete activity_index.csv.The index covers every merged PR, closed-unmerged PR, closed issue, and commit with specified activity, attribution, date, and status fields.
B.2 Exploration … D End-to-End Case Study
The benchmark’s tasks span exploration, long-context reasoning, multimodal understanding, and cross-platform coordination, with role prompts defining a closed-loop supervisor and user-simulation process. Together, the supplied sections emphasize auditable evidence, sustained state tracking, multimodal outputs, and synchronized actions across real tools.
- B.2 Exploration: Exploration tasks require investigating noisy inputs, rejecting incorrect alternatives, documenting negative evidence, and validating results through real scenarios.The examples include auditing and refactoring a legacy zsh configuration while preserving behavior, and conducting an offline license-provenance audit under explicit policy rules.
- B.3 Long-Context Reasoning: Long-context tasks aggregate heterogeneous evidence or track state across extended action sequences to produce globally consistent, auditable syntheses.Examples include setting up OpenClaw in Slack with tutorial and screenshot artifacts, and researching a constrained Santa Monica itinerary and budget without irreversible bookings.
- B.4 Multimodal Understanding: Multimodal tasks require extracting and generating information from real images, videos, or audio while combining perception with tool use.The examples require faithfully recreating a paper figure after visual comparison and revision, and collecting artwork screenshots with image and metadata evidence.
- B.5 Cross-Platform: Cross-platform tasks synchronize state and evidence across heterogeneous applications, interfaces, files, and calendars rather than producing only a textual answer.Examples combine PDF reading, Zotero metadata and attachments, BibTeX export, Obsidian notes, real web verification, and creation of a calendar event with evidence files.
- C Role Prompts: The role system uses a session wrapper, supervisor prompt, and user simulator prompt, with isolated workspaces and per-attempt placeholders.The supervisor and user simulator mirror the executor’s visible artifacts, while only the supervisor also receives hidden judging references.
- C.2 Supervisor Prompt: The hidden supervisor derives its judging standard from hidden references and applies it to visible transcripts, tool actions, and saved artifacts.Its instructions require strict rubric adherence, prohibit invented deductions, and prioritize verifiable checkpoints and supported end results.
- C.3 User Simulator Prompt: When the supervisor returns continue, the user simulator generates the next turn in the original end-user’s voice, language, and register rather than the agent’s internal role-play style.The simulator explicitly speaks in first person as the human and avoids addressing internal sub-agents or mimicking their stylized workflow language.
D.1 Task File · D.2 Eval Rule
D.1 defines a five-record Library of Congress image-rights selection task requiring official-source validation, while D.2 evaluates correctness, exploration, evidence, and adherence to task-specific score caps. High scores require results that are both correct and auditable.
- D.1 Task File: The executor must select five Library of Congress records with clear item-level rights statements and genuinely accessible image resources, rather than simply choosing the first search results.Thumbnails alone do not establish full-image availability.
- D.1 Task File: The task specifies the exploration category, Claude Opus 4.6 executor, 1200-second timeout, 1800-second maximum, and 0.9 success threshold.Its task identifier is task_009_loc_rights_images.
- D.1 Task File: Each record requires validation through the item page, official JSON/API or resource fields, an IIIF manifest or equivalent, and item-level rights evidence.The task also requires title, date, identifier, URLs, rights evidence, and a deduplication key for each final record.
- D.1 Task File: The task explicitly guards against unavailable full images, collection-level rights substituted for item-level rights, ambiguous advisories, non-image OCR items, and duplicate images.These traps determine whether candidate records are genuinely eligible.
- D.2 Eval Rule: The evaluation rubric assigns 30% to task completion and correctness, 20% to exploration breadth and candidate handling, and 25% to source and artifact evidence.Correctness alone is insufficient when exploration or auditable outputs are weak.
- D.2 Eval Rule: Results are capped at 0.40 when they are simply the first five search hits, 0.55 when thumbnails are treated as full-image evidence, and 0.70 when rights statements are unclear or unofficially tied.These caps directly enforce the task’s selection and evidence requirements.
- D.2 Eval Rule: Additional caps penalize missing exploration evidence, untied source evidence, absent negative evidence, and incomplete artifacts, with a 0.75 cap for lacking negative evidence on major traps.Scores near 1.0 are reserved for correct, auditable submissions whose artifacts directly support the final conclusion.
D.3 Cycle 1 – Executor Trace
Cycle 1 shows the executor probing the LOC site, recovering from a Cloudflare challenge by switching request strategies, and producing the required artifacts through subsequent processing.
- D.3 Cycle 1 – Executor Trace: The executor probes the LOC site during the first eight assistant turns of cycle 1.Table 3 summarizes these initial turns.
- D.3 Cycle 1 – Executor Trace: The model recovers from the Cloudflare block by switching to direct API and HEAD requests without abandoning the probe.This strategy change allows execution to continue despite the challenge.
- D.3 Cycle 1 – Executor Trace: The executor writes all five required artifacts after reading candidate JSON, normalizing titles and dates, and deduplicating by image stem.The outputs include loc_rights_images.json, loc_image_exclusions.json, methodology notes, the Python script, and the run log.
D.4 Cycle 1 – Supervisor Verdict · D.5 Cycle 1 – User Simulator Follow-up · D.6 Cycle 2 – Executor Trace
Cycle 1’s supervisor found complete, auditable artifacts but judged the final selection noncompliant and internally inconsistent, prompting a constrained user follow-up. In Cycle 2, the executor rebuilt the validation pipeline, uncovered two additional errors, and regenerated all five output files.
- D.4 Cycle 1 – Supervisor Verdict: The supervisor reviewed the executor’s full visible trajectory against the hidden evaluation rule and ground truth, then issued one structured decision.The visible evidence included official LOC API exploration, direct image and IIIF checks, negative trap evidence, and methodology notes.
- D.4 Cycle 1 – Supervisor Verdict: The final five selections violated the requirement not to use only the first five search results, because the executor fetched only page sp=1 and accepted five picks sequentially from 25 results.Broader exclusion notes were added afterward and were not connected to the actual selection process.
- D.4 Cycle 1 – Supervisor Verdict: The exclusions file claimed 7 exclusions but enumerated 1, misclassified item 2006686829, and asserted duplicate logic for two Trail Creek Ranch records without evidence.The supervisor considered these issues recoverable through item-level JSON revalidation and consistency checks.
- D.5 Cycle 1 – User Simulator Follow-up: Because the verdict was continue, the user simulator received only verdict, attempt state, recoverability, and score, alongside the executor’s visible trajectory and saved artifacts.The supervisor’s rationale, missing_artifacts, guidance_tags, and hidden evaluation rule were withheld from the simulator.
- D.5 Cycle 1 – User Simulator Follow-up: The simulator instructed the executor to revalidate official item JSON and pages, replace first-page inferences, correct the three documented inconsistencies, and support all five records and exclusions.The resulting message was deterministically post-rewritten before being sent as the next user turn.
- D.6 Cycle 2 – Executor Trace: The Cycle 2 executor used a tightened pipeline and rebuilt the results rather than patching the saved files in place.Table 4 summarizes ten assistant turns in this cycle.
- D.6 Cycle 2 – Executor Trace: In Cycle 2, the executor re-fetched official item JSON for every candidate, discovered two additional substantive errors, and regenerated all five output files from the corrected pipeline.The additional issues concerned the Koussevitzky rights field and the Feraille-o item type.
D.7 Cycle 2 – Supervisor Verdict
The supervisor’s second decision raises the score from 0.65 to 0.96, crossing the rubric’s pass band. The attempt completes after exactly two cycles with all required artifacts present and no missing artifacts.
- D.7 Cycle 2 – Supervisor Verdict: 0.96 finalScore passes the rubric, completing the attempt after exactly two cycles without invoking the user simulator again.The verdict is pass, the attempt state is complete_and_passed, recoverable is false, and confidence is high.
- D.7 Cycle 2 – Supervisor Verdict: The final selection file contains 5 LOC records with required metadata, while run logs provide official item JSON URLs, rights evidence, and image verification.The Highsmith item also includes IIIF evidence, and all required outputs are present in visible/result/.
- D.7 Cycle 2 – Supervisor Verdict: The exclusions file lists 7 items with concrete trap-matching reasons, correctly classifying item 2006686829 and documenting the Trail-Creek-Ranch duplicate through image-path comparison.The listed trap categories include thumbnail-only, collection-level rights mismatch, OCR-only text item, and duplicate image stem.
- D.7 Cycle 2 – Supervisor Verdict: The submission demonstrates broad exploration, internal consistency, and avoidance of every named trap; inferred commentary in loc_images_notes.json is the only deduction.The exploration covers multiple result pages and rights advisories.
- D.7 Cycle 2 – Supervisor Verdict: No artifacts are missing, and no guidance tags are recorded.The result reports missing_artifacts: [] and guidance_tags: [].
D.8 What This Trace Demonstrates · E Societal Impacts
The trace demonstrates a closed-loop supervisor that preserves progress on recoverable failures while structurally limiting information passed to the user simulator. The work may improve real-world agent evaluation and safety, but also carries potential misuse risks.
- D.8 What This Trace Demonstrates: The closed-loop design exposes three properties in a single attempt.
- D.8 What This Trace Demonstrates: The supervisor catches recoverable failures instead of rejecting them outright.
- D.8 What This Trace Demonstrates: A continue decision at score = 0.65 preserves executor progress while flagging remediable gaps.Cycle 1 had the correct pipeline and data, but insufficient selection rigor and exclusion-file consistency.
- D.8 What This Trace Demonstrates: The information firewall is structural rather than stylistic.The supervisor produces a seven-field decision, while only four fields cross to the user simulator.
- E Societal Impacts: The work can enable more realistic and rigorous evaluation of AI agents in real-world environments.This could improve agent reliability, transparency, and safety before deployment.
- E Societal Impacts: Identifying capability gaps in long-context and multimodal abilities can guide development of more robust systems.The passage names research assistance, software engineering, and information processing as potential applications.
- E Societal Impacts: The framework also carries potential risks because it could be used in harmful ways.