Source-linked AI summary
ClawBench: Can AI Agents Complete Everyday Online Tasks?
Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, Zhengkang Guo, Yuanzhe Shen, Dingjie Song, Han Zhou, Tuney Zheng, Xian Wu, Hao Yu, Songcheng Cai, Yi Lu, Yunzhuo Hao, Minyi Lei, Liang Chen, Kai Zou, Huifeng Yin, Wendong Xu, Dongfu Jiang, Ping Nie, Jiaheng Liu, Wenhu Chen, Kelsey R. Allen
TL;DR
AI agents still lack reliable evidence of completing everyday, write-heavy workflows on real websites. The paper introduces CLAWBench, a live-web benchmark with safe final-request interception and trace-based judging, and finds low success across eight frontier models, with recurring failures beyond high-level reasoning.
Problem
Existing benchmarks do not jointly test everyday, write-heavy task completion on live production websites while preserving safe, traceable evaluation.
Method
CLAWBENCH evaluates 153 tasks across live platforms using final-request interception, five-layer trajectory recording, human references, and Agent-as-Judge verdicts.
Results
Current agents complete only a small portion of tasks; Claude Sonnet 4.6 achieves the highest success rate at 33.3% among eight evaluated frontier models.
Takeaways & Limitations
Progress toward reliable online assistants requires better planning and perception alongside behaviorally grounded interaction and robust handling of production-site constraints.
Takeaways & Limitations
Live websites trade rerun determinism for ecological validity, and their cost limits repeated trials, prompt variants, and ablations under a fixed budget.
Abstract
from arXiv · showhide
AI agents may be able to assist with emails and documents, but can they reliably complete everyday online workflows on real websites? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework comprising 153 everyday online tasks that people need to accomplish regularly in their lives and work, spanning 144 platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and interaction challenges of real-world web environments. An interception layer captures and blocks the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 8 frontier models show that both proprietary and open-source models complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%, which exposes gaps in current AI agents. Progress on ClawBench brings us closer to AI agents that can function as general-purpose assistants.
1 Introduction
CLAWBENCH evaluates whether agents can complete everyday, write-heavy workflows on live production websites while safely blocking irreversible actions. Across eight models, results remain low and reveal failures involving production constraints, interaction behavior, safety refusals, and incomplete transactions.
- CLAWBENCH contains 153 everyday tasks across 15 life categories and 144 live platforms, targeting write-heavy workflows largely avoided by existing benchmarks.
- Agents operate on production websites with dynamic interfaces, authentication flows, validation logic, and anti-bot defenses, while final-request interception blocks real-world side effects.
- 33.3% is Claude Sonnet 4.6’s highest success rate among eight evaluated frontier models, while GPT-5.4 reaches 6.5%.
- The evaluation records five-layer trajectories and compares agent runs with human references to produce traceable binary task-level verdicts.
- Trace analysis identifies anti-bot blocking, unnatural interaction patterns, safety refusals, and premature stopping before final transactional actions as recurring failure modes.
2 Benchmark
The benchmark is designed to test everyday state-changing web assistance across heterogeneous live platforms, using task-scoped safety controls and trace-based judging. Its evaluation combines human references, synchronized evidence, and binary outcome scoring.
- Task Design and Collection: CLAWBENCH focuses on delegated write-heavy tasks such as submitting information, reserving services, placing orders, and filling applications.
- Task Design and Collection: Human annotators formalize user-grounded tasks with instructions, starting URLs, required files or profile fields, expected final states, and terminal submission targets.
- Task Design and Collection: The benchmark contains 153 tasks across 144 live production websites, favoring breadth across heterogeneous interfaces, authentication flows, forms, validation logic, and anti-automation defenses.
- Safe Live-Web Execution: Final-request interception monitors outgoing traffic and blocks the annotated terminal request, while other website traffic proceeds normally.
- Safe Live-Web Execution: The safety envelope is task-scoped and excludes workflows whose side effects cannot be bounded by one annotated terminal request.
- Trace-Based Evaluation: Each run records video, screenshots, HTTP traffic, agent messages, and browser actions for comparison with human reference trajectories.
- Trace-Based Evaluation: Agent-as-Judge maps task instructions, human references, agent traces, and intercepted payloads to binary pass/fail verdicts with structured justifications.
- Trace-Based Evaluation: 84.97–93.46% raw agreement was reported between Agent-as-Judge and human verdicts on human-adjudicated subsets.
3 Experiments
CLAWBENCH evaluates eight frontier models on 153 live, write-heavy tasks using a shared browser harness and trace-based success judgments. Results remain low and uneven across domains, costs, and failure behaviors, with many failures stopping short of final commitment.
- Main Results: 33.3% is Claude Sonnet 4.6’s overall success rate, leading Qwen 3.5 at 26.1% and GLM-5 at 24.2%.The primary metric is the percentage of tasks receiving a binary Agent-as-Judge pass verdict.
- Main Results: 68 of 153 tasks (44.4%) are solved by no model, while no task is solved by all eight models.Only one task is solved by seven of the eight models, showing low task-level saturation.
- Main Results: Model strengths vary by domain: Sonnet 4.6 leads Daily, Academic, and Social, GLM-5 leads Work, and Haiku 4.5 leads Dev.Sonnet 4.6 ties Qwen 3.5 on Finance, while Qwen 3.5 ties Gemini 3 Flash on Travel.
- Main Results: GLM-5 is the most cost-efficient capable model at $0.64 per task and 24.2% SR, while Sonnet 4.6 reaches 33.3% SR at 12× that cost.Qwen 3.5 uses 2.52M tokens per task for 26.1% SR and averages 42 tool calls, making it another budget-oriented operating point.
- Trace-Level Diagnosis: Failed trajectories use more tool calls than successful ones: Sonnet 4.6 has medians of 120 versus 64, and Gemini 3 Flash has longer failed-run tails.The diagnosis treats this as unproductive interaction rather than insufficient exploration.
- Trace-Level Diagnosis: Six of eight models concentrate many failures at S5, where agents reach final confirmation but do not commit the state-changing request.Failures often contain substantial ordinary interaction but a smaller terminal-commitment slice, indicating last-mile hesitation.
- Trace-Level Diagnosis: Failure termination regimes differ: Gemini 3.1 Flash Lite is agent_idle on 131 runs, GLM-5 hits the time limit on 69, and GPT-5.4 agent_exited on 85.These counts motivate separating premature exit, persistence through site friction, and safe final-step commitment.
4 Conclusion
CLAWBENCH provides a benchmark of real-world everyday web tasks across live platforms and combines safe interception, trajectory recording, and Agent-as-Judge evaluation. Its eight-model experiments show that performance on existing web-agent benchmarks does not transfer to these tasks, while the benchmark is intended to support more reliable and safe assistance.
- Conclusion: CLAWBENCH covers 153 real-world tasks across 144 live platforms and uses live, write-heavy workflows to test web competence.The benchmark is designed as a more realistic testbed than static-page or sandbox-based evaluations.
- Conclusion: The framework combines final-request interception, five-layer trajectory recording, and Agent-as-Judge evaluation.These components support safe execution and trace-based assessment of agent behavior.
- Conclusion: Experiments on eight models show that strong performance on existing web-agent benchmarks does not transfer to CLAWBENCH.The reported finding highlights a gap between controlled benchmark success and real-world web competence.
- Conclusion: CLAWBENCH is intended to guide web agents toward reliably and safely completing everyday online tasks people want to delegate.The stated goal is dependable assistance in daily life.
Limitations
The benchmark’s live-web setting improves ecological validity but sacrifices rerun determinism, while its scope and scoring impose important boundaries. Results are also conditioned on the shared agent stack and a task-scoped safety envelope.
- Scope and setting: Live production websites introduce layout changes, A/B tests, regional variation, account-state differences, and anti-automation defenses, so exact rerun reproduction is not guaranteed.Recorded trajectories and intercepted payloads make outcomes auditable despite site-state drift.
- Scoring: Binary Agent-as-Judge scoring is conservative for write-heavy tasks but abstracts away intermediate behavior such as partial form completion or stopping at verification.Released traces and structured justifications support complementary behavioral analyses.
- Coverage: The 153-task suite emphasizes cross-site generalization over dense per-domain coverage and underrepresents mobile-only, non-English, and accessibility-dependent workflows.Future versions can extend these areas under the same protocol.
B.2 Task-Level and Category Breakdowns
The extended breakdowns show substantial variation across categories and tasks, with most tasks remaining unsolved across the model panel. Effort diagnostics further distinguish early exits from prolonged, budget-consuming failures.
- Category breakdowns: Category leadership varies across models: Sonnet 4.6 leads four of eight categories, GLM-5 leads Work, Gemini 3 Flash and Qwen 3.5 share Travel, and Haiku 4.5 leads Dev.Figure 9 represents these results as per-column ranks over overall success and eight high-level categories.
- Task-level breakdowns: 68 of 153 tasks (44.4%) are solved by no model, only one task is passed by seven models, and no task is passed by all eight.The mean task solve rate is 0.176 and the median is 0.125.
- Effort diagnostics: Table 5 places all eight agents on a common behavioral scale using duration, tool-call, message, request, timeout, and pass/fail duration measures.The table complements Figure 6’s close-up of two representative models.
- Effort diagnostics: Pass-versus-fail distributions separate early exits with almost no actions or requests from budget-burning trajectories that accumulate duration near the 1,800-second wall.These regimes are obscured when only medians are reported.
B.4 Failure Case Evidence
Trace inspection makes the aggregate failures concrete: even strong models often stop before commitment, while anti-bot barriers, verification gates, platform drift, and inefficient execution create distinct failure regimes.
- Last-mile non-commitment: Seven sampled Sonnet 4.6 failures reached the penultimate step but never issued the terminal Submit, Confirm, or Place-Order action.The strongest model can complete navigation and preparation yet fail at last-mile commitment.
- Structural blockers: Among 40 universally hard tasks, recurring blockers include anti-bot walls, final-step verification gates, and substitution of a nearby platform.These patterns span services including Instacart, Zillow, Indeed, Crunchyroll, Asana, Uber-Eats, and others.
- Weak-model regimes: Gemini 3.1 Flash Lite’s median failure uses 20 actions over 454 seconds and often ends with agent_idle amid CAPTCHA or Cloudflare barriers.Low success can therefore reflect inefficient interaction and blocked navigation, not only reasoning errors.
- Trace-level insights: Representative cases also include false completion from malformed submissions, efficient success versus thrashing on one task, and open-source completion of long-horizon forms.These cases expose behavioral distinctions hidden by aggregate metrics.
Failure Case (121 actions, 27 turns, 173 s)
Two production-web failures illustrate why reaching a workflow’s endpoint is insufficient: agents may abandon the required platform after an anti-bot wall or appear complete while issuing an invalid commit.
- Last-mile non-commitment: The Soko Glam run added both products and reached checkout, but ended while entering the address without issuing the required terminal submission.The human-alignment judge therefore marked it a failure despite correct preparatory steps.
- Anti-bot wall and off-platform drift: Anti-bot infrastructure blocked Zillow access, and the agent abandoned the instructed platform for viewit.ca without submitting a rental application.The platform switch violated the task constraint as well as failing to complete the goal.
- False completion: A committing request can be invalid even when an agent reaches the correct platform, fills the cart, and believes the task is complete.A missing required field means no order is placed, motivating verification beyond apparent completion.
Failure Case (161 actions, 130 turns, 768 s)
The agent appeared to complete the order, but the terminal request was malformed and the order was not committed. This exposes why task success must be verified against server-facing evidence rather than self-reported completion.
- The terminal dropoffOptions request lacked the required cartId field, so no order was committed.The validator returned an error identifying cartId as required.
- The judge found that the agent added an item but did not reach the same final submission point as the reference.
- The agent’s claim that both items were correct diverged from the malformed request and unchanged server state.
- Self-reported completion is therefore not a reliable proxy for task success.Verification must use the intercepted committing request and a human reference.
C.4 Case 4: Execution Regimes on a Shared Task (Doodle)
Three models understood the Doodle task, but their outcomes diverged because of how they executed through the difficult time-selection interface. Qwen completed the workflow linearly, whereas GPT-5.4 and Gemini 3 Flash stalled through workaround escalation or looping.
- Qwen 3.5: Qwen 3.5 completed the workflow by switching to Month view, entering four dates and times linearly, and self-correcting an inefficient path.
- GPT-5.4: GPT-5.4 stalled at the time grid after escalating to JavaScript, React-fiber introspection, and roughly 12 unsuccessful hacks.
- Gemini 3 Flash: Gemini 3 Flash produced a malformed action and then entered a ref-mutation loop that repeated for about 108 turns until the time limit.
- Execution regimes: All three models planned the Doodle task correctly at the outset, making execution discipline—not goal knowledge—the separating factor.
- Insurify comparison: Qwen 3.5 also completed the long Insurify form by advancing one screen per answer and recovering from two dependent-field snags.
E Implementation Details
The implementation combines isolated browser-agent runs with an evidence-based Agent-as-Judge protocol and a calibrated evaluation prompt. Interception allows agents to attempt consequential workflows while preventing irreversible final actions.
- Each task run is evaluated as PASS or FAIL according to whether the agent completed the task, with explicit handling for verification, checkout, CAPTCHA, and interception cases.
- The interceptor blocks dangerous final actions, while correct prior inputs can still receive PASS when interception prevents completion.
- Runs use an isolated container with a clean Chromium profile, shared user data, and a 1,800 s task budget.
- Agent-as-Judge calibrates its evaluation prompt against held-out trajectories with human verdicts until agreement reaches the specified threshold.
- The browser harness exposes seven core actions and returns validation errors with the current page snapshot for malformed inputs.
F Human–Agent Interaction Dynamics
Agents and humans interact with websites in sharply different ways. Agents exhibit synthetic, implausibly fast, and coarse interaction patterns, indicating that live-site reliability involves interaction behavior in addition to planning.
- Agents generate implausibly fast typing, lack continuous mouse trajectories, and use coarse scrolling compared with humans.
- These interaction differences indicate that live-site failures are not only a planning problem.The paper identifies behaviorally grounded interaction models as an additional need.
G Reproducibility and Ethical Considerations
CLAWBENCH limits risk through intercepted terminal requests, but live-site interaction still requires careful use and leaves important scope boundaries. Its artifacts also raise trace-privacy and deployment constraints.
- Potential Risks: The benchmark excludes workflows whose side effects cannot be controlled through an annotated terminal request.Final-request interception mitigates risk but does not make arbitrary browsing side-effect-free.
- Responsible Use: Released artifacts are not intended for deploying autonomous agents on third-party websites or bypassing access controls, anti-bot systems, or terms of service.Use is governed by accompanying licenses and documentation.
- Data and Privacy: The benchmark uses synthetic profiles and benchmark-specific credentials, while released traces may contain screenshots, interaction logs, HTTP metadata, and generated agent text.Private credentials and secrets should be excluded and retained trace structures documented.
- Consent and Website Interaction: Researchers should respect website policies, avoid high-volume automation, and avoid workflows creating real-world obligations or uncontrolled side effects.Agents still load pages and interact with live services before the terminal action is blocked.