Source-linked AI summary
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Chenxin Li, Zhengyang Tang, Mingxin Huang, Yunlong Lin, Shijue Huang, Shengyuan Liu, Bowen Ye, Rang Li, Lei Li, Benyou Wang, Yixuan Yuan
TL;DR
Many agent benchmarks use static task mixtures and final-text grading, limiting evaluation of evolving workflow demand and observable execution. Claw-Eval-Live addresses this with refreshable, time-stamped releases built from public workflow signals and action-grounded grading. The current results show reliable workflow automation remains unsolved: the best model passes 66.7% of tasks, while HR, management, and multi-system workflows remain difficult.
Problem
Many agent benchmarks freeze task mixtures and emphasize final responses, limiting evidence about evolving workflow demand and whether agents changed the intended state or artifact.
Method
Claw-Eval-Live builds time-stamped, reproducible releases from public workflow signals and evaluates controlled service and workspace tasks using action-grounded graders.
Results
66.7% is the leading model’s pass rate; service-backed workflows remain harder than local workspace repair, with HR, management, and multi-system coordination persistent bottlenecks.
Takeaways & Limitations
Workflow-agent evaluation should be grounded in both fresh external demand and verifiable agent action, while each release is interpreted as a time-stamped snapshot.
Takeaways & Limitations
Using GPT-5.4 as a judge can introduce judge-model bias, and the judge is not an independent human adjudicator.
Abstract
from arXiv · showhide
LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow agents that separates a refreshable signal layer, updated across releases from public workflow-demand signals, from a reproducible, time-stamped release snapshot. Each release is constructed from public workflow-demand signals, with ClawHub Top-500 skills used in the current release, and materialized as controlled tasks with fixed fixtures, services, workspaces, and graders. For grading, Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts, using deterministic checks when evidence is sufficient and structured LLM judging only for semantic dimensions. The release contains 105 tasks spanning controlled business services and local workspace repair, and evaluates 13 frontier models under a shared public pass rule. Experiments reveal that reliable workflow automation remains far from solved: the leading model passes only 66.7% of tasks and no model reaches 70%. Failures are structured by task family and execution surface, with HR, management, and multi-system business workflows as persistent bottlenecks and local workspace repair comparatively easier but unsaturated. Leaderboard rank alone is insufficient because models with similar pass rates can diverge in overall completion, and task-level discrimination concentrates in a middle band of tasks. Claw-Eval-Live suggests that workflow-agent evaluation should be grounded twice, in fresh external demand and in verifiable agent action.
1 Introduction
As agents move toward multi-step workflows, benchmarks must track changing real-world demand and verify observable execution rather than rely on final-text plausibility. Claw-Eval-Live addresses both needs through time-stamped, rerunnable releases covering service-backed workflows and workspace repair.
- Modern agents perform workflows requiring cross-system retrieval, state-changing writes, artifact edits, and auditable evidence of actions.Examples include reconciling transactions, preparing multi-system meeting briefs, coordinating approvals, and repairing development workspaces.
- Static benchmark releases can become stale as tool stacks evolve, enterprise bottlenecks shift, and real-world workflow mixes change.Prior suites often freeze a particular task mixture at publication time.
- Final-response scoring cannot reliably establish whether agents retrieved correct information, changed the right state, or repaired the intended artifact.A polished memo may not show that the underlying records, entities, objects, or broken artifacts were handled correctly.
- Claw-Eval-Live calibrates evaluation twice: task distributions follow evolving workflow demand, while task scores rely on observable execution evidence.Each public release is treated as a time-stamped benchmark snapshot produced by a rerunnable signal-to-task pipeline.
- 105 tasks, 13 public models, and 18 controlled services plus sandboxed workspaces comprise the current benchmark snapshot.The release evaluates service-backed workflows and workspace repair in a single benchmark.
2 Related Work
Prior agent benchmarks span tool, web, desktop, professional, code, and workspace tasks, while Claw-Eval-Live builds on trajectory-aware and hybrid evaluation. Its primary distinction is constructing time-stamped releases from live public workflow signals rather than fixed curated pools.
- Agent benchmarks: Agent benchmarks cover heterogeneous capabilities across tool use, web interaction, desktop environments, and professional tasks.General suites test broad capabilities, while browser- and desktop-centered benchmarks make interaction central.
- Code and workspace agent benchmarks: API and code benchmarks progress from tool manipulation and function-level generation toward repository and command-line execution.These benchmarks provide the closest precedent for Claw-Eval-Live’s workspace-repair tasks.
- Evaluation methodology: Output-only grading can overlook plausible artifacts that do not faithfully execute the intended workflow.Claw-Eval is the closest methodological relative because it emphasizes trajectory-aware evidence, hybrid grading, and multidimensional evaluation.
- Positioning: Claw-Eval-Live constructs time-stamped benchmark snapshots from live public workflow signals, distinguishing it from benchmarks built from fixed curated pools.Its release construction calibrates the task mix to external workflow demand before materializing the snapshot.
3 Benchmark Construction
Claw-Eval-Live separates refreshable public workflow-demand signals from fixed, time-stamped release snapshots to balance external alignment with reproducibility. Its construction turns signal-derived workflow patterns into executable tasks and selects a discrimination-aware public subset with fixed fixtures, services, and graders.
- Signal layer: ClawHub Top-500 skills provide an externally inspectable prior over contemporary workflow demand, not ground-truth measurements of deployment frequency, economic value, or task difficulty.The snapshot is ranked by downloads and popularity at release construction.
- Release layer: Published releases keep tasks, fixtures, and graders fixed, while future releases refresh the upstream signal snapshot and task mixture.This preserves stable model comparison while allowing the benchmark to absorb shifts in the workflow ecosystem.
- Construction pipeline: The five-stage pipeline collects signals, clusters them into workflow patterns, weights families, expands seeds into executable candidates, and selects the public subset.Pattern clustering groups signals by user goal, artifact or state, and execution surface; seed implementation adds prompts, tool definitions, fixtures, and task-specific graders.
- Public release selection: 105 executable tasks comprise the current public release, selected to balance release size, family coverage, and leaderboard resolution.Brittle, ambiguous, or non-discriminative candidates are excluded after screening.
- Executable task design: Each released task is a complete executable evaluation unit containing a YAML definition, fixtures, tool schemas, and a task-specific grader.Tasks span controlled service workflows and local workspace repair, and are graded from observable execution evidence rather than prompts alone.
4 Evaluation Setup and Methodology
Claw-Eval-Live evaluates 13 public models on a fixed 105-task release under a unified, task-specific protocol with controlled services, sandboxed workspaces, and recorded execution evidence. Grading prioritizes deterministic checks of observable actions and post-run state, adding rubric-bound LLM judging only for semantic dimensions that exact checks cannot capture.
- Evaluation protocol: The release contains 105 tasks and evaluates 13 public models under shared task limits, prompts, tool schemas, fixed fixtures, and no model-specific prompt tuning.The default workflow budget is 24 turns and 300 seconds, with larger budgets allowed for some workspace-repair tasks.
- Evaluation protocol: Each run loads fixtures and services, executes the agent in controlled environments, records traces and artifacts, and produces a task-specific score in [0, 1].Recorded evidence includes tool calls, responses, tokens, wall time, and environment-side artifacts.
- Grading methodology: Grading starts from deterministic evidence and uses structured LLM judging only when explicit checks do not fully cover semantic dimensions such as completeness or report coherence.Judge inputs include the task prompt, agent trace, observable-action summaries, and task-specific rubric; GPT-5.4 is used as the judge model, creating potential judge-model bias.
- Grading methodology: Workspace-repair grading is fully deterministic: verification scripts re-check file contents, service health, configuration integrity, or command outputs after execution.For SHELL and W-family tasks, success is determined by post-run workspace state rather than self-report or final textual claims.
- Metrics and ranking: The public metrics are Pass Rate and Overall Completion Score, with τ = 0.80 as the pass threshold; models rank first by Pass Rate and then by Overall Completion Score.Pass Rate captures tasks meeting the public pass rule, while Overall Completion Score averages task scores across all 105 tasks on a 0–100 scale.
5 Main Results
Claw-Eval-Live shows that reliable workflow automation remains unsolved: no model reaches 70% pass rate, and business-facing, multi-system execution is substantially harder than local workspace repair. Overall completion, action-grounded evidence, task-family performance, and resource use reveal differences that aggregate leaderboard rank alone obscures.
- Leaderboard: 66.7% pass rate is the ceiling: Claude Opus 4.6 leads, with 83.6 overall completion, while GPT-5.4 reaches 63.8% and 81.7.No publicly reported model crosses the 70% pass-rate mark.
- Leaderboard: 22.9 percentage points separate the top and bottom models in pass rate, while models tied at 53.3% differ in overall completion.MiMo V2 Pro, Kimi K2.5, and Gemini 3.1 Pro demonstrate why pass rate alone is insufficient.
- Task families: Development / Terminal is near ceiling, whereas HR / People remains extremely difficult and Productivity has the widest spread across models.Development / Terminal reaches 100% for three models, HR / People stays below 22.2%, and Productivity ranges from 88.0% to 48.0%.
- Execution surfaces: 72.2% is the minimum workspace-slice pass rate across models, while no model exceeds 59.8% on service-backed workflows.Claude Opus 4.6 leads service-backed workflows at 59.8%, followed by GPT-5.4 at 56.3% and GLM-5 at 55.2%.
- Action-grounded evaluation: Action-grounded grading penalizes fluent outputs when required tool calls, evidence, record linkage, writes, or post-run state integrity are missing.This produces a “can say” versus “can do” distinction, especially in HR, management, and workflow tasks.
- Discrimination and efficiency: 19 tasks are all-pass and 27 all-fail under the public rule, concentrating discrimination in a middle band and requiring pass rate to be read with overall completion.Strong discriminators include ecommerce_monthly_reconcile, first_response_time_audit, and multi_doc_merge.
6 Conclusion
Claw-Eval-Live is introduced as a live agent benchmark designed around workflow calibration and evidence alignment. Its current snapshot shows the importance of these goals and that current agents remain limited, with the best model passing only 66.7% of tasks.
- Design goals: Claw-Eval-Live is built around workflow calibration and evidence alignment.Workflow calibration keeps the released task mix close to workflows users currently want to automate.
- Design goals: Evidence alignment grounds scores in what the agent observably did rather than final-text plausibility alone.The benchmark evaluates observable agent behavior instead of relying only on whether the final response appears plausible.
- Current limitations: 66.7% is the best model’s pass rate in the current snapshot.The result illustrates that current agents still fall short on the benchmark’s workflow tasks.
B Full Score Matrix
The full Claw-Eval-Live score matrix covers 105 tasks and 13 models, with a representative subset presented in Table 7 and the complete matrix released as a CSV file.
- B Full Score Matrix: 105 × 13: The full task × model score matrix is available as benchmark/results/raw_matrix_v3.csv, while Table 7 presents a representative subset.The release provides the complete matrix for all benchmark tasks and evaluated models.
C Grader Architecture Details
The grader architecture combines a shared AbstractGrader interface with task-specific evidence logic. It evaluates API use, tool-call robustness, agent traces, audit-log actions, and assistant text through reusable mechanisms.
- Shared grading interface: All task graders inherit from AbstractGrader, establishing a shared base for grading behavior.The base class provides common grader functionality.
- Execution evidence: _tool_gate(dispatches) applies a multiplicative penalty based on whether the agent called the required APIs.This makes required API usage part of the grading signal.
- Execution evidence: compute_robustness(dispatches) measures the fraction of tool calls returning HTTP 2xx status, while helper methods format traces, summarize audit-log actions, and concatenate assistant text for keyword matching.These utilities expose execution outcomes and observable actions to deterministic or semantic checks.
- Task-specific grading: The registry dynamically imports each task’s grader.py at scoring time and locates its AbstractGrader subclass by reflection.This preserves a shared interface while permitting task-specific evidence logic.
D Mixed-Integer Linear Programming Solver Details
The public-subset selection problem is formulated as a tractable MILP with binary task-inclusion variables, linear objective and constraints, and a standard branch-and-bound solver. Its pilot ordering guides release construction without asserting a permanent global ranking.
- Problem formulation: 157 binary variables and 22 family-coverage constraints define the MILP, which is solved using a standard branch-and-bound solver.Binary variables represent task-inclusion indicators, while the objective and constraints remain linear.
- Computational tractability: O(K2 · |T |) pairwise-order terms appear in the objective, with K = 5 for the pilot ordering panel, making the optimization readily tractable.The tractability assessment applies at this problem scale with off-the-shelf MILP solvers.
- Pilot ordering: The pilot ordering was Claude Opus ≻GPT-5.4 ≻Claude Sonnet ≻GLM-5 ≻ Kimi K2.5, used as a release-construction device rather than a permanent global ranking.The ordering was derived from preliminary evaluation on the screened candidate pool.