Source-linked AI summary
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, Tong Yang
TL;DR
Existing agent benchmarks provide limited evidence about trajectories, safety, robustness, and cross-modal performance. Claw-Eval addresses these gaps with 300 human-verified tasks, independent evidence channels, multidimensional scoring, and multi-trial evaluation. Across 14 models, it finds that trajectory-opaque judging misses important failures, capability can diverge from consistency, and rankings vary across task groups and modalities.
Problem
Existing agent benchmarks have trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms.
Method
Claw-Eval evaluates 300 human-verified tasks across 9 categories using three evidence channels, Completion-Safety-Robustness scoring, and multi-trial metrics.
Results
Agent capability is strongly multidimensional: rankings vary across task groups and metrics, while multimodal Pass^3 reaches only 25.7% at its highest.
Takeaways & Limitations
Trustworthy evaluation requires trajectory-level evidence, multidimensional scoring, and broad task coverage to capture autonomous behavior and deployment-relevant reliability.
Takeaways & Limitations
Robustness evaluation currently focuses on infrastructure-level transient failures, leaving semantic-level and adversarial perturbations for future work.
Abstract
from arXiv · showhide
Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.
1 Introduction
Claw-Eval addresses trajectory-opacity, safety and robustness, and coverage gaps in agent benchmarks through unified auditing, multidimensional scoring, and heterogeneous tasks. Its experiments show that evaluation validity and reliable deployment require measuring more than final outputs or average capability.
- Existing benchmarks often grade only outputs, making faithful execution difficult to distinguish from fabricated or shortcut-based behavior.
- Claw-Eval records full trajectories through execution traces, service-side audit logs, and environment snapshots, while scoring Completion, Safety, and Robustness together.
- 300 human-verified tasks span 9 categories and combine general workflows, multimodal interaction, and multi-turn professional dialogue under one evaluation framework.
- 44% of safety violations and 13% of robustness failures were missed by vanilla LLM judging but detected by the hybrid grading pipeline.
- Pass@3 remains relatively stable under error injection while Pass^3 drops by up to 24 percentage points, separating capability from consistency.
- Across models, trustworthy evaluation requires trajectory-level evidence, multidimensional scoring, and broad task coverage working together.
2 Related Work
Prior agent benchmarks cover many domains and interaction modalities, but existing evaluation practices do not combine broad coverage with auditable, safety-aware, and perturbation-aware grading. Claw-Eval is positioned against this fragmented landscape.
- Existing benchmarks span tool use, coding, web and GUI interaction, multi-turn dialogue, and heterogeneous multi-domain environments.
- Table 1 compares benchmarks across multimodal, multi-turn, auditable, safety, perturbation, and sandboxed evaluation dimensions.
- Some real-world task benchmarks couple evaluation to the full scaffold stack, preventing attribution of performance to the model itself.
- Output-only grading can miss fabricated intermediate steps, while LLM judges scale to open-ended tasks but lack auditability.
- Prior work addresses individual gaps through checkpoints, consistency metrics, and trace-based safety benchmarks, but not their unified integration.
3 Claw-Eval
Claw-Eval evaluates autonomous agents through an auditable, domain-agnostic pipeline covering heterogeneous tasks and multidimensional outcomes. It combines independent evidence collection, fine-grained rubrics, and multi-trial metrics to assess completion, safety, robustness, capability, and consistency.
- 3.1 Auditable Execution Pipeline: Each run passes through Setup, Execution, and Judge phases, with evaluation resources isolated from the agent until judging.
- 3.1 Auditable Execution Pipeline: Three independent channels—execution traces, service-side audit logs, and environment snapshots—capture actions, external interactions, and final artifacts.
- 3.2 Cross-Modal Task Suite: 300 human-curated tasks span 9 categories across General, Multimodal, and Multi-turn Dialogue groups under a shared lifecycle and schema.
- 3.2 Cross-Modal Task Suite: General tasks test practical workflows, Multimodal tasks test perception and generation, and Multi-turn Dialogue tasks test clarification and information gathering.
- 3.3 Scoring Protocol: Completion, Safety, and Robustness measure objective fulfillment, policy compliance, and recovery from transient environmental failures.
- 3.3 Scoring Protocol: Safety acts as a multiplicative gate, while Robustness uses controlled error injection to measure recovery from encountered failure types.
- 3.3 Scoring Protocol: 2,159 independently verifiable rubric items decompose tasks into deterministic checks and LLM-judged open-ended criteria.
- 3.3 Scoring Protocol: Average Score, Pass@k, and Pass^k respectively measure mean performance, capability ceiling, and reliability floor across multiple trials.
4 Evaluation
Claw-Eval evaluates 14 models across General, Multi-turn, and Multimodal tasks using three independent trials and metrics that expose both performance and consistency. Results show substantial difficulty, modality-specific capability, and remaining headroom.
- Evaluation setup: 14 frontier models are evaluated under a uniform protocol across General, Multi-turn, and Multimodal task groups.Multimodal evaluation is restricted to the 9 models supporting visual input.
- Main results: 70.4% is the strongest Overall Pass^3, showing that even the top model does not consistently pass all three trials.Claude-Opus-4.6 leads Pass^3, while Claude-Sonnet-4.6 leads Score at 81.4%.
- Difficulty: All models degrade monotonically from Easy to Hard as tasks require longer tool-call chains and more complex coordination.The difficulty range spans 14% to 75% Pass^3 on Easy, while Claude-Opus-4.6 retains 65.1% on Hard.
- Multimodal performance: 25.7% is the highest Multimodal Pass^3, far below Claude-Opus-4.6’s 70.8% on General tasks.The result indicates that text-based tool use is more reliable than visual perception and generation in these evaluations.
5 Analysis
Targeted analyses show that trajectory-aware grading detects failures missed by vanilla judging, while error injection separates peak capability from consistency. Multi-turn and multimodal breakdowns further reveal that effective performance depends on questioning precision and domain-specific strengths.
- Hybrid grading: 44% of safety violations and 13% of robustness issues are missed by the vanilla judge across five models and 2,000+ traces.The hybrid pipeline combines deterministic checks for safety-critical criteria with LLM judgment for open-ended assessment.
- Injected failures: Pass@3 remains nearly flat while Pass^3 drops sharply as error-injection rates rise from 0.0 to 0.6.Agents often find one successful path but rarely complete all three trials successfully.
- Injected failures: 56.5% Pass^3 is retained by Claude-Opus-4.6 at error rate 0.6, with its gap expansion limited to 9.9%→20.5%.The model is the most resilient in this analysis, but resilience does not track baseline performance.
- Multi-turn dialogue: 76% of Pass^3 variance is explained by question precision, compared with under 1% from dialogue round count.Question precision combines clarification targeting with the logical progression of information gathering.
- Multimodal analysis: 10.7% average Video Pass^3 trails Doc & Image at 32.3% and Code at 23.9%, with domain leaders differing by modality.Video is led by Claude-Opus-4.6/Sonnet-4.6, Doc & Image by GPT-5.4, and Code by MiMo-V2-Omni.
6 Conclusions
Claw-Eval shows that trustworthy agent evaluation requires trajectory-aware, multidimensional assessment across diverse tasks. Its results expose missed safety and robustness failures, inconsistent performance under perturbation, and domain-specific capability differences.
- 44% of safety violations and 13% of robustness issues were missed by trajectory-opaque judging but caught by the hybrid pipeline.
- Pass^3 dropped by up to 24 percentage points under error injection while Pass@3 remained stable, separating capability from consistency.
- Question quality explained 76% of multi-turn Pass^3 variance, compared with under 1% for round count.
- No single model led across all multimodal domains, indicating that agent capability varies across evaluation dimensions.
A Limitations
The study identifies boundaries in robustness coverage, model-based judgment, and benchmark longevity. These limits constrain which failures, future task types, and deployment conditions Claw-Eval currently represents.
- Robustness evaluation focuses on infrastructure-level transient failures such as HTTP 429/500 responses and latency spikes.Semantic-level and adversarial perturbations are left for future work.
- LLM judges achieve 95% exact agreement with human scoring, but model-based judgment may not generalize to all future task types.
- Future models may be optimized against Claw-Eval’s task distribution, creating a benchmark-longevity concern.Trajectory evidence, deterministic safety gates, and an extensible task schema provide mechanisms to detect shortcut-based gaming and expand evaluation headroom.
C Additional Task Group Details
Claw-Eval organizes tasks into general workflow execution, multimodal perception and generation, and multi-turn professional dialogue. Tasks undergo design, implementation, review, and pilot validation before release.
- General tasks: General tasks range from single-service queries through cross-service coordination to multi-system workflows, alongside standalone analytical scenarios.They use mock service APIs, injected files, and built-in sandbox tools.
- Multimodal tasks: Multimodal tasks cover videos, documents, images, and code-generated media, requiring agents to select visual content and produce functional artifacts.Examples include dynamic webpages, SVG animations, and edited video clips.
- Multi-turn dialogue tasks: Multi-turn dialogue tasks use simulated users with hidden intents and information-revealing strategies, requiring active questioning, clarification, and synthesis.
- Construction and validation: 250 of 300 released tasks were original designs, while the remainder came from established benchmarks.Each task is implemented, formalized into a rubric, reviewed, and pilot-tested before release.
D.3 Reproducibility
Claw-Eval supports reproducibility by releasing its complete benchmark, grading infrastructure, runtime, fixtures, and evaluation scripts. Researchers can inspect rubric items and reproduce reported results or evaluate new models.
- The open-source release includes all 300 task definitions, grader implementations, task fixtures, the sandbox runtime, and end-to-end evaluation scripts.
- The released materials cover prompts, tool specifications, sandbox configurations, environment setup, deterministic checks, and judgment-based rubric logic.
- Researchers can independently inspect every rubric item and evaluate new models using the publicly available codebase, data, and leaderboard.
E.1 Multi-dimensional Task Score.
Claw-Eval scores each task across Completion, Safety, and Robustness, grounding rubric judgments in independently verifiable evidence. Its pipeline combines deterministic and judgment-based checks with human validation of rubric adequacy and grading disagreements.
- Score dimensions: Completion, Safety, and Robustness jointly determine each task attempt’s score.Safety acts as a multiplicative gate, while robustness measures recovery across injected failure types.
- Score dimensions: Safety violations directly suppress the final score, and most safety constraints are embedded in ordinary workflow tasks.This integrates safety assessment with normal task execution rather than isolating it in standalone safety evaluations.
- Rubric construction: Each rubric verdict is grounded in execution traces, service audit logs, or environment snapshots rather than agent self-report.The stored supporting artifact creates an audit trail from the final score to behavioral evidence.
- Rubric construction: 2,159 rubric items evaluate 300 tasks, averaging 7.2 independently verifiable items per task.Items include deterministic checks of files, API calls, and audit logs, alongside judgment-based checks of open-ended outputs.
- Validation: Blinded adjudication matched the hybrid pipeline in 12 of 12 safety disagreements and 13 of 15 robustness disagreements.The vanilla judge matched human judgment in none of the safety cases and 1 of 15 robustness cases.
- Validation: Human audits found most sampled rubrics aligned with intended capabilities, covered key success conditions, and used available evidence.Remaining issues clustered in a small number of borderline cases involving partial-credit boundaries or underspecified criterion scope.
F.3 Alignment Between the Deployed LLM Judges and Human Scoring
Claw-Eval compares deployed LLM judges with adjudicated human reference scores on sampled judgment-based rubric items. The validation supports practical alignment while explicitly stopping short of treating the pipeline as perfect ground truth.
- Evaluation protocol: Gemini-3-Flash was deployed for General and Multimodal tasks, while Claude-Opus-4.6 was deployed for Multi-turn tasks.Exact agreement with human reference scores was reported by task group.
- Evaluation protocol: 100 judgment-based rubric items from each task group were scored by two human annotators and compared with deployed LLM judges.Disagreements were resolved through adjudication to form the human reference score.
- Findings: The deployed judges tracked human scoring closely across General, Multimodal, and Multi-turn task groups.Gemini-3-Flash showed particularly strong agreement on General and Multimodal items, while Claude-Opus-4.6 remained aligned on Multi-turn items.
- Interpretation: The validation supports practical reliability of model-based judging but not perfect ground truth.The supported claim is narrower: the hybrid pipeline is a more evidence-grounded approximation than conversational-trace-only judging.
- Stability context: General-task trial variance was low, with median σ = 1.2%, while Multi-turn and Multimodal variance was systematically higher.The passage attributes higher variance to lower conversion ratios and stochastic simulated-user interaction.
H Compute Resources
The evaluation used public model-provider APIs and ran the harness and sandbox on a single cloud VM. The supplied passages also illustrate the benchmark’s rubric case studies across general, multi-turn, and multimodal tasks.
- Compute resources: 2,310 M input and 79 M output tokens were consumed across the evaluation, averaging 212.3 seconds per task trial.All rollouts and judge calls used public model-provider APIs, with no GPU resources.
- Compute resources: The evaluation harness and sandbox ran on one cloud VM with 64 vCPU and 512 GB RAM.The passage reports no GPU use.
- Case studies: Four case studies covered general retrieval and classification, multi-turn dialogue, and multimodal video understanding with artifact generation and verification.These cases illustrated complete rubric decompositions with independently verifiable per-criterion judgments.
- Case studies: The case-study materials included email triage, statistical-analysis dialogue, video-to-floor-plan reconstruction, and machine-operation localization with dog-frame cropping.The supplied table descriptions show rubric-specific verification of tool use, visual artifacts, spatial accuracy, temporal overlap, and subject centering.