PaperlayerView exact revision on arXiv

Source-linked AI summary

ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

YuanHang Xiao

arXiv:2608.22510v1cs.AI

TL;DR

Agent benchmarks can under-specify evaluation when they score only final answers from stateful runtimes. ClawProBench addresses this with trace-aware, safety-gated evaluation of declared model-plus-runtime configurations across a broad OpenClaw profile and a frozen workspace holdout, finding gaps in native routing and repeated reliability.

  • Problem

    Existing benchmarks under-measure whether a declared model-runtime configuration coordinates native surfaces while preserving reliability, status, and process evidence under one auditable protocol.

  • Method

    ClawProBench evaluates declared model-plus-runtime configurations using a 102-scenario OpenClaw profile, a frozen 68-scenario workspace holdout, and trace-based scoring of correctness, process, efficiency, and safety.

  • Results

    The benchmark reports non-saturation, native-slice and repeated-reliability gaps, scoring sensitivity, weak cross-track alignment, and same-model configuration sensitivity across runtimes.

  • Takeaways & Limitations

    Model-plus-runtime behavior should be compared as a declared configuration because answer-only rankings can hide native-surface weaknesses, repeated-trial instability, and trace-local failures.

  • Takeaways & Limitations

    The comparison does not isolate wrapper, routing, tool-schema, safety-layer, or controller mechanisms inside the runtime bundle, so it supports system-level rather than single-mechanism claims.

Abstract

from arXiv · show

Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.

1 Introduction

CLAWPROBENCH evaluates declared model-plus-runtime configurations through trace-aware, safety-gated execution rather than final answers alone. Its two-track design combines broad runtime coverage with frozen-contract reliability analysis and joint diagnostic views.

  • Agent evaluation must account for runtime operation because correct answers can still involve wrong surfaces, missing evidence, unsafe approvals, wasteful retries, or uninspectable traces.
  • CLAWPROBENCH introduces trace-aware evaluation for declared model-plus-runtime configurations.
  • The full profile contains 102 active scenarios, including 66 workspace-live tasks and 36 native OpenClaw-surface tasks.
  • The frozen holdout contains 68 workspace-style closed-world JSON scenarios with fixed identities, end-state contracts, and checker interfaces for repeated-trial and cross-runtime comparison.
  • The benchmark combines trace scoring, bounded process credit, safety gates, efficiency, status semantics, hashes, and runtime adaptation interfaces.
  • Current evaluations show non-saturation, native-slice and repeated-reliability gaps, scoring sensitivity, weak cross-track alignment, and configuration sensitivity.

2 Related Work

Related benchmarks cover browser, desktop, workplace, tool-use, live, trajectory-aware, and safety-focused evaluation. CLAWPROBENCH distinguishes itself by integrating these concerns into joint diagnosis of native surfaces, process, status, reliability, and declared-runtime comparisons.

  • Web and interface-agent benchmarks: Browser and interface-agent benchmarks evaluate grounding, navigation, and online interaction, but generally do not cover the broader runtime surfaces of a full agent product.
  • Desktop, workplace, and tool-use evaluation: Desktop, workplace, and tool-use benchmarks study applications, service-backed workflows, high-fidelity mock services, dynamic services, APIs, MCP servers, and multi-environment skills.
  • Positioning: CLAWPROBENCH reports native-surface, process, status, and reliability diagnostics together in a model-plus-runtime setting.
  • Live and native-runtime benchmarks: Live and native-runtime benchmarks already emphasize realistic execution, trajectories, safety, robustness, or preserved artifacts; CLAWPROBENCH integrates these into joint per-configuration diagnosis.
  • Benchmark exposure and contamination: The benchmark treats exposure and execution status as reportable metadata rather than claiming to eliminate contamination risk.
  • Benchmark composition: Its full profile covers six capability dimensions and separates 66 workspace-live tasks from 36 OpenClaw-native tasks.

3 Benchmark

ClawProBench evaluates declared model-plus-runtime configurations through structured scenarios, trace-based scoring, and explicit runtime-adaptation semantics. Its benchmark combines broad live coverage with a separately frozen workspace holdout for inspectable, repeatable evaluation.

  • Benchmark scope: The benchmark unit is a declared model-plus-runtime configuration, not a model name alone.The configuration includes the model endpoint, wrapper, controller, runtime, tools, schemas, safety filters, execution policy, checker bundle, and scoring code.
  • Scenario inventory: The full profile contains 102 active scenarios across six capability dimensions, including 66 workspace-live and 36 native-runtime tasks.Native tasks span skills, browser, memory, messages, sessions, directory, cron, and delegated agents.
  • Scenario design: Each scenario specifies metadata, workspace inputs, expected runtime surfaces, grading checks, and optional custom Python graders.Workspace-live tasks test evidence synthesis, planning, safety, or recovery, while native tasks require routing through specified OpenClaw surfaces.
  • Scenario inventory: The frozen holdout contains 68 workspace-live scenarios with inline files and closed-world JSON contracts selected by a stable selector after calibration and pruning.The fixed identities, contracts, and checker interfaces support repeated-trial reliability and cross-runtime comparison.
  • Runtime adaptation: Cross-runtime adaptation preserves scenario identity, contracts, trace fields, evidence metadata, postconditions, safety labels, checker interfaces, and three-trial status reporting.The runtime-specific native partition is not claimed portable through tool renaming.
  • Trace scoring: The trial score combines safety-gated correctness, process quality, and efficiency, with trace checks covering outputs, tool use, ordering, recovery, artifacts, and audit-state matches.The score does not require exact trace identity, and process disagreement alone is bounded at 0.35 when correctness is perfect and no safety or efficiency penalty applies.

4 Experiments

Experiments examine saturation, configuration sensitivity, native-slice difficulty, holdout reliability, rank uncertainty, and scoring-view sensitivity. Results show meaningful runtime and task-slice differences, weak cross-track rank stability, and substantial dependence on scoring perspective.

  • Setup and saturation: The full-profile diagnostic panel reports a manifest-wide maximum overall score of 0.7671 and capability score of 0.7796.Execution status is audited separately because clean runs, reruns, and execution failures represent different measurement conditions.
  • Runtime configuration sensitivity: The largest within-model OpenClaw-release aggregate-score range is approximately 0.052, with a strict-3/3 range of 10/68 scenarios.The version sweep covered four model identities and four OpenClaw releases with successful statuses for all 3,264 trial records.
  • Runtime configuration sensitivity: The largest same-model cross-harness aggregate-score range is 0.0716, while the largest strict-3/3 range is 13/68 scenarios.No harness dominates every model, and the jointly varying harness bundles provide configuration-level rather than causal-ablation evidence.
  • Native-slice difficulty: Native scenarios average 0.5238 versus 0.6415 for workspace-live scenarios.The workspace-over-native gap remains above 0.10 under several observable stratifications and in a report-level bootstrap.
  • Holdout reliability and rank robustness: Mean pass@k-any is 0.6638, while mean strict three-trial pass is 0.2890.The expanded cross-profile versus holdout Spearman correlation is 0.1754 with a wide interval spanning zero, whereas full-vs-core and full-vs-native rankings are moderately aligned.
  • Scoring and trace sensitivity: Correctness-only scores correlate with the manifest ranking at Spearman 0.8334 but shift one model by as many as 42 positions in the positive-score view.Using all 66 component-resolved rows gives Spearman 0.8060 and a maximum shift of 46, showing that scoring views are not interchangeable.
  • Scoring and trace sensitivity: Trace-derived low scores include exact-output or structured-constraint failures, missing evidence, approval or temporal-boundary mistakes, and tool or runtime-routing failures.These categories come from failed checker details and are treated as audit priorities rather than final prevalence estimates.

5 Discussion

The benchmark should be interpreted as measuring declared model–runtime configurations, with native routing, repeated reliability, and trace-local failures treated as diagnostic dimensions. Its rankings are therefore diagnostic rather than deployment approval, and cross-track ordering remains uncertain.

  • CLAWPROBENCH measures model-plus-runtime behavior rather than harness-independent model intelligence.
  • Native scenarios localize runtime routing and surface-use weaknesses, while the frozen workspace holdout diagnoses repeated reliability under a fixed contract.
  • Trace components and status metadata distinguish how configurations fail and preserve conditions that affect leaderboard interpretation.
  • Cross-track ordering is not precise enough to support a stable law, whereas repeated-trial and runtime-sensitivity gaps are directly observed.
  • Scores are diagnostic slices across profile, holdout, status, process, safety, reliability, and runtime views, not deployment approval.

6 Conclusion

The conclusion presents CLAWPROBENCH as a trace-aware benchmark for diagnosing declared model–runtime configurations across a live OpenClaw profile and frozen workspace holdout. It argues that answer-only rankings can conceal runtime, reliability, comparability, and trace-local differences that become auditable through the benchmark protocol.

  • CLAWPROBENCH jointly diagnoses declared model-plus-runtime configurations using a 102-scenario OpenClaw profile and a frozen 68-scenario workspace holdout.
  • Answer-only rankings can hide native-surface weaknesses, repeated-trial instability, status-dependent comparability, and heterogeneous trace-local failures.
  • Under a fixed workspace contract, changing the declared runtime changes the same model identity’s score and strict reliability.
  • Harness ordering depends on the model, so the meaningful comparison is the declared configuration rather than a framework winner.

Limitations

The paper’s conclusions are bounded by configuration-level measurement, limited coverage, public calibrated holdout selection, and analyses that are often descriptive rather than causal. These constraints limit substrate-free, universally generalizable, and stable-ranking claims.

  • The benchmark does not isolate wrapper, routing, tool-schema, safety-layer, or controller mechanisms within a runtime bundle.
  • Coverage is limited to current task domains, languages, model-access pathways, and provider configurations; native evidence remains OpenClaw-specific.
  • The frozen holdout is a calibrated selector freeze rather than a permanently unseen private test set, leaving public-release overfitting risk.
  • Native-versus-workspace differences and failure groupings are descriptive, while the expanded full-versus-holdout correlation interval spans zero.
  • The paper frames its limitations as construct, internal, external, statistical, and release-contamination validity threats.

Ethical Considerations

The ethical and governance discussion treats safety, privacy boundaries, prompt injection, credentials, approvals, release exposure, and auditability as part of responsible agent evaluation. It emphasizes reproducible artifacts and trace evidence while limiting interpretations to research diagnostics rather than deployment approval.

  • Ethical safeguards: CLAWPROBENCH includes safety, privacy-boundary, prompt-injection, credential, approval, and no-write scenarios.
  • Ethical safeguards: Benchmark artifacts use manually designed synthetic records and secret-like strings rather than real user data or credentials.
  • Governance: Hidden execution failures, reruns, model settings, and prior exposure can mislead leaderboard interpretation, so status and exposure labels are reported.
  • Governance: Scores are research diagnostics, not substitutes for deployment-specific safety review, access control, or human oversight.
  • Auditability: The appendix serves as an audit trail documenting scenario construction, release contracts, scoring stress tests, and trace-level evidence.
  • Measurement scope: Runtime surfaces create a distinct measurement target, motivating native-surface diagnostics and native-gap sensitivity views.
  • Measurement scope: Runtime provenance affects measured outcomes, making the native-runtime claim an empirical diagnostic question rather than a naming artifact.
  • Measurement scope: Cross-runtime evaluation uses a workspace adaptation contract, matched-model comparisons, and status labels to assess interpretability.

B Benchmark Positioning

ClawProBench positions its benchmark artifact as an auditable, executable evaluation framework rather than a collection of isolated prompts. Its governance and schema bind tasks to runtime behavior, checks, provenance, and release controls.

  • Artifact and schema: The benchmark artifact includes executable scenario schemas, representative cards, release-governance tables, and audit-oriented supporting tables.These materials document task structure, provenance, status labels, freezing, hashes, reproducibility, validity threats, and disclosure controls.
  • Artifact and schema: Each benchmark item binds a task contract, workspace state, expected runtime behavior, deterministic checks, and release metadata.The schema supports freezing the 68-task holdout while non-holdout tasks continue evolving under explicit exposure labels.
  • Governance and positioning: ClawProBench combines OpenClaw native-surface taxonomy, per-configuration diagnosis, status-aware scoring, a frozen reliability holdout, and a workspace adaptation contract.The positioning is descriptive rather than a win/loss checklist against prior benchmarks.
  • Governance and positioning: The benchmark’s trust model treats status semantics, exposure labels, row provenance, and release hygiene as central to interpreting results.Governance tables separate paper-snapshot choices from durable public-release obligations.

E Additional Diagnostics

Additional diagnostics treat benchmark results as auditable rank and failure evidence, not merely as a single leaderboard. They emphasize status-aware comparison, uncertainty, and trace-localized mechanisms.

  • Rank and status diagnostics: Status filtering changes which result rows are directly comparable, so leaderboard interpretation depends on execution status.The status table and analysis views keep clean, rerun, and unresolved execution-failure cases distinct.
  • Trace-localized failures: Trace-level evidence identifies how a final-answer failure occurred, including a checker-derived vendor-injection secret-leak mechanism.The checker evidence records the failure as custom_safety:vendor_injection_secret_leak rather than as an unconstrained human label.
  • Uncertainty: Bootstrap intervals support stable descriptive gaps for native performance and holdout reliability but leave full-versus-holdout rank alignment wide.The authors therefore avoid precise holdout ordering or stable cross-track misalignment claims.
  • Rank and status diagnostics: The frozen holdout provides an additional ranking diagnostic alongside a clearer repeated-trial reliability signal.Its cross-profile ordering remains uncertain, while reliability differences are more interpretable.

F Native Surface Details

Native-surface diagnostics isolate runtime-affordance use from aggregate performance and show why declared runtime and adapter fields matter for comparison. The holdout complements, rather than replaces, the broader live profile.

  • Native-surface diagnostics: Native-surface diagnostics ask whether agents can discover and use OpenClaw affordances, rather than simply measuring aggregate leaderboard performance.Surface counts are not mutually exclusive, and sessions, agents, and memory have the lowest mean native slices.
  • Native-surface diagnostics: The workspace-over-native gap remains between 0.1008 and 0.1306 under the simple stratifications in Table 35.These stratified comparisons are diagnostic, not causal estimates.
  • Holdout and configuration comparison: The frozen holdout fixes a realistic task set for repeated-trial reliability analysis without later inventory drift.It is intended as a complementary diagnostic to the larger full profile, not merely a cheaper substitute.
  • Holdout and configuration comparison: Alternative aggregation views preserve top-10 overlap at 10/10 while placing Spearman in a narrow 0.1123–0.1496 range.These alternatives prevent a hidden aggregation detail from becoming an implicit ranking policy.

J Scoring and Failure Details

The scoring and failure analyses make process, safety, status, uncertainty, and trace evidence explicit. They show that reliability and ranking conclusions depend on the chosen diagnostic view and comparison denominator.

  • Failure analysis: Trace-failure counts are failed-check instances rather than unique scenario or model failures, making the taxonomy an audit lens over traces.The approach exposes recurring breakdowns without treating each failed check as a distinct model-level failure.
  • Scoring: The scoring contract combines correctness, process quality, efficiency, and a non-compensatory safety gate.Process can assess tool appropriateness, ordered-subsequence coverage, redundant-step control, or scenario-specific rubrics, while efficiency penalizes excess calls.
  • Ranking uncertainty: Expanded denominators and bootstrap analyses preserve a low cross-track point estimate but leave rank-order signals uncertain.The expanded Spearman analysis uses configuration-row bootstrap resampling, while the 27-model filtered view remains secondary.
  • Reliability and status: Holdout pass@k-any substantially exceeds strict pass, showing that occasional success and stable three-trial reliability are distinct views.Execution status labels retain clean, rerun, and unresolved failure rows separately.
  • Scenario checks: The benchmark distinguishes plausible schedules from schedules violating deadlines, blockers, or conflict constraints.This illustrates how declared checks can evaluate constraint satisfaction beyond final-answer plausibility.
Loading 2608.22510v1…