ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

Introduction

• Existing benchmarks under-measure coworker work by evaluating single sessions with static or agent-caused state changes, while office evidence often extends beyond text [1] [2] [3] [4] [5].Some benchmarks add images, but the passage identifies raw multimodal evidence as a broader requirement.Source: §1 · pp. 1–2
• ClawMark contains 100 tasks across 13 professional scenarios, each spanning multiple workdays across five stateful sandboxed services with external announced or silent changes between turns.Evidence is delivered untranscribed; the services are filesystem, email, calendar, knowledge base, and spreadsheet. Table 1 describes ClawMark as the only representative benchmark combining multi-day tasks, dynamic external state, full raw-office multimodality, and rule-based scoring.Source: §1 · p. 2, Table 1 · p. 2
• 75.8 weighted score and 20.0 strict Task Success were the respective highs among seven frontier systems; six of seven models dropped on Day 2 across 73 three-turn tasks.The Claude Sonnet 4.6–GPT-5.4 gap narrowed from +6.5 percentage points on Day 1 to +4.0 on Day 3.Source: §1 · p. 2
• Scoring uses 1,537 deterministic Python checkers on post-execution service state, and corpus tasks require two independent reruns with bit-identical verdicts and diagnostics.The scoring protocol avoids LLM-as-judge evaluation.Source: §1 · p. 2
• ClawMark releases its benchmark, evaluation harness, and construction pipeline to support reproducible evaluation of coworker agents.Similar aggregate scores can conceal different adaptation trajectories over time.Source: §1 · p. 2

2.1. Agent benchmarks

• WebArena [6], Mind2Web [11], VisualWebArena [4], and OSWorld [5] establish realistic single-episode settings; other benchmarks extend tool coverage or domains but largely retain fixed episodes.The latter group includes MCPMark [8], MCP-Bench [12], SWE-bench [13], AgentBench [14], GAIA [15], and Terminal-Bench [7].Source: §2.1 · p. 3
• tau-bench [1], WorkArena [2], and TheAgentCompany [3] add multi-turn execution, but ClawMark combines multi-day tasks, exogenous state changes, raw multimodal evidence, and rule-based verification.In the cited multi-turn benchmarks, later state changes arise primarily from interaction rather than exogenous changes between workdays.Source: §2.1 · p. 3

2.2. LLM agents

• Agent frameworks and scaffolds support tool use, code execution, and cross-service orchestration, but most systems are evaluated episodically rather than as persistent coworkers.ClawMark's five-service tool schema is harness-agnostic, while its reported seven-model comparison uses one OpenClaw harness to isolate model-side differences; this does not establish empirical equivalence across frameworks.Source: §2.2 · p. 3

3.1. Overview

• Tasks span two to six working days across five stateful services, with announced or silent changes between turns and raw multimodal evidence.The release contains 100 tasks across 13 professional scenarios, 87 distinct in-task roles, and 1,072 raw multimodal artifacts. Tasks average 3.6 turns and 15.4 checkers.Source: §3.1 · p. 4

3.2. Evaluation

• ClawMark scores post-turn service state with deterministic checkers, reporting both a weight-normalized score and all-or-nothing Task Success.Its 1,537 checkers include 55 high-weight red-line constraints; no LLM judge is used. Task Success requires every checker to pass, while weighted score rewards partial progress.Source: §3.2 · p. 5

3.3. Design principles

• Its design combines multi-turn timelines, between-turn environment mutations, and raw multimodal evidence that agents must parse with their own tools.Mutations include new workspace files and service-side changes to email, spreadsheets, knowledge-base entries, and calendar events, requiring agents to refresh state each turn.Source: §3.3 · p. 5

4.1. Task distribution

• The 13 scenarios span general office roles and specialized fields, with 87 substantive in-task roles, including four distinct clinical-assistant roles.Per-scenario counts and dynamic-environment composition statistics are reported in Table 2.Source: §4.1 · p. 6

4.2. Construction pipeline

• A task-first, four-phase pipeline authors tasks before sourcing evidence, then reviews each task iteratively and applies a release gate.Each task undergoes 3–5 review rounds; the phases are authoring steps, distinct from the in-task turns that represent working days.Source: §4.2 · p. 6
• Task authors define turns, service seeds, between-turn changes, and weighted checker rubrics, tying every silent mutation to a checker and every cross-modal contradiction to at least two modalities.Red lines must be expressed as deterministic state checks rather than prose matching.Source: §4.2 · pp. 6–7
• Required evidence artifacts are sourced from public web documents, original recordings, or targeted AI synthesis, each with a provenance tag.Tasks are authored before artifacts are sourced so checker coverage is designed rather than incidental.Source: §4.2 · p. 7
• Each task undergoes 3–5 review rounds that combine human artifact inspection and three AI audits with end-to-end trajectories from two reference models reviewed for runtime design flaws.Findings return to the author for revision; audits cover multimodal integrity, checker-hacking, and task–checker correspondence.Source: §4.2 · p. 7
• Release requires artifact sign-off, clean task-review audits, no trajectory design-flaw findings across two reference models, and bit-identical checker outputs on two reruns against frozen service state.This final condition operationally guarantees the no-LLM-as-judge claim; tasks failing it more than twice are redesigned or dropped.Source: §4.2 · p. 7

5.2. Main Results

• 75.8 weighted score (Claude Sonnet 4.6) led the seven-model leaderboard, while Claude Opus 4.6 led Task Success at 20.0, followed by Sonnet 4.6 at 14.0 and GPT-5.4 at 9.0.Each model’s result comes from a single full sweep, so rankings within the narrow weighted-score band should be interpreted cautiously; run-to-run variance is not reported.Source: Table 3 · p. 8, §5.2 · p. 8, §5.1 · p. 8

5.1. Experimental Setup

• All models ran under OpenClaw with identical tool schemas and no per-model prompt engineering, in isolated Docker-compose task environments torn down between tasks.Evaluation used provider-default inference settings, with extended thinking and prompt caching enabled where supported.Source: §5.1 · p. 8

5.2. Main Results

• 92.6 was Claude Opus 4.6’s highest per-scenario score outside single-task EDA, while every model scored below 44.0 on project management, averaging about 35.1.Task Success gives no credit when a trajectory misses a silent mutation, backend writeback, or turn-specific rubric item, helping explain why partial progress does not count as full completion.Source: §5.2 · p. 9
• Per-scenario bests split across four models: Claude Sonnet 4.6 and Claude Opus 4.6 each lead five scenarios, GPT-5.4 leads executive assistant, and Gemini leads investment analyst.The Anthropic models tie at 100.0 on the single EDA task; Gemini scores 91.3 versus Kimi K2.6’s 8.7 there, after a vision-dependent task was routed incorrectly. EDA is a case-level result, not a stable scenario trend.Source: §5.2 · p. 9, Table 4 · p. 9
• 13.2 score per thousand tool calls put Sonnet 4.6 first, ahead of Opus 4.6 (12.2), Kimi K2.5 (11.7), Gemini (11.6), Kimi K2.6 (11.3), GPT-5.4 (10.2), and Qwen (9.3).Score does not rise monotonically with tool or token consumption; tool calls and tokens are compute-side proxies, not direct dollar-cost measures.Source: §5.1 · p. 8, §5.2 · p. 9

6.1. Turn-by-turn trajectory

• Six of seven models fell on Day 2 after the first external mutation; Qwen 3.6 Plus alone rose from 56.7 to 57.9 (+1.2 pp).By Day 3, six of seven models remained below Day 1; only Qwen 3.6 Plus returned to essentially parity (+0.2 pp).Source: Figure 4 · p. 10, §6.1 · p. 10

6.2. Failure-mode taxonomy

• Other categories failed at 34.0% for cross-source consistency, 31.4% for deliverable correctness, 23.6% for evidence extraction, and 21.5% for compliance guardrails.Scenario-specific rubric items accounted for 67.0% of failures because they were numerous, while their fail rate was 29.5%, close to the benchmark-wide average.Source: §6.2 · p. 10
• Red-line checkers failed 7.1% of the time (26 of 364 evaluations), concentrated in 13 tasks and 23 task-model pairs; Qwen 3.6 Plus had a 14.5% fail rate.Claude Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro each had 3.6% red-line fail rates; every evaluated model tripped at least one red-line on pm_task2.Source: §6.2 · pp. 10–11
• Across 10,759 checker evaluations, 3,404 failed (31.6%); silent-change detection failed at 56.5% and backend writeback at 53.6%.Both structural failure rates were nearly twice the benchmark-wide average; backend writeback contributed 567 failures, or 16.7% of all failures.Source: §6.2 · p. 10, Table 5 · p. 11
• Red-line fail rates varied by subtype: compliance-bypass 10.4%, data-exfiltration 8.6%, premature-decision 6.1%, and irreversible-write 3.3%.Red-lines were unevenly distributed across scenarios, so their fail rates should be interpreted against scenario-specific denominators; models failed more often on judgment- and confidentiality-sensitive cases than hard do-not-modify constraints.Source: §6.2 · p. 11

7. Conclusion

• Silent-change detection fails in 56.5% of evaluations and backend writeback fails in 53.6%, the two dominant failure modes in ClawMark’s taxonomy.The paper notes that agents must refresh external state after exogenous updates and commit results to the right service to be trusted with real professional workflows.Source: §7 · p. 11
• ClawMark evaluates three undermeasured dimensions: multi-turn, multi-day timelines, exogenous between-turn environment changes, and raw multimodal evidence.Scoring uses deterministic rules over post-turn state in stateful sandboxed services, with bit-identical checker verdicts across independent reruns.Source: §7 · p. 11
• The authors release ClawMark, its harness, and 700 execution traces to support targeted progress on silent-change detection and backend writeback.Source: §7 · p. 11

• Multi-turn tasks contain multiple independent episodes, each involving multiple tool calls, and the environment may change between episodes.Examples include new emails, system notifications, and calendar shifts.Source: A. Multi-turn evaluation: terminology… · p. 14
• In ClawMark, one turn equals one in-universe working day, with a wake-up message at its start; tasks span two to six turns.Plots and tables label turns as days, so Day 1, Day 2, and Day 3 correspond to the first three turns.Source: A. Multi-turn evaluation: terminology… · p. 14
• “Stage” is retained only for legacy task-source and result fields that denote per-turn structures, not steps within an episode.The paper does not use “stage” in the agent-literature sense of an intra-episode step.Source: A. Multi-turn evaluation: terminology… · p. 14
• A construction “phase” is distinct from an execution “turn”: phases describe task authoring, while turns describe agent task execution.The four authoring phases are task authoring, evidence sourcing, review loop, and release gate.Source: A. Multi-turn evaluation: terminology… · p. 14

B. Task definition, parsing, and checking

• Each task bundle defines per-turn prompts and mutation hooks, injected evidence, deterministic rubric checkers, and checker weights and turn assignments.The loader parses these files into runtime task objects, and the orchestrator aggregates per-turn outcomes into a final task score.Source: B. Task definition, parsing, and… · p. 14, Figure 5 · p. 15
• The orchestrator runs each turn in an isolated Docker Compose stack with five services and checks post-turn state, regardless of prior failures.After the final turn, rubric items are aggregated under Eqs. 1 and 2.Source: B. Task definition, parsing, and… · p. 14
• ClawMark specifies wake-up messages, loud and silent updates, and rubric items once for both task description and execution, enabling 3–5 author–reviewer iteration rounds across 100 tasks and 13 scenarios.The passage attributes the affordability of iteration to keeping intent and execution closely aligned.Source: B. Task definition, parsing, and… · p. 14
• ClawMark makes services freely composable and turns first-class evaluation episodes, designing for multi-service, multi-turn benchmarks rather than retrofitting a single-service or single-turn substrate.Each turn has its own prompts and rubric items.Source: B. Task definition, parsing, and… · p. 14

C. Reproducibility and framework patches

• Seven model-specific OpenClaw routing and tool-definition patches were applied uniformly and did not advantage any particular model.The patches handled tool-call IDs and arguments, text-only inputs, model-specific API routing, and Gemini JSON-Schema compatibility.Source: C. Reproducibility and framework patches · p. 15
• Each turn had a forced two-hour timeout and a 30-minute LLM idle timeout, with 4–8 concurrent compose stacks and containers torn down between tasks.Per-task runs therefore did not share container state.Source: C. Reproducibility and framework patches · p. 15
• All seven models used provider-default sampling, supported extended thinking and prompt caching, and received no per-model prompt engineering.Wall-clock, token, and tool-call totals in Table 3 reflect one full sweep per model.Source: C. Reproducibility and framework patches · p. 15

D. Run-to-run stability

• Across three full sweeps, weighted scores varied by 2.8 pp for Kimi K2.6 (68.4, 70.8, 71.2) and 1.0 pp for GPT-5.4 (72.0, 72.5, 73.0), versus 19.8 pp cross-model spread.The sweeps used the same harness, container limits, and inference settings as the main sweep.Source: D. Run-to-run stability · p. 16

Case 1: cross-modal reasoning chain on content_operation_task7

• GPT-5.4’s highest-scoring content_operation_task7 run (80.0%) linked an audio-derived capacity concern to targeted video-frame inspection, a chain unique among inspected runs.The task combined a voice memo, walkthrough video, PDF quotes, floor plans, and an Excel budget; the model converted the video into frames with ffmpeg before searching them.Source: E. Case studies · p. 16

Case 2: red-line violation on insurance_task1

• Kimi K2.5’s day-3 approval violated the red-line constraint, lowering its task score from 58.1% (counterfactual) to 48.8% (actual), a −9.3 pp reduction.The model still passed seven other turn-3 checkers; across the benchmark, 26 red-line trips occurred in 23 distinct (task, model) pairs.Source: E. Case studies · p. 16