Source-linked AI summary
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, Rui Huang, Ziqi Zhao, Shengyuan Ding, Ailing Yu, Bo Peng, Bowei Xia, Hao Sun, Haotian Liang, Ji Xie, Jiajun Chen, Jiajun Song, Liu Yang, Ming Xu, Qionglin Qiu, Runhao Fu, Shengfang Zhai, Shijian Wang, Tengfei Ma, Tianyi Wu, Weiyang Jin, Yan Wang, Yang Dai, Yao Lai, Youwei Shu, Yue Liu, Yunzhuo Hao, Yuwei Niu, Jinkai Huang, Jiayuan Zhuo, Zhennan Shen, Linyu Wu, Hannah Yao, Charles Chen, Cihang Xie, Yuyin Zhou, Jiaheng Zhang, Zeyu Zheng, Mengkang Hu, Michael Qizhe Shieh
TL;DR
Persistent coworker agents must operate across changing, multimodal work environments, but existing benchmarks largely evaluate static, text-centric sessions. ClawMark introduces multi-day, multi-turn evaluation with stateful services, exogenous updates, and deterministic scoring; Claude Opus 4.6 achieves 20.0 Task Success, showing that complete workflows remain rare.
Problem
Existing benchmarks inadequately measure coworker agents across multi-day timelines, exogenous environment changes, and multimodal evidence.
Method
ClawMark evaluates multi-turn, multi-day workflows in five stateful sandboxed services with independent between-turn changes and deterministic rule-based scoring.
Results
20.0 Task Success for Claude Opus 4.6 shows fully correct end-to-end completion is much rarer than partial progress across the 100-task benchmark.
Takeaways & Limitations
Coworker-agent evaluation must test adaptation to changing state and reliable backend writeback, not only progress within static interactions.
Takeaways & Limitations
Main-table results use a single full sweep per model, so rankings within a narrow weighted-score band should be interpreted cautiously without run-to-run variance.
Abstract
from arXiv · showhide
Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.
1. Introduction
ClawMark targets persistent coworker agents operating across multiple working days in evolving, multimodal environments—a regime existing largely static, single-session benchmarks insufficiently measure. It combines multi-turn workflows, stateful sandboxed services, exogenous between-turn changes, and deterministic rule-based verification in one executable benchmark.
- Motivation: Persistent coworkers must handle independently changing environments and evidence distributed across images, scanned PDFs, audio, video, and spreadsheets.Examples of environmental change include arriving emails, shifting schedules, and updated knowledge-base records.
- Limitations of Existing Benchmarks: Existing benchmarks primarily score single sessions under static-environment assumptions, leaving multi-day adaptation, external change, and multimodal evidence integration insufficiently measured.A file read in one step may differ in a later step when the environment evolves between turns.
- ClawMark: ClawMark evaluates multi-turn workflows spanning multiple in-universe workdays, with one turn per working day, across five stateful sandboxed services.The services are filesystem, email, calendar, knowledge base, and spreadsheet.
- Contributions: ClawMark combines multi-turn multi-day tasks, stateful sandboxed services, exogenous between-turn changes, and deterministic rule-based scoring in one executable setting.These elements are presented as the benchmark’s central contribution for coworker-agent evaluation.
- Contributions: 1,537 deterministic Python checkers inspect post-execution service state, and released tasks require two independent re-runs with bit-identical checksums.The protocol operationalises scoring without an LLM-as-judge.
2. Related work
Prior agent benchmarks largely evaluate tool use and workflow progress within fixed episodes, while newer multi-turn benchmarks still lack independently changing state between workdays. ClawMark addresses this gap by evaluating persistent coworker agents in evolving, multimodal environments.
- Fixed-episode evaluation: Existing benchmarks such as WebArena, Mind2Web, VisualWebArena, OSWorld, MCPMark, MCP-Bench, SWE-bench, AgentBench, GAIA, and Terminal-Bench largely evaluate progress within fixed episodes.These benchmarks establish realistic web, computer-use, tool-coverage, and execution-domain evaluations, but generally do not model independently evolving environments.
- Multi-turn evaluation: Multi-turn benchmarks including tau-bench, WorkArena, and TheAgentCompany move beyond single-turn execution, but later state changes primarily result from interaction rather than exogenous between-day updates.Concurrent benchmarks address adjacent gaps through trajectory-aware scoring, live websites, or evolving information streams.
- Persistent coworker agents: Most LLM-agent systems are still evaluated episodically, with the environment resetting between tasks rather than persisting across in-universe working days.ClawMark instead refreshes independently mutating external state at each turn and exposes agents to raw multimodal evidence.
- Persistent coworker agents: ClawMark targets the complementary coworker-agent regime by combining persistent operation, independently mutating state, and raw multimodal evidence.Its five-service tool schema is designed to remain compatible with different agent frameworks.
3. ClawMark
ClawMark evaluates coworker agents on realistic office workflows unfolding across multiple working days in a stateful, multimodal environment that changes independently between turns. It combines five sandboxed services with deterministic post-execution scoring that distinguishes partial progress from complete workflow success.
- Corpus scale: 100 tasks cover 13 professional scenarios and 87 distinct in-task roles, using 1,072 multimodal artifacts and 1,537 deterministic Python checkers.The artifacts include PDFs, images, audio, video, and spreadsheets; 55 checkers are red-line constraints.
- Scoring: 1,537 checkers produce a weight-normalised score from post-turn sandbox state, with no LLM judge involved.Each task includes 6–29 weighted Python checkers, and verdicts are deterministic pass/fail checks across service state, artifact inspection, and tolerance or equivalence rules.
- Scoring: Weighted Score rewards partial progress, whereas strict Task Success requires completing the entire workflow and serves as the deployment-relevant all-or-nothing measure.Both metrics are reported on a 0–100 percentage scale and are evaluated against the post-turn service state rather than a cached snapshot.
- Task design: Each task spans two to six working days, with one turn per day and independently mutated external state between turns.The framework delivers wake-up messages and executes tasks against five stateful sandboxed services: filesystem, email, calendar, knowledge base, and spreadsheet.
4. Benchmark construction
ClawMark’s benchmark construction combines 100 tasks across 13 heterogeneous professional scenarios with multimodal evidence and deterministic state-based verification. A four-phase pipeline authors tasks, sources artifacts, conducts review, and admits tasks only after strict release-gate checks.
- Scenario coverage: 13 scenarios span eight general office roles and five specialised domains, including clinical assistance, insurance, legal assistance, investment analysis, and electronic design automation.The benchmark includes 87 substantive in-task roles rather than merely cosmetic role labels.
- Authoring pipeline: Task authoring defines turns, service seeds, between-turn injections, and weighted checker rubrics while enforcing silent-mutation coverage, cross-modal contradictions, and deterministic red-line checks.Each task is authored as a single task.py, and the resulting artifact list is passed to evidence sourcing.
- Evidence sourcing: Required artifacts come from web-collected documents, original recordings, or targeted AI synthesis, each carrying a provenance tag.Examples include policy PDFs, voice memos, walkthrough videos, whiteboard photos, forms, and spreadsheets.
- Review loop: 3–5 review rounds alternate task review with trajectory review, combining human inspection, three AI audits, two reference-model runs, and independent runtime-flaw checks.The audits target multimodal integrity, checker-hacking, and task–checker correspondence; trajectory review checks issues such as ambiguous prompts and inject–checker races.
- Release gate: Four simultaneous release-gate conditions require artifact sign-off, clean task-review audits, no trajectory design flaws, and bit-identical checker outputs across two independent reruns.The reruns use the same frozen service state, including identical verdicts and detail messages.
5. Experiments
The experiments evaluate seven frontier models under a shared agent framework and controlled infrastructure, using single-sweep results across 100 tasks. Results show strong partial performance, limited complete task success, distributed scenario leadership, and no monotone link between score and resource consumption.
- Experimental setup: Seven frontier models run under OpenClaw with identical tool schemas, no per-model prompt engineering, and isolated docker-compose environments.The evaluation uses provider-default inference settings, with extended thinking and prompt caching where supported.
- Overall results: 75.8 weighted score is the highest overall result, with Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.4 within a 3.8 pp band.Each model receives one full sweep, so rankings within this narrow band should be interpreted cautiously.
- Overall results: 20.0% Task Success is the strongest model’s complete-task rate, while Sonnet 4.6 reaches 14.0%, GPT-5.4 9.0%, and Kimi K2.5 0.0%.The stricter metric reveals substantial headroom despite weighted-score performance.
- Per-scenario results: Four models achieve per-scenario bests: Sonnet and Opus lead five scenarios each, while GPT-5.4 and Gemini lead one each.The two Anthropic models tie on the single EDA task at 100.0.
- Efficiency analysis: +23% tool calls at −3.8 pp score, +31% input tokens at −7.6 pp, and 1.8× output tokens at −18.6 pp show no monotone score–consumption relationship.Sonnet 4.6 leads score per thousand tool calls at 13.2, followed by Opus 4.6 at 12.2.
6. Analysis
ClawMark exposes substantial adaptation failures after exogenous state changes, while failures concentrate in silent-change detection and backend writeback. Red-line failures are infrequent overall but concentrated in a small set of tasks and model pairs.
- Turn-by-turn trajectory: −11.5 pp was Claude Opus 4.6’s largest Day-1 → Day-2 decline, while Qwen 3.6 Plus alone gained, increasing 1.2 pp.Day 2 is the first external mutation; six of seven models dropped there.
- Turn-by-turn trajectory: Six of seven models finished Day 3 below Day 1, with only Qwen 3.6 Plus returning to near parity at +0.2 pp.Recovery was partial and uneven; Kimi K2.5 had the largest Day-2 → Day-3 rebound at +3.7 pp.
- Failure taxonomy: 31.6% was the benchmark-wide per-evaluation failure rate, based on 3,404 failures across 10,759 checker evaluations.Failures concentrate in the two structural stressors ClawMark is designed to test.
- Failure taxonomy: 56.5% of silent-change detection evaluations and 53.6% of backend writeback evaluations failed, with backend writeback contributing 567 failures.Both failure rates were nearly twice the benchmark-wide average; backend writeback was the largest absolute failure bucket.
- Red-line incidents: 7.1% of red-line evaluations failed, but incidents concentrated in 13 tasks and 23 distinct task-model pairs.The three frontier systems each had a 3.6% red-line fail rate, while Qwen 3.6 Plus reached 14.5%.
7. Conclusion
ClawMark evaluates coworker agents on multi-day, multi-turn workflows with exogenous environment changes and raw multimodal evidence, using deterministic scoring over stateful sandboxed services. Its release process guarantees bit-identical checker verdicts across independent reruns.
- Evaluation scope: ClawMark measures three under-evaluated axes: multi-turn multi-day timelines, exogenous between-turn environment changes, and raw multimodal evidence.These axes target coworker-agent behavior beyond static, text-centric benchmark settings.
- Evaluation scope: Deterministic rule-based scoring evaluates post-turn state in stateful sandboxed services.The measurement is grounded in service state after each turn rather than subjective judgment.
- Reproducibility: ClawMark guarantees bit-identical checker verdicts across independent reruns through a release-gate procedure.This supports reproducibility of benchmark evaluation.
A. Multi-turn evaluation: terminology and conventions
ClawMark defines multi-turn tasks as multiple multi-step interaction episodes separated by possible environmental changes. Its conventions equate each turn with one in-universe working day, while distinguishing legacy “stage” fields and authoring-pipeline “phase” from execution turns.
- Multi-turn: Multi-turn tasks contain multiple independent, multi-step interaction episodes, with possible environmental changes between episodes.Examples include new emails, system notifications, and calendar shifts.
- Turn and day: Turn = Day: each turn is exactly one in-universe working day, with one agent wake-up message at its start.Tasks span two to six working days when they contain two to six turns.
- Legacy terminology: Stage is reserved for legacy task-source and result fields that denote the same per-turn structures, not steps within a single episode.The paper preserves names such as stage0 and stage1 for code compatibility.
- Phase: Phase denotes one of four task-authoring pipeline steps and is intentionally distinct from turn, which describes agent task execution.The phases are task authoring, evidence sourcing, review loop, and release gate.
B. Task definition, parsing, and checking
ClawMark represents each task as a compact, executable bundle of per-turn prompts, evidence updates, service mutations, and deterministic checkers. An isolated orchestrator executes first-class turns across five sandboxed services, evaluates post-turn state, and continues subsequent turns regardless of failures.
- Task representation: A task.py defines per-turn async entries, evidence inject layers, service-side mutation hooks, and one deterministic pass/fail checker per rubric item.The corresponding assets and inject/stage{k}/ files provide static evidence and between-turn updates.
- Turn execution: The orchestrator runs each turn end-to-end in an isolated Docker Compose stack containing the agent and five stateful services.These services include a mounted filesystem, GreenMail, Notion, Google Sheets, and Radicale CalDAV.
- Turn execution: At each turn’s end, checkers inspect post-turn sandboxed-service state, record outcomes, and allow the next turn to proceed regardless of failure.This separates per-turn verification from continued multi-turn execution.
- Framework design: Turns are first-class evaluation episodes with their own prompts and rubric items, while services compose freely rather than being retrofitted onto a single-service or single-turn substrate.The framework keeps natural-language descriptions and executable task components in tight correspondence, supporting 3–5 rounds of author–reviewer iteration across the 100-task, 13-scenario corpus.
C. Reproducibility and framework patches
The evaluation standardized OpenClaw builds, runtime limits, inference settings, and task isolation across seven-model sweeps. A GPT-5.4 case study scored 80.0 and uniquely transitioned from audio to video-frame vision.
- Framework patches: The seven-model sweep used one OpenClaw build with model-specific patches for tool-call IDs, null arguments, text-only inputs, and GPT-5 routing.The patches disabled tool-call-ID sanitisation, replaced null arguments with {}, declared text inputs, and routed GPT-5 to the openai-responses API with high thinking effort.
- Runtime controls: Each task ran in isolated containers, with a forced two-hour per-turn timeout, a 30-minute LLM idle timeout, and default parallelism of 4–8 stacks.Containers were torn down between tasks, preventing per-task state sharing.
- Inference settings: All seven models used provider-default sampling, extended thinking where supported, prompt caching where supported, and no per-model prompt engineering.Reported wall-clock, token, and tool-call totals reflect a single full sweep per model.
- Case study: GPT-5.4’s highest-scoring trajectory on content_operation_task7 achieved score 80.0 and uniquely transitioned from audio to video-frame vision.This causal transition was unique among evaluated models.
D. Run-to-run stability
Three independent full-corpus sweeps show small run-to-run variation for both Kimi K2.6 and GPT-5.4 under settings matching the main sweep.
- D. Run-to-run stability: Three independent full sweeps covered the 100-task corpus for Kimi K2.6 and GPT-5.4.The models represent opposite sides of the open/proprietary divide, and the harness, container limits, and inference settings matched the main sweep.
- D. Run-to-run stability: 2.8 pp: Kimi K2.6’s three weighted scores were 68.4, 70.8, and 71.2.Kimi K2.6 is the open-source model in this stability analysis.
- D. Run-to-run stability: 1.0 pp: GPT-5.4’s three weighted scores were 72.0, 72.5, and 73.0.GPT-5.4 is the proprietary model in this stability analysis.
E. Case studies · F. Author list
The case studies illustrate cross-modal reasoning in event operations and compliance-sensitive adaptation in insurance claims. The paper also provides its full author list, affiliations, and contribution acknowledgments.
- E. Case studies: Case 1 combines a voice memo, walkthrough video, PDF quotes, floor plans, and an Excel budget in a cross-modal DevSummit event-operations task.GPT-5.4 resolves the task through a cross-modal reasoning chain; its highest-scoring run is 80.0%.
- E. Case studies: Case 2 is a four-turn auto-insurance claim-adjudication task spanning Thursday 3/21 and Friday 3/22.A revised repair-shop quote and claimant pressure arrive on turn 3, while the technical report needed for the final decision arrives on turn 4.
- E. Case studies: The insurance case uses the red-line checker S3_redline_no_direct_approve, weighted 2.0, to encode a compliance constraint.The constraint concerns approving the claim on day 3 before the technical report arrives on day 4.
- F. Author list: The author list begins with Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, and Xiangyan Liu, followed by additional contributors.The passage marks the first five listed authors with superscript affiliations and contribution stars for the first three.
- F. Author list: The remaining author list includes Yang Dai, Yao Lai, Youwei Shu, Yue Liu, Yunzhuo Hao, and many other contributors.The passage concludes with Mengkang Hu and Michael Qizhe Shieh, both marked with the corresponding-author symbol.
- F. Author list: The affiliations span Evolvent AI and universities including the National University of Singapore, MIT, and the University of Hong Kong.The passage lists 21 numbered institutional affiliations across the author list.
- F. Author list: The contribution notes define ∗ as equal contribution and † as corresponding authors.These symbols appear alongside author names in the author list.