Source-linked AI summary
ClawArena: Benchmarking AI Agents in Evolving Information Environments
Haonian Ji, Kaiwen Xiong, Siwei Han, Peng Xia, Shi Qiu, Yiyang Zhou, Jiaqi Liu, Jinlong Li, Bingzhou Li, Zeyu Zheng, Cihang Xie, Huaxiu Yao
TL;DR
Persistent assistants need to maintain correct beliefs despite conflicting sources, evolving evidence, and implicitly expressed preferences. ClawArena addresses this gap with hidden-ground-truth scenarios, coupled evaluation dimensions, and complementary reasoning and workspace checks, finding that model capability varies more than framework design and that revision difficulty depends on update design.
Problem
Existing benchmarks do not jointly evaluate conflict resolution, belief revision after new evidence, and silent preference retention in evolving information environments.
Method
ClawArena exposes agents to noisy traces over multi-channel sessions, workspace files, and staged updates while evaluating three coupled dimensions across layered scenarios.
Results
Model capability produces a 29.46-point CRS range versus 23.78 points for framework design, while MetaClaw improves Robustness without degrading accuracy.
Takeaways & Limitations
Belief revision difficulty is governed by update design and the required bridging inference rather than update volume alone.
Abstract
from arXiv · showhide
AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.
1 Introduction
ClawArena addresses the difficulty of maintaining reliable beliefs when evidence conflicts, changes over time, and reveals user preferences implicitly. It combines these challenges in a benchmark designed to evaluate reasoning, revision, personalization, and behavioral consistency.
- Motivation: Persistent assistants must weigh conflicting evidence, revise conclusions after new information, and infer preferences from corrections and interaction patterns.These challenges are coupled: handling only some of them can still produce unreliable output.
- Motivation: Existing benchmarks typically isolate these capabilities through single-authority environments, static evidence, or recall-focused evaluations.They do not evaluate the full combination of conflict resolution, dynamic revision, and silent preference retention.
- Benchmark contribution: ClawArena exposes agents to noisy, partial, contradictory traces while verifying answers against complete hidden ground truth.Scenarios combine multi-channel histories, workspace files, staged updates, and a four-stage personalization protocol.
- Benchmark contribution: 14 categories arise from interactions among three evaluation dimensions, while multi-choice set-selection and shell-based executable checks test reasoning and workspace grounding.The benchmark comprises 12 multi-turn scenarios, 337 evaluation rounds, and 45 dynamic updates.
2 Related Work
Related benchmarks cover tool use, retrieval, memory, or harness interaction, but generally omit the joint setting of conflicting sources, evolving evidence, and silently learned preferences. ClawArena targets this missing combination while comparing support across harness-native evaluations.
- Agent benchmarks: Task-oriented agent benchmarks evaluate tool use and execution mainly in single-authority environments, without adjudication across conflicting sources.Examples include SWE-bench, AgentBench, WebArena, OSWorld, and GAIA.
- Static-evidence benchmarks: Long-context and multi-hop benchmarks stress retrieval and composition over static evidence, while ConflictQA keeps conflicting claims fixed at inference time.These settings do not test revision after new evidence arrives.
- Memory and personalization: Memory and persona benchmarks capture long-horizon recall or user-model consistency but do not jointly require conflict resolution, dynamic updates, and silent preference retention.ClawArena explicitly targets this joint setting.
- Harness-native benchmarks: Harness-native benchmarks evaluate models inside concrete agent environments, including browser, productivity, mock-service, and workspace simulations.Table 1 distinguishes support for multi-source conversations, dynamic updates, multi-turn user re-engagement, and implicit preferences.
3 ClawArena
ClawArena models persistent-assistant workspaces with hidden ground truth, conflicting evidence, staged updates, and implicit preferences. Its taxonomy, scenario layers, construction pipeline, and exact executable scoring jointly test retrieval, reasoning, revision, personalization, and workspace grounding.
- Scenario design: Each scenario combines multi-channel histories, workspace files, staged updates, and silent-exam personalization against a hidden ground truth.Agents observe noisy, partial, and sometimes contradictory traces rather than the ground truth itself.
- Evaluation formats: The benchmark includes 12 professional-context scenarios, 337 evaluation rounds, and 45 dynamic updates across multi-choice and shell-based executable checks.The two formats test complementary reasoning and workspace-grounding capabilities.
- Evaluation dimensions: The three dimensions are multi-source conflict reasoning, dynamic belief revision, and implicit personalization.They measure source adjudication, updating after contradictory evidence, and preference compliance without reminders.
- Question taxonomy: Pairwise and three-way combinations of the dimensions produce 14 recall and reasoning categories, preventing strong performance on one dimension from masking failures elsewhere.The taxonomy contains seven non-empty dimension combinations, each split into recall and reasoning variants.
- Scenario layers: Six scenario layers separate hidden truth from observable files, histories, questions, and updates, with an internal generation guide controlling formatting and noise.Layer 0 enables reliable verification while the visible layers remain noisy reflections of the same reality.
- Conflict design: Conflict design distinguishes integration from adjudication across factual, authority, non-conflict, and temporal/process relations.Non-conflict slots prevent agents from treating every disagreement as a contradiction.
- Update design: Staged updates begin with plausible incomplete narratives and later add contradictions or authoritative confirmations, scoring revision after contradictions rather than pre-emptive skepticism.Updates may shift credibility through appended messages or alter the factual record through file modifications.
- Personalization protocol: Personalization proceeds through calibration, feedback, session-implicit signals, and silent exams, with only the final stage scored.Preferences cover formatting, naming, document structure, analytical style, and communication tone.
4 Experiments
Experiments across models, frameworks, self-evolution, and error categories show that both model capability and framework design shape performance, while several failures arise from compound reasoning, revision, personalization, and output-format demands.
- Cross-model comparison: GPT-5.5 leads proprietary models on CRS (68.28) and TCR (78.34%), while Gemma-4-31B leads open-weight models on CRS (63.80) and MC (81.05%).Gemma-4-31B comes within 4.5 CRS of GPT-5.5, but its EC trails by 6.2 points (73.14 vs. 79.34).
- Cross-model comparison: 29.46 points is the model-induced CRS range, exceeding the largest framework-induced range of 23.78 points under GPT-5.5.MC and EC are only moderately correlated, indicating that workspace grounding and reasoning are partially independent.
- Skill-driven self-evolution: MetaClaw improves CRS by 0.33–1.19 without degrading TCR across four matched pairs.The gains come through increased Success Cohesion and Failure Dispersion, reducing isolated failures and long failure streaks rather than changing raw correct-answer rates.
- Error analysis: Belief-revision failures cluster around update installments requiring bridging inferences, whereas updates that merely extend prior evidence are handled reliably.Difficulty depends on whether an update forces reinterpretation of earlier claims, not monotonically on update count.
- Error analysis: Compound structural constraints produce 0/8 pass rates in a small cluster of executable-check rounds despite usually correct factual content.Implicit-preference compliance also becomes brittle under stricter successor checks, and framework choice does not compensate for model-level schema errors.
5 Conclusion
CLAWARENA evaluates persistent AI agents in evolving information environments and finds that model capability, skill overlays, and update design shape performance.
- 29 versus 24 points: model capability spans a wider CRS range than framework design across the evaluation.
- MetaClaw’s skill overlay improves Robustness without degrading accuracy across all four tested model families.
- Belief revision difficulty is governed by update specificity rather than update volume.
- Per-option diagnostics expose failure modes that aggregate scores conceal.
- A proposed extension moves from staged updates toward live environments where agents formulate queries against real-time sources.
A Data Sample Overview
CLAWARENA presents scenarios across diverse professional contexts, combining workspaces, multi-channel sources, evaluation questions, and evidence chains involving conflicting and evolving observations. The benchmark compares persistent assistant frameworks operating on a shared workspace environment.
- Scenario surface form: Each scenario tile displays a workspace, multi-channel session sources, an evaluation question, and an evidence chain leading toward hidden ground truth.The gallery provides a qualitative orientation to CLAWARENA’s data style across professional contexts.
- Framework coverage: Five AI agent frameworks receive unified conversational input and operate as persistent assistants on the same workspace environment.The frameworks differ in design while sharing the evaluation environment.
- Framework coverage: OpenClaw combines multi-channel session routing, structured memory with semantic retrieval, and configurable tool pipelines.It uses a TypeScript Agent Client Protocol architecture for session management and tool binding.
- Framework coverage: MetaClaw retrieves procedural skills distilled from prior failure trajectories and injects them into prompts without modifying model weights.MetaClaw is built on OpenClaw, isolating skill-injection effects over the same tool-binding and memory stack.
- Framework coverage: Claude Code relies on a single long-context window and in-context reasoning for code and file workspace operations.Its interaction is terminal-native and does not use external memory modules.
- Framework coverage: NanoBot and PicoClaw provide lightweight baselines using simple memory or event-driven designs that prioritize resource efficiency over retrieval sophistication.NanoBot uses MEMORY.md and HISTORY.md, while PicoClaw uses date-organized memory and under 10 MB runtime memory.
C Implementation Details
The evaluation supports proprietary, community-accessible, and self-hosted model deployments while testing exact set selection and workspace-grounded executable checks. Its Composite Reliability Score combines correctness with success cohesion, failure dispersion, and robustness across rounds.
- Model serving: Claude Haiku-4.5 uses Anthropic’s provider-native Claude Code harness because it cannot be served into OpenClaw’s tool-binding stack.The model is exercised with a Claude OAuth credential and no third-party API endpoint.
- Model serving: Two open-weight models are served locally with vLLM on four NVIDIA RTX 6000 Ada GPUs under tensor parallelism of 4.Both deployments use a 204,800-token maximum sequence length and a 0.92 GPU memory utilization target.
- Question formats: Multi-choice questions require selecting the exact correct subset from 7–9 candidate statements; partial-overlap diagnostics do not affect reported scores.Answers are enclosed in a \bbox{} token, and exact-match scoring prevents selecting all options for partial credit.
- Question formats: Executable checks are binary sandboxed shell tests of workspace-level claims, independently measuring workspace grounding rather than reasoning.Examples test file contents or consistency between timestamps in documents.
- Reliability metrics: CRS averages TCR and Robustness, combining mean correctness with behavioral consistency across macro-averaged scenarios.Robustness is the product of Success Cohesion and Failure Dispersion.
- Reliability metrics: Success Cohesion measures concentration of correct rounds into unbroken runs, while Failure Dispersion measures how evenly failures are distributed.Alternating success and failure minimizes cohesion, whereas a prolonged failure run lowers failure dispersion.
- Reliability metrics: For scores [1, 0, 1, 0, 1], TCR = 0.6 but Robustness = 0, producing CRS = 0.3.The example shows how alternating performance reduces reliability despite partial correctness.
- Reliability metrics: Although TCR is 60%, an alternating pattern yields Robustness = 0.375 and CRS = 0.488 in the worked contrasting example.Coherent success with contained failure produces a substantially higher CRS at the same TCR.
E Benchmark Statistics
CLAWARENA contains 12 scenarios and 337 rounds, with staged updates and implicit preferences distributed across balanced professional contexts. Most benchmark tokens come from workspace content, emphasizing file-grounded reasoning over dialogue history.
- Measurement procedure: Each evaluation round is a full multi-turn agent interaction whose API-call count ranges from one call to several dozen.The count varies with tool use and reasoning depth.
- Measurement procedure: Token counts use OpenAI’s cl100k_base tokenizer, with plain artifacts encoded directly and jsonl transcripts first normalized into role-content message sequences.The tokenizer serves as a cross-model reference.
- Benchmark scale: CLAWARENA spans 12 scenarios and 337 rounds, including 45 staged-update rounds and 24 rounds carrying implicit user preferences.Per-scenario counts range from 24 to 30, with a mean of 28.1 rounds.
- Token composition: OpenClaw processes 4,739,550 cumulative input tokens, while the other three compatible frameworks remain within a 0.4% band.Workspace content dominates the benchmark input under the OpenClaw layout.
- Token composition: Workspace content accounts for 90.1–90.3% of tokens across compatible frameworks, underscoring the benchmark’s emphasis on file-grounded reasoning.Session-side updates contribute a further 2.6% of tokens.
E.3 Question statistics
ClawArena combines multi-answer selection rounds with executable workspace checks, emphasizing evidence integration and state-dependent validation rather than single-answer or string-matching performance.
- Every multi-choice round requires subset selection, with a mean selection ratio of 4.46-of-6.38 that makes random guessing uninformative.
- Executable-check rounds assert exit status within a bounded timeout, while expect_stdout appears in only 1/242 rounds.
- Workspace state, rather than output-string formatting, determines pass/fail outcomes in executable checks.
- Implicit preferences occur in 24 rounds (7.1%), concentrated in P-R.
E.4 Staged updates
Staged updates test belief revision through changes to both workspace files and conversation histories, balancing accumulation and discovery while varying update intensity across rounds.
- 45 update-bearing rounds contain 83 update events affecting 25 distinct workspace files.
- Updates are roughly balanced between workspace and session targets, exercising filesystem evidence changes and newly introduced conversation turns.
- Append and new file actions are similarly balanced, preventing either pure accumulation or pure discovery from dominating revision workload.
- Per-round file mutations average 0.30 and reach a maximum of four modified files.
E.5 Per-scenario breakdown
The benchmark uses standardized six-layer scenario specifications and maintains a deliberately varied per-scenario workload, including one workspace-heavy token outlier.
- All 12 scenarios occupy a narrow 24–30-round range, while token totals vary by nearly two orders of magnitude.
- hil_s1 contributes 91.7% of benchmark tokens through staged updates to 25 files atop a 24-file baseline; the other 11 scenarios use 395,385 tokens.
- Layer 0 defines the objective timeline, truth gaps, contradiction map, bias design, and evaluation traps without being shown to evaluated systems.
- The evidence emission map distributes objective events across multiple channels so no single source contains the full truth.
- Layer 3 specifies evaluation rounds, reversals, and personalization scoring, while Layer 4 specifies staged evidence injection, actions, and runtime checks.
G Per-Option Case Studies
Figures 4–8 examine ten per-option cases across four interaction categories and show failures involving omitted evidence, relational reasoning, format compliance, revision, and synthesis.
- Figures 4–8 cover ten cases from 12 scenarios across MS-R, DU-R, P-R, and exec_check in security, clinical, HR, and e-commerce domains.
- GPT-5.1 configurations omit a 2,340-record confirmation despite absorbing five other log-derived facts, while q25 exposes five failure modes.
- The only benchmark round with 0% exact match requires compound relational statements, while most models miss a single IRB pre-registration option.
- GPT-5.1 fails filename-prefix checks, and all three NexaFlow GDPR questions have 0/8 pass rates under compound output-format requirements.
- Update-specific failures include conflating partial refunds with full restitution, while schema pass rates track documentation style.
- Four models miss a two-element termination claim, whereas seven of eight achieve perfect synthesis on a separate two-part statement.