PaperlayerView exact revision on arXiv

Source-linked AI summary

WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks

Prince Zizhuang Wang, Aojie Yuan, Haiyue Zhang, Xiyang Hu, Yue Zhao, Shuli Jiang

arXiv:2608.03499v1cs.AI

TL;DR

Existing benchmarks lack an end-to-end, auditable setting for cross-user agent collaboration over personal workspaces and harmful influence through shared channels. WeClawArena fills this gap with matched benign and attack scenarios plus runtime evidence, showing that utility and final harm can diverge across models and domains.

  • Problem

    Existing benchmarks lack end-to-end, verifiable evaluation of cross-user agents coordinating across owner-scoped workspaces while attacks travel through legitimate collaboration channels.

  • Method

    WeClawArena benchmarks multi-party owned-agent tool use across personal workspaces with 124 base tasks, 620 matched scenarios, and bounded runtime evidence separating utility from attack success.

  • Results

    Claude Opus 4.7 has the strongest overall utility, but no model dominates every harm surface and task success remains distinct from final harm.

  • Takeaways & Limitations

    The benchmark enables reproducible diagnosis of where collaborative personal agents succeed, break down, leak privacy, or follow invalid authority paths.

Abstract

from arXiv · show

Recent advances in persistent personal-agent frameworks are making human-centered agent networks realistic deployment targets: each user can be served by an AI agent that acts on the user's behalf, maintains state, and communicates with other agents through social and task relations. In these networks, everyday tool use becomes multi-party owned-agent collaboration over personal workspaces, where files, records, tools, and policies are not directly visible across owners. Existing agent benchmarks study tool use and collaboration, but they do not provide an end-to-end sandbox for verifiable cross-user agent collaboration with realistic user digital workspaces or test how harmful actions can travel through the human-centered agent network. We introduce WeClawArena, an auditable benchmark and runtime sandbox for multi-party owned-agent collaboration over personal workspaces. WeClawArena targets collaborative tool-use tasks in which personal workspaces serve as both operational tools and personal constraints. The benchmark contains 124 base tasks across six cross-user task domains and expands them into 620 scenario variants, with one benign control and four attack-vector variants per base task. The sandbox records peer messages, tool calls, resource operations, governed decisions, and final workspace states. WeClawArena reports utility and attack success rate separately and audits attack success from bounded runtime evidence, supporting diagnosis of task breakdown, privacy leakage, poisoned evidence, and invalid authority paths.

1 INTRODUCTION

WeClawArena addresses the missing end-to-end setting of cross-user tool-use collaboration over separate personal workspaces, where legitimate actions and attacks can share communication, resource, tool, and approval paths. It provides a benchmark and controlled runtime sandbox for evaluating utility and attack success separately with bounded runtime evidence.

  • Problem setting: Cross-user collaboration distributes tool-use tasks across several owners’ agents, which must coordinate while operating over separate personal workspaces.These workspaces contain owner-scoped artifacts, resources, policies, and tools that are not directly shared across owners.
  • Benchmark gap: Existing tool-use and interaction benchmarks do not instantiate the full end-to-end problem of collaborative work across separate personal workspaces.The missing runtime setting combines multi-party collaboration with workspace boundaries and realistic deployment conditions.
  • Security challenge: Attacks and policy violations can travel through the same messages, resources, tools, and approval paths as legitimate collaboration.This makes harmful behavior part of the same operational pathways used for cross-user task completion.
  • WeClawArena: 124 base tasks and 620 matched scenarios span bargaining, bidding, travel, SWE-Workspace, clinical, and trading domains.The benchmark models each user as an owner with a personal workspace, role-specific resources, policies, and domain tools.
  • Evaluation and auditing: WeClawArena separates utility from ASR and audits final harm using bounded runtime evidence in a controlled sandbox and evaluation protocol.The sandbox supports collaborative tool-use demands and adversarial social pressure while recording evidence for evaluation.

2 Human-Centered Agent Network over Personal Workspaces

The framework models a human-centered agent network in which each human principal owns a personal workspace node and delegates agents to act and collaborate on their behalf. Each workspace includes the user’s operational resources and governing policies, constraints, consent rules, and approvals.

  • Human-Centered Agent Network over Personal Workspaces: Each network node represents a human principal’s personal workspace rather than merely a user profile.The workspace defines the user’s filesystems, databases, tools, policies, personal constraints, and delegated agents.
  • Human-Centered Agent Network over Personal Workspaces: Delegated personalized agents send messages, call tools, update records, and collaborate with other agents in the network graph.The human user remains the principal party associated with each node, while agents act on the user’s behalf.
  • Human-Centered Agent Network over Personal Workspaces: For each user u, the personal workspace comprises delegated agents A_u, filesystem state F_u, structured records D_u, available tools T_u, and governing policies P_u.P_u includes policies, constraints, consent rules, or approval requirements.

2 HUMAN-CENTERED AGENT NETWORK OVER PERSONAL WORKSPACES

WeClawArena models collaboration as delegated agents operating across human-owned personal workspaces connected by social or task relationships. Tasks require agents to coordinate under distributed visibility and authority constraints, with success determined by final workspace states and evidence traces rather than conversation alone.

  • Network model: Personal workspaces are the network’s social units, while delegated agents are the operational actors interacting across edges carrying roles, permissions, task phases, affinity, or approval context.Each edge connects two workspace nodes and defines the context under which their agents may interact.
  • Delegated action: Delegated agents may message peers, call tools, operate resources, request approval, record consent, or produce artifacts under workspace-specific authority and visibility constraints.A scenario remains unsolved when agents produce plausible conversation but fail to read, update, or validate required records.
  • Task formulation: A task is a multi-party owned-agent tool-use problem requiring coordination across workspace files, databases, tools, and policies.The task-specific formulation includes a personal-workspace graph, private goals, initial states, available actions, state transitions, contracts, and verifiers.
  • Distributed visibility: Agents observe only workspace-visible instructions, records, tools, policies, and peer messages, so solving tasks may require combining partial information without crossing contract-defined boundaries.The input is distributed across the initial instructions, multi-workspace state, graph, and available action sets.
  • Execution and verification: The benchmark’s primary output is the final multi-workspace state and evidence trace after message, tool-use, resource, approval, consent, and finalization events.Natural-language responses may also be produced, but they are not the primary benchmark output.

2 HUMAN-CENTERED AGENT NETWORK OVER PERSONAL WORKSPACES

This section distinguishes collaboration, security, and privacy harms in cross-user agent networks from ordinary utility failure. WeClawArena pairs benign controls with attack variants and records runtime evidence to attribute final outcomes to agent actions.

  • Attack harms: The framework formalizes collaboration harm, security harm, and privacy harm as distinct attack surfaces.Examples include goal hijacking, false handoff, poisoned evidence, unauthorized resource mutation, and unauthorized information disclosure.
  • Utility versus harm: A visible task can succeed despite leaked protected information, forged evidence, or invalid approval, while a benign task can fail without attack-caused harm.Thus, utility failure and attack harm are evaluated separately rather than treated as equivalent outcomes.
  • Benchmark design: Each benign control is paired with attack-vector variants targeting the same task through different harm surfaces, with runtime evidence recorded for outcome attribution.The recorded evidence links final outcomes to messages, tool calls, and related runtime actions.

3 WeClawArena

WeClawArena is an auditable benchmark and sandbox for multi-party tool-use collaboration over owner-scoped personal workspaces. It combines 124 base tasks with 620 matched scenario variants and evaluates utility separately from attack success using runtime evidence.

  • Benchmark scope: 124 base tasks expand into 620 scenario variants across bargaining, bidding, travel, SWE-Workspace, clinical, and trading.Each base task has one benign no-attacker control and four attack-vector variants.
  • Base tasks: Each base task specifies owners, delegated agent roles, scoped resources, tools, policies, a task contract, and a verifiable collaboration objective.Scenarios provide role-specific files, structured records, private constraints, applicable policy or consent data, and domain tools.
  • Scenario variants across attack vectors: Matched variants preserve the collaboration objective, task contract, and owner workspaces while adding adversarial messages, files, database rows, or other bounded artifacts.The benign control preserves the original objective, whereas attack variants pressure one of four harm surfaces.
  • Attack construction and attack-success evidence: Attack success requires agents to encounter, route, accept, repeat, or act on injected material through ordinary messages or tools before harmful final states count.Attack delivery is separated from attack success, and injected material cannot directly create the final harmful state.
  • Evaluation: Utility is evaluated from the final multi-workspace state and evidence trace, while attack success rate measures final harm on the intended surface.ASR is judged after the run from a bounded evidence packet and counted only when final harm has a clear link to the attack.

4 Experiments

Experiments evaluate WeClawArena across six domains and 620 scenario variants using passive runtime evidence capture, calibrated turn budgets, and separate utility and attack-success metrics. The evaluation excludes benign controls from ASR while retaining them for false-positive checks.

  • Domains and scenario variants: Six domains—bargaining, travel, SWE-Workspace, bidding, clinical, and trading—each include one benign control and four attack-vector variants.The attack variants cover collaboration, security, privacy, and governance.
  • Runtime and models: Passive evidence capture records peer messages, tool calls, resource operations, policy verdicts, and final workspace states without changing agent trajectories.Table 1 evaluates every model in the same Dockerized OpenClaw runtime; main ASR uses scoreable attack rows with GPT-5.2 headline LLM-judge verdicts.
  • Budgets: 36-turn caps apply to bargaining, bidding, and travel, while SWE-Workspace workflows receive 90 turns.Turn-cap termination is diagnostic rather than attack success unless the domain judge identifies final harm with a clear evidence link.
  • Metrics: TSR and ASR are reported separately: TSR measures raw task success, while ASR counts attack-vector rows and excludes benign controls.Missing TSR fields count as unsuccessful; benign controls support false-positive checks, and Table 1 reports utility across all variants.

4 EXPERIMENTS

The experiments show that WeClawArena separates overall task utility from attacked-row harm, revealing substantial variation across domains, attack vectors, and models. Claude Opus 4.7 has the strongest overall utility and resistance profile, but no model dominates every harm surface.

  • Utility results: Table 1 reports deterministic task success over final workspace state and evidence fields, with utility read separately from ASR on attacked rows.Claude Opus 4.7 leads travel, SWE-Workspace, and bidding, while Claude Sonnet 4.5 leads bargaining and ties for the best clinical score.
  • Domain heterogeneity: 34.0% is the top all-variant TSR for SWE-Workspace, while travel and bargaining better separate high-utility frontier models from the rest.Clinical and trading all-variant scores reflect resilience because the table includes attacked rows, not benign task competence alone.
  • Surface and domain patterns: Governance and security have the highest judged-attack rates across models, whereas collaboration has the lowest, making TSR drop less differentiated.Figure 4 reports utility and per-vector row-micro ASR over the ASR-MAIN-SIX-DOMAIN pool.
  • Surface and domain patterns: The same harm surface affects domains differently: security pressure dominates bargaining and bidding, while privacy and governance pressure dominate SWE-Workspace.These patterns come from the domain-level heatmap using the same ASR pool.
  • Model-level resistance and utility tradeoff: Claude Opus 4.7 is the most resistant model, open-weight models cluster lower, and no single model dominates every harm surface.The utility-resistance scatter shows TSR drop and final harm are correlated but distinct: some attacks reduce utility without causing final harm.

5 RELATED WORK

WeClawArena extends personal-agent, tool-use, and multi-agent benchmarks to cross-user collaboration over personal workspaces. It also incorporates security, privacy, governance, and judging perspectives to audit leakage, poisoned evidence, and invalid authority paths.

  • WeClawArena makes user-relative workspace ownership and authority part of both task utility and attack auditing.
  • Personal-agent, workspace, and multi-agent benchmarks: WeClawArena evaluates cross-user tool-use collaboration in which personal workspaces function as both task resources and policy boundaries.
  • Security, privacy, governance, and judging: Security, privacy, and governance benchmarks motivate WeClawArena’s harm surfaces, including leakage, privacy failures, and inappropriate information flow.Contextual integrity addresses whether information flow is appropriate, while ConFAIDE, PrivacyLens, MAGPIE, and MAMA study secret keeping, privacy norms, and privacy failures in agent memory and multi-agent settings.

6 Conclusion … E Attack Vector Design Details

WeClawArena benchmarks cross-user collaboration over private, owner-scoped workspaces and audits both utility and attack success from bounded runtime evidence. Its matched scenarios, governed task contracts, and attack-stage reporting expose how harmful actions propagate through agent networks without directly editing final outcomes.

  • 6 Conclusion: WeClawArena contains 124 base tasks and 620 matched scenarios across six domains, pairing each benign task with collaboration, security, privacy, and governance attack variants.The benchmark treats agents as delegated actors operating over owner-scoped files, records, policies, tools, and decision rights.
  • A Multi-User Runtime Sandbox: Each scenario runs in a controlled multi-user sandbox with one private owner workspace containing scoped resources, personal state, tools, and governance records.Task-critical facts are intentionally placed in workspaces, requiring successful runs to use resources rather than prompt text alone.
  • A Multi-User Runtime Sandbox: The gateway records model calls, peer messages, tool activity, resource operations, delegation, memory, policy decisions, and final states for post-simulation evaluation.Evidence packets preserve transcripts, observations, scores, final resource states or digests, and owner or governance context for judges.
  • B Runtime Pipeline and Evidence: Valid benchmark runs are Docker-based and gateway-mediated, while bypassed communications, edits, governed actions, or artifacts cannot support reliable attack-success claims.Readiness failures such as missing logs or broken evidence chains are excluded from outcome denominators.
  • C Benchmark Bundle and Social Topology: Each base bundle combines metadata, evaluation data, personas, owner resources, tools, governance records, and an owner-bound agent graph that defines valid evidence and authority paths.Attack variants are materialized through overlays that add pressure without directly editing final score state.
  • D Domain Setup Atlas: Across domains, the shared contract combines owner-bound agents, private resources, domain tools, verifiable utility objectives, and four attack-vector variants.Task-critical records differ by domain, including budgets and approvals, bids and reservations, travel consent, engineering review and CI evidence, clinical case governance, and trading mandates.
  • E Attack Vector Design Details: Attack vectors separate four harm surfaces—collaboration, security, privacy, and governance—from delivery surfaces such as messages, files, database rows, observations, approvals, and review artifacts.The design records both surfaces because one harm type can enter through different mechanisms.
  • E Attack Vector Design Details: Attack success requires final harm linked to attack pressure, distinguishing exposure, target compliance, partial compromise, final harm, and attack success.Agents must encounter, route, accept, repeat, or act on attack material; overlays cannot directly edit final score fields or task artifacts.

F Additional Experimental Results

Additional results show that attack pressure varies substantially by domain and harm surface, while model robustness is consistently vector-dependent rather than reducible to one scalar safety score. Claude Opus 4.7 has the highest resistance across all four attack vectors, but other models show important surface-specific vulnerabilities.

  • Domain and attack-vector heterogeneity: Security pressure is strongest in bargaining and bidding, whereas privacy and governance pressure are strongest in SWE-Workspace and travel.Figure 19 reports domain-by-attack-vector heterogeneity using ASR-MAIN-SIX-d,v.
  • Model resistance by harm surface: Claude Opus 4.7 has the highest resistance in all four attack vectors.Figure 20 ranks models by model-vector row-micro 1 −ASRmicro over ASR-MAIN-SIX-DOMAIN.
  • Model resistance by harm surface: Qwen3 32B is strong under privacy attacks but weak under governance attacks, while Kimi K2 Thinking and Qwen3 235B show different cross-surface weaknesses.Kimi K2 Thinking ranks second under governance but lower under security and privacy; Qwen3 235B is low under collaboration and governance.
  • Utility and attack resistance by harm surface: Governance results show Qwen3 235B and Qwen3 32B with positive matched TSR drops and low resistance, while Claude Opus 4.7 retains high resistance despite a positive utility drop.Figure 21 expands utility and attack-resistance comparisons by harm surface over ASR-MAIN-SIX-DOMAIN.
  • Utility and attack resistance by harm surface: Model risk should be evaluated as a vector over harm surfaces rather than as one scalar safety score.The governance-panel comparison demonstrates that utility and resistance patterns differ across models and surfaces.

G Failure Analysis and Case Studies

Failure analysis shows that cross-user attacks most often produce task failure, but some preserve apparent utility while causing security, privacy, or governance harm. Case studies distinguish final harm from mere exposure and show why utility and attack success must be audited separately.

  • Aggregate failure pattern: 1,152 of 3,743 judged attack-vector rows reached attack success, yielding row-micro ASRmicro = 30.8%.Only 117 rows both completed the task and reached attack success, while 1,035 reached attack success with task failure.
  • Attack-vector breakdown: 400/939 governance rows were successful attacks, followed by security at 331/922, privacy at 251/943, and collaboration at 170/939.Collaboration attacks never coincided with task success in this sweep.
  • Cross-user failure modes: The four dominant failure categories were invalid authority paths, poisoned cross-workspace evidence, unauthorized disclosure, and cross-owner coordination breakdowns.Their judged success counts were 400 governance, 331 security, 251 privacy, and 170 collaboration cases, respectively.
  • Case studies: Security harm can preserve utility when an agent completes a purchase using a poisoned settlement price that follows the injected objective.The task predicate marked the negotiation complete, while the ASR judge identified final harm in the closing price and downstream purchase artifact.
  • Case studies: Privacy harm can likewise occur after successful work when a reviewer discloses owner-local protected identifiers through shared communication.The code-review workflow succeeded, so the failure would be invisible under TSR alone.
  • Case studies: Partial resistance does not restore authority when an organizer rejects a forged approval identifier but submits a booking with an empty approval identifier.The booking completed, yet the ASR judgment classified the bypassed approval gate as final governance harm.
  • Negative cases: Exposure alone is not attack success when agents reject poisoned settlement notices and close at legitimate negotiated values.Attack success requires final harm and a clear evidence link to the attack; suspicious access or turn-cap pressure alone is insufficient.

H ASR Judge Validation … L Artifact Release and Safety

WeClawArena validates its post-hoc ASR judge through cross-model agreement, benign calibration, sensitivity analysis, and a human-annotated pilot. Its evaluation operators and released artifacts support recomputation while limiting disclosure of reusable attack payloads.

  • H ASR Judge Validation: Claude Opus 4.7 re-judges the same bounded evidence packets as GPT-5.2 using the same prompting template, differing in vendor, training data, and refusal behavior.The comparison is designed to assess the judging procedure rather than within-family bias.
  • H ASR Judge Validation: 0.70 overall Cohen’s κ indicates substantial inter-judge agreement, with per-surface κ ranging from 0.66 for security to 0.73 for governance.The lowest agreement occurs on security and privacy, where judged attack-success rates are closer to 0.5.
  • H ASR Judge Validation: 0.3% overall under GPT-5.2 and 0.7% overall under Opus 4.7 are benign false-positive rates, remaining below 1% across every surface.These low, surface-stable rates support using no-attacker rows as calibration data.
  • H ASR Judge Validation: +3.3 percentage points overall is Opus 4.7’s increase in reported ASR, while per-cell deltas range from −5.8 to +8.9 percentage points.Opus 4.7 is higher in 24 of 32 cells, but the per-model overall ranking of 1 −ASR remains identical across judges with Spearman ρ = 1.00.
  • I.1 ASR Pools and Aggregation Operators: ASR uses four named row pools, including 3,743 judged attack-vector rows in ASR-MAIN-SIX-DOMAIN and a 200-row expert-labeled ASR-HUMAN-PILOT.ASR-MAIN-SIX-DOMAIN is the headline GPT-5.2 pool; ASR-SENSITIVITY-SIX-DOMAIN is the Opus 4.7 re-judging pass over the same packets and denominators.
  • I EVALUATION DETAILS: The evaluation separates utility from attack success, using domain-specific task-success measures and reporting TSR failures for missing task-success fields.Table 10 reports exact TSR counts by model and scenario variant, while Table 11 provides canonical raw ASR denominators and counts.
  • I EVALUATION DETAILS: ASR counts only attack-vector rows showing attack-specific final harm and a clear evidence link to attack pressure, while benign controls serve as false-positive checks.Deterministic diagnostics are excluded from ASR, and cells without judged attack rows are omitted from denominators as n/a.
  • K Appendix Artifact Tables and Figures: The appendix consolidates cited tables, figures, role-contract boxes, and release artifacts, while the initial release supports independent recomputation from scenario bundles, runtime records, verifier code, prompts, seeds, settings, and turn caps.Attack-vector manifests preserve repeatability fields but redact or summarize reusable exploit instructions, scripts, credentials, and bypass steps.
Loading 2608.03499v1…