PaperlayerView exact revision on arXiv

Source-linked AI summary

CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies

Sen Zhao, Ruiqi Kong, Zuyu Zhang, Lifeng Shen, Xinyu He, Ding Zou, Xu Zhang, Qinghua Zhang

arXiv:2609.32192v2cs.AI

TL;DR

Existing benchmarks largely separate memory evaluation from multi-agent coordination, leaving topology-conditioned sharing and isolation insufficiently measured. CoMemBench builds executable workflows and matched isolation challenges to assess both sides of that boundary. Its experiments find that broader access can improve progress while weakening isolation, with system behavior varying across topology and violation type.

  • Problem

    Existing memory and multi-agent benchmarks do not jointly measure required sharing and resistance to inapplicable information across topology-diverse executable workflows.

    Source: §1 · p. 2

  • Method

    CoMemBench evaluates 800 source-grounded workflows across four domains using declared handoffs, matched isolation challenges, and native evaluators.

    Source: §1 · p. 2

  • Results

    Broader context can improve progress while weakening isolation, and system rankings vary across topologies and artifact violations.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Completion, verified progress, required-handoff reliability, and isolation robustness remain distinct dimensions of collaborative memory behavior.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated. We call this task-conditioned scope of information a collaborative memory boundary. Workflow topology determines which intermediate artifacts are applicable to which downstream workers and when they cease to be valid, thereby providing a structural stress dimension for sharing and isolation. Existing memory benchmarks primarily evaluate retention and retrieval, whereas multi-agent benchmarks emphasize coordination and end-to-end completion, leaving topology-conditioned memory boundaries largely unmeasured. We introduce CoMemBench, an execution-grounded benchmark for collaborative memory sharing and isolation across multi-agent workflow topologies. It constructs 800 composite workflows across four domains from source-grounded dependency graphs, with node-local specifications, verifiable artifact handoffs, native evaluators, and matched isolation challenges. CoMemBench measures workflow completion, verified node progress, required-handoff reliability, isolation robustness, and token cost. Experiments reveal a sharing-isolation trade-off: broader context improves information availability but can weaken isolation, while system rankings shift across topologies and artifact violations.

1 INTRODUCTION

Collaborative memory boundaries determine which information workers need and which records must remain isolated, a distinction existing benchmarks do not jointly evaluate across topology-diverse executable workflows. CoMemBench operationalizes this distinction and finds that broader availability can improve progress without preserving isolation.

1 INTRODUCTION

  • A collaborative memory boundary separates evidence a downstream task requires from inapplicable records that must not alter its verified behavior.Examples include sibling-branch results, unverified outputs, and superseded states.Source: §1 · p. 1
  • Five dependency motifs change where workflow records are required, branch-specific, reused, partially shared, or superseded.Topology thus conditions which intermediate artifacts apply to downstream workers.Source: Figure 1 · p. 2
  • Existing benchmarks do not jointly test required information delivery and resistance to inapplicable information across topology-diverse executable workflows.End-to-end success alone cannot distinguish a missing handoff from an inappropriate record altering a consumer.Source: §1 · p. 2
  • CoMemBench evaluates required information delivery and resistance to inapplicable information across 800 source-grounded workflows in four domains.Its measures separate terminal completion, verified progress, required sharing, and isolation robustness.Source: §1 · p. 2

2 RELATED WORK

Prior work studies memory retention and selective disclosure, multi-agent organization, and executable outcomes, but does not directly evaluate topology-conditioned information boundaries across interdependent tasks.

2.1 AGENT MEMORY AND INFORMATION BOUNDARIES

  • Memory benchmarks primarily study recall or disclosure by one agent, or scope shared memory by speakers, identities, roles, and access policies.CoMemBench instead asks which artifact a consumer needs at its task state and which should remain outside its context.Source: §2.1 · pp. 2–3

2.2 MULTI-AGENT AND WORKFLOW BENCHMARKS

  • Multi-agent and workflow benchmarks vary interaction protocols or task dependencies, but do not reveal whether memory follows those dependencies selectively.End-to-end success does not establish whether information was exposed appropriately.Source: §2.2 · p. 3
  • Executable workflow benchmarks ground outcomes or collaboration, but lack controlled tests of required sharing and isolation.CoMemBench uses source-derived workflows, native validation, typed versioned handoffs, and matched clean–polluted runs.Source: §2.2 · p. 3

3 THE COMEMBENCH BENCHMARK

CoMemBench represents workflows as executable dependency graphs with local task specifications, typed handoffs, resources, and native evaluation. It constructs and certifies topology-controlled workflows with matched isolation challenges across four domains.

3 THE COMEMBENCH BENCHMARK

  • Each instance is an executable task graph with node-local specifications, typed and version-aware handoffs, resources, and a domain-native evaluator.The pipeline recovers domain graphs, extracts topology-controlled workflows, adds matched isolation challenges, and certifies execution.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 EXECUTABLE ENVIRONMENTS

  • The benchmark executes workflows in domain-native environments and records node-level receipts and terminal workflow verdicts.Examples include the official SWE-bench harness, document-level and exact-span retrieval verification, and compiled Lean proofs.Source: §3.2 · p. 4

3.3 RECOVERING DOMAIN TASK GRAPHS

  • Source task graphs are recovered from four domains using native structures, pinned assets, and dependency edges witnessed by domain evidence.Recovery covers stateful tool use, repository code change, offline retrieval, and formal mathematics; it excludes declarations transitively depending on sorryAx.Source: §3.3 · pp. 4–5

3.4 TOPOLOGY-CONTROLLED WORKFLOW AND CHALLENGE CONSTRUCTION

  • Workflow extraction selects connected, scenario-coherent subgraphs with one terminal objective, dependency closure, necessary witnessed edges, and native-evaluator coverage.Labels cover chain, split–join, reuse, overlap, and bounded revision; workflows may receive multiple labels.Source: §3.4 · p. 5
  • Each extracted graph receives a natural request and node-local tasks whose declared incoming artifacts and environment views contain the information needed for execution.Handoff contracts record semantic type, entity scope, version, and verification requirement.Source: §3.4 · p. 5
  • Matched clean and polluted runs test whether an inapplicable artifact changes the verified behavior of the same consumer.The clean run establishes that required handoffs support a solvable target.Source: §3.4 · p. 6

3.5 CERTIFICATION AND CORPUS PROFILE

  • Certification combines automated structural and evaluator checks, private Gold executions with negative mutations, and human review before workflows enter the scored split.Rejected workflows are repaired and recertified or excluded; infrastructure defects quarantine affected families rather than count as method failures.Source: §3.5 · p. 6
  • The certified corpus contains 800 clean workflows and 800 matched isolation challenges, averaging 10.10 executable task units, 17.66 required handoffs, and 4.50 topology types per workflow.The four domains use distinct native artifacts and verification, so the profile reflects graph coordination and domain-specific evaluation.Source: §3.5 · p. 6

4 EVALUATION PROTOCOL

The protocol separates workflow completion and verified node progress from required handoff reliability and isolation success, with results stratified by domain and challenge conditions.

4 EVALUATION PROTOCOL

  • VHS measures successful eligible consumers after verified required predecessors and artifact delivery, while ICS measures polluted-target success among matched pairs with successful clean targets.For the system comparison, ICS micro-pools eligible pairs across domains so each pair receives equal weight.Source: §4 · p. 6
  • SR measures clean terminal success and VNCR the macro-average verified fraction of required nodes; results are also stratified by target topology and pollution operator.Token cost is analyzed separately, and benchmark or infrastructure failures are separated from method failures.Source: §4 · p. 6

5 EXPERIMENTS

Across systems, memory mechanisms, and workflow topologies, progress, required sharing, and isolation do not reliably improve together. Greater exposure can improve availability while weakening isolation, and both rankings and token costs vary across conditions.

5.1 EXPERIMENTAL SETUP

  • The evaluation compares ten baselines across domains and three backbones, then fixes a topology-preserving executor to isolate memory mechanisms.The controlled comparison holds prompts, workers, tools, graph, evaluator, budgets, and retries fixed.Source: §5 · p. 7, §5.1 · p. 7

5.2 SYSTEM AND BACKBONE COMPARISON

  • Zero-SR configurations still reach 84.2 VNCR and 89.4 VHS, while the best SR is 8.0 in Repository Code and 18.0 in Offline Retrieval.These results distinguish verified subgraph progress and successful required handoffs from closing the terminal objective.Source: §5.2 · p. 7, Table 3 · p. 8
  • With Qwen3.8-27B, SMoA reaches 43.8/72.4 VNCR/VHS on Stateful Tool but 10.1 ICS, while Deep Agents reaches 38.1 ICS despite weak progress.With DeepSeek-V4-Flash, OMAC leads progress in Stateful Tool, Code, and Retrieval but reaches 4.4 ICS, below Peer Review's 17.1; rankings shift again under GPT-6-Luna.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 MEMORY MECHANISM COMPARISON

  • FULL CONTEXT raises Stateful Tool VNCR from 74.6 to 86.9 and Code SR from 0.0 to 16.0, but lowers ICS from 97.1 to 72.4 and from 89.4 to 71.9.SHARED FLAT RAG reaches 90.8 Code VHS but only 73.9 ICS.Source: §5.3 · p. 7, Table 4 · p. 8
  • A-Mem reaches 18.4 Code SR but only 63.9 ICS, while PlugMem reaches 91.3 Code ICS with 0.0 SR and 52.9 Retrieval ICS.Formal Mathematics remains unsolved at workflow level; structured memory approaches do not consistently preserve required access and isolation.Source: §5.3 · pp. 7–8

5.4 TOPOLOGY AND ISOLATION ANALYSIS

  • Chain consumers have the highest VHS for every system, but EvoMAS reaches 72.7 ICS on split–join and 14.3 on reuse, while Deep Agents reaches 80.0 on overlap and 28.6 on reuse.Topology and violation strata describe complementary boundary pressures, not causal topology–operator comparisons.Source: §5.4 · pp. 8–9, Table 5 · p. 9

5.5 COST ANALYSIS

  • At equal verifier-accepted progress, Deep Agents averages 0.31M tokens per task, versus 8.01M for OMAC and 3.61M for Peer Review.Compared with LOCAL-ONLY at 489k tokens, FULL CONTEXT uses 672k and SHARED FLAT RAG 621k without consistently improving sharing and isolation; PlugMem costs 521k and varies by domain.Source: §5.5 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The conclusion frames collaborative memory as selective, task- and state-conditioned exposure across workflow consumers. CoMemBench evaluates that boundary through executable workflows, and its findings show that broader access can improve progress without preserving isolation.

6 CONCLUSION

  • Effective collaborative memory requires selective, task- and state-conditioned exposure rather than simply more context.The benchmark separates completion, verified progress, required-handoff reliability, and isolation, with rankings varying by topology and violation.Source: §6 · p. 10
  • The authors report that they designed the study, verified the data and results, and take responsibility for the paper and released artifacts.Generative AI tools were used for language and presentation editing and routine implementation assistance.Source: AI USE STATEMENT · p. 10
  • The authors will release benchmark data, clean–polluted pair specifications, schemas, evaluation code, prompts, and environment configurations.Run records include system and model versions, decoding and budget settings, source and evaluator revisions, and asset checksums.Source: REPRODUCIBILITY STATEMENT · p. 10

A BENCHMARK SCHEMA AND RUNTIME PROTOCOL

The runtime separates reusable workflow definitions from clean and polluted benchmark instances. Execution records expose declared inputs, artifacts, verification, handoffs, and consumer exposure so behavior can be replayed without assuming a shared internal memory architecture.

A BENCHMARK SCHEMA AND RUNTIME PROTOCOL

  • A workflow_family stores source references, tasks, dependency edges, topology witnesses, execution contract, and split; instances bind a family to a clean control or matched polluted variant.The public package contains 800 families, 1,600 instances, and 8,975 node-local contexts.Source: A BENCHMARK SCHEMA AND RUNTIME PROTOCOL · p. 13
  • Each worker receives a node_view with its task, authorized context, incoming artifacts, handoff contract, output type, and capability token.Only a verified receipt permits a required artifact to satisfy a downstream dependency; the harness records proposed and actually exposed artifacts.Source: A BENCHMARK SCHEMA AND RUNTIME PROTOCOL · p. 13
  • Replay follows manifest-fixed initialization, node views, actions, receipts, artifacts, handoffs, consumed-artifact identifiers, and pollution events.Schema validation, content hashes, and receipt signatures make exposure traces replayable without treating private reasoning or memory-store contents as ground truth.Source: A BENCHMARK SCHEMA AND RUNTIME PROTOCOL · p. 14

B DATA SOURCES AND CONSTRUCTION DETAILS

Workflows are extracted as connected, source-grounded subgraphs with closed dependencies and native evaluators, then reviewed and certified through machine gates and replay. Domain-specific execution substrates preserve their native interfaces while emitting shared benchmark records.

B DATA SOURCES AND CONSTRUCTION DETAILS

  • Each workflow is a connected source subgraph with one terminal objective, complete ancestor closure, evidenced dependencies, and a native evaluator, without joining unrelated cases or domains.Node tasks expose local information and declared predecessors while preserving the source task's executable semantics.Source: B DATA SOURCES AND CONSTRUCTION DETAILS · p. 14

C QUALITY ASSURANCE AND AUDIT PROTOCOL

  • All 800 families pass graph and execution-contract audits, and replay accepts known-good traces while rejecting targeted negative traces.All 800 matched isolation plans pass the candidate gate; the final release audit reports no schema, graph, oracle-replay, source-invariant, or public leakage findings.Source: C QUALITY ASSURANCE AND AUDIT PROTOCOL · pp. 14–15

D.1 MCP APPLICATION ENVIRONMENT

  • Before each workflow, the harness materializes state in an isolated workspace, database, and service namespace, then starts only servers allowed by the node capability contract.A task with an unavailable declared server is excluded from method-level scoring.Source: §D.1 · p. 15

D.2 NON-MCP NATIVE SUBSTRATES

  • Code tasks use the official SWE-bench evaluation entrypoint, mathematical proofs compile in isolated Lean 4.7 copies, and retrieval uses immutable corpora with document and exact-span verification.All three substrates emit the canonical action, artifact, receipt, and failure records used by the MCP path.Source: §D.2 · pp. 15–16

E BASELINE INSTANTIATIONS AND RUN MANIFESTS

Baseline implementations differ in how they expose or retrieve workflow artifacts, while a shared adapter supplies node-local tasks and records traces. Immutable run manifests document each evaluated configuration.

E BASELINE INSTANTIATIONS AND RUN MANIFESTS

  • The adapter supplies node-local tasks, authorized actions, and declared incoming artifacts, then records traces without selecting answers, routing, schedules, or memory contents.Table 8 records the exact implementation evaluated for each system.Source: E BASELINE INSTANTIATIONS AND RUN… · p. 16, Table 8 · p. 16
  • Six memory mechanisms range from worker-local state and full context to retrieval systems and linked-note or graph-based memory.External stores start empty, and only the memory policy changes across rows.Source: E BASELINE INSTANTIATIONS AND RUN… · p. 16
  • Each immutable run manifest records revisions, model and decoding settings, budgets, configuration hashes, memory policy, split, environment and evaluator hashes, seed, and trace schema.This separates each baseline’s organization from the common execution and scoring substrate.Source: E BASELINE INSTANTIATIONS AND RUN… · p. 16

F METRIC DEFINITIONS AND SCORING DETAILS

The scoring definitions distinguish terminal success, verified node progress, required handoff reliability, and isolation robustness. Token accounting includes execution and memory-related costs, with metric eligibility and failure denominators specified separately.

F METRIC DEFINITIONS AND SCORING DETAILS

  • VNCR averages, across queries, the fraction of canonical required nodes with verified completion; retries count once and method-private nodes are excluded.Terminal success is separately defined as the fraction of queries with terminal success.Source: F METRIC DEFINITIONS AND SCORING DETAILS · p. 16
  • VHS is the share of eligible consumers whose required predecessor artifacts are exposed at the declared boundary, current, receipt-valid, consumed, and followed by verifier success.Unsupported slices are marked “–”.Source: F METRIC DEFINITIONS AND SCORING DETAILS · p. 16
  • ICS measures how often a target consumer succeeding in a clean isolation pair also succeeds in its matched polluted variant.Eligibility requires clean target success, not full clean-workflow terminal success; only one evaluator-hidden inapplicable artifact differs between variants.Source: F METRIC DEFINITIONS AND SCORING DETAILS · p. 17
  • Token accounting covers execution, coordination, retrieval context, recovery or repair, and meta-level search, with several distribution and per-successful-workflow statistics reported.Benchmark and infrastructure failures are excluded from method-failure denominators.Source: F METRIC DEFINITIONS AND SCORING DETAILS · p. 17

G ADDITIONAL EVALUATION DETAILS

Additional evaluation details break down isolation by pollution operator and document token accounting and failure attribution. Operator-specific results show that robustness varies across artifact violations.

G ADDITIONAL EVALUATION DETAILS

  • EvoMAS is comparatively robust to sibling crosswires but remains vulnerable to unverified results and other observed violations; Deep Agents performs strongly on wrongentity challenges but less reliably on unverified results.Isolation robustness does not transfer uniformly between artifact violations; Table 9 presents the operator breakdown.Source: Table 9 · p. 17, G ADDITIONAL EVALUATION DETAILS · p. 17
  • Table 10 reports aggregate token statistics in thousands, covering model calls for execution, coordination, recovery, and memory operations with complete usage telemetry.The same aggregate statistics are represented in Figure 4.Source: G ADDITIONAL EVALUATION DETAILS · p. 17, Table 10 · p. 17
  • Only method failures enter method-performance denominators, while benchmark and infrastructure failures remain separately auditable.Table 11 states the run-outcome attribution used before metric aggregation.Source: G ADDITIONAL EVALUATION DETAILS · p. 18, Table 11 · p. 18

Related papers

Loading 2609.32192v2…