PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace

Seonglae Cho, Donghyun Lee

arXiv:2608.23740v1cs.AIcs.SE

TL;DR

Existing multi-agent coding systems often serialize agents or pool independent samples without explicit coordination, leaving open whether coordinated concurrency improves coding outcomes at matched compute. AgentRoom combines a CRDT-merged shared filesystem with MCP-based claims, broadcasts, and state management, and evaluates it across coding models and backend tasks. The reported comparisons place AgentRoom above parallel-merge and partial variants, while the paper concludes that explicit coordination—not parallelism or CRDT merge alone—is the identified contributor within its setting.

  • Problem

    Existing systems predominantly use sequential turn-taking, while implicit CRDT coordination has shown mixed results from 21% speedup to 39% slowdown across tasks.

  • Method

    AgentRoom provides concurrent agents a CRDT-merged shared workspace with explicit MCP tools for file claims, broadcasts, and peer state.

  • Results

    At matched compute, AgentRoom ranks above parallel-merge and each tested partial case, with the paper identifying coordination as the key contributor in its bundle probe.

  • Takeaways & Limitations

    Within the evaluated setting, explicit room-level state management distinguishes coordinated concurrency from parallelism or CRDT merge alone.

  • Takeaways & Limitations

    Headline cells share one Express.js/TypeScript runtime; Python and Rust checks are bounded portability spot checks, while Gemini is underpowered and Codex-mini is excluded.

Abstract

from arXiv · show

Concurrent multi-agent coding promises division of labor across modules, robustness through redundancy, and parallel exploration at the natural granularity of multi-file projects. Realtime collaborative editing protocols solve this coordination problem for human teams via Conflict-free Replicated Data Types (CRDTs), but the LLMs underneath generate one token at a time and existing multi-agent coding systems inherit this serial limit: they either sequence agents through phase handoffs or pool independent samples without coordination, and a single agent abandons up to half of hard tasks with a one-file stub-and-exit. AgentRoom is a realtime collaborative editing protocol for concurrent coding agents. Its runtime layer exposes file-level claim, status, and broadcast as MCP tools on a CRDT-merged shared filesystem. Five frontier coding-CLI models ran four backend coding tasks, with cross-language checks in Python DevBench and Rust+axum. For CLI-stable models, AgentRoom with 2 agents abandons fewer tasks than Solo and has less run-to-run variation. At matched-compute, one positive mean LLM-judge contrast puts AgentRoom over parallel-merge. The other contrast, a bundle probe, puts full AgentRoom above each partial case: an ordering rather than a percentage split. Coordination, not parallelism or CRDT-merge, bears the load.

1 Introduction

Existing multi-agent coding systems commonly serialize agents through turn-taking or use implicit CRDT coordination with mixed results. AgentRoom addresses whether explicit coordination on a CRDT-merged shared workspace can outperform both alternatives at matched compute.

  • Most multi-agent coding systems use sequential design, implementation, and review, with each agent waiting for the previous one.
  • Implicit concurrent CRDT coordination has produced mixed task-level results, ranging from 21% speedup to 39% slowdown.
  • AgentRoom exposes file-level claims, agent status, and an append-only broadcast log through MCP tools on a CRDT-merged shared filesystem.
  • Unlike CodeCRDT, AgentRoom uses an explicit room and advisory protocol without pre-assigned roles or orchestrated handoffs.

2 Method

AgentRoom combines a CRDT-backed shared workspace with explicit coordination tools and advisory workflow rules. The method motivates coordination by distinguishing byte-level merge from intent-level compatibility and evaluates matched-compute alternatives with multiple quality scorers.

  • AgentRoom architecture: AgentRoom is an integrated primitive combining a CRDT-mediated shared filesystem, concurrent agents, and a coordination interface.
  • AgentRoom architecture: The architecture uses a CRDT-mediated workspace in which concurrent character-level edits remain visible to all agents.
  • AgentRoom architecture: The MCP server exposes claim, release, broadcast, state, and read operations for shared-room coordination.
  • Coordination interface: room_claim(path) atomically assigns file ownership and rejects claims held by another agent, while broadcast/read maintain an append-only message log.
  • CRDT substrate: CRDT strong eventual consistency prevents data loss under concurrent edits but does not ensure semantic compatibility between incompatible changes.
  • Collision rate: At N=2 and K_t≈5, the lower-bound collision probability is ≈0.20, while the claim primitive drives collision rate to zero at O(1) per-edit cost.
  • Workflow: The advisory workflow asks agents to inspect room state, claim files, revise after conflicts, write claimed files, poll for updates, and broadcast completion.
  • Evaluation: The study compares Solo, Shared-only, and Parallel-merge with AgentRoom at matched compute and evaluates quality using LLM-judge, regex, and AST scorers.

3 Experiments

Experiments use vendor CLI coding models across matched-compute conditions and report quality, abandonment, test, and coordination measures. The design includes validity filters, confidence intervals, and evidence tiers for interpreting results.

  • Models: Five frontier coding models from three providers were evaluated, with headline cells using four CLI-stable models and excluding GPT-5.4-mini after a concurrent-MCP CLI crash.
  • Conditions: AgentRoom launches N agents within 1 s on a shared workspace and room server, while alternatives remove collaboration tools or use post-hoc file union.
  • Validity: Runs lasting at least 30 s were pooled, excluding only authentication and network failures that exited earlier.
  • Statistical reporting: Headline cells use bootstrap 95% confidence intervals with 10,000 resamples, while underpowered cells are flagged as exploratory.
  • Measures: The study measures continuous quality, emergence, passing-test count, room-message count, and CRDT-conflict count across three evidence tiers.

4 Results

AgentRoom’s strongest results are lower run-to-run variance and fewer lone-agent abandonments across CLI-stable models, while matched-compute ablations favor explicit coordination over parallel merging. Scaling results identify two agents as the operational quality sweet spot, with coordination overhead increasing beyond three agents.

  • Tier I: Variance and Abandonment: ∼30–45% lower run-to-run standard deviation under AgentRoom ×2 replicated across all three CLI-stable models and both Tier I tasks.On T4 Sonnet, σ fell from 0.23 to 0.14; on T5 Sonnet, from 0.23 to 0.13.
  • Tier I: Variance and Abandonment: 34% versus 6%: T4 Haiku’s 1-file abandonment rate fell from 12/35 Solo attempts to 1/17 AgentRoom ×2 attempts.The reduction was significant under Fisher’s exact one-sided test (p=0.025).
  • Tier II.a: Six-Condition Ablation: +0.213 LLM-judge composite: AgentRoom exceeded parallel-merge at matched compute, with Welch’s t=3.35 and p=0.003.The comparison isolates concurrent coding with explicit CRDT+MCP coordination from concurrent coding without coordination.
  • Tier II.a: Six-Condition Ablation: The six-condition ablation formed a monotonic ordering from ChatDev-style sequential coding at 0.333 to AgentRoom at 0.669.The intermediate conditions were parallel-merge 0.456, Solo 0.544, shared-only 0.575, and shared+collab without MCP 0.588.
  • Scaling and Heterogeneity: AgentRoom quality peaked at ×2, while mean tests passing peaked at ×3; message traffic rose super-linearly past ×3.The reported quality means were 0.544, 0.669, 0.553, and 0.489 for ×1 through ×4.
  • Backend Comparison: The MCP-chat baseline failed 1 of 2 T5 runs, whereas CRDT-state AgentRoom succeeded in 7 of 7 runs.The qualitative contrast was attributed to room_claim() returning a system-level conflict error when simultaneous file claims occurred.

5 Discussion

AgentRoom’s results indicate that explicit coordination, rather than concurrency or CRDT merging alone, is associated with the observed gains. Evidence spans matched-compute comparisons, component probes, structured-state tests, and limited cross-domain checks.

  • The AgentRoom bundle probe reports shared-only < prompt-only < AgentRoom, with the largest observed step attributed to the explicit coordination channel.The MCP-tool step rises from 0.588 to 0.669, but its interval spans zero, so the paper claims an ordering rather than a percentage split.
  • +0.213 is the AgentRoom contrast over parallel-merge on the T4 Sonnet matched-compute comparison.The reported Welch test is t=3.35, p=0.003.
  • 2×Haiku AgentRoom reaches T4 mean 0.662 versus 1×Sonnet’s 0.544 at roughly half the dollar cost.This comparison is reported as suggestive, with Welch t=2.10, p=0.036; only Haiku T4 clears its bootstrap confidence interval in the broader matrix.
  • AgentRoom logs show claim-discipline behaviors in 6 of 6 runs, while agents fixed one another’s bugs in only 2 of 6.The paper describes apologies, gap-filling, warnings, and dependency documentation without attributing them to a specific training-data source.
  • CRDT-state AgentRoom succeeds 7/7 on T5, whereas MCP-chat fails 1/2 runs with score 0.088 when claims are broadcast as free text.The paper attributes the difference to room_claim() returning a system-level conflict error.
  • Cross-domain checks reproduce the direction: Rust+axum scores 0.740 versus Solo’s 0.714, while Python DevBench passes 8/10 versus 7/10.Both checks are individually underpowered, and the Rust comparison is described as descriptive.
  • Compared with CodeCRDT and other systems, AgentRoom uses explicit runtime coordination rather than implicit observation or sequential staged roles.CodeCRDT reports a bimodal 21% speedup / 39% slowdown and a 5–10% semantic conflict rate.

6 Conclusion

The paper argues that concurrent multi-agent coding is primarily an explicit-coordination problem, not merely a parallelism or merge-correctness problem. Its scope remains bounded to the reported coding setting, with generalization beyond coding tasks left open.

  • AgentRoom identifies file-level claims, broadcast logs, and per-agent status as the runtime operations associated with its contribution.The paper separates these coordination operations from agent count and the CRDT substrate in its matched-compute setting.
  • The paper leaves generalization beyond coding tasks open.

7 Limitations

The paper’s evidence is strongest for reduced abandonment and variance in a defined CLI-stable setting, while quality, cross-model, cost, and portability claims remain exploratory or bounded. Several comparisons rely on limited samples, particular scorers, or a shared runtime.

  • Scope: Headline cells span four backend domains and three powered CLI-stable models, but all headline cells share one Express.js/TypeScript runtime.Python and Rust checks are bounded mechanism-portability spot checks rather than claims of runtime-general magnitude.
  • Measurement: Quality is assessed by an LLM-judge composite, not an execution oracle, and the tasks lack a held-out correctness suite.Regex and TypeScript-AST scorers provide cross-validation, but do not replace the missing execution-based correctness evaluation.
  • Ablation interpretation: The bundle-component probe supports an ordering between substrate and coordination conditions, not a precise variance decomposition.The probe reuses ablation cells, and both reported steps carry wide intervals at the available sample sizes.
  • Cross-model comparisons: The cross-model effect-size analysis is exploratory: only Haiku T4 has a 95% bootstrap CI strictly above 1, while the other five cells include 1.The analysis uses an AST-scored subsample, excludes two CLI-unstable models, and is explicitly reported as an exploratory conjecture.
  • Evaluation sensitivity: The agent-count sweep yields scorer-dependent optima: judged quality peaks at ×2, whereas mean tests passing peaks at ×3.This disagreement makes the preferred agent count dependent on the evaluation measure.
  • Cost-equivalent comparison: The cost-equivalent comparison is suggestive rather than categorical, and its exposed bias limits generalization beyond this difficulty band.The reported direction is not expected to hold on tasks where solo Haiku already collapses.

B.7 Compute Cost (T4, Sonnet 4.6)

AgentRoom keeps wall-clock time near Solo while increasing cumulative compute with concurrent agents. In the T4 Sonnet results, ×2 reaches the highest quality column at matched wall-clock budget, while ×3 adds compute for a small regression.

  • Compute cost: AgentRoom shares the wall-clock budget concurrently, so elapsed time stays near Solo while cumulative compute scales with agent count.Cumulative compute is the sum of per-agent CLI durations and serves as a billable-cost proxy.
  • Compute cost: AgentRoom ×2 reaches the top quality column at the same wall-clock budget as Solo and Parallel-merge.The comparison uses the T4 Sonnet LLM-judge composite and budget-fair pool.
  • Compute cost: AgentRoom ×3 adds approximately 500 s of compute for a small quality regression.The additional agent therefore does not improve the reported quality column in this comparison.
  • Observed coordination: Across six valid ×2 T4 Sonnet runs, agents commonly claimed modules and adjusted plans, while cross-agent bug fixes and apologies appeared in only one-third of runs.These behaviors were coded as unprompted collaborative categories; apologies and gap-filling were not in the prompt.
  • Claim coverage: Across 65 changed-source-file runs, the mean unclaimed-write rate was 0.11, and 39/65 runs had every changed file covered by a claim.The unclaimed-write rate was uncorrelated with changed-file count (r=0.02) and modestly positively correlated with the LLM-judge composite.
  • Claim coverage: The claim-coverage quality estimate is correlational, zero-inflated, and conditioned on rooms whose coordination used the observable channel.The room surfaces unclaimed writes rather than blocking them, and some such writes served as unblocking edits.

B.11 Cross-Model Ablation (T4, all CLI-stable models)

The cross-model ablation broadly places AgentRoom above alternatives for Sonnet and Haiku, while Codex departs from that pattern. Threshold sensitivity supports the Tier I abandonment finding across tested codebook settings.

  • Evaluation design: The cross-model ablation uses the LLM-judge composite on a budget-fair 30–700 s pool.The study ran Solo, Shared-only, Parallel-merge, and AgentRoom conditions where collected.
  • Cross-model ablation: Codex multi-agent pairings hurt regardless of whether the coordination channel is enabled.The result is consistent with Codex’s T5 regression and its strong monolithic-architecture default.
  • Cross-model ablation: For Sonnet and Haiku, AgentRoom is the top condition, while Parallel-merge falls below Solo.These endpoints replicate the ordering across the two models, although intermediate Haiku cells are noisier.
  • Threshold sensitivity: Across all 12 codebook threshold combinations, the T4-only CMH common odds ratio was at least 6.3 with CMH p < 0.001.The threshold sweep supports robustness of the Tier I abandonment result to the tested classification thresholds.
  • Bundle probe: The CRDT-plus-prompt probe without MCP tools produced an LLM-judge mean of 0.588 with σ=0.188 and fell between comparison conditions.The probe tests whether the full AgentRoom bundle can be separated into components.
  • Bundle probe: The prompt contribution is sign-dependent on the substrate rather than strictly additive across conditions.The prompt-only control without teammates fell below bare Solo, so the prompt’s effect depends on the surrounding substrate.

B.14 Pool Sensitivity (T4 Sonnet ablation, LLM-judge)

The T4 Sonnet ablation preserves its substrate-to-AgentRoom ordering under both budget-fair and unrestricted pools. Cross-domain checks extend the direction to Python and Rust, but remain descriptive and underpowered.

  • Pool sensitivity: AgentRoom > Shared-only > Solo > Parallel-merge survives under both budget-fair and unrestricted pools.The budget-fair pool remains primary because it enforces matched compute; the unrestricted pool includes attempts exceeding 700 s.
  • Pool sensitivity: The wider unrestricted pool narrows the gap modestly because some long-running Solo attempts occasionally produce more code.Both AgentRoom and substrate baselines shift slightly under the wider pool.
  • Cross-domain validation: On 10 Python DevBench projects, AgentRoom ×2 passes 8/10 tasks versus Solo’s 7/10, with zero CRDT semantic conflicts.The +1 pass differential is descriptive at n=10; geotext drives the difference because ×2 finishes within budget while Solo times out.
  • Cross-language validation: On the Rust+axum T4 transposition, AgentRoom ×2 scores 0.740 versus Solo’s 0.714 on the LLM-judge composite.This comparison uses Codex GPT-5.4 and is descriptive with n=4 versus 5.
  • Cross-language validation: One Rust AgentRoom ×2 run scoring 0.580 was excluded because four enums were duplicated across two files with slightly different shapes.The duplicated-enum pattern was unique in the Rust pool and remains in the archive.
  • Protocol trace: Figure 9 maps broadcasts to per-file ownership over time for one representative T4 Sonnet AgentRoom ×2 run.The swim lanes show agent activity, the Gantt shows file ownership, and the leaders connect messages to ownership timing.

C.1 T4 ×3 Sonnet 4.6 : Complete AgentRoom Transcript

The T4 ×3 transcript shows agents claiming modules, detecting conflicts, revising ownership, and filling gaps while building a multi-file backend. The run ends with most tests passing despite one resolved CRDT conflict.

  • Concurrent work: Agents initially claim disjoint module groups and report progress through timestamps, statuses, and completion messages.The transcript includes claims for core types, storage, accounting, audit, transactions, reconciliation, fraud, and routes.
  • Cross-agent repair: Agent-C fixes an Express 5 route pattern in another agent’s file and adds comprehensive API integration tests.The agent explicitly describes the one-line change as a critical fix intended to unblock tests.
  • Concurrent work: Agent-A completes its claimed module set with all 66 tests passing before integration proceeds.Its completion message states that the modules are ready for integration.
  • Conflict handling: Conflicts on types.ts and transactions.ts trigger revised claims so agents can continue on non-overlapping files.The transcript records conflict detection, dropped claims, re-claiming, and a warning to coordinate.
  • Gap filling: Developer-C fills an uncreated-file gap when Developer-3 has not yet produced transactions.ts.This illustrates progress through adaptive reassignment rather than waiting for the stalled module.
  • Final outcome: The final result reports 65/66 tests passing across 13 source files, with 1 CRDT conflict resolved and 14 room messages.The transcript therefore records substantial completion alongside a documented merge conflict.

D Task Specifications

The evaluation uses four backend coding tasks that vary in modularity, cross-module invariants, and organizational freedom. All tasks run under a shared TypeScript/Express sandbox with fixed time limits, while agents receive requirements without reference solutions.

  • Common setup: All tasks start from the same Express.js 5 sandbox with Node 22, vitest 2, TypeScript 5.6, ESM, and a stub server.Agents receive the test command, requirements, and prescribed file structure where applicable, but not reference solutions.
  • Task specifications: The authentication task requires user storage, SHA-256 password hashing, JWT authentication, three auth routes, and seven tests.The tests cover registration, duplicates, login outcomes, authenticated and unauthenticated access, and expired tokens.
  • Task specifications: The marketplace task combines accounts, product listings, orders, verified reviews, filtered search, and appropriate 400, 401, 404, and 409 responses.Agents choose the module organization because no file structure is prescribed, making coordination especially relevant.
  • Task specifications: The e-commerce task stresses cross-module agreement through purchase-verified reviews and atomic stock decrements during checkout.It includes users, products, orders, reviews, validation, error handling, and integration tests.
  • Task specifications: The financial ledger task requires double-entry balancing, multi-currency integer-cent amounts, atomic optimistic-locking transactions, hash-chained audits, reconciliation, and fraud detection.The specification continues beyond the supplied passage, which ends during the fraud-detection requirements.

E Quality Scorer Details

Quality is measured with complementary LLM-judge, regex, and AST-based composites, with the LLM judge primary for budget-fair T4 results. Cross-checks show meaningful but imperfect scorer agreement, while sensitivity analyses preserve the reported condition ordering; scorer bias and lexical gaming remain limitations.

  • Primary scorer: The LLM judge is the primary T4 budget-fair scorer, using one fixed Sonnet 4.6 invocation per run to assess four rubric dimensions.It receives the task specification, test status, and a truncated repository snapshot.
  • Cross-validation: Pearson r=0.67 across 246 jointly scored T4 runs indicates that regex and LLM-judge composites measure related but non-interchangeable quantities.The paper therefore uses one scorer throughout and reports the other as a cross-check rather than averaging them.
  • Limitations: The LLM judge may favor Sonnet outputs, although the same-family concern does not affect the Sonnet-versus-Sonnet AgentRoom comparison.The regex/AST cross-validation provides only a partial guard against scorer bias.
  • Regex scorer: The regex scorer uses fixed, author-selected weights, including 0.25 for Type Safety and 0.25 for Defensive Coding.Its patterns were fixed before evaluating multi-agent runs and were not tuned to favor any condition.
  • Cross-validation: The AST cross-check reports Pearson r=0.79 and Spearman ρ=0.62 over 30 runs, supporting regex-score use while showing that agreement is not perfect.The AST scorer is harsher on Codex’s compact single-file implementations.
  • Sensitivity analysis: The leave-one-out sensitivity check preserves AgentRoom > solo under every single-dimension ablation, with the ratio staying within 1.12–1.25 on the smaller subset.That subset contains four solo runs and is retained as a robustness check rather than the canonical comparison.
  • Limitations: Regex scoring is reproducible across runtime variants but remains lexical and can reward defensive code that is never exercised.This is identified as a known limitation and future-work direction.
Loading 2608.23740v1…