Source-linked AI summary
Decentralized Multi-Agent Systems with Shared Context
Yuzhen Mao, Azalia Mirhoseini
TL;DR
Existing multi-agent systems depend on centralized coordination that can bottleneck communication and integration as subtasks grow. DeLM replaces this controller with asynchronous agents sharing verified context through a task queue, achieving stronger results across software engineering and long-context reasoning benchmarks while reducing SWE-bench cost.
Problem
Centralized, prompt-routed coordination in multi-agent systems can constrain communication and integration as parallel subtasks or evidence clusters increase.
Method
DeLM uses parallel agents, a task queue, and a shared context that admits compact, evidence-verified updates for asynchronous coordination.
Results
DeLM achieves strongest SWE-bench Verified test-time scaling and highest LongBench-v2 average accuracy across four frontier models, improving over the best baseline by up to 5.7 percentage points.
Takeaways & Limitations
Shared verified state makes intermediate discoveries, failures, and constraints reusable across agents, supporting higher benchmark performance and lower SWE-bench cost.
Takeaways & Limitations
DeLM incurs verification overhead and depends on agents producing appropriately decomposed subtasks.
Abstract
from arXiv · showhide
Multi-agent systems (MAS) can scale large language model reasoning at test time by decomposing complex problems into parallel subtasks. However, most existing MAS rely on centralized orchestration, where a main agent assigns work, collects outputs, and merges results. As the number of subtasks grows, this controller becomes a communication and integration bottleneck. We propose Decentralized Language Models (DeLM), a MAS framework that decentralizes coordination through parallel agents, a shared verified context, and a task queue. Agents asynchronously claim subtasks, read accumulated progress, perform local reasoning, and write back compact verified updates. The shared context acts as a common communication substrate, enabling agents to build on one another's verified progress without routing every update through a central controller. Empirically, DeLM improves both software-engineering test-time scaling and long-context reasoning. On SWE-bench Verified, DeLM achieves the best performance across Avg.@1, Pass@2, and Pass@4, with gains of up to 10.5 percentage points over the strongest baseline, while reducing cost per task by roughly 50%. On LongBench-v2 Multi-Doc QA, DeLM achieves the highest average accuracy across four frontier model families, improving over the strongest baseline by up to 5.7 percentage points. The code is available on our project website at https://yuzhenmao.github.io/DeLM/.
1 Introduction
DELM addresses centralized multi-agent coordination bottlenecks by enabling asynchronous task claiming and shared verified-context updates. It achieves strong results across software-engineering scaling, long-context QA, and programmatic reasoning, including 77.4% pass@4 on SWE-bench Verified at roughly half the baseline cost.
- Coordination mechanism: DELM replaces synchronous centralized scatter–gather with asynchronous coordination through parallel agents, a shared verified context, and a task queue.Agents claim tasks and write compact verified results that remain visible to other workers without central merging and rebroadcasting.
- Evaluation settings: The strongest results span software-engineering test-time scaling, long-context multi-document QA, and aggregation-heavy long-context reasoning.These settings stress parallel reasoning trajectories, concurrent evidence-cluster inspection, and exact aggregation, respectively.
- Evaluation results: 77.4% pass@4 is achieved on SWE-bench Verified while cost falls to $0.12 per task, roughly half that of the baselines.Shared context lets agents reuse discoveries across parallel software-engineering attempts.
- Evaluation results: Up to 5.7 percentage points improvement over the best baseline is achieved on LongBench-v2, with the highest average accuracy across four frontier models.Admission-time verification and hierarchical summarization both contribute to these gains.
- Evaluation results: Combining RLM with DELM yields the best accuracy and lowest cost on OOLONG, although vanilla DELM underperforms RLM on exact row-level aggregation.The result indicates that decentralized verified context can complement code-mediated execution in programmatic reasoning workflows.
2 Motivation and Design Principles
The section motivates replacing prompt-routed centralized communication with a shared context, while addressing the scalability and reliability challenges of shared state. DeLM uses compact gists by default and verifies updates before admission to limit context overload and error propagation.
- Motivation: Centralized MAS route intermediate progress through the main agent, so parallel execution does not guarantee parallel progress sharing.This communication bottleneck motivates a different communication substrate.
- Design Principles: Sharing raw documents or full traces preserves information but overwhelms context windows and raises cost, whereas compact summaries risk losing reasoning details and evidence.The section frames shared-context design as a trade-off between usability and information preservation.
- Design Principles: Errors in shared state can propagate widely by becoming reusable problem state that misleads later agents and shapes downstream reasoning.Because post-hoc answer checking may occur after intermediate decisions are influenced, DeLM uses admission-time verification before updates enter shared context.
3 Decentralized Language Models (DELM)
DELM decentralizes coordination with a shared context C and task queue T: agents claim subtasks in parallel, build on verified progress, and produce a final answer from the accumulated context. Its pipeline compresses and verifies completed results before admitting reusable gists, with optional generation of additional subtasks.
- End-to-end pipeline: The DELM pipeline initializes subtasks, runs agents, admits verified gists, optionally generates more subtasks, and finalizes the answer from C.Algorithm 1 repeats execution and admission until the task queue is empty, then produces the final answer.
- Parallel execution: Agents execute queued subtasks in parallel, read the verified shared context, and perform local reasoning before returning results.The overview describes asynchronous task claiming and shared visibility of reusable progress.
- Coordination structures: DELM coordinates agents through shared context C and task queue T, where agents claim pending subtasks and communicate progress through admitted context entries.C stores compact, reusable problem state, while T stores pending subtasks available for parallel execution.
- Verified updates: Completed results are compressed into reusable gists and verified against supporting evidence before becoming visible in the shared context.DELM treats each context update as an admission problem and rejects raw or unverified outputs from direct insertion.
4 Experiments
Section 4 evaluates DELM on software-engineering test-time scaling and long-context multi-document reasoning, then explains its gains through shared-state coordination mechanisms and component ablations. The experiments show that verified, compact progress helps agents avoid redundant exploration while improving accuracy and cost efficiency.
- Benchmarks: DELM is evaluated on SWE-bench Verified and long-context multi-document reasoning, covering complementary coordination challenges.SWE-bench Verified uses realistic GitHub issues, repository environments, and human-verified tests.
- SWE-bench Verified: On SWE-bench Verified, scaling runs across independent trajectories because tasks are largely sequential and offer little within-trajectory parallelism.A task is solved when any of X ∈ {2, 4} attempts produces a correct patch; reported metrics include Avg.@1.
- SWE-bench Verified: Shared progress improves test-time scaling by letting later agents avoid redundant exploration, reuse failures, and focus on unresolved task components.Unlike isolated attempts, DELM exposes compact verified updates across trajectories through shared state.
- Trace-level mechanisms: DELM preserves admitted constraints as reusable shared state, preventing later agents from reopening globally invalid simplifications.A trace shows agents retaining the constraint that a single filter is unsafe for multi-valued relations while optimizing valid foreign-key cases.
- Trace-level mechanisms: Compact patch summaries reduce communication cost by sharing evidence-backed discoveries without exposing peers to full raw trajectories.Summaries omit command histories, file dumps, failed edits, and intermediate reasoning while retaining reusable findings.
- Ablations: 60.1% falls to 55.2% without verification, while hierarchical summarization lowers accuracy to 57.7%.The ablation indicates that unchecked unsupported claims can corrupt downstream reasoning, and removing summarization harms performance by losing useful gist.
5 Combining DELM with RLM
DELM and RLM have complementary strengths across LongBench-v2 Multi-Doc QA and OOLONG, motivating a hybrid that retains RLM reasoning while adding DELM’s decentralized coordination mechanisms.
- Separate Evaluation: DELM achieves higher average accuracy than RLM on LongBench-v2 at roughly the same cost per task.The comparison uses GPT-5 with medium reasoning; RLM uses GPT-5-mini for sub-calls with recursion depth=1.
- Separate Evaluation: On OOLONG, DELM underperforms RLM in both average accuracy and cost because the benchmark resembles structured data processing.OOLONG is described as aggregation-heavy, making RLM’s recursive, code-mediated processing a natural fit.
- Hybrid Design: The hybrid retains RLM as the underlying reasoner while adding DELM’s verified shared context and decentralized task queue.All agents are equally ranked RLM instances rather than a main-agent/sub-agent hierarchy.
- Hybrid Design: An initial RLM call creates a compact work plan, parallel RLM workers claim queued subtasks, and outputs enter shared context only after verification.The hybrid coordinates through shared state without a central hierarchy.
6 Related Work
Related work spans multi-agent systems, programmatic agentic systems, and long-context language-model systems. DeLM differs by organizing shared information hierarchically through compact verified gists, selectively unfolded summaries, and raw evidence.
- Multi-Agent Systems: Recent multi-agent systems decompose problems across specialized or interacting agents, including CAMEL’s role-playing agents and MetaGPT’s agent collaboration.The passage also situates other systems by their instructions, context, tools, and models.
- Multi-Agent Systems: Combee identifies centralized aggregation as a bottleneck under high parallelism, but studies parallel prompt learning rather than decentralized agent coordination.Recent blackboard-style multi-agent systems are also noted in this related-work discussion.
- Programmatic Agentic Systems: Recursive Language Models treat the prompt as an external environment and recursively select snippets for inspection through a code-mediated process.This exemplifies programmatic interfaces that let language models inspect state, invoke tools, manipulate artifacts, and recursively delegate computation.
- Long-context LM systems: Long-context systems compress, retrieve, or store information beyond the model’s current context window, but lossy gist lookup can miss unrecovered evidence.Examples include gist representations, compression with retrieval, and external-memory mechanisms such as LongMem, MemGPT, Mem0, and MemOS.
- Long-context LM systems: DeLM organizes shared information by abstraction level: agents reason over compact verified gists, selectively unfold relevant entries, and recover detailed summaries and raw evidence.This hierarchy makes the shared context both global and recoverable.
7 Limitations and Future Work
DELM’s future work centers on reducing verification overhead and improving task decomposition. Its reliability gains currently involve admission-time verification overhead, while performance depends on avoiding both under-specified and unnecessarily numerous subtasks.
- Verification: Future work should explore lighter-weight admission-time verifiers to reduce overhead while preserving grounding benefits.Suggested approaches include learned models and rule-based checks for common claim types.
- Task decomposition: Overly coarse decompositions leave agents with under-specified subtasks, limiting the quality of decentralized reasoning.
- Task decomposition: Overly aggressive decompositions can spawn unnecessary agents, making decomposition quality an open direction for improvement.
8 Conclusion
DELM is a decentralized multi-agent system that coordinates agents through shared context and a task queue instead of a central controller. Compact, verified updates make intermediate progress reusable, enabling agents to build on findings, avoid repeated failures, preserve constraints, and recover evidence when needed.
- 8 Conclusion: DELM coordinates agents through a shared context and task queue rather than a central controller.This design decentralizes coordination across agents.
- 8 Conclusion: Compact, verified updates turn intermediate progress into reusable problem state.Agents can build on prior findings, avoid repeated failures, preserve constraints, and recover detailed evidence only when needed.
- 8 Conclusion: On SWE-bench Verified, DELM improves test-time scaling by making discoveries and failures reusable across parallel attempts.The passage states that this produces higher pass rates while reducing cost.
Supplementary Material
The shared context stores long source units as reference-grounded summaries and compact gists, which agents read by default and selectively expand when needed. Entries are admitted only after verification.
- Shared-context compression: Long source units are compressed into reference-grounded summaries S_i and compact gists G_i stored in the shared context.The gists provide a compact representation of the source material.
- Selective evidence access: Agents read gists by default and selectively unfold them to summaries or raw evidence when needed.This supports progressively retrieving more detailed source information.
- Verified context: Entries enter the shared context only after verification.Verification governs admission of summaries and gists into the shared context.
A Method Details · A.1 Hierarchical Summarization
DELM hierarchically represents each long source unit as raw content, a reference-grounded evidence map, and a compact gist. Only gists enter shared context, while detailed summaries and raw sources remain externally addressable for selective inspection.
- A.1 Hierarchical Summarization: Each long source unit r_i is transformed through three levels: r_i → S_i → G_i.The shared context C stores only compact entries (ℓ_i, G_i), while L[ℓ_i] stores S_i and R[ℓ_i] stores the raw source unit.
- A.1 Hierarchical Summarization: The hierarchy avoids using a lossy gist to select raw evidence directly and avoids broad unfolding of raw context.This separates lightweight global navigation from detailed, reference-grounded and raw-source access.
- A.1 Hierarchical Summarization: Stage 1 produces S_i as a reference-grounded evidence map of short bullets, each expressing one atomic claim.Every bullet includes a ref-tag identifying its supporting span in the raw text.
- A.1 Hierarchical Summarization: Each ref-tag copies the first and last N ≥5 words of its supporting span verbatim.The tag is written as [ref: h . . . t], where h and t denote the span boundaries.
- A.1 Hierarchical Summarization: The first-stage summarizer ϕ1 is conditioned on question Q and requires full source-unit coverage.This preserves potentially critical evidence while extracting question-relevant content at higher fidelity; each bullet is addressable by local index k in L[ℓ_i].
- A.1 Hierarchical Summarization: Stage 2 compresses S_i into G_i, a short, highly compact gist for the shared context.The gist is designed to capture a source unit’s query relevance at a density that lets downstream agents decide whether further inspection is needed.
- A.1 Hierarchical Summarization: Only G_i is admitted to C, whereas S_i and raw content remain in L and R for selective unfolding.Agents access the detailed representations only when further inspection is needed.
A.2 Selective Unfolding · A.3 Admission-Time Verification · A.4 Parallel Execution and Systems Optimizations
DELM combines opt-in coarse-to-fine unfolding with admission-time verification and parallel execution mechanisms. These designs let agents retrieve grounded detail selectively, admit only verified updates, and execute dependent work efficiently with disciplined concurrency and shared-context reuse.
- A.2 Selective Unfolding: Agents default to compact gists G_i and selectively unfold to reference-grounded summaries S_i, then raw source content when additional detail is required.Unfolding is a two-stage, coarse-to-fine retrieval process.
- A.2 Selective Unfolding: The UNFOLD directive retrieves labeled summaries and evidence maps, enabling reasoning over grounded evidence rather than compressed gists.Agents request a small set of labels at any reasoning step.
- A.2 Selective Unfolding: DEEP_UNFOLD retrieves labeled raw content after summary reasoning when subtle qualifiers or exact wording require finer-grained detail.The follow-up call includes the original task, shared context, prior summaries, and requested raw content.
- A.3 Admission-Time Verification: Verification follows the compression path, checking gists against original results or checking long-source summaries and gists across both hierarchy levels.Stage-1 summaries are grounded in raw source content before Stage-2 gists are checked against them.
- A.3 Admission-Time Verification: Only updates that pass verification enter shared context; failed updates are rejected, regenerated, or returned to the task queue with a rejection reason.Retries continue only up to a fixed retry limit.
- A.4 Parallel Execution and Systems Optimizations: DELM supports parallel execution with concurrent admission, a dependency-aware task queue, and KV-cache reuse across calls.These are three lightweight mechanisms for safe and efficient parallel execution.
- A.4 Parallel Execution and Systems Optimizations: Compression and verification run concurrently across completed results, while verified updates are admitted through an atomic write after backing content is stored.Agents independently propose updates, and admission is the only synchronized step.
- A.4 Parallel Execution and Systems Optimizations: The dependency-aware queue dispatches tasks without dependencies first and unlocks subsequent tasks as upstream dependencies complete.When the queue is exhausted, a single agent can acquire a queue lock to request more work.