PaperlayerView exact revision on arXiv

Source-linked AI summary

DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation

Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee, Branislav Kveton, Seunghyun Yoon, Franck Dernoncourt, Qizhe Xie, Trung Bui

arXiv:2609.33485v1cs.CL

TL;DR

Long-context reasoning can collapse as inputs grow, and monolithic models entangle evidence search with reasoning. DISCO distributes local grounding across Workers and uses a trained Driver to plan and synthesize evidence. It reports strong performance across long-context benchmarks alongside lower inference costs.

  • Problem

    Reasoning quality can collapse as inputs grow because monolithic architectures entangle contextual grounding with answer reasoning.

    Source: §1 · p. 1

  • Method

    DISCO partitions long contexts across Workers for parallel grounding and uses a GRPO-trained Driver to plan query actions and synthesize gathered evidence.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    78.4% accuracy on RULER-QA (1M tokens) versus 10.9% for standard retrieval baselines; DISCO gains up to 9.8 points over full-context models on LongBench v2 and matches Gemini-3-Pro-Preview while reducing inference costs over 80%.

    Source: §1 · p. 2, §4.1 · p. 7

  • Takeaways & Limitations

    The reported results support distributed long-context inference as a way to maintain fidelity at million-token scale while reducing inference costs.

    Source: §6 · p. 9

Abstract

from arXiv · show

While Large Language Models (LLMs) advertise million-token context windows, reasoning quality often collapses as inputs grow -- a phenomenon termed context rot. This failure stems from a structural entanglement in monolithic architectures, where the massive search burden of contextual grounding exhausts the representational capacity needed for complex reasoning. To resolve this, we propose Grounding-Reasoning Disaggregation via DIStributed long COntext scaling (DISCO). Inspired by distributed computing frameworks like Apache Spark, DISCO partitions long context across a fleet of Worker LLMs dedicated exclusively to parallel, localized grounding. A central Driver LLM, trained via Reinforcement Learning (GRPO) to optimize planning, orchestrates execution by dynamically mapping queries into atomic extraction tasks and reducing the gathered evidence to synthesize a final answer. By isolating reasoning from raw context noise, DISCO effectively eliminates context rot. On RULER-QA (1M tokens), it maintains 78.4% accuracy where standard baselines collapse. Furthermore, it outperforms full-context models by up to 9.8 points on LongBench v2 and matches frontier models like Gemini-3-Pro-Preview while reducing inference costs by over 80%, establishing a highly efficient paradigm for robust long-context inference.

1 Introduction

The paper argues that long-context reasoning degrades when grounding and reasoning are entangled, and introduces DISCO to separate them. Its reported results show strong accuracy at one million tokens, gains over full-context baselines, and lower inference costs.

1 Introduction

  • As inputs grow, reasoning quality can collapse even within advertised context limits, constraining effective context and real-world reliability.The paper notes that memory-augmented alternatives demand pre-training from scratch, with generalization remaining underexplored.Source: §1 · pp. 1–2
  • The authors attribute context rot to monolithic attention entangling grounding’s growing search burden with reasoning, exhausting capacity needed for multi-hop reasoning.They characterize grounding as scaling with text length while reasoning complexity typically remains constant.Source: §1 · p. 1
  • DISCO separates contextual grounding from reasoning by distributing context shards across Workers and assigning planning and synthesis to a central Driver.The Driver dynamically decomposes queries into extraction tasks and reduces gathered evidence into an answer; GRPO trains its planning.Source: §1 · p. 2
  • 78.4% accuracy on RULER-QA (1M tokens) contrasts with 10.9% for standard retrieval baselines; DISCO also gains up to 9.8 points on LongBench v2 and cuts costs over 80% at frontier-model performance.The LongBench v2 gain is reported for Qwen3-14B; the cost comparison is with Gemini-3-Pro-Preview.Source: §1 · p. 2

2 Method

The method treats long-context inference as distributed computation: Workers ground evidence locally, while a Driver synthesizes it. The Driver plans a query DAG to coordinate these roles.

2 Method

  • The method frames long-context processing as distributed inference, with a data-parallel architecture, dynamic DAG execution, iterative Driver–Worker orchestration, and Driver optimization via reinforcement learning.The section develops grounding–reasoning motivation, architecture, DAG formalization, orchestration, and RL optimization.Source: §2 · p. 2

2.1 The Grounding-Reasoning Interference

  • As context length grows, grounding sparse evidence becomes the bottleneck; monolithic models couple that search with answer reasoning in one attention window.The paper links this coupling to context rot and lost-in-the-middle failures, rather than a lack of reasoning capability.Source: §2.1 · pp. 2–3

2.2 DISCO: A Distributed Long Context Scaling Paradigm

  • DISCO partitions context into shards for local grounding, then aggregates evidence for global synthesis, replacing global extraction with parallel shard-level extraction.The paper characterizes the transformation as replacing O(L^2) global extraction with K parallel O(|x(k)|^2) local extractions.Source: §2.2 · p. 3

2.3 Query Decomposition via Directed Acyclic Graphs

  • Because lightweight Workers may lack multi-hop reasoning depth, the Driver decomposes queries into DAG actions with data dependencies.The architecture assigns local grounding to Workers and has the Driver reason over aggregated Worker outputs without accessing raw context.Source: §2.3 · p. 3, §2.2 · p. 3

1. Narrow Actions (Map: Atomic Extraction).

Narrow actions assign local, entity-centric extraction to Workers, keeping their task focused on finding evidence rather than performing complex reasoning.

1. Narrow Actions (Map: Atomic Extraction).

  • Narrow actions are atomic, entity-centric extraction tasks executed by Workers using only their assigned local context.The instruction is framed as a direct lookup, such as finding an entity’s birth date.Source: §1 · pp. 3–4
  • The lookup directive is deliberately not a complex reasoning question, limiting the work assigned to each Worker.Workers process their persistent shards independently.Source: §1 · p. 4
  • Workers act as high-recall information scanners, separating evidence extraction from logical deduction.The execution pipeline dispatches Map/Narrow tasks to parallel Workers.Source: §1 · p. 4, Figure 1 · p. 4

2. Wide Actions (Reduce: Logical Synthesis).

Wide actions let the Driver synthesize evidence across shards, while iterative DAG refinement and reinforcement learning support planning and execution.

2. Wide Actions (Reduce: Logical Synthesis).

  • Wide actions aggregate prior outputs to resolve cross-shard dependencies through Driver-based reasoning and synthesis.They operate on distilled evidence, supporting multi-hop deduction and conflict resolution within a concise window.Source: §2 · p. 4

2.4 Dynamic Orchestration: Iterative DAG Refinement

  • The Driver iteratively constructs and executes DAGs, then evaluates the terminal output and replans when its evidence is insufficient.Narrow tasks run on Workers in parallel; Wide tasks synthesize dependent outputs into the evidence pool.Source: §2.4 · pp. 4–5

2.5 Optimizing the Driver via Reinforcement Learning

  • GRPO trains the Driver against answer-accuracy, evidence-support, and DAG-format rewards, with terminal accuracy and evidence rewards propagated to intermediate planning steps.The evidence reward penalizes answers unsupported by retrieved context, while the format reward penalizes syntax errors or invalid dependencies.Source: §2.5 · p. 5

3 Experimental Setup

The evaluation compares DISCO with direct-processing, retrieval, and agentic baselines across three long-context benchmarks. Implementation details cover model configurations and GRPO training on synthetically expanded contexts.

3.1 Evaluation Protocol

  • Baselines span full-context processing, dense retrieval, sequential Chain-of-Agents, LLM×MapReduce, and Recursive Language Models.Standard RAG retrieves the top 16 256-token chunks using Qwen3-Embedding-4B.Source: §3.1 · p. 5
  • LongBench v2, RULER-QA, and ∞Bench evaluate contexts from tens of thousands of tokens to 1M tokens across long-context reasoning tasks.LongBench v2 includes Medium and Long subsets; ∞Bench covers long-book QA and multiple-choice QA.Source: §3.1 · p. 5
  • Evaluations use Qwen3-8B, Qwen3-14B, and Gemini-3-Pro-Preview, with Qwen3-4B-Instruct-2507 Workers and 4,096-token shards in DISCO.Open-weight baselines use SGLang and extend context to 128K with YaRN; DISCO allows up to 10 replanning stages.Source: §3.1 · p. 5

3.2 Implementation Details.

  • Recursive context augmentation expands MuSiQue examples to 16K–512K tokens by replacing entity mentions with linked Wikipedia content, then filters contradictions to retain 4.5K samples.Qwen3-235B-A22B-Think-2507 supports the expansion process.Source: §3.2 · p. 6
  • GRPO trains Qwen3-8B and Qwen3-14B thinking models for two epochs on the curated dataset, using AdamW with a constant 1e−6 learning rate.Training samples eight trajectories per prompt, with a 24K-token context limit and 16K-token generation limit.Source: §3.2 · p. 6

4 Results and Analyses

Across the evaluated benchmarks, DISCO combines strong long-context accuracy with lower inference costs and near-flat latency as context grows. Results also examine reasoning gains, Worker scale, reward design, and efficiency.

4.1 Main Results

  • +13.5 vs. +5.6 for Qwen3-14B: DISCO amplifies Thinking Mode accuracy gains on LongBench v2 compared with Full Long Context.With DISCO, Thinking Mode gains are +5.5 for 8B and +13.5 for 14B; full-context gains are +0.9 and +5.6, respectively.Source: Figure 2 · p. 7, §4.1 · p. 7
  • In the million-token regime, DISCO sustains 78.4% accuracy while performance collapses to 10.9% with Qwen3-8B; on LongBench v2, Qwen3-14B gains 9.8 points over Full Long Context (48.7% vs 38.9%).DISCO also outperforms the strongest baseline, LLM×MapReduce, by 6% in the million-token regime.Source: §4.1 · p. 7
  • 71.3% vs. 72.2% accuracy: Gemini-3-Pro with DISCO retains near-parity with Full Context while evaluation cost falls 6.2× ($126.7 → $20.4).The comparison is on LongBench v2 (Long).Source: Figure 3 · p. 7, §4.1 · p. 7

4.2 Impact of Worker LLMs

  • 2.9% accuracy gain (50.9% → 53.8%) from larger Workers is smaller than the Driver-scaling gain (39.8% → 50.9%); execution stages fall from 3.96 to 3.02.Despite the larger Worker’s higher per-token cost, total latency is comparable: 415s vs. 432s.Source: Figure 4 · p. 7, §4.2 · p. 8

4.3 Impact of Reward Design

  • Removing the Evidence reward reduces accuracy and increases execution length, while omitting the Format reward yields the lowest accuracy and highest latency.Without the Evidence reward, the Driver accepts poorly grounded answers and enters costly replanning loops; without the Format reward, invalid DAGs trigger retries.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Inference Efficiency and Scalability

  • Near-flat latency as context grows to 1M tokens: under an 8×H100 iso-compute budget, DISCO outperforms agentic baselines whose latency grows steeply.The RL-tuned variant has the highest reported efficiency by reducing execution steps.Source: Figure 5 · p. 8, §4.4 · p. 8

5 Related Work

Prior long-context approaches extend model windows, retrieve or manage selected information, or process chunks through agent workflows. DISCO instead treats context as distributed data and combines parallel grounding with central orchestration.

Distributed Computing Paradigms

  • Distributed computing systems such as MapReduce and Spark use Map and Reduce operations, with DAG scheduling enabling iterative processing.DISCO adapts these primitives to orchestrate processing across distributed long-context data.Source: §5 · p. 8

Long-Context Processing in LLMs

  • Architectural extrapolation and memory-augmented models expand nominal context windows, but may still face context rot or memory-module generalization challenges.The cited approaches include length extrapolation, memory architectures, and test-time training.Source: §5 · pp. 8–9
  • Retrieval and agentic-memory systems scale through selective retrieval, but similarity-based selection can discard implicitly relevant evidence needed for global reasoning.The related-work discussion also notes that just-in-time mechanisms rely on imperfect recall.Source: §5 · p. 9
  • Sequential agent workflows incur latency and cascading information loss, while LLM×MapReduce may lack scalability for complex multi-hop reasoning and Recursive LMs rely on massive models.The discussion describes Chain-of-Agents as sequential chunk processing with information passed along a chain.Source: §5 · p. 9
  • DISCO uses smaller Workers for parallel chunk processing and a central Driver for orchestration, aiming to avoid sequential-agent latency and retrieval precision loss.The paper describes these operations as Map and Shuffle/Reduce.Source: §5 · p. 9

6 Conclusion

The conclusion presents DISCO as a distributed approach to long-context scaling that separates grounding from reasoning. It reports robust fidelity up to one million tokens and lower inference costs alongside frontier-model accuracy.

6 Conclusion

  • DISCO reframes long-context processing as a distributed computing problem and structurally separates contextual grounding from logical reasoning.Massive contexts are partitioned across Worker LLMs orchestrated by a central GRPO-trained Driver.Source: §6 · p. 9
  • The paper reports robust fidelity up to one million tokens where monolithic architectures collapse.It also reports frontier-model accuracy at a fraction of the inference cost.Source: §6 · p. 9
  • The conclusion advocates active distributed systems for navigating large information spaces rather than passively expanding single-node context windows.It links the approach to isolating reasoning from raw data noise for complex, multi-hop tasks.Source: §6 · p. 9

A.2 Evaluation Details

The evaluation combines long-context data construction and baseline protocols with experiments on worker choice, model-family generalization, information loss, and RL-driven planning behavior.

A.2 Evaluation Details

  • Baselines follow official implementations except Chain-of-Agents, which was reproduced; En.QA predictions are judged semantically by Qwen3-235B-A22B-Instruct on a binary scale.The full-context baseline is truncated within 128K; Chain-of-Agents and LLM×MapReduce use 4096-token chunks.Source: §A.2 · p. 11
  • DISCO partitions context across Workers, plans a DAG of parallel Narrow extractions and Driver-side Wide synthesis, and iteratively adds evidence until answering or reaching Tmax.The inference algorithm maintains an evidence pool and returns a best-effort response if the stage limit is reached.Source: §A.2 · p. 11

A.3 Training Data Construction

  • Recursive Context Augmentation expands MuSiQue supporting-fact entities with linked Wikipedia content, then filters contradictions to produce 4.5k samples spanning 16K–512K tokens.The expanded contexts disperse query-linked information to test long-range retrieval.Source: §A.3 · pp. 11–12

A.4 More Experimental Results

  • At p ≤0.2 fact dropout, accuracy stays near 51% at about four stages; at p = 0.5, accuracy falls to 33% as stages rise to about seven.The inference budget is Tmax = 10, and excessive loss can terminate runs before convergence.Source: §A.4 · p. 12, Figure 6 · p. 12
  • 47.3 on LongBench v2 for DISCO, versus 39.6 for the equivalent-scale dense-retriever baseline.The comparison uses a Qwen3-14B Driver after RL training.Source: §A.4 · p. 12, Table 3 · p. 12
  • Across model families, DISCO improves on Full-Context CoT, with gains of 2.8 points for LLaMA-3.1-70B and 5.55 for Ministral-8B-Reasoning.Worker family is described as fungible, while the Driver sets the reasoning ceiling.Source: §A.4 · p. 13, Table 4 · p. 13

A.5 Case Studies of RL Benefits

  • RL-trained planning atomizes multi-hop questions, reducing average planning stages to 3.39; RL also stabilizes DAG formatting, with removing Rfmt raising latency from 381 s to 499 s.Across 15 samples, the After-RL Driver converges in one stage on every sample, while the Before-RL Driver hits the stage cap in 40%.Source: §A.5 · pp. 13–14
  • Across translation cases, After-RL extraction keys target the unique source sentence or partition phrases into disjoint word-level segments, whereas Before-RL generic/topic keys yield no match or no satisfiable chunk.For Kalamang, After RL produced 317 facts and converged in 1 stage; for Zhuang, After RL produced 37 facts in 1 stage.Source: §A.5 · pp. 13–14

Related papers

Loading 2609.33485v1…