PaperlayerView exact revision on arXiv

Source-linked AI summary

ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning

Jingwei Song, Meng Chen, Jie Xiao, Qingnan Ren, Jiaqi Huang, Yangshen Deng, Chris Tong, Wanyi Chen, Suli Wang, Zhisheng Chen, Ziqian Bi, Shuo Lu, Yiqun Duan, Xu Wang, Rymon Yu, Lynn Ai, Eric Yang, Tianyu Shi

arXiv:2602.02192v5cs.LGcs.DC

TL;DR

LLM RL post-training can use cheaper distributed inference resources, but wide-area coordination and policy dissemination complicate centralized learning. ECHO-2 combines centralized learning with distributed rollouts, bounded staleness, overlap-aware provisioning, peer-assisted broadcast, and cost-aware worker activation; experiments report lower cost while preserving comparable RL quality.

  • Problem

    Conventional RL pipelines use expensive co-located rollout resources, while distributed inference introduces latency, heterogeneity, and dissemination challenges that complicate continuous learner utilization.

  • Method

    ECHO-2 combines centralized learning with distributed rollouts, treating bounded staleness as a control parameter and using overlap-based capacity planning, peer-assisted broadcast, and cost-aware activation.

  • Results

    ECHO-2 reduces cumulative cost by 33.3%–36.3% for Qwen3-8B at matched AIME accuracy and achieves a 4.6% end-to-end cost reduction on QwQ-32B while tracking Prime-RL reward trajectories.

  • Takeaways & Limitations

    ECHO-2 shows that centralized learning with distributed wide-area rollouts can lower RL post-training cost while maintaining learning quality comparable to baselines.

  • Takeaways & Limitations

    ECHO-2 provides no formal guarantees for bounded-staleness robustness, whose safe range may depend on task and reward signal; validation beyond 70B models remains future work.

Abstract

from arXiv · show

Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.

1 Introduction

ECHO-2 addresses the cost and utilization problems of conventional LLM RL pipelines by separating centralized learning from distributed rollout generation. It combines bounded staleness, overlap-aware provisioning, and peer-assisted dissemination to reduce cost while preserving RL quality.

  • Conventional LLM RL pipelines co-locate learners and rollout workers, although rollout generation often dominates wall-clock time and leaves learners intermittently idle.
  • Distributed inference resources are cheaper and abundant, but heterogeneous throughput, wide-area latency, and dynamic availability make naive distribution inefficient.
  • ECHO-2 uses centralized learning with distributed rollouts and treats policy staleness S as a user-specified budget that absorbs latency and enables overlap.
  • Peer-forwarding chains relay policy snapshots and use aggregate fleet bandwidth to reduce tail broadcast latency.
  • ECHO-2 provides a closed-form capacity rule linking training time, dissemination latency, and rollout throughput to learner-saturating capacity.
  • Experiments on GRPO post-training show substantially lower end-to-end cost while matching the RL quality of strong centralized baselines.

2 Related Work

Prior work improves asynchronous RL under controlled, co-located deployment or studies stale-data optimization, whereas ECHO-2 targets centralized learning with heterogeneous wide-area rollout workers. Its peer-assisted dissemination addresses policy-transfer costs that data-center connectivity assumptions leave unresolved.

  • Asynchronous RL frameworks under controlled deployment: Existing asynchronous RL frameworks improve utilization but largely assume co-located learners and rollout workers with data-center-grade connectivity.
  • Under wide-area constraints, star-topology dissemination faces learner-uplink and tail-receiver bottlenecks, motivating peer-forwarding broadcast.
  • Algorithm-level stabilization under staleness: Algorithm-level methods such as M2PO, BAPO, and GAC modify optimization under staleness and are orthogonal to ECHO-2's system-level pipeline.
  • Decentralized RL: ECHO-2 differs from fully decentralized RL by retaining centralized learning while distributing rollout inference across heterogeneous wide-area workers.

3 Cost-Efficient Distributed RL

ECHO-2 turns bounded staleness into a controllable systems parameter: rollout generation, dissemination, and training overlap while capacity is provisioned to keep the learner saturated. Cost-aware scheduling then activates the cheapest workers meeting the required throughput.

  • 3.1 Design Rationale: Bounded staleness allows distributed rollouts to tolerate variable throughput and wide-area latency without imposing strict on-policy synchronization.
  • 3.1 Design Rationale: The learner consumes rollouts at most S steps stale and publishes snapshots every κ steps, making (S, κ) explicit execution controls.
  • 3.3 Overlap Condition and Capacity Requirement: The capacity model relates Ttrain, Tbcast, R, and worker throughputs to the aggregate capacity needed for continuous learner utilization.
  • 3.3 Overlap Condition and Capacity Requirement: Under the conservative no-new-snapshot rollout assumption, κ ≤ S −1 suffices when the overlap-condition ceiling term is at most 2.
  • 3.4 Cost-Aware Provisioning and Scheduling: ECHO-2 ranks workers by increasing unit throughput cost ρi and activates the cheapest prefix whose cumulative throughput exceeds µmin(κ).
  • 3.4 Cost-Aware Provisioning and Scheduling: Closed-loop scheduling adjusts active workers using throughput and availability estimates, targeting µtarget = γµmin(κ) to absorb variability.
  • 3.4 Cost-Aware Provisioning and Scheduling: The rollout plane uses Nch ≈⌊B0/Bw⌋ parallel forwarding chains to exploit aggregate worker bandwidth.

4 System Architecture and Implementation

ECHO-2 implements distributed RL through separate rollout, learning, and data planes connected by versioned messages and a shared replay buffer. Workers asynchronously generate and forward trajectories, while the learner enforces bounded staleness and publishes policy snapshots periodically.

  • The three-plane architecture separates rollout inference, centralized learning, and task-specific data handling.
  • Rollout Plane: Rollout workers generate rewarded, version-tagged trajectories under locally installed policy snapshots and push them to the shared buffer.
  • Learning Plane: The centralized learner samples eligible trajectories, performs training updates, and publishes immutable snapshots every κ updates.
  • Data and Learning Planes: The replay buffer discards trajectories older than S learner steps, enforcing bounded staleness without global synchronization.
  • Rollout Plane: Peer-forwarding workers relay snapshot chunks immediately and start generating under a new version after installation completes.
  • Rollout Plane: Heartbeat statistics support chain ranking, failure recovery, and closed-loop worker management under fluctuating throughput and availability.

5 Experiments

ECHO-2 is evaluated on GRPO post-training across model sizes, WAN regimes, staleness budgets, dissemination strategies, and provisioning mechanisms. The experiments show lower cost with comparable RL quality, robust moderate staleness, accurate overlap-based provisioning, and scalable peer-assisted broadcast.

  • Cost–Quality Efficiency: ECHO-2 reduces cumulative cost by 33.3%–36.3% at matched AIME accuracy for Qwen3-8B while achieving comparable final accuracy at matched cost.Qwen3-4B shows the same trend.
  • Cost–Quality Efficiency: A 4.6% end-to-end cost reduction is achieved on QwQ-32B versus Prime-RL despite ECHO-2 using 76×RTX 5090 workers over a WAN.Reward trajectories track Prime-RL throughout training.
  • RL Quality under Bounded Staleness: For S ≤6, reward and accuracy remain within roughly 5% fluctuation of the synchronous baseline, whereas S = 11 can lead to instability in standard GRPO.Moderate staleness also shows similar convergence trends and lower cost.
  • Learner Utilization and Provisioning: As the rollout pool grows, learner bubble ratio drops toward zero near the predicted threshold, validating the overlap model as a provisioning rule.Larger S shifts the transition left, trading policy freshness for reduced rollout capacity.
  • Broadcast under Bandwidth Constraints: TREE-PIPELINED keeps Tbcast within a small constant factor of STAR-UNLIMITED across fleet sizes, while STAR-LIMITED latency grows rapidly with N.Chunked peer forwarding lets each chain operate at line rate Bw after pipeline warm-up.
  • Ablations: Removing peer-assisted broadcast increases dissemination and learner waiting time, while disabling cost-aware provisioning raises the cost of reaching a quality target.Both mechanisms are therefore necessary for the reported end-to-end cost efficiency.

6 Limitations and Future Work

ECHO-2’s staleness range is supported empirically rather than by formal guarantees, and its scope remains centralized learning with distributed rollouts. Broader learner topologies and larger models require further validation.

  • Limitations: ECHO-2 provides no formal guarantees for bounded policy lag, and the safe staleness range may depend on the task and reward signal.The authors identify theoretical staleness control as future work.
  • Future Work: Extending ECHO-2 to multiple or geographically replicated learners introduces synchronization and policy-consistency challenges, while validation beyond 70B remains future work.The current framework focuses on centralized learning with distributed rollouts.

7 Conclusion

ECHO-2 combines centralized learning with distributed rollouts, bounded staleness, overlap-based provisioning, cost-aware activation, and peer-assisted broadcast. Experiments across 4B, 8B, and 32B models report lower cost with comparable RL quality, while the appendix tightens the staleness bound under the overlap condition.

  • Conclusion: ECHO-2 separates centralized learning from distributed rollouts to support cost-aware LLM post-training under wide-area execution.The framework treats bounded staleness as a control knob and activates workers on demand based on cost.
  • Conclusion: Across 4B, 8B, and 32B models, ECHO-2 significantly lowers training cost while preserving RL quality comparable to baselines.The conclusion summarizes the reported cross-model outcome.
  • Comparison with INTELLECT-2: ECHO-2 and INTELLECT-2 address orthogonal design questions: cost-efficient hybrid execution under bounded staleness versus permissionless contributor participation.The comparison concerns system scope rather than a direct performance ranking.
  • Tightening the Bound: The appendix derives a conservative upper bound on maximum policy staleness and shows that the overlap condition tightens it.The derivation uses the execution semantics and rollout-throughput assumptions stated in the appendix.
  • Tightening the Bound: The bound assumes snapshots publish every κ learner steps, batches form at step start, and new-policy rollouts begin only after Tbcast.Ignoring progressive dissemination and early rollout start makes the resulting bound conservative.

B.4 Implication for κ = 2

For κ = 2, ECHO-2 uses overlapping dissemination, rollout generation, and training to maintain learner utilization while controlling policy staleness. Its scheduler provisions heterogeneous workers using throughput, availability, and cost signals, supported by peer-forwarding chains.

  • B.4 Implication for κ = 2: For κ = 2, the overlap condition requires Tbcast < 2Ttrain.The bound is presented for the common case κ = 2.
  • B.4 Implication for κ = 2: ECHO-2 configures the publication period as κ = S −1, making S the sole staleness control parameter.Substitution into the conservative bound justifies this single-parameter configuration.
  • B.4 Implication for κ = 2: The learner, rollout workers, and dissemination operate concurrently while batches are restricted to trajectories within the configured staleness window.The execution loop samples admissible trajectories, performs policy updates, and publishes snapshots periodically.
  • B.4 Implication for κ = 2: ECHO-2 activates low-cost workers when aggregate capacity falls below target and releases expensive workers when capacity exceeds it.The scheduler uses worker availability and effective throughput to maintain saturation while limiting unnecessary rollout cost.
  • B.4 Implication for κ = 2: Peer-assisted broadcast uses parallel forwarding chains, with workers ranked by speed, success rate, and recent stability.Each chain streams snapshot chunks through one upstream parent and at most one downstream child; ranking weights are 0.5, 0.35, and 0.15.
  • B.4 Implication for κ = 2: The ranking coefficients and operational thresholds are defaults rather than tuned hyperparameters, and moderate perturbations leave the system robust.The stated defaults include weights (0.5, 0.35, 0.15), a failure threshold of 3, and a success saturation count of 10.

D Fault Tolerance under Single-Worker Failure

ECHO-2 recovers from isolated worker failures through replacement and forwarding-chain reconstruction, but broadcast-stage failures can temporarily affect downstream workers. The reported fault results characterize a minimally provisioned, no-redundancy setting rather than large concurrent failures.

  • D Fault Tolerance under Single-Worker Failure: 240–312 s recovery restores full saturation across all four single-worker fault scenarios, with bubble-ratio increases below 3%.These recovery windows equal roughly 0.15–0.20 of one 8B training step at Ttrain ≈1500 s.
  • D Fault Tolerance under Single-Worker Failure: Broadcast-stage failures cause cascading downstream policy-installation delays, reducing throughput beyond the 1/8 loss from one worker.Downstream workers temporarily continue generating rollouts under their previous policy until the chain is rebuilt.
  • D Fault Tolerance under Single-Worker Failure: The fault-injection table measures peak throughput drop, recovery to µtarget, and bubble-ratio increase against a fault-free baseline.The deployment contains 8 workers with γ = 1.0.
  • D Fault Tolerance under Single-Worker Failure: The default γ = 1.1 redundancy absorbs single-worker faults without replacement, while robustness to large-scale concurrent failures is outside the targeted regime.The no-redundancy results represent the worst case and use approximately 10% overprovisioning in the default setting.

E.2 Wide-Range Benchmarks

Across five math reasoning benchmarks and both Qwen3 model scales, ECHO-2 with S = 3 maintains reward performance comparable to VERL under the same training configuration.

  • E.2 Wide-Range Benchmarks: Across five datasets and both Qwen3-4B and Qwen3-8B scales, ECHO-2 with S = 3 maintains reward performance comparable to VERL.The comparison uses the same reward model and training configuration.
  • E.2 Wide-Range Benchmarks: The evaluation covers AIME24, OmniMath, JEE, HardMath, and IMO-answer-400 with dataset-specific average-reward metrics.AIME24 uses avg@64, JEE uses avg@8, and the remaining benchmarks use avg@1.

E.3 Validation on a Larger Model and Comparison with Prime-RL

On QwQ-32B, ECHO-2 is evaluated against Prime-RL with matched training settings but different rollout deployments. ECHO-2 achieves a 4.6% end-to-end cost reduction under realistic wide-area networking while maintaining comparable reward trajectories.

  • E.3 Validation on a Larger Model and Comparison with Prime-RL: ECHO-2 uses 8×H100 training GPUs and 76×RTX 5090 rollout workers, whereas Prime-RL uses 10×H100 rollout workers on one intranet node.The rollout deployment is the stated system difference in the QwQ-32B comparison.
  • E.3 Validation on a Larger Model and Comparison with Prime-RL: 4.6% end-to-end cost reduction is achieved by ECHO-2 relative to Prime-RL’s idealized intranet deployment.The comparison covers a 100-step GRPO run with matched hyperparameters; ECHO-2 uses 76×RTX 5090 wide-area rollout workers.
  • E.3 Validation on a Larger Model and Comparison with Prime-RL: The cost gap reflects overlap-aware provisioning and cost-aware activation that make commodity RTX 5090 workers viable without sacrificing learner saturation.Prime-RL does not use unit-throughput cost ρi as a scheduling input.
  • E.3 Validation on a Larger Model and Comparison with Prime-RL: The 4.6% Prime-RL comparison isolates wide-area system mechanisms, whereas the 33–36% VERL comparison combines disaggregated rollouts with those mechanisms.The Prime-RL evaluation is conservative because it uses an idealized intranet; wide-area Prime-RL broadcast cost would grow with fleet size.
  • E.3 Validation on a Larger Model and Comparison with Prime-RL: Reward trajectories remain close to Prime-RL throughout 100 steps, and final reward matches its endpoint within run-to-run variance.The result is attributed to separating snapshot delivery timing from the rollout version consumed under bounded staleness.
  • E.3 Validation on a Larger Model and Comparison with Prime-RL: Direct end-to-end evaluation beyond 32B was not performed because of compute budget constraints.The authors leave empirical study at 70B+ scales to future work.

F Data Plane Implementation and Poker Game Alignment via Sandbox Integration

ECHO-2 integrates poker through a decoupled Data Plane that converts sandbox interactions and heterogeneous logs into version-tagged rollouts for unchanged generic learning and rollout components. The adapter also computes poker-specific rewards, masks, advantages, and off-policy corrections for GRPO training.

  • Sandbox Integration: A task-specific Data Plane adapter connects the poker sandbox to ECHO-2 while leaving the generic Rollout Plane and Learning Plane unchanged.The adapter standardizes game logs into a unified rollout tuple consumable by both planes.
  • Versioned Rollout Records: Each trajectory record τ = (x, y, r, v, Ω) stores prompt-response data, reward, generating-policy version, and optional task metadata.The replay buffer indexes records by version and enforces bounded staleness during sampling.
  • Log Standardization: The adapter converts raw poker logs into standardized chat messages and token sequences, using left padding for variable-length episodes.States become user messages and actions become assistant messages before tokenization.
  • Reward and Training Signals: Poker training restricts loss computation to assistant action tokens and derives advantages from episodic chip outcomes.Turn-aware masks isolate response tokens, while group-wise normalization reduces variance in policy updates.
  • Reward and Training Signals: The trajectory-level reward is the episode’s net chip profit or loss, and normalized advantages are broadcast only to response tokens.Grouping can use initial state or batch-level strategies for GRPO standardized returns.
  • Off-Policy Learning: Truncated importance sampling stabilizes GRPO updates when replayed data comes from a sampling policy different from the current learner.This enables the Learning Plane to reuse diverse off-policy experiences from the Data Plane.
  • Poker Evaluation: GRPO changes Qwen3-0.6B’s result against the LLM opponent from −1.677 net loss to +1.245 net profit.Performance against three rule-based opponents becomes negative after GRPO, indicating a trade-off addressed by broader opponent sampling.
Loading 2602.02192v5…