PaperlayerView exact revision on arXiv

Source-linked AI summary

Miles v0.1: Production-Level Post-Training

RadixArk, :, Tom Chen, Mao Cheng, Shi Dong, Kangrui Du, Yanbin Jiang, Jiajun Li, Yiming Li, Tao Lin, Yusheng Su, Andy Ye, Yueming Yuan, Zhichen Zeng

arXiv:2609.08368v1cs.LGcs.CL

TL;DR

Frontier-scale post-training requires reliable coordination between increasingly complex rollouts and training, including control of numerical and historical mismatches. Miles v0.1 presents a full-stack architecture with modular rollout, training, synchronization, and extension mechanisms, and demonstrates fully asynchronous agentic RL at 744B scale on 64 NVIDIA GB300 GPUs.

  • Problem

    Frontier post-training must coordinate multi-turn, tool-using rollouts with training while preventing rollout-training numerical and history mismatches that can invalidate or silently corrupt objectives.

  • Method

    Miles provides modular rollout, training, and weight-update components with configurable comparison policies, true-on-policy alignment, and extensions beyond full-parameter RL.

  • Results

    A 744B-parameter model trains on 32 GPUs within a 64-GPU fleet, while the median training step takes 263 seconds over the first 30 measured steps.

  • Takeaways & Limitations

    Miles combines numerical fidelity, execution efficiency, and extensibility across frontier-scale post-training while making supported model, hardware, and recipe coverage explicit.

  • Takeaways & Limitations

    True-on-policy alignment currently covers only dense Qwen3 0.6B and 4B variants under specified Megatron or FSDP profiles, and its guarantee covers sampled-token log-probabilities rather than the full output distribution.

Abstract

from arXiv · show

We present Miles v0.1, a full-stack, production-ready system for frontier post-training. Building upon the clean design of slime, Miles designs each stage of the reinforcement-learning (RL) training loop around a single principle: components should be verified, clean, and customizable. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and enterprises alike. This report walks through the system end to end: rollout engines built on SGLang, a trainer with a choice of two backends (NVIDIA Megatron-LM and PyTorch FSDP), and three weight-synchronization transports for different deployment topologies. Beyond full-parameter RL, Miles also supports LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment, and extends the same architecture to diffusion models. We close with an end-to-end case study: fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps. Miles is open-sourced at https://github.com/radixark/miles, with the project website at https://miles.radixark.com.

1 Introduction

Miles v0.1 is a production-ready full-stack system for frontier post-training that organizes rollout, training, and weight synchronization around verified, clean, customizable components. It supports asynchronous RL and extends the same architecture to additional post-training paradigms and diffusion models.

  • Motivation: Frontier-scale RL must coordinate multi-turn, tool-using rollouts in external environments with training on trillion-parameter MoE models.These demands create hardware-utilization challenges and can produce numerical gaps between rollout engines and trainers large enough to invalidate the objective.
  • System: Miles v0.1 provides a full-stack, production-ready post-training system built around verified, clean, and customizable components.The system targets accuracy, efficiency, reliability, and scalability across the post-training loop.
  • Core RL loop: The core RL loop generates trajectories, updates the policy, and synchronizes updated weights back to rollout engines.SGLang performs rollout generation, Megatron-LM or PyTorch FSDP performs training, and weight updates minimize interruption to in-flight rollouts.
  • Core RL loop: Fully asynchronous execution lets rollout engines continue generating while the trainer works, allowing both sides to progress concurrently rather than taking turns.Miles uses this mode to overlap throughput-oriented generation with training and reduce hardware idle time.
  • Scope: The architecture also covers LoRA RL, on-policy distillation, supervised fine-tuning, true-on-policy alignment, and diffusion models.The report describes these as extensions that reuse shared rollout, trainer, and weight-update components while changing selected parts of the loop.

2 Rollout

Miles addresses agentic rollout throughput and fidelity by preserving cache locality, token-exact multi-turn histories, and consistent expert routing. Its asynchronous rollout stack decouples generation from training with configurable replenishment, buffering, and staleness controls.

  • Design goals: Agentic rollout generation must solve both throughput and fidelity because slow generation limits wall-clock speed while trajectory divergence can corrupt gradients silently.Miles addresses throughput through routing and decoupled generation, and fidelity through environment integration, token-exact recording, and expert-routing consistency.
  • Terminology: Miles defines prompts, trajectories, groups, and sessions so multi-turn episodes and same-prompt comparisons can be scheduled and consumed correctly.A trajectory is a single-turn completion or multi-turn episode; a group contains trajectories from one prompt, while a session is the ordered requests for one episode.
  • Fast agentic rollout: Affinity keeps each multi-turn session on the rollout engine holding its KV-cache prefix, so later turns prefill only the newly generated suffix.Session-aware and data-parallel-rank-aware routing preserve prefix reuse, while load-based routing can separate turns and lose that benefit.
  • Fast agentic rollout: The session server preserves conversation history, controls tokenization, and retains exact token IDs so the trainer sees what the policy sampled.It also supplies session identity for routing; least-loaded initial placement helps balance the fleet while preserving affinity for the session lifetime.
  • Asynchronous rollout: Fully asynchronous rollout lets engines generate continuously while the trainer consumes completed trajectory groups from a buffer using separate GPU pools.This removes turn-taking idle time, while new weights temporarily pause rollout engines only during installation.
  • Asynchronous rollout: Sample granularity immediately replaces each finished trajectory, keeping generation near its limit even when trajectory lengths differ by an order of magnitude.Group granularity instead waits for every trajectory in a group, so one slow trajectory can leave rollout capacity underused.
  • Buffering and staleness: The bounded data buffer absorbs rate differences between rollout and training and drops groups when generation fails, filters reject them, or their weights become too stale.Staleness is checked when the trainer collects a group and is based on the current weight version versus the oldest version used during generation; dropped prompts may be retried except for filter-rejected groups.

2.3 Agentic Environments

Miles supports agentic environments with different control scopes through nested plug-in layers and independent sandbox backends. This accommodates varied framework responsibilities while preserving configurable environment integration, though the integrations remain experimental and screenshot recording is unavailable.

  • Different environments may own only episode control or also batching, rewards, and token recording, making one common interface unsuitable.
  • Miles uses three nested plug-in layers, with each connector replacing exactly one layer to match an environment’s scope of control.
  • Connectors can attach at the agent, generate, or rollout function, determining which token-recording responsibilities Miles or the external framework retains.
  • Sandbox providers are independent of connector layers, and recipes can choose sandbox lifetime, including one sandbox per episode that is deleted afterward.
  • The named connectors and sandbox integrations are experimental and continue to evolve.
  • The session server does not record screenshots, so computer-use connectors must record their own trajectories at the generate-function layer.

2.4 Token-In-Token-Out (TITO)

Miles’ token-in-token-out design makes the trainer’s sequence match the policy’s sampled tokens by centralizing tokenization and checkpointing multi-turn history. Linear and branching session rules, configurable replay matching, and model-registration checks extend this fidelity across diverse harnesses, while vision-language inputs remain unsupported.

  • Multi-turn pipelines can silently alter sampled sequences through message parsing, tool execution, and chat-template rendering, breaking training fidelity.
  • The TITO session server controls tokenization, checkpoints prompt and output token IDs with log-probabilities and routed experts, and assembles contiguous training sequences.
  • Linear and Branching Sessions: Linear sessions permit only tail extensions and yield exactly one training sequence, whereas branching sessions retain an append-only tree whose leaves can yield multiple trajectories.
  • When Harnesses Do Not Replay Verbatim: Configurable replay matching ranges from comparing template-consumed fields to accepting identical tool-call JSON despite different serialization.
  • When Harnesses Do Not Replay Verbatim: Looser matching can merge genuinely different tool histories, leaving the stored prefix authoritative and silently preserving a call the agent never made.
  • Verification and Current Limits: Miles verifies new model registrations with CPU append-only checks and GPU live-inference checks, because CPU validation alone cannot guarantee parser compatibility.
  • Verification and Current Limits: The session server cannot carry image or video inputs, so vision-language models must use SGLang’s lower-level token-in, token-out interface.

2.5 Efficient Rollout Routing Replay (R3)

MoE rollout and training can diverge because numerical differences change expert assignments even when tokens and weights match. R3 records rollout routing decisions and replays them during training, extending exact routing across multi-turn episodes at substantial memory and transport cost.

  • Small numerical differences can route the same token to different MoE experts during rollout and training, changing which experts receive updates.
  • R3 treats each token’s expert assignments as rollout data and replays those assignments during training instead of recomputing them.
  • Because the session server records routed experts with token IDs and log-probabilities, R3 covers entire multi-turn episodes rather than single completions.
  • A 32K-token sequence over 60 layers at k = 8 requires roughly 60 MB for routing assignments, and the payload must remain in memory and move with the trajectory.
  • R3 is a per-recipe choice, has no effect for dense models, and may have limited effects in asynchronous RL because other mismatch factors remain.

3 Training

Miles structures training around numerical precision, memory management, backend choice, and objectives. Its low-precision recipes require rollout and training to use a shared quantization contract, while offloading and backend selection address frontier-scale memory and model-support constraints.

  • 3.1 Low-Precision Training: Low-precision training requires rollout and training to quantize the same weights in the same way.Miles applies one quantization path across rollout and training, with end-to-end recipes for three formats and additional INT4 and BF16-train, FP8-serve options.
  • 3.1 Low-Precision Training: NVFP4 requires every stage touching the weights to quantize them, so pairing NVFP4 rollout with a BF16 trainer is unsupported.The general rule permits matching formats or BF16 training with quantized rollout, but NVFP4 is the exception.
  • 3.1 Low-Precision Training: Miles checks checkpoint conversion, trainer forward, SGLang rollout, and live weight export to enforce end-to-end precision agreement.The MXFP8 and NVFP4 contracts use the same quantization logic across these four stages.
  • 3.1 Low-Precision Training: Reward curves track the BF16 baseline closely while measured low-precision configurations reduce rollout time significantly.The evidence covers tested configurations, but the authors note that behavior may differ across models; MXFP8 and NVFP4 remain Beta.
  • 3.2 Memory Efficiency: Streaming optimizer state cuts Qwen3-30B-A3B offloading from 24 s to 5.2 s and reloading from 8.9 s to 1.3 s.The mechanism keeps optimizer state off GPU during training and fetches it as needed, whereas actor eviction moves the entire paused training process.
  • 3.3 Two Training Backends: Megatron-LM suits large MoE models and multi-rack jobs, whereas FSDP suits direct Hugging Face loading and models fitting under data parallelism.FSDP avoids conversion and architecture flags, while Megatron’s backend is positioned for larger model-parallel deployments.

4 Weight Update

Miles treats weight synchronization as a configurable handoff from trainer to rollout engines, optimized for different hardware topologies. Broadcast is the default, while P2P and disk-delta updates serve wider multi-node fleets or environments connected only by shared storage.

  • 4 Weight Update: Weight synchronization prepares new trainer weights, transfers them to every rollout engine, and resumes generation under the updated policy.The handoff occurs after every training step by default and can become a major bottleneck at frontier scale.
  • 4.2 Peer-to-Peer Weight Transfer: P2P lets multiple sources write only needed shards directly to rollout targets over RDMA, unlike broadcast’s one-copy-to-every-rank pattern.The number of active senders is the smaller of the source and target fleets.
  • 4.2 Peer-to-Peer Weight Transfer: P2P is up to about 70% slower than broadcast on single-node deployments and is useful mainly when both fleets span more than one node.Its advantage grows with fleet width, while host-side reshaping and pinned-memory staging impose overhead.
  • 4.2 Peer-to-Peer Weight Transfer: P2P model coverage is limited to architectures with unified Megatron-to-SGLang weight-name mappings.Current mappings cover Qwen2, Qwen3 dense and MoE families, GLM4-MoE, and DeepSeek-V3 derivatives.
  • 4.3 Disk-Delta Updates: Disk-delta updates send only changed checkpoint bytes through shared storage, avoiding direct NCCL or RDMA connectivity.Each rollout host patches a local copy from a shared base checkpoint and receives numbered policy versions.
  • 4.4 Pausing Generation, and Checking the Result: Miles retracts in-flight rollout requests by default so they roll back and resume against new weights.Abort is disallowed when session server is enabled, and FSDP always retracts in-flight requests.

5 Other Post-Training Recipes

Miles reuses its rollout, training, and synchronization components for LoRA RL, on-policy distillation, and true-on-policy alignment. These recipes reduce update cost, add teacher-based signals, or enforce exact rollout-training probability agreement within stated coverage limits.

  • 5.1 LoRA RL: LoRA freezes the base model and makes a small adapter the unit of training, synchronization, and rollout.The adapter consists of low-rank matrices whose product adds a correction to selected frozen modules.
  • 5.1 LoRA RL: LoRA lightens training and weight updates because only adapter gradients, optimizer state, arithmetic, communication, and synchronization are required.Colocated jobs use IPC, while disaggregated jobs broadcast serving-ready adapters over NCCL; P2P and disk-delta do not carry adapters.
  • 5.1 LoRA RL: LoRA support requires compatible adapter implementations, module-name mapping and export, and serving-time application in SGLang.A name match alone does not establish support for expert, linear-attention, or model-specific projections.
  • 5.2 On-Policy Distillation (OPD): On-policy distillation trains a student on its own trajectories using a teacher-derived reverse-KL signal folded into token-level advantages.The policy-gradient update remains unchanged, and Miles supports one-sample and top-K divergence estimates.
  • 5.2 On-Policy Distillation (OPD): 56% response-length reduction was observed on held-out DAPO prompts, from 14,070 to 6,132 tokens, with no reliable accuracy change.Accuracy moved from 84.0% to 85.2%, within the evaluation’s roughly 1.6-point standard error.
  • 5.3 True-On-Policy Alignment: True-on-policy alignment addresses probability gaps caused by differing rollout and trainer kernels, fusions, and batch shapes.In the documented Qwen3-4B-Base run, the reward curve matched baseline while rollout took longer.
  • 5.3 True-On-Policy Alignment: True-on-policy alignment currently covers dense Qwen3 0.6B and 4B profiles and guarantees agreement only for sampled-token log-probabilities.Miles refuses models outside the registered profile rather than offering a partial guarantee.

6 Miles-Diffusion

Miles-Diffusion extends the same three-stage post-training architecture to image and video diffusion models. It overlaps generation, decoding, and reward scoring, while using precision controls and explicit evidence levels to manage numerical sensitivity and recipe maturity.

  • 6 Miles-Diffusion: Miles-Diffusion applies the generate, update, and weight-return loop to image and video diffusion models.A diffusion trajectory is the sequence of denoising steps that transforms noise into one image or video.
  • 6 Miles-Diffusion: Generation, decoding, and reward scoring overlap through microgroups that are deserialized and scored while the engine processes the next batch.This reduces serialized phases and accommodates video trajectories that can occupy gigabytes of tensors.
  • 6 Miles-Diffusion: Diffusion RL uses deterministic and per-parameter dtype controls because rounding differences between engine and trainer forward passes become update noise.Precision-sensitive parameters such as timestep embedders and modulation tables can remain in FP32 while the rest runs in BF16.
  • 6 Miles-Diffusion: Fully gated recipes require a complete training curve and nightly deterministic end-to-end tests whose registered metrics exactly match a committed standard.Proxy gated uses a documented scaled-down proxy; verified lacks either higher-level deterministic standard; not verified lacks a complete training curve.
  • 6 Miles-Diffusion: Current main lists SD3.5 and LTX-2.3 as fully gated, Wan2.2 as proxy gated, Qwen-Image and Cosmos 3 as verified, and Wan2.2 LoRA recipes as not verified.The evidence level applies to the exact script and topology named in the model guide.

7 Verified Coverage: Models and Hardware

Miles reports verified coverage across frontier model architectures and NVIDIA and AMD hardware, while distinguishing support levels and hardware-dependent precision constraints.

  • Model coverage: Six frontier models received day-0 support, with Miles and SGLang developing RL recipes and inference paths in parallel.
  • Model coverage: Miles documents recipes for nine model families, but “supported” ranges from full-scale validation to reduced-layer CI tests or unit tests.
  • GPU coverage: NVIDIA coverage spans A100 through GB300, with Hopper and Blackwell designated production hardware and the reference run using 64 GB300 GPUs.
  • GPU coverage: AMD support is native to ROCm and covers MI300X, MI325, MI350X, and MI355X, with end-to-end CI tests on MI350 runners.
  • Compatibility constraints: A workable run requires simultaneous compatibility among the checkpoint, GPU, and supported number format.

8 Code Quality Principle

Miles organizes its implementation around readability and extensibility, using a simple driver, typed interfaces, modular layers, and automated enforcement.

  • Miles adopts a code-quality principle that the system should be easy to read and easy to extend.
  • Readability: The training driver presents each synchronous iteration as a short sequence of high-level operations while hiding implementation details in modules.
  • Extensibility: Small typed interfaces let users replace rollout, data, reward, loss, and importance-ratio correction logic through import-path flags.
  • Extensibility: Layered agent, generation, and rollout components allow custom RL loops without forking or rewriting Miles.
  • Enforcement: Pre-commit hooks, CI checks, banned-pattern hooks, and metric gates enforce formatting, API usage, and reproducibility practices.

9 Case Study: GLM-5.2 Agentic Training with 64 NVIDIA GB300 GPUs

The case study runs asynchronous agentic RL for GLM-5.2 744B-A40B across 64 GB300 GPUs, combining memory-aware parallelism with overlapping generation and training. It reports a 263-second median step and stable numerical alignment, while reward rises are treated as observations from a single run.

  • Configuration: The reference run trains GLM-5.2 744B-A40B on terminal-use coding tasks with fully asynchronous RL across 64 GB300 GPUs split evenly between generation and training.
  • Configuration: Each generation node serves an FP8 model copy with an FP8 KV cache, while the trainer uses BF16 weights.
  • Memory and parallelism: About 279 GB of optimizer state per rank is streamed to node-local disk because it exceeds GPU memory.
  • Agentic workload: Episodes solve terminal-bench-2 tasks in sandboxes for up to 30 turns or one hour, with GRPO comparing each trajectory against seven attempts at the same task.
  • Scheduling: Up to 128 trajectories remain in flight independently of the 64-trajectory training batch, with evaluation every ten steps on held-out tasks.
  • Results: 263 seconds is the median training-step time across the first 30 measured steps, excluding the clipped 1,042-second warm-up.
  • Results: Generation and training overlap between weight updates, while affinity maintains a 96% prefix-cache hit rate.
  • Numerical health and reward: Log-probability divergence averages 0.0369 over 100 steps, while the nine-step reward moving average rises from 0.438 to 0.556 but is reported only as an observation.

10 Conclusion

Miles combines numerical fidelity, execution efficiency, and extensibility in a full-stack post-training architecture. Its scope spans multiple objectives and diffusion models, while explicit coverage levels and current limitations make supported use cases clear.

  • Miles combines token-exact trajectories, consistent expert routing, low-precision execution, memory-efficient state management, two training backends, and rollout-mismatch corrections.
  • Weight-update transports preserve one synchronization contract across deployment topologies, allowing the same components to support synchronous and asynchronous schedules.
  • The architecture extends to LoRA RL, on-policy distillation, supervised fine-tuning, true-on-policy alignment, and diffusion models.
  • Incomplete vision-language session support, early-stage precision formats, and model-family limits on weight-transfer paths bound the currently supported scope.

11 Contributions and Acknowledgments

Miles acknowledges its core contributors, additional supporters, and the broader Miles and SGLang communities for their contributions and support.

  • Miles identifies Tom Chen, Mao Cheng, Shi Dong, Kangrui Du, Yanbin Jiang, Jiajun Li, Yiming Li, Tao Lin, Yusheng Su, Andy Ye, Yueming Yuan, and Zhichen Zeng as core contributors.
  • The report extends sincere gratitude to Haoguang Cai, Richard Chen, Jiadong Guo, and Jiadong Guo's listed collaborators.
  • Miles thanks all community contributors in the Miles and SGLang ecosystems for their invaluable contributions and support.
Loading 2609.08368v1…