Source-linked AI summary
Belayer: Efficient Fault Tolerance for LLM Agentic RL Training
Jiecheng Zhou, Qinghao Hu, Peng Sun, Xingcheng Zhang, Weiming Zhang
TL;DR
Agentic RL lacks efficient, correct recovery for failures spanning rollout engines and stateful environments. Belayer combines selective rollout-worker recovery with prefix-consistent environment checkpoint/restore, achieving up to 42× faster worker recovery and 1.5×–3.5× faster environment recovery than baselines.
Problem
Agentic RL needs recovery mechanisms that preserve long-horizon trajectory context and stateful sandbox environments across rollout and environment failures.
Method
Belayer combines standby-worker recovery using selective GPU-state reuse with full container file-system/runtime checkpointing and prefix-consistent environment restoration.
Results
Up to 42× faster worker recovery and 1.5×–3.5× faster environment recovery were achieved with low failure-free overhead under evaluated workloads.
Takeaways & Limitations
Belayer provides an end-to-end fault-tolerance framework for LLM agentic RL training across rollout, environment, and training components.
Takeaways & Limitations
Environment recovery is limited to supported Linux container mechanisms and container-local state, while shadow-worker handover does not guarantee warm recovery from CUDA context corruption or driver resets.
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly trained with reinforcement learning in long-horizon, sandboxed environments. Unlike conventional RL, agentic RL couples GPU-intensive rollout engines with stateful environment containers whose actions may produce visible side effects, such as file edits, command execution, and dependency installation. A single trajectory can span many rounds of gen- eration and environment interaction, so a component failure can discard completed work or expose the model to an environment state that is inconsistent with its context. However, existing systems lack efficient and correct recovery mechanisms for this distributed execution model. This paper presents Belayer, an efficient fault-tolerant system for LLM agentic RL training. Belayer handles failures in both rollout engines and environment execution while targeting low failure-free overhead. For scoped worker-local rollout failures, Belayer equips each pre-initialized shadow worker with a selective GPU-state reuse protocol that retains independently owned weights and raw KV-arena allocations after owner and GPU health checks, reinitializes worker-local state, and rebuilds request-specific KV contents from logged token prefixes. For environment failures, Belayer introduces full checkpoint and full restore to jointly capture and restore container file-system and runtime state, and coordinates the recovered environment with the LLM context to preserve prefix consistency. An adaptive policy opportunistically overlaps full-state checkpointing with natural LLM inference bubbles when the predicted interval is long enough. Empirical results show low measured overhead during failure-free training, a worker-recovery-time reduction of up to 42 times faster compared with a full engine cold start, and 1.5 to 3.5 times faster recovery from environment failures.
1 Introduction
LLM agentic RL combines long-horizon generation with stateful environments whose persistent side effects create coupled recovery requirements. Belayer addresses rollout-engine and environment failures with selective GPU-state reuse, full-state checkpoint/restore, and low failure-free overhead.
- Workload challenges: LLM agentic RL trajectories span repeated token generation and environment execution, with semantic state distributed across LLM context and sandbox file-system and runtime state.Actions can execute commands, modify files, and create persistent side effects inside isolated environments.
- Workload challenges: 460.8 seconds is the median trajectory duration, while 1% of environment action failures can extend rollout duration by 48.7%.Restarting an entire trajectory after environment failure can therefore discard substantial completed work.
- Belayer: Belayer uses pre-initialized standby workers for rollout recovery and full-state checkpoint/restore for environment recovery, preserving completed interactions and enabling handover within 1 second.The recovery design targets both heavyweight inference-engine reconstruction and aligned recovery across context and environment state.
- Belayer: Belayer selectively reuses independently owned model weights and raw KV-cache allocations after health checks, then reconstructs volatile worker state and request-specific KV contents.The shadow worker remaps graph buffers and rebuilds KV cache from prefix tokens rather than retaining complete process context and request-specific contents.
- Evaluation: Up to a 42× worker-recovery speedup is achieved over a full engine cold start, with 1.5×–3.5× recovery speedup over the environment failure baseline and low failure-free overhead.Belayer is implemented on Slime with ∼6K lines of code and evaluated on Qwen3 models from 4B to 32B parameters.
2 Background and Motivation
Agentic RL couples multi-step LLM generation with stateful sandbox execution, so valid trajectories require consistency between logged context and environment state. Failures therefore create both performance and correctness challenges that existing checkpointing, restart, and standby approaches do not adequately resolve.
- Agentic RL workflow: Agentic RL alternates LLM token generation with sandbox tool execution, then rewards trajectories and updates model parameters for the next rollout.Each trajectory contains generated tokens and environment observations.
- Trajectory consistency: A valid trajectory depends jointly on its logged LLM context and persistent file-system and runtime state because tool observations shape later actions and rewards.The environment state is represented as E_k = (F_k, R_k) at committed action boundary k.
- Failure consequences: Approximately 30 seconds of failure detection, 38.5 seconds of engine reconstruction, and tens of seconds of request regeneration can delay an affected trajectory after rollout-worker failure.These costs arise in the Qwen3-32B setup because failures interrupt stateful in-flight requests.
- Failure consequences: Environment failures can alter training-sample semantics or leave uncertain partial effects, causing the LLM to misinterpret infrastructure errors and making retries unsafe.An injected OOM traceback may appear as an ordinary observation, while interrupted actions may have had no effect, completed, or partially changed state.
- Recovery requirements: Recovery must limit critical-path delay and roll back LLM context and environment state to a consistent prefix, while checkpointing both states at the same committed action boundary.Frequent checkpoints raise failure-free latency, whereas sparse checkpoints increase rollback and regeneration costs.
- Standby recovery motivation: 82.8% of 58 fail-stop reports left model weights and raw KV backing arenas unimplicated, while only 1.7% left process context and 15.5% left complete KV images reusable.These findings motivate selective reuse rather than full replication or indiscriminate sharing across worker failure boundaries.
3 System Overview
Belayer is an end-to-end fault-tolerance framework for LLM agentic RL training, combining fast rollout recovery with prefix-consistent environment-state recovery. Its loosely coupled control and execution planes coordinate rollout, environment, and training workloads while supporting fault isolation.
- Architecture: Belayer separates control functions from rollout, environment, and training workloads across loosely coupled components for flexible deployment and fault isolation.The control plane manages rollout logic, the training pipeline, and data processing, while the execution plane runs workloads.
- Training workflow: For each prompt, the training workflow creates an environment container, invokes LLM inference and environment interaction APIs, and generates a trajectory through routed rollout engines.A breadth-first reorganizer loads datasets and traverses prompts before passing them to the custom rollout function.
- Fault-tolerance scope: Belayer provides fault tolerance across rollout, environment, and training clusters, with rollout and environment recovery as its primary focus.The rollout and environment recovery mechanisms are described in Sections 4 and 5, respectively.
- Training recovery: For training failures, Belayer asynchronously checkpoints model parameters and optimizer state, then restores the learner from the latest checkpoint at a step boundary.Checkpointing occurs at the end of each training step and can overlap with rollout generation in the next step.
4 Fast Rollout-Worker Recovery
Belayer provides warm handover for detected rollout-worker fail-stop failures and stalls by reusing independently owned GPU allocations through a pre-initialized shadow worker. It restores interrupted inference through token-level context recovery while flushing and refilling potentially inconsistent KV-cache state.
- Failure model: Belayer’s warm handover targets detected worker-local fail-stop failures and stalls, excluding silent data corruption and fail-slow behavior.Examples include Python exceptions, process crashes, and worker hangs.
- State isolation: Belayer isolates model weights and KV-cache memory in independent servers, reusing them only after owner-process and GPU-health checks.The servers do not execute inference and expose CUDA IPC handles to rollout workers.
- Handover protocol: A pre-initialized shadow worker performs handover by checking health, flushing KV state, remapping CUDA-graph buffers, replacing the failed worker, and rerouting requests.The control plane terminates residual main-worker processes to prevent concurrent service by old and replacement workers.
- KV-cache recovery: Belayer flushes and refills the KV cache because the complete pool may be inconsistent after failure, avoiding fine-grained consistency tracking.The paper states that refill overhead is relatively small in RL workloads and leaves consistency tracking to future work.
- LLM context recovery: Token-level recovery records streamed tokens and metadata, updating the preserved prefix every k=256 tokens to recover long interrupted responses more finely than response-level checkpoints.This targets long reasoning segments in which a single response may contain thousands of tokens.
- Hardware-fault handling: Repeated shadow-worker failure shortly after handover marks the GPU unhealthy, excludes it from the cluster, and redistributes affected requests across healthy engines.Training continues on remaining healthy GPUs while Belayer alerts for manual intervention and uses token-level context recovery.
5 Environment Fault Tolerance
Belayer provides prefix-consistent recovery for container-local environment state under fail-stop failures, avoiding unsafe action replay by restoring a consistent trajectory prefix and regenerating subsequent actions. Its full-state checkpointing and adaptive overlap policy preserve correctness while limiting interference with failure-free training.
- Failure model and recovery goals: Belayer targets fail-stop environment failures and restores container-local file-system state, runtime state, and LLM context from the same ready trajectory prefix.The scope excludes external-system state through sandboxing.
- Adaptive checkpointing: Belayer launches checkpoints online when expected regeneration benefit exceeds visible overhead, opportunistically hiding checkpoint work inside natural LLM-response bubbles.The policy accounts for failure probability, regeneration cost, elapsed response time, and predicted checkpoint duration; timeout triggers a typically 20-second cooldown.
- Failure model and recovery goals: Unsafe action replay can duplicate persistent effects, lose observed effects, or interact with partially updated runtimes, so recovery rolls back to a consistent prefix and regenerates actions.This also handles effects that depend on nondeterministic inputs such as wall-clock time.
- Full-state checkpoint and restore: Belayer’s full_checkpoint jointly captures the container writable file-system layer and process runtime state at an action boundary, publishing a checkpoint only after both artifacts complete.The container is frozen during capture, and incomplete checkpoints are discarded.
- Full-state checkpoint and restore: full_restore creates a fresh container, installs the checkpointed file-system layer, and restores the runtime so resumed processes observe state from the same captured boundary.This preserves recovery alignment between the reconstructed environment and restored LLM context.
6 Implementation
Belayer is implemented on Slime, with rollout and shadow-worker components built on SGLang and environments on mini-swe-agent. Its implementation integrates selective shadow-worker initialization and asynchronous full checkpoint/restore into rollout execution.
- System implementation: Belayer is implemented on Slime v0.2.1, with approximately 6K lines of code, using SGLang v0.5.9 for rollout and shadow workers and mini-swe-agent for environments.Token-level context recovery is implemented in the router with streaming request handling.
- Shadow worker integration: The rollout engine integrates shadow-worker initialization through CUDA IPC weight handles and KV-cache pointers and metadata obtained from dedicated servers.This initialization path is integrated into Slime after implementing the weight and KV cache servers.
- Full checkpoint/restore integration: Full checkpoint and restore are implemented as a standalone module and integrated into rollout execution after checkpoint decisions or environment failures.Checkpointing runs asynchronously to overlap with LLM generation, with execution waiting for completion when necessary.
7 Evaluation
Belayer achieves near-baseline failure-free training performance and substantially improves recovery from rollout-worker and environment failures. Across tested failures, it preserves execution consistency while limiting training and rollout-time increases.
- 7.2 Interference-free execution: In Math configurations, Belayer’s measured end-to-end training times differ from the baseline by only 1%, indicating low failure-free overhead.Checkpoint and non-checkpoint scenarios show little difference in training time.
- 7.3 Rollout-worker recovery: Belayer’s shadow worker consumes only 1.84GB of GPU memory and maintains the main worker’s throughput, with approximately 22.8 seconds of extra initialization.The overhead includes CUDA context and communication group memory.
- 7.3 Rollout-worker recovery: Approximately 1 second is required to restart a failed rollout worker with Belayer, compared with tens of seconds for a cold restart.Cold-restart latency is dominated by process initialization and its associated engine components.
- 7.3 Rollout-worker recovery: After an injected rollout-worker software failure, Belayer increases training time by only 1.16%, versus 8.72% for the baseline.Token-level recovery further reduces the training-time increase by 3.46%.
- 7.4 Environment recovery: Across 593 active arbitrary-time fail-stop trials, all executions recovered to the golden final state, with no observed consistency counterexample.The protocol restored a ready checkpoint or the initial state, replayed remaining actions exactly once, and converged to the fault-free runtime and file-system state.
- 7.4 Environment recovery: Without environment-state recovery, rollout time increases by 48.7% on average, whereas Belayer’s full-state recovery incurs a 1.5% rollout-time increase.Restoring the latest ready checkpoint avoids regenerating the trajectory from the base image.
8 Discussion
Belayer’s recovery guarantees depend on Linux-container infrastructure and do not provide exactly-once semantics for effects outside the sandbox. Future work targets broader recovery coverage, more optimistic mechanisms, and hardware-failure handling.
- Limitations: Belayer’s environment recovery relies on Linux containers, Docker overlay storage, and CRIU, while supporting virtual machines or other isolation backends would require analogous snapshot mechanisms.The required mechanisms cover both storage and runtime state.
- Limitations: Belayer provides prefix-consistent recovery for container-local file-system state and CRIU-restorable runtime state, but not exactly-once semantics for effects outside the sandbox.The guarantee excludes external-service effects.
- Future work: Future work will investigate more optimistic recovery strategies and extend full-state recovery to environments interacting with external services.The paper also identifies flush-free KV-cache recovery for shadow workers as a future direction.
- Future work: Future work will address hardware failures through techniques such as live migration and elastic rollout scaling.These techniques are listed alongside flush-free KV-cache recovery for shadow workers.
9 Related Work
Prior work improves LLM RL training efficiency, failover latency, and inference-engine startup, while training fault-tolerance research largely targets conventional DNN or elastic-cluster settings. Belayer addresses the distinct recovery requirements of agentic RL by combining shadow workers, selective state reuse, and trajectory-consistent restoration.
- LLM RL training systems: LLM RL training systems orchestrate complex workflows and improve efficiency through asynchronous training, stage fusion, long-tail batching, and online context learning.
- Fast failover and LLM cold start: Primary-backup handover, pre-initialized standby workers, and partial-state recovery reduce failover latency by retaining replacement capacity or selected state.
- Fast failover and LLM cold start: Belayer combines pre-initialized shadow workers with independently owned GPU allocations, health checks, KV invalidation, and token-prefix reconstruction for failed rollout engines.
- LLM training fault-tolerance: Checkpointing and elastic-cluster methods support fault tolerance in DNN and LLM training, but mainly target training phases rather than agentic RL trajectory consistency.
10 Conclusion
Belayer is a fault-tolerant, efficient system for LLM agentic RL training that enables fast rollout recovery and prefix-consistent recovery of container-local environment state. Experiments show significantly reduced recovery time with low failure-free overhead under evaluated workloads.
- Belayer provides fault tolerance and efficiency for LLM agentic RL training.The paper presents Belayer as a system designed for this training setting.
- Belayer enables fast rollout recovery and prefix-consistent recovery of container-local environment state.Environment recovery uses composite file-system and runtime checkpoints.
- Experiments show significantly reduced recovery time with low failure-free overhead under the evaluated workloads.