PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies

Yi Wang, Xinchen Li, Pengwei Xie, Pu Yang, Buqing Nie, Yunuo Cai, Qinglin Zhang, Chendi Qu, Jeffrey Wu, Jianheng Song, Xinlin Ren, Jingshun Huang, Mingjie Pan, Siyuan Feng, Zhi Chen, Jianlan Luo

arXiv:2605.00416v2cs.RO

TL;DR

Generalist robot policies need continual learning from diverse deployment experience because fixed offline demonstrations incompletely capture real-world variation and failures. LWD uses fleet-scale offline-to-online reinforcement learning with DIVL and QAM, reaching 0.95 performance across evaluated tasks and strongest gains on long-horizon tasks.

  • Problem

    Fixed demonstration datasets provide limited coverage of deployment distributions, task variations, failures, recoveries, and human interventions for generalist robot policies.

  • Method

    LWD continually improves one shared generalist VLA policy through fleet-collected autonomous rollouts and interventions, using DIVL for value learning and QAM for policy extraction.

  • Results

    LWD reached 0.95 and outperformed all baselines across evaluated tasks, with larger gains on long-horizon tasks.

  • Takeaways & Limitations

    Within the evaluated scope, deployment serves both as the setting for policy evaluation and as a source of data for continued cross-task improvement.

  • Takeaways & Limitations

    The current policy learning framework does not explicitly model execution safety, leaving safety-aware learning and control as important future work.

Abstract

from arXiv · show

Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.

I. INTRODUCTION · II. RELATED WORK · A. Post-Training of Robot Generalist Policies

LWD reframes robot deployment as a continual offline-to-online learning loop in which a fleet aggregates real-world experience to improve and redeploy a shared generalist VLA policy. It combines distributional value learning with QAM-based policy extraction to adapt generalist policies across diverse real-world tasks, including long-horizon settings.

  • I. INTRODUCTION: Offline pretraining alone cannot make generalist VLA policies deployment-ready because real-world operation introduces distribution shifts, long-tail failures, task variations, and human correction opportunities.
  • I. INTRODUCTION: Fleet-scale deployment creates a closed-loop data flywheel: robots collect target-distribution experience, the shared policy improves from aggregated data, and the improved policy is redeployed.
  • I. INTRODUCTION: LWD combines DIVL for distributional value estimation with QAM for stable policy extraction in flow-based VLA action generators.DIVL addresses heterogeneous, asynchronous, multi-modal, and heavy-tailed returns, while QAM converts critic gradients into step-wise supervision for the flow model.
  • I. INTRODUCTION: The system uses the same RL objective during offline pretraining and rapid online finetuning, reducing offline-to-online mismatch in critic calibration and value extrapolation.
  • II. RELATED WORK: The related-work landscape includes large-scale pretraining of robot generalist policies followed by offline, online, or combined offline-to-online post-training strategies.
  • A. Post-Training of Robot Generalist Policies: Prior offline post-training methods improve policies from stored rollouts but cannot immediately use deployment experience, slowing adaptation to distribution shifts.π∗0.6 combines offline value learning with iterative offline RL, while RLDG uses specialist RL data for policy distillation.
  • A. Post-Training of Robot Generalist Policies: Online RL methods can improve specialist policies but typically depend on costly, sample-inefficient on-policy collection, whereas LWD uses offline data and off-policy online replay.
  • A. Post-Training of Robot Generalist Policies: Prior offline-to-online methods generally learn task-specific specialists, while LWD post-trains a single generalist policy across multiple real-world tasks, including long-horizon tasks with sparse rewards.

B. Offline-to-Online Reinforcement Learning … B. Implicit Q-Learning

LWD combines fleet-scale offline-to-online reinforcement learning with a shared generalist VLA policy, using deployment experience to drive continual policy improvement. Its formulation uses sparse-reward, chunk-based control and extends implicit Q-learning with distributional value estimation and quantile extraction.

  • B. Offline-to-Online Reinforcement Learning: LWD trains a shared generalist policy across diverse tasks with a unified offline-to-online objective, addressing the task specificity, inconsistent stages, and limited scale of prior methods.The framework combines offline and online replay in one learning loop to improve stability and scalability.
  • C. Large-Scale Robotic RL Systems: Fleet-scale deployment creates an RL-driven data flywheel in which distributed robot experience continually supplies data for centralized policy improvement.Large-scale robotic RL systems aggregate experience from distributed actors and train centralized learners, extending beyond isolated task-level collection.
  • A. Problem Setting and Notation: LWD formulates control as a Markov decision process M = (S, A, T, r, γ), with language-conditioned states and task indices k ∈ K.Each state contains a robot observation and language instruction, with high-level commands used for long-horizon tasks.
  • A. Problem Setting and Notation: The framework uses sparse binary rewards, assigning r = 1 only when an episode terminates successfully.This reward design is used for the task formulation described in the paper.
  • A. Problem Setting and Notation: The generalist VLA policy outputs action chunks conditioned on the current state, executes each chunk before replanning, and operates across all tasks.The policy is shared across tasks and represented by πθ(· | st).
  • A. Problem Setting and Notation: Replay samples use chunk transitions (st, at, rt, st+H), with offline data from Boff and online data from mixed replays in Boff ∪ Bon.Both the policy and critic operate on action chunks throughout the method.
  • B. Implicit Q-Learning: Implicit Q-learning fits a scalar state-value function to a high expectile of dataset action-values, avoiding explicit action maximization.For τ > 1/2, the estimate favors higher-valued dataset actions and provides an implicit improvement target without a maxa Q(s, a) backup.
  • B. Implicit Q-Learning: LWD retains IQL’s asymmetric bootstrap principle but replaces scalar expectile regression with a distributional value model and quantile-based value extraction.The critic target uses a target network updated by exponential moving average.

C. Flow Matching and Q-learning with Adjoint Matching · IV. LEARNING WHILE DEPLOYING · A. Distributional Implicit Value Learning

LWD combines fleet-scale offline-to-online training with DIVL value learning and QAM policy extraction for flow-based VLA policies. DIVL models replay action-value distributions, extracts uncertainty-aware quantile targets, and preserves the optimum of asymmetric scalar value learning.

  • C. Flow Matching and Q-learning with Adjoint Matching: QAM extracts policies from flow-based VLA generators by combining TD critic learning with adjoint-matching updates against a KL-regularized improvement target.It avoids costly and potentially unstable direct critic backpropagation through multi-step generation.
  • C. Flow Matching and Q-learning with Adjoint Matching: LWD uses QAM to update the flow policy with local regression targets formed from the critic learned by DIVL.The flow policy is optimized along trajectories of a pretrained reference flow.
  • IV. LEARNING WHILE DEPLOYING: LWD alternates offline initialization with online fleet deployment, mixed replay learning, and periodic redeployment of the updated policy.Offline training uses Boff, while online actors add policy transitions and optional human interventions to Bon; updates use Boff ∪ Bon.
  • A. Distributional Implicit Value Learning: DIVL trains the critic Qϕ and distributional value model Vψ, using a replay action-value distribution instead of a single scalar expectile target.A quantile of the learned distribution bootstraps the chunk-level critic while retaining IQL’s asymmetric bootstrap principle.
  • A. Distributional Implicit Value Learning: The τ-quantile provides an in-distribution optimistic bootstrap over replay actions, favoring high-value data actions without an explicit full-action-space max backup.DIVL realizes the asymmetric value-learning principle through distribution modeling and quantile extraction.
  • A. Distributional Implicit Value Learning: DIVL’s two-step distribution fitting and asymmetric-statistic extraction has the same optimal scalar value as corresponding direct asymmetric regression.This equivalence holds for any fixed asymmetric loss in the stated family; p = 2 corresponds to IQL’s expectile and p = 1 to DIVL’s quantile.
  • A. Distributional Implicit Value Learning: Uncertainty-adaptive τ schedules lower optimism for diffuse value distributions and retain more optimistic targets for concentrated distributions.The schedule uses normalized categorical-distribution entropy, with τbase for confident states and α ≥0 controlling uncertainty sensitivity; τ(st+H) is stop-gradient in the TD target.

B. Policy Extraction via QAM

LWD extracts policies from flow-matching VLAs with QAM, avoiding costly likelihood evaluation and unstable full-process critic backpropagation. QAM uses DIVL’s reward-informed action gradient to guide local regression along reference flow trajectories.

  • Policy Extraction via QAM: Existing advantage-weighted regression is poorly matched to flow-based VLAs because it requires action-chunk likelihoods under multi-step denoising.This motivates a policy-extraction method that avoids differentiating through those likelihood calculations.
  • Policy Extraction via QAM: Directly backpropagating ∇aQϕ(s, a) through multi-step flow generation is computationally expensive and numerically unstable.These limitations make direct critic backpropagation difficult for large VLA policies.
  • Policy Extraction via QAM: QAM reformulates trajectory-level policy optimization as a local regression objective along the reference flow.The DIVL critic initializes the terminal adjoint state, which guides refinement of the policy vector field.
  • Policy Extraction via QAM: During offline and online training, fβ remains fixed while fθ is optimized toward adjoint-derived local targets.Each minibatch samples states and Gaussian noise, generates reference trajectories with fβ, evaluates the endpoint action gradient, solves adjoint dynamics, and regresses fθ.

C. Offline to Online RL Training Pipeline

The LWD pipeline performs offline initialization from heterogeneous replay, then continuously improves the policy through asynchronous fleet rollouts, human interventions, and mixed offline-online learning. It uses longer chunk-level targets offline for sparse long-horizon rewards but reverts to one-step targets online.

  • Offline stage: Offline training initializes the policy, value models, and critic from demonstrations, historical-policy rollouts, and human-guided failure exploration stored in Boff.These sources are converted into a common chunked transition format for learning.
  • Offline stage: n = 1 for short tasks and n = 10 for long-horizon tasks, with terminal truncation removing bootstrapping when episodes end within the target window.The n-step chunk-level target accelerates sparse reward propagation through fixed offline replay.
  • Online stage: Online training uses 1-step chunk-level TD targets because multi-step targets are less effective when trajectories combine policy actions with human interventions.Offline initialization of the critic and value model makes one-step targets sufficient for online updates.
  • Online stage: The online stage deploys the current policy across the robot fleet, collecting autonomous transitions and human corrective actions asynchronously in Bon.Rollouts terminate on time limits, failures, or successes, and intervention segments are stored as regular replay transitions.
  • Online stage: Online updates sample mixed replay from Boff ∪ Bon, apply the same value-learning and policy-extraction objectives, and periodically redeploy the latest checkpoint to every robot.The central learner and robot actors operate asynchronously, with deployment controlled by the actor-sync period Nsync.

D. Architectures

LWD separates policy action generation from centralized value and critic learning, using shared multimodal backbones with task-specific heads. Offline training fine-tunes all networks, while online QAM updates freeze the policy VLM backbone and update only the action expert.

  • Module separation: LWD uses separate policy and value/critic modules, distributing only the policy checkpoint asynchronously to the robot fleet while retaining value and critic networks centrally.This isolates action generation from value and critic optimization.
  • Value and critic networks: The value and critic networks share a Gemma3–SigLIP VLM backbone but use separate prediction heads initialized alongside scratch-trained visual projection layers.The language and vision components initialize from Gemma 3-270M-IT and SigLIP-So400M checkpoints.
  • Value and critic networks: The value head predicts logits over a fixed categorical support, while the critic uses state and action-chunk representations with clipped double-Q estimates.The critic encodes action chunks through learned temporal attention pooling and uses the minimum critic estimate for DIVL target construction and TD backups.
  • Actor: The actor follows a flow-based VLA architecture combining a PaliGemma vision-language backbone with a Gemma-300M action expert for flow-based action generation.The backbone uses a Gemma-2B language model and a SigLIP vision encoder.
  • Training updates: During online QAM updates, the policy VLM backbone is frozen and only the action expert is updated, while value and critic networks remain fully fine-tuned on mixed replay.Offline training fully fine-tunes the actor and value/critic networks, whose resulting weights initialize online training.

V. EXPERIMENTAL EVALUATIONS … B. Main Results

LWD is evaluated across eight real-world manipulation tasks on a 16-robot dual-arm fleet against static and post-training baselines. Online learning reaches a 0.95 average score, with especially strong gains on long-horizon tasks and reduced cycle time.

  • 1) Tasks, Evaluation, and Robots:: The evaluation covers eight real-world tasks: four grocery-restocking scenarios and four 3–5-minute long-horizon tasks with 5–8 annotated subtasks.Long-horizon tasks include Gongfu Tea, Fruit Juice, Cocktail, and Shoebox packing, requiring multi-stage planning, manipulation, and recovery.
  • 1) Tasks, Evaluation, and Robots:: Grocery tasks use binary success rates, whereas long-horizon tasks use step-wise scores assigning 1 for autonomous success and 0.5 for minor intervention.The protocols distinguish language-following and timed completion from annotated-substep performance.
  • 1) Tasks, Evaluation, and Robots:: Experiments use Agibot G1 dual-arm robots with two 7-DoF arms, parallel-jaw grippers, three RGB cameras, and 30 Hz joint-position control.A fleet of 16 robots collects concurrent online rollouts: four for grocery tasks and three for each long-horizon task.
  • 2) Baselines and Reference Policies:: LWD is compared with SFT, RECAP, and HG-DAgger, which respectively use demonstrations, autonomous-rollout post-training, and online successful-rollout training.RECAP and HG-DAgger both start from the reference policy, while SFT uses standard flow-matching loss.
  • B. Main Results: 0.95 is LWD (Online)’s average score across all eight tasks, outperforming all baselines while maintaining strong short-horizon and long-horizon performance.Table I reports task-level results and the average across all eight tasks.
  • B. Main Results: 0.91 is LWD (Online)’s average long-horizon step-wise score, versus 0.68 for SFT, 0.77 for RECAP, 0.73 for HG-DAgger, and 0.79 for LWD (Offline).The improvement is attributed to an offline-to-online RL pipeline using successful demonstrations, play data, and successful and failed online trajectories.
  • B. Main Results: LWD (Online) remains at or near the best result on every grocery task, despite limited improvement room because most methods already achieve high scores.This indicates benefits beyond long-horizon tasks while preserving shared-policy generalist behavior.
  • B. Main Results: 23.75 seconds is the reduction in mean long-horizon cycle time versus the reference policy, as critic-guided updates favor action chunks that make reliable progress.The value estimate rises with successful task progress and remains lower after failures stop advancing annotated milestones.

C. Ablation Study … VI. CONCLUSION

The ablations show that DIVL and adaptive-τ value learning improve robustness, particularly for long-horizon tasks. LWD then extends this design into fleet-scale deployment, where shared interaction data supports continual generalist-policy improvement while leaving update efficiency and language-driven task decomposition as limitations.

  • 1) Value Learning Design:: DIVL outperforms scalar expectile value regression on all tasks, with gains of 9.7% offline and 16.7% online for long-horizon tasks.The comparison keeps all other components fixed; scalar values can blur heterogeneous successful, failed, and intervention outcomes.
  • 2) Adaptive τ Strategy:: Adaptive τ improves average offline performance over a constant-τ baseline while producing more consistent gains across tasks.The constant baseline uses τ = 0.52, the empirical average from the adaptive-τ run.
  • 2) Adaptive τ Strategy:: Conditioning τ on distributional entropy calibrates bootstrap optimism, making targets more conservative under high uncertainty and more optimistic when value estimates are confident.The strongest task-specific gains occur on Restocking, Correction, and Cocktail.
  • VI. CONCLUSION: LWD initializes a generalist policy from previously collected robot data and improves it through online reinforcement learning during deployment.The framework combines DIVL for value learning with QAM for policy extraction.
  • VI. CONCLUSION: Across eight real-world manipulation tasks spanning grocery restocking and long-horizon manipulation, LWD delivers the best overall performance.The conclusion characterizes LWD as a large-scale real-world reinforcement-learning framework for post-training generalist robot policies.
  • VI. CONCLUSION: Fleet deployment turns evaluation into learning by aggregating interaction data into a shared process that enables one generalist policy to improve across heterogeneous tasks.This provides a practical path toward continuously improving robot systems in unstructured environments.
  • VI. CONCLUSION: The online pipeline uses a straightforward real-time update schedule that may be suboptimal for larger-scale deployment or long-term continual improvement.More efficient and stable update strategies remain an important direction for future work.
  • VI. CONCLUSION: Long-horizon experiments rely on a single short language instruction per task, whereas complex tasks require stronger vision-language reasoning for task decomposition.This is identified as a limitation of the current method.

APPENDIX … 2) Proof of the Distributional View of Asymmetric Value Estimation:

The appendix specifies DIVL’s categorical distributional value representation and proves that, under idealized assumptions, distribution fitting followed by asymmetric statistic extraction is equivalent to direct asymmetric value optimization.

  • 1) Discretization of Distributional Value Model:: The distributional value model uses a fixed categorical support spanning [vmin, vmax], with the value head predicting logits over K atoms.The policy defines pψ(i | s) by applying softmax to the value-head logits.
  • 1) Discretization of Distributional Value Model:: K = 201 atoms over [−0.1, 1.1] are used in the real-robot experiments.
  • 1) Discretization of Distributional Value Model:: Replay scalar targets are clipped to [vmin, vmax] and linearly projected onto neighboring atoms using the C51 projection, producing a target distribution for cross-entropy training.
  • 2) Proof of the Distributional View of Asymmetric Value Estimation:: Under idealized conditions, direct asymmetric optimization and fitting the state-conditioned distribution of dataset Q-values before extracting its statistic yield the same optimal scalar value.
  • 2) Proof of the Distributional View of Asymmetric Value Estimation:: At optimum, DIVL recovers the pushforward distribution of dataset Q-values induced by sampling actions from D(· | s) and mapping them through Q(s, a).
  • 2) Proof of the Distributional View of Asymmetric Value Estimation:: The direct and distributional objectives have the same minimizer because their optimality conditions coincide after changing variables from actions to Q-values.
  • 2) Proof of the Distributional View of Asymmetric Value Estimation:: p = 2 recovers the expectile statistic used in standard IQL, while p = 1 recovers the quantile statistic used by DIVL.

3) Analysis of Direct Backpropagation for Flow-Based Policy: … 2) Training Hyperparameters:

The analysis shows that direct policy-gradient fine-tuning for flow-based policies requires differentiating through the full ODE trajectory, making optimization expensive and numerically fragile. The implementation uses heterogeneous offline data and specified flow-policy, value-learning, temporal-difference, and online-mixture hyperparameters.

  • 3) Analysis of Direct Backpropagation for Flow-Based Policy:: The flow-based policy generates action x_1 by integrating vector field f_θ(x_t, t) from t = 0 to 1, starting from x_0 ∼ N.The terminal sample is written as x_1(x_0; θ).
  • 3) Analysis of Direct Backpropagation for Flow-Based Policy:: Vanilla policy gradient requires differentiating through the entire ODE trajectory and its sensitivity matrix along the flow.This connects reward fine-tuning to trajectory-level derivatives of the generated action.
  • 3) Analysis of Direct Backpropagation for Flow-Based Policy:: Direct backpropagation is computationally expensive and numerically fragile because it backpropagates through the full ODE solver.Adjoint Matching instead reformulates trajectory-level optimization as local regression targets along the flow path.
  • B. Implementation and Training Details: The offline buffer combines successful human demonstrations, historical-policy rollouts containing successes and failures, and unsuccessful human exploratory play data.These sources cover demonstrations, prior evaluations, and failure modes or edge cases.
  • 2) Training Hyperparameters:: The policy emits action chunks with H = 30 and uses AdamW at 2 × 10^-5 with cosine decay, while value and critic networks use Adam at 5 × 10^-4 with cosine decay.The passage specifies separate optimizers and base learning rates for policy versus value and critic networks.
  • 2) Training Hyperparameters:: γ = 0.9999, DIVL uses α = 0.3 with τbase = 0.6 offline and 0.9 online, EMA updates use 0.005, and QAM temperature is λ = 2.The α value remains 0.3 across offline and online training.
  • 2) Training Hyperparameters:: Offline value learning uses 10-step chunk-level TD for long-horizon tasks and 1-step TD for grocery restocking, whereas online training uses 1-step TD for all tasks.Each online learner update samples from Boff ∪ Bon with an approximately balanced 1:1 ratio.

3) Checkpoint Initialization: · C. Additional Experimental Details · 1) Reference Policy and Baseline Implementations:

The paper initializes LWD from a behavior-cloned π0.5 VLA checkpoint, then separates offline and online continuation through DIVL-based value learning and Adjoint Matching. It benchmarks against a supervised-fine-tuned reference policy and adapted RECAP and HG-DAgger baselines using shared data, task, and optimization conditions.

  • 3) Checkpoint Initialization:: LWD (Offline) starts from a behavior-cloned pretrained π0.5 VLA checkpoint and jointly trains its policy with Adjoint Matching and its critic and distributional value model with DIVL.The imitation-learning checkpoint is adapted from pretrained π0.5 using demonstration data.
  • 3) Checkpoint Initialization:: LWD (Online) initializes from the LWD (Offline) checkpoint, retaining both policy and value-learning modules while training on mixed offline-online replay.This stage continues the offline-to-online training process rather than restarting from the reference policy.
  • 1) Reference Policy and Baseline Implementations:: 336.6 hours of demonstration data are used to supervised-fine-tune pretrained π0.5 into the reference policy with a flow-matching objective.The objective trains a conditional vector field to match the velocity a1−a0 for interpolated noisy actions aw.
  • 1) Reference Policy and Baseline Implementations:: 652.5 hours of offline data comprise 18.8% Grocery Restocking and 81.2% Long-Horizon tasks, with 65.2% successful and 34.8% failed data by source outcome.The buffer combines expert demonstrations, historical-policy rollouts, and human-guided failure-mode play.
  • 1) Reference Policy and Baseline Implementations:: The reference policy is used for all post-training methods, including RECAP and HG-DAgger, which are initialized from the same checkpoint.RECAP is adapted to the eight-task generalist setting, while HG-DAgger performs interactive imitation learning on the same suite.
  • 1) Reference Policy and Baseline Implementations:: RECAP collects two approximately 60 robot-hour rollout rounds across eight tasks, trains on demonstrations plus rollouts, and treats human interventions or corrections as positive examples.Round 1 uses the SFT checkpoint and Round 2 uses the RECAP checkpoint trained on Round 1; RECAP uses H = 30 and a shared threshold selecting 30% positive-advantage transitions.
  • 1) Reference Policy and Baseline Implementations:: HG-DAgger aggregates human intervention segments and autonomous rollouts into an approximately 60 robot-hour online buffer pooled across all eight real-world tasks.The online buffer is combined with the offline demonstration-data buffer for training.
  • C. Additional Experimental Details: The post-training baselines use the same policy optimizer and learning-rate schedule as LWD for fair comparison.This controls optimization settings across the compared post-training methods.

2) Complete Value-Estimation Ablation Results: … 2) Operational Latency:

The paper reports complete value-estimation ablations, qualitative DIVL value-distribution behavior, and a distributed fleet infrastructure designed for reliable episode delivery and timely actor–learner updates. The infrastructure uses versioned snapshots, scalable ingestion and learning, at-least-once delivery, and two operational latency measures.

  • 2) Complete Value-Estimation Ablation Results:: The value-learning ablation replaces DIVL with scalar expectile value regression while keeping the remaining training setup fixed.The complete comparison covers grocery-restocking and long-horizon tasks under offline and online settings, reporting task success rates for each task and the eight-task average.
  • 3) Complementary Qualitative Results of DIVL:: DIVL’s predicted value distribution is unimodal on a successful episode, with its mode increasing from approximately 0.4 to 1.0 as the task progresses.On a failure episode, the mode increases only from approximately 0.5 to 0.6 before plateauing, providing a fine-grained signal of policy progress.
  • D. Distributed Data Infrastructure: LWD links fleet robot actors to a multi-host learner through a versioned-snapshot data plane.Actors upload complete episodes to distributed object storage, persist metadata, and publish event notifications; a Coordinator commits monotonically increasing snapshots that define each training-data view.
  • D. Distributed Data Infrastructure: The learner uses a multi-host SPMD JAX program with one Distributed Replay Buffer Reader per node and parallel prefetching from object storage.One prefetcher per node is sufficient to saturate the per-node read bandwidth available from the distributed filesystem in the reported deployment.
  • D. Distributed Data Infrastructure: Updated model parameters fan out through a publish-subscribe channel, and robot actors reload the new policy at episode boundaries.The Coordinator is the only orchestration singleton, while the actor fleet and learner scale independently.
  • 1) End-to-End Reliability:: The system provides at-least-once end-to-end delivery for every actor-produced episode.Atomic object-storage uploads are retried, metadata uses transactional insertion and acknowledged durable-queue publication, and prefetcher download tasks are requeued on failure.
  • 2) Operational Latency:: The actor–learner loop is evaluated using episode-to-learner and model-to-actor latency on the same 8-hour, 16-actor online-RL run.Episode-to-learner measures availability for learner sampling after production, while model-to-actor measures policy loading before the next rollout; both are dominated by object-storage I/O on the actor-to-cloud link.
Loading 2605.00416v2…