PaperlayerView exact revision on arXiv

Source-linked AI summary

SOP: A Scalable Online Post-Training System for Vision-Language-Action Models

Mingjie Pan, Siyuan Feng, Qinglin Zhang, Xinchen Li, Jianheng Song, Chendi Qu, Yi Wang, Chuankang Li, Ziyu Xiong, Zhi Chen, Yi Liu, Jianlan Luo

arXiv:2601.03044v1cs.RO

TL;DR

VLA models generalize broadly but still need expert-level proficiency, while existing post-training is often offline, single-robot, or task-specific. SOP couples fleet-scale on-policy interaction with centralized online learning for multi-task post-training, improving large pretrained VLA models within hours while preserving a shared policy and near-linear fleet scaling. The paper argues that execution–learning coupling is a key complement to algorithmic progress.

  • Problem

    VLA models need expert-level task proficiency in addition to broad generalization, but existing post-training remains largely offline, single-robot, and task-specific.

  • Method

    SOP uses a closed-loop architecture in which robot fleets stream on-policy experience and human interventions to a centralized learner that asynchronously returns updated policies, supporting HG-DAgger and RECAP.

  • Results

    SOP substantially improves pretrained VLA performance across diverse manipulation tasks while maintaining one shared policy, achieving effective post-training within hours and near-linear scaling with fleet size.

  • Takeaways & Limitations

    Tightly coupling online learning with fleet-scale deployment supports efficient, reliable, and scalable post-training of generalist robot policies in the physical world.

Abstract

from arXiv · show

Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality. Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific, limiting effective on-policy adaptation and scalable learning from real-world interaction. We introduce a Scalable Online Post-training (SOP) system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world. SOP tightly couples execution and learning through a closed-loop architecture in which a fleet of robots continuously streams on-policy experience and human intervention signals to a centralized cloud learner, and asynchronously receives updated policies. This design supports prompt on-policy correction, scales experience collection through parallel deployment, and preserves generality during adaptation. SOP is agnostic to the choice of post-training algorithm; we instantiate it with both interactive imitation learning (HG-DAgger) and reinforcement learning (RECAP). Across a range of real-world manipulation tasks including cloth folding, box assembly, and grocery restocking, we show that SOP substantially improves the performance of large pretrained VLA models while maintaining a single shared policy across tasks. Effective post-training can be achieved within hours of real-world interaction, and performance scales near-linearly with the number of robots in the fleet. These results suggest that tightly coupling online learning with fleet-scale deployment is instrumental to enabling efficient, reliable, and scalable post-training of generalist robot policies in the physical world.

I. INTRODUCTION

SOP addresses the need for VLA models that combine broad generalization with expert-level proficiency by coupling fleet-scale real-world execution to centralized online learning. The system improves diverse manipulation tasks while preserving a single generalist policy and supports rapid, scalable adaptation.

  • Real-world deployment requires VLA systems to combine generalization across diverse tasks with expert-level proficiency in each setting.
  • SOP couples a robot fleet’s on-policy experience and human intervention signals with a centralized learner in a closed-loop collect–train–deploy architecture.The design targets timely on-policy correction, parallel experience collection, and low-latency adaptation.
  • SOP instantiates both HG-DAgger and RECAP, representing interactive imitation learning and reinforcement learning, respectively.
  • SOP substantially improves pretrained VLA performance across cloth folding, box assembly, and shelf restocking while jointly post-training one model across tasks.The shelf-restocking setting includes a large and diverse object set, and the reported gains preserve generality.
  • Effective post-training occurs on the order of hours rather than multi-day cycles, with near-linear fleet scaling and often 2× or greater success-rate improvements over non-SOP counterparts.Some tasks approach near-perfect performance, while long-horizon laundry folding and box assembly run for over 36 hours without degradation.
  • SOP frames shared fleet experience as a route toward scalable robot learning, where deployment and learning jointly refine a continuously evolving VLA.The authors suggest fleet-scale deployment complements advances in algorithms and data.

II. RELATED WORKS

Prior VLA post-training work is organized around online learning, distributed data collection, and multi-task training, while SOP integrates these axes into one generalist-policy framework. Its distinguishing approach is continuous online updating from distributed multitask experience rather than offline, single-robot, task-specific training.

  • SOP surveys related work across online learning, distributed data collection, and multi-task training because it integrates all three system dimensions.
  • Static supervised-finetuning datasets limit adaptation to distribution shift and improvement beyond demonstration coverage.
  • SOP’s actor–learner design streams on-policy rollouts to a cloud learner, mixes online and offline buffers, applies plug-in modules, and asynchronously broadcasts refreshed weights.
  • SOP directly updates one generalist policy through continuous online learning with distributed multitask data collection, preserving generality during performance improvement.
  • Online learning trains on states encountered by the current policy, and SOP extends this principle to parallel state-space coverage across robot fleets.

C. Distributed and Multi-task Robot Learning

SOP addresses the need to combine distributed robot learning with multi-task coverage for generalist VLA post-training in the physical world. It collects fleet experience and updates a shared policy through a post-training objective.

  • Distributed and Multi-task Robot Learning: SOP is framed as the first framework combining online, distributed, multi-task post-training for generalist models in the physical world.
  • Distributed and Multi-task Robot Learning: A heterogeneous robot fleet is modeled as multiple deployment domains, with each robot interacting under its own MDP while adapting a shared pretrained policy.
  • Distributed and Multi-task Robot Learning: SOP initializes an online buffer, broadcasts the pretrained policy to actors, samples batches from online and offline data, and streams updated policies back to robots.
  • Distributed and Multi-task Robot Learning: The training pipeline collects autonomous rollouts and human interventions, then updates policy parameters by minimizing an algorithm-specific post-training loss.

IV. SCALABLE ONLINE POST-TRAINING

SOP is a closed-loop actor–learner system that adapts pretrained VLA policies through continual real-world interaction. Distributed actors provide on-policy data, while a centralized learner mixes online and offline experience and asynchronously returns updated parameters.

  • SCALABLE ONLINE POST-TRAINING: SOP combines distributed on-policy collection, centralized cloud optimization, and low-latency synchronization in an algorithm-agnostic framework.
  • SCALABLE ONLINE POST-TRAINING: Each robot executes the latest policy locally, uploads trajectories to a shared online buffer, and operates in parallel with the other actors.
  • SCALABLE ONLINE POST-TRAINING: The cloud learner samples batches from a mixture of online and static offline buffers, applies a post-training algorithm, and asynchronously streams updated parameters to actors.
  • SCALABLE ONLINE POST-TRAINING: The online buffer aggregates episodes uploaded by all robots up to each wall-clock time, supporting fleet-wide experience for subsequent training batches.

B. System Infrastructure

SOP’s infrastructure supports asynchronous experience transfer, model synchronization, and task-balanced adaptive sampling. It preserves equal task coverage while adjusting online/offline data composition from recent losses.

  • System Infrastructure: Robot clients buffer episodes locally and upload them asynchronously to cloud storage, where episodes enter an online buffer consumed independently by the learner.
  • System Infrastructure: Updated checkpoints reach actors through publish–subscribe synchronization and are applied at safe episode boundaries to avoid mid-episode policy changes.
  • System Infrastructure: Task-balanced sampling assigns uniform weights across tasks while adjusting each task’s online/offline ratio using recent training losses.
  • System Infrastructure: Sliding-window online and offline loss estimates drive the adaptive mixture, with a boost factor prioritizing online data to accelerate adaptation under distribution shift.
  • System Infrastructure: The sampling strategy maintains equal task coverage while enabling loss-driven adaptation of the online/offline data mixture.

D. Post-training Learning Module

SOP separates system-level dataflow from the post-training algorithm, allowing existing methods to operate online through continuous streaming and asynchronous updates. The paper instantiates this design with HG-DAgger and RECAP.

  • Post-training Learning Module: SOP’s post-training module is algorithm-agnostic: any method that consumes logged experience and returns updated parameters can be integrated.
  • Post-training Learning Module: HG-DAgger uses real-time human interventions near failures to provide corrective supervision on difficult on-policy states with reduced effort versus full teleoperation.
  • Post-training Learning Module: Within SOP, HG-DAgger streams intervention segments, autonomous rollouts, and offline demonstrations into a shared buffer for frequent asynchronous updates.
  • Post-training Learning Module: RECAP is an offline RL method for improving large VLA policies from experience, including autonomous rollouts and optional human corrections.
  • Post-training Learning Module: SOP converts RECAP into an online workflow by continually adding trajectories from the latest policy and applying asynchronous updates on the evolving dataset.

V. EXPERIMENTAL EVALUATION

SOP is evaluated on three manipulation task families designed to test dexterous control, semantic understanding, and long-horizon execution. The evaluation measures success rate and throughput under defined trial and timing protocols.

  • Task families: The evaluation covers Grocery Restocking, Laundry Folding, and Box Assembly across three challenging manipulation task families.These tasks stress both fine-grained dexterity and semantic generalization.
  • Task families: Grocery Restocking tests semantic generalization across a cluttered retail environment with more than 500 pretrained objects and 40 sampled evaluation objects.The task includes restocking, picking, hanging, and item rearrangement across diverse shelf configurations.
  • Task families: Laundry Folding evaluates bimanual manipulation of a disordered T-shirt, requiring correct folding and placement within 500 s.
  • Task families: Box Assembly evaluates precise multi-step execution by folding a flat cardboard sheet into a 3D box within 300 s without errors.
  • Metrics and protocol: Laundry Folding and Box Assembly trials begin from randomized disordered states and focus quantitatively on the core folding or assembly skill rather than upstream preparation.
  • Metrics and protocol: Success rate is the fraction of successful episodes, while throughput counts completed episodes per hour and captures speed and reliability under fixed time budgets.Episodes terminate on success, failure, or timeout; policy-side throughput excludes human operation time.

B. Experiment Setup

The experiments compare SOP-based and non-SOP post-training across three task families, multiple actor counts, and pretrained models with different data scales. Results show broad performance gains, faster fleet-scaled learning, and a strong dependence on pretraining quality.

  • Experiment setup: A shared learner aggregates experience from 10 dual-arm robots distributed across Grocery Restocking, Laundry Folding, and Box Assembly.The fleet is partitioned as 4, 3, and 3 robots across the three tasks.
  • Multi-task post-training: SOP + HG-DAgger achieves success rates of 0.94, 0.96, and 0.98 across the three task families.Across all tasks, post-trained models outperform the pretrained baseline, and SOP improves both HG-DAgger and RECAP relative to their non-SOP counterparts.
  • Multi-task post-training: Throughput typically improves by approximately 2× across pretrained models when SOP targets deployed-policy failures through prompt on-policy correction.In laundry folding, on-policy feedback corrects repeated missed grasps and reduces cycle time.
  • Fleet scaling: Increasing active actors from one to four raises final success after 180 minutes from 0.805 to 0.925.Parallel data collection provides more diverse on-policy experience and reduces sensitivity to station-specific noise.
  • Fleet scaling: Fleet scaling improves both final performance and wall-clock learning efficiency, supporting favorable scaling of data efficiency and training time.The reported time-to-target metric measures the time to first reach success level 0.8.
  • Pretraining quality and data efficiency: SOP improves all pretrained variants, but models trained on larger pretraining datasets start higher and reach higher asymptotic performance.The compared variants use 1/8, 1/2, or the full pretraining data scale, with the full dataset comprising approximately 160 hours.

VI. DISCUSSION AND FUTURE WORK

SOP’s closed-loop coupling of execution and learning makes on-policy experience central to efficient post-training, while its scaling and generality boundaries remain open. The discussion also reports that larger pretraining datasets improve both initial and final performance.

  • Discussion: SOP’s system-level coupling of execution and learning is described as critical to post-training success, beyond the choice of underlying algorithm.The closed loop streams on-policy experience to the learner and updated policies back to robots.
  • Discussion: On-policy correction provides greater marginal utility than additional offline data, reflecting the deployed policy’s induced state distribution.The discussion contrasts dynamic on-policy experience with static datasets that cannot fully anticipate deployment states.
  • Future Work: SOP currently relies on human interventions or task-specific rewards, motivating learned reward models and foundation-model-based success detection.Reducing this supervisory burden is identified as an important direction.
  • Future Work: Whether near-linear fleet scaling extends to much larger fleets and supports continual skill acquisition without catastrophic forgetting remains open.These are explicitly identified as unresolved questions.
  • Discussion: Larger pretraining datasets yield higher initial success and higher final performance after SOP online post-training.The comparison is summarized in Figure 5.

APPENDIX

The appendix describes a distributed robot infrastructure that asynchronously moves episodes from actors to a cloud learner and broadcasts updated policies back to the fleet. Its design supports independent scaling, fault tolerance, and memory-efficient sampling.

  • Data Infrastructure Details: Each robot buffers observations into episodes, uploads completed episodes to object storage, and publishes message-queue notifications.The actor-side client performs serialization and asynchronous upload at episode termination.
  • Data Infrastructure Details: The learner consumes notifications, retrieves episodes, indexes frame-level metadata, and fetches payload data on demand for training.This separates lightweight sampling metadata from the corresponding heavy episode contents.
  • Data Infrastructure Details: Updated model parameters are broadcast through a publish–subscribe channel so actors refresh local policies at episode boundaries without interrupting collection.This completes the actor–learner feedback loop.
  • Data Infrastructure Details: Separating actor-side production from learner-side consumption lets both pipelines scale independently, while the message queue absorbs transient load imbalance and network disruption.The queue functions as the decoupling layer between data generation and consumption.
  • Data Infrastructure Details: New robot actors can join through the message queue without code changes or system reconfiguration, enabling scaling from one robot to fleets of hundreds.Consumer groups provide automatic discovery and load balancing.
  • Data Infrastructure Details: Durable object storage with atomic writes and guaranteed-delivery queue retries protect episode integrity during network partitions or node failures.The appendix identifies this reliability as important for long-running training campaigns.
  • Data Infrastructure Details: Keeping metadata in memory and lazily loading frame payloads reduces memory footprint by over two orders of magnitude compared with full data loading.The design targets efficient sampling at million-scale episode volumes.

C. Implementation Details

Implementation initializes SOP from a pretrained multi-task VLA policy and uses a centralized learner to update selected vision and action components. RECAP is adapted for multi-task post-training with task-dependent advantage thresholds, while ablations compare single-task and multi-task variants.

  • Implementation Details: The initial policy πθ0 is obtained by tuning π0.5 on about 160 hours of multi-task robot data.The dataset includes 100 hours of Grocery Restocking, 30 hours of Laundry Folding, and 30 hours of Box Assembly.
  • Implementation Details: The cloud learner uses 8 NVIDIA H100 GPUs and publishes updated parameters every 25 training steps.Robot actors refresh their local policies from the latest published checkpoint.
  • Implementation Details: Post-training freezes the LLM backbone and updates the vision components and action experts, with approximately 780 MB transmitted per checkpoint.Only the necessary updated weights are distributed to keep deployment practical.
  • Implementation Details: RECAP is implemented as a multi-task post-training recipe rather than task-specific fine-tuning.This design supports a generalist policy across tasks.
  • Implementation Details: RECAP uses task-dependent advantage thresholds because episode lengths vary across tasks.The advantage condition is approximated by a value function relative to the behavior policy πref.
  • Implementation Details: The SOP-RECAP setup updates vision components and the action expert while keeping the pretrained value function fixed during policy training.The corresponding RECAP-alone comparison uses two iterations.
  • Implementation Details: Single-task and multi-task RECAP achieve 0.75 versus 0.75 success on freezer restocking and 0.86 versus 0.8 on open-cooler restocking.These ablations evaluate two tasks within the Grocery Restocking category.
Loading 2601.03044v1…