Source-linked AI summary
Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Yuqi Xie, Fengyuan Hu, Jimmy Wu, Zhengyi Luo, Linxi "Jim" Fan, Guanya Shi, Yuke Zhu
TL;DR
Human demonstrations make VLA post-training costly and leave deployment-distribution coverage gaps. PLD uses residual RL to probe failures, distribution-aware hybrid rollouts to collect recovery data, and SFT to distill it back into the generalist, achieving strong simulation and real-world results without additional oracle demonstrations.
Problem
Costly human demonstrations limit scalable VLA post-training, while teleoperation data can be decoupled from deployed policies and leave coverage gaps.
Method
PLD freezes a VLA generalist, trains lightweight residual RL specialists, collects hybrid generalist-specialist recovery trajectories, and distills them through SFT.
Results
PLD improves seen and unseen-task performance across simulation and real-world manipulation, including near-saturated ∼99% LIBERO success, >50% SimplerEnv gains, and robust hardware results.
Takeaways & Limitations
Residual probing and distribution-aware replay are identified as key to stability, sample efficiency, and generalization, supporting autonomous and scalable VLA post-training.
Takeaways & Limitations
The RL setting assumes sparse binary rewards for language-conditioned manipulation, and the paper reports an initial PLD exploration performance drop.
Abstract
from arXiv · showhide
Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose Probe, Learn, Distill (PLD), a three-stage plug-and-play framework that improves VLAs through residual reinforcement learning (RL) and distribution-aware data collection. In Stage 1, we train lightweight residual actors to probe failure regions of the VLA generalist. In Stage 2, we use a hybrid rollout scheme that aligns collected trajectories with the generalist's deployment distribution while capturing recovery behaviors. In Stage 3, we distill the curated trajectories back into the generalist with standard SFT. PLD achieves near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world Franka and YAM arm manipulation tasks. Ablations show that residual probing and distribution-aware replay are key to collecting deployment-aligned data that improves both seen and unseen tasks, offering a scalable path toward self-improving VLA models.
1 INTRODUCTION
PLD addresses the cost and deployment-distribution gaps of human-demonstration-based VLA fine-tuning with a three-stage residual-RL and SFT pipeline. It autonomously collects recovery-aware, generalist-aligned data and reports strong simulation and real-world results without additional oracle demonstrations.
- Motivation: The motivating gap is that teleoperated demonstrations are costly and may be decoupled from deployed VLA policies, leaving coverage gaps.Independent RL experts can additionally suffer distributional mismatch and insufficient behavioral diversity for robust SFT coverage.
- Method: PLD trains lightweight residual actors with off-policy RL while freezing the VLA generalist, then uses them to collect task-specific data.The residual actors can take over at arbitrary states and are trained for multiple tasks.
- Method: Hybrid rollouts first follow the VLA generalist and then transfer control to specialists, preserving deployment-aligned behavior while collecting recovery trajectories.The scheme biases takeovers toward states frequently visited by the base model to mitigate distribution shift.
- Method: PLD distills the collected multi-task trajectories back into the generalist through architecture-agnostic supervised fine-tuning.The reported implementation supports both flow-matching and autoregressive action heads.
- Results: 99% success on LIBERO and over 50% performance gains in SimplerEnv demonstrate improvements on seen and unseen tasks without additional oracle demonstrations.The contribution statement frames these results as an autonomous post-training recipe.
- Results: PLD also reports over 99% success on Franka and YAM dexterous manipulation tasks, including one hour of continuous GPU insertion and unplugging without human intervention.These results extend validation from simulation to real robot hardware.
2 PRELIMINARIES
The paper formulates sparse-reward, language-conditioned manipulation as goal-conditioned continuous control with VLA policies mapping visual-proprioceptive observations and language goals to actions. It reviews SFT objectives across action-head families and presents the MDP and stationary-goal formulation used for RL.
- 2.1 TASK FORMULATION: The task uses partial RGB and proprioceptive observations plus a language goal, with the VLA outputting a 7-DoF action.The action comprises a 6-DoF delta pose and a 1-DoF continuous gripper command.
- 2.1 TASK FORMULATION: VLA policies combine a vision-language backbone with an action head, which may be diffusion- or flow-based for continuous control or token-based for autoregressive decoding.The paper aims to maximize success rate by tuning both backbone and action-head parameters.
- 2.2 SUPERVISED FINE-TUNING: SFT adapts a VLA by maximizing conditional action likelihood on demonstrations, with the loss determined by the action-head architecture.Autoregressive heads use sequence negative log-likelihood over action tokens.
- 2.2 SUPERVISED FINE-TUNING: Diffusion heads learn conditional denoising through score-matching MSE, while flow-matching heads learn a continuous velocity field with an L2 flow-matching loss.Both are continuous-control formulations, but they parameterize action generation differently.
- 2.3 GOAL-CONDITIONED RL: The RL formulation is a goal-conditioned MDP with state transitions, initial-state distribution, reward, discount, and stationary goals.The goal-augmented transition preserves the goal across steps.
- 2.3 GOAL-CONDITIONED RL: The considered manipulation setting uses sparse binary rewards based on task success.Success is defined through a goal-relevant state representation, a metric, and tolerance ε.
3 METHODS
PLD uses a frozen VLA generalist with lightweight residual actors and hybrid rollouts to generate diverse, deployment-aligned demonstrations for SFT. The method combines policy-prior warm-starting, residual exploration, and base-policy probing to capture recovery behavior while improving fine-tuning data.
- Policy-prior warm-start: PLD freezes the base policy and trains lightweight residual actors with off-policy RL, using the residual to explore near the base policy behavior.The residual action is conditioned on the base action and its magnitude is scheduled within [−ξ, ξ] to limit early deviation.
- Policy-prior warm-start: Successful base-policy rollouts populate an offline replay buffer, while online experience is replayed symmetrically during residual training.Equal sampling from offline and online buffers supports value learning from prior successful behavior together with newly collected experience.
- Policy-prior warm-start: A warm-up stage collects data solely with the base policy to stabilize off-policy learning and mitigate forgetting.The Q-function is initialized conservatively, and the policy loss does not explicitly impose behavior constraints.
- Distribution-aware data collection: Base-policy probing first rolls out the generalist for random steps, then transfers control to the residual expert to produce recovery trajectories from potentially suboptimal states.The probing states initialize the RL expert and are excluded from its replay buffer.
- Distribution-aware data collection: Increasing the probing horizon creates longer, more diverse successful episodes, improving fine-tuning performance until performance saturates.The hybrid collection scheme addresses the narrow distribution of highly optimal RL-expert demonstrations, which can underrepresent failure and out-of-distribution states.
4 EXPERIMENTS
PLD is evaluated for RL sample efficiency, in-distribution performance, generalization, and real-world robustness across simulation and robotic manipulation settings. Results indicate strong performance against baseline data-collection methods and recovery behavior in randomized real-world tasks.
- Effectiveness and efficiency of learning RL specialist: PLD outperforms policy-prior and data-prior RL baselines across 8 LIBERO-90 manipulation tasks at low interaction budgets.The evaluation reports mean rollout performance with 95% confidence intervals across 3 seeds.
- Effectiveness and efficiency of learning RL specialist: PLD achieves over 95% performance on every reported task used to fine-tune performance across more than 120 manipulation tasks.An initial performance drop occurs during exploration as residual policies diverge from the base policy and visit potentially suboptimal states.
- In-distribution performance: PLD improves VLA performance across LIBERO subsets and SimplerEnv using both OpenVLA and π0 base architectures.The experiments cover LIBERO-Object, LIBERO-Spatial, LIBERO-Goal, and a four-task SimplerEnv suite.
- Generalization: π0 fine-tuned on DPLD has the strongest in-distribution performance across coverage levels while maintaining robust zero-shot transfer to unseen LIBERO tasks.Human-data SFT has similar zero-shot generalization but lags on in-distribution tasks, whereas self-bootstrapping data fails to generalize out of distribution.
- Generalization: PLD data outperforms self-bootstrapped rollout data but remains below human-expert demonstrations on LIBERO-10 long-horizon tasks.The comparison uses π0 fine-tuned on LIBERO-90 and one-shot evaluation on held-out long-horizon tasks.
- Real-world performance: 100% success is achieved on both real-world Franka tasks within 2 hours of RL training, and +DPLD maintains 30/30 cube pick-up success under randomized trials.Peg insertion was 30/30 for all methods, while +DRLPD and +DHuman achieved 16/30 and 10/30 cube pick-up success, respectively; PLD recovered from corner-state failures.
5 RELATED WORK
Related work situates PLD within scalable VLA post-training, offline-to-online RL, and guide-policy methods for improving exploration and reducing distribution shift.
- Diffusion-based action generation and transformer-based VLA architectures reflect broader efforts to scale robot foundation models through data and model design.
- Offline-to-online RL uses offline initialization followed by online fine-tuning to improve sample efficiency and alleviate distributional shift.
- VLA post-training commonly combines broad robot-data pretraining with task-specific teleoperation demonstrations, which are expensive to acquire.
6 CONCLUSIONS
The conclusion presents PLD as an autonomous VLA post-training pipeline that combines residual RL with curated-data distillation and reports strong simulation and real-world performance.
- PLD couples a frozen VLA generalist with lightweight residual RL specialists and distills curated successes into the base model using standard SFT.
- PLD improves VLA performance without additional oracle human demonstrations across large-scale simulation experiments and real-world deployment.
- ∼99% success on LIBERO, >50% gains in SimplerEnv, and robust real-world performance are reported for PLD.
- Ablations identify residual policy probing and distribution-aware replay as key factors for stability, sample efficiency, and generalization.
B IMPLEMENTATION
The implementation compares hybrid and guide-policy RL baselines, uses offline initialization for sparse-reward learning, and describes conservative value estimation and in-sample alternatives.
- RL baselines: RLPD equally samples online and offline buffers and uses LayerNorm to address Q-value blow-ups under high update-to-data ratios.
- RL baselines: WSRL pretrains action and critic networks with Cal-QL, then discards offline data and warms up with pretrained-policy rollouts.
- RL baselines: JSRL uses guide-policy visits to shape the learner’s initial-state distribution, progressing from near-goal to farther states without directly imitating guide actions.
- Value initialization: Cal-QL learns a conservative value function that underestimates out-of-distribution action values while keeping them within a reasonable scale.
- Value initialization: IQL avoids querying Q on out-of-distribution actions by fitting state values with expectile regression and using advantage-weighted policy improvement.
B.2 DESIGN CHOICES OF PLD
The design ablations examine reward shaping, residual action scale, critic pretraining, update frequency, on-the-fly action sampling, and JSRL convergence across selected LIBERO-90 tasks.
- Reward shaping: A slight reward bias has little impact but can increase convergence speed in 2 of 8 tasks, whereas a large bias can significantly hinder performance.
- Action scale: Residual action scale trades off exploration and stability: large initial scales can degrade early performance, while small scales reduce exploration and asymptotic performance.
- Action scale: ξ = 0.5 is suggested for LIBERO and ξ = 0.1 for SimplerEnv in single-arm manipulation.
- Critic pre-training: Cal-QL critic pretraining performs consistently better and remains robust to conservative coefficient α, while CQL shows severe forgetting.
- Update frequency: Update frequency from 1 to 500 gradient steps produces largely insensitive overall performance across synchronization cadences.
- On-the-fly policy: On-the-fly policy sampling improves sample efficiency, with sample sizes greater than 20 showing significant performance gains.
- Baseline comparison: JSRL is generally data-efficient but can fail to converge on some tasks, whereas PLD reliably provides solutions for all tasks.
C.1 RL ALGORITHM
The RL algorithm uses a shared architecture and hyperparameter setting across tasks, with rollout performance compared across policy variants and against JSRL.
- Implementation and evaluation: All baselines use the same 3-layer MLP Gaussian policy, Clipped Double Q-networks, LayerNorm, and pretrained ResNetV1-10 visual encoder.The shared architecture supports apples-to-apples comparisons.
- Ablations: On-the-fly policy ablation reports mean rollout performance with 95% confidence intervals across 3 seeds.
- Ablations: Comparison with JSRL reports mean rollout performance with 95% confidence intervals across 3 seeds.
- Implementation and evaluation: RL hyperparameter settings are shared across all tasks.
- Fine-tuning setup: OpenVLA and π0 fine-tuning uses 8 NVIDIA L40 GPUs with LoRA rank 32.OpenVLA-OFT and π0 use default hyperparameters from their open-source codebases.
D.1 EXPERIMENT SETUP
The real-world evaluation uses a 7-DoF Franka arm for manipulation tasks, with an operational human intervention for peg-insertion diversity and a comparison of PLD with human data on unseen tasks.
- Experimental setup: PLD performs automatic reset, residual RL training, and SFT without human supervision.For peg insertion, human supervisors must randomly move the hole position to increase diversity.
- Experimental setup: Human supervisors must randomly move the peg-insertion hole to increase task diversity.
- Generalization performance: PLD data achieves better generalization than human teleoperation data on real-world unseen tasks.π0 is trained on clean-environment blue-cube pickup and peg-insertion data, then evaluated on cluttered blue- and red-cube pickup tasks.
- Generalization performance: Table 6 compares PLD and human data using success rate on real-world unseen tasks.