PaperlayerView exact revision on arXiv

Source-linked AI summary

$π^{*}_{0.6}$: a VLA That Learns From Experience

Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Ben Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Alex Swerdlow, James Tanner, Marcel Torne, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, Ury Zhilinsky, Zhiyuan Zhou

arXiv:2511.14759v2cs.LGcs.RO

TL;DR

VLA models need to improve from real-world deployment experience rather than relying only on demonstrations. RECAP uses advantage-conditioned RL with demonstrations, autonomous rollouts, and expert interventions, producing a specialized π∗_0.6 model that improves performance on realistic manipulation tasks. On some hard tasks, it more than doubles throughput and roughly halves failures, while the system still depends on human effort for feedback, interventions, and resets.

  • Problem

    VLA models need experiential data to correct deployment mistakes and improve beyond demonstration data, but scalable real-world RL with heterogeneous data remains challenging.

  • Method

    RECAP trains VLAs with advantage-conditioned RL using demonstrations, autonomous experience, reward feedback, and expert interventions, beginning with offline RL pre-training.

  • Results

    RECAP improves success rate and throughput on laundry folding, box assembly, and espresso tasks, more than doubling throughput on some hard tasks and decreasing failures by roughly 2×.

  • Takeaways & Limitations

    The trained π∗_0.6 reaches practically useful robustness in homes and factories while performing complex manipulation tasks.

Abstract

from arXiv · show

We study how vision-language-action (VLA) models can improve through real-world deployments via reinforcement learning (RL). We present a general-purpose method, RL with Experience and Corrections via Advantage-conditioned Policies (RECAP), that provides for RL training of VLAs via advantage conditioning. Our method incorporates heterogeneous data into the self-improvement process, including demonstrations, data from on-policy collection, and expert teleoperated interventions provided during autonomous execution. RECAP starts by pre-training a generalist VLA with offline RL, which we call $π^{*}_{0.6}$, that can then be specialized to attain high performance on downstream tasks through on-robot data collection. We show that the $π^{*}_{0.6}$ model trained with the full RECAP method can fold laundry in real homes, reliably assemble boxes, and make espresso drinks using a professional espresso machine. On some of the hardest tasks, RECAP more than doubles task throughput and roughly halves the task failure rate.

I. INTRODUCTION

RECAP is designed to help VLA models improve through real-world practice by combining demonstrations, autonomous experience, reward feedback, and expert interventions. The resulting π∗_0.6 model is evaluated on demanding manipulation tasks and shows gains in robustness and throughput.

  • Motivation: VLA models need experiential data to correct deployment mistakes and improve speed, robustness, and adaptation beyond demonstrations.The paper identifies scalable RL, heterogeneous data, and real-world deployment as key challenges.
  • Method: RECAP combines demonstrations, autonomous experience, reward feedback, and expert interventions throughout the VLA training pipeline.It begins with offline RL pre-training and continues with training on deployment data.
  • Experiments: RECAP trains policies for diverse laundry folding, box assembly, and espresso-making tasks involving long-horizon, fine-grained manipulation.The tasks include pouring liquids, manipulating cloth and cardboard, and generalizing across varied objects.
  • Method: Advantage conditioning enables a value function to improve the policy while supporting training of large flow-matching VLA models.The method avoids much of the complexity of policy-gradient objectives for expressive VLA architectures.
  • Results: More than 2× throughput and roughly 2× lower failure rates are reported on some of the hardest tasks.The reported improvements concern successful task completions and task failures under RECAP training.
  • Results: The model reaches practically useful robustness by operating for 13 hours making espresso drinks, over two hours folding novel laundry, and assembling factory packaging boxes.These deployments span real homes and industrial packaging use cases.

III. PRELIMINARIES

The paper formulates RL around trajectory rewards, value functions, and action advantages, then uses regularized advantage conditioning to extract improved policies. RECAP repeats data collection, value-function training, and advantage-conditioned training, with a simplifying Markov-state assumption.

  • Reinforcement learning: RL models trajectories generated by a policy and stochastic dynamics, with the objective of maximizing cumulative reward.Rewards may be summarized through the return over a trajectory.
  • Reinforcement learning: A value function estimates future cumulative reward from an observation, while an advantage estimates an action’s contribution relative to that value.The paper defines advantage using an n-step return estimate.
  • Regularized reinforcement learning: Regularized RL balances reward maximization against divergence from a reference policy, yielding an improved policy weighted by exponentiated advantages.The reference policy typically corresponds to the behavior policy that collected the data.
  • RECAP procedure: RECAP repeatedly collects outcome-labeled episodes, optionally adds human corrections, trains a multi-task value function, and performs advantage-conditioned training.The value function detects failures and estimates expected time to task completion.
  • Assumption: The method assumes each observation constitutes a valid Markovian state, although this simplification is not generally true in robotic RL.The paper identifies this as a modeling assumption for simplicity.

A. Distributional value function training

The method trains a multi-task distributional value function from trajectory returns, then uses its continuous estimates to identify improved actions for advantage-conditioned policy extraction. The estimator is simple and reliable but less optimal than classic off-policy Q-learning.

  • Distributional value function: The value function maps observations and language commands to a distribution over discretized value bins.The implementation uses B = 201 bins and a smaller VLM backbone than the policy.
  • Distributional value function: Continuous values are extracted as the probability-weighted average of the learned value distribution across bins.These values provide the basis for computing advantages relative to the reference policy.
  • Distributional value function: The dataset-dependent value function represents expected returns from human demonstrations during pre-training and a mixture of demonstrations and learned-policy returns thereafter.This allows the critic to evolve as deployment data accumulates.
  • Policy extraction: The on-policy value estimator is less optimal than a classic off-policy Q-function estimator but is reported to be simple, reliable, and capable of improving over imitation learning.The authors identify off-policy estimators as a possible future extension.
  • Policy extraction: Advantage conditioning trains the policy to distinguish actions above a task-dependent improvement threshold from other actions.The threshold controls the optimality indicator and reduces reliance on test-time attenuation tuning.
  • Policy extraction: The method incorporates demonstrations, autonomous attempts, and human corrections into the policy-extraction dataset.Human corrections are assigned a positive indicator under the assumption that experts provide good corrective actions.

C. Method summary

RECAP alternates data collection, value-function training, and advantage-conditioned policy improvement. Pre-training uses demonstrations, while later iterations add autonomous experience and optional expert corrections for task specialists.

  • Method summary: RECAP is defined by three recurring subroutines: autonomous data collection with optional interventions, value-function training, and advantage-conditioned policy training.The data supplied to these subroutines changes between pre-training and downstream specialization.
  • Pre-training: Pre-training applies value and policy training to a multi-task demonstration dataset before downstream specialization.The resulting specialists are fine-tuned from the pre-trained model.

A. The π0.6 model

The π0.6 model combines multimodal perception, language reasoning, and flow-matching action generation, then adds a binarized advantage indicator for RECAP training. Its architecture predicts high-level subtasks alongside chunked robot actions.

  • Base VLA: π0.6 derives from π0.5 and uses Knowledge Insulation to train continuous actions and discretized tokens end-to-end.A stop gradient prevents the flow-matching action expert from affecting the rest of the model.
  • Base VLA: π0.6 augments its pre-training data, uses a Gemma 3 4B base VLM, and increases the action expert to 860M parameters.These are the stated changes relative to π0.5.
  • Model inputs and outputs: The model consumes camera images, robot configuration, and language metadata, producing action chunks at 50 Hz and tokenized subtask predictions.The predicted subtask precedes action generation and provides high-level guidance.
  • RECAP conditioning: π∗0.6 extends π0.6 with a binarized advantage indicator so a value function can condition policy training.The indicator is inserted before action outputs, affecting their log-likelihoods.
  • RECAP conditioning: Training can omit the advantage indicator to support either direct positive-indicator sampling or classifier-free guidance at inference.The latter enables guidance with β > 1.

C. Reward definition and value function training

RECAP uses sparse episode-level success labels to train a task-normalized value function, then applies that critic within an iterative deployment loop. The loop combines demonstrations, autonomous attempts, and optional interventions while restarting updates from the pre-trained checkpoint.

  • Reward definition: Each episode receives a success label from which RECAP derives a sparse reward applicable across tasks.The value function represents negative time to successful completion and assigns failed episodes a large negative value.
  • Value function training: The value function outputs are normalized to (−1, 0) per task using the task’s maximum episode length.This accounts for tasks with different typical durations.
  • Pre-training: Pre-training fits the value function on diverse multi-task, multi-robot data and estimates each task’s improvement threshold from predicted values.The threshold is set to the 30% percentile for task ℓ.
  • Deployment loop: For a target task, RECAP first fine-tunes π∗0.6 on demonstrations, then uses the resulting policy to collect autonomous data and optional expert interventions.The policy begins with the advantage indicator fixed to positive during demonstration fine-tuning.
  • Experimental setup: The experiments use a static bimanual robot with two 6-DoF arms, parallel-jaw grippers, 50 Hz joint-position control, and three cameras.The setup can be mounted flexibly, including on a table.
  • Deployment loop: The value function and policy are fine-tuned on accumulated task data from the pre-trained checkpoint rather than the previous iteration’s checkpoint.This was found useful for avoiding drift across iterations.

VI. EXPERIMENTAL EVALUATION

The evaluation tests RECAP on realistic, multi-step laundry, espresso, and box-assembly tasks, comparing its offline-RL foundation and on-robot training against supervised and alternative policy-extraction baselines. RECAP improves throughput across tasks, with especially large gains on diverse laundry and espresso.

  • Evaluation tasks: The experiments cover laundry folding, espresso preparation, and box assembly, including complex manipulation and 5–15-minute task durations.The evaluation emphasizes realistic tasks requiring constrained force, liquid pouring, cloth and cardboard manipulation, and fast execution.
  • Baselines: RECAP is compared with supervised π0.5 and π0.6, RL-pre-trained π*0.6, offline RL + SFT π*0.6, AWR, and PPO.AWR and PPO use the same on-robot data as RECAP but different policy-learning methods.
  • Baselines: Offline RL + high-quality SFT on π*0.6 outperforms standard SFT without offline RL pre-training.This combination provides a stronger starting point for subsequent reinforcement learning with on-robot data.
  • Main comparison: RECAP applied to π*0.6 produces substantial throughput improvements, more than doubling successful completions per hour on diverse laundry and espresso tasks.The throughput metric combines task success and execution speed.

C. Quantitative results

Quantitative results show that RECAP improves both throughput and success rate, with gains accumulating across iterations and varying by task difficulty. The largest throughput gains occur on challenging tasks, while box assembly improves substantially after additional data.

  • Metrics: Throughput measures successful task executions per hour, combining speed and success rate into one practical metric.Success rate separately measures the proportion of episodes that succeed using human-provided annotations.
  • Comparisons: Throughput more than doubles on diverse laundry folding and espresso tasks, while the failure rate decreases by about a factor of two.These gains compare the final RECAP policy with the offline RL + SFT π*0.6 model.
  • Success rates: The final π*0.6 model reaches 90%+ success on all tasks except diverse laundry and achieves higher success across every box-assembly stage.Box assembly is broken down into picking up the sheet, building, labeling, and placing the box in a crate.
  • Iterative improvement: After two iterations, laundry throughput improves by 50%, while long-horizon box assembly throughput improves 2×.Box assembly requires more data before a significant improvement appears.
  • Iterative improvement: The first laundry iteration raises success above 90%, whereas the second iteration mainly improves throughput.Box assembly success improves across both iterations, reaching about 90% for folding and labeling within 600 seconds.
  • Policy extraction: RECAP achieves the highest laundry throughput among the compared policy-extraction methods, outperforming AWR and PPO.The comparison uses the same task setting while changing the policy-extraction method.
  • Failure modes: RECAP is particularly effective at removing strict-criterion laundry failure modes, enabling behavior changes with relatively little data.The result comes from a one-item laundry variant with unusually strict success criteria.

3) How does the advantage-conditioned policy extraction method in RECAP compare to other methods?:

RECAP’s advantage-conditioned policy extraction is presented as a scalable alternative to policy-gradient extraction for end-to-end VLA training. Comparisons report that PPO performs poorly in this off-policy setting, while AWR achieves reasonable success but reduces policy speed and throughput.

  • Advantage-conditioned extraction versus baselines: AWR achieves a reasonable success rate but produces slower policies with lower throughput.The comparison uses the T-shirts and Shorts task and the same data used to train the final model.
  • Advantage-conditioned extraction versus baselines: PPO requires a small trust-region constraint (η = 0.01) for stable training but does not achieve good performance.The constraint is used because training occurs in an off-policy setting.
  • Overall RECAP outcomes: RECAP combines autonomous trials, reward feedback, and human interventions to improve VLA success rate and throughput.The reported results include more than doubling throughput on some harder tasks and decreasing failures by roughly 2×.
  • Limitations: RECAP’s approach remains partly dependent on human labeling, interventions, and episode resets.The authors identify greater automation of these components as a direction for improvement.
  • Limitations: RECAP uses largely greedy exploration and iterated offline updates rather than a fully concurrent online RL loop.The authors describe fully concurrent online RL as a promising future direction.

APPENDIX

The appendix provides supplementary visualizations, likelihood-decomposition details, and derivations connecting flow matching, diffusion, and autoregressive terms. It also shows how the value function’s changes are visualized across multiple robotic tasks.

  • Additional visualizations: Figure 13 visualizes value-function changes across five tasks using red for decreases, green for increases, and yellow for oscillations.The visualizations pair highlighted regions with corresponding episode frames and descriptions.
  • Likelihood derivation: The likelihood derivation decomposes the full model likelihood into autoregressive and diffusion terms.The autoregressive terms cover discretized actions and text, while the continuous action likelihood is handled through a diffusion-based bound.
  • Likelihood derivation: The continuous action likelihood is approximated with a one-step Gaussian diffusion process and an evidence lower bound.The derivation introduces noise-dependent weighting before combining the bound with autoregressive likelihoods.
  • Likelihood derivation: The resulting bound combines flow-matching, discretized-action, and text likelihood components.The weighting terms are subsumed into α in the final expression.

D. PPO implementation

The PPO implementation separates autoregressive and diffusion-based action terms while using a likelihood bound based on the single-step diffusion objective. The appendix reports practical stability issues with trust-region constraints and cautions against overly aggressive test-time sharpening.

  • PPO implementation: The PPO variant estimates likelihoods from the single-step diffusion objective to train both autoregressive and diffusion action components efficiently.Its loss combines separated diffusion and autoregressive PPO-style terms.
  • PPO implementation: Standard PPO clipping is difficult to enforce on the unbounded diffusion action head in this off-policy setting.The authors report this as a practical difficulty in their preliminary experiments.
  • PPO implementation: The implementation uses α as a trade-off parameter and ϵ_ar and ϵ_flow as trust-region parameters for the two model parts.The same variant is used for training on evaluation data from the π0.6 checkpoint.
  • Test-time policy improvement: β > 1 can sharpen the evaluation policy without additional training by combining learned conditional and unconditional policies.The authors primarily rely on the training-time advantage threshold and use moderate β values where useful.
  • Test-time policy improvement: High β settings can push actions toward the boundaries of learned support and produce overly aggressive motions.The authors give β ∈ [1.5, 2.5] as a moderate range used where useful.

F. Additional algorithm details

The appendix specifies advantage estimation, conditioning dropout, task-specific thresholds, and task-dependent dataset aggregation. These choices support pre-training and downstream fine-tuning across diverse robotic tasks and deployment settings.

  • Advantage estimation: Post-training advantages use an N = 50 lookahead, while pre-training uses whole-episode returns with N = T.The pre-training estimate has higher variance but can be computed online with a single value-function inference call.
  • Advantage estimation: The whole-episode pre-training advantage estimate works empirically with large, diverse multi-task datasets.This estimate is chosen for efficient on-the-fly calculation during pre-training.
  • Conditioning dropout: Advantage conditioning is randomly dropped 30% of the time during training.This enables conditional or unconditional inference and supports classifier-free guidance at test time.
  • Advantage thresholds: Pre-training thresholds target approximately 30% positive-advantage demonstrations, while fine-tuning thresholds generally target approximately 40% positive-advantage evaluation rollouts.Thresholds are set separately by task and training stage.
  • Dataset composition: Dataset aggregation varies by task because episode lengths, initial model performance, deployment setting, and available demonstrations differ.The appendix reports autonomous and correction data collected in different amounts for laundry tasks.
Loading 2511.14759v2…