Source-linked AI summary
$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, Ury Zhilinsky
TL;DR
Robots need to generalize beyond laboratory settings, but the extent of VLA generalization in the wild remains open. π0.5 co-trains heterogeneous robot, semantic, language, and web data, and demonstrates long-horizon dexterous manipulation in entirely new homes.
Problem
Robots must perform practically relevant tasks outside the lab, while how far VLA models generalize in the wild remains an open question.
Method
π0.5 co-trains heterogeneous data from multiple robots, semantic subtask prediction, supervisor language, web examples, and mobile manipulation.
Results
π0.5 performs long-horizon and dexterous multi-stage manipulation, including cleaning kitchens and bedrooms in homes unseen during training.
Takeaways & Limitations
The experiments show that transferring knowledge from diverse co-training sources is essential for effective generalization.
Takeaways & Limitations
π0.5 still makes mistakes in unfamiliar environments, under partial observability, and when high-level subtask inference is distracted.
Abstract
from arXiv · showhide
In order for robots to be useful, they must perform practically relevant tasks in the real world, outside of the lab. While vision-language-action (VLA) models have demonstrated impressive results for end-to-end robot control, it remains an open question how far such models can generalize in the wild. We describe $π_{0.5}$, a new model based on $π_{0}$ that uses co-training on heterogeneous tasks to enable broad generalization. $π_{0.5}$\ uses data from multiple robots, high-level semantic prediction, web data, and other sources to enable broadly generalizable real-world robotic manipulation. Our system uses a combination of co-training and hybrid multi-modal examples that combine image observations, language commands, object detections, semantic subtask prediction, and low-level actions. Our experiments show that this kind of knowledge transfer is essential for effective generalization, and we demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills, such as cleaning a kitchen or bedroom, in entirely new homes.
I. INTRODUCTION
π0.5 addresses open-world robotic generalization by co-training on heterogeneous knowledge sources. The resulting system targets long-horizon, dexterous manipulation in homes unseen during training.
- Open-world generalization remains a major challenge because useful robots must handle diverse real-world situations beyond laboratory settings.
- Broad robotic generalization requires training recipes that provide knowledge across multiple levels of abstraction, not merely increased scale.
- π0.5 uses co-training across heterogeneous knowledge sources to enable broad generalization in vision-language-action models.
- The model combines data from mobile manipulators in real homes, other robots, semantic prediction, and web sources.
- π0.5 controls mobile manipulators across household tasks, including tasks in homes absent from the training data.
- The work provides a proof of concept and empirical evaluation of generalization and the relevance of different co-training ingredients.
II. RELATED WORK
Related work shows that diverse robot training data can broaden task coverage and improve generalization, while prior non-robot sources and specialized methods address narrower aspects of the problem.
- Generalist manipulation policies trained on diverse scenes and tasks can solve more tasks out of the box and generalize to new scenes and tasks.
- Non-robot data co-training: Prior co-training approaches use non-robot data such as computer-vision datasets, task planners, and pretrained vision-language models.
- Generalist robot manipulation policies: Methods based on task-specific assumptions can generalize broadly, including to entirely new homes, but do not readily cover the full range of generalist-robot tasks.
- Vision-language-action models: VLAs are commonly trained by imitation learning to maximize the likelihood of action outputs conditioned on observations and language instructions.
IV. THE π0.5 MODEL AND TRAINING RECIPE
π0.5 is a transformer-based VLA trained in two stages to combine diverse robotic and semantic data with specialized mobile-manipulation control. It represents both textual outputs and action chunks, enabling hierarchical inference through a shared model.
- Training recipe: The model is initialized from a web-trained vision-language model and then pre-trained on diverse robotic tasks before mobile-manipulation post-training.
- Training recipe: Pre-training represents all tasks with discrete tokens, while post-training adds an action expert for mobile manipulation and efficient inference.
- Hierarchical inference: Text outputs can answer web-data questions or predict high-level subtasks, while action outputs control low-level robot behavior.
- Model architecture: π0.5 maps images, robot configuration, and a task prompt to textual outputs and predicted action chunks within one multimodal model.
- Hierarchical inference: Hierarchical inference separates high-level prediction πθ(ˆℓ|ot, ℓ) from low-level action prediction πθ(at:t+H|ot, ˆℓ), using the same model for both.
- Model architecture: The transformer processes heterogeneous token types with modality-specific encoders or expert weights and bidirectional attention for image, text-prompt, and continuous-action tokens.
B. Combining discrete & continuous action representations
π0.5 combines discrete action-token prediction with continuous flow matching. This design supports stable pre-training while retaining fast continuous-action inference.
- Flow matching trains π0.5 to predict a continuous action flow from noisy action chunks.The model uses the flow vector field ω − a_t:t+H with noise interpolation controlled by τ.
- The hybrid design addresses a trade-off: discrete actions accelerate training, whereas continuous flow matching is better suited to real-time inference.Autoregressive decoding of discrete actions is expensive at inference time.
- The model predicts actions both autoregressively through FAST tokens and non-autoregressively through iterative flow-field integration.Attention masking prevents the two action representations from attending to each other.
- π0.5 first pre-trains as a standard vision-language transformer with actions mapped to text tokens, then adds continuous action-expert weights during post-training.The action expert enables non-autoregressive prediction for fast inference while preserving language-following abilities.
- The pre-training and post-training task mixtures differ, with verbal instructions added later and laboratory cross-embodiment data omitted during post-training.Post-training focuses the model on mobile manipulation and diverse environments.
C. Pre-training
π0.5 is pre-trained on a heterogeneous mixture of robot, semantic, and web data. The mixture spans mobile manipulation, diverse environments, cross-embodiment experience, subtask prediction, and object localization.
- π0.5 is trained as an autoregressive transformer to predict text, object locations, and FAST-encoded action tokens.This unified objective supports multiple modalities and task types during pre-training.
- Mobile-manipulator data contributes about 400 hours collected across about 100 home environments.This slice is most directly relevant to evaluation tasks involving cleaning and tidying in unseen homes.
- Non-mobile robot data broadens environmental coverage through single-arm and dual-arm systems deployed across varied homes.These robots differ in embodiment from the mobile manipulators.
- The pre-training mixture includes cross-embodiment laboratory data, high-level subtask prediction, and additional web data with indoor-scene and household-object bounding boxes.Subtask examples decompose commands such as cleaning a bedroom into shorter actions such as adjusting a blanket or picking up a pillow.
- Action data uses target joint and end-effector poses, with prompts indicating the control mode and normalization based on dataset-specific quantiles.Action dimensionality is fixed to accommodate the largest action space among the datasets.
D. Post-training
Post-training specializes π0.5 for mobile manipulation while adding continuous flow-matching control. Evaluations in unseen mock and real homes show successful long-horizon household manipulation across novel environments.
- D. Post-training: Post-training specializes the model for mobile manipulation, jointly preserving text prediction and training a flow-matching action expert.It follows 280k discrete-token pre-training with 80k additional steps using α = 10.0.
- D. Post-training: The post-training data uses successful mobile and non-mobile robot episodes, selected web and semantic data, and expert verbal instruction demonstrations.Laboratory cross-embodiment data is omitted in the post-training mixture.
- E. Robot system details: π0.5 directly controls two mobile-manipulator platforms with 18-19 DoF state and action spaces.Each platform has two 6 DoF arms, grippers, a mobile base, a torso lift, and four cameras.
- E. Robot system details: The end-to-end controller commands arm, gripper, torso, and base targets at 50 Hz using action chunking and simple PD tracking.No additional trajectory planning or collision detection is used.
- A. Can π0.5 generalize to real homes?: All experiments use environments unseen during training, including controlled mock homes and three real homes for final evaluation.The environments contain novel objects, backgrounds, and layouts.
- A. Can π0.5 generalize to real homes?: π0.5 consistently succeeds across varied multi-stage tasks in each evaluated real home, with trials lasting about 2 to 5 minutes.High-level commands are converted autonomously into appropriate subtasks such as picking up a cup.
B. How does generalization scale with the number of scenes?
Generalization improves as the number of training environments increases, with the strongest model reaching performance similar to a model trained directly on the test homes. The evaluation covers multi-stage household tasks and language following on both seen and unseen object categories.
- Overall task performance: Performance on four multi-stage household tasks generally improves as the number of training locations increases.The tasks are dishes in sink, items in drawer, laundry basket, and make bed, evaluated in mock homes unseen during training.
- Overall task performance: The 104-location model achieves similar performance to a control trained on the test homes, despite never seeing test-home data.Baselines trained without the full co-training recipe perform significantly worse, indicating that additional data sources are important for generalization.
- Language following: Language following and placement success improve as more training locations are added for both in-distribution and out-of-distribution objects.Performance improves more quickly for in-distribution objects, while unseen categories require broader semantic generalization.
- Language following: Each additional environment introduces new household items, increasing robustness and supporting generalization to task categories absent from training.
C. How important is each part of our co-training recipe?
Cross-embodiment and diverse-environment data are important across evaluations, while web data contributes especially to out-of-distribution language following. The full π0.5 recipe also significantly outperforms π0 and π0-FAST+Flow in mock homes.
- Training-mixture ablations: Removing cross-embodiment or diverse-environment data causes large performance degradation on the four mock-home tasks.Removing both sources produces the strongest evidence that data from other robots is important beyond the target mobile manipulator platform.
- Training-mixture ablations: Excluding cross-embodiment or diverse-environment data significantly degrades language following for both in-distribution and out-of-distribution objects.
- Training-mixture ablations: Removing web data does not significantly affect mock-home task performance in this experiment but causes significantly worse out-of-distribution language following.The authors conjecture that web data provides broad knowledge of physical objects and helps interpret commands involving unseen categories.
- Comparison with other VLAs: The full π0.5 model significantly outperforms both π0 and π0-FAST+Flow in mock-home test environments.The comparison uses the same cross-embodiment robot training set and comparable training steps, while π0.5 additionally uses high-level and web data.
E. How important is high-level inference?
Explicit high-level inference gives the best performance, but much of its benefit comes from training on subtask-prediction data. Verbal-instruction and web data are also important, while zero-shot GPT-4 performs worst.
- Inference design: π0.5 predicts a semantic subtask from a high-level command and uses it as context for inferring lower-level actions.
- High-level inference: The full π0.5 model with high-level and low-level inference performs best, even exceeding the human high-level-policy oracle.
- High-level inference: Training with high-level subtask examples benefits performance even when high-level inference is omitted at runtime.The implicit-HL ablation is the second-best model, indicating that subtask knowledge can be learned without explicit runtime inference.
- Training ingredients: Excluding high-level task data significantly worsens performance, while excluding verbal-instruction data also produces a significant degradation.Verbal-instruction data constitutes about 11% of the high-level mobile-manipulation examples.
- Training ingredients: Excluding web data significantly degrades performance, and zero-shot GPT-4 achieves the worst result among the evaluated high-level inference methods.The results indicate that web data contributes to the high-level policy and that adapting VLMs with robot data is important in this evaluation.
APPENDIX
The appendix defines the quantitative evaluation protocol for four household cleanup tasks in new scenes and configurations. Policies are compared using rubric points across kitchen and bedroom tasks, with controlled sample sizes and interleaved execution.
- Evaluation protocol: The quantitative evaluation covers four training-included tasks tested in entirely new scenes and configurations.The set contains two kitchen cleanup tasks and two bedroom cleanup tasks, with results reported as percentages of rubric points.
- Evaluation protocol: Each policy receives 40 standard evaluations across four locations, with executions interleaved to control for environmental changes.Cancelled episodes from robot failures, time limits, or other causes are removed, and sample sizes are kept close across policies.
- Kitchen cleanup tasks: The Dishes in Sink task scores item pickup and placement, with a maximum of 8 points.
- Kitchen cleanup tasks: The Items in Drawer task requires picking up an item, opening the drawer, placing the item inside, and closing the drawer when applicable.The maximum score is 4 points.
- Bedroom cleanup tasks: The Laundry in Basket task scores navigating to and picking up clothing, placing it in or on the basket, and getting it fully inside.The maximum score is 3 points.
- Bedroom cleanup tasks: The Make the Bed task requires straightening the blanket and placing two pillows at the head of the bed.The maximum score is 5 points, including neatness criteria.
C. Language following experiment setup
The language-following experiments test whether π0.5 can interpret specific commands and move the correct object in two unseen kitchen scenes. Compared with π0 and π0-FAST+Flow, π0.5 performs best, with a particularly large margin over π0.
- Experiment design: The experiments use two unseen kitchen scenes containing five objects each, with commands requiring selection and manipulation of a specified object.The scenarios cover Items in the drawer and Items in the sink.
- Experiment design: The target object is placed farther away than distractors, limiting language-following accuracy to approximately 20% for policies that ignore the command.
- Metrics: The evaluation reports language following rate for correct object selection and task success rate for completing the instructed action.
- Results: π0.5 follows language slightly better than π0-FAST+Flow and much better than π0.The passage attributes the result to the importance of discrete token training for language-following abilities.
D. Per-task performance breakdown
Per-task ablations show that generalization depends on both cross-embodiment and diverse data, while explicit high-level inference improves performance across household tasks. The importance of each ingredient varies with task demands, especially semantic breadth and horizon.
- Co-training recipe ablations: Cross-embodiment transfer and diverse-data co-training are critical for generalization across the four evaluated household tasks.The tasks are Items in Drawer, Dishes in Sink, Laundry Basket, and Make Bed.
- Co-training recipe ablations: Items in Drawer degrades substantially without cross-embodiment or web data, with the largest drop when all are excluded.Its broad common-object recognition requirements may draw on diverse data sources.
- Co-training recipe ablations: Dishes in Sink is relatively robust to web-data removal but degrades when cross-embodiment data is excluded, consistent with reliance on robotic manipulation strategies.
- Co-training recipe ablations: Laundry Basket and Make Bed degrade when cross-embodiment data is removed but are less sensitive to other mixture changes.
- High-level model analysis: Explicit high-level inference improves performance across tasks, and the full π0.5 model achieves the best overall results.The comparison evaluates the full model and high-level inference baselines across the same four household tasks.
- High-level model analysis: Items in Drawer and Dishes in Sink decline substantially without high-level inference, indicating the importance of structured subtask prediction and long-horizon planning.π0.5 also outperforms GPT-4 HL on these tasks, while sensitivity is lower for shorter or less semantically demanding tasks.
E. Model technical details
π0.5 combines a vision-language backbone with a separate action expert and processes visual, linguistic, proprioceptive, semantic, and action information through structured token and attention pathways. Its action expert predicts a 50-step action horizon using flow matching with low-timestep-focused sampling.
- Architecture: The model uses PaliGemma as the vision-language backbone and adds an action expert for fast action generation.The backbone processes image and language inputs, while the action expert generates actions.
- Inputs and outputs: The vision-language backbone receives image sequences, a language prompt, and tokenized proprioceptive state, while actions are predicted as tokens.
- Action generation: The action expert processes noisy action chunks with horizon H = 49, corresponding to an action horizon of 50, and is trained with flow matching.
- Action generation: A separate timestep MLP and adaptive RMSNorm inject flow-matching timestep information into each action-expert layer.This differs from π0, which fuses the timestep with the noisy action before transformer processing.
- Action generation: The action expert outputs action tokens that are decoded into the target vector field through a final linear projection.
- Attention structure: The attention mask allows action representations to attend to the prefix and one another while preventing attention to FAST action tokens.The separation is intended to avoid information leakage between the two action representations.
- Timestep sampling: Flow-matching timesteps are sampled from a distribution emphasizing low timesteps, with values above s excluded; experiments use s = 0.999.This setting accommodates up to 1,000 integration steps when δ > 0.001.