Source-linked AI summary
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, Ury Zhilinsky
TL;DR
Robot foundation models must overcome limited data, generalization, and robustness while supporting versatile physical behavior. The paper proposes π0, a flow-matching VLA built on a pre-trained VLM and trained across diverse robot data. π0 performs strongly across zero-shot and fine-tuned dexterous tasks, though dataset composition and reliable performance remain open issues.
Problem
Robot learning lacks the data, generalization, and robustness needed for broadly capable real-world systems.
Method
π0 combines a pre-trained VLM with flow matching and diverse multi-robot pre-training, followed by out-of-box evaluation or fine-tuning.
Results
π0 achieves the best results across all reported out-of-box tasks and solves many difficult multi-stage tasks, with the full pre-training and fine-tuning recipe performing best.
Takeaways & Limitations
The results support π0 as a broad robot foundation-model framework spanning dexterous, generalized, and temporally extended behaviors.
Takeaways & Limitations
The study does not establish how pre-training datasets should be composed or weighted, and not all evaluated tasks work reliably.
Abstract
from arXiv · showhide
Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence. However, bringing robot learning to the level of generality required for effective real-world systems faces major obstacles in terms of data, generalization, and robustness. In this paper, we discuss how generalist robot policies (i.e., robot foundation models) can address these challenges, and how we can design effective generalist robot policies for complex and highly dexterous tasks. We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We then discuss how this model can be trained on a large and diverse dataset from multiple dexterous robot platforms, including single-arm robots, dual-arm robots, and mobile manipulators. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people and from a high-level VLM policy, and its ability to acquire new skills via fine-tuning. Our results cover a wide variety of tasks, such as laundry folding, table cleaning, and assembling boxes.
I. INTRODUCTION
The paper frames generalist robot policies as a response to data, generalization, and robustness challenges in versatile robot learning. It introduces π0, combining VLM pre-training, cross-embodiment data, flow matching, and staged training for dexterous tasks.
- Robot learning still lags human versatility across diverse physical environments, language commands, constraints, and unexpected perturbations.
- Generalist policies may address data scarcity by learning from diverse tasks, robots, and non-robot sources, while improving observation and action coverage.
- Developing robot foundation models requires large-scale training, architectures that represent intricate behaviors, and effective training recipes.
- π0 combines a pre-trained VLM, cross-embodiment training, action chunking, and flow matching to incorporate semantic knowledge and model dexterous actions.
- The study pre-trains on over 10,000 hours of robot data and evaluates zero-shot commands, high-level policy control, and fine-tuning across dexterous tasks.
II. RELATED WORK
π0 extends prior VLA and diffusion-based robot-learning approaches by integrating VLM initialization, flow matching, cross-embodiment training, and a broad training framework. The resulting model targets long, dexterous, multi-stage behaviors across robot platforms.
- Prior VLA models typically fine-tune pre-trained VLMs while representing actions autoregressively as discrete tokens.
- π0 instead uses flow matching with a pre-trained VLM backbone to generate high-frequency action chunks for dexterous control.
- The framework combines architecture, pre-training and post-training phases, cross-embodiment data, and real-world experiments.
- π0 learns very long tasks, sometimes tens of minutes, combining physical dexterity and combinatorial complexity such as laundry folding and table bussing.
- The pre-training mixture includes dexterous datasets from 7 robot configurations and 68 tasks, alongside the OXE dataset spanning 22 robots.
IV. THE π0 MODEL
π0 augments a language-model transformer and image encoders with proprioception, action outputs, and a separate action expert. Conditional flow matching models continuous action chunks from multimodal observations.
- π0 uses a transformer VLM backbone with image, language, proprioceptive-state, and robot-action inputs and outputs.
- A separate action expert processes robotics-specific tokens, while continuous actions use flow-matching supervision and discrete outputs use cross-entropy.
- The model represents future actions as a chunk of H = 50 actions conditioned on images, language commands, and joint angles.
- Flow matching trains on noisy actions across flow-matching timesteps τ ∈ [0, 1], with sampling emphasizing lower, noisier timesteps.
- At inference, actions are generated by integrating the learned vector field from random noise using forward Euler steps.
- The main model uses a 3 billion-parameter PaliGemma backbone plus a 300M-parameter action expert, totaling 3.3 billion parameters.
V. DATA COLLECTION AND TRAINING RECIPE
The training procedure separates broad pre-training from later post-training, exposing the model to diverse tasks before inducing more targeted robot behaviors.
- π0 uses a multi-stage procedure modeled on language-model training, with pre-training followed by post-training.
- Pre-training exposes the model to diverse tasks to acquire broadly applicable physical capabilities.
A. Pre-training and post-training
π0 is pre-trained on a weighted mixture of diverse robot data and then fine-tuned on smaller, task-specific datasets, with language-based high-level decomposition supporting complex behaviors.
- Pre-training: The pre-training mixture combines open-source datasets with 903M proprietary timesteps from 7 robot configurations and 68 tasks.The proprietary data includes 106M single-arm and 797M dual-arm timesteps.
- Pre-training: Task-robot combinations are reweighted by n^0.43 to reduce the influence of over-represented combinations.The action and configuration vectors are zero-padded to the largest robot dimensionality, 18.
- Post-training: Post-training specializes the model on smaller task-specific datasets, ranging from 5 hours for simpler tasks to 100 or more hours for complex tasks.The recipe separates broad pre-training from focused fine-tuning for dexterity, efficiency, and robustness.
- Language-guided control: A high-level policy can decompose semantic tasks into immediate language subtasks that π0 executes.Examples include converting “bus the table” into commands such as picking up or discarding specific objects.
- Cross-embodiment training: π0 is trained jointly across single-arm, dual-arm, and mobile manipulator platforms with different configuration spaces and action representations.The platforms include 6-DoF and 7-DoF arms and holonomic and nonholonomic mobile manipulators.
VI. EXPERIMENTAL EVALUATION
The evaluation tests π0 after pre-training on dexterous, multi-stage, language-commanded tasks and compares it with robot foundation models and alternative architectures under matched or near-matched training budgets.
- Evaluation design: The out-of-box evaluation covers shirt folding, two bussing variants, grocery bagging, and removing toast from a toaster.These tasks combine dexterous manipulation, multi-stage behavior, and semantic recognition.
- Evaluation design: The study compares full π0, a 160k-step compute-parity π0, π0-small, OpenVLA, and Octo, including an OpenVLA model trained only on UR5e data.The baselines use different parameterizations and action-generation architectures.
- Evaluation metric: Scores average 10 episodes per task and method, assigning full success a score of 1.0 and partial success a fractional score.For bussing, the score is the fraction of objects placed in the correct receptacle.
- Results: π0 achieves the best results across all out-of-box tasks, with near-perfect success on shirt folding and easier bussing.The 160k-step parity model still outperforms all baselines, and π0-small also exceeds OpenVLA and Octo.
- Results: The results indicate that combining expressive architectures with action chunks and flow matching supports complex dexterous control.OpenVLA struggles without action chunks, while Octo has comparatively limited representational capacity.
B. Following language commands
Language evaluation measures π0 on multi-stage tasks requiring intermediate commands, while fine-tuning evaluation tests transfer to tasks increasingly different from pre-training. π0 generally benefits from VLM initialization and pre-training, though gains vary with task similarity and data amount.
- Language evaluation: The language evaluation uses bussing, table setting, and grocery bagging tasks composed of sequences of intermediate language commands.Conditions include flat task commands, human intermediate commands, and high-level VLM guidance.
- Fine-tuning evaluation: Fine-tuning tasks range from pre-training-similar activities such as stacking bowls and towel folding to tasks involving unseen objects, motions, and object types.Examples include a microwave task and replacing paper towels or handling Franka items in a drawer.
- Fine-tuning results: π0 generally outperforms other methods across fine-tuning tasks and data amounts, although its 5-hour Tupperware policy performs similarly to baselines.OpenVLA and Octo were evaluated on fewer dataset sizes because of their significantly worse performance and real-world evaluation cost.
- Fine-tuning results: Pre-training produces larger improvements on tasks more similar to the pre-training data, with gains sometimes reaching 2x over non-pre-trained models.The evaluation compares π0 with models trained from scratch and other pre-trained baselines.
D. Mastering complex multi-stage tasks
The paper evaluates π0 on difficult, temporally extended tasks that combine many behaviors, varied object configurations, and challenging physical properties. Full pre-training plus fine-tuning performs best across these tasks, while pre-training is especially valuable on harder ones.
- Evaluation: Average scores are computed over 10 trials, with perfect execution scored as 1.0 and partial completion receiving proportional scores.For example, 0.5 indicates that half the objects were bussed correctly.
- Task suite: The evaluation covers laundry folding, lunch-table bussing, box assembly, egg packing, and food packing across complex robot settings.The tasks involve stationary or mobile robots and require behaviors including grasping, stacking, folding, and flattening.
- Task suite: These tasks require dozens of coordinated behaviors, broad generalization across object configurations, and manipulation of deformable or flexible materials.Examples include clothing, dense clutter, delicate glasses, and flexible cardboard.
- Results: Full pre-training and fine-tuning performs best across the evaluated tasks, while many difficult tasks improve substantially when using the pre-trained model.The study compares full π0, out-of-box evaluation after pre-training, scratch training, and ablations.
VII. DISCUSSION, LIMITATIONS, AND FUTURE WORK
The paper frames π0 as a robot foundation-model framework combining large-scale diverse pre-training with downstream evaluation and fine-tuning. Results indicate broad task performance and some zero-shot capability, but dataset composition, reliability, transfer, and domain breadth remain unresolved.
- Framework: π0 combines highly diverse pre-training with either out-of-box evaluation or fine-tuning for complex downstream robot tasks.The framework is presented as a training procedure for a robot foundation model.
- Scale and evaluation: The pre-training mixture contains 10,000 hours of dexterous manipulation data spanning 7 robot configurations and 68 tasks, plus data from OXE, DROID, and Bridge.The authors describe this as the largest pre-training mixture used for a robot manipulation model to their knowledge.
- Scale and evaluation: Fine-tuning experiments cover over 20 tasks and outperform a variety of baselines, including prior VLA and dexterous-manipulation models.Post-training also addresses complex tasks such as folding multiple clothing articles and assembling boxes.
- Training behavior: Pre-trained models show some zero-shot capability, while complex tasks such as laundry folding require fine-tuning with high-quality data.Training only on high-quality data produces brittleness, whereas zero-shot pre-training may lack fluent post-training strategies.
- Limitations and future work: The authors leave open how to compose and weight pre-training data, predict data requirements, assess positive transfer, and extend universality beyond manipulation.Not all evaluation tasks work reliably, and broader domains such as driving, navigation, and legged locomotion remain future work.
APPENDIX
The appendix details π0’s robotics-specific transformer inputs, action-expert design, flow-matching supervision, and timestep sampling. It also describes the architecture’s action horizon and its emphasis on noisy-action timesteps.
- Architecture: The model adds robot state, action chunks, flow-timestep information, and a smaller action-expert weight set to a PaliGemma-style VLM.The action expert handles robotics-specific inputs and outputs while the VLM backbone processes images and language.
- Inputs and outputs: Robot state is linearly projected into the transformer embedding space, while noisy action chunks are embedded with an MLP incorporating timestep τ.For the evaluated tasks, the action horizon is H = 50.
- Action prediction: The model predicts H action outputs from noisy action tokens using conditional flow-matching supervision.Each action in the chunk has a corresponding action token processed by the action expert.
- Attention: π0 uses blockwise causal attention across input, state, and noisy-action blocks, with the action block attending to the full input sequence.Within each block, attention is bidirectional, while blocks cannot attend to future blocks.
- Timestep sampling: Flow-matching timesteps are sampled from a shifted beta distribution emphasizing lower, noisier timesteps and excluding values above cutoff s = 0.999.The cutoff permits integration steps δ > 1/1000, or up to 1,000 integration steps.
C. Non-VLM Baseline Architecture
π0-small is a non-VLM baseline designed to test the contribution of VLM pre-training while remaining expressive enough for the large robot dataset. Its observation and action-processing components differ from the main π0 architecture.
- Baseline purpose: π0-small does not use a VLM backbone, enabling evaluation of the benefits of VLM pre-training.It is designed to fit the large dataset while supporting training from scratch.
- Observation processing: The baseline uses a traditional encoder-decoder transformer with smaller pre-trained ViT image encoders and no Internet-pre-trained transformer backbone.Its image encoders do not share weights.
- Action processing: π0-small uses a DiT action expert with AdaLN-Zero timestep conditioning rather than the main model’s Gemma-based action expert.The baseline remains broadly similar to π0 in other respects.
D. Inference
During inference, π0 encodes observations, predicts a continuous vector field through flow matching, and integrates it over multiple steps to produce action chunks. Computation is measured on an RTX 4090, with mobile inference adding Wi-Fi latency.
- Inference pipeline: π0 encodes camera images and observation tokens before running flow-matching inference for each new action chunk.Each flow-matching step requires a forward pass on action tokens, while observation keys and values are cached.
- Inference pipeline: The model runs 10 flow-matching steps for each predicted action chunk.
- Runtime: Inference time is summarized for three camera images on an NVIDIA GeForce RTX 4090 GPU.Mobile-robot inference runs off-board over Wi-Fi, adding a small amount of network latency.
- Action execution: Because π0 predicts an entire H-step action chunk, it can execute up to H actions before running inference again.The authors execute chunks open-loop after temporal ensembling reduced policy performance.
E. Evaluation Details
The evaluation uses task-specific rubrics that quantify progress across manipulation tasks, including folding, sorting, stacking, and microwave placement. Tasks vary from binary success criteria to scores based on multiple subtasks and final-product neatness.
- Scoring framework: Each task uses a designed score rubric to measure progress quantitatively.
- Folding tasks: Shirt folding is evaluated as success or failure based on sleeve folding and one lengthwise half-fold.The evaluation covers four small T-shirts and one medium T-shirt, with two trials per item.
- Other tasks: Other tasks use multi-point rubrics, including 3-point bowl stacking and towel folding and 4-point Tupperware placement in a microwave.Points reflect subtasks such as stacking, folding stages, neatness, opening components, placing items, and closing the drawer.
- Folding tasks: Laundry folding is scored out of 4 for retrieval, flattening, folding, and placing or stacking each clothing item.Trials begin with randomly crumpled items and run for a maximum of 15000 steps, approximately 5 minutes.
- Sorting tasks: Table bussing is scored out of 12, assigning one point for each correctly sorted object.The task includes challenging arrangements such as a chopstick resting on trash.