PaperlayerView exact revision on arXiv

Source-linked AI summary

$π_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, George Bokinsky, Shihao Cao, Thomas Charbonnier, Vedant Choudhary, Foster Collins, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Maitrayee Dhaka, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachlan Groom, Haroun Habeeb, Hunter Hancock, Karol Hausman, Gashon Hussein, Victor Hwang, Brian Ichter, Connor Jacobsen, Szymon Jakubczak, Rowan Jen, Tim Jones, Gregg Kammerer, Ben Katz, Liyiming Ke, Mairbek Khadikov, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Brendon LeCount, Sergey Levine, Xinyu Li, Adrian Li-Bell, Vladislav Lialin, Zhonglin Liang, Wallace Lim, Yao Lu, Enyu Luo, Vishnu Mano, Nandan Marwaha, Aikys Mongush, Liam Murphy, Suraj Nair, Tyler Patterson, Karl Pertsch, Allen Z. Ren, Gavin Schelske, Charvi Sharma, Baifeng Shi, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Jiaming Tang, Jimmy Tanner, Shalom Tekeste, Marcel Torne, Kyle Vedder, Quan Vuong, Anna Walling, Haohuan Wang, Jason Wang, XuDong Wang, Chris Whalen, Samuel Whitmore, Blake Williams, Charles Xu, Sukwon Yoo, Lili Yu, Wuming Zhang, Zhuoyang Zhang, Ury Zhilinsky

arXiv:2604.15483v2cs.LGcs.RO

TL;DR

Robotic foundation models have struggled with compositional generalization, broad instruction following, and strong performance without task-specific fine-tuning. π0.7 addresses this with diverse multimodal context conditioning over heterogeneous data, and demonstrates out-of-the-box dexterous performance, unseen-environment instruction following, and zero-shot cross-embodiment transfer. A limitation is that the breadth of its dataset makes truly unseen tasks difficult to establish definitively.

  • Problem

    Prior robotic foundation models struggle to solve new tasks and often need task-specific fine-tuning even for instructions seen during training.

  • Method

    π0.7 uses diverse multimodal prompts containing expressive language, episode metadata, and subgoal images to contextualize demonstrations, failures, autonomous rollouts, and non-robot data.

  • Results

    π0.7 performs dexterous tasks out of the box, follows instructions in unseen environments, transfers tasks zero-shot across embodiments, and shows compositional generalization.

  • Takeaways & Limitations

    Diverse prompting and datasets allow one pre-trained model to represent policies of different qualities and distill specialist performance into a generalist robot model.

  • Takeaways & Limitations

    Because the dataset contains many related scenes and behaviors, it is difficult to determine which evaluation tasks are truly unseen.

Abstract

from arXiv · show

We present a new robotic foundation model, called $π_{0.7}$, that can enable strong out-of-the-box performance in a wide range of scenarios. $π_{0.7}$ can follow diverse language instructions in unseen environments, including multi-stage tasks with various kitchen appliances, provide zero-shot cross-embodiment generalization, for example enabling a robot to fold laundry without seeing the task before, and perform challenging tasks such as operating an espresso machine out of the box at a level of performance that matches much more specialized RL-finetuned models. The main idea behind $π_{0.7}$ is to use diverse context conditioning during training. This conditioning information, contained in the prompt, makes it possible to steer the model precisely to perform many tasks with different strategies. It is conditioned not just on a language command that describes what it should do, but on additional multimodal information that also describes the manner or strategy in which it should do it, including metadata about task performance and subgoal images. This enables $π_{0.7}$ to use very diverse data, including demonstrations, potentially suboptimal (autonomous) data including failures, and data from non-robot sources. Our experiments evaluate $π_{0.7}$ across numerous tasks with multiple robot platforms, on tasks that require speed and dexterity, language following, and compositional task generalization.

I. INTRODUCTION

π0.7 targets compositional generalization in robotics through diverse data and detailed multimodal context conditioning. It demonstrates broad instruction following, dexterous out-of-the-box performance, cross-embodiment transfer, and learning from heterogeneous data.

  • Prior robotic foundation models often struggle to solve new tasks and to fluently perform instructions seen during training without task-specific fine-tuning.
  • π0.7 uses detailed context annotations describing both what to do and how to do it, conditioning the model with multimodal signals.
  • π0.7 performs dexterous long-horizon tasks out of the box, including espresso-machine use, laundry folding, trash-bag removal, box folding, and vegetable peeling.
  • π0.7 follows diverse language instructions in unseen kitchen and bedroom environments.
  • π0.7 transfers dexterous tasks such as t-shirt folding zero-shot to a robot never trained on laundry folding, matching expert teleoperators on initial attempts.
  • Ablations and scaling studies show that diverse datasets and detailed context work synergistically, allowing mixed-quality and non-robot data without hurting performance.

III. FLOW-BASED VISION-LANGUAGE-ACTION MODELS

π0.7 extends a flow-based vision-language-action architecture with multimodal context conditioning and diverse training data. Its VLA predicts action chunks from observation histories while using a memory encoder, VLM backbone, and action expert.

  • VLA training data consists of robot trajectories containing observations, including camera images and joint configuration, and corresponding actions.
  • VLAs predict short future action chunks from recent observation histories, with an action expert enabling fast runtime inference.
  • A conventional VLA context is a language instruction, while π0.7 adds expressive language, episode metadata, and subgoal images.
  • The model is trained on broader data including failed episodes, suboptimal autonomous rollouts, and egocentric human videos, using detailed context to incorporate them.
  • π0.7 builds on π0.6-MEM with a Gemma3-based VLM backbone, MEM video-history encoder, and 860M-parameter flow-matching action expert.
  • The architecture is a 5B-parameter VLA whose context combines language commands, episode metadata, and subgoal images.

V. DIVERSIFYING THE PROMPT

π0.7 diversifies its prompts with subtask instructions, subgoal images, and episode metadata so the model can represent task intent, desired future states, and execution quality. These components are trained with flexible dropout and can be produced or used at runtime.

  • V. DIVERSIFYING THE PROMPT: The prompt components are randomly dropped during training, allowing π0.7 to handle any subset of them at test time.
  • A. Subtask instructions: Subtask instructions add intermediate semantic goals, such as “open the fridge door,” alongside the overall task description.
  • A. Subtask instructions: Semantic subtask conditioning supports step-by-step verbal coaching and can provide data for fine-tuning a high-level policy that guides new tasks.
  • B. Subgoal images: Subgoal images depict the desired near-future scene state, specifying execution details that language instructions may omit.
  • B. Subgoal images: Multi-view subgoals jointly specify environment or object outcomes and arm or gripper outcomes, improving spatial grounding for control.
  • B. Subgoal images: A lightweight world model generates subgoal images from the semantic subtask instruction and can generalize across diverse tasks and scenarios.
  • B. Subgoal images: The world model uses segment-end frames as ground-truth subgoals and is initialized from BAGEL, a web-pretrained image understanding, editing, and generation model.

C. Episode metadata

π0.7 uses episode metadata to contextualize diverse-quality trajectories and steer execution toward desired speed, quality, and mistake profiles. The prompt also specifies whether actions should be expressed in joint or end-effector control mode.

  • Metadata motivation: Episode metadata labels diverse trajectories with task-performance information so π0.7 can contextualize lower-quality and autonomous data.The metadata includes episode attributes such as speed, quality, and mistakes.
  • Metadata fields: Overall speed records episode length in timesteps and is discretized into 500-step bins.For example, episodes between 1750 and 2250 steps are binned as 2000 steps.
  • Metadata fields: Overall quality is represented by a score from 1 to 5, with 5 indicating the highest execution quality.
  • Metadata fields: Mistake labels identify whether the robot made an error within an action segment, using human annotations for events such as failed grasps or wrong subtasks.
  • Runtime prompting: At runtime, metadata prompting requests high speed, high quality, and mistake-free execution.The model uses correlations learned from episodes with varying speed and annotated quality or mistakes.
  • Control mode: The prompt includes a control-mode identifier selecting either joint-level or end-effector actions for runtime execution.

E. Full prompt and training details

π0.7 combines multimodal prompt context with diverse robot and non-robot data, using dropout so different context subsets remain usable at test time. Its architecture integrates historical vision, subgoal images, proprioception, and continuous action prediction with latency-aware action chunking.

  • E. Full prompt and training details: The full prompt can combine observations, subtask instructions, subgoal images, episode metadata, control mode, and proprioception.An example prompt includes a vegetable-peeling task, a subtask instruction, speed, quality, mistake, control mode, and multimodal observations.
  • E. Full prompt and training details: Prompt-component dropout lets π0.7 use arbitrary subsets of context at test time, while subgoal images substantially accelerate training.With subgoal images, action prediction becomes essentially an inverse-dynamics problem; visual subgoals are therefore added to only 25% of examples.
  • A. Training datasets: The training mixture spans demonstrations from many robot platforms and environments, autonomous evaluations, interventions, open-source datasets, human videos, and web-based multimodal tasks.The data includes static and mobile robots, single-arm and bimanual systems, video-language tasks, localization, attribute prediction, visual question answering, and text-only prediction.
  • A. Training datasets: π0.7 deliberately incorporates failures, mistake-heavy demonstrations, and autonomous data collected by prior models.This represents a departure from classic VLA pipelines that primarily rely on high-quality demonstrations.
  • A. Training datasets: Episode metadata allows the model to learn from heterogeneous evaluation data while distilling behaviors from RL-trained specialists.The authors report that this can support performance comparable to task-specialized RL models and sometimes stronger robustness.
  • B. Model architecture: The architecture adds historical vision encoding and visual subgoal conditioning, processing up to four camera views with history frames and up to three subgoal images.The history frames are compressed to the token count of a single frame, and subgoals use the same vision encoder.
  • A. Training datasets: Autonomous data from generalization-focused evaluation tasks is excluded from training.
  • B. Model architecture: An 860M-parameter action-expert transformer predicts 50-step continuous action chunks with flow matching and bidirectional action-token attention.The action expert also attends to activations from the VLM backbone.

C. Training with subgoal images

Subgoal images are generated and refreshed as multimodal goals that guide π0.7 through changing subtasks, while runtime prompts select metadata, control, and optional guidance settings. The model produces smooth action chunks asynchronously under simulated inference delays.

  • C. Training with subgoal images: Subgoal training combines real future-trajectory images with generated images to handle varying goal delays and image quality.
  • Runtime prompting: Runtime prompts always include control mode and episode metadata, while subtask instructions and subgoal images are added according to the desired behavior.
  • Runtime prompting: Runtime metadata sets speed to the task’s 15th-percentile episode length, quality to 5, and mistakes to false.
  • Runtime prompting: The system refreshes subgoal images when the semantic intent changes or after four seconds, whichever occurs first.Subtask instructions can come from a learned high-level policy or a human supervisor coaching the robot.
  • Inference: π0.7 generates 50-step action chunks with five denoising steps and executes either 15 or 25 steps before reinference.
  • Inference: The runtime workflow initializes a subtask from a high-level policy or human coaching before action generation begins.
  • Inference: At each inference cycle, the policy conditions actions on recent observations and context, while an asynchronous world model optionally generates subgoal images.
  • Inference: After the selected action steps elapse, the system asynchronously regenerates actions with real-time action chunking and executes them continuously.

VIII. ROBOT SYSTEM DETAILS

The experiments span mobile and static bimanual platforms, a morphologically different UR5e system, and long-horizon tasks. Evaluations include coarse-instruction execution and stepwise coaching for tasks absent from training.

  • Robot platforms: The robot suite includes bimanual mobile manipulators, static bimanual robots, and a bimanual UR5e setup for cross-embodiment experiments.
  • Evaluation tasks: Selected evaluations include long-horizon tasks such as taking out trash and toasting a bagel.
  • Evaluation tasks: π0.7 can execute a coarse instruction for taking out trash as a full long-horizon task, while toasting a bagel is coached through detailed stepwise instructions despite being absent from training.
  • Robot platforms: Additional language-following and generalization experiments use a single-arm 6 DoF system based on the BiPi arms.
  • Robot platforms: The UR5e arms are longer, heavier, and morphologically different from the BiPi-like arms used for much of the data, requiring a different manipulation strategy.

IX. EXPERIMENTAL EVALUATION

The evaluation examines π0.7’s out-of-the-box dexterity, memory, and training-recipe effects across challenging tasks. A single general-purpose model performs competitively with specialized policies, while metadata and evaluation data improve results.

  • Out-of-the-box dexterity: π0.7 matches or closely approaches specialized policies across the evaluated dexterous tasks without task-specific post-training.It even exceeds RL-specialist throughput on laundry folding and box building.
  • Out-of-the-box dexterity: π0.7 achieves higher throughput than RL specialists on diverse laundry-folding and box-building tasks.
  • Training ablations: π0.7 outperforms ablations without episode metadata and without autonomous evaluation data, with the largest gap in normalized throughput.Throughput is measured as successes per hour relative to π0.7.
  • Memory tasks: The same π0.7 model achieves similar or better performance than memory-fine-tuned specialists on tasks requiring prior context.

B. Instruction following

π0.7 follows multi-step language instructions in unseen environments and handles unusual references and instructions that conflict with dataset biases. These results indicate stronger instruction following than prior models.

  • Referential instructions: π0.7 handles out-of-distribution referential instructions involving unusual language and complex spatial references.
  • Bias-breaking instructions: π0.7 significantly improves over prior models on tasks requiring instructions to contradict strong dataset biases.On “Reverse Fridge to Microwave,” generated subgoal images are critical for success.

C. Cross-embodiment transfer

π0.7 transfers skills to robots with no task-specific target-embodiment data, including dexterous laundry folding. Successful transfer can involve discovering strategies adapted to the target robot’s morphology.

  • Zero-shot transfer: π0.7 succeeds out of the box on several target embodiments for tasks with no target-robot training data.Prior models show some transfer for smaller embodiment differences, while π0.7 handles larger gaps more effectively.
  • Emergent strategies: On the UR5e, π0.7 discovers a single-arm bag-insertion strategy instead of replicating the source robot’s two-arm behavior.The strategy changes with the target robot’s reach and morphology.
  • Laundry transfer: π0.7 successfully folds towels and shirts on the UR5e despite laundry data being collected on different robots.For shirt folding, it uses vertical grasps better suited to the UR5e’s kinematics.
  • Visual conditioning: Generated subgoal images significantly improve cross-embodiment performance by providing visual cues about suitable grasps and clothing configurations.
  • Dexterous transfer: π0.7 achieves 85.6% task progress and 80% success on shirt folding, versus expert teleoperators’ 90.9% and 80.6%.

D. Compositional task generalization

π0.7 compositionally generalizes to new tasks, directly handling some short-horizon tasks and using language coaching for longer-horizon tasks. Coaching episodes can then support autonomous execution without additional teleoperation data.

  • Short-horizon generalization: π0.7 performs some unfamiliar short-horizon tasks out of the box, including wiping headphones and spinning a desk fan.
  • Scaling with diverse data: With metadata, π0.7 improves continuously as mixed-quality datasets grow, whereas removing rich conditioning can make performance degrade.Episode metadata disambiguates data quality and strategies during training.
  • Language coaching: Language coaching enables π0.7 to perform unseen longer-horizon tasks that are too complex for direct prompting.These tasks can involve multiple stages and up to five minutes of interaction.
  • Autonomous capability acquisition: A high-level language policy trained from coaching episodes produces autonomous policies that roughly match coached performance on five unseen tasks.No additional teleoperation or low-level action data is collected.

E. Can π0.7 learn effectively from diverse and mixed-quality data?

The ablations show that diverse prompting helps π0.7 exploit larger datasets even when added data is lower quality, while task diversity improves generalization. The experiments also identify zero-shot performance as lower on unseen tasks than on seen tasks and note difficulty determining whether tasks are truly unseen.

  • Mixed-quality data: π0.7 without metadata can worsen on larger mixed-quality datasets, whereas metadata enables continued improvement.The comparison uses laundry data divided into four quality-and-speed buckets ranging from the top 30% to all data.
  • Mixed-quality data: π0.7 with metadata continuously improves as more data is added, even though average data quality decreases.Episode metadata helps distinguish data quality and strategies, enabling the model to prompt for desired behavior at test time.
  • Task diversity: Across short-horizon unseen tasks, removing the most diverse 20% of data performs worse than removing a random 20% or removing nothing.The comparison controls dataset quantity while isolating the contribution of highly diverse task data.
  • Implications: The authors conclude that diverse prompting and larger, more diverse datasets help π0.7 represent policies of different qualities and distill specialist performance into one pretrained model.This conclusion connects the ablation findings to compositional generalization and cross-robot skill transfer.
  • Scope boundary: Zero-shot success rates are lower on unseen tasks or unseen task-robot combinations than on in-distribution tasks.Seen tasks often exceed 90% success, while unseen settings fall in the 60–80% range.
  • Scope boundary: It is difficult to determine whether tasks are truly unseen because large datasets may contain related skills under different labels or within other behaviors.The authors frame this ambiguity as a limitation of measuring generalization in large, diverse datasets.

APPENDIX

The appendix documents implementation details for π0.7, its world model, control choices, human-study operators, and inference costs. It also reports that end-effector control offers no clear advantage over joint-space control and that richer visual context increases latency.

  • Attention and world model: The world model combines robot and egocentric human video data with high-quality language labels, plus open-source image and video datasets.The authors report that temporal segmentation and label quality strongly affect subgoal-image quality.
  • Attention and world model: π0.7 and its world model use several attention patterns that change depending on whether image goals are present.Without image goals, the model uses global bidirectional attention across memory-aware image views; FAST tokens and flow actions do not attend to each other.
  • Inference cost: Inference time is 38ms for the minimal π0.7 variant and 127ms in the worst case with the MEM vision encoder and subgoal images.The reported timings use one H100 GPU, three camera inputs, and five denoising steps for the minimal variant.
  • Control space: Joint-space and end-effector control show no substantial performance difference across the evaluated cross-embodiment tasks.The main cross-embodiment experiments therefore focus on joint-space control.
  • Human study: The human-study operators were selected from the top 2% of the operator fleet by teleoperation experience.The study recruited ten operators across UR5e, static bimanual, and combined robot categories.
  • Inference cost: Generating subgoal images requires computational optimizations including four-way tensor parallelism across four H100 GPUs and 8-bit matrix multiplications.At inference time, π0.7 continues executing asynchronously while the world model generates the next subgoal.

E. Comparison of action spaces in cross-embodiment transfer

The cross-embodiment comparison finds no clear advantage for end-effector control over joint-space control, so the main experiments use joint-space control. The human study provides a strong zero-shot reference for the shirt-folding task on UR5e.

  • Action-space comparison: End-effector control does not show a clear advantage over joint-space control across cross-embodiment tasks.The authors use joint-space control in the main experiments for clarity.
  • Human comparison: The human study evaluates expert teleoperators on the same UR5e shirt-folding task and uses identical metrics, time limits, and evaluation criteria.Ten highly experienced operators completed three trials each without practice or a warm-up period.
  • Task scope: The evaluated task suite includes laundry folding, peeling vegetables, trash removal, object sorting, and appliance manipulation.Examples include reversing a fridge-to-microwave sequence and completing a multi-step oven task.
Loading 2604.15483v2…