PaperlayerView exact revision on arXiv

Source-linked AI summary

$π_0$: A Vision-Language-Action Flow Model for General Robot Control

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, Ury Zhilinsky

arXiv:2410.24164v4cs.LGcs.RO

TL;DR

Versatile robot control faces challenges in data scarcity, generalization, and robustness. π0 combines a pre-trained vision-language model with flow matching, diverse-data pre-training, and optional downstream fine-tuning. It outperformed baselines across out-of-box tasks and solved many difficult multi-stage tasks, with pre-training plus fine-tuning performing best among tested training recipes.

  • Problem

    Robot learning faces data scarcity, robustness, and generalization challenges in building versatile systems for diverse physical tasks.

    Source: §I · p. 2

  • Method

    π0 combines a pre-trained vision-language model with flow matching for robot actions, using diverse-data pre-training and optional fine-tuning for downstream tasks.

    Source: §IV · p. 4, §VII · p. 11

  • Results

    π0 outperformed all baselines on every out-of-box task and solved many difficult multi-stage tasks, with pre-training plus fine-tuning performing best among tested training recipes.

    Source: A. Evaluating the base model · p. 7, D. Mastering complex multi-stage tasks · p. 11

  • Takeaways & Limitations

    Some complex tasks require high-quality fine-tuning data, but training only on those data produces brittle models that do not reliably recover from mistakes.

    Source: §VII · p. 12

    Not all evaluated tasks work reliably, and the amount and type of data needed for near-perfect performance remain unclear.

    Source: §VII · p. 12

Abstract

from arXiv · show

Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence. However, bringing robot learning to the level of generality required for effective real-world systems faces major obstacles in terms of data, generalization, and robustness. In this paper, we discuss how generalist robot policies (i.e., robot foundation models) can address these challenges, and how we can design effective generalist robot policies for complex and highly dexterous tasks. We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We then discuss how this model can be trained on a large and diverse dataset from multiple dexterous robot platforms, including single-arm robots, dual-arm robots, and mobile manipulators. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people and from a high-level VLM policy, and its ability to acquire new skills via fine-tuning. Our results cover a wide variety of tasks, such as laundry folding, table cleaning, and assembling boxes.

I. INTRODUCTION

π0 is a generalist robot policy combining a pre-trained vision-language model with flow-matched action generation and diverse robot training. Its pre-training/post-training recipe targets dexterity, efficiency, and robustness.

I. INTRODUCTION

  • Embodied data is needed for physically situated versatility, while diverse robot pre-training could ease data scarcity and improve generalization and robustness.Data from other tasks, robots, and non-robot sources expands available training material and broadens observation/action coverage, including corrections and recovery behaviors.Source: §I · p. 2
  • Generalist robot policies face three bottlenecks: sufficiently large-scale training, architectures accommodating diverse data and intricate behaviors, and effective pre-training/post-training data recipes.The proposed framework addresses these through VLM initialization, robot-action modeling, and investigation of training recipes.Source: §I · pp. 2–3
  • Up to 50 Hz control supports dexterous tasks through action chunking [57], flow-matched continuous action distributions [28] [32], and a novel VLM action expert.The action expert augments the standard VLM with flow-based outputs.Source: §I · p. 3
  • Diverse pre-training followed by carefully curated fine-tuning combines recovery behaviors with the dexterity, efficiency, and robustness sought from high-quality demonstrations.High-quality data alone rarely exposes mistakes, while lower-quality pre-training data alone does not teach efficient, robust behavior; combining them preserves corrections and recoveries.Source: §I · p. 3
  • π0 combines VLM pre-training and flow matching, with evaluation demonstrating greater dexterity and generality than previously demonstrated robot foundation models.Evaluation covers direct language commands, downstream fine-tuning, and a high-level semantic policy issuing intermediate language commands for temporally extended tasks.Source: §I · p. 3
  • Over 10,000 hours of robot data support pre-training, while cross-embodiment training [10] combines single-arm, dual-arm, and mobile-manipulator data.The pre-trained VLM supplies Internet-scale knowledge, semantic reasoning, and problem-solving abilities. Downstream tasks include laundry folding, table clearing, handling dishes and eggs, box assembly, and grocery bagging.Source: §I · pp. 2–3

II. RELATED WORK · III. OVERVIEW

π0 combines a pre-trained vision-language backbone with flow matching to generate high-frequency continuous robot actions. Its framework pairs large-scale, cross-embodiment pre-training with curated post-training for complex dexterous tasks, including behaviors lasting tens of minutes.

II. RELATED WORK

  • Up to 50 Hz action chunks are supported by flow matching, rather than the autoregressive action discretization used by prior VLA models [7] [24] [55].Unlike action diffusion methods [9] [60], this model uses a pre-trained VLM backbone [5]. The contribution integrates architecture, a pre-training recipe, pre-training and post-training phases, and real-world experiments.Source: §II · p. 3
  • The model adapts elementwise diffusion-style losses from Zhou et al. [59] and separate diffusion-token weights from Liu et al. [29].The authors identify it, to their knowledge, as the first flow-matching VLA producing high-frequency action chunks for dexterous control.Source: §II · p. 3
  • About 10,000 hours of demonstrations, complemented by OXE [10], substantially exceed the tens or hundreds of trajectories typical of earlier dexterous-task studies [57].Those earlier datasets represented 10 or fewer hours. The authors describe their experiment as the largest by robot-data volume, to their knowledge.Source: §II · p. 3
  • Tens of minutes of dexterous behavior are learned for some tasks, with a single cross-embodiment model serving as the base for these tasks.These tasks combine physical dexterity and combinatorial complexity: laundry folding handles varied garments in arbitrary configurations and sequences multiple items, while table bussing classifies novel objects as trash or dishes.Source: §II · p. 4

III. OVERVIEW

  • 68 tasks across 7 robot configurations supply the authors’ pre-training data, combined with the entire OXE dataset [10], containing data from 22 robots.The weighted mixture uses diverse language labels, combining task names with fine-grained sub-trajectory annotations typically covering about 2 seconds.Source: §III · p. 4
  • Pre-training develops broad capabilities and rudimentary language-conditioned task proficiency; curated post-training adapts the base model for specialized, complex dexterous tasks.The study examines both efficient adaptation with small-to-moderate datasets and high-quality post-training with larger datasets for laundry folding and mobile manipulation.Source: §III · p. 4
  • PaliGemma [5] supplies the vision-language backbone, while a smaller action expert processes robot states and actions for continuous action generation via flow matching.PaliGemma was chosen for convenience and its comparatively small size, useful for real-time control; the framework can use any pre-trained VLM.Source: §III · p. 4, Figure 3 · p. 4

IV. THE π0 MODEL

π0 augments a vision-language transformer with robot state inputs and a separate action expert, using conditional flow matching to generate action chunks. A non-VLM-initialized baseline evaluates the benefit of VLM pretraining.

IV. THE π0 MODEL

  • π0 combines a vision-language transformer with proprioceptive inputs and continuous action outputs modeled through conditional flow matching [28] [32].The implementation uses PaliGemma [5], a 3 billion parameter VLM, plus a randomly initialized 300M-parameter action expert, totaling 3.3 billion parameters.Source: §IV · pp. 4–5
  • Separate weights for robotics-specific state and action tokens improve performance, forming an action expert alongside the image-and-text pathway.The architecture draws inspiration from Transfusion [59], which supervises continuous outputs with flow matching and discrete outputs with cross-entropy.Source: §IV · pp. 4–5
  • 50-action chunks are conditioned on language commands, joint angles, and 2 or 3 RGB images per robot.Images and proprioceptive state pass through their respective encoders and linear projections into the language-token embedding space.Source: §IV · p. 5
  • π0 trains its action expert to predict a denoising vector field from interpolations between Gaussian noise and demonstrated actions.Training uses A_t^τ = τA_t + (1 − τ)ϵ with target A_t − ϵ, bidirectional attention across action tokens, and beta-distributed τ emphasizing noisier timesteps. Inference integrates from noise using 10 forward-Euler steps (δ = 0.1), caching observation-prefix attention keys and values.Source: §IV · p. 5
  • 470M-parameter π0-small omits VLM initialization and serves as an ablation baseline for evaluating the benefits of VLM pretraining.It also includes small architectural differences found helpful for training without VLM initialization, so initialization is not its only difference from the main model.Source: §IV · p. 5

V. DATA COLLECTION AND TRAINING RECIPE · A. Pre-training and post-training

π_0 uses diverse pre-training data to develop general physical capabilities, followed by task-specific post-training for fluent execution. Its training recipe combines open-source and in-house robot data, reweights imbalanced task–robot combinations, and accommodates different robot configurations.

V. DATA COLLECTION AND TRAINING RECIPE

  • Pre-training targets broad task and behavioral diversity, while post-training emphasizes consistent, fluent strategies for effective downstream execution.Diverse, lower-quality pre-training data supports mistake recovery and handling varied situations that may be absent from high-quality post-training data.Source: §V · p. 5

A. Pre-training and post-training

  • 9.1% of the training mixture consists of open-source datasets, including OXE [10], Bridge v2 [52], and DROID [23].These datasets cover varied objects and environments, typically using one or two cameras and control frequencies between 2 and 10 Hz.Source: A. Pre-training and post-training · pp. 5–6
  • 903M in-house timesteps cover 68 broadly defined tasks containing complex behaviors, extending beyond the narrow noun–verb task definitions used in prior work.Single-arm robots contribute 106M steps and dual-arm robots contribute 797M steps. Each task can encompass diverse objects and behaviors, so the task count understates behavioral breadth.Source: A. Pre-training and post-training · p. 6
  • n^0.43 weighting reduces the relative contribution of overrepresented task–robot combinations, where n is the sample count for each combination.This addresses dataset imbalance, including overrepresentation of difficult laundry-folding tasks.Source: A. Pre-training and post-training · p. 6
  • 18 dimensions standardize configuration and action vectors across robots, with zero-padding for smaller spaces and masking for missing image slots.The largest configuration accommodates two 6-DoF arms, two grippers, a mobile base, and a vertically actuated torso. Missing image slots are masked for robots with fewer than three images.Source: A. Pre-training and post-training · p. 6
  • 5 hours of task-specific data suffice for the simplest post-training tasks, while the most complex use 100 or more hours.Post-training fine-tunes the model on a smaller dataset to specialize it for a downstream application.Source: A. Pre-training and post-training · p. 6

B. Language and high-level policies · C. Robot system details

π0 can use a high-level VLM to decompose complex tasks into immediate language-guided subtasks. Its dexterous manipulation datasets cover 7 robot configurations and 68 tasks, with joint training across single-arm, dual-arm, and mobile platforms.

B. Language and high-level policies

  • A high-level VLM supplies semantic reasoning and strategy by decomposing complex tasks into immediate subtasks for π0, analogously to SayCan [2].Language inputs let the high-level policy guide π0 through subtasks such as picking up a napkin and throwing it away; this assists several experimental tasks.Source: B. Language and high-level policies · p. 6

C. Robot system details

  • 7 robot configurations and 68 tasks comprise the dexterous manipulation datasets, with π0 trained jointly across single-arm, dual-arm, and mobile platforms.The platforms include 6-DoF and 7-DoF arms, plus holonomic and nonholonomic mobile manipulators.Source: C. Robot system details · p. 6, Figure 5 · p. 6
  • 7-dimensional UR5e and 8-dimensional Franka configuration and action spaces each accompany two camera images.The UR5e uses a parallel-jaw gripper, a wrist-mounted camera, and an over-the-shoulder camera.Source: C. Robot system details · p. 6
  • 14-dimensional configuration and action spaces and three camera images characterize the bimanual UR5e, Trossen, ARX, and AgileX setups.Trossen uses paired 6-DoF ViperX arms in an ALOHA-based arrangement [4] [57]; paired 6-DoF ARX and AgileX arms are grouped for similar kinematics. Trossen, ARX, and AgileX use two wrist cameras and one base camera.Source: C. Robot system details · p. 6
  • 16-dimensional actions and 14-dimensional configurations characterize mobile Trossen and ARX, whose nonholonomic base adds two action dimensions.Based on Mobile ALOHA [57], these platforms combine two 6-DoF arms with two wrist cameras and one base camera. They are grouped because of similar kinematic properties.Source: C. Robot system details · p. 6
  • 17-dimensional actions and 14-dimensional configurations characterize Mobile Fibocom, combining two 6-DoF ARX arms with a holonomic base.The base contributes three action dimensions: two for translation and one for orientation.Source: C. Robot system details · p. 6

VI. EXPERIMENTAL EVALUATION · A. Evaluating the base model

Pretrained π0 achieves the best results across five out-of-box tasks, and its 160k-step compute-parity version also outperforms all baselines. The broader evaluation examines language following, dexterous-task adaptation, and fine-tuning for complex multistage tasks.

VI. EXPERIMENTAL EVALUATION

  • The evaluation investigates pretrained performance, language following, dexterous-task adaptation, and fine-tuning for complex multistage tasks.Language tests use human commands and a high-level VLM policy; downstream comparisons examine architecture and pretraining through fine-tuning versus training from scratch. Complex tasks take 5–20 minutes, and some require high-level guidance.Source: §VI · p. 7

A. Evaluating the base model

  • Without post-training, one language-commanded base model tackles shirt folding, easy and hard bussing, grocery bagging, and toast removal.The tasks combine dexterity, multistage behavior, and semantic recognition. Hard bussing includes obstructing or stacked objects and objects absent from the pretraining dataset.Source: A. Evaluating the base model · p. 7, Figure 6 · p. 7
  • OpenVLA [24] and Octo [50] are trained on π0’s full mixture, alongside comparisons with UR5e-only OpenVLA and the smaller π0-small without VLM pretraining.OpenVLA has 7B parameters, while Octo has 93M; the UR5e-only baseline excludes cross-embodiment training.Source: A. Evaluating the base model · p. 7
  • 10 episodes per task and method determine the average normalized score, with 1.0 for full success and fractional credit for partial success.For bussing, the score is the fraction of objects placed in the correct receptacle.Source: A. Evaluating the base model · p. 7
  • π0 outperforms all baselines across the out-of-box tasks, with near-perfect success on shirt folding and easy bussing.The full model achieves the best results by a large margin across all tasks.Source: A. Evaluating the base model · p. 7, Figure 7 · p. 8
  • 160k training steps suffice for compute-parity π0 to outperform every baseline; π0-small also outperforms OpenVLA [24] and Octo [50].The full π0 model trains for 700k steps, compared with 160k for OpenVLA and 320k for Octo; time constraints prevented matching training epochs.Source: A. Evaluating the base model · p. 7, Figure 7 · p. 8
  • The authors attribute baseline weaknesses to OpenVLA’s [24] lack of action chunking and high-frequency control, and Octo’s [50] limited representational capacity.UR5e-only training improves OpenVLA but leaves it far below π0. The authors argue for combining expressive architectures with flow matching or diffusion.Source: A. Evaluating the base model · pp. 7–8
  • π0’s advantage over π0-small supports the importance of VLM pretraining, but π0-small’s lower parameter count complicates a fair comparison.The authors note that larger models are difficult to use without pretraining.Source: A. Evaluating the base model · p. 8

B. Following language commands · C. Learning new dexterous tasks

Fine-tuned π0 generally outperforms comparison methods on new dexterous tasks, with pre-training frequently improving performance over training without it. π0 also follows language commands better than π0-small, improving task performance with human or autonomous high-level guidance.

B. Following language commands

  • Language evaluation fine-tunes π0 and non-VLM π0-small on table bussing, table setting, and grocery bagging.Commands identify objects and destinations in roughly two-second labeled segments. π0-small is also substantially smaller, confounding attribution of the performance difference to VLM initialization alone.Source: B. Following language commands · p. 8, Figure 8 · p. 8
  • π0-flat and π0-small-flat receive overall task commands, π0-human and π0-small-human receive expert steps, and π0-HL receives autonomous high-level VLM commands.Intermediate commands specify which object to pick up and where to place it, testing whether the policies can exploit detailed language guidance.Source: B. Following language commands · p. 8
  • π0 follows language significantly better than π0-small, translating intermediate human commands and autonomous high-level VLM guidance into improved task performance.Evaluations average 10 trials per task, with human guidance producing larger gains than autonomous high-level guidance. π0-small does not improve overall with expert intermediate commands because of its limited language-following ability.Source: B. Following language commands · p. 8, Figure 9 · p. 9

C. Learning new dexterous tasks

  • Five downstream tasks span familiar manipulations, an unseen microwave, and new objects or motions, testing adaptation beyond pre-training.Bowl stacking and towel folding form the easy tier; microwave loading introduces a new appliance, while paper-towel replacement and Franka drawer packing are hard. Bowl and container evaluations mix seen and unseen objects.Source: C. Learning new dexterous tasks · p. 9, Figure 10 · p. 9
  • Comparisons distinguish fine-tuned pre-trained policies, task-only specialist policies, and π0 trained with versus without robot pre-training.OpenVLA [24] and Octo [50] use publicly available OXE [10]-pre-trained checkpoints, while ACT [57] and Diffusion Policy [9] train only on target-task datasets. The π0 comparison separates the benefits of its architecture and VLM initialization from robot pre-training.Source: C. Learning new dexterous tasks · p. 9
  • Fine-tuned π0 generally outperforms other methods across tasks evaluated with varying amounts of fine-tuning data, with performance averaged over 10 trials per task.The strongest prior models were trained entirely from scratch on the target tasks. On Tupperware in microwave, π0’s 5-hour policy performs similarly to baselines, while its 1-hour version is significantly better.Source: C. Learning new dexterous tasks · p. 9
  • Up to 2x the non-pre-trained model’s performance is achieved by pre-trained π0, with larger pre-training benefits on tasks resembling pre-training data.The pre-trained model frequently outperforms the non-pre-trained model, although the benefit varies across tasks.Source: C. Learning new dexterous tasks · pp. 9–10

D. Mastering complex multi-stage tasks

π0’s full pre-training and fine-tuning recipe performs best across challenging multi-stage laundry, bussing, assembly, and packing tasks. These tasks require sequencing many behaviors while handling varied object configurations and demanding physical interactions.

D. Mastering complex multi-stage tasks

  • Laundry evaluations cover stationary folding, mobile folding, and dryer unloading, all represented in pre-training but requiring fine-tuning for mastery.Stationary folding starts with randomly crumpled clothing in a bin and ends with folded items stacked; mobile folding additionally requires controlling robot orientation and translation.Source: D. Mastering complex multi-stage tasks · p. 10
  • Table bussing, absent from pre-training, tests generalization to novel objects, dexterous grasping, dense clutter, and ordered behavior sequences.The robot must handle large plates and delicate glasses, and may need to pick up a plate, empty its trash, then place it in a bin.Source: D. Mastering complex multi-stage tasks · p. 10
  • Box assembly, absent from pre-training, requires coordinated bimanual folding of flexible cardboard, bracing against the table, and reactive retries.Across the illustrated tasks, robots must combine dozens of behaviors while generalizing to varied object configurations and handling deformable materials.Source: D. Mastering complex multi-stage tasks · p. 10, Figure 12 · p. 11
  • Food and egg packing, both absent from pre-training, require careful placement followed by bimanual container closure.Food must fit inside a to-go box without protruding. Egg packing requires transferring six slippery eggs from varying poses in a bowl into open carton slots.Source: D. Mastering complex multi-stage tasks · p. 10
  • 10 trials per task determine average scores, with 1.0 indicating perfect execution and partial scores reflecting partial completion.For example, 0.5 means half the objects were correctly bussed.Source: D. Mastering complex multi-stage tasks · pp. 10–11
  • π0 with pre-training and fine-tuning performs best across all evaluated tasks, outperforming pre-training alone and training from scratch on fine-tuning data.Other methods did not solve these tasks, so comparisons focus on these ablations; pre-training yields especially large improvements on harder tasks. Absolute performance varies, likely reflecting task difficulty and representation in pre-training.Source: D. Mastering complex multi-stage tasks · p. 11

VII. DISCUSSION, LIMITATIONS, AND FUTURE WORK

π0 combines diverse robot-data pre-training, Internet-scale VLM knowledge, and flow matching, with fine-tuning enabling complex tasks. Dataset composition, reliable performance, and transfer to more distinct domains remain unresolved.

Results

  • More than 50% of the maximum score is attained on every complex post-training task by full pre-trained π0, typically outperforming ablations.Scores average 10 trials, with especially significant improvements on the hardest tasks. Fine-tuning experiments include over 20 tasks and show improvements over prior VLA models [24] and dexterous-manipulation models [57] [9].Source: Figure 13 · p. 11, §VII · p. 11

Framework

  • 10,000 hours of dexterous manipulation across 7 robot configurations and 68 tasks underpin pre-training, supplemented by OXE [10], DROID [23], and Bridge [52] data.π0 combines Internet-scale VLM pre-training with flow matching to represent complex, high-frequency action chunks. Diverse pre-training is followed by either out-of-box evaluation or fine-tuning for complex downstream tasks.Source: §VII · p. 11

Pre-training and post-training

  • Complex tasks require high-quality fine-tuning: high-quality-only training produces brittle models, while zero-shot pre-trained models do not consistently exhibit fluent post-training strategies.Pre-trained models nevertheless have some zero-shot capabilities, whereas high-quality-only models struggle to recover reliably from mistakes. The authors suggest an analogy with language models, where pre-training acquires knowledge and post-training shapes its use.Source: §VII · pp. 11–12

Limitations

  • Dataset composition, weighting, and the amount and type of data needed for near-perfect performance remain unresolved, while some evaluated tasks remain unreliable.The pre-training mixture combined all available data rather than establishing which additions or weights are most helpful.Source: §VII · p. 12

Future work

  • Positive transfer across diverse tasks and robots remains insufficiently understood, and universality beyond manipulation—including autonomous driving, navigation, and legged locomotion—remains untested.The results suggest universal robot foundation models may be possible, but do not establish generality across these more distinct domains.Source: §VII · p. 12

APPENDIX · A. Contributions · B. Model Architecture Details

π_0 extends the PaliGemma VLM [5] with robotics-specific projections, timestep conditioning, and a smaller action expert. The appendix also details attention and timestep-sampling schemes and lists author responsibilities.

A. Contributions

  • Author responsibilities span data and operations, evaluation, model design, pre-training, post-training, robot hardware and software, training infrastructure, and writing and illustration.Contributors are listed alphabetically within each area.Source: A. Contributions · p. 15

B. Model Architecture Details

  • H = 50 defines the action-chunk horizon; robot state is linearly embedded, and action-token outputs are linearly decoded into the predicted vector field.Only transformer outputs corresponding to the noisy-action tokens are decoded.Source: B. Model Architecture Details · p. 15
  • Each noisy-action embedding combines a projected action with sinusoidal timestep encoding [51] through an MLP with a swish nonlinearity.The MLP concatenates the projected action and timestep encoding before applying two further learned projections with swish between them.Source: B. Model Architecture Details · p. 15
  • Three attention blocks separate images/language, robot state, and noisy actions, with bidirectional attention inside blocks and no attention to future blocks.Isolating VLM inputs minimizes pre-training distribution shift, while isolating robot state enables caching its keys and values across integration steps. Noisy actions can attend to the full input sequence.Source: B. Model Architecture Details · p. 15
  • π_0 extends PaliGemma [5] with a smaller action expert for robot state and noisy actions, while images and language use the pretrained backbone.The two experts form one transformer and interact only through self-attention. The action expert uses width=1024 and mlp dim=4096, totaling ∼300M parameters to accelerate repeated inference passes.Source: B. Model Architecture Details · p. 15
  • Flow-matching training uses a shifted beta timestep distribution that emphasizes low timesteps, corresponding to high-noise actions, and excludes timesteps above a cutoff.The authors hypothesize that observation-conditioned mean-action prediction is harder than text-conditioned mean-image prediction. They use s = 0.999; excluding timesteps above s is sufficient when δ > 1−s, allowing up to 1,000 integration steps.Source: B. Model Architecture Details · p. 16, Figure 14 · p. 16

C. Non-VLM Baseline Architecture

π0-small is a roughly 470M-parameter non-VLM baseline designed to evaluate the benefits of VLM pre-training. It changes the language, vision, and action architectures while retaining the main model’s observation format and 10-step flow-matching prediction.

C. Non-VLM Baseline Architecture

  • 470M parameters, approximately, give π0-small capacity to fit the large dataset as a non-VLM baseline for evaluating VLM pre-training benefits.The architecture is designed to provide good performance when trained from scratch.Source: C. Non-VLM Baseline Architecture · p. 16
  • Language commands use DistilBERT [42], while images use smaller pre-trained R26-S-32 ResNet-ViT hybrid encoders from Steiner et al. [47] without shared weights.The observation transformer following the image encoders is not pre-trained on Internet data.Source: C. Non-VLM Baseline Architecture · p. 16
  • The action expert cross-attends to observation-encoder outputs like an encoder-decoder transformer [51], unlike the main model’s decoder-only mixture-of-experts-style architecture [45].Both models use separate weights for the observation encoder and action expert.Source: C. Non-VLM Baseline Architecture · p. 16
  • The action expert uses DiT [36] rather than Gemma, incorporating the flow-matching timestep through AdaLN-Zero layers.This replaces the main model’s action-expert architecture.Source: C. Non-VLM Baseline Architecture · p. 16
  • 10 flow-matching steps predict each action chunk in both models, which also share the same observation format and use pre-trained ViT image encoders.Source: C. Non-VLM Baseline Architecture · p. 16

D. Inference

Inference generates action chunks using 10 flow-matching steps with cached observation keys and values. The policy executes chunks open-loop without action aggregation because temporal ensembling [57] hurt performance.

D. Inference

  • 10 flow-matching steps generate each action chunk after image and observation encoding, with observation keys and values cached across steps.Each flow-matching step requires a forward pass over the action tokens.Source: D. Inference · p. 16
  • Inference timing used three camera images on an NVIDIA GeForce RTX 4090; mobile-robot inference ran off-board over Wi-Fi, adding network latency.Further optimization and quantization might reduce inference times.Source: D. Inference · p. 16, Table I · p. 16
  • Up to H actions can execute per inference call, but the policy uses unaggregated, open-loop chunks after temporal ensembling [57] hurt performance.The 20Hz UR5e and Franka robots run inference every 0.8 seconds after 16 actions. All other robots run at 50Hz and infer every 0.5 seconds after 25 actions.Source: D. Inference · p. 16

E. Evaluation Details … D. Mastering complex multi-stage tasks

Evaluation uses task-specific rubrics to measure progress across base-model tasks, language instruction following, new dexterous skills, and complex multi-stage tasks. Scores reward completed actions, correct object placement, and, for selected tasks, final neatness.

E. Evaluation Details

  • Task-specific scoring rubrics measure progress and provide the basis for quantitative evaluation.The supplied passages describe evaluation protocols rather than model performance results.Source: E. Evaluation Details · pp. 16–17

A. Evaluating the base model

  • 7 points assess easy bussing, 12 points assess hard bussing, 7 points assess grocery bagging, and 4 points assess toast retrieval.Sorting and bagging award one point per correctly placed object; toast retrieval awards separate points for extraction and plating. Shirt folding is binary success after sleeve folds and one lengthwise half-fold, starting flat; four small shirts and one medium shirt receive two trials each, capped at 15000 steps or approximately five minutes.Source: A. Evaluating the base model · p. 17
  • Language-following evaluation scores both successful object repositioning and compliance with instructions across bussing, table setting, and grocery bagging.Each episode includes 12 objects and around 30 instructions for bussing, seven objects and around 20 instructions for table setting, or seven objects and around 14 instructions for grocery bagging.Source: A. Evaluating the base model · p. 17

C. Learning new dexterous tasks

  • 3 points assess bowl stacking, and 3 points assess towel folding, with both rubrics explicitly rewarding final neatness.Bowl stacking awards one point for each of two bowls stacked inside larger bowls; towel folding awards one point for each of two half-folds. Each task reserves its remaining point for neatness.Source: C. Learning new dexterous tasks · p. 17
  • 4 points assess microwave loading, 4 points assess paper-towel replacement, and 5 points assess placing items in a drawer.Microwave loading rewards opening, picking up the Tupperware, inserting it, and closing; paper-towel replacement rewards grasping and removing the old roll, then grasping and installing the new roll. Drawer evaluation rewards opening, placing each of three items inside, and closing.Source: C. Learning new dexterous tasks · p. 17

D. Mastering complex multi-stage tasks

  • 4 points assess laundry folding through retrieval, flattening, folding, and placement or stacking; mobile laundry follows the same protocol.Evaluation uses five garments, with two trials per item and unevaluated items initially crumpled in a bin. Each trial allows 15000 steps, approximately five minutes.Source: D. Mastering complex multi-stage tasks · p. 17
  • 5 points assess box building, 7 points assess egg packing, and 5 points assess food packing.Box building rewards pickup, folding, closing each flap, and neat centering; egg packing rewards correct egg placement and lid closure. Food packing rewards plate pickup, transferring each of three food items, and closing the container.Source: D. Mastering complex multi-stage tasks · p. 17
  • 5 points assess dryer unloading, while 12 points assess table bussing through one point per correctly sorted object.Dryer unloading rewards approaching, positioning the basket, opening the dryer, transferring all clothes, and closing the dryer; three shirts and two shorts start randomly arranged inside. Bussing includes challenging arrangements such as a chopstick atop trash.Source: D. Mastering complex multi-stage tasks · p. 17

Related papers

Loading 2410.24164v4…