PaperlayerView exact revision on arXiv

Source-linked AI summary

$Ψ_0$: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation

Songlin Wei, Hongyi Jing, Boqian Li, Zhenyu Zhao, Jiageng Mao, Zhenhao Ni, Sicheng He, Jie Liu, Xiawei Liu, Kaidi Kang, Sheng Zang, Weiduo Yuan, Marco Pavone, Di Huang, Yue Wang

arXiv:2603.12263v1cs.RO

TL;DR

Humanoid loco-manipulation is constrained by costly robot data and mismatched human–robot embodiments. Ψ0 separates learning objectives by pre-training a VLM on egocentric human videos and post-training a flow-based action expert on real humanoid data, achieving stronger multi-task performance with far less data than baselines.

  • Problem

    Humanoid VLA training is limited by expensive teleoperation data and heterogeneous human–robot action distributions that make direct co-training suboptimal.

  • Method

    Ψ0 pre-trains a VLM on high-quality egocentric videos, post-trains a flow-based MM-DiT action expert on humanoid data, and deploys the model with a lower-body tracking controller.

  • Results

    At least 40% higher average overall success rate than GR00T-N1.6 was achieved across eight long-horizon tasks using 800 hours of human video and 30 hours of robot data.

  • Takeaways & Limitations

    The results support scaling high-quality, task-relevant human video together with smaller amounts of domain-specific humanoid data rather than simply enlarging heterogeneous datasets.

  • Takeaways & Limitations

    Training was not scaled to larger human-video or real-robot collections, and the hardware payload limits potentially more capable manipulation behaviors.

Abstract

from arXiv · show

We introduce $Ψ_0$ (Psi-Zero), an open foundation model to address challenging humanoid loco-manipulation tasks. While existing approaches often attempt to address this fundamental problem by co-training on large and diverse human and humanoid data, we argue that this strategy is suboptimal due to the fundamental kinematic and motion disparities between humans and humanoid robots. Therefore, data efficiency and model performance remain unsatisfactory despite the considerable data volume. To address this challenge, \ours\;decouples the learning process to maximize the utility of heterogeneous data sources. Specifically, we propose a staged training paradigm with different learning objectives: First, we autoregressively pre-train a VLM backbone on large-scale egocentric human videos to acquire generalizable visual-action representations. Then, we post-train a flow-based action expert on high-quality humanoid robot data to learn precise robot joint control. Our research further identifies a critical yet often overlooked data recipe: in contrast to approaches that scale with noisy Internet clips or heterogeneous cross-embodiment robot datasets, we demonstrate that pre-training on high-quality egocentric human manipulation data followed by post-training on domain-specific real-world humanoid trajectories yields superior performance. Extensive real-world experiments demonstrate that \ours\ achieves the best performance using only about 800 hours of human video data and 30 hours of real-world robot data, outperforming baselines pre-trained on more than 10$\times$ as much data by over 40\% in overall success rate across multiple tasks. We will open-source the entire ecosystem to the community, including a data processing and training pipeline, a humanoid foundation model, and a real-time action inference engine.

I. INTRODUCTION

Ψ0 addresses humanoid loco-manipulation by separating learning across heterogeneous human-video and robot data, rather than co-training them directly. Its staged recipe and real-time control design target long-horizon whole-body skills with substantially less data than competing approaches.

  • Humanoid dexterous loco-manipulation remains more challenging than locomotion despite progress in whole-body humanoid control.
  • Human egocentric videos provide scalable motion and behavior information, but embodiment differences in action frequency, dynamics, and degrees of freedom complicate transfer.
  • Ψ0 pre-trains a VLM on human data for task-level representations, then trains a flow-based action expert on humanoid data for joint-space control.
  • The system combines a manipulation-oriented teleoperation pipeline, real-time action chunking, and whole-body control to improve deployment on complex tasks.
  • The paper positions higher-quality, embodiment-relevant data as more valuable than simply scaling heterogeneous datasets, while targeting unified whole-body dexterous manipulation.

C. Learning From Egocentric Videos

Ψ0 uses human egocentric video to learn task semantics and visual representations, then connects those representations to whole-body humanoid actions through a staged architecture. The design uses unified task-space actions for pre-training and a triple-system controller for deployment.

  • Co-training heterogeneous human and robot action distributions is suboptimal, so Ψ0 pre-trains on task semantics before modeling humanoid joint-space actions.
  • Ψ0 predicts 36-dimensional whole-body action chunks from language, visual observations, and proprioceptive state.
  • Its triple-system architecture combines a VLM backbone, a roughly 500M-parameter flow-based MM-DiT action expert, and an RL-based lower-body tracking policy.
  • Training proceeds through VLM pre-training on egocentric video, action-expert post-training on cross-task humanoid data, and small in-domain fine-tuning.
  • EgoDex contributes approximately 829 hours of human video, while Humanoid Everyday contributes 31 hours across 260 humanoid tasks for visual-gap mitigation.
  • The VLM predicts single next-step actions because this suffices to learn task semantics and downstream visual representations while avoiding expensive multi-chunk autoregression.

2) Post-Training on Cross-Task Real Humanoid Data:

After VLM training, Ψ0 freezes the backbone and trains a separate flow-based action expert on humanoid data. The expert uses flow-matching with multimodal attention to generate joint-space action chunks conditioned on vision-language features.

  • The action expert is trained from scratch after the VLM backbone is frozen, using hidden VLM features as conditioning.
  • Flow-matching training uses a uniformly sampled flow timestep and a noised action formed by interpolating the target action with noise.
  • MM-DiT separately modulates action and vision-language features with the flow timestep, then fuses their tokens through joint global attention.
  • The resulting action expert produces joint-space action chunks and is evaluated within eight real-world tasks, most exceeding 2,000 steps at 30Hz.

C. Real-Time Action Chunking

Ψ0 addresses inference latency and teleoperation challenges in humanoid loco-manipulation by combining real-time action chunking with a structured whole-body control setup. These designs target smoother execution and more stable, practical data collection.

  • Real-Time Action Chunking: 160 ms per forward pass motivates real-time chunking to preserve smooth, reactive humanoid control.The model contains over 2.5 billion parameters, and training simulates inference delay by masking randomly delayed action tokens.
  • Real-Time Action Chunking: Real-time chunking conditions each prediction on the previously committed action chunk, suppressing divergence between consecutive actions.The figure contrasts large action divergence and control jitter without RTC against smoother, more stable behavior with RTC.
  • Tailoring Teleoperation for Loco-Manipulation: The teleoperation framework separates upper-body pose tracking, dexterous hand control, and locomotion commands for single-operator whole-body control.A headset and wrist trackers support inverse kinematics, MANUS gloves control finger degrees of freedom, and additional trackers provide locomotion commands.
  • Tailoring Teleoperation for Loco-Manipulation: Separating locomotion from in-place whole-body actions improves locomotion stability across diverse task scenarios.Wrist trackers and MANUS gloves also alleviate occlusion and out-of-view issues in vision-based VR tracking.
  • Real-World Platform: The real-world experiments use a Unitree G1 with 29 degrees of freedom and 7-DoF Dex3-1 hands on each arm.Visual observations come from the robot’s head-mounted Intel RealSense D435i camera.

2) Data Preparation:

The data preparation pipeline combines large-scale human egocentric data processing with action-tokenizer training and real-world task evaluation. Ψ0 is tested on diverse long-horizon tasks using standardized teleoperated data and reproduced open-source baselines.

  • Data Preparation: EgoDex contains approximately 900M frames with per-frame upper-body transformation matrices for pre-training data preparation.Actions are transformed into the head-camera frame, upsampled threefold, and normalized using the 1st and 99th quantiles.
  • Data Preparation: A FAST tokenizer trained on 500,000 EgoDex actions achieves approximately 0.005 held-out L1 reconstruction loss.This improves on the approximately 0.01 loss reported for the original open-source FAST tokenizer.
  • Data Preparation: The FAST tokenizer compresses each action sequence into 20 tokens to accelerate subsequent training.Pre-training fine-tunes Qwen3-VL-2B-Instruct, while post-training uses an approximately 500M-parameter action expert on Humanoid Everyday data.
  • Task Evaluation: Eight real-world long-horizon tasks span pick-and-place, pushing, wiping, dexterous finger-object coordination, whole-body motion, and locomotion.Examples include turning a faucet, pulling out a chip tray, torso rotation, squatting, and turning.
  • Task Evaluation: Each task uses 80 teleoperated trajectories and contains three to five sub-tasks involving dexterous manipulation, dual-arm coordination, and locomotion.All baselines are fine-tuned on the same dataset with identical image observations and action and state representations.
  • Benchmarking: The study benchmarks Ψ0 against recent open-source baselines using efforts to reproduce the best possible results for each method.The supplied passage describes the benchmarking procedure but does not report comparative performance values.

3) Baselines:

The benchmark compares Ψ0 with multiple humanoid and manipulation baselines across eight long-horizon loco-manipulation tasks. Ψ0 shows the most stable performance and achieves an average overall success rate at least 40% higher than GR00T-N1.6.

  • Baseline Characteristics: InternVLA-M1 demonstrates strong spatial reasoning but exhibits action jitter across consecutive action chunks, resulting in unstable executions.Its pre-training focuses on spatial reasoning and robotic arm data, limiting performance on humanoid tasks.
  • Baseline Characteristics: H-RDT excels at tasks that do not require precise movements but struggles with high-precision manipulation across many joints.H-RDT is a single 2B-parameter DiT action expert.
  • Baseline Characteristics: EgoVLA shows limited performance on lower-body commands because its pre-training primarily captures upper-body and hand manipulation skills.The reported limitation concerns coordinated lower-body motion.
  • Real-World Benchmark: Ψ0 outperforms all baselines by a large margin across eight long-horizon dexterous loco-manipulation tasks.The benchmark reports task-wise and aggregated skill-level success rates.
  • Real-World Benchmark: 40% higher average overall success rate is achieved by Ψ0 than by the second-best baseline, GR00T-N1.6.GR00T-N1.6 is identified as the most recently released humanoid foundation model in the comparison.
  • Real-World Benchmark: Ψ0 exhibits the most stable performance across all eight long-horizon dexterous loco-manipulation tasks.The evaluated tasks include distinct manipulation and locomotion challenges.

C. Ablation Studies

The ablations examine pre-training, post-training, action-expert design, and real-time chunking. They show that human-data pre-training and robot-data post-training improve performance, while MM-DiT strengthens VLM-conditioned action prediction and RTC addresses inference latency.

  • Pre-Training and Post-Training: Pre-training the VLM on EgoDex and post-training the action expert on high-quality robot data progressively improve overall performance.Freezing the text-pre-trained Qwen3-VL backbone and fine-tuning only the action head yields an overall success rate of 0.2.
  • Pre-Training and Post-Training: 0.2 overall success rate is obtained when the text-pre-trained Qwen3-VL backbone is frozen and only the action head is fine-tuned.The ablation studies a dual-arm long-horizon task involving right-arm pick-and-place, left-arm pick-and-place, and dual-arm lifting.
  • MM-DiT versus Naive DiT: MM-DiT consistently outperforms the naive DiT variant for action prediction.The authors attribute the improvement to dual modulation and joint attention integrating VLM features with action-branch representations.
  • Real-Time Chunking: Training-time real-time chunking is introduced because large VLAs can take more than 200 ms to generate actions, causing pauses and jitter.The method modifies training by randomly masking the first d action tokens, with dmax set to 6 in the experiments.
  • Conclusion: Ψ0 combines high-quality egocentric videos, real-robot data, and real-time control techniques to achieve state-of-the-art performance with substantially less real-world robotic data.The conclusion frames the recipe as scaling the right data in the right way rather than blindly increasing teleoperation volume.
  • Limitations: Training cannot yet be scaled to larger human-video and real-world-robot datasets, and the hardware payload limits potentially more capable manipulation behaviors.These are stated as limitations caused by compute and time constraints and by the hardware platform.

A. Pre-Training

The training pipeline processes human and humanoid data for efficient pre-training and post-training, while real-time control techniques address latency and action-chunk transitions. The implementation includes tokenizer retraining, data normalization, embodiment alignment, and asynchronous inference.

  • Data Processing: The FAST tokenizer is retrained from scratch on 500,000 randomly sampled actions after the original tokenizer shows a reconstruction loss of 0.583×10^-4.The original tokenizer performs particularly poorly under noisy token settings.
  • Post-Training Data: Humanoid Everyday data from two embodiments is aligned by reordering joint indices, producing a 28-DoF action representation and padding it to 36 DoF for future fine-tuning.The padding includes lower-body control dimensions absent from the Humanoid Everyday dataset.
  • Post-Training: During post-training, the VLM backbone is frozen while only the action expert is optimized for 30k steps using a global batch size of 2048.Training takes approximately 30 hours on 32 A100 GPUs.
  • Real-Time Control: Naive synchronous inference causes pauses and jitter because modern VLAs have billions of parameters and substantial inference latency.Naive action chunking reduces delay but can introduce jittery transitions due to randomness and discontinuity.
  • Real-Time Control: Training-time real-time chunking randomly masks the first d action tokens to improve action-chunk transitions during deployment.The masking range is d ∈[1, dmax], with dmax set to 6 in the reported experiments.

B. System Implementation

The deployment system separates observation and execution from control and inference, using a 30Hz control loop and asynchronous action generation. A headset-based multi-target IK pipeline supplies whole-body control variables to a locomotion policy for stable coordinated motion.

  • System Architecture: The real-time action chunking system uses a client for observation collection and action execution and a server for control and inference.The Control Loop runs at 30Hz and coordinates observation updates and action dispatch.
  • System Architecture: Asynchronous inference is triggered when the current action chunk passes a threshold and switches chunks before the previous one finishes.This design prevents system interruptions during action execution.
  • Whole-Body Control: Head and wrist poses from a PICO4U headset and two wrist trackers are treated as three end-effectors in a multi-target inverse-kinematics problem.The solver outputs arm joint positions, torso orientation, and pelvis height for downstream control.
  • Whole-Body Control: The intermediate torso and pelvis variables are provided to a low-level locomotion RL policy, enabling coordinated whole-body control while maintaining balance and locomotion stability.The hierarchical design connects upper-body teleoperation signals with locomotion control.

B. Dexterous Manipulation

Ψ0 uses a teleoperation pipeline that separately tracks dexterous upper-body motion and commands locomotion, enabling stable whole-body execution across diverse long-horizon tasks.

  • Teleoperation Pipeline: MANUS gloves and PICO wrist trackers provide precise finger, hand, and wrist tracking for dexterous humanoid manipulation.Finger motions are retargeted to the G1’s three-finger hands, avoiding occlusion and out-of-view issues from vision-based tracking.
  • Teleoperation Pipeline: Ψ0 controls locomotion through high-level commands rather than directly retargeting whole-body SMPL motion, reducing foot drift and unstable lower-body behavior.The authors report that direct whole-body tracking can produce excessive corrective steps that hinder policy learning.
  • Teleoperation Pipeline: The teleoperation pipeline enables a single operator to perform stable whole-body teleoperation and complex dexterous loco-manipulation tasks.The framework maps upper-body motion to arm and hand control while generating lower-body pose through an RL-based policy.
  • Task Evaluation: Ψ0 is evaluated on eight real-world long-horizon tasks involving pick-and-place, pushing, wiping, dexterous manipulation, torso motion, squatting, and locomotion.The task set includes faucet turning, chip-tray extraction, cart pushing, object transfer, and coordinated whole-body actions.
  • Task Evaluation: The benchmark includes multi-step tasks requiring grasping, turning, walking, object placement, handing objects to a person, squatting, and recycling actions.Each task contains extended action sequences rather than isolated manipulation primitives.

B. Detail Evaluation Metrics

The evaluation examines deployment stability and reports that real-time action chunking improves Ψ0’s smoothness and stability, while showing comparable effects when added to GR00T.

  • Evaluation Protocol: The benchmark reports detailed sub-task progress, counting a trial as successful only when all three to five subtasks are completed.Boldface marks the best performance and underlining marks the second-best performance.
  • Deployment: The real-time deployment system separates policy inference from low-level control using asynchronous threads and a shared action buffer.The low-level control thread runs at 60 Hz to maintain stable lower-body locomotion and robot stability.
  • RTC Evaluation: RTC improves action smoothness and stability and can reduce failures such as collisions, with a slight empirical improvement in Ψ0 performance.The authors describe the possible contribution to task success rates as indirect.
  • RTC Evaluation: RTC achieves comparable performance on GR00T with and without RTC.The comparison is reported for the three-step task used in the GR00T evaluation.

B. Pre-Training on only 10% EgoDex

Ablations show that reducing EgoDex pre-training to 10% worsens performance, while omitting EgoDex particularly harms subtasks requiring precise manipulation; multi-task fine-tuning also lowers individual-task performance.

  • Pre-Training on only 10% EgoDex: Using only 10% of EgoDex leads to significantly worse performance on certain tasks and inferior overall performance than baseline Ψ0.Post-training and fine-tuning protocols are kept unchanged in this comparison.
  • Pre-Training on only 10% EgoDex: The Humanoid Everyday-only variant performs well on tasks without fine-grained manipulation but lags behind Ψ0 on precision-demanding subtasks.This ablation keeps the post-training and fine-tuning protocols the same as the baseline.
  • Multi-Task Fine-Tuning: Multi-task fine-tuning reduces performance on each individual task compared with single-task fine-tuning.The authors hypothesize that dispersing the learning objective causes underfitting.
Loading 2603.12263v1…