PaperlayerView exact revision on arXiv

Source-linked AI summary

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, Yuke Zhu

arXiv:2503.14734v2cs.ROcs.AIcs.LG

TL;DR

General-purpose robots need models that can handle diverse real-world situations despite fragmented robot data. GR00T N1 addresses this with a cross-embodiment VLA model whose dual-system architecture is trained end-to-end on heterogeneous video, synthetic, and real-robot data. It outperforms imitation-learning baselines across simulation benchmarks and shows data-efficient performance on real GR-1 manipulation tasks.

  • Problem

    Robot foundation-model training is hindered by fragmented “data islands,” while general-purpose autonomy requires diverse data for varied embodiments and real-world tasks.

  • Method

    GR00T N1 combines a vision-language reasoning module with a Diffusion Transformer action module and co-trains them end-to-end across annotated videos, synthetic datasets, and real-robot trajectories.

  • Results

    GR00T N1 supports diverse manipulation across single-arm, bimanual, and humanoid embodiments, outperforming state-of-the-art imitation-learning baselines in simulation and performing strongly on real GR-1 experiments.

  • Takeaways & Limitations

    The open GR00T-N1-2B model, training data, and simulation benchmarks provide a generalist policy framework for multi-embodiment robot manipulation and further research.

  • Takeaways & Limitations

    The current model focuses primarily on short-horizon tabletop manipulation, while synthetic data generation remains limited in producing diverse counterfactual data consistent with physics.

Abstract

from arXiv · show

General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy in the human world. A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapidly learn new tasks. To this end, we introduce GR00T N1, an open foundation model for humanoid robots. GR00T N1 is a Vision-Language-Action (VLA) model with a dual-system architecture. The vision-language module (System 2) interprets the environment through vision and language instructions. The subsequent diffusion transformer module (System 1) generates fluid motor actions in real time. Both modules are tightly coupled and jointly trained end-to-end. We train GR00T N1 with a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets. We show that our generalist robot model GR00T N1 outperforms the state-of-the-art imitation learning baselines on standard simulation benchmarks across multiple robot embodiments. Furthermore, we deploy our model on the Fourier GR-1 humanoid robot for language-conditioned bimanual manipulation tasks, achieving strong performance with high data efficiency.

1. Introduction

GR00T N1 addresses the need for generalist robot autonomy with an open, cross-embodiment VLA model that combines dual-system reasoning and action generation. Its training strategy unifies heterogeneous web, video, synthetic, and real-robot data to support diverse manipulation behaviors and benchmark performance.

  • Model and motivation: GR00T N1 is an open VLA foundation model for generalist humanoid robots with cross-embodiment support from tabletop arms to dexterous humanoids.It is designed to generate actions from image observations and language instructions.
  • Training strategy: GR00T N1 organizes heterogeneous training data as a pyramid, combining large-scale web and human-video data, synthetic data, and real-robot trajectories.Data quantity decreases while embodiment specificity increases from the pyramid’s base to its top.
  • Training strategy: Action-less human and neural-generated videos are annotated with latent-action codebooks and inverse-dynamics-model pseudo-actions for unified model training.This lets action-less videos function as additional robot embodiments in the training mixture.
  • Results and scope: A unified model and single set of weights generate manipulation behaviors across single-arm, bimanual, and humanoid embodiments.On standard simulation benchmarks, GR00T N1 achieves superior results compared with state-of-the-art imitation-learning baselines and also performs strongly on real GR-1 robots.
  • Model and motivation: The dual-system architecture uses a VLM to interpret visual and language inputs, then a Diffusion Transformer to generate motor actions.The VLM processes tokenized image and language inputs; its outputs are combined with robot state and action encodings for action generation.

2. GR00T N1 Foundation Model

GR00T N1 combines vision-language reasoning with diffusion-based action generation in a unified, cross-embodiment policy. Its training strategy unifies human videos, synthetic trajectories, simulation data, and real-robot demonstrations, while embodiment-specific encoders support varied state and action dimensions.

  • Model Architecture: GR00T N1 integrates a VLM-based reasoning module with a DiT-based action module in one jointly trained framework.The VLM processes language and images, while the DiT generates actions through cross-attention to vision-language tokens.
  • Model Architecture: Embodiment-specific state and action encoders project varying robot dimensions into a shared embedding space for the DiT.This design supports training across single-arm robots, bimanual systems, and humanoid dexterous hands.
  • Model Architecture: The policy uses flow matching to iteratively denoise 16-action chunks conditioned on proprioceptive state, image tokens, and text tokens.During inference, four denoising steps work well across all embodiments.
  • Model Architecture: Eagle-2 supplies vision-language representations that condition the DiT, with GR00T-N1-2B using middle-layer representations from layer 12.The model processes images and task descriptions through the VLM before cross-attending to those features during action generation.
  • Training Data Generation: The data pyramid combines web and human videos, synthetic neural trajectories, simulation trajectories, and real-robot data across pre- and post-training.Latent-action representations and inverse dynamics provide action supervision for action-less videos, while neural and simulated trajectories expand training diversity.

3. Pre-Training Datasets

GR00T N1’s pre-training corpus combines real-robot, synthetic, and human-video datasets, with data-generation strategies matched to their available sources. These sources span teleoperated robot behavior, simulation and neural trajectories, and diverse human-object interactions.

  • The pre-training corpus is organized into real-robot, synthetic, and human video datasets, corresponding to the peak, middle, and base of the data pyramid.
  • 3.1. Real-World Datasets: Real-robot data centers on Fourier GR-1 teleoperation, capturing wrist poses, finger movements, images, proprioception, actions, and hierarchical task annotations.Teleoperator movements are retargeted to humanoid actions through inverse kinematics at 20Hz.
  • 3.2. Synthetic Datasets: Synthetic data includes simulation trajectories multiplied from human demonstrations and neural trajectories derived from videos produced by off-the-shelf neural generation models.Simulation trajectories use physics simulators, while neural trajectories are generated from video data.
  • 3.2. Synthetic Datasets: DexMimicGen generates large-scale GR-1 manipulation datasets from a few dozen teleoperated demonstrations by retargeting wrist and finger poses to a whole-body inverse-kinematics controller.
  • 3.2. Synthetic Datasets: Neural trajectories are produced by fine-tuning open-source image-to-video models on 3,000 language-annotated real-world robot samples.The trained model generates counterfactual trajectories from novel language prompts.
  • 3.3. Human Video Datasets: Human video datasets provide action-unlabeled sequences of grasping, tool use, cooking, assembly, and other human-object interactions from first-person and complementary exocentric perspectives.The collection includes Ego4D, Ego-Exo4D, Assembly-101, EPIC-KITCHENS, HOI4D, HoloAssist, and RH20T-Human.

4. Evaluation

GR00T N1 is evaluated across diverse simulated embodiments and real GR-1 manipulation tasks, where it consistently outperforms baselines and benefits from neural-trajectory co-training.

  • Evaluation setup: The evaluation spans three simulation benchmarks, multiple robot embodiments, and tabletop manipulation tasks on the GR-1 humanoid.Simulation includes two prior open-source benchmarks and a newly developed tabletop suite; real-world experiments use GR-1.
  • Pre-training evaluations: 76.6% and 73.3% success rates were achieved on GR-1 tasks requiring coordinated bimanual transfer and manipulation of novel objects.Each task used five objects with three trials per object.
  • Post-training evaluations: GR00T N1 consistently outperforms from-scratch baselines across simulation benchmarks and demonstration counts of 30, 100, and 300 per task.The comparison covers RoboCasa, DexMG, and GR-1 simulation benchmarks.
  • Post-training evaluations: 32.4% and 30.4% are the improvements over Diffusion Policy with 10% and full real-robot training data, respectively.With only 10% of the data, GR00T-N1-2B performs 3.8% below Diffusion Policy trained on the full dataset.
  • Neural trajectories: +4.2%, +8.8%, and +6.8% average gains result from neural-trajectory co-training at 30, 100, and 300 RoboCasa demonstrations.GR-1 real-world tasks show a +5.8% average gain across eight tasks.
  • Neural trajectories: LAPA slightly outperforms IDM at 30 demonstrations, while IDM’s advantage widens at 100 and 300 demonstrations.The authors attribute this trend to improved alignment of IDM pseudo-actions with real-world actions as IDM training data increases.
  • Limitations: The current model primarily targets short-horizon tabletop manipulation, while long-horizon loco-manipulation remains future work.Future extensions require advances in humanoid hardware, model architecture, and training corpora.

5. Related Work

Prior robotics work uses pretrained reasoning modules, teleoperated or instrumented demonstrations, simulation, and generative augmentation to address data scarcity and scalability.

  • Foundation models in robotics: Robotics foundation-model approaches commonly pair pretrained high-level reasoning modules with low-level robot-specific policies.This design allows robots to plan sequences of low-level actions through a separate policy.
  • Robot-learning datasets: Teleoperation produces high-quality embodied demonstrations but requires substantial time, human effort, and collection cost.Scaling with operator teams and robot fleets has yielded datasets containing thousands of demonstration hours.
  • Robot-learning datasets: Instrumented human demonstrations accelerate collection but introduce a mismatch with the downstream robot compared with direct teleoperation.Examples include handheld grippers, exoskeletons, and special glasses for capturing robot-relevant hand motions.
  • Synthetic data generation: Simulation offers more efficient data collection, but simulation-generated datasets face a simulation-to-reality gap.Synthetic data can be generated at large scale, while deployment to physical robots remains challenging.
  • Synthetic data generation: Neural video generation enables approximately 300k neural trajectories, equivalent to 827 hours of robot trajectories.GR00T N1 uses these trajectories alongside simulation and real-world data for co-training.

6. Conclusions

GR00T N1 is presented as an open foundation model for generalist humanoid robots, combining dual-system design, heterogeneous training data, and multi-embodiment support. Experiments report strong generalization and data-efficient learning of diverse manipulation skills.

  • GR00T N1 combines a dual-system model design, heterogeneous training data, and support for multiple robot embodiments.
  • The model is evaluated as a generalist policy across simulation benchmarks and on the real GR-1 humanoid robot.
  • Experiments demonstrate strong generalization capabilities for learning diverse manipulation skills with high data efficiency.
  • The authors release the GR00T-N1-2B model, training datasets, and simulation environments to support community progress.

A.1. Core Contributors

The core contributors cover model training, research leadership, infrastructure, experiments, data, open-sourcing, and program operations. Their roles span the main technical and organizational components of the project.

  • Model Training: Model training was led by Scott Reed, Ruijie Zheng, Guanzhi Wang, Johan Bjorck, Joel Jang, and collaborators.
  • Infrastructure and Experiments: Contributors handled real-robot and teleoperation infrastructure, simulation infrastructure, and simulation experiments.
  • Experiments and Data: Additional contributors worked on real-robot experiments, video generation models and latent actions, and data infrastructure and curation.
  • Operations and Open-Sourcing: Compute infrastructure and open-sourcing, along with program management and operations, were assigned to dedicated contributors.
  • Leadership: Research leads were Linxi “Jim” Fan and Yuke Zhu, while Spencer Huang served as product lead.

A.2. Contributors

This contributor listing names Ajay Mandlekar, Jan Kautz, Dieter Fox, Edith Llontop, Hao Zhang, and Guilin Liu. The passage provides no individual roles or task assignments.

  • Ajay Mandlekar and Jan Kautz are listed among the contributors.
  • Dieter Fox and Edith Llontop are listed among the contributors.
  • Hao Zhang and Guilin Liu are listed among the contributors.

B. Detailed Experiment Results

GR00T-N1-2B is compared with Diffusion Policy across simulation and real-world benchmarks using multiple dataset sizes. Performance improves with more demonstrations, while GR00T-N1-2B consistently outperforms the baseline.

  • GR00T-N1-2B consistently outperforms Diffusion Policy across all reported benchmarks and dataset sizes.The comparison covers simulation and real-world benchmarks.
  • Performance improves steadily for both GR00T-N1-2B and Diffusion Policy as dataset sizes increase.Simulation uses 30, 100, and 300 demonstrations; real-world benchmarks use 10% and full data.
  • Figure 10 reports average policy success rates on simulated manipulation tasks with varying numbers of demonstrations.

C. Additional Qualitative Results

Qualitative rollouts show that GR00T-N1-2B follows language instructions and generalizes to unseen bimanual manipulation situations. Post-training examples also show successful object placement where Diffusion Policy fails or remains stuck.

  • GR00T-N1-2B demonstrates language following and generalization to unseen situations in bimanual manipulation tasks.
  • Despite an unseen apple placement setup, the pretrained model completes a two-handed handover into a basket, with jerkier motion.
  • The post-trained model places a cucumber into a basket while Diffusion Policy fails because of an inaccurate grasp.
  • The post-trained model picks a lemon from a cutting board and puts it into a pan, whereas Diffusion Policy remains stuck.
  • Post-training uses smaller batch sizes than pre-training to reduce overfitting in data-limited fine-tuning settings.

E.1. Dataset Formats

The dataset pipeline extends the LeRobot format to represent heterogeneous, cross-embodiment robotics data with semantically structured modalities and standardized action spaces. These choices support multi-view and neural trajectory data while preserving training compatibility.

  • E.1. Dataset Formats: The training corpora build on LeRobot, an open-source format for storing, sharing, and using robot demonstration data.
  • E.1. Dataset Formats: Dataset examples pair language prompts with multi-view observations, such as left, right, and wrist camera views.
  • E.1. Dataset Formats: Multi-view trajectories are represented by concatenating views into a 4-grid video, and consecutive sequences can connect through shared boundary frames.
  • E.1. Dataset Formats: LeRobot stores robot states, actions, and metadata in parquet files, while visual observations use MP4 videos or PNG image sequences referenced from those files.
  • E.1. Dataset Formats: Demonstrations are organized into episodes containing synchronized observation-action pairs, including camera imagery, robot state, and control commands.
  • E.1. Dataset Formats: The extended format adds modality configuration and fine-grained semantic fields to support cross-embodiment data and richer VLA training regimes.
  • E.1. Dataset Formats: Semantic modality definitions improve interpretability and enable field-specific normalization and transformation during VLA training.
  • E.2. Standardized Action Spaces: Action and state spaces are unified across embodiments using 6D rotation states, axis-angle rotation actions, and scaling normalization.

F. Additional Training Details

Additional training details combine auxiliary spatial supervision, synthetic neural trajectories, and inverse dynamics modeling with diverse human-video and evaluation datasets. The appendix also reports dataset sizes, hyperparameters, and real-world task tables.

  • An auxiliary object-detection loss trains the model to localize the instructed target object while predicting actions.
  • Neural trajectories are generated by LoRA-finetuning WAN2.1-I2V-14B on teleoperation data and filtering videos that fail to follow language instructions.
  • The inverse dynamics model predicts action chunks between current and future images using the System 1 diffusion transformer module.
  • Pre-training includes seven human video datasets with corresponding language annotations, including Ego4D and EgoeXO-4D examples.
  • The appendix reports simulation results by dataset size, GR-1 real-world success rates, training hyperparameters, and pre-training dataset statistics.
Loading 2503.14734v2…