Source-linked AI summary
GigaBrain-0: A World Model-Powered Vision-Language-Action Model
GigaBrain Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jie Li, Jiagang Zhu, Lv Feng, Peng Li, Qiuping Deng, Runqi Ouyang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yang Wang, Yifan Li, Yilong Li, Yiran Ding, Yuan Xu, Yun Ye, Yukun Zhou, Zhehao Dong, Zhenan Wang, Zhichao Liu, Zheng Zhu
TL;DR
Real-world robot data collection is costly, time-consuming, and limited in diversity, constraining scalable VLA training. GigaBrain-0 uses world model-generated data together with RGB-D modeling and embodied Chain-of-Thought supervision, achieving strong generalization and performance across diverse real-world manipulation tasks while offering a lightweight edge variant.
Problem
Training generalist VLA robots depends on large-scale real-world interaction data that is expensive and time-consuming to collect and limited in diversity and scalability.
Method
GigaBrain-0 trains a VLA foundation model on diverse world model-generated trajectories and adds RGB-D input modeling plus embodied Chain-of-Thought supervision.
Results
GigaBrain-0 shows strong real-world performance and generalization across dexterous, long-horizon, and mobile manipulation, including appearance, placement, and viewpoint changes.
Takeaways & Limitations
World model-generated data provides a scalable alternative to traditional robot data collection, while GigaBrain-0-Small supports deployment on edge hardware such as NVIDIA Jetson AGX Orin.
Abstract
from arXiv · showhide
Training Vision-Language-Action (VLA) models for generalist robots typically requires large-scale real-world robot data, which is expensive and time-consuming to collect. The inefficiency of physical data collection severely limits the scalability, and generalization capacity of current VLA systems. To address this challenge, we introduce GigaBrain-0, a novel VLA foundation model empowered by world model-generated data (e.g., video generation, real2real transfer, human transfer, view transfer, sim2real transfer data). By leveraging world models to generate diverse data at scale, GigaBrain-0 significantly reduces reliance on real robot data while improving cross-task generalization. Our approach further improves policy robustness through RGBD input modeling and embodied Chain-of-Thought (CoT) supervision, enabling the model to reason about spatial geometry, object states, and long-horizon dependencies during task execution. This leads to substantial gains in real-world performance on dexterous, long-horizon, and mobile manipulation tasks. Extensive experiments demonstrate that GigaBrain-0 achieves superior generalization across variations in appearances (e.g., textures, colors), object placements, and camera viewpoints. Additionally, we present GigaBrain-0-Small, an optimized lightweight variant designed to run efficiently on devices such as the NVIDIA Jetson AGX Orin.
1. Introduction
GigaBrain-0 addresses the cost, time, diversity, and scalability limits of real-world robot data by combining world model-generated trajectories with RGB-D inputs and embodied reasoning. The model is evaluated across dexterous, long-horizon, and mobile manipulation, including appearance, placement, and viewpoint variations.
- World model-generated trajectories reduce reliance on costly real-world robot data while expanding variation in materials, colors, lighting, and viewpoints.This scalable pipeline is intended to support robust representations that transfer to real-world environments.
- RGB-D input improves spatial perception, while embodied Chain-of-Thought supervision supports long-horizon and fine-grained sequential manipulation.Depth contributes 3D geometry and spatial layout information, whereas intermediate reasoning represents trajectories and subgoals.
- Real-world deployments cover dexterous tasks such as laundry folding, long-horizon tasks such as table bussing, and mobile tasks such as moving boxes and laundry baskets.The evaluation spans multiple manipulation settings rather than a single task family.
- GigaBrain-0 generalizes across changes in appearance, object placement, and camera viewpoint, while GigaBrain-0-Small targets efficient deployment on NVIDIA Jetson AGX Orin.The lightweight variant is designed for edge hardware.
2. Related Works
Related work frames general-purpose robotic manipulation as a challenge of learning transferable, language-grounded policies from heterogeneous and increasingly cross-embodiment data. Recent world models extend this direction by generating synthetic experiences for embodied intelligence and autonomous-driving scenarios.
- Prior robotic manipulation approaches use heterogeneous datasets to learn transferable representations for robustness in complex and unseen scenarios.
- VLA frameworks combine visual observations, language instructions, and action sequences through autoregressive token prediction or continuous flow-matching models.
- Cross-embodiment datasets aggregate publicly available interaction logs to support data-hungry vision-language-action models.
- World model advances have encouraged synthetic data generation to narrow the sim-to-real divide in embodied intelligence, including complex traffic simulation for autonomous driving.
3. GigaBrain-0 Model
GigaBrain-0 is an end-to-end VLA model that combines multimodal semantic encoding, continuous action generation, RGB-D perception, and embodied intermediate reasoning. Its training objective jointly supervises trajectories, subgoals, discrete actions, and continuous action chunks while using Knowledge Insulation to reduce optimization interference.
- GigaBrain-0 combines PaliGemma2 multimodal encoding with a flow-matching action Diffusion Transformer to generate continuous action chunks.The mixture-of-transformers design separates semantic understanding from continuous action generation while retaining synergistic processing.
- RGB-D inputs extend visual perception with a depth channel, and random depth dropping preserves compatibility with RGB-only inference.SigLIP’s first convolution is expanded with zero-initialized depth kernels, while the encoder remains trainable.
- Embodied CoT represents manipulation trajectories, subgoal language, and discrete action tokens as intermediate reasoning signals.Ten learnable trajectory tokens use bidirectional visual attention, and a GRU decoder regresses 2D end-effector keypoints.
- Trajectory regression, autoregressive subgoals and discrete actions, and DiT-predicted action chunks are jointly optimized under one objective.The objective includes flow-matching noise and timestep terms, CoT masking, and predicted-versus-ground-truth trajectory keypoints.
- Knowledge Insulation prevents interference between action prediction and Embodied CoT optimization, avoiding manually assigned language-action loss weights.
4. GigaBrain-0 Data
GigaBrain-0 combines public and proprietary robot data with diverse world-model-generated trajectories to reduce dependence on narrowly collected real-world data. GigaWorld expands variation through appearance, viewpoint, simulation, human-video, and video-generation transfers while preserving physically plausible structure.
- GigaBrain-0 combines public datasets with 1182 hours of proprietary robot data collected across 3100m2 and five environment categories.
- GigaWorld expands training diversity through multiple complementary pipelines, including Real2Real, view, Sim2Real, human-video, multiview, and inverse-dynamics-conditioned video generation.
- Real2Real transfer changes textures, colors, lighting, and materials while using depth and edge controls to preserve motion, layout, and object structure.
- View transfer re-renders real demonstrations from novel camera perspectives while maintaining 3D scene consistency and task-relevant robot actions.
- Sim2Real transfer synthesizes physically plausible manipulation sequences in Isaac Sim and adapts simulated appearances to narrow the simulation-to-real domain gap.
- Human-video transfer maps wrist positions to simulated robot end-effectors and renders stabilized robot demonstrations that preserve task intent while removing visual and kinematic gaps.
- GigaWorld’s NATTEN, step distillation, and FP8 inference reduce video-generation denoising to one step and provide over a 50× speedup over baseline diffusion models.
5. Experiment
GigaBrain-0 is evaluated across dexterous, long-horizon, and mobile manipulation tasks, with world-model-generated data improving generalization to appearance, placement, and viewpoint shifts.
- 5.1–5.3. Real-world manipulation experiments: GigaBrain-0 achieves the highest success rate across dexterous, long-horizon, and mobile manipulation tasks, surpassing π0 by 10–30% where quantified.It is evaluated on PiPER and G1 platforms across laundry folding, paper towel preparation, table bussing, juice preparation, boxes moving, and laundry baskets moving.
- 5.1. Dexterous Manipulation Experiment: 30% and 10% higher task success rates than π0 are reported for laundry folding and paper towel preparation, respectively.Depth sensing is associated with improved spatial awareness for precise coordination in contact-rich dexterous tasks.
- 5.2. Long-horizon Experiment: Embodied Chain-of-Thought reasoning supports temporally ordered planning for reliable execution in table bussing and juice preparation.The reported workflows involve multiple ordered steps, including sorting dishes, dispensing water, and stirring mixtures.
- 5.3. Mobile Manipulation Experiment: GigaBrain-0 surpasses π0 by 10% in both boxes moving and laundry baskets moving while combining navigation and local manipulation policies.The system also recovers from minor localization or grasping errors through real-time replanning.
- 5.4. Manipulation Generalization Experiment: Increasing world-model-generated data sampling from 0% to 90% raises garment-folding success from poor single-appearance generalization to above 80%.Appearance-transferred data pushes success to nearly 70% at 50% sampling and above 80% at 75% and 90%.
- 5.4. Manipulation Generalization Experiment: View-transferred data improves robustness to novel camera viewpoints, with success exceeding 80% as generated-data sampling increases from 0% to 90%.The passage attributes this to viewpoint-invariant representations and geometrically consistent multi-view renderings.
6. Conclusion and Future Work
The paper concludes that world-model-generated trajectories address scalability and diversity limits in robot data, while RGBD modeling and embodied Chain-of-Thought support broad real-world generalization. It also introduces GigaBrain-0-Small for practical edge deployment and proposes using world models as interactive reinforcement-learning environments.
- 6. Conclusion: GigaBrain-0 uses photorealistic world-model trajectories to generalize across scene appearances, object placements, viewpoints, and diverse real-world robotic tasks.The conclusion covers dexterous manipulation, long-horizon operations, and mobile operations.
- 6. Conclusion: RGBD input modeling and embodied Chain-of-Thought supervision enhance spatial reasoning and sequential decision-making capabilities.
- 6. Conclusion: GigaBrain-0-Small is optimized for edge deployment on platforms such as the NVIDIA Jetson AGX Orin, supporting real-time, on-device robotic control.
- 6. Future Work: Future work proposes using world models as interactive reinforcement-learning environments for trajectory rollouts, reward signals, and policy refinement.This direction is intended to reduce reliance on real-world trial-and-error while supporting simulated experience.