Source-linked AI summary
GR-3 Technical Report
Chilam Cheang, Sijin Chen, Zhongren Cui, Yingdong Hu, Liqun Huang, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Xiao Ma, Hao Niu, Wenxuan Ou, Wanli Peng, Zeyu Ren, Haixin Shi, Jiawen Tian, Hongtao Wu, Xin Xiao, Yuyang Xiao, Jiafeng Xu, Yichu Yang
TL;DR
Generalist robot policies must operate across diverse novel scenarios while following complex instructions and executing long-horizon dexterous tasks. GR-3 combines vision-language, robot-trajectory, and human-trajectory learning in an end-to-end VLA model, and real-world experiments show it outperforms π0 across three challenging task settings. The report also identifies remaining failures on some unseen concepts, shapes, and out-of-distribution rollout states.
Problem
Robot policies need strong generalization, instruction following, robustness, and reliability, while VLA models still face out-of-distribution instructions and large demonstration requirements.
Method
GR-3 combines co-training on large-scale vision-language and robot-trajectory data with few-shot fine-tuning from human trajectories in an end-to-end VLA model.
Results
GR-3 consistently outperforms π0 across generalizable pick-and-place, long-horizon table bussing, and dexterous cloth manipulation experiments.
Takeaways & Limitations
GR-3 generalizes to novel objects, environments, and instructions, adapts from minimal human trajectories, and performs long-horizon dexterous tasks robustly.
Takeaways & Limitations
GR-3 still makes mistakes on unseen instructions involving novel concepts and objects, struggles with unseen object shapes, and may fail to recover from out-of-distribution rollout states.
Abstract
from arXiv · showhide
We report our recent progress towards building generalist robot policies, the development of GR-3. GR-3 is a large-scale vision-language-action (VLA) model. It showcases exceptional capabilities in generalizing to novel objects, environments, and instructions involving abstract concepts. Furthermore, it can be efficiently fine-tuned with minimal human trajectory data, enabling rapid and cost-effective adaptation to new settings. GR-3 also excels in handling long-horizon and dexterous tasks, including those requiring bi-manual manipulation and mobile movement, showcasing robust and reliable performance. These capabilities are achieved through a multi-faceted training recipe that includes co-training with web-scale vision-language data, efficient fine-tuning from human trajectory data collected via VR devices, and effective imitation learning with robot trajectory data. In addition, we introduce ByteMini, a versatile bi-manual mobile robot designed with exceptional flexibility and reliability, capable of accomplishing a wide range of tasks when integrated with GR-3. Through extensive real-world experiments, we show GR-3 surpasses the state-of-the-art baseline method, $π_0$, on a wide variety of challenging tasks. We hope GR-3 can serve as a step towards building generalist robots capable of assisting humans in daily life.
1 Introduction
GR-3 addresses the need for generalist robot policies that can generalize across diverse scenarios, follow complex instructions, and robustly perform long-horizon dexterous tasks. It combines vision-language, robot-trajectory, and human-trajectory data to support these capabilities.
- Motivation: Generalist robots must handle novel scenarios, long-horizon tasks, and complex dexterous manipulation in diverse real-world settings.These requirements make strong generalization, robustness, and reliability central challenges.
- Motivation: VLA models enable robots to perform tasks from natural-language instructions, but out-of-distribution instruction following and large demonstration requirements remain challenges.The difficulties include unseen object categories and complex concepts requiring sophisticated reasoning.
- GR-3 Overview: GR-3 combines vision-language, robot-trajectory, and human-trajectory data in a large-scale VLA model for generalization, adaptation, and robust task execution.The model is designed to generalize to novel objects, environments, and instructions while learning from few-shot human trajectories.
- Experimental Results: Extensive real-world experiments across generalizable pick-and-place, long-horizon table bussing, and dexterous cloth manipulation show GR-3 consistently outperforms π0.GR-3 also adapts to novel objects with only 10 human trajectories per object and achieves high average task progress on challenging tasks.
- Capabilities: GR-3 strictly follows instructions involving abstract concepts and performs robustly on long-horizon table bussing and dexterous cloth manipulation.These capabilities are highlighted as central examples of the model’s generalization and dexterity.
- Training Recipe: GR-3 is co-trained on robot trajectories and vision-language data using flow matching and next-token prediction, respectively.The two objectives correspond to action learning and vision-language learning.
2 The GR-3 Model
GR-3 is an end-to-end VLA model that processes language, multi-view observations, and robot state to generate action chunks for a bi-manual mobile robot. Its architecture uses a pre-trained VLM and an action diffusion transformer with flow matching, while design changes improve training stability and inference speed.
- Architecture: GR-3 generates k-length action chunks conditioned on language instructions, observations, and robot state to control a bi-manual mobile robot.It uses multiple camera views and processes the inputs end to end.
- Architecture: The model combines a pre-trained VLM with an action diffusion transformer that predicts actions using flow matching.The action transformer receives the robot state and VLM key-value cache as conditioning information.
- Action Prediction: Causal attention in the action diffusion transformer models temporal dependencies within each action chunk.The chunk is represented as k tokens and combined with a robot-state token.
- Efficiency: The action diffusion transformer uses half as many layers as the VLM backbone and only its latter-half key-value cache for fast inference.GR-3 contains 4B parameters in total.
- Training Stability: Additional RMSNorm layers inside attention and feed-forward networks substantially improve training stability and language-following capability.The improvement addresses instability observed during early training explorations.
3 Training Recipe
GR-3 combines robot, human, and vision-language data through imitation learning, co-training, and few-shot adaptation. This recipe targets generalization, efficient adaptation, and robust performance on long-horizon and dexterous tasks.
- Training data: GR-3 trains on robot trajectory, human trajectory, and web-scale vision-language data to support generalization and robust task performance.Robot trajectories provide imitation-learning data, while vision-language and human trajectories support co-training and few-shot generalization.
- Imitation learning with robot trajectory data: GR-3 uses flow matching to supervise action prediction, adding noise to action chunks during training and integrating from random noise with Euler steps during inference.The flow-matching timestep is sampled uniformly, and multiple timesteps can be evaluated in one VLM forward pass to accelerate training.
- Auxiliary supervision: A task-status action dimension provides auxiliary supervision intended to reduce reliance on spurious multi-view correlations during action prediction.The status labels are Ongoing, Terminated, and Invalid.
- Co-training with vision-language data: Vision-language co-training jointly optimizes next-token prediction and flow matching across vision-language and robot-trajectory mini-batches.Vision-language data trains the VLM backbone, while robot trajectories train both the backbone and action DiT.
- Few-shot generalization with human trajectory data: Human trajectories enable few-shot adaptation to novel settings using approximately 450 trajectories per hour from VR-based collection.The human data is incorporated alongside robot trajectories and vision-language data during additional co-training.
4 Hardware & System
ByteMini is a 22-DoF bi-manual mobile robot designed for flexible manipulation, reliable operation, and user-friendly deployment. Its whole-body control and teleoperation systems support dexterous, mobile, and long-horizon policy data collection and rollout.
- The ByteMini Robot: ByteMini is a 22-DoF bi-manual mobile robot designed for flexible manipulation, high reliability, and user-friendliness.It is used for GR-3 data collection and policy rollout.
- Flexible manipulation: The sphere wrist and 7-DoF arms provide human-like dexterity in confined spaces, with elbow adduction reaching 2.53 rad.The compact wrist addresses dimensional limitations of traditional SRS-configured arms.
- High reliability: An omni mobile platform, lift mechanism, and QDD-based actuators support stable spatial mobility, height adjustment, and consistent arm motion.The design targets the demanding workload of data collection and policy rollout.
- System & control: Whole-body compliance control jointly handles manipulability, singularity avoidance, and joint limits to retarget human motion into feasible robot motion.The resulting controller supports continuous motion across large workspaces and physical interaction with the environment.
- System & control: Whole-body VR teleoperation simultaneously controls the arms, lift, gripper, and mobile base for complex long-horizon tasks.Policy rollout controls 19 DoFs and applies pure pursuit plus trajectory optimization to reduce jerk and smooth waypoint transitions.
5 Experiments
The experiments evaluate GR-3 on instruction following, out-of-distribution generalization, few-shot adaptation, and robust long-horizon manipulation across challenging real-world tasks.
- Experimental goals: The evaluation targets instruction following, out-of-distribution generalization, few-shot human-data learning, and robust long-horizon dexterous control.These questions are tested across generalizable pick-and-place, table bussing, and cloth manipulation tasks.
- Generalizable pick-and-place: The pick-and-place study uses 35k robot trajectories covering 101 objects and co-trains GR-3 with robot trajectories and vision-language data.Photometric augmentation is applied to improve robustness to changing environments, alongside a no-co-training ablation.
- Generalizable pick-and-place: The study evaluates four pick-and-place settings: Basic, Unseen Environments, Unseen Instructions, and Unseen Objects.Unseen Objects uses 45 objects absent from robot trajectory data, while Unseen Instructions test abstract concepts and complex semantics.
- Generalizable pick-and-place: GR-3 surpasses π0 in Basic and Unseen Environments for both instruction-following and success rates, with only modest degradation under environment changes.Co-training does not significantly affect performance on seen objects in these settings.
- Generalizable pick-and-place: GR-3 increases success from 40% to 77.1% on Unseen Instructions and from 40% to 57.8% on Unseen Objects, outperforming π0 and the no-co-training variant.The results indicate that vision-language co-training contributes to generalization in novel settings.
- Few-shot generalization: Few-shot adaptation is evaluated with human trajectories collected using VR devices, focusing on transfer across embodiments and scarce data.The evaluation compares performance across different numbers of human demonstrations.
1) the model
GR-3 is evaluated for few-shot adaptation, long-horizon table bussing, and dexterous cloth manipulation, where it achieves strong performance across varied settings.
- Few-shot generalization: With 10 human trajectories per unseen object, GR-3 raises success from 57.8% to 86.7% without evident performance loss on seen objects.The study evaluates 1-, 5-, and 10-shot settings and uses up to 450 human trajectories lasting about 30 minutes.
- Long-horizon table bussing: Table bussing requires packing food, sorting utensils and trash, and moving the mobile base across a large workspace.The study evaluates both autonomous flat execution and instruction following with multiple sub-task descriptions.
- Long-horizon table bussing: The flat setting measures average task progress, while the instruction-following setting measures average sub-task success across six instruction sets.Average task progress is the ratio of completed sub-tasks to total sub-tasks; 1.0 denotes complete success.
- Long-horizon table bussing: GR-3 achieves 97.5% success versus π0’s 53.8% in instruction-following table bussing and follows instructions across all six test sets.It also generalizes to multiple objects and destinations and rejects invalid tasks.
- Long-horizon table bussing: Removing RMSNorm or task-status supervision reduces instruction-following performance, especially for novel destinations and the instruction-following setting.The ablations identify both components as contributors to the reported instruction-following capability.
- Dexterous cloth manipulation: GR-3 hangs clothes through coordinated hanger pickup, cloth placement, and mobile-base movement between the table and drying rack.Evaluation covers seen clothes, challenging positional layouts, and unseen clothes with novel patterns or sleeve lengths.
- Dexterous cloth manipulation: GR-3 reaches 86.7% average task progress in Basic, 83.9% in Position, and 75.8% on unseen clothes.The hardest milestone is placing the left shoulder after the right shoulder, partly because the left collar is often folded behind the hanger.
6 Related Work
Prior work develops generalist manipulation through large-scale representations and multimodal data, while human data is increasingly used to improve policy training efficiency.
- Generalist Manipulation Policies: Generalist manipulation policies aim to follow instructions and interact effectively with the physical world despite the longstanding difficulty of robust behavior.Prior approaches learn representations from large-scale data for downstream policy learning.
- Multi-Modal Co-Training for Robot Manipulation: Multimodal co-training broadens policy-training data beyond costly real-world robot trajectories.Earlier approaches initialize policies from pretrained vision or vision-language encoders, while other systems combine modalities directly.
- Leveraging Human Data for Policy Training: Human videos, large-scale video pretraining, hand tracking, and VR devices represent complementary strategies for leveraging human data in policy training.These approaches seek to improve data efficiency by incorporating human-derived representations or trajectories.
7 Limitations & Conclusions
GR-3 combines vision-language co-training, few-shot human trajectories, and robot imitation learning to support generalist manipulation, while retaining important limitations.
- Limitations: GR-3 makes mistakes on unseen instructions involving novel concepts and objects and struggles to grasp objects with unseen shapes.Future work proposes scaling the model and training data to improve handling of novel scenarios.
- Limitations: Like other imitation-learning methods, GR-3 can become stuck in out-of-distribution rollout states and fail to recover from failure.The authors propose reinforcement learning as a future direction for improving robustness and performance beyond imitation learning.
- Conclusions: The report concludes that GR-3 combines multimodal training and few-shot adaptation with end-to-end control of a bi-manual mobile robot.Real-world experiments cover abstract instructions, novel objects and environments, few-shot learning, and long-horizon dexterous tasks.
8 Contributions and Acknowledgements
The paper credits contributions across data, model architecture, training, evaluation, robot development, robot systems, writing, and team leadership, while acknowledging support for curation, deployment, maintenance, data collection, and annotation.
- Contributors are credited for data, model architecture, training, evaluation, robot development, robot systems, writing, and team leadership.
- Writing contributions are credited to Sijin Chen, Yuxiao Liu, Xiao Ma, Zeyu Ren, Hongtao Wu, and Yichu Yang.
- Team leadership is credited to Tao Kong and Hang Li.
- Data curation, model deployment, robot development and maintenance, data collection, and annotation received additional acknowledged support.