Source-linked AI summary
GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, Minzhao Zhu
TL;DR
GR-2 addresses the challenge of building a generalist manipulation agent when large-scale robot data collection is costly and difficult to scale. It pre-trains on 38 million Internet videos, then fine-tunes on robot trajectories for joint video generation and action prediction, achieving strong multi-task learning and generalization across unseen scenarios.
Problem
Large-scale robot data collection remains costly and difficult to scale, motivating a foundation generalist manipulation agent trained on comprehensive data.
Method
GR-2 uses two-stage training: video generative pre-training on large-scale Internet videos, followed by robot-data fine-tuning to predict action trajectories and videos in tandem.
Results
97.7% success rate on 105 tasks demonstrates strong multi-task learning, while GR-2 also generalizes to unseen backgrounds, environments, objects, and tasks.
Takeaways & Limitations
GR-2 efficiently learns many tasks from limited robot data and shows potential for industrial manipulation through robust end-to-end bin picking of unseen objects.
Abstract
from arXiv · showhide
We present GR-2, a state-of-the-art generalist robot agent for versatile and generalizable robot manipulation. GR-2 is first pre-trained on a vast number of Internet videos to capture the dynamics of the world. This large-scale pre-training, involving 38 million video clips and over 50 billion tokens, equips GR-2 with the ability to generalize across a wide range of robotic tasks and environments during subsequent policy learning. Following this, GR-2 is fine-tuned for both video generation and action prediction using robot trajectories. It exhibits impressive multi-task learning capabilities, achieving an average success rate of 97.7% across more than 100 tasks. Moreover, GR-2 demonstrates exceptional generalization to new, previously unseen scenarios, including novel backgrounds, environments, objects, and tasks. Notably, GR-2 scales effectively with model size, underscoring its potential for continued growth and application. Project page: \url{https://gr2-manipulation.github.io}.
1 Introduction
GR-2 develops a generalist manipulation agent through large-scale video pre-training followed by robot-data fine-tuning. It targets broad task coverage, adaptation to novel scenarios, and scalable performance.
- Motivation: Foundation-model pre-training motivates adapting large, diverse datasets to versatile downstream manipulation tasks.The paper identifies large-scale robot-data collection as difficult because real-robot systems are inefficient and difficult to scale.
- Approach: GR-2 pre-trains on diverse daily-activity videos using text-conditioned autoregressive future-frame prediction.The pre-training data spans household, outdoor, workplace, and leisure contexts.
- Capabilities: Fine-tuning on robot trajectories enables learning multiple manipulation tasks and adapting to unseen backgrounds, environments, objects, and tasks.The paper reports learning over 100 tasks from 5,000 trajectories, averaging 50 trajectories per task.
- Contributions: GR-2 is pre-trained on 38 million text-video data amounting to over 50 billion tokens, while supporting over 100 manipulation tasks and bin-picking over 100 objects.The paper also introduces an architecture intended to transfer pre-training knowledge to downstream fine-tuning without loss.
2 Methods
GR-2 is a language-conditioned GPT-style visual manipulation policy trained in two stages: video generative pre-training and robot-data fine-tuning. It combines multimodal inputs with future-video and action-trajectory prediction for generalist control.
- Policy formulation: A universal policy maps language instructions, observation histories, and robot-state histories to action trajectories.The observation-history length is h, and the action-trajectory length is k.
- Model & Training: GR-2 first performs video generative pre-training, then fine-tunes on robot data to predict videos and action trajectories in tandem.The model is described as a language-conditioned GPT-style visual manipulation policy.
- Inputs: The training inputs include language, video frames, and robot states; robot states encode end-effector position, rotation, and binary gripper state.A frozen text encoder and frozen VQGAN tokenize language and images, while state encoders are trainable during fine-tuning.
- Video pre-training: During pre-training, a GPT-style transformer predicts future image tokens from tokenized text and image sequences, which are decoded into future images.This objective is intended to develop a prior for predicting future events and improve action prediction.
- Robot fine-tuning: Robot fine-tuning handles multiple camera views and outputs future images for each view together with an action trajectory generated by a conditional VAE.The real-robot deployment uses a WBC algorithm to optimize Cartesian trajectories and convert them into low-level joint actions.
3 Experiments
GR-2 is evaluated on real-robot multi-task manipulation, end-to-end bin picking, CALVIN, video generation, and model scaling. Across these settings, it shows strong task performance and generalization, including on unseen scenarios and with limited demonstrations.
- Real-World Multi-Task Learning: 97.7% success rate is achieved on 105 tasks in the Simple multi-task setting.With approximately 50 trajectories per task, success reaches 73.9% in Simple.
- Real-World Multi-Task Learning: 74.7% average success is achieved across three generalization settings with data augmentation.GR-2 reaches 87.0% in Unseen Environments, while the unaugmented model reaches 71.4% in Unseen Backgrounds and 71.7% in Unseen Environments.
- End-to-End Bin Picking: 79.0% average success improves substantially over GR-1's 33.3% in end-to-end bin picking.GR-2's success rates in Unseen and both Cluttered settings are comparable to its Seen-setting performance, including for transparent, deformable, and reflective objects.
- CALVIN Benchmark: 98.6% and 85.9% success are reported for completing 1 and 5 CALVIN tasks in a row, improving over GR-1's 94.9% and 73.1%.GR-2 establishes a new state of the art and outperforms the compared baselines in success rates and average length.
- Autoregressive Video Generation: Generated videos align faithfully with real-world rollouts, while actions are predicted alongside the videos.The predicted action can be viewed as replaying the trajectory in the generated video.
- Scaling: Success rate scales well as model size increases from 30M to 719M trainable parameters.Validation loss for video prediction also decreases with model size.
4 Related Work
Related work develops generalist robot manipulation through language-conditioned policies, large-scale robot data, cross-domain pre-training, and 3D or goal-image conditioning. Pre-training methods commonly learn visual representations before downstream policy learning.
- Generalist robot manipulation aims to accomplish diverse tasks across varied environments, often using natural language to specify tasks.
- Prior approaches learn generalist policies from large-scale robot datasets and combine robot data with other domains to improve generalization in unseen scenarios.
- Other methods condition policies on goal images or exploit 3D geometry, while some align goal-image and language latent spaces.
- Robot-learning pre-training commonly uses masked modeling or contrastive learning to obtain visual representations for downstream policy learning.
5 Conclusions
GR-2 is a generative video-language-action model pre-trained on Internet videos and fine-tuned on robot data to predict actions and videos. It learns over 100 real-world manipulation tasks, generalizes to unseen scenarios and objects, and shows correlated video and action predictions.
- GR-2 is pre-trained on 38 million Internet videos, then fine-tuned on robot data to predict action trajectories and videos together.
- GR-2 successfully completes more than 100 real-world manipulation tasks with a high success rate and generalizes to unseen backgrounds, environments, objects, and tasks.
- GR-2 performs end-to-end bin picking across over 100 objects and handles unseen objects with reported robustness.
- Generated video and the concurrently predicted action show a strong correlation.
- The authors identify improving robustness and generalization for unseen manipulation as future priorities.
Contributions & Acknowledgements
The acknowledgements recognize contributors across evaluation, model training, data collection, paper writing, engineering, and data processing. They also thank collaborators for discussion and advice.
- The evaluation team is credited for evaluation work.
- The model and training, data collection and curation, and paper writing teams are separately credited.
- ByteDance engineering and data teams are thanked for technical work, data collection, annotation, and processing.
- Xiao Ma is thanked for discussion and advice on paper writing.