Source-linked AI summary
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong
TL;DR
Visual robot manipulation faces limited robot data, so this paper studies whether large-scale video generative pre-training can transfer useful representations to the task. It introduces GR-1, a unified GPT-style model that predicts actions and future images before fine-tuning on robot data, and reports improved multi-task performance and generalization on CALVIN and a real robot.
Problem
Robot data is sparse because its collection often requires costly and time-consuming human demonstrations, motivating investigation of large-scale generative pre-training for visual manipulation.
Method
GR-1 is a GPT-style model that takes language instructions, observation images, and robot states, predicts actions and future images, and is pre-trained on video before robot-data fine-tuning.
Results
GR-1 outperforms baseline methods across CALVIN multi-task, unseen-scene, and data-efficiency settings, and also outperforms baselines in real-robot experiments.
Takeaways & Limitations
Large-scale video generative pre-training benefits visual robot manipulation learning and supports generalization to unseen scenes and objects within the evaluated settings.
Abstract
from arXiv · showhide
Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io
1 INTRODUCTION
The paper applies large-scale video generative pre-training to language-conditioned visual robot manipulation, introducing GR-1 as a unified GPT-style model fine-tuned on robot data. It reports gains on simulation and real-robot evaluations, including multi-task and generalization settings.
- Robot data is sparse because collecting demonstrations is costly and time-consuming, motivating scalable pre-training approaches.
- Video generative pre-training is presented as closely related to robot action learning because robot trajectories contain video sequences.Forecasting future frames from past images and language instructions is intended to support anticipation of forthcoming events and relevant actions.
- GR-1 takes language instructions, observation images, and robot states, then predicts robot actions and future images end-to-end.The unified GPT-style model is first pre-trained on video prediction and subsequently fine-tuned with robot data.
- 94.9% success rate is reported for GR-1, improving over the 88.9% baseline on the CALVIN benchmark.
- GR-1 is evaluated extensively in simulation and the real world across multi-task visual robot manipulation settings.
2 RELATED WORK
Related work studies language-conditioned visual robot manipulation, transformer-based sequential decision-making, and pre-training approaches for robot learning. Existing methods combine language models, action policies, multimodal transformers, or large-scale data generation in different ways.
- Language-conditioned visual robot manipulation lets non-experts instruct robots to perform different tasks.
- Some methods use large language models to plan over task domains and pass instructions to low-level action policies.
- Other methods use language models to scale data generation and leverage diffusion policies for robot action generation.
- Transformer-based approaches model sequential decision-making, including autoregressive action prediction and multimodal prompting for generalist manipulation.
- Pre-training for robot learning has become an active research topic across numerous recent studies.
3 METHOD
GR-1 formulates language-conditioned visual robot manipulation as a multimodal sequence-prediction problem, mapping instructions, observations, and robot states to future images and actions. Its GPT-style architecture supports video generative pre-training followed by end-to-end robot-data finetuning.
- 3.1 PROBLEM FORMULATION: Video generative pre-training predicts a future frame from a language description and preceding video frames.The model receives frames from timestep t−h through t and predicts the frame at t+∆t.
- 3.1 PROBLEM FORMULATION: Language-conditioned manipulation maps an instruction, observation-image sequence, and robot-state sequence to a future image and robot action.Robot states include the end-effector pose and binary gripper status.
- 3.2 ARCHITECTURE: GR-1 is a GPT-style multimodal transformer that accepts different modalities and outputs future images and actions.Language uses CLIP, visual observations use a MAE-pretrained ViT with a perceiver resampler, and robot states use linear layers.
- 3.2.1 INPUTS: The model uses [ACT] tokens for arm and gripper action prediction and [OBS] tokens for future-frame prediction.The architecture includes language, robot-state, and image encoders, plus image and action decoders.
- 3.3 TRAINING: Robot-data finetuning optimizes GR-1 end-to-end with causal behavior-cloning and video-prediction losses.The training setup first pre-trains on video prediction, then finetunes on robot data; the CLIP text and MAE image encoders remain frozen.
4 EXPERIMENT
Experiments evaluate GR-1 on CALVIN and real-robot manipulation, including long-horizon tasks, data efficiency, unseen-scene generalization, and video prediction. GR-1 consistently outperforms the compared baselines, while retaining some failures on difficult articulated and unseen-object settings.
- CALVIN Benchmark: GR-1 improves CALVIN long-horizon average length from 3.06 for the best baseline to 4.21.It outperforms all baselines when completing sequences of 1–5 tasks on the ABCD→D split.
- CALVIN Benchmark: 85.4% success rate on ABC→D exceeds the best baseline's 53.3% for zero-shot unseen-scene generalization.The authors hypothesize that large-scale egocentric video pre-training provides robust visual-textual alignment across environments.
- CALVIN Benchmark: 77.8% success rate and 2.00 average length show GR-1's data efficiency using 10% of the full training dataset.The 10% setting contains 2,244 trajectories, compared with 22,966 total training trajectories.
- Real Robot Experiments: GR-1 outperforms baselines in all three real-robot object-transportation settings, with only a modest drop on unseen instances.On unseen categories, a typical error is confusing a bell pepper with a similarly colored peach.
- Real Robot Experiments: GR-1 outperforms two baselines by a large margin on drawer manipulation.Typical failures include incompletely closing the drawer and failing to engage the handle when opening it.
- Video Prediction: GR-1 reconstructs future frames on both CALVIN and real-robot data, although occluded-object details can be missing.The video-prediction signal can serve as a strong guide for action prediction.
5 CONCLUSION
The paper proposes GR-1, a GPT-style transformer that uses large-scale video generative pre-training for language-conditioned visual robot manipulation. Experiments on CALVIN and a real robot report improved performance across multiple generalization settings.
- GR-1 takes language instructions, observation images, and robot states, then predicts actions and future images end-to-end.
- GR-1 is pre-trained on language-conditioned video prediction and subsequently finetuned on robot data.
- GR-1 improves state-of-the-art methods on CALVIN across multi-task learning, unseen-scene, small-dataset, and unseen-language settings.
- Real-robot experiments show GR-1 outperforming a state-of-the-art method and generalizing to unseen scenes and objects.
A.1 NETWORK AND TRAINING DETAILS
GR-1 uses a causal transformer with separate output components for robot actions and video prediction. The model has 195M total parameters, including 46M trainable parameters.
- The causal transformer contains 12 layers, 12 heads, and a hidden size of 384.
- GR-1 contains 195M parameters, of which 46M are trainable.
- Action prediction uses a three-layer MLP with separate heads for arm and gripper actions.
- Video prediction uses a transformer composed of self-attention blocks and linear layers.
A.2 CALVIN BENCHMARK EXPERIMENTS
The CALVIN evaluation tests language-conditioned long-horizon manipulation through 1000 instruction chains, each containing up to five sequential tasks.
- The evaluation follows CALVIN’s protocol and uses 1000 unique sequence instruction chains.
- Each sequence requires the robot to understand five language instructions and continuously solve up to five tasks.
- A task is considered a failure if it is not completed within 360 timesteps.
- The next task is provided only after the current task is successfully completed.
A.3 REAL ROBOT EXPERIMENTS
Real-robot experiments use a 7-DoF Kinova Gen2 with wrist-mounted and static cameras to evaluate articulated-object manipulation and object transportation. Figure 7 summarizes ablations using CALVIN average task length and real-robot picking and transporting success rates.
- The real-robot platform is a 7-DoF Kinova Gen2 equipped with a wrist-mounted RealSense camera and a static Kinect Azure camera.
- Articulated-object experiments evaluate opening and closing a drawer.
- Object transportation experiments evaluate ten tasks involving placing objects onto a desk or plate.
- Figure 7 reports CALVIN average completed-task length and real-robot picking and transporting success rates for ablation studies.
- The real-robot experiments distinguish unseen instances from unseen categories, including tomato and yellow peach examples for unseen categories.
A.4 ABLATION STUDIES
The ablations examine how video prediction and the choice of prediction horizon affect GR-1’s visual robot manipulation learning. Predicting moderately future frames helps, while overly distant predictions provide diminishing guidance for immediate actions.
- Video Prediction & Pre-training: GR-1 is compared with variants trained from scratch that either remove video prediction or retain it without video pre-training.The first variant removes [OBS] tokens, while the second retains video prediction.
- Different Future Predictions: Increasing the prediction horizon from 1 to 3 steps improves CALVIN success rates.The comparison is reported in Table 5 without pre-training.
- Different Future Predictions: The improvement from longer-horizon prediction saturates soon.The authors hypothesize that predicting frames too far ahead may not guide immediate local action prediction well.
A.5 TASK SUCCESS RATES
The task-success evaluation reports improvements from video generative pre-training, especially on difficult block-manipulation tasks, alongside benchmark, rollout, and video-prediction evaluations.
- Task Success Rates: Video generative pre-training improves performance on tasks involving difficult block manipulation.These tasks require grasping the correct block and manipulating it according to the language instruction.
- Task Success Rates: Increasing training data improves block-manipulation tasks and substantially raises success on turning lights on or off.The comparison uses 10% data from the ABCD→D split.
- Task Success Rates: Table 7 reports task success rates for the evaluated manipulation tasks.The supplied passage identifies the table but does not provide its individual values.
- Evaluation Visualizations: The CALVIN evaluation includes rollouts and video-prediction results, with green boxes denoting ground truth and blue boxes denoting predictions.The supplied figures cover CALVIN rollouts and video prediction.
- Evaluation Visualizations: Real-robot rollout visualizations include object transportation and articulated object manipulation experiments.The first five rows show object transportation, while the last two show articulated manipulation.