Source-linked AI summary
Vision-Language Foundation Models as Effective Robot Imitators
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, Tao Kong
TL;DR
RoboFlamingo addresses the difficulty of using VLMs for low-level robot control, where static image-language training and language-token outputs do not directly match sequential action prediction. It adapts OpenFlamingo by separating single-step vision-language understanding from temporal decision-making with a policy head and imitation learning, achieving state-of-the-art benchmark performance and strong zero-shot generalization.
Problem
VLMs trained on static image-language pairs face challenges in robotics, which requires video comprehension and action representations for low-level control.
Method
RoboFlamingo uses OpenFlamingo for single-step visual-language understanding, an explicit policy head for history and actions, and imitation learning only on language-conditioned manipulation datasets.
Results
RoboFlamingo achieves state-of-the-art performance with a large margin over previous works on the CALVIN benchmark, including 2x improvement over the previous state-of-the-art method.
Takeaways & Limitations
RoboFlamingo is presented as a cost-effective, competitive alternative for adapting VLMs to robot control, with data efficiency and zero-shot generalization benefits.
Abstract
from arXiv · showhide
Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.
1 INTRODUCTION
RoboFlamingo addresses the challenge of adapting vision-language models to low-level robot control with a lightweight framework and reports strong benchmark performance.
- VLMs trained on static image-language pairs face challenges in robotics, which requires video comprehension for closed-loop control and action prediction beyond language tokens.
- RoboFlamingo decouples vision-language understanding from decision-making using OpenFlamingo, a policy head, and imitation learning on language-conditioned manipulation data.The backbone processes observations and instructions at each decision step, while the policy head models historical features.
- A small amount of robotics demonstration is used to adapt the model, while the decomposition supports open-loop control and deployment on low-performance platforms.
- 2x performance improvements compared with the previous state-of-the-art method are reported on the CALVIN benchmark for long-horizon language-conditioned tasks.
2 RELATED WORK
Prior approaches incorporate language and vision models into manipulation through planners, large end-to-end models, or fine-tuning, while RoboFlamingo emphasizes simpler adaptation with fewer resources.
- Language-conditioned manipulation requires robots to understand both visual observations and word-based instructions.
- Some approaches use LLMs as zero-shot planners that generate step-by-step plans for pre-trained low-level skill policies.
- PaLM-E co-fine-tunes a large vision-language model end-to-end using mobile-manipulation question-answering and auxiliary vision-language data.
- RoboFlamingo adapts OpenFlamingo with fine-tuning on a small number of manipulation demonstrations, reducing data-collection costs and computing consumption relative to prior approaches.
3 BACKGROUND
The background frames robot manipulation as partially observable, language-conditioned control and describes imitation learning from instruction-labeled expert trajectories.
- In CALVIN, robot observations include simulated camera views, while actions control a 7-DoF Franka Panda arm with a parallel gripper.
- RoboFlamingo uses a Flamingo backbone for single-step observations and a policy head for temporal features.
- Imitation learning trains agents to mimic manipulation plans from instruction-labeled expert play trajectories.
- The learning objective is maximum-likelihood, goal-conditioned imitation learning for the policy.
4 ROBOFLAMINGO
RoboFlamingo adapts the OpenFlamingo vision-language model to language-conditioned robot manipulation by separating vision-language understanding from sequential action prediction. It fuses visual observations with language goals, models history through a policy head, and trains pose and gripper outputs with imitation learning.
- 4 ROBOFLAMINGO: RoboFlamingo builds manipulation policies from OpenFlamingo by decoupling vision-language understanding from decision-making.The backbone processes visual observations and language goals, while a separate policy head predicts robot actions.
- 4.3 POLICY HEAD: A policy head converts the fused representation into low-level actions while modeling historical observations with an explicit sequential mechanism.The tested policy-head strategies include LSTM, decoder-only transformer, and single-step MLP variants; the LSTM example uses max pooling followed by recurrent processing.
- 4.2 VISION-LANGUAGE BACKBONE: At each decision step, the Flamingo backbone encodes camera observations and language goals into a fused vision-language representation.Two-view images are encoded by a ViT, compressed by a perceiver resampler, and fused with language through the decoder.
- 4.2.1 VISION ENCODER: The perceiver resampler uses learnable parameters and attention to compress the visual token sequence from N tokens to K tokens.The resampler parameters provide query vectors, while transformed visual features provide keys and values.
- 4.3 POLICY HEAD: RoboFlamingo predicts 7 DoF end-effector pose and gripper status, training pose with MSE regression and gripper status with BCE classification.The model fine-tunes the resampler, gated cross-attention modules, and policy head while freezing the remaining parameters.
5 EXPERIMENTS
Experiments evaluate RoboFlamingo on CALVIN for imitation, zero-shot generalization, and ablations of policy design and VLM pre-training. Results show strong benchmark performance, benefits from history, VL pre-training, larger models under limited data, and instruction fine-tuning.
- 5.1 BENCHMARK AND BASELINES: CALVIN evaluates 34 tasks through 1000 instruction chains, requiring consecutive completion of sequences up to five language instructions.The benchmark uses simulated long-horizon language-conditioned tasks and imitation-learning demonstrations.
- 5.2 IMITATION PERFORMANCE: RoboFlamingo outperforms all baseline methods across metrics in the ABCD →D imitation setting.The model is trained on language-annotated demonstrations from splits A, B, C, and D, then evaluated on split D.
- 5.2 IMITATION PERFORMANCE: RoboFlamingo achieves the highest success rate on later tasks, whose more diverse initial states test manipulation-policy generalizability.Later tasks depend on the ending states of earlier tasks and therefore expose greater variation in initial conditions.
- 5.3 ZERO-SHOT GENERALIZATION: RoboFlamingo significantly outperforms baselines in both vision generalization on ABC →D and language generalization with 50 synonymous instructions per task.Evaluation uses unseen visual contexts for vision generalization and randomly sampled GPT4-generated instructions for language generalization.
- 5.3 ZERO-SHOT GENERALIZATION: Freezing the fusion-decoder embedding layer improves generalization and reduces the performance drop on subsequent tasks.The reported drop is larger than HULC’s, which the authors associate with directly training on word tokens and their sensitivity to synonymous sentences.
- 5.4.1 POLICY FORMULATIONS: History-aware GPT and LSTM policy heads outperform MLP formulations, while MLP without history performs worst.The comparison is conducted on ABCD →D; the results indicate that observation history is important, but how history is integrated also matters.
- 5.4.2 VL PRE-TRAINING: VL pre-training improves downstream manipulation by a large margin, and fine-tuning the VLM is indispensable because the policy head has limited capacity.The ablations compare removing OpenFlamingo pre-training with freezing the pre-trained VLM and training only the policy head.
- 5.4.3 VL PRE-TRAINING FACTORS: With 10% of language-annotated CALVIN data, larger VLMs achieve much higher performance, indicating greater data efficiency.With full training data, the smaller model is competitive with the larger model.
6 CONCLUSION AND FUTURE WORK
RoboFlamingo demonstrates strong performance for language-conditioned robotic manipulation using pre-trained vision-language models. The paper also identifies data efficiency and zero-shot generalization as benefits, while leaving real-world deployment for future work.
- RoboFlamingo achieves state-of-the-art performance on a benchmark dataset for language-conditioned robotic manipulation.
- Pre-trained models provide benefits in data efficiency and zero-shot generalization ability.
- The paper does not deploy RoboFlamingo on real-world robotics because real-robot data is unavailable.
- Future work will examine RoboFlamingo on real-world tasks using large-scale real robotics data.
A.1 THE CALVIN BENCHMARK
CALVIN is an open-source simulated benchmark for long-horizon language-conditioned manipulation, organized into four environment splits and sequential instruction chains. Its evaluation tests whether robots can complete up to five consecutive goals, including under varied language expressions.
- CALVIN is an open-source simulated benchmark for evaluating long-horizon language-conditioned tasks.
- The benchmark contains four environments, each with 6 hours of teleoperated data and around 24 thousand language-annotated trajectories.
- CALVIN evaluates 1000 unique instruction chains across sequences of up to five consecutive language instructions.
- A robot advances to the next goal only after successfully completing the current instruction-conditioned task.
- GPT4 augments CALVIN instructions with differently worded expressions that retain the original meaning, and RoboFlamingo still outperforms HULC.
B.1 CO-TRAINING
The co-training experiments examine whether jointly preserving vision-language capabilities during robotics adaptation improves RoboFlamingo. Results indicate that ordinary fine-tuning can overfit, while comparisons favor pre-trained VLM adaptation over selected robotics representation models.
- Fine-tuning can cause foundation-capability loss through over-fitting, as shown by performance degradation on enriched instructions and auxiliary evaluations.
- Co-training is selected to prevent the foundation-capability loss observed with fine-tuned RoboFlamingo.
- The comparison evaluates co-trained and fine-tuned models on CALVIN, COCO image captioning, and VQAv2.
- Fine-tuning pre-trained VLMs shows a clear advantage over the evaluated R3M and Voltron robotics representation models.
- RoboFlamingo-style fine-tuning trains the resampler, gated cross-attention modules, and policy head while freezing the remaining parameters.
B.4 PERFORMANCE CURVES IN TRAINING OF DIFFERENT BACKBONES
Performance curves compare RoboFlamingo variants across training epochs, backbones, and CALVIN splits. Most variants converge within five epochs, while a task-sequence visualization contrasts RoboFlamingo’s completion with HULC’s failure on the third task.
- Most RoboFlamingo variants converge within 5 training epochs on the ABC →D and ABCD →D settings.
- The performance curves report results for different VLM backbones on the ABC →D split.
- On the illustrated ABC →D sequence, RoboFlamingo completes the entire task sequence while HULC becomes stuck at the third task.
- RoboFlamingo reaches the drawer in a dozen steps and releases the gripper, whereas HULC moves above the desktop for hundreds of steps without locating it.
B.6 DETAILED IMITATION PERFORMANCES ON EACH TASK
This section reports task-level imitation performance and rollout examples for RoboFlamingo on the ABCD →D split of CALVIN, alongside comparisons across policy-head formulations and training epochs.
- The ABCD →D split is used for rollout examples and for tracking VLM performance at each epoch.
- Table 9 reports success rates by task for RoboFlamingo variants, with each task evaluated 100 times.The reported models use their best checkpoint.
- MLP w/o hist uses only the current observation to predict actions, omitting observation history.
- MLP w hist incorporates history frames through positional embeddings, vision encoding, and cross-attention in the feature-fusion decoder.
- GPT and LSTM use the VLM backbone to process single-frame observations before integrating them through their respective policy formulations.