Source-linked AI summary
RoboCLIP: One Demonstration is Enough to Learn Robot Policies
Sumedh A Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, Laurent Itti
TL;DR
Reward design and conventional imitation learning require substantial supervision or large in-domain demonstration datasets. RoboCLIP uses a pretrained Video-and-Language Model to generate rewards from one language or video demonstration, achieving 2–3× higher zero-shot performance than competing imitation-learning methods on downstream robot manipulation tasks.
Problem
Reward design is difficult and expert demonstrations used by imitation-learning methods typically require large, expensive, in-domain datasets.
Method
RoboCLIP uses pretrained Video-and-Language Models to compare online agent rollouts with a single language or video demonstration and generate rewards without manual reward design.
Results
2–3× higher zero-shot task success than state-of-the-art imitation-learning baselines was observed across the evaluated Metaworld and Franka Kitchen environments.
Takeaways & Limitations
RoboCLIP can train robot policies from a single video demonstration or textual task description, including out-of-domain videos, with performance exceeding the evaluated baselines.
Takeaways & Limitations
VLM biases may propagate into reinforcement-learning agents, and downstream reward fine-tuning can be unstable because of the scale of VLM rewards.
Abstract
from arXiv · showhide
Reward specification is a notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imitation learning (IL) methods attempt to circumvent these problems by utilizing expert demonstrations but typically require a large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Language Models (VLMs), we present RoboCLIP, an online imitation learning method that uses a single demonstration (overcoming the large data requirement) in the form of a video demonstration or a textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains. RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration.
1 Introduction
RoboCLIP addresses the data and reward-design burden in imitation learning by using pretrained Video-and-Language Models to turn a single language or video demonstration into a reward. It achieves 2–3× higher zero-shot task success than state-of-the-art imitation-learning baselines on evaluated robotic environments.
- Reward design for complex, long-horizon reinforcement-learning tasks requires substantial human expertise and is often not principled.Manual reward evaluation may also require knowledge of the environment’s true state.
- Imitation learning reduces manual reward design but typically depends on massive expert datasets and tedious collection by robot operators.Performance and generalization hinge on demonstration coverage and dataset size.
- Naturalistic demonstrations, such as a human opening a cabinet in a separate environment, can be more intuitive than thousands of teleoperated target-environment trajectories.
- RoboCLIP uses pretrained VLM embeddings to compare an agent rollout with a single language instruction or video demonstration and treats the mismatch as a proxy reward.This avoids relying on large-scale, in-domain datasets for reward generation.
- 2–3× higher zero-shot task success was achieved than with state-of-the-art imitation-learning baselines on the Metaworld and Franka Kitchen environments.The rewards require no expert reward specification and support natural-language descriptions or human demonstrations.
2 Related Work
Prior imitation-learning methods learn from expert demonstrations, but large datasets are expensive, inverse reinforcement learning is difficult in high-dimensional spaces, and existing language-model reward approaches are not grounded in physical-world tasks.
- Behavioral-cloning imitation-learning methods formulate learning as supervised prediction over state-action pairs from large expert trajectory datasets.
- Large demonstration datasets are expensive to collect, while inverse reinforcement learning is difficult to apply in high-dimensional state and action spaces.
- Prior language-model reward-design methods study text-based games rather than physical-world tasks.RoboCLIP instead uses video-and-language models to assess alignment between robot-policy videos and expert demonstrations.
3 Method
RoboCLIP embeds agent interaction videos and task specifications in a shared pretrained S3D space, then uses their scalar-product similarity as a sparse episode reward for online reinforcement learning.
- RoboCLIP embeds agent-environment interaction videos into a semantically meaningful latent space using a pretrained S3D video-and-language model.The S3D model was pretrained on diverse human activity videos from HowTo100M.
- The encoded agent video and task descriptor are multiplied with a scalar product to produce the similarity score used as the episode reward.
- The method stores an episode’s observations, downsamples them to the model’s required video length, and computes a sparse trajectory-level reward.
- Task descriptors can be encoded from natural-language tokens into the same latent space as the agent video.
- For video task descriptions, RoboCLIP encodes the demonstration with the video encoder and compares it with the agent episode at the episode’s end.Intermediate timesteps receive zero reward.
- The resulting reward can train agents online with standard reinforcement-learning algorithms, including PPO, before zero-shot evaluation or optional true-reward fine-tuning.
4 Experiments
RoboCLIP experiments test whether pretrained VLMs can generate rewards from language, in-domain videos, out-of-domain videos, and multimodal task specifications. Across these settings, RoboCLIP enables zero-shot task completion or improves performance over imitation-learning baselines, while harder tasks and some conditioning choices expose limitations.
- Experimental Questions: The experiments evaluate VLM alignment and reward generation from language, expert videos, out-of-domain videos, and combined video-text task specifications.The study also examines which components are crucial for success, using PPO for RoboCLIP and baselines.
- Domain Alignment: Higher diagonal than off-diagonal similarities indicate that the pretrained VLM is aligned with Metaworld task descriptions and videos.The analysis compares pair-wise similarities between textual task descriptions and videos across tasks.
- Language for Reward Generation: Language-conditioned RoboCLIP rewards let agents complete tasks before true-reward finetuning and achieve more than double zero-shot rewards across environments.The reward is the VLM similarity between an encoded interaction video and a natural-language task description.
- In-Domain Videos for Reward Generation: Using one in-domain video demonstration, RoboCLIP solves Kettle, Hinge, and Slide tasks, whereas baselines generally fail to interact with the target objects.The learned policies can also reproduce motion characteristics from demonstrations, including wide swipes and wrist rotations, although they may use unconventional contact strategies.
- Out-of-Domain Videos for Reward Generation: Out-of-domain human or animated videos produce rewards that outperform GAIL and AIRL despite both baselines receiving the same single expert demonstration.The comparison uses videos recorded in environments different from the robot deployment environment.
- Multimodal Specification and Ablations: Multimodal task editing outperforms string-only task specification, while single-video conditioning works best and static-image CLIP provides little useful pretraining signal.The multimodal experiment edits a button-pushing video embedding toward a drawer-closing task; harder tasks can require single-demonstration finetuning, which improves rewards by 200%.
5 Conclusion
RoboCLIP distills pretrained Video-and-Language Model knowledge into online reinforcement-learning rewards, while the conclusion identifies model bias, finetuning instability, and deployment-time pretraining selection as unresolved limitations.
- Conclusion: RoboCLIP generates rewards from pretrained Video-and-Language Models and supports single video or textual task specifications, including out-of-domain videos and multimodal demonstrations.The method is presented as outperforming baselines across robotic environments.
- Conclusion: RoboCLIP outperforms the baselines in various robotic environments.
- Limitations and Broader Impact: VLM implicit biases may percolate into reinforcement-learning agents, but their form in those agents remains unclear.
- Limitations and Broader Impact: Finetuning on downstream task rewards can become unstable, potentially because low-magnitude VLM rewards distort normalized PPO Q-values.The authors state this matters especially for longer-horizon deployment, although RoboCLIP rewards already produce task-completing policies without deployment-environment finetuning in their experiments.
- Limitations and Broader Impact: RoboCLIP has no fixed pretraining length and currently selects the best model using the true task reward, requiring human monitoring when that reward is unavailable.The authors leave automated selection for future work.