Source-linked AI summary
Real-World Robot Learning with Masked Visual Pre-training
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, Trevor Darrell
TL;DR
Real-world robot learning needs visual representations that transfer across tasks and embodiments from diverse, in-the-wild data. The paper pre-trains a masked autoencoder, freezes its encoder, and learns control policies on top, finding stronger performance and sample efficiency than several baselines while benefiting from scale. Its evidence remains bounded by mostly clean lab scenes, single-object tasks, and the absence of feedback control and multi-step planning.
Problem
The paper asks how representation learning from diverse real-world visual data can transfer effectively to real-world robotic tasks across tasks and embodiments.
Method
The method pre-trains a masked autoencoder on Internet and egocentric images and videos, freezes the vision encoder, and learns behavior-cloned control policies on top.
Results
The encoder consistently outperforms CLIP by up to 75%, supervised ImageNet pre-training by up to 81%, and training from scratch by up to 81% across real-world evaluations.
Takeaways & Limitations
Scaling to a 307M-parameter encoder trained on 4.5M images demonstrates benefits of visual pre-training scale for robot learning.
Takeaways & Limitations
The study mostly uses toy objects in relatively clean lab environments, with single-object tasks that do not require feedback control or multi-step planning.
Abstract
from arXiv · showhide
In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%). Finally, we train a 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
1 Introduction
The paper investigates shared visual representations for robotics and shows that masked visual pre-training on diverse Internet and egocentric data transfers effectively to real-world tasks. Frozen representations support control across tasks and embodiments, outperforming several baselines while improving demonstration efficiency and scaling with data and model size.
- Motivation: Shared visual representations are studied as a way to transfer representation-learning advances from computer vision to robotics.The paper focuses on shared representations on the perception side rather than shared action representations.
- Contribution: 7× larger data, 15× bigger models, and extensive real-world experiments extend the authors’ prior self-supervised visual pre-training framework.The paper evaluates whether the framework transfers from prior simulation results to real-world robotic tasks.
- Approach: The framework pre-trains visual representations with masked autoencoders, freezes the encoder, and learns task-specific control policies on top.Representations are learned from Internet and egocentric images and videos, then reused across downstream robotic tasks and embodiments.
- Results: 981 real-world experiments show higher performance than CLIP by up to 75%, supervised pre-training by up to 81%, and training from scratch by up to 81%.The evaluation covers reach, push, pick, and tasks with scene and object variations.
- Results: The representations reach the strongest baseline performance with half the number of demonstrations.The study uses 20–80 human-provided demonstrations per task.
- Scaling: A 307M-parameter vision encoder trained on 4.5M images demonstrates benefits from scaling visual pre-training for robot learning.The images come from ImageNet, Epic Kitchens, Something Something, 100 Days of Hands, and Ego4D.
2 Related Work
Prior robotics work uses end-to-end, supervised, language-supervised, and self-supervised representations, but this paper emphasizes a shared frozen vision encoder across tasks and robots.
- End-to-end control: End-to-end control predicts robot actions directly from observations without explicit pose estimation, grasp planning, or motion planning.These approaches are described as tending to be sample inefficient for real-world training.
- Supervised pre-training: Supervised robotics pre-training transfers representations learned from strongly supervised pretext tasks to downstream robotic tasks.Examples include semantic detection and segmentation representations and language-supervised CLIP policies.
- Self-supervised learning: Self-supervised robotics methods have learned dynamics, visual representations, policies, spatiotemporal features, correspondences, and retrieval mechanisms.The cited approaches use interaction data, self-collected trajectories, videos, or pre-collected visual data.
- Shared representations: The paper uses one frozen vision encoder while training separate control policies for all downstream robotic tasks and embodiments.This differs from approaches that share representations on the action side.
3 Real-World Robot Learning with Masked Visual Pre-training
The method combines large-scale real-world visual data, masked image modeling with ViTs, frozen representations, and behavior-cloned control for diverse robot tasks.
- Data collection: The visual pre-training dataset combines Ego4D, ImageNet, Epic Kitchens, 100 Days of Hands, and other hand-object interaction data.Ego4D contributes egocentric daily-life activity data, while the remaining sources broaden visual coverage.
- Masked visual pre-training: MAE masks random image patches and reconstructs the missing content with a vision transformer.A high masking ratio and asymmetric heavy-encoder/light-decoder design support efficient representation learning.
- Architecture: The study scales the vision encoder to a ViT-Large model containing 307M parameters for deployment on a real robot.The model runs at approximately 64 gigaflops for 224×224 inputs.
- Control learning: Behavior cloning learns control policies from demonstration trajectories containing wrist-camera RGB images and robot joint states.The controller is trained on top of the frozen visual encoder.
- Real-world tasks: Evaluations use a 7-DoF arm with a parallel-jaw gripper across basic motor skills and variations in scenes and objects.Tasks include reaching, pushing, picking, closing a fridge, picking fruit, and picking from a cluttered sink.
4 Experimental Setup
The experiments compare visual encoders and demonstration efficiency across controlled real-robot tasks, using a standardized setup and varied demonstration counts.
- Pre-training data: The combined Ego dataset contains 4.5M images: 2.6M from Ego4D, 1.2M from ImageNet, and 700k hand-object interaction images.The dataset is formed by extracting frames and combining them with ImageNet and YouTube hand-object data.
- Encoders: The compared ViT-Small, ViT-Base, and ViT-Large encoders contain 22M, 86M, and 307M parameters, respectively.ViT-Large has approximately 15× as many parameters as ResNet-50.
- Pre-training: Models are pre-trained with MAE on the combined Ego, hand-object interaction, and ImageNet datasets using dataset-specific training schedules.The combined Ego models use 400 epochs, while the other listed MAE studies use 1600 epochs.
- Controllers: The controller receives image features and joint positions and outputs delta joint angles through a four-layer MLP.Velocity and explicitly appended end-effector pose are not used in the proprioceptive state.
- Robot setup: The robot setup uses a 7-DoF xArm7, a 1-DoF parallel-jaw gripper, a wrist-mounted camera, and no depth information.Control and demonstration collection run at 5 Hz.
- Demonstrations: Each task uses 80 demonstrations, with motion-tracked VR for most tasks and kinesthetic teaching for closing the fridge.Trajectory replay is used for trajectory pruning.
- Evaluation protocol: Models are evaluated on the same 16 environment variations within each task to improve consistency across comparisons.Sequential evaluation helps maintain similar lighting, robot conditions, and object initial locations.
5 Experimental Results
Across 981 real-world experiments, the proposed visual representations outperform competing encoders across basic and visually diverse robotic tasks, while scaling benefits depend on jointly increasing model and data size.
- 981 real-world experiments evaluate visual encoders across robotic tasks, objects, and environments.
- The proposed encoder consistently outperforms CLIP, supervised ImageNet pre-training, and training from scratch across benchmark tasks.CLIP is the strongest baseline among the comparisons.
- Sample complexity: 50% fewer demonstrations are sufficient for the proposed approach to reach CLIP’s performance.
- Visually Diverse Scenes and Objects: In visually diverse settings, the approach outperforms baselines on closing a fridge, picking varied fruits, and picking from a cluttered sink.The fruit task achieves a nearly perfect score, while the cluttered-sink task uses the same ViT-B encoder for comparison.
- Scaling Model and Data Size: Scaling ViT-S to ViT-B without more data does not improve performance and can hurt, whereas jointly scaling data and model size improves results.
- Scaling Model and Data Size: The 307M-parameter ViT-L further improves over the 86M ViT-B, with larger gains on the visually challenging PickSink task.
- Ablation Studies: The wrist-mounted camera outperforms a third-person camera, while removing either images or proprioception produces zero success rate.The third-person setup struggles with fine-grained localization and scale ambiguity, and neither modality alone fully observes the task.
- Ablation Studies: Common policy-training augmentations provide no observed benefit, and smaller ResNet architectures trained from scratch do not improve results.Finetuning smaller pretrained models yields only marginal gains, while CLIP ViT-B outperforms CLIP ResNet-50.
6 Discussion
The paper reports broad real-world benefits from masked visual pre-training while identifying important limits in task realism and complexity. Its demonstrations use consumer VR devices, and its evaluation covers extensive real-world experiments.
- The study primarily uses toy objects in relatively clean lab environments rather than real objects in real-world scene contexts.
- The tasks involve a single object and do not require feedback control or multi-step planning.
- 981 real-world experiments evaluate basic motor control and tasks involving scene and object variations.
- The approach achieves higher success rates and better sample complexity than CLIP, supervised ImageNet pre-training, and training from scratch.
- Demonstrations are collected with HTC Vive and Meta Quest 2 devices for the xArm and Allegro hand.
Appendix A: Data Collection
The paper collects robot demonstrations through consumer-grade VR interfaces tailored to the xArm and Allegro hand. The setups map tracked human motion to robot control commands.
- xArm demonstrations use an HTC Vive VR controller to control the robot end-effector pose.
- The xArm setup estimates the controller’s 6-DoF pose and maps end-effector pose to joint angles through inverse kinematics.
- Allegro hand demonstrations use Meta Quest 2 hand tracking to obtain 3D human-hand keypoints.
- The demonstration system relies exclusively on consumer-grade VR software and hardware.
- Consumer-grade tracking is presented as accurate, updateable, and easy for researchers worldwide to acquire and set up.
Appendix B: Evaluation
The evaluation controls confounding factors that can affect comparisons between pre-training methods. It accounts for changes in robot operation and lighting conditions.
- Robot performance can vary non-trivially with robot up-time as extended operation reduces action precision.
- Lighting-condition differences may affect the performance of different methods.
- The evaluation aims to control confounding factors that might impact pre-training comparisons.