Source-linked AI summary
Masked Visual Pre-training for Motor Control
Tete Xiao, Ilija Radosavovic, Trevor Darrell, Jitendra Malik
TL;DR
Motor control from pixels remains difficult, motivating representations that transfer across tasks rather than being learned end-to-end from environment experience. The paper pre-trains a visual encoder with masked modeling on real-world images, freezes it, and trains controllers with reinforcement learning; this approach outperforms supervised representations by up to 80% absolute success rate and can match oracle state performance, while in-the-wild data benefits manipulation tasks.
Problem
Motor control remains a notable exception to progress from learned representations, while complex tasks involve high-dimensional pixel observations and control inputs.
Method
The paper pre-trains one visual encoder by masked modeling on real-world images, freezes it, and trains task-specific controllers with reinforcement learning across tasks.
Results
Up to 80% absolute success rate improves over supervised representations, with oracle state performance matched in some cases; in-the-wild images also outperform ImageNet images for manipulation tasks.
Takeaways & Limitations
A single self-supervised visual representation can support varied motor control tasks from pixels without per-task fine-tuning, explicit state estimation, or expert demonstrations.
Abstract
from arXiv · showhide
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We first train the visual representations by masked modeling of natural images. We then freeze the visual encoder and train neural network controllers on top with reinforcement learning. We do not perform any task-specific fine-tuning of the encoder; the same visual representations are used for all motor control tasks. To the best of our knowledge, this is the first self-supervised model to exploit real-world images at scale for motor control. To accelerate progress in learning from pixels, we contribute a benchmark suite of hand-designed tasks varying in movements, scenes, and robots. Without relying on labels, state-estimation, or expert demonstrations, we consistently outperform supervised encoders by up to 80% absolute success rate, sometimes even matching the oracle state performance. We also find that in-the-wild images, e.g., from YouTube or Egocentric videos, lead to better visual representations for various manipulation tasks than ImageNet images.
1. Introduction
Motor control from pixels remains challenging because high-dimensional actions and observations make end-to-end reinforcement learning costly and prone to overfitting. The paper addresses this with frozen visual representations learned by masked modeling on real-world images, reporting broad gains over supervised encoders and strong generalization across tasks.
- Motor control remains a notable exception to the broader progress from large-scale learned representations.
- High-dimensional pixel observations and control inputs make complex tasks difficult to learn, especially with robots having many degrees of freedom.The task suite varies movement types, robots, scenes, and objects, including a 23 DoF robot with a multi-finger hand.
- End-to-end reinforcement learning is computationally expensive, has poor sample complexity, and typically overfits to the training setting.The paper identifies these as practical challenges for learning policies directly from pixels.
- MVP pre-trains a visual encoder with masked image modeling on natural images, freezes it, and trains task-specific controllers with reinforcement learning.The same visual representations are used across motor control tasks without task-specific encoder fine-tuning.
- The paper contributes PixMC, a benchmark suite covering different movement types, scene configurations, and robots.It includes reward functions, baselines, and a multi-GPU implementation for learning from pixels.
- A single real-world-image encoder solves varied motor control tasks without per-task fine-tuning, state estimation, or demonstrations.
- Up to 80% absolute success rate separates the self-supervised approach from supervised representations, with oracle performance matched in some cases.
- In-the-wild images from YouTube or Egocentric videos produce better representations for manipulation tasks than ImageNet images.
2. Masked Visual Pre-training for MC
MVP pre-trains visual representations on unlabeled real-world images with masked modeling, freezes the encoder, and trains task-specific controllers with reinforcement learning. The framework combines visual and proprioceptive embeddings and uses PPO policies for pixel-based motor control.
- Masked Visual Pre-training: MVP pre-trains visual representations on roughly 700K unlabeled images from egocentric, YouTube, and crowd-sourced video datasets.The HOI collection excludes human labels and temporal information, while ImageNet supports controlled comparisons.
- Masked Visual Pre-training: Masked autoencoders reconstruct randomly masked image patches, using a high masking ratio and a heavy encoder with a light decoder.Only unmasked patches enter the MAE during pre-training, improving training efficiency.
- Learning Motor Control from Pixels: The visual encoder remains frozen across downstream tasks, preventing task-specific fine-tuning and preserving one shared representation.The paper also identifies memory and runtime savings from this design.
- Learning Motor Control from Pixels: Controllers combine fixed image features with proprioceptive information before predicting actions for motor control.Proprioception consists of robot joint positions and velocities, concatenated with image features into the controller embedding.
- Learning Motor Control from Pixels: Task-specific policies use model-free reinforcement learning with PPO and small MLP actor and critic networks.The policy and critic use the same representations but do not share weights.
- Benchmark context: The benchmark design combines hand-designed tasks, dense rewards, complex robots, fast simulation, and distributed training for pixel-based motor control.These features are presented as a distinctive combination relative to existing benchmarks.
3. PixMC Benchmark
PixMC is a benchmark suite for pixel-based motor control spanning varied movements, scenes, objects, and robots. It provides high-resolution observations, task rewards and metrics, fast GPU simulation, and distributed training, while MVP outperforms supervised baselines across most tasks.
- Motivation: PixMC addresses the lack of a benchmark combining high-resolution images, realistic robots, fast physics simulation, efficient training, and suitable rewards and metrics.The suite was introduced because existing motor-control benchmarks fall short on one or more of these requirements.
- Simulation and training: GPU-based IsaacGym enables approximately 12-minute oracle-state training and approximately 5-hour image-model training on one NVIDIA 2080 Ti GPU.The image-model figure reports roughly 8 million environment steps for the latter setup.
- Robots: PixMC covers Franka and Kuka robot configurations, including a 7-DoF Franka with a 2-DoF gripper and a 23-DoF Kuka with an Allegro hand.The Kuka configuration combines a 7-DoF arm with a 16-DoF, four-finger hand.
- Observations and control: The benchmark renders high-resolution wrist-camera observations and provides proprioceptive and hand-engineered state information for both robot setups.Default control uses joint-angle position control at 60Hz.
- Tasks, rewards, and metrics: The suite includes movement types from reaching to object interaction, with variation in object position, scale, color, and shape.Tasks use hand-designed dense rewards and reward-independent success metrics based on distances to goals over time.
- Results: MVP significantly outperforms the supervised baseline on 7 of 8 PixMC tasks and closely matches the oracle state model on 5 tasks at convergence.The sample-complexity comparison plots success rate against environment steps for tasks using Franka or Kuka robots.
4. Experimental Setup
The experiments compare pre-training data and encoder variants, then use fixed ViT representations with lightweight controllers trained by reinforcement learning. The setup also evaluates robustness to distractors.
- Pre-training data: HOI pre-training combines Epic-Kitchens, Something-Something, and 100 Days of Hands, yielding 700k sampled images.
- Encoder: The visual encoder is a ViT-Small with 16×16 patches, 384 hidden dimensions, 6 attention heads, and 12 Transformer blocks.
- Controller: Controllers receive projected image features and proprioceptive state, then predict delta joint angles through a four-layer MLP.
- Training with RL: The visual encoder remains frozen throughout reinforcement-learning training, with task-specific iteration budgets for reach, cabinet, pick, and relocate.
- Robustness evaluation: Distractor tests vary color, shape, or size relative to a blue 4.5cm training box.
5. Experimental Results
MVP improves performance and robustness across PixMC tasks, generalizes to varied objects, and benefits from in-the-wild manipulation imagery. Additional comparisons examine stability, scaling, environment-trained representations, and encoder fine-tuning.
- PixMC benchmark: MVP significantly outperforms supervised representations on 7 of 8 PixMC tasks and matches the oracle state model on 5 tasks at convergence.
- Robustness: Color and shape distractors only marginally reduce success, whereas scale variation yields 50% success, consistent with scale ambiguity from a single first-person camera.
- Generalization: The Kuka robot with the Allegro hand achieves at least 50% success on box, can, mug, and banana objects after retraining per category.
- Pre-training data: HOI-trained MVP outperforms ImageNet-trained MVP on 7 of 8 tasks.
- Random features: Random frozen features produce flat-zero or near-zero success on 6 of 8 tasks, showing that random representations are insufficient for complex motor control.
- Encoder fine-tuning: Unfreezing the encoder substantially increases GPU compute and memory consumption and produces unstable training in the tested setting.
- Larger encoders: A ViT-Base encoder does not improve performance in preliminary pick-task scaling experiments.
- Comparisons: For a fixed number of steps, MVP outperforms reimplemented CURL and RAD while using 1 rather than 8 GPUs for training.
6. Related Work
Related work develops representations through end-to-end task learning, auxiliary self-supervision, robotics interaction, supervised vision tasks, and computer-vision pretext objectives. These approaches motivate learning transferable visual representations for motor control.
- Dexterous manipulation: Dexterous manipulation studies often rely on domain randomization, explicit state estimation, expert policies, demonstrations, or human priors.
- Representations in RL: End-to-end reinforcement learning learns representations from task signals but has high sample complexity and may overfit to particular settings.
- RL with self-supervision: Auxiliary objectives can improve reinforcement learning representations, although environment-specific experience remains their training source.
- Self-supervision in robotics: Robotics work has learned representations from interaction, multiview video, dense descriptors, and environment-data methods such as RAD and CURL.
- Supervised pre-training: Supervised mid-level vision representations have been transferred to navigation, manipulation, driving, and affordance prediction.
- Self-supervised pre-training: Computer-vision self-supervision commonly uses pretext tasks and contrastive learning for representation pre-training.
7. Conclusion
The paper concludes that self-supervised visual pre-training can support motor control from pixels using one frozen encoder across tasks. It reports gains over supervised baselines and stronger performance from in-the-wild imagery than ImageNet imagery.
- A single self-supervised vision encoder supports varied motor-control tasks without per-task fine-tuning, explicit state estimation, or expert demonstrations.
- MVP improves over supervised baselines by up to 80% absolute success rate and sometimes matches oracle state performance.
- In-the-wild images from YouTube or egocentric videos can produce better visual representations than ImageNet images.