Source-linked AI summary
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, Sergey Levine
TL;DR
Deep reinforcement learning still struggles to generalize robotic skills across tasks and objects, while real-world settings often lack external rewards. This paper proposes self-supervised visual MPC, which predicts future images for planning and demonstrates broad manipulation generalization with one model, subject to visibility and planning-horizon limits.
Problem
Deep reinforcement learning has not yet broadly generalized to new robotic tasks and objects, and real-world settings typically lack ground-truth reward signals.
Method
Visual MPC learns action-conditioned predictive models from autonomously collected raw-image data and uses them for sampling-based, model-predictive planning toward pixel, image, or classifier goals.
Results
Visual MPC generalizes to a wide range of unseen tasks and performs multi-object manipulation, pushing, picking and placing, and cloth folding within one framework.
Takeaways & Limitations
A single visual model can support user-defined manipulation of previously unseen rigid and deformable objects across diverse task settings.
Takeaways & Limitations
The framework requires target objects to remain visible, handles only medium-duration tasks involving one or two objects, and has lower positioning fidelity than humans.
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.
1 INTRODUCTION
The paper addresses limited generalization in deep reinforcement learning by learning predictive models from raw sensory observations and using them to control real-world manipulation. Visual MPC combines diverse autonomous data, pixel-based goal specification, and planning to handle new tasks and objects.
- Generalizing learned robotic behaviors to new tasks and objects remains an open problem, especially beyond narrow training environments.
- Predicting raw sensory observations can avoid requiring low-dimensional state information or an extrinsic reward signal.
- Visual MPC is a framework that uses sensory prediction models for deep reinforcement learning in diverse, open-world environments.
- The framework specifies and evaluates goals through designated pixel positions, goal-image registration, and success classifiers.
- Experiments combine these components to perform manipulation involving unseen objects, clutter, obstructions, deformable objects, and perturbation recovery.
2 RELATED WORK
Related work situates visual MPC within model-based reinforcement learning and action-conditioned sensory prediction. The paper distinguishes its approach by targeting diverse real-world manipulation without requiring reward or success signals during data collection.
- Model-based reinforcement learning predicts the future with a learned model and uses that model to select actions.
- Prior image-based model-based reinforcement learning has focused mainly on simple synthetic images and video-game environments.
- Prior sensory-prediction work includes action-conditioned prediction of game images, point clouds, and real-world images.
- Visual MPC learns predictive models for varied manipulation skills without success measures, event indicators, or reward functions during data collection.
3 OVERVIEW
Visual MPC has three phases: autonomous data collection, predictive-model training, and planning-based control at test time. Planning evaluates candidate futures with costs matched to how the user specifies the goal.
- The method consists of unsupervised data collection, predictive-model training, and planning-based control via the model at test time.
- Training data is collected autonomously using random actions sampled from a distribution designed to visit task-relevant parts of the state space.
- Designated-pixel goals use pixel-distance costs, while goal images use image-to-image registration to evaluate predicted progress.
- The computation graph represents actions, recurrent hidden states, warping fields, real images, predicted images, and a pairwise training loss.
- At test time, the model plans multiple future steps, executes the lowest-cost sequence’s first action, and replans after each real-world timestep.
4 VIDEO PREDICTION FOR CONTROL
The transformation-based SNA video predictor forecasts future frames by warping image content and composing it with learned masks. Temporal skip connections support recovery of pixels after occlusion and enable pixel-motion prediction without external supervision.
- Transformation-based prediction produces a 2D flow field that can forecast pixel motion without external pixel-motion supervision.
- The DNA model cannot recover object shapes after occlusion, limiting planning when selected pixels become hidden.
- Each predicted frame is formed by compositing transformed images with predicted masks, reusing appearance from prior frames.
- The SNA model uses temporal skip connections so occluded pixels can reappear later in the predicted sequence.
- The simplified implementation transforms only the previous image and the first image, and can use the first image directly for background generation.
5 PLANNING COST FUNCTIONS
The paper develops three planning-cost mechanisms for visual MPC: designated-pixel distances, goal-image registration, and few-shot goal classifiers. These mechanisms trade off precise relocation, robust correspondence, and abstract task specification while addressing distractors and clutter.
- Pixel Distance Cost: Naive pixel-wise image error can be dominated by large objects such as the robot arm and shadows, causing planners to neglect smaller target objects.This failure motivates more selective goal-evaluation mechanisms.
- Pixel Distance Cost: Pixel-distance costs measure the expected Euclidean distance between predicted designated pixels and user-specified goal positions.They provide smooth objectives for longer-horizon planning and incorporate predictor uncertainty.
- Registration-Based Cost: Goal-image registration warps the current image to start and goal views, estimating designated-pixel locations while ignoring distractor pixels.Registration to both views improves accuracy and robustness, although either view is sufficient in principle.
- Registration-Based Cost: Registration reliability is estimated from local photometric error and used to weight the corresponding planning costs.Low photometric error indicates similar warped color but can still select the wrong object when multiple objects share a color.
- Classifier-Based Cost Functions: Few-shot goal classifiers infer new task goals from positive success examples and support abstract objectives through learned invariances.They can ignore irrelevant factors such as arm position and represent relations such as placing a cup in front of a plate.
- When to Use Which Cost Function?: Pixel costs are easy to specify and robust to clutter, whereas goal images improve closed-loop precision and classifiers support broader goal sets at greater user effort.The three interfaces therefore offer different balances between precision, robustness, abstraction, and specification burden.
6 TRAJECTORY OPTIMIZER
Visual MPC plans by optimizing action sequences against predicted future-image costs, using CEM to iteratively refine promising sequences while keeping actions within the model's training distribution.
- Trajectory optimization: CEM searches over action sequences by repeatedly resampling and refitting Gaussian distributions around sequences with the lowest predicted costs.The optimizer is gradient-free and operates over the planning horizon T.
- Trajectory optimization: Sampled actions are checked against admissible ranges and resampled when invalid, preventing the model from receiving out-of-distribution actions.This constraint is intended to keep planning within action regimes encountered during training.
- Trajectory optimization: At each planning iteration, the predictive model generates future image sequences, which are evaluated by a planning cost function.The procedure samples candidate sequences, predicts their consequences, scores them, and updates the sampling distribution.
- Trajectory optimization: The controller applies only the first action of the best sequence before replanning.This receding-horizon procedure repeats across control steps.
7 CUSTOM ACTION SAMPLING DISTRIBUTIONS
Simple random action distributions mainly produce pushing and dragging, so the data-collection policy adds a gripper-closing reflex to expose grasping and folding states without task-specific policy engineering.
- Motivation: Random sampling alone rarely visits grasping, placing, or cloth-folding states, restricting emergent skills largely to pushing and dragging.The resulting model would also be imprecise in these underrepresented states, and planning would rarely discover the corresponding actions.
- Action sampling: A reflex automatically closes the gripper when the wrist drops below a small height threshold, increasing exposure to grasping and folding behaviors.With rigid objects, about 20% of trajectories included some form of grasp.
- Results: The joint pushing, grasping, and folding policy emerges through planning without grasping-specific or folding-specific engineering beyond the reflex.Experiments compare data collected with and without the grasping reflex.
8 MULTI-VIEW VISUAL MPC
Multi-view visual MPC uses two cameras to overcome the ambiguity of single-view task specifications and define manipulation goals in 3D.
- Motivation: A single camera view cannot fully specify tasks such as lifting an object to a particular 3D position because depth is ambiguous.The original formulation supports tasks specified in 2D, such as rearranging objects on a table.
- Multi-view formulation: Two cameras arranged from different viewing angles jointly define a 3D manipulation task.The robot setup uses two standard webcams observing the workspace.
- Multi-view formulation: Planning costs from the two views are combined by weighted averaging, with weights supplied by the registration network.Registration is performed separately for each view to support dynamic retrying and temporally extended tasks.
9 EXPERIMENTAL EVALUATION
Visual MPC is evaluated across occlusion handling, tracking, classifier-based goals, and diverse multi-task manipulation settings. Across these experiments, it generalizes to novel objects and tasks while outperforming comparison methods in several settings.
- Prediction Models: Temporal skip-connections help SNA handle occlusions in pushing and obstacle-avoidance tasks with seen and novel objects.SNA usually found valid trajectories, whereas DNA was mostly unable to find solutions.
- Belief Updates: Visual MPC recovers from unexpected object motion and still reaches the goal after an initial execution failure.In one example, the object moved downward instead of rightward before the system recovered.
- Registration-Based Cost Functions: Registration-based tracking improves long-distance pushing, while short-distance performance is comparable across methods.The evaluation used 20 long-distance and 15 short-distance tasks.
- Classifier-Based Cost Functions: Classifier-based costs outperform pixel-distance and latent-space metrics across 20 relative-arrangement tasks, including distractors and sequential goals.Evaluation used novel held-out objects, and success required more than half of an object to be correctly positioned relative to another.
- Multi-Task Performance: A single visual MPC model solves diverse rigid and deformable manipulation tasks, including placement, obstacle avoidance, object relocation, towel wrapping, and shorts folding.Quantitative comparisons show substantial improvement over a hand-crafted baseline, which failed on almost all object-relocation tasks.
10 CONCLUSION
The paper presents visual model-predictive control as a self-supervised image-dynamics approach for diverse robotic manipulation. It demonstrates multiple manipulation skills within one framework while identifying important scope limitations.
- Visual MPC leverages self-supervised visual prediction to learn a deep dynamics model on images and embed it into planning.
- The framework successfully performs multi-object manipulation, pushing, picking and placing, and cloth-folding tasks within a single framework.
- Limitations: All target objects must remain visible throughout execution, limiting the framework in partially observed tasks involving occlusion.The paper gives placing an object in a box and closing it as an example.
- Limitations: The demonstrated tasks are medium duration and usually involve only one or two objects, leaving longer-term planning unresolved.
- Limitations: Object-positioning fidelity remains significantly below human performance.
APPENDIX A SKIP CONNECTION NEURAL ADVECTION MODEL
The appendix describes a transformation-based video-prediction model that conditions predictions on robot actions and uses skip connections to preserve object information through occlusions. Its predictions support designated-pixel tracking during planning.
- The SNA video-prediction model uses a convolutional LSTM to predict 13 future frames from an initial sequence of 2 ground-truth frames.
- Robot actions are spatially replicated and concatenated to the inputs of all convolutional layers for action conditioning.
- SNA incorporates instance normalization, ReLU activations, normalized LSTM pre-activations, and modified spatial downsampling and upsampling.
- In the occlusion example, SNA predicts that the occluded object remains stationary, unlike DNA, which incorrectly moves the object with the occluding arm.
- Designated-pixel predictions are visualized through predicted frames and probability distributions obtained by repeatedly applying transformations.
APPENDIX B IMPROVED ACTION SAMPLING DISTRIBUTIONS FOR DATA COLLECTION
The appendix addresses action-distribution design for autonomous data collection and replanning. The method supports recovery by tracking objects and can combine pushing with grasping when an initial manipulation fails.
- Naive replanning can alternate indefinitely between pushing and grasping actions, causing the agent to get stuck.
- CEM mitigates this behavior by using the previous timestep’s best action sequence to initialize the current sampling distribution.
- During pushing, tracking allows the robot to retry after the object initially moves unexpectedly downward and eventually reach the goal.
- After losing an object during pushing, the method can switch to grasping and pull it to the goal.
- Obstacle tracking can preserve a designated-pixel distribution through occlusion while planning a trajectory that avoids the obstacle.
- Training used 20,000 pushing trajectories and 15,000 gripper-control trajectories collected autonomously, with human intervention only to change objects.
APPENDIX E EXPERIMENTAL EVALUATION
The experimental evaluation includes an example in which SNA predicts an obstacle’s position through occlusion and finds an avoiding trajectory, alongside a controlled MuJoCo simulation benchmark.
- SNA successfully predicts the obstacle position through occlusion and finds a trajectory that avoids the obstacle.
- The controlled simulation uses a MuJoCo robotic manipulator to push randomly generated, randomly colored L-shaped objects toward target locations.