Source-linked AI summary
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
Frederik Ebert, Sudeep Dasari, Alex X. Lee, Sergey Levine, Chelsea Finn
TL;DR
The paper asks how robots can plan complex tasks from raw visual observations when predictions are inaccurate. It combines video-prediction control with self-supervised image registration to track goals and retry; experiments report improved long-distance manipulation performance over prior visual MPC and a supervised tracker.
Problem
Predictive models for control from raw images raise challenges in using predictions when they are inaccurate and in evaluating predicted futures with a grounded mechanism.
Source: §1 · p. 1Method
Results
On long-distance relocation, learned registration outperforms prior visual MPC and a supervised tracker, while methods perform comparably on short-distance pushing.
Source: §6 · p. 8Takeaways & Limitations
The approach supports self-supervised prehensile and non-prehensile relocation, with the robot able to plan grasping or pushing as needed.
Source: §7 · p. 8Whether sharing parameters between the registration and video-prediction models would be beneficial is left for future work.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Prediction is an appealing objective for self-supervised learning of behavioral skills, particularly for autonomous robots. However, effectively utilizing predictive models for control, especially with raw image inputs, poses a number of major challenges. How should the predictions be used? What happens when they are inaccurate? In this paper, we tackle these questions by proposing a method for learning robotic skills from raw image observations, using only autonomously collected experience. We show that even an imperfect model can complete complex tasks if it can continuously retry, but this requires the model to not lose track of the objective (e.g., the object of interest). To enable a robot to continuously retry a task, we devise a self-supervised algorithm for learning image registration, which can keep track of objects of interest for the duration of the trial. We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation. Our real-world experiments demonstrate that a model trained with 160 robot hours of autonomously collected, unlabeled data is able to successfully perform complex manipulation tasks with a wide range of objects not seen during training.
1 Introduction
The paper asks how robots can plan complex, temporally extended tasks from raw visual observations despite inaccurate predictions. It proposes visual MPC with learned image registration so the robot can track its goal and retry during execution.
1 Introduction
- The central challenge is planning from raw sensory observations when predictive models are imperfect and predicted futures need grounded evaluation.Source: §1 · p. 1
- Registration-based costs let visual MPC evaluate progress toward the goal and retry until completion, even when video predictions are flawed.The registration model is learned without supervision from the same data used to train the predictive model.Source: §1 · p. 2
- The robot plans actions from a video prediction model toward a desired image configuration, with goals specified by an image and user-annotated object or point locations.Goal commands are not provided during training, and the model has no prior knowledge of objects or goals.Source: §1 · p. 2
- The method uses learned image registration to match current observations with initial and goal images, enabling closed-loop attempts until the task is complete.The paper combines this cost with video prediction for longer-horizon manipulation than prior short-horizon pushing work [4] [5].Source: §1 · p. 2
2 Related Work
The paper contrasts video-prediction planning with approaches tied to hand-designed representations, single predicted events, or reactive visual servoing. It emphasizes reusable goal-conditioned planning and grounded costs that support persistent replanning.
2 Related Work
- Video prediction can support multiple goals and behaviors from raw RGB video, unlike methods that predict a single event such as grasp success [1] [2] [3] or crashing [8] [9].The paper reports complex manipulation of previously unseen objects and longer tasks than prior visual planning work [4] [5].Source: §2 · p. 2
- Learned registration measures pixel movement toward a goal image rather than distance in an unconstrained feature space [12] [13].The paper argues registration provides a calibrated goal-distance measure because it captures how pixels should move.Source: §2 · p. 2
- Unlike visual servoing approaches, the method uses multi-step predictive models for temporally extended goals while retaining continuous visual feedback.It also performs non-prehensile manipulation, whereas visual servoing typically assumes fully actuated control and often a known Jacobian.Source: §2 · p. 3
- MPC supports persistent replanning, but visual foresight prior work [4] [5] remained short-term because it lacked an accurate planning cost throughout an episode.The paper presents grounded evaluation of visual predictions as the missing mechanism for persistent replanning.Source: §2 · p. 3
- Autonomously collected data can train the registration model, which can continue improving as the robot gathers more data, unlike off-the-shelf trackers that may lose track during occlusion.Source: §2 · p. 3
3 Preliminaries
Visual MPC plans actions to move selected pixels from an initial image toward corresponding goal-image positions. Its cost uses predicted pixel locations, while replanning with the current observation addresses tracking errors that accumulate when earlier predictions are reused.
3 Preliminaries
- An action-conditioned video model predicts future camera observations from the current frame and candidate actions, and CEM optimizes sampled action sequences for lower cost.The prediction model transforms past images to generate future images.Source: §3 · p. 3
- Visual MPC represents a task as moving selected source pixels to goal-image locations, with an object moving when its selected pixel lies on it.This can require long-horizon reasoning and prehensile or non-prehensile manipulation.Source: §3 · p. 3
- The cost measures the distance between predicted pixel positions and goal positions, with per-pixel costs weighted by λ_i.Distance-based costs are described as well-shaped and efficiently optimizable.Source: §3 · p. 3
- Reusing the previous step’s predicted pixel location to initialize the next step can accumulate errors and make the model lose track of the designated object.The method instead replans at each real-world step and executes the first action of the best sequence.Source: §3 · pp. 3–4
4 Retrying by Registration
The section makes visual MPC retryable by learning registration between current, start, and goal images, then using tracked points and registration confidence to plan. Registration is trained from the same trajectory data as video prediction, without supervision.
4 Retrying by Registration
- The learned registration cost supports persistent retries by tracking the objective across current, start, and goal images; pixel error and whole-image warp costs have important drawbacks.Pixel-wise error can lack a useful action gradient for distant objects and become meaningless with blurry predictions, while whole-image metrics can prioritize the robot arm over smaller objects.Source: §4 · p. 4
4.1 Test time procedure
- At test time, the network registers the current image to both start and goal images, but either registration can fail when objects are far apart.It tracks user-designated pixels rather than evaluating the whole image, and uses a neighborhood median of flow vectors for more stable tracking.Source: §4.1 · pp. 4–5, Figure 2 · p. 4
4.2 Planning Costs
- The planning cost weights registered-point distances by estimated registration quality, using photometric error as a local success signal.Low photometric error can still point to the wrong object, such as another object with a similar color.Source: §4.2 · pp. 5–6
4.3 Training procedure
- The registration network learns image correspondences from randomly sampled trajectory frames, trained separately from the video predictor on the same data.Training uses bidirectional photometric reconstruction and flow smoothness; the temporal gap grows from 1 to 8 steps by 20k SGD steps, across 60k iterations.Source: §4.3 · p. 6
5 Scaling up Visual Model-Predictive Control
The section extends visual MPC to multiple camera views and combines pushing with grasping for more complex manipulation. A simple data-collection reflex increases the frequency of grasping experience without grasp-specific policy engineering.
Extension to multiple cameras
- Multiple appropriately oriented camera views combine 2D image-space tasks into 3D manipulation tasks, including lifting tasks ambiguous from one view.Registration is applied separately per view, and view costs are combined by weighted averaging.Source: §5 · p. 6
Combined prehensile and non-prehensile manipulation
- The video-prediction and registration networks do not share parameters; exploring parameter sharing is left for future work.Source: §5 · p. 6
- A wrist-height reflex closes the gripper during data collection, and about 20% of training trajectories include some grasping, enabling a jointly learned pushing-and-grasping policy.The illustrated behavior pushes, loses the object, then grasps and pulls it to the goal.Source: §5 · pp. 6–7, Figure 5 · p. 7
6 Experiments
Experiments evaluate self-supervised visual MPC on novel-object pushing and combined grasping-and-pushing tasks, using autonomously collected data. Learned registration outperforms comparison methods on long pushes, while short-push methods perform comparably; combined manipulation is comparable to supervised tracking.
6 Experiments
- The evaluation uses two Sawyer arms and autonomously collected pushing and gripper-control trajectories, with novel objects reserved for testing.The pushing benchmark disables gripper control; comparisons include OpenCV MIL tracking, which uses additional human supervision, and predictor propagation.Source: §6 · pp. 7–8
- On long-distance pushing, with average object-to-goal distance 30 cm, learned registration outperforms OpenCV tracking and predictor propagation across 20 objects not seen during training.Figure 6 reports the fraction of runs with final distance below threshold.Source: Figure 6 · p. 7, §6 · p. 8
- On short pushes averaging 15 cm, all evaluated methods perform comparably.Figure 7 reports the fraction of runs below final-distance thresholds.Source: Figure 7 · p. 7, §6 · p. 8
- With grasping enabled, registration-based visual MPC is comparable to the supervised OpenCV tracker on long-distance relocation tasks.In most cases, the robot chooses to grasp the object.Source: §6 · p. 8
7 Discussion
The discussion reports improved performance on temporally extended manipulation from closing the control loop with learned image registration. A simple grasping reflex supports self-supervised learning of prehensile and non-prehensile relocation, while multi-object relocation remains a future challenge.
7 Discussion
- Closing the control loop with learned image registration substantially improves performance on temporally extended object manipulation tasks.A simple grasping reflex also supports self-supervised learning of prehensile and non-prehensile relocation; multi-object relocation is identified as future work.Source: §7 · p. 8, Figure 9 · p. 8
Simulated Experiments
A MuJoCo simulation evaluates pushing randomly generated L-shaped objects to target locations. Around a 0.1 distance threshold, registration-based visual MPC substantially outperforms predictor propagation and has a modest gap to simulator ground-truth tracking.
Simulated Experiments
- The simulation uses a MuJoCo arm with Cartesian position control to push randomly generated, randomly colored L-shaped objects.The evaluation comprises 50 tasks with maximum episode lengths of 120 steps; the illustrated arm is in Figure 10.Source: Simulated Experiments · p. 12, Figure 10 · p. 12
- Around a 0.1 distance threshold, registration-based control substantially outperforms predictor propagation and has a modest gap to ground-truth tracking.Figure 11 plots the fraction of evaluation scenarios within each distance threshold; ground-truth registration is an oracle upper bound.Source: Simulated Experiments · p. 12
Planner Implementation Details
The planner uses the cross-entropy method to optimize a short sequence of actions against the cost function.
Planner Implementation Details
- The planner runs three cross-entropy iterations, optimizing 20 variables for five independent actions over a 15-step horizon.It samples 400 candidates initially and 200 thereafter, fitting a Gaussian to the best 5%; each action is repeated three times.Source: Planner Implementation Details · p. 12
Improvements
The online optimization procedure addresses stalled progress from inconsistent replanning by biasing CEM toward its previous plan while retaining exploration at the horizon’s final step.
to online optimization procedure
- Figure 11 measures simulated success as the fraction of trajectories ending below a final-object-distance threshold; higher is better.Source: Figure 11 · p. 12
- Replanning can alternate between pushing and grasping indefinitely, leaving the robot stuck without progress toward the goal.Source: to online optimization procedure · p. 12
- The first CEM iteration uses the previous plan as its mean, narrows covariance for the first T−1 steps, and preserves final-step exploration.It omits the previous plan’s first action because that sequence was optimized from the preceding time step.Source: to online optimization procedure · p. 12