Source-linked AI summary
Self-Supervised Correspondence in Visuomotor Policy Learning
Peter Florence, Lucas Manuelli, Russ Tedrake
TL;DR
Visuomotor policy learning needs visual representations that generalize across tasks and objects while using limited human supervision. This paper trains dense visual correspondence representations and pairs them with imitation learning, finding strong simulation and hardware performance across challenging manipulation settings. The authors also identify scope boundaries where spatial correspondence alone is insufficient.
Problem
Visuomotor policy learning needs scalable visual training with limited human supervision, while prior approaches have mainly used end-to-end training, autoencoding, and pose-based losses.
Method
The paper trains visuomotor policies using self-supervised dense visual correspondence paired with imitation learning.
Results
Simulation comparisons suggest correspondence training improves generalization and sample complexity over autoencoding and end-to-end training, while hardware tests cover deformable, class-general, and visually challenging manipulation.
Takeaways & Limitations
Dense correspondence supports a broad range of spatial manipulation tasks, including policies that approach ground-truth-information performance in simulation and achieve high reliability on several hardware tasks.
Takeaways & Limitations
The representation does not explicitly handle simultaneously viewing multiple instances of the same object class, motivating future combination with instance-level segmentation.
Abstract
from arXiv · showhide
In this paper we explore using self-supervised correspondence for improving the generalization performance and sample efficiency of visuomotor policy learning. Prior work has primarily used approaches such as autoencoding, pose-based losses, and end-to-end policy optimization in order to train the visual portion of visuomotor policies. We instead propose an approach using self-supervised dense visual correspondence training, and show this enables visuomotor policy learning with surprisingly high generalization performance with modest amounts of data: using imitation learning, we demonstrate extensive hardware validation on challenging manipulation tasks with as few as 50 demonstrations. Our learned policies can generalize across classes of objects, react to deformable object configurations, and manipulate textureless symmetrical objects in a variety of backgrounds, all with closed-loop, real-time vision-based policies. Simulated imitation learning experiments suggest that correspondence training offers sample complexity and generalization benefits compared to autoencoding and end-to-end training.
I. INTRODUCTION
The paper proposes self-supervised dense correspondence as a visual training signal for scalable visuomotor learning with limited human supervision. It evaluates this formulation through simulation and hardware experiments targeting broad generalization, challenging manipulation, and modest data requirements.
- Motivation: Prior visuomotor visual training commonly uses end-to-end learning, autoencoding, or pose-based losses, leaving dense three-dimensional correspondence underused as robot-automatable self-supervision.Correspondence learning can provide visual supervision without additional human input.
- Contributions: The primary contribution is a self-supervised correspondence formulation paired with imitation learning for visuomotor policy learning.The paper also introduces behavior-cloning noise augmentation and multi-camera time-synchronized correspondence learning.
- Hardware validation: Hardware validation covers non-prehensile, class-general, and deformable manipulation, including visual challenges such as textureless objects, clutter, occlusion, and lighting variation.The reported hardware demonstrations use between 50 and 150 demonstrations.
- Scope: The method’s scope is primarily spatial manipulation: spatial correspondence alone cannot determine non-spatial completion conditions such as when cooking should finish.The paper presents correspondence as useful for many spatial tasks rather than sufficient for every task.
- Evaluation: Simulation comparisons measure generalization and sample-complexity benefits against end-to-end training and autoencoding.The experiments are designed to compare correspondence learning with benchmarked visual training methods.
B. Methods for Learning Vision-Based Closed-Loop Policies
Vision-based closed-loop policies must support diverse tasks, limited incremental human effort, and real-time operation, but prior learning signals face reward, reliability, sample-complexity, and generalization challenges. The paper frames correspondence-based visual training within a factored visuomotor architecture that compresses high-dimensional images into policy inputs.
- Design goals: The target policy-learning setting combines broad task coverage, potentially deformable or unknown objects, roughly one human-hour of incremental effort per task, and real-time closed-loop vision.These criteria motivate scalable alternatives to methods requiring substantial task-specific engineering.
- Prior policy-learning signals: Reinforcement learning is limited by the difficulty of measuring real-world rewards, while sim-to-real methods can require significant engineering for each new simulation scenario.The paper also considers imitation learning and observation-based approaches as alternative sources of policy-learning signal.
- Open challenges: Prior vision-based closed-loop methods have unclear reliability and sample-complexity requirements, and relatively little work characterizes their training and expected generalization distributions.The paper argues for greater simulation-based comparison alongside hardware validation.
- Visuomotor factorization: A common visuomotor architecture factors image processing from policy learning: a visual model maps high-dimensional images to a compact representation that the policy combines with other observations.This factorization is practical because the image input can have hundreds of thousands of dimensions while robot measurements are much smaller.
- Proposed model: The proposed model replaces conventional visual training within this factorized architecture with visual models trained on correspondence.Figure 2 contrasts the common factorization with the correspondence-trained proposal.
B. Visual Correspondence Models for Visuomotor Policy Learning
The paper uses self-supervised dense correspondence to produce spatial visual representations for visuomotor policies, supporting localization in image or 3D space. It compares fixed, optimized, and end-to-end uses of correspondence features while retaining a generic policy-learning loss.
- Visual representation: Self-supervised correspondence directly trains spatially informative pixel descriptors for RGB-based visuomotor policy learning.Unlike end-to-end, pose-based, and autoencoding signals, the representation is explicitly trained to find correspondences between RGB images.
- Model variants: All four correspondence-based methods optimize a generic policy loss and differ in their learnable parameters and how the visual representation z is acquired.The first three methods use the correspondence function; the fourth applies a differentiable operation to a correspondence-pretrained model.
- Model variants: Fixed Descriptor Set optimizes only policy parameters, leaving both the descriptor set and dense visual model fixed after initialization.Descriptors are sampled from a masked reference descriptor image; the method is effective but does not further optimize the visual representation.
- Model variants: Descriptor Set Optimization jointly learns which descriptors to use with policy parameters while keeping the dense correspondence mapping pretrained.It can improve validation error over a fixed descriptor set with minimal additional computational cost and parameters.
- Model variants: End-to-end dense optimization includes the visual parameters but has not shown a performance advantage over Descriptor Set Optimization.Using a fixed or optimized descriptor set also increases policy-training speed by avoiding forward-backward optimization through the deep vision network, reported as 1 to 2 orders of magnitude faster.
IV. VISUAL IMITATION FORMULATION
The paper applies its general correspondence-based visuomotor policy framework to a specific imitation-learning formulation for robot manipulation.
- The visual correspondence framework is instantiated for a specific type of imitation learning in robot manipulation.
A. Robot Observation and Action Spaces
The robot receives image and proprioceptive observations and produces relative end-effector pose and gripper commands tracked by low-level controllers.
- Action space: Teleoperation demonstrations specify relative-to-current desired end-effector poses and gripper width, which low-level controllers track.The action space is a=(T∆,cmd,wgripper)∈SE(3)×R+.
- Observation space: The proprioceptive observation contains three hand points, axis-angle rotation relative to the task start, and gripper width.
- Observation space: The image observation is a high-dimensional RGB input represented as oimage∈R921,600.
B. Imitation Learning Visuomotor Policies
The imitation-learning formulation uses behavioral cloning to train visuomotor policies from observation-action trajectories, with weighted action losses and data augmentation for local feedback stabilization.
- Behavioral cloning: Behavioral cloning optimizes policy parameters on Ntrain observation-sequence-to-action trajectories.
- Behavioral cloning: The behavioral-cloning loss combines weighted l1 and l2 action errors with λ=0.1.Action components are scaled to balance 1.0m translation, 0.1 radians rotation, and 1.0m gripper translation.
- Data augmentation: The augmentation method partially addresses cascading errors and supports local feedback stabilization, but not recovery from discrete environmental changes.Noise standard deviations of 1mm, 1 degree, and 1cm are used for translational, rotational, and gripper components, respectively.
- Data augmentation: On-the-fly noise augmentation creates observation-action pairs that command the same global-frame setpoint under small robot-state perturbations.The method uses the low-level controller’s tracking behavior to stabilize trajectories locally.
D. Multi-View Time-Synchronized Correspondence Training
The paper trains correspondence models for dynamic scenes using approximately time-synchronized camera views, then evaluates their visual representations in simulated visuomotor policies.
- Correspondence training: Dynamic-scene correspondence matches pixels between approximately time-synchronized images from two camera views.Training requires depth images, camera poses, and camera intrinsics.
- Policy models: The visual model produces a feature vector z that serves as input to policy learning.The evaluations use MLPs with current observations and LSTMs with full observation sequences.
- Evaluation: The simulated tasks include reaching with translation or rotation variation and pushing under external disturbances.Expert demonstrations use hand-designed policies with ground-truth object state.
- Evaluation: Simulation compares visual representations from ground-truth points, autoencoding, end-to-end training, and dense correspondence.Dense descriptors encode expected image-space or 3D-space locations of descriptor points.
B. Simulation Results
Simulation results show that dense correspondence supports strong generalization and closed-loop control, often approaching ground-truth representations and outperforming autoencoding and end-to-end baselines.
- Overall simulation results: Dense correspondence policies perform remarkably close to policies receiving ground-truth world-state information, while AE and E2E performance varies more across tasks.Masked and end-to-end autoencoder variants produce mixed results depending on the task.
- Ablation: A deeper end-to-end vision backbone reduces validation error but does not improve closed-loop performance.The authors interpret this as evidence that correspondence training, rather than model capacity, drives their method’s advantage.
- Generalization and sample complexity: 3D representations generalize better outside the training distribution than corresponding 2D representations, including DD-3D over DD-2D.GT-2D and DD-2D show similar extrapolation limitations relative to their 3D counterparts.
- Closed-loop pushing: The LSTM is required for “Push box” policies, even with ground-truth object state, because demonstrations use a dynamic state-feedback controller.The pushing tasks include disturbances during both demonstration collection and policy deployment.
- Closed-loop pushing: On the harder “Push plate” task, DD-2D nearly matches GT-2D, outperforms AE and E2E, and DD-3D improves performance further.The task requires higher precision because the robot finger has point contact with the plate.
- Noise augmentation: Noise augmentation markedly improves behavior-cloned task success across ground-truth and proposed methods with 30 to 200 demonstrations.Table II evaluates the “Reach, T + R” task with and without translational and rotational noise.
C. Hardware Experimental Setup
The hardware experiments use an offboard RGBD setup and human teleoperation to validate correspondence learning and imitation policies in real-world manipulation.
- Hardware platform: The platform is a Kuka IIWA LBR robot equipped with a Schunk WSG 50 parallel-jaw gripper.The experiments cover five imitation-learning tasks.
- Correspondence validation: Figure 5 visualizes learned correspondences as heatmaps from a source pixel to predicted locations in target scenes.The broader hardware evaluation tests class variation, deformable configurations, and diverse backgrounds.
- Sensing and calibration: RealSense D415 cameras are rigidly mounted offboard and calibrated to the robot coordinate frame.Effective correspondence learning benefits from overlapping but sufficiently different camera views.
- Policy inputs: Trained hardware policies use only RGB input, with DD-2D representations and descriptor-set optimization.The visual and policy learning stages use demonstration data without additional data collection.
- Experimental challenges: The real-world evaluation introduces visual complexity, including textures, lighting, backgrounds, and clutter, plus human demonstrations.These conditions complement the controlled simulation comparisons.
1) Learned Correspondences from Dynamic Scenes:
Learned correspondences support visual recognition across object classes, deformable configurations, and varied backgrounds, while hardware policies handle challenging manipulation scenarios. Simulated comparisons suggest improved generalization and sample complexity, with no additional human supervision.
- Learned Correspondences from Dynamic Scenes:: Learned visual models identify correspondences across object classes, deformable configurations, and diverse backgrounds despite imperfect sensing and calibration.The models use time-synchronized image pairs and remain effective under depth sensor noise.
- Hardware Validation: Several hardware tasks achieve over 95% reliability, including sugar-box pushing, single-instance shoe flipping, and undisturbed plate pushing.The reported tasks include non-prehensile manipulation such as pushing and flipping.
- Hardware Validation: The policies react to deformable hat configurations after disturbances and manipulate a symmetric, textureless plate amid visual clutter.Plate manipulation uses a wood block to enable sliding the gripper into position for grasping.
- Simulation Comparisons: Simulation comparisons empirically suggest correspondence training improves generalization and sample complexity over two vision-based baselines.The approach achieves results comparable to baselines using ground-truth information in the experimental setting.
- Supervision: Because the approach is self-supervised, it does not require additional human supervision.This supports efficient policy learning in the real world without increasing human-labeling effort.
- Limitations: The representation does not explicitly handle simultaneously viewing multiple object instances of the same class.The authors suggest combining instance-level segmentation with intra-instance visual representations as future work.
APPENDIX
The appendix describes simulation settings, policy-network training, and vision-network architectures used to evaluate the visuomotor methods. It specifies task success criteria, optimization schedules, and the correspondence-model variants.
- Simulation Tasks: The simulation uses Drake to model a 7-DOF arm, gripper, objects, and multi-view RGBD sensing matched to the hardware experiments.Example tasks include reaching relative to a sugar box and pushing a plate to a target location.
- Simulation Tasks: Reach T only succeeds within 1.2cm of the target, while Push plate succeeds when the plate center is within 1cm of its target position.The plate task includes external disturbances; the box pose varies in translation but not rotation.
- Policy Networks: MLP policies use two layers of 128 hidden units, 20% dropout, ReLU nonlinearities, RMSProp, and scheduled learning-rate decay.Training uses 75,000 steps, batch size 16, and an initial learning rate of 1e−4.
- Policy Networks: LSTM training uses truncated backpropagation with a maximum length of 50 steps and downsampled trajectories at 5 Hz.The optimization uses RMSProp with gradient clipping and a decaying learning rate.
- Vision Networks: AE and E2E vision methods share an architecture, while AE adds a decoder; DD-2D and DD-3D use 16 descriptors for spatial expectations.DD-2D computes image-space expectations, whereas DD-3D computes 3D-space expectations using depth.