Source-linked AI summary
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg
TL;DR
Contact-rich manipulation requires combining visual and haptic feedback, but heterogeneous sensing and reinforcement-learning sample complexity make robot deployment difficult. The paper learns a compact multimodal representation with self-supervision, then uses it for policy learning, showing peg-insertion generalization across task variations and robustness to perturbations in simulation and on a robot.
Problem
Contact-rich manipulation needs complementary visual and haptic feedback, yet manually combining heterogeneous modalities and deploying deep reinforcement learning on real robots remain difficult.
Method
The paper learns a compact joint representation of visual, haptic, depth, and proprioceptive inputs through self-supervised predictive objectives, then trains a policy on the representation.
Results
22% insertion rates resulted from removing the pairing loss, while the full approach generalized across peg-insertion variations and transferred to a real robot.
Takeaways & Limitations
Multimodal self-supervised representations can support contact-rich peg insertion across varying geometries, configurations, clearances, and perturbations.
Abstract
from arXiv · showhide
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.
I. INTRODUCTION
The paper addresses the difficulty of combining complementary visual and haptic feedback for contact-rich manipulation without relying on handcrafted features or prior knowledge. It proposes self-supervised multimodal representation learning and evaluates peg insertion across task variations, perturbations, modalities, and model designs.
- Vision supplies semantic and geometric information, while haptics reveals contact conditions under occlusion, making the modalities complementary during manipulation.
- The proposed policy learns a joint representation of haptic, RGB-D, and proprioceptive information through self-supervised objectives.The objectives predict optical flow, imminent contact, future end-effector state, and visual-haptic concurrency.
- The peg insertion evaluation tests generalization across geometries, configurations, and clearances, along with robustness to external perturbations.
- The work introduces a variational multimodal representation model and demonstrates haptic and visual feedback for hole search, peg alignment, and insertion.
- The study compares modalities, representation objectives, architectures, latent dimensions, depth and roll inputs, and reproduction on a Franka Panda robot.
A. Contact-Rich Manipulation
Prior contact-rich manipulation methods often depend on force control, accurate state estimation, handcrafted structure, demonstrations, or object models. Representation learning offers a route to compact task-relevant states, while concurrent heterogeneous modalities remain challenging to fuse effectively.
- Traditional contact-rich policies often rely on haptic feedback and force control, accurate state estimation, or manually defined contact configurations.
- Existing multimodal approaches may require manipulation graphs, human demonstrations, CAD models, or remain limited to single tasks.
- Simulation-to-real learning has rarely addressed contact-rich tasks with haptic feedback, partly because contact simulation and collision modeling are difficult.
- A compact, low-dimensional representation can encode task-relevant state information and make reinforcement learning more data efficient.
- Reconstruction is a common baseline that can improve stability and speed but is data intensive and prone to overfitting.
- The paper builds on product-of-experts fusion to exploit concurrent heterogeneous modalities in a learned latent space.
III. PROBLEM STATEMENT AND METHOD OVERVIEW
The paper learns a compact multisensory representation before policy training, then evaluates multimodal fusion and representation transfer for contact-rich manipulation. It formulates control as model-free reinforcement learning over a latent state.
- The central evaluation asks whether multisensory information and learned multimodal representations improve contact-rich manipulation and transfer across tasks.
- A neural network first learns a compact feature vector from multisensory data, which then serves as input to a deep reinforcement learning policy.
- The manipulation task is modeled as a finite-horizon, discounted Markov Decision Process with states, actions, transitions, rewards, and a discount factor.
- The policy state is a learned latent representation of visual and haptic data, while actions are continuous 3D position and roll-angle displacements.
IV. MULTI-MODAL REPRESENTATION MODEL
The model learns a compact multimodal latent representation using variational inference and self-supervised predictive objectives rather than reconstructing raw inputs. This representation is intended to capture information relevant for contact-rich manipulation policy learning.
- The approach addresses the need for automatic supervision when learning representations from high-dimensional visual and haptic data.
- The framework learns an approximate posterior over latent representations from sensor readings, self-supervised labels, and robot actions.
- Variational inference optimizes an ELBO whose likelihood term trains neural decoders and whose KL term regularizes the learned posterior toward a standard Gaussian prior.
- Each data point is assumed to map to a unique latent representation, allowing the ELBO to decompose into per-data-point terms.
- Unlike a conventional VAE, the decoders predict self-supervised action-conditional targets instead of reconstructing input data.
B. Modality Encoders
The model uses modality-specific encoders for heterogeneous RGB, depth, haptic, and proprioceptive inputs, then fuses their estimated distributions into one latent representation. Product-of-experts fusion combines modality posteriors under conditional-independence and isotropic-Gaussian assumptions.
- Modality Encoders: The model encodes RGB, depth, wrist force-torque, and joint-encoder data into a single d-dimensional latent vector.
- Modality Encoders: RGB and depth images use separate CNN encoders that output 2×d-dimensional variational parameter vectors.
- Multimodal Fusion: Product-of-experts fusion combines modality-specific Gaussian posteriors into a multimodal latent distribution.
- Multimodal Fusion: For each modality, µ_i j denotes the posterior mean and σ2_i j denotes the posterior variance for latent dimension j.
D. Self-Supervised Predictions and Decoder Architecture
The representation is trained with action-conditional predictions that connect current multimodal observations to future visual motion, contact, end-effector state, and sensor-stream alignment. Decoder architectures are specialized to the target modality, while the controller converts low-rate policy actions into high-rate torque commands.
- Self-Supervised Predictions: The model predicts action-conditional optical flow, flow masks, future contact, and future end-effector positions from the current representation and next action.
- Decoder Architecture: The flow decoder uses four convolutional layers with upsampling and skip connections, producing a 32×32×2 prediction before upsampling to 128×128×2.
- Self-Supervised Predictions: A fifth objective classifies whether two sensor streams are temporally aligned using aligned and randomly shifted training examples.
- Controller: The policy supplies end-effector position and z-axis orientation displacements at 20Hz, while the controller outputs torque commands at 1000Hz.
- Controller: A trajectory generator interpolates policy actions, an impedance PD controller tracks them, and an operational-space controller transforms accelerations into joint torques.
E. Loss Functions and Training Details
Training minimizes six loss terms end-to-end: five self-supervised prediction losses and a KL regularizer. The study also compares the full variational model with deterministic and reconstruction-based representation baselines.
- Loss Functions: Binary prediction tasks use Bernoulli likelihoods and cross-entropy losses, while continuous predictions use Gaussian likelihoods and mean squared error.
- Loss Functions: Optical flow uses endpoint error, end-effector prediction uses mean squared error, and contact, alignment, and mask prediction use cross-entropy.
- Loss Functions: The objective combines five prediction losses with a KL-divergence term, giving six loss terms optimized jointly.
- Training Details: The six losses are minimized end-to-end with stochastic gradient descent on rolled-out trajectories using the reparameterization trick.
- Baselines: The deterministic baseline uses deterministic encoders and decoders, whereas the reconstruction baseline reconstructs RGB, force, and proprioceptive inputs.
- Baselines: The reconstruction model measures reconstruction error across its three modalities with mean squared error.
VI. POLICY LEARNING AND CONTROLLER DESIGN
The paper formulates contact-rich manipulation as model-free reinforcement learning over multimodal sensory representations. TRPO maps the learned representation to end-effector displacements, while a high-rate controller converts these into torque commands.
- Policy Learning: Model-free reinforcement learning avoids requiring an accurate dynamics model for rich-contact manipulation under uncertain geometry, clearance, and configuration.
- Policy Learning: TRPO constrains each policy update by bounding its KL-divergence from the previous policy.
- Policy Learning: The policy network is a 2-layer MLP that consumes a d-dimensional multimodal representation and outputs 3D position and 1D orientation displacements.
- Controller Design: The controller runs policy commands at 20Hz and outputs direct robot torque commands at 500Hz through trajectory generation, impedance control, and operational space control.
- Experimental Design: The study compares representations using different sensory modalities, loss functions, representation models, and latent-space dimensions in simulation.
- Experimental Setup: Experiments use four peg and hole geometries with nominal clearance around 2mm, and evaluate normalized episode rewards alongside staged insertion outcomes.
VIII. EXPERIMENTS: RESULTS
The experiments first ablate modalities, representation techniques, and latent dimensionality in simulation, then apply the full multimodal model to real-robot peg insertion. Removing RGB, depth, or force input reduces task completion, with removing depth performing worst.
- Experimental Design: The experiments ablate sensory modalities, representation learning techniques, and latent-space dimensionality before evaluating the full model on a real robot.
- Modality Ablation: The representation model encodes and fuses RGB images, depth images, force readings, and proprioception for square-peg insertion policies.
- Evaluation Procedure: Training uses 2.0k episodes of 500 steps, with policy updates every four episodes and evaluation based on 50 rollouts per selected checkpoint.
- Modality Ablation: Removing RGB images, depth, or force input negatively affects task completion, and the No Depth baseline performs worst.
2) Representation Learning Model:
The Full Model combines variational multimodal representation learning with self-supervised objectives for contact-rich peg insertion. Experiments evaluate objective ablations, representation size, real-robot performance, transfer to novel geometries, and perturbation recovery.
- The Full Model uses variational encoders to predict action-conditional optical flow, contacts, end-effector pose, and time-aligned sensory pairing.
- No Pairing reduces insertion rates to 22%, while the Reconstruction Model reaches 36%, still below the Full Model.The pairing objective is especially important because it learns co-occurring patterns across modalities, such as contact events.
- 78% full insertion is achieved with a 128-dimensional representation, whereas d=16 reaches 18% and d=256 reduces performance by more than a third.The d=16 model also has 95.3% contact-prediction accuracy and a 1.56E-02 end-effector pose prediction loss.
- B. Real Robot Experiments: The real robot achieves success similar to simulation across triangular and semicircular pegs using a learned reach-search-align-insert strategy.Each policy is trained for 450 episodes of 1000 steps, approximately 7 hours of wall-clock time, and evaluated over 50 episodes.
- B. Real Robot Experiments: Transferred policies exceed 70% success on unseen hexagonal and square pegs without additional policy training.Reusing the triangular-peg representation while training new policies improves performance by 8% for hexagonal and 10% for square pegs.
- B. Real Robot Experiments: The policy recovers from periodic camera occlusion and external pushes during trajectory roll-outs.
IX. DISCUSSION AND CONCLUSION
The study finds that compact, time-aligned multimodal representations support contact-rich peg insertion and transfer across task instances. It also identifies latent-space size as an unresolved trade-off and points toward broader task and sensing extensions.
- A fixed compact latent representation learned from time-aligned multisensory data enables shallow-policy reinforcement learning without manual annotation.
- Vision and touch are both required for the evaluated tight-clearance peg insertion tasks.
- Action-conditional and time-alignment pairing objectives outperform reconstruction objectives, with pairing especially important for policy performance.
- Latent-space size trades off capturing sufficient state information against keeping the policy state space compact.The paper identifies the need for more principled methods to make this trade-off.
- Future work targets other contact-rich tasks, full 6-DoF control, richer sensing modalities, and new self-supervision sources.
APPENDIX A REPRESENTATION TRAINING DETAILS
The appendix specifies the optimization settings and reports representation-learning losses for the self-supervised and reinforcement-learning experiments. It also documents a real-robot TRPO adjustment used to stabilize late training.
- The appendix reports hyperparameters for representation learning using Adam and reinforcement learning using TRPO.
- Representation learning uses Adam-based stochastic gradient descent over the prediction objectives for simulation and real-robot data.
- Representation prediction losses and test-to-training loss ratios are reported after 20 training epochs.
- During the final real-robot training hour, λ is increased and γ decreased because this stabilized learning.