PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg

arXiv:1810.10191v2cs.ROcs.AIcs.LG

TL;DR

Contact-rich manipulation requires combining visual and haptic feedback, yet manually integrating heterogeneous modalities and training deep policies on real robots is difficult. The paper learns a compact multimodal representation with self-supervised, action-conditional predictions, then uses it for reinforcement-learning control. In peg insertion, the approach achieves nearly 80% simulated completion versus below 5% for all baselines and transfers across unseen peg geometries.

  • Problem

    Contact-rich manipulation requires complementary visual and haptic feedback, but combining heterogeneous modalities and training high-dimensional reinforcement-learning policies is challenging because of sample complexity.

  • Method

    The method learns a compact representation from RGB, haptic, and proprioceptive data using self-supervised action-conditional predictions, then feeds it to a reinforcement-learning policy.

  • Results

    Nearly 80% completion was achieved by the Full model in simulation, compared with below 5% for all baselines; transferred policies exceeded 60% success on two unseen peg shapes.

  • Takeaways & Limitations

    Multimodal representations using vision and touch support peg insertion across geometry variations and transfer to new task instances, while remaining robust to perturbations.

Abstract

from arXiv · show

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

I. INTRODUCTION

Contact-rich manipulation benefits from combining complementary visual and haptic feedback, but existing approaches are often task-specific and difficult to integrate. The paper learns a self-supervised multimodal representation and uses it for manipulation policies that generalize across task variations and perturbations.

  • Vision supplies semantic and geometric information, while haptics reveals contact conditions during occluded manipulation.
  • The proposed policy combines vision, touch, and proprioception despite their differing dimensions, frequencies, and characteristics.
  • Self-supervised objectives predict optical flow, imminent contact, and visual-haptic concurrency while conditioning representation learning on the next action.
  • The resulting compact representation supports deep reinforcement-learning policies for contact-rich manipulation with practical sample efficiency on a real robot.
  • The experiments evaluate multimodal feedback for hole search, peg alignment, insertion, geometry generalization, perturbation robustness, and sensor noise.

II. RELATED WORK AND BACKGROUND

Prior manipulation methods use visual, haptic, or fused modalities, but often depend on task-specific policies, manual contact configurations, or assumptions about clearance and state estimation. Learning-based representation methods offer alternatives, while explicitly exploiting temporal concurrency between sensory streams remains uncommon.

  • Traditional contact-rich policies often rely on haptic feedback and accurate state estimation, and new geometries may require entirely new policies.
  • RGB-proprioceptive policies can handle varied tasks, but large object clearances may make visual data sufficient compared with tighter automation settings.
  • Hand-designed features and sensor fusion become challenging as robotics systems combine heterogeneous modalities such as vision, haptics, proprioception, audio, range, and language.
  • Raw-sensory reconstruction can improve representation stability and speed but is data intensive and prone to overfitting.
  • Few studies explicitly exploit concurrent sensory streams in representation learning, motivating self-supervised fusion of visual and haptic data.

III. PROBLEM STATEMENT AND METHOD OVERVIEW

The paper formulates contact-rich manipulation as reinforcement learning over a finite-horizon discounted MDP. A learned low-dimensional representation of multisensory inputs serves as the policy state to improve sample efficiency.

  • The method first learns a neural feature representation of multisensory data, then uses its compact feature vector as input to a reinforcement-learning policy.
  • The manipulation problem is modeled as a finite-horizon discounted MDP with state space S, action space A, transitions T, initial distribution ρ0, reward r, horizon T, and discount γ.
  • The policy state is the low-dimensional representation learned from high-dimensional visual and haptic data, while actions are continuous 3D Cartesian displacements Δx.

IV. MULTI-MODAL REPRESENTATION MODEL

The representation model addresses the data demands of deep networks by learning visual and haptic features through automatically generated self-supervised predictive tasks. It encodes and fuses RGB, force, and proprioceptive inputs into a controller-ready representation.

  • The model uses predictive tasks whose supervision is generated automatically, avoiding laborious human annotation for visual and haptic representation learning.
  • RGB images, force readings over a 32ms window, and end-effector position and velocity are encoded and fused into a multimodal representation.
  • The representation-learning network is trained end-to-end through self-supervision to provide features for learning contact-rich manipulation controllers.

A. Modality Encoders

The model encodes visual, haptic, and proprioceptive inputs into a compact representation using self-supervised, action-conditional prediction objectives. These objectives predict optical flow, contact, and temporal alignment before supplying the resulting feature vector to policy learning.

  • A. Modality Encoders: The model uses domain-specific encoders for RGB images, wrist force-torque feedback, and proprioceptive data.The visual encoder processes 128×128×3 RGB images with a six-layer CNN.
  • A. Modality Encoders: Self-supervision replaces manual labels while encouraging the representation to encode action-related information.The training objectives use automatically generated annotations and condition predictions on the next robot action.
  • A. Modality Encoders: The action-conditional prediction module predicts optical flow and binary contact from the multimodal representation and next end-effector motion.A convolutional decoder produces a 128 × 128 × 2 flow map, while an MLP performs contact classification.
  • A. Modality Encoders: A third objective predicts whether visual and haptic streams are temporally aligned by contrasting aligned data with randomly shifted data.The alignment predictor receives the low-dimensional representation and learns cross-modal concurrency.
  • A. Modality Encoders: The three losses are optimized end-to-end, producing a 128-d feature vector that compactly represents the multimodal data for the manipulation policy.Training uses optical-flow endpoint error and cross-entropy losses for contact and alignment prediction.

V. POLICY LEARNING AND CONTROLLER DESIGN

The paper formulates contact-rich manipulation as model-free TRPO policy learning over a learned low-dimensional state representation. A trajectory-generation and impedance-control stack converts low-frequency Cartesian actions into high-frequency, compliant torque commands.

  • V. POLICY LEARNING AND CONTROLLER DESIGN: The controller receives end-effector displacement actions from the policy at 20Hz and produces robot torque commands at 200Hz.A trajectory generator interpolates high-bandwidth trajectories from low-bandwidth policy outputs.
  • V. POLICY LEARNING AND CONTROLLER DESIGN: The method uses model-free reinforcement learning to handle uncertainty in geometry, clearance, and configuration without requiring an accurate dynamics model.The policy-learning algorithm is trust-region policy optimization, which constrains each update by a KL-divergence bound.
  • V. POLICY LEARNING AND CONTROLLER DESIGN: Compliance supports peg insertion under position uncertainty by allowing the robot to slide along the box surface while pushing downward.This behavior is described as making the insertion task easier to accomplish when position is uncertain.
  • V. POLICY LEARNING AND CONTROLLER DESIGN: The trajectory generator interpolates position, velocity, and acceleration targets, which a PD impedance controller converts into task-space acceleration commands.The controller uses manually tuned proportional and derivative gains, kp and kv.
  • V. POLICY LEARNING AND CONTROLLER DESIGN: The operational-space controller maps Cartesian accelerations to joint torques using the robot’s dynamics, end-effector inertia, and Jacobian.It computes F = Λau and then τu = J^T(q)F.

VI. EXPERIMENTS: DESIGN AND SETUP

The experiments test multimodal representation learning for peg insertion in simulation and on a Kuka torque-controlled robot. The design compares sensor ablations, uses staged rewards, and evaluates task completion across insertion stages.

  • VI. EXPERIMENTS: DESIGN AND SETUP: The experiments ask whether combining all modalities improves learning, whether real-robot policy learning is practical, and whether representations generalize and recover from perturbations.The task setup varies peg and hole geometry, configuration, and clearance.
  • VI. EXPERIMENTS: DESIGN AND SETUP: The peg-insertion suite contains five 3D-printed peg and hole geometries with nominal clearance of around 2mm.The peg shapes are round, square, triangular, semicircular, and hexagonal.
  • VI. EXPERIMENTS: DESIGN AND SETUP: Both simulation and real hardware provide proprioception, RGB vision, and force-torque sensing on a 7-DoF Kuka LBR IIWA robot.Proprioception includes end-effector pose and linear and angular velocity.
  • VI. EXPERIMENTS: DESIGN AND SETUP: Simulation uses an ablative comparison between the full visual-haptic-proprioceptive model and models missing vision, haptics, or both.The figure reports partial task completion rates for the different feedback combinations.
  • VI. EXPERIMENTS: DESIGN AND SETUP: The staged reward function guides reinforcement learning through reaching, alignment, and insertion sub-tasks.Its purpose is to simplify exploration and improve learning efficiency.
  • VI. EXPERIMENTS: DESIGN AND SETUP: Performance is measured by normalized episode reward and the percentage of episodes reaching four peg-insertion stages.The categories range from completed insertion to failure before reaching the box.

VII. EXPERIMENTS: RESULTS

The evaluation first studies modality contributions in simulation, then tests the full multimodal model on a real robot. It also examines sample-efficient policy learning, representation structure, and robustness to shape and clearance variations.

  • VII. EXPERIMENTS: RESULTS: The experiments begin with a simulation ablation and then train real-robot reinforcement-learning policies from the full learned multimodal representation.This sequence separates modality analysis from practical deployment of the complete model.
  • VII. EXPERIMENTS: RESULTS: The evaluation additionally visualizes the learned representations and analyzes robustness to variations in shape and clearance.These analyses complement the policy-learning experiments by probing representation structure and task variation.

A. Simulation Experiments

Simulation experiments show that combining RGB, force, and proprioceptive inputs produces the strongest peg-insertion performance, while removing modalities degrades completion.

  • The ablation study encodes RGB images, force readings, and proprioception in different combinations before feeding representations to TRPO policies.
  • Removing either vision or force feedback negatively affects task completion, with the vision-and-haptics-absent baseline performing worst.
  • The Full model achieves nearly 80% completion, while all baseline methods remain below 5%.
  • The No haptics baseline relies solely on visual feedback and localizes the hole, but completes full insertion in few episodes.

B. Real Robot Experiments

Real-robot experiments evaluate multimodal policies across peg shapes and transfer to unseen geometries. The policies achieve high success, transfer effectively, and recover from occlusion and physical disturbances.

  • The Full model is evaluated on real hardware with round, triangular, and semicircular pegs despite sensing, synchronization, delay, and dynamics challenges.
  • The robot learns a strategy of reaching the box, sliding to search for the hole, aligning the peg, and performing insertion.
  • Transferred policies exceed 60% success on unseen hexagonal and square pegs without further policy training.
  • Using a transferred representation with a newly trained policy increases performance by 19% for the hexagonal peg and 30% for the square peg.
  • The policy recovers when the camera is periodically occluded and the robot arm is pushed during trajectory rollout.

VIII. DISCUSSION AND CONCLUSION

The paper concludes that compact, self-supervised multimodal representations support efficient reinforcement-learning control for contact-rich peg insertion and transfer across task instances.

  • The method encodes heterogeneous sensory inputs into a compact representation that remains fixed while a shallow policy is trained with reinforcement learning.
  • Self-supervision eliminates manual annotation while the representation combines time-aligned multisensory feedback for tight-clearance peg insertion.
  • Experiments indicate that peg insertion requires multimodal feedback from both vision and touch, and that the representations transfer to new task instances.
  • Future work targets other contact-rich tasks, richer modalities such as depth and sound, and a full 6-DoF controller.
Loading 1810.10191v2…