Source-linked AI summary
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao, Jiajun Wu
TL;DR
Existing robot-manipulation research has mostly used one or two modalities, leaving vision, audio, and touch unstudied together in challenging tasks. This paper fuses all three with a self-attention model for dense packing and pouring, and reports that the system outperforms baseline methods on both tasks.
Problem
Prior robotic manipulation work had limited use of multiple sensory modalities, with no prior work simultaneously studying vision, audio, and touch in challenging manipulation tasks.
Source: §1 · p. 1Method
The robot uses a multisensory self-attention model to fuse visual, acoustic, and tactile data across modalities and time for imitation-learning action prediction.
Source: §1 · p. 1, §3.2 · p. 4Results
The all-modality system outperforms baseline methods on dense packing and pouring.
Source: §4.2 · p. 7, §5 · p. 8, §4.3 · p. 8Takeaways & Limitations
Vision provides global information but can be occluded, audio gives immediate feedback of occluded events, and touch supplies precise local geometry.
Source: §1 · p. 1The behavioral-cloning framework requires collecting human demonstrations.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.
1 Introduction
The paper studies how vision, audio, and touch can complement one another in complex robotic manipulation, addressing the lack of prior work combining all three modalities. It introduces a multisensory self-attention approach and examines it on dense packing and pouring tasks.
1 Introduction
- No prior work had simultaneously incorporated visual, acoustic, and tactile signals to study their respective roles in challenging robotic manipulation tasks.Earlier attempts combined audio or touch with vision, but not all three modalities together.Source: §1 · p. 1
- The study equips a Franka Emika Panda arm with a camera, contact microphone, and vision-based tactile sensor for dense packing and pouring.Dense packing inserts a target object into a cluttered box; pouring transfers tiny beads between cylinders to reach a target level.Source: §1 · p. 1, Figure 1 · p. 2
- The proposed multisensory self-attention model fuses vision, audio, and touch for imitation-learning action prediction and outperforms prior fusion methods and ablated versions.The paper reports quantitative and qualitative analyses of the modalities’ characteristics.Source: §1 · p. 1
- The analyses uncover modality characteristics in robotic tasks that correlate with human behaviors and agree with intuition.The introduction frames vision as global but occlusion-prone, audio as providing immediate alerts even for occluded events, and touch as offering precise local geometry.Source: §1 · pp. 1–2
2 Related Work
Prior research covers multimodal learning and robotic manipulation, including work using audio or touch alongside vision. This paper distinguishes its study of all three senses in real-robot, complex manipulation tasks from prior work focused on simulated simple tasks or other objectives.
2 Related Work
- Prior multimodal research spans representation learning, embodied learning, video understanding, and cross-modality prediction; this paper instead studies vision, audio, and touch in robotic manipulation.Related robotics work has used audio or touch for tasks including object classification, shape reconstruction, grasping, and peg insertion.Source: §2 · p. 2
- Earlier manipulation work includes peg-in-hole and pouring, while this paper uses multiple sensory modalities for dense packing and pouring to a target amount.The tasks require attention to different modalities at different stages.Source: §2 · p. 2
- Unlike ObjectFolder, which creates a neural-object dataset and uses simulated multisensory data for simple tasks, this work studies two complex tasks using a real robot.The paper presents an in-depth study of fusing all three sensory modalities and their roles in robot learning.Source: §2 · p. 3
3 Multisensory Robot Learning
The section presents a robot-learning pipeline that captures visual, acoustic, and tactile data, fuses them with a multisensory self-attention model, and applies them to dense packing and pouring.
3.1 Multisensory Data Capture
- The robot captures time-aligned visual, acoustic, and tactile observations using a third-person camera, a contact microphone, and GelSight fingertips.The camera provides a global view and end-effector position; the microphone captures contact vibrations, while GelSight measures contact geometry and forces.Source: §3.1 · p. 3
3.2 Multisensory Self-Attention Model
- MULSA applies self-attention across modalities and time, then concatenates the attended features and uses an MLP to predict actions.The model first extracts modality features with ResNet-18; its attention includes cross-modality, cross-time, and combined attention.Source: §3.2 · p. 4
- The policy is trained by imitation learning on time-aligned multisensory demonstrations, using cross-entropy between expert and predicted discrete actions.The expert supplies the demonstration action labels.Source: §3.2 · p. 4
3.3 Task Setup
- Dense packing requires moving an object above a crowded box, locating an empty space while avoiding obstacles, and inserting it; setups include printed bumps and diverse real-world objects.The printed setup uses a cylinder peg and a two-wall base with four bump configurations; the real-object setup packs a glass among objects such as a metal board, toy, towel, and plate.Source: §3.3 · pp. 4–5
- Pouring requires aligning two cups, rotating the held cup to pour beads, and stopping at the target level; the robot predicts horizontal movement and rotation actions.The setup uses 1 mm beads, starts with either 60 g or 100 g, and targets 40 g in the fixed cup, with an electronic scale measuring the result.Source: §3.3 · p. 5
4 Experiments
Experiments evaluate multisensory manipulation on dense packing and pouring, using a Panda robot equipped with visual, acoustic, and tactile sensing. Across controlled and real-world settings, the three-modality model outperforms the baselines, while attention shifts among senses with task demands.
4.1 Physical Setup and Implementation Details
- Experiments use a Panda arm with camera, contact microphone, and GelSight sensing, running Cartesian policy commands at 10 Hz and low-level control at 200 Hz.Visual and tactile frames are paired with audio segments across six observations; dense-packing and pouring policies are trained from human demonstrations.Source: §4.1 · p. 6, §4.2 · p. 6, §4.3 · p. 8
4.2 Dense Packing
- 100% success in all four 3D-printed bump scenarios; vision-only, vision+audio, and vision+touch average 25%, 60%, and 75%, respectively.Vision and touch suffice on flat bumps; audio distinguishes hard from soft slanted bumps, while touch helps distinguish the two flat bumps.Source: Table 1 · p. 7, §4.2 · p. 7
- Attention shifts from vision to audio and touch when packing walls occlude the peg, then returns to vision after it clears the bump.During pouring, attention moves toward audio and touch as beads fall and the cup’s center of mass changes; acoustic feedback is especially important for predicting when to stop.Source: §4.2 · p. 7, §4.3 · p. 8
- MULSA consistently outperforms all baselines in dense packing with real-world objects of diverse shapes and material properties.Observed failures include pushing too hard on hard obstacles and missing the empty-space range during alignment.Source: §4.2 · p. 7, Table 2 · p. 8
4.3 Pouring
- MULSA with all three modalities achieves the best pouring results, outperforming both fusion baselines; adding audio or touch to vision yields large gains over vision-only.Vision-only keeps pouring because similar visual feedback does not reveal the current fill status.Source: Table 3 · p. 8, §4.3 · p. 8
5 Conclusion and Limitations
The multisensory framework solves dense packing and pouring, and its self-attention model outperforms baseline methods. The authors describe complementary roles for vision, audio, and touch, while noting limitations in learning and generalization.
5 Conclusion and Limitations
- The multisensory self-attention framework solves dense packing and pouring and strongly outperforms baseline methods.It fuses and filters useful information from visual, acoustic, and tactile data.Source: §5 · p. 8
- Vision provides general position but often suffers occlusion; audio gives immediate feedback and object-material cues but is less informative without contact.Touch captures in-hand dynamics, local geometry, and contact-force direction and magnitude.Source: §5 · p. 8
- Behavioral cloning requires human demonstrations, and the robot's multisensory perception may differ from how humans use senses during demonstrations.The authors identify reinforcement learning and a more general robot setup as potential future improvements.Source: §5 · p. 8