Source-linked AI summary
ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Kuppuswamy, Benjamin Burchfiel, Shuran Song
TL;DR
Vision-based manipulation systems have limited ability to sense and use contact, while prior robot audio collection has relied on teleoperated demonstrations in controlled settings. ManiWAV collects synchronous audio-visual demonstrations with a handheld gripper and learns a policy from them. The system outperforms alternatives across four contact-rich tasks and generalizes to unseen in-the-wild environments.
Problem
Method
Results
Takeaways & Limitations
Diverse in-the-wild human demonstrations can support robot-policy generalization to unseen in-the-wild environments.
Source: §1 · p. 2Audio may be ineffective when interactions produce no salient signals or when robot motor noise makes those signals imperceptible; the method also does not use audio’s higher frequency for more reactive behavior.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations.
1 Introduction
ManiWAV treats audio as a useful contact signal for manipulation, addressing vision-based systems’ limited ability to sense and use contact. It pairs in-the-wild audio-visual demonstrations with a policy designed to bridge deployment audio differences.
1 Introduction
- Audio feedback can reveal contact events and modes, surface materials, and object states or properties beyond visual observation.These signals are relevant when visual information is ambiguous or incomplete.Source: §1 · pp. 1–2
- Audio collection is presented as scalable because contact microphones are cheap, robust, and readily integrated into standard video recording pipelines.The handheld approach avoids the need for a robot during data collection.Source: §1 · p. 2
- ManiWAV embeds a piezoelectric contact microphone in a handheld gripper to collect synchronous audio and vision during human demonstrations.The signals are stored together through the camera, enabling collection without a robot.Source: §1 · p. 2, Figure 2 · p. 2
- The policy uses audio augmentation to address deployment-domain noise and an end-to-end audio-visual network with a diffusion-policy head for action prediction.Source: §1 · p. 2
- The system is demonstrated on four contact-rich tasks and is reported to generalize to unseen in-the-wild environments using diverse demonstrations.Source: §1 · p. 2
2 Related Work
Prior work studies tactile sensing and acoustic sensing for manipulation, but tactile sensors can be difficult to reproduce and prior passive-audio collection has relied on controlled teleoperation. ManiWAV proposes in-the-wild audio-visual collection and end-to-end policy learning as alternatives.
Tactile Sensing for Contact-rich Manipulation
- Tactile sensing spans force/torque sensors, camera-based sensors, and tactile skins, but cost and domain expertise constrain reproducibility.Prior robot-learning work uses tactile feedback to learn visuo-tactile representations for contact-rich tasks.Source: §2 · p. 2
Acoustic Sensing in Controlled Environments
- Prior passive acoustic robot-learning studies use audio inputs, but collect data in controlled settings through teleoperation with sensors on the robot or object.ManiWAV instead proposes a passive-sensing handheld gripper for human demonstrations without a robot.Source: §2 · p. 3
Policy Learning from Multisensory Data
- ManiWAV extends end-to-end multisensory behavior cloning with audio augmentation to bridge the gap between in-the-wild demonstrations and robot deployment.The related work contrasts this approach with policies requiring predefined state abstractions.Source: §2 · p. 3
3 Method
ManiWAV combines a handheld audio-visual data-collection device with a policy that learns task-relevant representations and addresses the audio gap between demonstrations and robot deployment.
3 Method
- The method pairs in-the-wild audio-visual demonstrations with augmentation and a transformer-based model to bridge the demonstration-to-deployment audio gap.Source: §3 · p. 3
3.1 Ear-in-Hand Hardware Design
- The handheld UMI-based gripper embeds a piezoelectric contact microphone beneath high-friction tape and records audio synchronously with images.The microphone connects to the camera’s external mic port; deployment uses the same gripper design on a UR5 arm.Source: §3.1 · p. 3
3.2 Policy Design
- The closed-loop model takes RGB images and audio as input and predicts 10-DoF robot actions.The action includes end-effector position and orientation plus gripper openness.Source: §3.2 · p. 3
- Audio augmentation adds environmental and robot motor noises to training signals to encourage focus on task-relevant audio despite deployment noise.The approach samples background sounds from ESC-50 and records motor noise under sampled robot trajectories.Source: §3.2 · p. 4
- Training the audio transformer from scratch outperforms pretrained and from-scratch CNN encoders in the reported experiments.Source: §3.2 · p. 4
- The policy fuses vision and audio features with a transformer and conditions a diffusion-policy head on the resulting observation representation.The fusion can weight modalities differently across task stages, such as vision for approaching an object and audio upon contact.Source: §3.2 · p. 4
4 Evaluation
The evaluation tests audio feedback across four contact-rich tasks, examining its roles in sensing contact events and modes, object states, and surface materials. It compares alternative approaches and tests robustness and generalization across scenarios.
4.1 Flipping Task
- In flipping, audio-enabled MLP outperforms vision-only diffusion, while action diffusion outperforms MLP; in-the-wild training improves results in two unseen environments.A transformer audio encoder also outperforms the ResNet and AVID CNN-based encoders; the authors suggest self-attention may focus on task-relevant frequencies.Source: §4.1 · p. 6
4.2 Wiping Task
- 40% to 85%: Contact audio improves wiping success; noise augmentation improves robustness to test-time sounds, and transformer fusion outperforms MLP fusion.Without noise augmentation, performance is less robust to test-time noise and generalization suffers; simply masking robot-noise frequencies helps somewhat but does not address other execution noises.Source: §4.2 · p. 7
4.3 Pouring Task
- For pouring, audio vibrations and an information-seeking action reveal whether the cup contains dice, with generalization to objects with similar sounds such as screws.A 1s audio history performs significantly worse because it lacks sufficient information; a 10s history contains sufficient information but increases learning complexity and model-capacity demands.Source: §4.3 · p. 7
4.4 Taping Task
- In taping, contact-microphone audio reliably guides correct tape selection, unlike vision-only and environmental-microphone approaches; vision-only achieves 20% success.Training-time noise augmentation is more effective than test-time noise reduction: the latter degrades the signal and creates a train-test domain gap.Source: §4.4 · p. 8
4.5 Limitations and Future Directions
- Contact audio may be unhelpful when interactions produce weak signals, such as with cloth or quasi-static tasks, and robot motor noise can make it imperceptible.The current policy also does not exploit audio's higher sampling frequency for more reactive behavior.Source: §4.5 · p. 8
5 Conclusion
ManiWAV combines in-the-wild audio-visual demonstration collection with policy learning for contact-rich manipulation. Across four tasks, it outperforms alternative approaches and generalizes to unseen in-the-wild environments.
5 Conclusion
- ManiWAV presents an in-the-wild audio-visual data collection and policy-learning framework for contact-rich manipulation.Audio signals provide information about robot interactions and object properties that can ease policy learning.Source: §5 · p. 8
- An ear-in-hand gripper collects human demonstrations that can directly train robot policies through behavior cloning.The learned audio-visual representation conditions a diffusion policy.Source: §5 · p. 8
- Across four contact-rich tasks, ManiWAV outperforms several alternative approaches and generalizes to unseen in-the-wild environments.Source: §5 · p. 8
A Method Details
The method synchronizes audio and video by calibrating their latency, then specifies augmentations, spectrogram settings, the fusion encoder, and end-to-end training configuration.
A.1 Audio Latency Calibration
- Audio arrives 0.06 s after the image, giving a total audio latency of 0.23 s after accounting for 0.17 s calibrated image latency.The authors align the streams by tapping the contact microphone and matching the audio signal to the frame showing contact.Source: §A.1 · p. 12
A.2 Model Details
- Images are randomly cropped at a 95% ratio and resized, with ColorJitter applied for lighting robustness.ColorJitter uses brightness 0.3, contrast 0.4, saturation 0.5, and hue 0.08.Source: §A.2 · p. 12
- At 16 kHz, the spectrogram uses a 400-sample window and 160-sample hop to slightly augment sparse contact patterns along time.The passage explains that smaller hops elongate the spectrogram’s time axis.Source: §A.2 · p. 12
- One 8-head transformer encoder layer fuses vision and audio features, with feedforward dimension 2048 and dropout 0.0.Source: §A.2 · p. 12
- The model is trained end-to-end for 60 epochs on 2 NVIDIA GeForce RTX 3090 GPUs, using batch size 64 and AdamW with EMA.The optimizer settings are lr=1e-4, betas=[0.95, 0.999], eps=1.0e-8, and weight decay=1.0e-6.Source: §A.2 · p. 12
B.1 In-the-Wild Data Collection
The bagel-flipping data collection spans 274 in-the-wild demonstrations across six venues and uses four different pans.
B.1 In-the-Wild Data Collection
- The bagel-flipping dataset contains 274 in-the-wild demonstrations collected across six locations.Counts are 52 in a conference room, 37 and 44 in two kitchens, 46 in an office, 76 on lounge tables, and 19 in a cafe.Source: §B.1 · p. 12
- The demonstrations cover a conference room, two kitchens, an office, lounge tables, and a cafe.Source: §B.1 · p. 12
- The demonstrations use 4 different pans.Source: §B.1 · p. 12
B.2.1 Pouring Task
In pouring, audio feedback helps the policy infer whether a cup contains objects, supporting reliable pouring and placement that the baselines do not achieve; the policy also generalizes to screws.
B.2.1 Pouring Task
- The method reliably guides pouring and cup placement, while baselines never execute the pour or place action and have low substep success.The table breaks down success for grasp, pour, and place.Source: Table 1 · p. 12, §B.2.1 · pp. 12–13
- Audio feedback helps infer whether the cup contains objects, and the policy generalizes to unseen screws.Shaking produces a distinctive spectrogram for an empty cup; dice and screws produce similar patterns.Source: §B.2.1 · pp. 12–13, Figure 11 · p. 13
B.2.2 Taping Task
In taping, audio feedback helps the policy choose the correct tape by inferring its surface material, while baselines make random choices; noise reduction leaves a signal-quality gap.
B.2.2 Taping Task
- Audio feedback guides the policy to choose the correct tape, whereas baselines make random decisions at the sensing step.The success-rate breakdown covers touch, sense, pick, and place.Source: Table 2 · p. 13, §B.2.2 · p. 13
- The sensing step identifies whether the tape is hook or loop from its surface material.Source: §B.2.2 · p. 13
- Non-stationary noise reduction removes much robot and background noise but distorts the original signal and leaves a training–deployment domain gap.The method smooths spectrograms, estimates noise thresholds by frequency band, and applies a mask to the original spectrogram.Source: §B.2.2 · pp. 13–14, Figure 12 · p. 14
B.2.3 Wiping Task
In wiping, noise masking improves performance but still leaves frequent pressing failures; vision-only control often floats or presses, especially at unseen table heights, while MLP fusion often stops wiping too early.
B.2.3 Wiping Task
- Vision-only wiping most often fails by floating above the board or pressing too hard, especially at unseen table heights.The authors attribute this pattern to insufficient contact inference from the top-down wrist camera image alone.Source: Table 3 · p. 14, §B.2.3 · p. 14
- The scenario breakdown distinguishes floating above the board, pressing too hard, and incomplete wiping or wiping in the wrong location.These labels describe different contact and task-progress failures.Source: §B.2.3 · p. 14, Table 3 · p. 14
- With MLP fusion, the most common failure is incomplete wiping, when the robot releases the eraser before finishing the shape.The authors hypothesize that MLP fusion loses information needed to infer task stage.Source: §B.2.3 · p. 15
- Noise masking improves wiping over no augmentation, but still fails half the time, mostly by pressing too hard.Without noise augmentation, the policy also exhibits incomplete wiping and floating failures.Source: §B.2.3 · p. 15
B.2.4 Flipping Task
In the flipping task, vision-only control generalizes poorly across randomized starting configurations, while ResNet and AVID audio encoders often fail to detect spatula contact with the pan bottom.
B.2.4 Flipping Task
- The scenario breakdown distinguishes side-poking, losing bagel contact before the flip, and spatula displacement during downward motion.The bagel can fall back to its original side after contact is lost.Source: §B.2.4 · p. 15, Table 4 · p. 15
- With randomized initial poses and object positions, vision-only control generalizes poorly, often poking the bagel’s side or losing contact before the flip.The authors note that the image alone makes it hard to infer whether the spatula contacts the pan bottom.Source: §B.2.4 · p. 15
- ResNet and AVID audio encoders most often displace the spatula, likely because they miss the sound of spatula contact with the pan bottom.The robot consequently continues moving downward instead of sliding the spatula along the pan bottom.Source: §B.2.4 · p. 16