Source-linked AI summary
Learning Robot Manipulation from Audio World Models
Fan Zhang, Michael Gienger
TL;DR
When visual cues are ambiguous, audio can reveal evolving physical states, but audio-driven world models have remained underexplored. The paper predicts future audio with a latent flow-matching model integrated into a robot policy, and reports better performance than methods without future lookahead across real-world and simulated tasks.
Problem
Audio can provide essential state information when visual cues are scarce or unclear, but audio-driven world models have remained largely underexplored.
Source: §1 · p. 1Method
A transformer-based latent flow-matching model predicts future audio states, which are used with current audio and image observations by a robot policy.
Source: §1 · p. 1, §2 · pp. 2–3Results
Across simulation and real-world experiments, using audio predictions for robot action generation outperformed methods without future lookahead.
Source: §1 · p. 1Takeaways & Limitations
The paper concludes that audio-based world models play a critical role in informing robot action policies when task audio captures intrinsic physical dynamics.
Source: §4 · p. 4
Abstract
from arXiv · showhide
World models have demonstrated impressive performance on robotic learning tasks. Many such tasks inherently demand multimodal reasoning; for example, filling a bottle with water will lead to visual information alone being ambiguous or incomplete, thereby requiring reasoning over the temporal evolution of audio, accounting for its underlying physical properties and pitch patterns. In this paper, we propose a generative latent flow matching model to anticipate future audio observations, enabling the system to reason about long-term consequences when integrated into a robot policy. We demonstrate the superior capabilities of our system through two manipulation tasks that require perceiving in-the-wild audio or music signals, compared to methods without future lookahead. We further emphasize that successful robot action learning for these tasks relies not merely on multi-modal input, but critically on the accurate prediction of future audio states that embody intrinsic rhythmic patterns.
1 INTRODUCTION
Audio-driven world models address a gap in robot learning: audio can reveal physical state when visual cues are ambiguous, yet audio world models remain underexplored. The proposed latent flow-matching model predicts future audio for robot policies and outperforms methods without future lookahead in simulation and real-world experiments.
1 INTRODUCTION
- Audio world models remain underexplored, although audio can reveal essential state information when visual cues are scarce or unclear.During bottle filling, pitch changes over time can indicate fullness even when images remain largely unchanged.Source: §1 · p. 1
- The method uses a transformer-based latent flow-matching model to predict future audio states for robot action policies.The predicted audio representations are decoded into spectrograms and used with current audio and image observations.Source: §1 · p. 1, Figure 1 · p. 2
- Across simulation and real-world experiments, using audio predictions for action generation outperforms methods without future lookahead.The paper emphasizes that these audio cues can encode intrinsic physical dynamics, including pitch and rhythmic patterns.Source: §1 · p. 1
2 METHODS
The method predicts future audio latents from current audio and uses the resulting audio frames, together with images, to generate robot actions. It combines an audio autoencoder, a latent flow-matching world model, and a robot policy, trained as decoupled components.
2 METHODS
- An audio autoencoder encodes spectrograms into a smooth latent space, and a flow-matching world model predicts future audio latents that are decoded into audio frames.The pipeline then uses those frames to predict robot actions.Source: §2 · p. 2
- The latent flow model reconstructs a target vector field, with frame-wise conditioning separated from time-axis attention to support temporally consistent generation.Training uses noisy audio latents and includes preceding-window latents to smooth transitions across windows.Source: §2 · p. 2
- A velocity loss supervises temporal consistency by comparing frame-wise changes in the predicted and target vector fields.The total objective combines this loss with the flow-matching loss; preceding audio latents are dropped with probability 0.5 to smooth the initial window.Source: §2 · p. 2
- At inference, random source-distribution waypoints are flowed toward target audio latents from t = 0 to t = 1 over multiple steps.The method can use 1/∆t steps for inference.Source: §2 · pp. 2–3
- The robot policy conditions on current audio, image representations, and predicted future audio to generate 16-step end-effector velocity chunks.It is trained end-to-end with supervised flow-prediction MSE on future robot end-effector velocity.Source: §2 · p. 3
- Audio-world-model and policy training are decoupled, allowing components to be adapted or replaced independently.For piano playing, the paper describes replacing AudioMAE with MusicVAE and the flow-matching policy with SAC.Source: §2 · p. 3
- The full pipeline was trained on 20,000 spectrograms in approximately one day on a single NVIDIA H100 GPU.The training setup used batch size 256, AdamW with a 1.5e-4 learning rate, and 3,000 epochs.Source: §2 · p. 3
3 EXPERIMENTS
The paper evaluates audio-world-model lookahead in real-world water filling and simulated piano playing. In water filling, generated audio anticipates a button release before the bottle fills; in piano playing, future generated goals improve performance over a basic RL baseline without lookahead.
3.1 FILLING WATER
- The real-world water-filling setup uses recent spectrograms and synchronized images to generate 6-DoF end-effector velocity commands in a closed loop.Each input covers roughly 1.28 seconds of audio, and the world model generates about 2.56 seconds of future spectrograms.Source: §3.1 · p. 3
- 100% success over 30 trials was achieved in water filling, where closed-loop generated spectrograms captured rising pitch and predicted button release before the bottle filled.Figure 2 presents ground-truth and generated water-filling spectrograms.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 PIANO PLAYING IN SIMULATION
- The simulated piano duet task requires the robot to anticipate forthcoming, unheard notes from auditory input before playing its own part.The SAC policy receives dynamically generated music as goal states rather than a complete predefined song.Source: §3.2 · p. 4
- The piano world model generates the next 64 MIDI time steps from 64 input steps and can extend the sequence autoregressively.The robot policy was trained and tested on Twinkle Twinkle Little Star and Chopin Nocturne in E Flat Major Op.9.Source: §3.2 · p. 4
- In piano playing, incorporating future goal states improved performance over a basic RL baseline without future lookahead.The authors suggest that access to future notes can help the policy plan, for example by pre-positioning non-finger joints such as the wrist.Source: §3.2 · p. 4
4 CONCLUSIONS
The proposed latent flow matching audio-generation model facilitates robot policy learning. In these tasks, audio cues can capture physical dynamics, including pitch and rhythmic patterns, and the authors identify more complex dexterous tasks as future work.
4 CONCLUSIONS
- The proposed latent flow matching audio-generation model facilitates robot policy learning.The conclusion characterizes audio cues as carrying task-relevant physical dynamics, not merely serving as additional modalities.Source: §4 · p. 4
- Audio cues in these tasks often capture intrinsic physical dynamics, such as pitch and rhythmic patterns, informing robot action policies.The authors say constructing an audio-based world model plays a critical role in informing those policies.Source: §4 · p. 4
- Extending the framework to more complex tasks requiring fine-grained and dexterous manipulation is an important direction for future research.Source: §4 · p. 4