Source-linked AI summary
The Sound of Motions
Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba
TL;DR
Audio-visual source separation must distinguish and associate sounds with objects, especially when visually similar sources are mixed. The paper introduces DDT and curriculum learning to capture motion cues, improving musical-instrument separation and handling same-instrument duets. Its scope includes assumptions about trajectory processing and curriculum use in the same-instrument task.
Problem
Audio-visual source separation must distinguish sound components and associate them with objects, while prior methods often overlook motion cues that matter for visually similar sources.
Method
The system uses an end-to-end Deep Dense Trajectory network with curriculum learning to model video motion for sound separation.
Results
The approach improves prior audio-visual source-separation methods and handles challenging duets involving instruments of the same kind.
Takeaways & Limitations
Motion cues provide useful information for more robust separation of different instruments and for separating same-category instrument sounds.
Takeaways & Limitations
The method retains dense, noisy trajectories without subsampling, smoothing, or normalization, assumes learning can handle them, and uses curriculum learning only for same-instrument separation.
Abstract
from arXiv · showhide
Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.
1. Introduction
Audio-visual source separation distinguishes sound components and associates them with objects, but appearance-only approaches overlook motion cues that help disambiguate visually similar sources. The paper introduces motion-aware separation inspired by how humans associate movement with musical tempo.
- Audio-visual source separation aims to distinguish sound components and associate them with their corresponding objects.
- Watching violinists’ movements over time can help associate each performer with melody or harmony through temporal repetition and musical beats.
- Appearance-only methods often ignore motion cues, which are especially important when sound-producing objects look similar.
- Deep Dense Trajectory (DDT) is an end-to-end learnable network designed to learn motion cues for audio-visual sound separation.
- A curriculum learning scheme progresses from different instruments toward same-type instruments to force motion-based differentiation.
- Experiments on MUSIC and URMP show improved separation over prior art, including challenging duets of the same instrument category.
2. Related Work
Prior audio-visual separation methods use visual signals to assist source separation, but commonly rely on static appearance. This work explicitly models motion and targets correspondence between sound components and video movements.
- Audio-visual source separation differs from conventional separation by conditioning separation on additional visual signals.
- Prior audio-visual music-separation methods mainly obtain visual cues from a single frame and therefore emphasize appearance.
- The paper explicitly models motion cues to make fuller use of video input for sound separation.
- Earlier sound-motion studies used mutual-information or canonical-correlation methods to discover audio-visual relations.
- Video motion representations include handcrafted spatio-temporal features such as space-time interest points, HOG3D, and dense trajectories.
- The framework combines motion, appearance, fusion, and sound-separation networks, with separation conditioned on fused visual features.
- Unlike action recognition or localization, this work seeks correspondence between sound components and movements in videos.
3. Approach
The approach combines self-supervised mix-and-separate training with motion and appearance representations, using DDT to model visual motion for sound separation. A curriculum progressively trains separation from different instruments to same-instrument mixtures.
- 3.1. Mix-and-Separate for Self-supervised Learning: The model separates mixed audio conditioned on corresponding video inputs using synthetic mixtures, without requiring labeled data.Mixture and separated-audio ground truths are created by mixing paired video clips and audios.
- 3.2. Learning Motions with Deep Dense Trajectories: The end-to-end learnable DDT network incorporates differentiable optical-flow and pixel-tracking operations.The system omits trajectory smoothing and normalization and tracks within detected video shots to reduce drift.
- 3.3. Model Architectures: The architecture combines motion and appearance networks through fusion before conditioning a U-Net sound separation network.The separation network predicts a spectrogram mask, which is thresholded and converted back to audio with iSTFT.
- 3.2. Learning Motions with Deep Dense Trajectories: DDT estimates dense optical flow, tracks pixels into trajectories, and extracts trajectory features with a CNN.The system uses PWC-Net for flow estimation and I3D for trajectory feature extraction.
- 3.4. Curriculum Learning: A three-stage curriculum bootstraps training on mixtures of different instruments before mixtures of the same kinds.Same-kind mixtures can share identical semantic and contextual cues, making motion the only useful cue in that stage.
4.1. Dataset
The evaluation uses MUSIC and URMP, combining an unlabeled web-video dataset with a smaller studio-recorded multi-instrument dataset. MUSIC is additionally expanded with ten musical instrument categories.
- Datasets: MUSIC contains unlabeled YouTube videos of instrument solos and duets collected by keyword query, while URMP is a small, studio-recorded multi-instrument dataset.The two datasets provide differing video settings for vision-guided sound separation.
- Datasets: The expanded MUSIC collection adds 10 instrument categories to the 11 categories originally defined by MUSIC.The added categories include piano, drum, electric bass, guzheng, pipa, and ukulele.
4.2. Sound Separation for Different Instruments
The evaluation compares visual-input variants and baselines for instrument sound separation, showing that temporal motion representations improve performance, especially as mixtures become more crowded.
- The motion variants differ by their temporal representation: optical flow, RGB frame sequences, or tracked dense trajectories.RGB+Trajectory is the full model; RGB+Flow removes tracking, and RGB multi-frame removes the flow network.
- All compared models use 3-second mixtures, N = 2 for the main validation evaluation, and 24 video frames when vision is used.Performance is measured on 256 validation mixture pairs using SDR, SIR, and SAR.
- RGB+Trajectory, RGB+Flow, and RGB multi-frame outperform all baseline methods, while RGB+Trajectory is the strongest variant.RGB+Trajectory outperforms Sound of Pixels by approximately 0.8 dB in the N = 2 setting.
- ≈0.8dB separates RGB+Trajectory from Sound of Pixels in the N = 2 mixture evaluation.The comparison uses SDR, SIR, and SAR metrics measured in dB on mixtures of two input audios.
- ≈1.2dB at N = 3 and ≈1.8dB at N = 4 measure the larger margins over Sound of Pixels in more highly mixed cases.These experiments evaluate separation with three and four mixed sound sources.
4.3. Sound Separation for the Same Instruments
The section evaluates same-instrument sound separation using curriculum learning, quantitative comparisons, qualitative embeddings, and human judgments. The motion-based model outperforms Sound of Pixels and separates duets of the same instruments.
- Same-instrument separation is evaluated on violin, cello, congas, erhu, and xylophone mixtures using curriculum learning.The curriculum strategy is applied because this task is particularly challenging.
- Curriculum learning: Curriculum learning greatly improves performance over single-stage training, with a total SDR improvement of ≈4dB.Training progresses from different-instrument mixtures toward same-instrument mixtures.
- Quantitative comparison: The motion-based model outperforms Sound of Pixels by > 3dB on SDR for same-instrument separation.The comparison uses the same curriculum training for both models.
- Qualitative comparison: Pixel-level sound embeddings show that the proposed model distinguishes sounds in same-instrument duets, whereas Sound of Pixels cannot.Spatial sound features are projected into three dimensions and visualized by color.
- Human evaluation: Human evaluations find that the motion-based model consistently outperforms Sound of Pixels across all five instruments.The evaluation compares separated results with ground truth using three independent workers per job.
4.4. Sounding object localization
The model localizes sounding objects with roughly correct spatial predictions, focusing on the instrument parts where people interact.
- Heatmaps show roughly correct sounding-object locations, but the predictions do not cover whole instruments.The model focuses on parts where humans are interacting with the instruments.
5. Conclusion
The paper argues that motion is important for audio-visual tasks and uses Deep Dense Trajectories with curriculum learning to separate instrument sounds. The system improves separation of different instruments and handles same-kind instrument mixtures.
- The system captures visual motion with Deep Dense Trajectories to perform audio-visual sound separation.Curriculum learning enables separation of sounds from instruments of the same kind.