Source-linked AI summary
Music Gesture for Visual Sound Separation
Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, Antonio Torralba
TL;DR
Existing visual sound-separation methods rely mainly on appearance or optical-flow-like motion features, which have limited ability to associate audio with visual points when separating same-type instruments. Music Gesture introduces structured keypoints for explicit musician body and finger movements, integrates body dynamics with semantic context, and fuses them with audio. Across three music-performance datasets, it improves heterogeneous separation benchmarks and enables effective piano, flute, and trumpet duet separation, while remaining vulnerable to viewpoint changes and occlusions.
Problem
Appearance- and optical-flow-like visual representations have limited ability to find audio–visual correlations, especially for separating multiple instruments of the same type.
Method
Music Gesture uses structured keypoint-based representations, a graph network integrating body dynamics with semantic context, and audio-visual fusion to associate movements with sound signals.
Results
The model improves heterogeneous instrument separation and achieves effective homo-musical separation for piano, flute, and trumpet duets across three music-performance datasets.
Takeaways & Limitations
Structured visual representations open opportunities for audio-visual source separation involving different instruments and challenging same-instrument duets.
Takeaways & Limitations
The method is not resilient to camera viewpoint changes and musician body-part occlusions.
Abstract
from arXiv · showhide
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks. However, these approaches are mostly built on appearance and optical flow like motion feature representations, which exhibit limited abilities to find the correlations between audio signals and visual points, especially when separating multiple instruments of the same types, such as multiple violins in a scene. To address this, we propose "Music Gesture," a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals. Experimental results on three music performance datasets show: 1) strong improvements upon benchmark metrics for hetero-musical separation tasks (i.e. different instruments); 2) new ability for effective homo-musical separation for piano, flute, and trumpet duets, which to our best knowledge has never been achieved with alternative methods. Project page: http://music-gesture.csail.mit.edu.
1 MIT, 2 MIT-IBM Watson AI Lab
Music Gesture uses explicit body-dynamics motion cues for visual sound separation in music performances, targeting both heterogeneous and homogeneous separation tasks.
- The model leverages explicit body-dynamics motion cues for visual sound separation in music performances.
- The proposed system is evaluated on both heterogeneous and homogeneous music separation tasks.
1. Introduction
Music Gesture treats music performance as a source of informative body and instrument interactions, using structured keypoint representations and audio-visual fusion to guide sound separation. Evaluations report favorable results across heterogeneous and homogeneous instrument-separation tasks.
- Music performance gestures connect musicians’ bodily movements with the sounds they produce, providing visual cues for distinguishing sources.
- Music Gesture uses structured keypoint-based visual representations to exploit explicit body and hand movements rather than only semantic or optical-flow features.
- The framework combines video analysis of body dynamics and instrument context with an audio-visual separation network conditioned on visual features.
- A dedicated audio-visual fusion module adjusts sound features according to visual features to better use body-motion cues.
- Experiments on URMP, MUSIC, and AtinPiano show favorable performance against state-of-the-art methods for heterogeneous and homogeneous separation.
- The paper reports large-margin improvements for heterogeneous separation and new opportunities for piano, flute, and trumpet duet separation.
2. Related Work
Prior work spans signal-based, visual-semantic, motion-based, and broader audio-visual learning methods, while explicit body-pose cues for sound separation remain comparatively underexplored. Music Gesture addresses this gap with structured keypoints integrated with semantic context and audio features.
- Classical sound-separation methods such as NMF rely on low-level signal correlations, while deep models predict time-frequency masks for music separation and enhancement.
- Visual sound-separation research has used audio-visual onset associations, semantic cues, deep networks with NMF, and vision-based speech-separation methods.
- Temporal trajectory and optical-flow-like motion features were limited in modeling human-object interactions and lacked broad applicability to real mixtures.
- Audio-visual learning has jointly or separately learned audio and image representations for cross-modal tasks including sounding-object localization.
- Body-pose correlations with sound were less explored than face-speech associations, and prior work mainly studied predicting body dynamics from music or speech rhythms.
3. Approach
Music Gesture separates sounds by combining structured body-and-hand keypoint dynamics with visual context and audio-visual fusion. Its pipeline extracts these representations, associates them with mixture spectrograms, predicts target masks, and reconstructs separated waveforms.
- System Pipeline: The mix-and-separate pipeline creates synthetic mixtures from paired video-audio clips and learns to recover each source conditioned on its visual context.During training, audios from N selected clips are linearly combined, while the model separates each source from the mixture.
- Video Analysis Network: The video analysis network combines global semantic context with structured keypoint representations of body and hand movements.ResNet-50 supplies a 2048-dimensional context vector, while AlphaPose estimates body joints and hand poses.
- Video Analysis Network: The context-aware graph CNN models spatial-temporal relationships among body and finger keypoints, then incorporates visual appearance context into node features.Graph convolutions encode poses at each time step, temporal convolution fuses dynamics, and context features are concatenated to the node representation.
- Audio-Visual Separation Network: The audio-visual separation network takes a mixture spectrogram and visual representation to predict a same-size binary spectrogram mask for the selected video.A U-Net-style encoder-decoder receives the spectrogram and fuses visual features into its middle layers to guide separation.
- Audio-Visual Separation Network: Self-attention cross-modal fusion weights visual features by their correlations with sound features before concatenation and residual MLP processing.The fused representation is computed at each time step, with attention applied across visual feature channels.
- Training and Inference: The model trains on binary dominant-source masks, thresholds predicted masks, applies them to complex STFT coefficients, and uses iSTFT to reconstruct waveforms.At test time, it localizes people and extracts a visual feature for each detected musician to isolate that musician’s portion of the mixed audio.
4. Experiments
Experiments evaluate Music Gesture on heterogeneous and homogeneous musical mixtures using quantitative metrics, human judgments, ablations, and visualizations. The method consistently improves heterogeneous separation and enables strong same-instrument duet separation, while remaining sensitive to viewpoint changes and occlusions.
- Experimental Setup: Experiments use MUSIC-21, URMP, and AtinPiano, with MUSIC evaluations including synthetic 2-mix and 3-mix mixtures of different instruments.MUSIC-21 is an untrimmed YouTube dataset; AtinPiano records piano performances from above the keyboard and hands.
- Hetero-musical Separation: Music Gesture outperforms previous methods by 1.8dB on 2-mix and 0.6dB on 3-mix separation under SDR.The method is reported to outperform all baselines consistently across separation metrics on MUSIC.
- Hetero-musical Separation: Human evaluators favor Music Gesture over the best baseline for both 2-mix and 3-mix heterogeneous sound separation.Each comparison was independently performed by three AMT workers, with results aggregated by majority vote.
- Homo-musical Separation: For same-instrument duets, Music Gesture consistently outperforms Sound of Motions across five instruments and both automatic and human evaluations.The advantage is especially pronounced for trumpet, piano, and flute, where the reported gap exceeds 3 dB.
- Visualizations: Attention visualizations show greater focus on hand keypoints for guitar and flute, but on elbows for cello and violin.These patterns indicate that the model uses different body regions for different instrument-separation cases.
- Limitations: The system is not resilient to camera viewpoint changes and musician body-part occlusions.The authors suggest unsupervised learning of keypoints from raw images as a direction for future work.
5. Conclusions and Future Work
The paper concludes that keypoint-based structured visual representations are powerful for visual sound separation and enable separation of sounds from the same instruments. It identifies more general audio-visual data with complex human-object interactions as future work.
- Keypoint-based structured visual representations are powerful for visual sound separation.
- The approach performs better than previous appearance- and low-level motion-based models for different-instrument separation.
- The method achieves remarkable results for separating piano, flute, and trumpet duets with the same instrument type, a task previously described as impossible.
- Future work will extend the approach to more general audio-visual data with complex human-object interactions.