Source-linked AI summary
The Sound of Pixels
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba
TL;DR
Manual supervision is costly for jointly recognizing, localizing, and separating sounds in visual scenes. PixelPlayer learns these capabilities from unlabeled videos by combining audio-visual analysis with Mix-and-Separate training. The system separates and localizes sound components, with qualitative and user-study evidence supporting visual grounding and perceptual separation.
Problem
Recognizing, localizing, and separating sounds through audio-visual learning typically requires substantial supervision, while natural audio-visual synchronization offers an unlabeled supervisory signal.
Method
PixelPlayer uses cross-modal video and audio networks with an audio synthesizer, trained by mixing sounds from unlabeled videos and recovering sources conditioned on their visual inputs.
Results
Quantitative, qualitative, and subjective evaluations demonstrate the effectiveness of PixelPlayer for audio separation and visual-sound correspondence.
Takeaways & Limitations
PixelPlayer can ground separated sound components in visual inputs, supporting applications such as independently adjusting the volume of different sound sources.
Takeaways & Limitations
NSDR, SIR, and SAR do not fully capture perceptual separation quality, so the paper supplements them with user studies.
Abstract
from arXiv · showhide
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel. Our approach capitalizes on the natural synchronization of the visual and audio modalities to learn models that jointly parse sounds and images, without requiring additional manual supervision. Experimental results on a newly collected MUSIC dataset show that our proposed Mix-and-Separate framework outperforms several baselines on source separation. Qualitative results suggest our model learns to ground sounds in vision, enabling applications such as independently adjusting the volume of sound sources.
1 Introduction
PixelPlayer uses naturally synchronized audio and vision in unlabeled videos to localize sound-producing regions and separate audio into spatially grounded components. The system supports recognizing and manipulating sounds associated with individual objects.
- Natural audio-visual synchronization provides supervision for grounding sounds in vision without manual annotations.
- PixelPlayer jointly separates accompanying audio into components and spatially localizes those components in video frames.
- In instrument videos, each estimated component corresponds to a spatial location and can represent an individual instrument or silence.
- The system estimates each pixel’s sound energy and distinguishes sounding instruments from the background in example videos.
- Separated and localized sources could support isolated auditory processing and video editing, including object-specific volume adjustment or audio removal.
2 Related Work
PixelPlayer connects sound source separation with visual and spatial grounding, building on cross-modal and self-supervised learning rather than treating separation as an audio-only problem.
- Sound source separation: Classical and deep learning approaches address sound source separation, whereas PixelPlayer separates sounds into visually and spatially grounded components.
- Learning visual-audio correspondence: Prior cross-modal work uses sound and vision to generate sounds, learn representations, and localize sounds, but the cited approaches address related subproblems.
- Self-supervised learning: Self-supervised methods derive supervisory signals from natural contextual structure instead of manual annotations.
- Learning visual-audio correspondence: Pixel-level sound generation combines visual features, audio components, masking, and inverse STFT to produce a sound for each pixel.
- Self-supervised learning: This work applies self-supervision to separate and ground sound in vision.
3 Audio-Visual Source Separation and Localization
PixelPlayer combines video and audio analysis with a synthesizer, then trains through Mix-and-Separate to recover source sounds conditioned on corresponding visual inputs. The framework uses spectrogram masks as its learning targets.
- Model architecture: PixelPlayer comprises video analysis, audio analysis, and audio synthesizer networks.
- Model architecture: The video network extracts pixel-level visual features from temporally pooled video-frame representations.
- Model architecture: The audio network uses a U-Net to split the input sound into K components represented in the time-frequency domain.
- Model architecture: The synthesizer combines pixel-level visual and audio features to estimate a mask applied to the input spectrogram, then reconstructs a waveform using the input phase.
- Mix-and-Separate training: Mix-and-Separate creates complex auditory scenes by adding audio from randomly sampled videos and trains the model to recover each source from its mixture and corresponding visual input.
- Mix-and-Separate training: Training remains unsupervised because it uses no data labels or assumptions about the sampled videos, despite having targets generated from the mixtures.
- Mask targets: The framework learns binary or ratio spectrogram masks, using dominant-component decisions for binary masks and magnitude ratios for ratio masks.
4 MUSIC Dataset
The MUSIC dataset provides unlabeled musical videos for audio-visual source separation, covering solos and duets across 11 instrument categories. Its naturally collected duet performances are less balanced than solo performances.
- Dataset construction: MUSIC is a dataset of 685 untrimmed musical solo and duet videos collected for audio-visual source separation.The videos were retrieved from YouTube using keyword queries.
- Dataset composition: Example videos pair visual frames with log-frequency spectrograms for solo and duet performances.Figure 4 places solos in the top row and duets in the bottom row.
- Dataset composition: The dataset covers 11 instrument categories, including accordion, acoustic guitar, cello, clarinet, erhu, flute, saxophone, trumpet, tuba, violin, and xylophone.
- Dataset limitations: Duet performances are less balanced than solo performances, with almost no tuba or violin duets but many guitar-and-violin duets.
5 Experiments
The experiments use convolutional audio and video analysis networks linked by a lightweight synthesizer that produces spectrogram masks. The model is configured for short visual inputs and multiple output feature channels.
- Audio representation: The audio pipeline uses log-frequency representations because they resemble auditory frequency decomposition and provide pitch translation invariance for harmonic sounds.
- Model configurations: The video analysis network uses a dilated ResNet-18 variant that produces K-dimensional visual features for each pixel.It removes final classification layers, dilates the last residual block, and adds a 3 × 3 convolution with K output channels.
- Model configurations: The audio analysis network is a U-Net with seven down-convolutions and seven up-convolutions, producing K feature maps from an input spectrogram.
- Model configurations: The audio synthesizer fuses video and audio features by weighted summation and outputs a spectrogram mask.Its linear layer has K weights and one bias, supporting interpretable intermediate representations.
- Model configurations: The best model uses 3 visual frames as input and sets the number of feature channels to K = 16.
5.3 Implementation details
The evaluation combines natural-video dataset splits with synthetic mixtures and measures separation using NSDR, SIR, and SAR. Qualitative results show successful component segmentation, while the authors note perceptual-quality limitations of these metrics.
- Data splits: The MUSIC dataset is split into 500 training videos, 130 validation videos, and 84 test videos, with solos used for validation and duets for testing.
- Training mixtures: Training samples are formed from two randomly selected videos, and mixtures may contain zero to four instruments.Silent background videos are added to regularize sound localization.
- Quantitative results: Binary masking in log-frequency scale performs best in most metrics, while masking generally outperforms direct spectrogram regression.The comparison includes NMF and DeepConvSep baselines using audio and ground-truth labels.
- Qualitative results: Qualitative results show that predicted binary masks can segment target instrument components from complex mixed spectrograms.
- Evaluation: The model is evaluated on synthetic validation mixtures using NSDR, SIR, and SAR, with Mix-and-Separate used to construct the mixtures.NSDR measures improvement over using the mixture as the separated source.
- Evaluation limitations: NSDR, SIR, and SAR are not considered the best metrics for perceptual separation quality, motivating complementary user studies.
5.5 Visual Grounding of Sounds
The paper frames PixelPlayer as addressing both sound localization and sound separation.
- PixelPlayer is designed to solve both localization and separation of sounds.
Sound localization.
PixelPlayer localizes sounding instruments in image space and associates different image regions with distinct sound features. Its channel activations also support object localization and visual and auditory instrument discrimination.
- Sound localization: The model accurately localizes sounding instruments by distributing each pixel’s sound energy as heatmaps.Fig. 7 addresses which pixels are making sounds by visualizing per-pixel sound volume.
- Sound clustering: Different instruments and background regions receive different color embeddings from their pixel-level sound features.Vectorized log spectrogram magnitudes are projected into 3D RGB space with PCA for visualization.
- Channel-based discrimination: 46.2% visual accuracy and 68.9% audio accuracy result when categories are assigned directly to the strongest activated channel.The evaluation uses no learning; the authors expect higher performance with a linear classifier.
- Object localization: Object localization is evaluated by thresholding channel activation maps and measuring bounding-box accuracy under different IoU criteria.The bounding box covers the largest connected component above 20% of the activation map’s maximum.
5.6 Visual-audio corresponding activations
PixelPlayer learns corresponding visual and audio channel activations that detect instrument-specific patterns across modalities. Some channels specialize in instruments, while others detect instrument features.
- 5.6 Visual-audio corresponding activations: Channels 6, 11, and 14 emerge as violin, guitar, and xylophone detectors in both visual and audio domains.Their activations correspond respectively to violin harmonics, guitar low frequencies, and xylophone pulse-like spectrogram patterns.
- 5.6 Visual-audio corresponding activations: The learned channels respond jointly to object appearance and characteristic sound patterns.Channel 6 responds to violin appearance and higher-order harmonics, channel 11 to guitars and low frequencies, and channel 14 to xylophone appearance and brief pulses.
6 Subjective Evaluations
Subjective studies complement objective evaluations of PixelPlayer’s separation and visual-sound correspondence. The binary masking model performs best among the evaluated models, while objective tests mainly use synthetic mixtures and standard metrics that do not closely track perceptual quality.
- Evaluation scope and caveats: Objective and quantitative evaluations mainly use synthetic mixture videos, leaving performance on natural videos for further investigation.The authors also note that NSDR, SIR, and SAR are not closely related to perceptual quality, motivating the subjective studies.
- 6.1 Sound separation quality: Human raters evaluate separation using 256 outputs per model, with each audio sample assessed by three independent workers.The study compares NMF, spectral regression, ratio masking, and binary masking using solo-instrument ground truth.
- 6.1 Sound separation quality: Binary masking achieves the best subjective sound separation performance among the evaluated models.It has the highest correct rate, lowest error rate, and lowest confusion rate in the human study.
- 6.2 Visual-sound correspondence evaluations: The binary masking model also gives the best performance on the visual-sound correspondence evaluation.Workers judge whether sounds generated for selected instrument and background pixels come from those pixels.
7 Conclusions
The paper introduces PixelPlayer, which learns from unlabeled videos to separate input sounds and locate them in visual inputs. Quantitative, qualitative, and subjective evaluations demonstrate the effectiveness of its cross-modal learning system.
- 7 Conclusions: PixelPlayer learns from unlabeled videos to separate input sounds and locate them in the visual input.The system uses both visual and auditory signals without requiring manual labels for these tasks.
- 7 Conclusions: Quantitative results, qualitative results, and subjective user studies demonstrate the effectiveness of the cross-modal learning system.The authors expect this work to open research avenues for sound source separation using visual and auditory signals.