Source-linked AI summary
Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, Michael Rubinstein
TL;DR
Audio-only speech separation struggles with mixed speech and does not associate separated signals with visible speakers. The paper combines visual face information with audio in a speaker-independent neural model trained using the AVSpeech dataset, and reports advantages over audio-only and prior speaker-dependent audio-visual methods in classic and real-world scenarios.
Problem
Audio-only separation is challenging and does not provide an easy association between separated audio sources and speakers visible in the video.
Method
A joint audio-visual neural network uses face information and sound mixtures, trained with roughly 4700 hours of clean web video clips in AVSpeech and synthetic cocktail-party mixtures.
Results
The model outperforms audio-only and audio-visual models on classic speech separation tasks and works in challenging natural scenes with overlapping speech and background noise.
Takeaways & Limitations
Speaker-independent audio-visual separation can generalize to unseen speakers and support enhanced audio streams for real-world video scenarios.
Takeaways & Limitations
The paper notes that classic speech enhancement with one speaker and non-speech noise produces results similar to its audio-only baseline.
Abstract
from arXiv · showhide
We present a joint audio-visual model for isolating a single speech signal from a mixture of sounds such as other speakers and background noise. Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video. In this paper, we present a deep network-based model that incorporates both visual and auditory signals to solve this task. The visual features are used to "focus" the audio on desired speakers in a scene and to improve the speech separation quality. To train our joint audio-visual model, we introduce AVSpeech, a new dataset comprised of thousands of hours of video segments from the Web. We demonstrate the applicability of our method to classic speech separation tasks, as well as real-world scenarios involving heated interviews, noisy bars, and screaming children, only requiring the user to specify the face of the person in the video whose speech they want to isolate. Our method shows clear advantage over state-of-the-art audio-only speech separation in cases of mixed speech. In addition, our model, which is speaker-independent (trained once, applicable to any speaker), produces better results than recent audio-visual speech separation methods that are speaker-dependent (require training a separate model for each speaker of interest).
1 INTRODUCTION
The paper addresses the difficulty of separating speech and associating separated audio with visible speakers by combining visual and auditory signals. It introduces a speaker-independent audio-visual model and AVSpeech dataset, showing advantages over audio-only separation in mixed speech and real-world mixtures.
- Audio-only separation is ill-posed and cannot easily associate separated sources with their corresponding speakers in video.
- The model uses detected face crops and the recorded sound mixture to split audio into streams for each detected speaker.Visual information focuses audio on desired speakers while improving source separation.
- AVSpeech contains roughly 4700 hours of clean video clips collected from 290,000 online lectures, talks, and how-to videos.The clips contain visible speakers and no interfering sounds; synthetic cocktail-party mixtures were generated for training.
- The approach outperforms a state-of-the-art audio-only method on pure speech mixtures and handles overlapping speech with background noise in real-world scenarios.
- The paper contributes a speaker-independent audio-visual model and a large-scale AVSpeech dataset for speech separation research.The model is intended to apply across speakers rather than requiring a separate model for each speaker.
2 RELATED WORK
Prior work includes audio-only and audio-visual separation methods, but existing audio-visual approaches are generally speaker-dependent and datasets often lack scale or clean speech. This paper positions its model and AVSpeech dataset as enabling speaker-independent separation with strong generalization and real-world applicability.
- Speech separation: Audio-only speech separation has been extensively studied, including denoising, deep clustering, and permutation-invariant training methods.
- Comparison with audio-only methods: The proposed approach jointly improves speech separation and assigns each separated speech signal to its corresponding face.
- Audio-visual speech separation: Existing audio-visual separation approaches are mainly speaker-dependent, requiring a dedicated model for each speaker.The paper identifies the lack of a sufficiently large and diverse dataset as a likely reason speaker-independent audio-visual separation had been less pursued.
- Generalization: The model separates and enhances previously unseen speakers, including speakers using languages absent from training data.
3 AVSPEECH DATASET
AVSpeech is a large-scale dataset of clean, single-speaker audio-visual clips created by automatically filtering online videos for visible speakers and low background interference.
- Dataset overview: AVSpeech contains roughly 4700 hours of 3–10-second video segments from approximately 150,000 distinct speakers.The clips span varied people, languages, and face poses, with one visible face and one corresponding audible speaker per clip.
- Dataset overview: The collection pipeline began with roughly 290,000 YouTube lectures, TED talks, and how-to videos, from which clean speaker clips were extracted.The training set later mixed these clean clips with other speech tracks and background noise to create synthetic cocktail parties.
- Video filtering: Face tracking retained segments with visible, sufficiently frontal speakers and rejected blurred, poorly illuminated, extreme-pose, or excessively incomplete face frames.A segment was discarded when more than 15% of its face frames were missing.
- Audio filtering: Speech refinement estimated each segment’s speech SNR and rejected segments below an empirically selected threshold using synthetic mixtures with known SNR.The estimator used a pretrained audio-only denoising network and compared estimated SNR with ground-truth SNR.
- Audio filtering: 17 dB was the filtering threshold, and a listening check found no noticeable background noise in a random sample of 100 clips that passed.Estimator accuracy diminished at higher SNR because the original noise became faint.
4 AUDIO-VISUAL SPEECH SEPARATION MODEL
The separation model combines face-derived visual streams with noisy audio to predict speaker-specific complex spectrogram masks and reconstruct isolated speech waveforms.
- Architecture: The multi-stream network takes detected face thumbnails and the soundtrack as inputs, then outputs one complex spectrogram mask for each detected speaker.Applying each mask to the noisy spectrogram and converting back to waveforms produces an isolated speech signal per speaker.
- Input representation: Visual streams extract pretrained face embeddings and temporal features, while the audio stream computes an STFT spectrogram and learns an audio representation.The visual features are temporally upsampled to match the audio spectrogram’s 100 Hz sampling rate.
- Mask output: The model uses multiplicative masks to represent time-frequency relationships between clean speech and background interference.The complex ideal ratio mask contains separately estimated real and imaginary components.
- Fusion and training: Audio and visual feature maps are concatenated, processed by a BLSTM and three fully connected layers, and trained with compressed-spectrogram squared error.The output mask has separate real and imaginary channels and is applied through complex multiplication.
- Multiple speakers: The model supports multiple visible speakers by using one visual stream per speaker, with shared convolutional weights across visual streams.A separate model is trained for each number of visible speakers.
- Implementation details: Implementation uses 16 kHz mono audio, 25 ms Hann windows, 10 ms hops, and FFT size 512, yielding 257 × 298 × 2 input scalars.Power-law compression uses p = 0.3, and visual inputs are resampled to 25 FPS.
5 EXPERIMENTS AND RESULTS
Experiments compare the audio-visual model with audio-only baselines across synthetic mixtures, real-world scenes, and additional analyses. The model improves mixed-speech separation, generalizes to unseen speakers, and remains partly robust to missing visual information, while low-SNR speech quality remains a limitation.
- Experimental setup: The audio-visual model was evaluated against audio-only and audio-visual methods using quantitative and qualitative tests.Experiments covered synthetic mixtures, speech enhancement, and real-world scenarios.
- Synthetic mixtures: 16 dB SDR was achieved for one-speaker-plus-noise enhancement by both the audio-visual and audio-only models.The paper attributes the lack of an audio-visual advantage to noise and speech frequencies being typically well separated.
- Synthetic mixtures: 1.3 dB SDR improvement over the audio-only baseline was obtained for two clean speakers with one visual stream, while two streams added 0.4 dB for 10.3 dB total improvement.The two-stream model processes both speakers jointly in a single forward pass.
- Synthetic mixtures: 10.6 dB total SDR improvement was reached for two speakers plus background noise using two visual streams, exceeding the audio-only baseline by 0.5 dB.The task mixes two speakers with AudioSet background noise and uses outputs for the speakers and background.
- Generalization: The model remained robust across gender combinations and generalized better than speaker-dependent methods despite never encountering their test speakers during training.The paper reports strong generalization on prior datasets and good performance on same-gender mixtures.
- Additional analysis and real-world scenes: Dropping two-thirds of visual embeddings reduced speech separation quality by only 0.8 dB on average, but low-SNR mixtures produced noticeably degraded speech quality.Heat maps showed the mouth contributed most, with eyes and cheeks also contributing; the bottleneck analysis found one scalar per timestep was only 0.5 dB below the full model.
6 CONCLUSION
The paper presents a speaker-independent audio-visual model for challenging multi-speaker mixtures and reports broader experimental applicability, while noting the need to validate separated speech carefully in non-occluded video.
- The proposed model performs single-channel, speaker-independent speech separation in challenging multi-speaker mixtures.
- The paper directs readers to supplementary validation that separated speech in non-occluded video is accurate.
- The model is evaluated with background noise and supports applications including video captioning and speech recognition.
- The authors conducted extensive experiments to analyze the model’s behavior and components.
A.1 SDR
This section defines signal-to-distortion ratio as a standard evaluation measure for speech separation, based on separating true signal content from several error sources.
- Signal-to-distortion ratio (SDR) evaluates blind audio source separation when the original source signals are available as ground truth.
- SDR decomposes each estimated source into a true source part plus interference, noise, and algorithmic-artifact errors.
- SDR is measured in dB and is commonly reported as a general score for speech separation algorithms.
- The authors found SDR to correlate well with the amount of noise remaining in separated audio.
A.2 ViSQOL
This section describes ViSQOL as an objective speech-quality metric based on spectrotemporal similarity between reference and degraded signals, aggregated into a 1–5 opinion score.
- ViSQOL models human speech-quality perception using spectrotemporal similarity between reference and degraded speech signals.
- NSIM compares reference and degraded spectrograms using their means and correlation coefficients.
- ViSQOL computes NSIM on corresponding spectrogram patches and aggregates the scores into a mean opinion score between 1 and 5.