Source-linked AI summary
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Ruohan Gao, Kristen Grauman
TL;DR
The paper asks how to isolate speech associated with a visible face amid background or overlapping sounds, beyond relying only on lip-speech alignment. VisualVoice jointly learns separation and cross-modal speaker embeddings from unlabeled video, combining lip motion with face-appearance priors. It reports state-of-the-art results on five benchmark datasets and generalizes to challenging real-world videos.
Problem
Speech separation must isolate a target speaker despite background noise or overlapping speakers, while lip-motion cues can fail under occlusion or head turns.
Method
VisualVoice jointly learns audio-visual speech separation and cross-modal speaker embeddings, using facial appearance, lip motion, and vocal audio without identity labels or speaker enrollment.
Results
VisualVoice achieves state-of-the-art audio-visual speech separation and enhancement results on five benchmark datasets and generalizes well to challenging real-world videos.
Takeaways & Limitations
Combining lip motion with cross-modal facial attributes provides complementary cues for speech separation, and the learned embedding also improves unsupervised cross-modal speaker verification.
Takeaways & Limitations
Lip-motion-based separation can degrade when the mouth is occluded or the speaker turns away, and real-world videos lack ground truth for quantitative evaluation.
Abstract
from arXiv · showhide
We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers. Whereas existing methods focus on learning the alignment between the speaker's lip movements and the sounds they generate, we propose to leverage the speaker's face appearance as an additional prior to isolate the corresponding vocal qualities they are likely to produce. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement, and generalizes well to challenging real-world videos of diverse scenarios. Our video results and code: http://vision.cs.utexas.edu/projects/VisualVoice/.
1. Introduction
VisualVoice addresses speech separation in overlapping, noisy environments by combining lip-motion cues with face-voice attributes. Its multi-task framework jointly learns separation and cross-modal embeddings from unlabeled video, achieving state-of-the-art results across five benchmark datasets.
- Problem and motivation: Lip movements provide audio-visual consistency cues for masking the target speech, but become unreliable when the mouth is occluded or the speaker turns away.These conditions motivate adding a complementary face-appearance prior.
- Core insight: Face appearance supplies priors about vocal characteristics such as tone, pitch, timbre, and articulation that can guide separation.The paper associates visible attributes including gender, age, nationality, and body weight with likely sound qualities.
- Core insight: Cross-modal matching and speech separation are mutually beneficial: cleaner separation improves voice-face embeddings, while better embeddings provide more distinctive separation priors.This feedback motivates jointly learning both tasks rather than treating them independently.
- Problem and motivation: VisualVoice separates a target speaker’s speech from background noise or overlapping speakers using video of the speaker talking.The task is motivated by speech’s frequent occurrence amid competing sounds and by visual information’s role in reducing auditory ambiguity.
- Method: The framework jointly learns audio-visual speech separation and cross-modal speaker embeddings without identity labels or speaker enrollment.Training and testing can use fully unlabeled video.
- Results: Experiments on 5 benchmark datasets report state-of-the-art audio-visual speech separation and enhancement results, while the learned embedding also improves unsupervised cross-modal speaker verification.The framework combines facial motion with cross-modal face-voice attributes as complementary cues.
2. Related Work
Prior work spans audio-only separation, audio-visual source separation, and face-voice cross-modal learning across several tasks. VisualVoice differs by combining lip motion and face-voice attributes in a multi-task framework for speech separation from unlabeled video.
- Audio-only speech separation: Audio-only speech separation includes microphone-based, monaural, and recent deep-learning approaches, whereas VisualVoice uses visual information to guide single-channel separation.The target signal is speech separated from non-speech noise or interfering speakers.
- Audio-visual source separation: Audio-visual source-separation research has used mutual information, independent component analysis, nonnegative matrix factorization, and deep learning for instruments, speech, and other sounds.The cited literature covers both conventional and modern approaches.
- VisualVoice: VisualVoice incorporates both lip motion and cross-modal face-voice attributes, unlike prior audio-visual methods described here.Its multi-task framework jointly learns audiovisual speech separation and cross-modal speaker embeddings.
- Cross-modal faces and voices: Face-voice cross-modal learning has addressed face reconstruction, talking-face generation, emotion recognition, diarization, speech recognition, and speaker identification, but not audio-visual speech separation.VisualVoice applies the face-voice link to separation rather than those established tasks.
- Broader audio-visual learning: Other audio-visual learning work includes representation learning, sound localization, sound generation, and action recognition; VisualVoice instead uses facial cues for speech separation.This positions the paper relative to broader audio-visual learning applications.
3. Approach
VisualVoice separates each speaker’s speech from mixed audio by combining vocal audio with lip-motion and facial-attribute cues. It jointly trains separation and cross-modal face-voice embeddings without identity labels.
- Problem formulation: The task separates each speaker’s sound from a single-channel mixture by leveraging visual cues from accompanying video.The framework also supports mixtures containing non-speech sounds for speech enhancement evaluation.
- Training data: Training examples mix speech segments from in-the-wild videos with automatically extracted face tracks, without identity labels.The same speaker can contribute multiple segments, while mixtures are formed from speech associated with different face tracks.
- Speech separation: The separator predicts complex ideal ratio masks from mixture spectrograms and reconstructs separated speech using complex masking followed by ISTFT.Masks are predicted separately for each target speaker and mixture.
- Visual cues: The visual stream combines temporally analyzed lip motion with facial attributes replicated across time to guide separation.Facial attributes encode expected voice properties, while lip motion isolates articulated speech specific to the segment.
- Network architecture: A U-Net-style audio network concatenates mixture-spectrogram features with visual features before decoding the audio-visual representation.The decoder uses frequency upsampling and produces the output mask representation.
- Multi-task learning: Multi-task learning jointly optimizes mask prediction, cross-modal face-voice matching, and speaker consistency to learn separation and embeddings together.The mask loss supplies supervision from ground-truth complex masks, while the cross-modal framework links separated speech embeddings with corresponding face embeddings.
4. Experiments
Experiments evaluate VisualVoice across benchmark datasets for speech separation, enhancement, and cross-modal verification, including reliable and corrupted lip-motion conditions. The method improves prior results while combining lip motion with face-voice embeddings to remain less vulnerable when lip cues are unreliable.
- Datasets: Experiments cover audio-visual speech separation, speech enhancement, and cross-modal speaker verification across 6 benchmark datasets.The datasets include VoxCeleb2, Mandarin, TCD-TIMIT, CUAVE, LRS2, and VoxCeleb1.
- Experimental settings: VoxCeleb2 separation tests compare reliable lip motion with sequences corrupted by temporal shifts and lip-region occlusions.The unreliable condition shifts mouth ROI sequences by up to 1s and occludes the lip region for up to 1s per segment during training and testing.
- Experimental settings: Speech enhancement uses mixtures containing AudioSet background sounds, while only the target speaker’s visual feature guides separation.Background interference includes laughter, car engines, wind, and other non-speech sounds.
- Quantitative results: In both reliable and unreliable scenarios, VisualVoice achieves the best separation results and outperforms AV-Conv by a good margin.The audio-only baseline has decent performance but cannot assign separated speech to the corresponding speaker.
- Ablations and qualitative results: VisualVoice combines complementary lip-motion and face-voice cues, making separation less vulnerable to unreliable lip motion than the lip-motion-only model.Lip-motion-only performance drops significantly when lip motion is unreliable, whereas static-face separation is possible but can be difficult when facial attributes are insufficiently distinctive.
- Comparison to prior work: The approach improves state-of-the-art results on all five datasets in the cross-method comparison and significantly improves FaceFilter by 4.68 SDR.The cross-dataset tests use a speaker-independent model trained on VoxCeleb2, including speakers never seen during training.
- Learned cross-modal embeddings: Cross-modal embeddings cluster speakers with shared gender and identity attributes despite using no identity or attribute labels during training.Joint training with speech separation produces large gains in cross-modal verification and more reliable face-voice embeddings.
5. Conclusion
VisualVoice jointly learns cross-modal speaker embeddings and audio-visual speech separation, using complementary lip-motion and facial-attribute cues. It achieves state-of-the-art separation results and generalizes to challenging real-world videos.
- VisualVoice simultaneously learns cross-modal speaker embeddings and speech separation in a multi-task framework.
- The approach exploits complementary cues from lip motion and cross-modal facial attributes.
- VisualVoice achieves state-of-the-art results on audio-visual speech separation and generalizes well to challenging real-world videos.
- The cross-modal matching and speaker consistency losses may also support speaker identification and sound source localization.
- Future work will model fine-grained cross-modal face–voice attributes to further enhance speech separation.