Source-linked AI summary
The Conversation: Deep Audio-Visual Speech Enhancement
Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman
TL;DR
The paper addresses isolating individual speakers from simultaneous speech in video, where prior approaches focused on known speakers and controlled environments. It proposes a lip-conditioned audio-visual enhancement network that predicts target magnitude and phase, achieving strong results with up to five voices in unconstrained settings, including unseen speakers and languages.
Problem
Existing multi-talker speech-separation work focused largely on known speakers and controlled environments, while noisy conditions also degrade speech recognition.
Method
The network uses the target speaker’s lip regions with noisy audio to predict a mask for magnitude filtering and a correction for phase enhancement.
Results
The model demonstrates strong qualitative and quantitative performance for up to five simultaneous voices in unconstrained environments and on speakers and languages unseen during training.
Takeaways & Limitations
The method produces realistic, intelligible speech from very noisy audio in unconstrained real-world environments.
Takeaways & Limitations
Performance is very sensitive to temporal alignment between the voice and video, and the alignment method can fail under extreme noise.
Abstract
from arXiv · showhide
Our goal is to isolate individual speakers from multi-talker simultaneous speech in videos. Existing works in this area have focussed on trying to separate utterances from known speakers in controlled environments. In this paper, we propose a deep audio-visual speech enhancement network that is able to separate a speaker's voice given lip regions in the corresponding video, by predicting both the magnitude and the phase of the target signal. The method is applicable to speakers unheard and unseen during training, and for unconstrained environments. We demonstrate strong quantitative and qualitative results, isolating extremely challenging real-world examples.
1. Introduction
The paper introduces an audio-visual network that isolates a target speaker using lip motion, correcting both magnitude and phase to handle noisy, multi-speaker speech. It targets unconstrained settings, including speakers and languages unseen during training, and reports strong qualitative and quantitative performance.
- Contribution: The proposed network isolates a target speaker’s voice from other speakers and background noise using visual information from the target speaker’s lips.It produces an enhanced audio signal containing only the target speaker’s voice.
- Method: The model predicts corrections for both spectrogram magnitude and phase instead of relying on noisy phase during reconstruction.The authors note that noisy phase becomes a poor approximation as signal-to-noise ratio decreases.
- Applications: The method addresses noisy-speech recognition applications such as ASR for mobile phones in crowded environments and automatic captioning for online videos.The paper motivates enhancement partly because recognition performance deteriorates in noisy environments.
- Evaluation: The model is evaluated with up to five simultaneous voices in unconstrained in-the-wild environments, including speakers and languages unseen during training.The authors report strong qualitative and quantitative performance under these conditions.
- Related work: Unlike audio-only approaches, the visual stream helps separate voices while assigning them to the corresponding speakers.The comparison is based on using visual information from the target speaker.
- Related work: Relative to prior deep audio-visual systems, the method uses temporal-signal spectrogram processing, soft-mask filtering, phase enhancement, and evaluation on unheard and unseen speakers.The paper lists these as distinguishing features from earlier works.
2. Architecture
The network combines visual lip features with audio spectrograms in temporal convolutional architectures, using separate magnitude and phase subnetworks to enhance the target speaker. It predicts a soft magnitude mask and refines noisy phase through a residual prediction conditioned on both modalities.
- 2.1. Video representation: Visual input is encoded by a pretrained spatio-temporal residual network into compact 512-dimensional features for each video frame.The visual encoder uses a 3D convolution followed by an 18-layer ResNet and operates on grayscale, appropriately scaled pre-cropped faces.
- 2.2. Audio representation: Audio is represented with Hann-window STFT magnitude and phase spectrograms, aligning each video frame with four spectrogram slices.The representation uses a 10 ms hop, 40 ms window, 16 kHz sampling, and 321 frequency bins from 0 to 8 kHz; magnitudes are converted to 80-bin melscale inputs before filtering on the linear spectrogram.
- 2.3. Audio-visual fusion: Temporal convolutional stacks process the visual and audio streams, treating spectrogram frequencies as channels and using stride-2 blocks for temporal down-sampling.The visual stream uses 10 convolutional blocks, while the audio stream uses 5; their fused representation is processed by 15 additional temporal convolutional blocks.
- 2.3. Magnitude sub-network: The magnitude subnetwork outputs a sigmoid soft mask that is multiplied element-wise with the noisy magnitude spectrogram to produce enhanced magnitudes.Two transposed convolutions up-sample the temporal dimension by a factor of 4 so the mask matches the input magnitude resolution.
- 2.4. Phase sub-network: The phase subnetwork uses noisy phase and predicted magnitudes to estimate a residual, adds it to the noisy phase, and L2-normalizes the result.The phase pathway is designed to refine rather than synthesize clean phase from scratch, and training initializes residuals near zero so noisy phase initially propagates to the output.
- 2.5. Loss function: Training combines an L1 magnitude loss with a ground-truth-magnitude-scaled cosine-similarity objective for phase prediction.The overall objective jointly trains the magnitude and phase subnetworks.
3. Experiments
Experiments evaluate audio-visual speech enhancement on synchronized LRS2 and VoxCeleb2 data, including mixtures of two to five speakers and tests with unseen speakers and languages. The model is assessed using perceptual, separation, intelligibility, and ASR metrics, showing effective cross-talk suppression but remaining phase and target-quality challenges.
- 3.1. Datasets: LRS2 and VoxCeleb2 provide complementary evaluation settings, with transcription-based intelligibility testing and speaker-disjoint testing for speaker-independent performance.LRS2 contains BBC speech with transcriptions, while VoxCeleb2 contains over 6,000 speakers but lacks text transcriptions.
- 3.2. Mixtures: Mixtures contain 2–5 simultaneous speakers, with equal-loudness speech interference posing a harder separation problem than background babble noise.The experiments add one to four interference speakers to a clean target signal.
- 3.3. Evaluation protocol: Evaluation combines SIR, SAR, SDR, PESQ, STOI, and ASR word error rate to measure suppression, artefacts, distortion, perceptual quality, intelligibility, and recognition.PESQ is reported as the metric most correlated with listening tests accounting for phase distortion.
- 3.4. Training: The training procedure uses visual pre-training, cached audio-visual features, curriculum learning over increasing speaker counts, separate magnitude and phase training, and end-to-end fine-tuning.The phase-loss weighting is fixed at λ = 1 rather than experimentally tuned.
- 3.5. Results: The model strongly suppresses cross-talk, but adding speakers increases distortion and creates a trade-off in target-voice quality; predicted phase improves on mixed phase yet remains below ground-truth phase.The authors report noticeable reduction of robotic off-sync harmonics, while the remaining ground-truth-phase gap indicates room for phase-network improvement.
- 3.5. Results: ASR results show highly intelligible enhanced speech from noisy audio that is incomprehensible to humans or an ASR system, and VoxCeleb2 results confirm generalization to unseen speakers.The LRS2-trained model also generalizes to languages not seen during training, although VoxCeleb2 performance is slightly worse and is attributed to unfine-tuned visual features and additional background noise.
4. Conclusion
The paper separates a target speaker’s speech from background noise and other speakers using lip movements, predicting both signal phase and magnitude. It produces intelligible speech from very noisy, unconstrained real-world recordings.
- The method separates a target speaker’s speech from background noise and other speakers using visual information from the speaker’s lips.
- The deep network predicts both the phase and magnitude of the target signal to produce realistic speech segments.
- The network generates intelligible speech from very noisy audio segments recorded in unconstrained ‘in the wild’ environments.