Source-linked AI summary
Visual Speech Enhancement
Aviv Gabbay, Asaph Shamir, Shmuel Peleg
TL;DR
This paper addresses speech enhancement when noisy audio alone is insufficient to separate a visible speaker’s voice from background speech. It combines mouth video with noisy audio and trains on examples containing same-speaker background speech; it reports better performance than prior methods on two public benchmarks and demonstrates enhancement on Obama’s weekly addresses.
Problem
Audio-only speech separation can be limited when the background and target voices are difficult to distinguish, motivating use of visible speech cues.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The compact model processes 200 ms speech segments in an average of 36 ms on an NVIDIA Tesla M60 GPU, supporting its suitability for real-time applications.
Source: §7 · p. 4The method fails when either the audio or video input modality is missing because training used both modalities.
Source: §7 · p. 4
Abstract
from arXiv · showhide
When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.
1. Introduction
The paper proposes visual speech enhancement that uses a visible speaker’s mouth movements alongside noisy audio to isolate that speaker’s voice. It positions this approach among audio-only, visually derived, and audio-visual speech methods, and reports comparisons on two lipreading datasets plus a demonstration on Obama’s weekly addresses.
1. Introduction
- The proposed end-to-end audio-visual network uses visible mouth movements with noisy audio to separate a visible speaker’s voice from background noise.The setup assumes the target speaker’s face is shown in the video.Source: §1 · p. 1
- Training with synthetic background noise drawn from the target speaker’s voice encourages visual-feature use and generalizes to different noise types.This addresses cases where audio alone cannot separate the target from their own voice.Source: §1 · p. 1
- The model outperforms prior work on GRID and TCD-TIMIT and demonstrates speech enhancement on Obama’s weekly addresses.The two benchmark datasets were designed for audio-visual speech recognition and lip reading.Source: §1 · p. 1
1.1.1. Audio-only deep learning based speech enhancement
- Audio-only enhancement includes spectral restoration, Wiener filtering, statistical models, and deep networks, while audio-only separation can struggle with similar human voices.Lower performance is commonly observed for same-gender mixtures.Source: §1.1.1 · p. 1, §1.1 · p. 1
1.1.2. Visually-derived speech and sound generation
- Prior work also generates speech or sound from silent video, including speech from speaking faces and sound from people striking or scratching objects.Source: §1.1.2 · p. 1
1.1.3. Audio-visual multi-modal learning
- Audio-visual neural networks have been used for cross-modal feature learning, lip reading, lip sync, and robust speech recognition.Earlier audio-visual enhancement work used visual masks or neural networks to produce enhanced speech.Source: §1.1.3 · p. 1, §1.1.4 · p. 2
2. Neural Network Architecture
The model encodes mouth-centered video frames and a noisy-audio spectrogram in separate convolutional streams, combines their representations, and decodes an enhanced-speech spectrogram. Its training minimizes mean-square error between the output and target spectrograms.
2. Neural Network Architecture
- The end-to-end encoder-decoder takes mouth video and noisy-audio spectrograms, combines their encodings in a shared embedding, and decodes an enhanced-speech spectrogram.Source: §2 · p. 2
2.1. Video encoder
- The video encoder processes five consecutive grayscale 128×128 frames cropped around the mouth using six convolutional layers.Each layer is followed by batch normalization, Leaky-ReLU, max pooling, and dropout of 0.25.Source: §2.1 · p. 2
2.2. Audio encoder
- The audio encoder uses five convolutional layers on the log mel-scale spectrogram, with strided convolutions used to maintain temporal order.The input and output spectrograms have 80 frequency intervals and 20 temporal steps spanning 200 ms.Source: §2.2 · p. 2
2.3. Shared representation
- The video and audio encoders produce 2,048- and 3,200-value vectors, concatenated into a 5,248-value shared embedding and passed through three fully connected layers.The fully connected layers have sizes 1,312, 1,312, and 3,200.Source: §2.3 · p. 2
2.4. Audio decoder
- A five-layer transposed-convolution audio decoder mirrors the audio encoder and outputs a spectrogram the size of the input.Source: §2.4 · p. 2
2.5. Optimization
- Training minimizes mean-square error between output and target speech spectrograms using Adam, with the learning rate halved after validation error stagnates for five epochs.The initial learning rate is 5e−4.Source: §2.5 · p. 3
3. Multi-Modal Training
The training strategy adds examples in which the background noise is another sentence spoken by the target speaker. Because audio alone cannot separate overlapping sentences from the same speaker, these examples encourage the model to use visual features and support generalization across noise types.
3. Multi-Modal Training
- Earlier strategies included occasionally zeroing an input modality or adding an auxiliary video output to encourage use of visual features.Source: §3 · p. 3
- Training examples with same-speaker speech as background noise encourage visual-feature use and help the model generalize across noise types.The authors state that audio alone cannot separate two overlapping sentences spoken by the same person.Source: §3 · p. 3
4. Implementation Details
Video and audio are segmented into aligned 200 ms inputs: five mouth-centered frames and a spectrogram with 20 temporal steps. The enhanced spectrogram is reconstructed into waveform segments and concatenated, using the noisy input’s original phase.
4.1. Video pre-processing
- Video is resampled to 25 fps, divided into five-frame segments, and cropped to 128×128 mouth-centered windows using facial landmarks.The inputs are normalized using the training-data mean frame and standard deviation.Source: §4.1 · p. 3
4.2. Audio pre-processing
- Audio is resampled to 16 kHz and represented by 80-bin log mel spectrograms in 200 ms segments, aligned with five video frames.The STFT uses a 40 ms window and 160-sample hop; phase is retained for reconstruction.Source: §4.2 · p. 3
4.3. Audio post-processing
- Enhanced segments are concatenated, then the waveform is reconstructed using the pseudo-inverse mel filterbank and inverse STFT with the noisy input’s original phase.Source: §4.3 · p. 3
5. Datasets
The evaluation uses GRID and TCD-TIMIT, a Mandarin sentences corpus, and Obama weekly addresses, which vary in recording conditions and are not designed specifically for lipreading.
5.1. GRID Corpus and TCD-TIMIT
- GRID contains 1,000 sentences from 34 speakers, while TCD-TIMIT includes 60 volunteer speakers and three lipspeakers.TCD-TIMIT speakers recorded sentences from the TIMIT dataset.Source: §5.1 · p. 3
5.2. Mandarin Sentences Corpus
- The Mandarin corpus contains recordings of a native speaker saying 10-character sentences, with approximately 3–4 seconds per utterance.The speaker was recorded frontally in a quiet, well-lit room.Source: §5.2 · p. 3
5.3. Obama Weekly Addresses
- The Obama dataset comprises 300 weekly-address videos, each 2–3 minutes long, with varied scale, background, lighting, face angle, and audio conditions.It also has an unbounded vocabulary.Source: §5.3 · p. 3
6. Experiments
The experiments compare audio-visual enhancement with prior work and audio-only baselines across several tasks. The model outperforms prior methods on GRID and TCD-TIMIT and improves SNR and PESQ on the Obama dataset.
6. Experiments
- Experiments mix target speech with LibriSpeech background speech, ambient noise, or the target speaker’s own speech; most training and testing mixtures use 0 dB SNR.SNR measures noise reduction and PESQ assesses speech-quality improvement.Source: §6 · p. 4
6.1. Baselines and previous work
- The audio-only baseline performs consistently worse, especially on self mixtures, where it yields no speech-quality improvement; without self mixtures in training, the visual model cannot separate same-voice speech.Table 2 presents the detailed results.Source: §6.1 · p. 4
- The model outperforms Vid2speech and Gabbay et al. on GRID and TCD-TIMIT, and significantly improves SNR and PESQ on the Obama dataset.Results were less convincing on TCD-TIMIT; the authors suggest its 20 minutes of clean training speech, versus 40–60 minutes in other experiments, as one possible explanation.Source: §6.1 · p. 4
7. Concluding remarks
The paper presents an audio-visual neural network and a training strategy using same-speaker overlapping speech as noise. It reports improvements on public benchmarks and demonstrates enhancement on a general dataset, while requiring both audio and video inputs.
7. Concluding remarks
- Training with overlapping sentences spoken by the target speaker builds robustness to similar vocal characteristics and supports effective use of visual information.The paper describes an end-to-end model that separates a visible speaker’s voice from background noise.Source: §7 · p. 4
- The model improves noisy-speech quality and intelligibility, outperforms previous methods on two public benchmarks, and is demonstrated on a dataset not designed for lipreading.The model is compact and processes short speech segments, which the authors describe as suitable for real-time applications.Source: §7 · p. 4
- The method fails when either the audio or video input is missing, because training used both modalities.Source: §7 · p. 4