Source-linked AI summary
Seeing Through Noise: Visually Driven Speaker Separation and Enhancement
Aviv Gabbay, Ariel Ephrat, Tavi Halperin, Shmuel Peleg
TL;DR
Audio-only separation has difficulty distinguishing similar human voices. The paper uses speaker-specific visual speech predictions to guide separation or enhancement of noisy audio. It reports improved speech over raw predictions on complex TCD-TIMIT and identifies visual information as useful for same-gender separation.
Problem
Audio-only approaches perform poorly when separating similar human voices, such as same-gender mixtures.
Source: §1 · p. 1Method
The method uses a known speaker’s video-to-speech prediction as prior information to mask a noisy mixture, using a frequency-threshold approach for single-speaker enhancement.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2Results
On complex TCD-TIMIT, the separation methods produce final speech that sounds much better than raw vid2speech predictions.
Source: §4.3 · p. 3Takeaways & Limitations
Visual information provides disambiguating power for similar speech vocal characteristics, including same-gender mixtures.
Source: §5 · p. 4The method assumes a previously known speaker, and the authors identify less-constrained real-world multi-speaker noisy recordings as future work.
Source: §3.2 · p. 2, §5 · p. 4
Abstract
from arXiv · showhide
Isolating the voice of a specific person while filtering out other voices or background noises is challenging when video is shot in noisy environments. We propose audio-visual methods to isolate the voice of a single speaker and eliminate unrelated sounds. First, face motions captured in the video are used to estimate the speaker's voice, by passing the silent video frames through a video-to-speech neural network-based model. Then the speech predictions are applied as a filter on the noisy input audio. This approach avoids using mixtures of sounds in the learning process, as the number of such possible mixtures is huge, and would inevitably bias the trained model. We evaluate our method on two audio-visual datasets, GRID and TCD-TIMIT, and show that our method attains significant SDR and PESQ improvements over the raw video-to-speech predictions, and a well-known audio-only method.
1. INTRODUCTION
Audio-only separation struggles with similar human voices, including same-gender mixtures. This work uses visual speech information to separate speakers and enhance speech, building on audio-visual speech-processing research.
1. INTRODUCTION
- Audio-only methods perform poorly on similar human voices, such as same-gender mixtures.The paper contrasts this difficulty with its use of visual information to distinguish speakers.Source: §1 · p. 1
- The approach is speaker-dependent and noise-invariant, allowing less training data while retaining good results even for overlapping voices from the same person.The authors contrast it with methods trained on mixtures of speech and noise or two voices.Source: §1 · p. 1
1.1. Related work
- Related work spans audio-only masking and deep-learning separation, alongside visual-feature masks and CNN-based audio-visual enhancement [20].Audio-visual speech research also includes lipreading, sound prediction, and unsupervised sound representations [13].Source: §1.1 · p. 1
2. VISUALLY-DERIVED SPEECH GENERATION
The paper uses vid2speech to generate speech predictions from silent video of a known speaker. The model maps video and optical flow through visual-feature and spectrogram-prediction stages.
2.1. Vid2speech
- Vid2speech generates speech spectrogram predictions from silent video frames of a speaking person.The paper uses this speaker-dependent method, which requires a dedicated model for each speaker.Source: §2.1 · p. 1, §2 · p. 1
- The model takes consecutive video frames and dense optical-flow fields between them as inputs.A dual-tower ResNet encodes visual features, followed by fully connected layers that predict mel-scale spectrograms.Source: §2.1 · p. 2
- A post-processing network aggregates consecutive predictions and maps them to a linear-scale spectrogram for the final speech prediction.Source: §2.1 · p. 2
3. AUDIO-VISUAL SPEECH SEPARATION
The method uses video-derived speech estimates as prior knowledge to assign mixture spectrogram components to sources. It applies this principle to two-speaker separation and to single-speaker enhancement using frequency-specific thresholds.
3. AUDIO-VISUAL SPEECH SEPARATION
- The method assigns time-frequency elements of a mixture spectrogram to sources, then reconstructs estimated individual signals.Video-to-speech estimates provide prior knowledge because they are imperfect predictions.Source: §3 · p. 2
3.1. Speech separation of two speakers
- For two known speakers, separate video-to-speech networks predict each speaker’s spectrogram, and a binary mask assigns each time-frequency bin to the speaker with the larger prediction.The separated spectrogram is formed by multiplying the mixture by the speaker mask; reconstruction preserves the original phase.Source: §3.1 · p. 2
- A ratio mask offers a continuous alternative to winner-takes-all assignment, giving each time-frequency bin a value between 0 and 1.The example thresholding function sets each frequency’s threshold to the 75th percentile of its training magnitudes.Source: §3.1 · p. 2, Figure 1 · p. 2
3.2. Speech enhancement of a single speaker
- For single-speaker enhancement, training-data Long-Term Speech Spectra set frequency-specific thresholds, and a mask filters the noisy spectrogram before speech reconstruction.This setting assumes a known speaker and uses one video-to-speech prediction; the background may include other, unseen speakers.Source: §3.2 · pp. 2–3
4. EXPERIMENTS
Experiments on GRID and TCD-TIMIT evaluate separation and enhancement using synthesized mixtures, objective scores, and qualitative listening. The results include comparisons with raw vid2speech predictions and an audio-only method, plus tests of enhancement and transfer learning.
4.1. Datasets; 4.2. Performance evaluation; 4.3. Results
- GRID and TCD-TIMIT experiments use synthesized same-gender speech mixtures with unseen signals; evaluation includes SDR, SIR, SAR, PESQ, and informal listening.GRID includes mixtures of two male speakers; TCD-TIMIT includes female speakers, including a speaker mixed with her own voice.Source: §4.1 · p. 3, §4.2 · p. 3, §4.3 · p. 3
4.3. Results
- On TCD-TIMIT, separation methods have real impact because raw vid2speech predictions are low-quality and mostly unintelligible; final signals sound much better.On GRID’s constrained-vocabulary data, raw predictions have reasonable PESQ quality; ground-truth spectra provide a performance ceiling.Source: §4.3 · p. 3
- Enhancement is also evaluated qualitatively on mixtures of speech and non-speech background noise.Table 2 reports synthesized-mixture enhancement experiments on GRID and TCD-TIMIT, using ground-truth spectra as an oracle upper bound.Source: §4.3 · pp. 3–4
- Unknown-speaker separation tests fine-tune a model trained on 40 minutes from known speaker S2 with 5 minutes from each unknown speaker before separating unseen sentences.The unknown GRID speakers are S3 and S5.Source: §4.3 · p. 4
5. CONCLUDING REMARKS
The paper concludes that visual information supports high-quality single-channel speech separation and enhancement, with visual cues addressing similar vocal characteristics in same-gender separation. It identifies less constrained recordings and end-to-end enhancement as future research directions.
5. CONCLUDING REMARKS
- Visual information supports high-quality single-channel separation and enhancement, and its disambiguating power addresses similar vocal characteristics in same-gender separation.The conclusion contrasts this with audio-only techniques.Source: §5 · p. 4
- The evaluation includes LTSS-thresholded enhancement and GRID unknown-speaker separation using transfer learning.The captions identify the scope of Tables 2 and 3, respectively.Source: Table 2 · p. 4, Table 3 · p. 4
- The paper identifies real-world multi-speaker noisy recordings, voice-recognition performance, and end-to-end enhancement as future research directions.These directions are proposed as extensions of the work.Source: §5 · p. 4