Time Domain Audio Visual Speech Separation

1. INTRODUCTION

• Blind audio-only methods require the target-speaker count to be known and fixed, and their outputs cannot be associated with speakers.Target-dependent features instead remove the need for a prior speaker count and avoid label permutation.Source: §1 · p. 1
• Speaker- or utterance-based target features used in earlier approaches [7] [8] may be insufficiently robust, especially with noise or same-gender speakers.The cited examples include VoiceFilter using d-vectors [8, 9] and an approach using a short anchor utterance [7].Source: §1 · p. 1
• Visual features are presented as acoustic-noise-insensitive and speech-content-linked, with prior work exploring them for recognition, enhancement, and separation [12] [15] [16].The paper specifically trains lip embeddings to represent phonetic information and investigates word-, CI-phone-, and CD-phone-level modeling units.Source: §1 · pp. 1–2
• Most prior audio-visual separation systems operate in the time-frequency domain, where mask-estimation accuracy is key and phase is less considered despite its effect on separation quality.Earlier systems used phase-refinement subnets or complex masks to incorporate phase information.Source: §1 · p. 1
• The proposed architecture generalizes TasNet to audio-visual target-speaker extraction directly in the time domain, using mixture speech and the target speaker’s video.The paper compares it with uPIT, Conv-TasNet, and Conv-FavsNet, and describes it as the first audio-visual separation work directly on the time domain.Source: §1 · p. 2

2. PROPOSED SYSTEM

• The system uses Si-SNR as its training objective and evaluation metric for separation quality.The paper describes Si-SNR as more robust than SDR.Source: §2 · p. 2

2.1. Overview

• TasNet encodes mixture chunks into non-negative feature sequences, estimates source masks, and decodes masked features back into time-domain signals.For target extraction, the separator conditions its target mask on audio and video features.Source: §2.1 · p. 2

2.2. Video encoder

• The video encoder uses lip regions to derive visual features through a lip embedding extractor followed by temporal convolutional blocks.The lip embedding extractor uses a 3D convolution layer and an 18-layer ResNet; the lip embeddings are 256-dimensional.Source: §2.2 · p. 2, Figure 1 · p. 3

2.3. Audio encoder/decoder

• The audio encoder and decoder apply 1D convolution and deconvolution to mixture audio and masked encoded sequences, respectively.The default convolution kernel size is 40 and stride is 20.Source: §2.3 · p. 3

2.4. Separation network

• The separation network estimates target masks from encoded audio and visual features using dilated convolutional blocks and feature fusion.It concatenates audio and visual features across channels, projects the result, and upsamples video when needed to synchronize time resolution; the setup uses D = 8.Source: §2.4 · p. 3

3.1. Dataset

• LRS2 mixtures used non-overlapping broadcast-date splits and randomly mixed two or three utterances at -5 dB to 5 dB SNR.The simulated data comprise 40k training, 5k validation, and 3k test utterances; utterances shorter than 2 seconds were excluded.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2. Training details

• Lip embeddings were trained first for word-level classification on LRW, then at phone level using LRS2 pretraining data because LRW lacks utterance-level transcripts.The LRW word-level extractor achieved 76.02% test classification accuracy.Source: §3.2 · p. 4

3.3. Results and comparisons

• The audio-only baselines include uPIT-BLSTM and Conv-TasNet, with oracle masks providing the upper bound for their corresponding mask-based methods.The LRS2 audio is less clean than WSJ0 audio, and audio-only results are worse than prior WSJ0-2mix results, especially as speaker count increases.Source: §3.3 · p. 4, Table 2 · p. 4

3.3.1. Results with different lip embeddings

• The audio-visual models with word-level lip embeddings significantly improve over audio-only methods, while CI-phone targets perform better, especially on three-speaker tests.Replacing batch normalization with global layer normalization improves both mixture sizes.Source: §3.3.1 · p. 4, Table 3 · p. 5

3.3.2. Impact of separation networks

• Na = 1 and Nf = 3 achieves the best separation-network performance; CD-phone targets are slightly worse than CI-phone targets.Increasing Nf supplies more fused-feature context when the total number of blocks is fixed at four.Source: §3.3.2 · pp. 4–5, Table 4 · p. 5

3.3.3. Comparision with the frequency-domain networks

• Oracle phase adds 3 dB Si-SNR over noisy phase in frequency-domain reconstruction, matching the time-domain audio-visual results.The comparison uses a frequency-domain audio-visual model with a similar parameter size on the same dataset.Source: Table 5 · p. 5, §3.3.3 · p. 5

3.3.4. Multi-speaker training

• 14.02 dB and 9.92 dB Si-SNR are the best multi-speaker-training results on two- and three-speaker test sets, respectively.Three-speaker-only training also generalizes to two-speaker mixtures, though slightly below the two-speaker model; mixed training improves two-speaker results across setups.Source: Table 6 · p. 5, §3.3.4 · p. 5

• The architecture combines the raw mixture waveform with the target speaker’s video to predict the target speech waveform directly.Source: §4 · p. 5
• Word-level and phoneme-level lip embeddings effectively benefit the separation network.The lip embedding extractor is pre-trained to extract movement information from video streams.Source: §4 · p. 6
• More than 3 dB and 4 dB in Si-SNR on two- and three-speaker test sets, respectively, versus audio-only and frequency-domain audio-visual methods.Source: §4 · p. 6