PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-modal Multi-channel Target Speech Separation

Rongzhi Gu, Shi-Xiong Zhang, Yong Xu, Lianwu Chen, Yuexian Zou, Dong Yu

arXiv:2003.07032v2eess.AScs.SDeess.IV

TL;DR

Audio-only separation can be challenged by acoustic interference, motivating use of complementary target-speaker cues. This paper combines direction, lip movement, and voice characteristics in a multi-modal separation framework, and reports steadier performance under video or audio modality interference.

  • Problem

    Audio-only target separation can suffer intense interference in complex acoustic environments, while prior multi-modal methods mostly addressed monaural separation.

    Source: §I · p. 1

  • Method

    The framework combines multi-channel mixture and target-direction information with lip movements and enrollment audio, and investigates multi-modal fusion methods.

    Source: §I · p. 1, A. Overview · p. 3

  • Results

    The proposed multi-modal model exhibits more stable and persistent performance under interferences from video or audio modality.

    Source: §I · p. 1, 2) Impact of sound direction… · p. 11

  • Takeaways & Limitations

    The evaluation supports the framework’s effectiveness and steadiness when some input modalities are absent or noisy.

    Source: §I · p. 1, §VI · p. 11

    Joint training of the video and audio streams may not produce sufficiently discriminative lip embeddings; the authors plan to pretrain the lip network with phonetic-transcribed data.

    Source: §VI · p. 11

Abstract

from arXiv · show

Target speech separation refers to extracting a target speaker's voice from an overlapped audio of simultaneous talkers. Previously the use of visual modality for target speech separation has demonstrated great potentials. This work proposes a general multi-modal framework for target speech separation by utilizing all the available information of the target speaker, including his/her spatial location, voice characteristics and lip movements. Also, under this framework, we investigate on the fusion methods for multi-modal joint modeling. A factorized attention-based fusion method is proposed to aggregate the high-level semantic information of multi-modalities at embedding level. This method firstly factorizes the mixture audio into a set of acoustic subspaces, then leverages the target's information from other modalities to enhance these subspace acoustic embeddings with a learnable attention scheme. To validate the robustness of proposed multi-modal separation model in practical scenarios, the system was evaluated under the condition that one of the modalities is temporarily missing, invalid or corrupted. Experiments are conducted on a large-scale audio-visual dataset collected from YouTube (to be released) that spatialized by simulated room impulse responses (RIRs). Experiment results illustrate that our proposed multi-modal framework significantly outperforms single-modal and bi-modal speech separation approaches, while can still support real-time processing.

I. INTRODUCTION

Target speech separation extracts a speaker of interest from a mixture, but audio-only systems can struggle with noise and reverberation. This work combines target-related modalities for far-field separation, building on prior work in audio-only and audio-visual separation.

I. INTRODUCTION

  • Audio-only spectrogram-masking approaches can suffer intense interference in complex acoustic environments such as noise and reverberation.The introduction frames this as a motivation for incorporating target-related information beyond the mixture audio.Source: §I · p. 1
  • Audio-only target separation estimates a target mask from the mixture spectrogram, while this paper’s stated contributions include robustness to absent or noisy modalities.The introduction defines the task as extracting the speech of interest from an observed mixture.Source: §I · p. 1
  • The paper proposes a far-field framework combining multi-channel mixture cues, lip movements, and an enrollment utterance, with factorized attention-based fusion and evaluation under noisy or unavailable modalities.Its stated contributions include integrating direction, lip movements, and voice characteristics, and investigating fusion methods.Source: §I · p. 1

B. Audio-visual target speech separation

  • Earlier multi-modal approaches mostly addressed monaural separation and reported state-of-the-art results on close-talk audio-visual datasets, leaving far-field multi-channel separation as this work’s focus.The paper also reviews audio-visual methods combining mixture audio with face or lip information, and work adding speaker embeddings to handle partial visual occlusion –.Source: §I · p. 1, B. Audio-visual target speech separation · p. 2

A. Audio-only speech separation

  • Prior audio-only work uses spatial cues for target separation, but close source directions can make those cues ambiguous and leave the target uncertain.Speaker embeddings provide another route to target conditioning, though the cited methods had been proven effective only in close-talk corpora.Source: A. Audio-only speech separation · p. 2

III. MULTI-MODAL MULTI-CHANNEL SEPARATION

The proposed system combines directional, visual, and speaker-voice information to produce estimated target speech. Its audio stream extracts spectral and spatial features from the multi-channel mixture and converts the estimated target spectrogram back to a waveform.

A. Overview

  • The system takes four inputs—multi-channel mixture waveforms, target direction, cropped lip video, and target-speaker enrollment audio—and outputs estimated monaural target speech while suppressing interferers.The overview describes a multi-stream architecture for integrating these target-related inputs.Source: A. Overview · pp. 2–3

B. Audio Stream

  • The audio stream maps mixture waveforms to complex spectrograms, extracts spectral, spatial, and direction-conditioned features, and processes them into acoustic embeddings.Stacked dilated convolutional layers provide a long receptive field for contextual information; an iSTFT layer converts the estimated target spectrogram to waveform.Source: B. Audio Stream · p. 3

1) spectral feature:

  • The spectral feature is the reference channel’s logarithm power spectrum, computed from the first channel of the multi-channel complex spectrogram.The STFT operation is reformulated as a convolution kernel for on-the-fly computation, [32],.Source: 1) spectral feature · pp. 3–4

2) spatial features:

The audio stream uses speaker-independent interchannel phase differences alongside a direction-conditioned feature. The directional feature compares steering-vector and IPD information for the target direction, with practical direction estimation relying on face tracking in the described real-recorded setup.

2) spatial features:

  • IPD features concatenate phase differences across selected microphone pairs to represent spatial information from all speakers in the mixture.The experiments select microphone pairs with different spacings rather than requiring all microphone pairs.Source: 2) spatial features · p. 4

3) directional feature:

  • The directional feature compares the steering vector with IPD: a T-F bin dominated by a source from the target direction approaches 1, otherwise it approaches 0.The formulation is adapted for general microphone-array topologies; it assumes speakers do not change location while speaking. For real recordings, the paper estimates direction from face detection and tracking, while audio localization may leave uncertainty about which direction belongs to which speaker.Source: 3) directional feature · p. 4

C. Video Stream

The video stream is trained jointly with audio, without a lip-reading objective, to learn lip embeddings aligned with speech. The paper also uses a pre-trained speaker model to represent the enrolled target speaker.

C. Video Stream

  • Joint audio-video training uses gray frames and a temporal convolutional network to produce lip embeddings, which are upsampled to synchronize with audio.The lip-reading network is trained from scratch using only the speech-separation objective.Source: C. Video Stream · p. 4
  • Audio-domain supervision encourages the video stream to learn correlations between lip movements and target speech, including mouth opening as a voice-activity cue.The authors note that phone-discriminative lip embeddings may be learned, although the training has no lip-reading supervision.Source: C. Video Stream · p. 4
  • Visualizations show that lip embeddings indicate speech boundaries and emphasize different dimensions for target and interfering speech; clusters across videos suggest shared frame information.The figure caption reports a 25-fps video stream and audio analyzed with a 32-ms window and 16-ms hop.Source: C. Video Stream · p. 5, Figure 3 · p. 5

D. Speaker Embedding

  • A speaker-verification model processes the target speaker’s enrollment utterance into an utterance-level embedding, which is tiled across audio time steps.The model uses self-attention to aggregate frame-level features.Source: D. Speaker Embedding · p. 5

E. Multi-modality Fusion

The paper investigates concatenation and factorized attention for combining acoustic, lip, and speaker embeddings. Factorized attention uses information from other modalities to select contributions from acoustic subspaces.

1) Concatenation:

  • Concatenation treats modalities equally, leaving the subsequent network to learn their interactions and potentially underusing cross-modal correlations.The paper identifies concatenation as a common approach in prior audio-visual separation work.Source: 1) Concatenation · p. 5

2) Factorized Attention:

  • Factorized attention splits acoustic embeddings into subspaces and uses other modalities to selectively aggregate them into a fused representation.Lip embeddings provide subspace weights, while speaker embeddings can be mapped into acoustic subspaces for audio-speaker fusion.Source: 2) Factorized Attention · pp. 5–6
  • Compared with direct concatenation, factorized attention sums over speaker or acoustic contexts guided by cross-modal information to capture and select semantic information across subspaces.For audio-visual fusion, lip embeddings are mapped to subspace posteriors that weight the acoustic subspaces.Source: 2) Factorized Attention · p. 6

3) Rule-based Attention:

The rule-based attention strategy adjusts the weight of spatial and directional features according to the angle difference between speakers. It lowers their weight when that difference is small.

3) Rule-based Attention:

  • The paper motivates attention by noting that modality effectiveness depends on the case, with spatial cues less discriminative for speakers from close directions.The strategy aims to emphasize discriminative modalities and ignore others.Source: 3) Rule-based Attention · p. 6
  • When speakers have a small angle difference, the rule-based strategy assigns less weight to spatial and directional features using a trainable sigmoid score.The text notes that other factors, such as whether the face is frontal-facing, could also be considered.Source: 3) Rule-based Attention · p. 6

4) Fusion of three modalities:

Three-modality fusion proceeds hierarchically, first combining unimodal embeddings into bimodal embeddings and then forming a trimodal embedding. The system estimates a target-speech mask and trains end to end with SI-SDR.

4) Fusion of three modalities:

  • Three-modality fusion uses two stages, from unimodal to bimodal embeddings and then from bimodal to trimodal embeddings.One example first fuses acoustic and speaker embeddings with factorized attention, then concatenates the result with lip embeddings.Source: 4) Fusion of three modalities · p. 6

F. End-to-End Training

  • A 1 × 1 convolution and ReLU produce the target-speech magnitude mask, which is applied to the reference-channel mixture spectrogram before iSTFT reconstruction.The estimated target speech is returned as a waveform.Source: F. End-to-End Training · p. 6
  • End-to-end training optimizes SI-SDR rather than time-domain MSE to directly optimize separation performance.The SI-SDR definition uses the reverberant clean and estimated target-speech waveforms with zero-mean normalization for scale invariance.Source: F. End-to-End Training · p. 6

IV. EXPERIMENTS PROCEDURES

Experiments use a simulated, speaker-independent Mandarin audio-visual dataset with multi-channel noisy, reverberant mixtures. The system combines spectral and spatial audio features with lip and speaker embeddings in convolutional processing blocks.

A. Dataset

  • The speaker-independent simulated dataset contains 160,000 training, 15,000 validation, and 1,200 test mixtures, with reverberation, noise, and varied speaker-to-array distances.It draws on about 1,000 speakers and 53,000 clean utterances; simulated mixtures use 1–3 speakers, 1–5 m distances, and 18–30 dB noise SNR.Source: A. Dataset · p. 7, C. Network structure · p. 8

B. Features

  • Acoustic features use the same microphone pairs for TPDs and direction features (DF), with ground-truth target-speaker direction used during training and evaluation; their total dimension is 7×257=1799.The 9-element non-uniform linear array has spacings of 4-3-2-1-1-2-3-4 cm.Source: B. Features · p. 7, A. Dataset · p. 7

C. Network structure

  • Audio and video streams use convolutional blocks, while factorized attention uses 10 acoustic subspaces and fusion blocks are repeated three times.Audio blocks use depth-wise separable convolutions with batch normalization; the video stream uses five convolutional blocks.Source: C. Network structure · pp. 7–8

D. Training procedure

The speaker model is pretrained for speaker verification and then frozen while the audio and video streams are jointly trained. Evaluation measures separation quality, intelligibility, and performance across mixture conditions.

D. Training procedure

  • The speaker model is pretrained on Mandarin speaker verification data, frozen to extract enrollment embeddings, and followed by joint training of the audio and video streams.Joint training uses utterance-level mixtures, Adam, early stopping, and learning-rate halving after four epochs without validation-loss improvement.Source: D. Training procedure · p. 8

E. Evaluation metrics

  • Evaluation uses SI-SDR as the main metric, alongside PESQ, STOI, SDR, and speaker-attributed WER, with reverberant clean speech as the reference.Models are also assessed across speaker-angle and mixing conditions without being given the number of mixture sources.Source: E. Evaluation metrics · p. 8

V. RESULTS AND ANALYSIS

The experiments compare concatenation, rule-based attention, and factorized attention for audio-speaker, audio-visual, and multimodal fusion. Factorized attention performs best overall in the audio-visual comparison, while the best multimodal configuration combines audio-speaker factorized attention with lip-feature concatenation.

A. Fusion approaches

  • Audio-speaker factorized attention raises SI-SDR from 7.1 dB to 7.7 dB for small angle differences, but neither it nor concatenation improves overall performance.The authors suggest speaker embeddings may provide target-related information where directional-feature discriminability decreases.Source: A. Fusion approaches · p. 8, Table I · p. 9
  • Among the audio-visual fusion methods, factorized attention has the best overall performance; concatenation and rule-based attention show no clear gain over DF-only.The comparison uses models trained only on overlapped data.Source: A. Fusion approaches · pp. 8–9, Table II · p. 9
  • The best multimodal result uses factorized attention for audio-speaker fusion followed by concatenation with lip embeddings.The authors report that factorized attention is more effective than concatenation for both audio-speaker and audio-visual fusion, but attribute the unsatisfactory multimodal result possibly to late lip fusion.Source: A. Fusion approaches · p. 9, Table III · p. 9

B. Impact of different modalities

Direction features contribute strongly to overall separation, while speaker and lip cues improve performance particularly for small speaker-angle differences. The multimodal system performs best across the reported speaker-count cases and remains robust to dropped lip frames.

B. Impact of different modalities

  • The multimodal model reports SI-SDR improvements of 3.7 dB, 11.1 dB, and 10.4 dB for 1-, 2-, and 3-speaker cases, respectively, and the lowest WER at 10%.Despite increased computation, the system remains real-time with RTF < 1.Source: B. Impact of different modalities · p. 9, Table IV · p. 10
  • Direction features contribute strongly overall, but lip-only separation performs better than DF-only at small angle differences; adding lip or speaker cues improves overall performance, especially in that range.DF-only has an RTF of 0.4% on the GPU.Source: B. Impact of different modalities · p. 9
  • In the illustrated example, DF-only over-suppresses target speech when the speaker turns away, while audio-visual separation leaks interference when the target opens their mouth without speaking.The multimodal example produces the best estimate among the compared outputs.Source: B. Impact of different modalities · pp. 9–10, Figure 8 · p. 10

1) Impact of missing lip information:

  • With 50% lip-frame dropout, the multimodal model’s performance decrease is less than 2%, while lip-only performance is negatively affected by frame dropping.Missing frames are filled with the latest previous frame in this experiment.Source: 1) Impact of missing lip information · p. 10, Table V · p. 10

2) Impact of sound direction estimation error:

When target and interfering speakers are close in direction, direction-estimation error sharply degrades DF-only performance, while combining target information makes separation more robust. The conclusion reports effectiveness and steadiness on a large-scale audio-visual dataset.

2) Impact of sound direction estimation error:

VI. CONCLUSION

  • Evaluation on a large-scale audio-visual dataset demonstrates the effectiveness and steadiness of the proposed multi-modal system.Source: §VI · p. 11
  • The authors identify insufficiently discriminative lip embeddings, a need for further robustness improvements, and room to improve the fusion methods.They propose pretraining the lip network with phonetic-transcribed data and note that data augmentation could further improve robustness.Source: §VI · p. 11
Loading 2003.07032v2…