Source-linked AI summary
Neural Target Speech Extraction: An Overview
Katerina Zmolikova, Marc Delcroix, Tsubasa Ochiai, Keisuke Kinoshita, Jan Černocký, Dong Yu
TL;DR
The paper addresses how to extract one speaker from mixtures containing interfering speech and noise, where target and non-target voices are difficult to distinguish. It surveys recent neural TSE approaches, organizing them by clue type and related design dimensions. The overview finds that multimodal clues can improve extraction robustness, while audio-clue systems remain practical but face speaker-identification challenges.
Problem
TSE addresses the challenge of isolating a desired speaker from mixtures with interfering speakers, noise, and other acoustic conditions using identifying clues.
Method
The paper provides an overview and taxonomy of recent neural TSE approaches, organized by clue type, channel count, speaker openness, and generative versus discriminative modeling.
Results
Audio-visual TSE can achieve superior extraction and greater robustness to clue corruption than using either audio or visual clues alone.
Takeaways & Limitations
TSE offers a practical framework for extracting a target speaker using audio, visual, or spatial clues, with multimodal combinations helping across supported conditions.
Takeaways & Limitations
Audio-clue systems must handle within-speaker variability across conditions and confusion between speakers with similar voices, requiring precise speaker modeling.
Abstract
from arXiv · showhide
Humans can listen to a target speaker even in challenging acoustic conditions that have noise, reverberation, and interfering speakers. This phenomenon is known as the cocktail-party effect. For decades, researchers have focused on approaching the listening ability of humans. One critical issue is handling interfering speakers because the target and non-target speech signals share similar characteristics, complicating their discrimination. Target speech/speaker extraction (TSE) isolates the speech signal of a target speaker from a mixture of several speakers with or without noises and reverberations using clues that identify the speaker in the mixture. Such clues might be a spatial clue indicating the direction of the target speaker, a video of the speaker's lips, or a pre-recorded enrollment utterance from which their voice characteristics can be derived. TSE is an emerging field of research that has received increased attention in recent years because it offers a practical approach to the cocktail-party problem and involves such aspects of signal processing as audio, visual, array processing, and deep learning. This paper focuses on recent neural-based approaches and presents an in-depth overview of TSE. We guide readers through the different major approaches, emphasizing the similarities among frameworks and discussing potential future directions.
I. INTRODUCTION
Target speech extraction (TSE) estimates one desired speaker’s speech from a mixture using auxiliary clues, addressing the cocktail-party problem. Unlike BSS and noise reduction, TSE uses clues to identify one target while avoiding source-count and permutation ambiguities.
- Motivation: Humans use spatial, spectral, visual, and semantic cues to attend to a desired speaker in complex acoustic scenes.The paper presents machine extraction based on auxiliary cues as a long-standing goal of speech-processing research.
- TSE problem: TSE isolates a target speaker’s speech from mixtures of several speakers by conditioning extraction on auxiliary clues.The paper discusses spatial direction, visual face or lip video, and pre-recorded enrollment speech as clue types.
- Neural TSE overview: Neural networks enabled complex conditioning on diverse clues using large speech datasets, producing increasingly capable TSE systems.The overview focuses on recent neural approaches and organizes implementations by audio, visual, and spatial clues.
- Relation to BSS and noise reduction: TSE estimates only the target signal, whereas BSS estimates all sources without clues and may require knowing or estimating their number.BSS also has a global permutation ambiguity because its outputs are not identified with the target speaker in advance.
- Relation to BSS and noise reduction: Noise reduction can suppress background noise without clues but cannot generally suppress interfering speakers because it cannot discriminate among speakers in a mixture.The paper treats clue-based personalized speech enhancement as a special TSE case when only the target speaker is actively speaking.
C. Historical context
TSE developed from early spatial beamforming and clue-based BSS toward neural systems that support flexible, efficient extraction. The historical progression also exposed constraints involving microphone arrays, target-speaker data, computational cost, and speaker coverage.
- Early spatial methods: Early TSE work used microphone arrays and fixed beamformers to enhance speech from a known target direction.The overview traces these systems to the 1980s and treats them as early spatial-clue TSE.
- BSS development: BSS history progressed from ICA and IVA to single-channel F-HMM and NMF methods that modeled speech spectra or speaker-specific sources.IVA addressed frequency permutation by modeling dependencies across frequency bins, while later single-channel approaches used pretrained spectral models.
- Clue-based methods: Some visual-clue methods required clues for all speakers, so the number of speakers had to be known beforehand despite avoiding global permutation ambiguity.This requirement reduced some advantages associated with TSE.
- Neural transition: Deep neural networks improved spectrum modeling and inference efficiency, rapidly influencing both BSS and TSE research.Neural BSS advances included deep clustering and permutation invariant training for single-channel speaker-open separation.
- Neural TSE: Early speaker-close neural TSE required substantial target-speaker data and could not extend to speakers unseen during training.Later TSE systems were conditioned on speaker characteristics rather than being tied solely to a known training speaker.
- Extensions: Recent TSE research extends beyond extraction to target-speaker ASR and personalized VAD or diarization.The field has also gained attention through evaluation campaigns including DNS and Clarity.
III. TSE TAXONOMY
The taxonomy organizes TSE approaches by clue type, channel count, speaker-open versus speaker-close setting, and generative versus discriminative modeling. It highlights audio, visual, and spatial clues and the trade-offs between multi-channel and single-channel systems.
- TSE approaches are categorized by clue type, number of channels, speaker-open versus speaker-close setting, and generative versus discriminative modeling.
- Type of clue: Audio, visual, and spatial clues are the three prominent clue types organizing the overview.
- Type of clue: Spatial clues use the target speaker’s location and require multiple microphones, but can identify stationary speakers reliably.
- Type of clue: Different clues have situation-dependent strengths, motivating systems that combine audio, visual, and spatial information.
- Number of microphones: Multi-channel systems generally provide better extracted-signal quality, whereas single-channel systems avoid microphone-array cost and remain useful when arrays are unavailable.
- Number of microphones: Recent single-channel neural TSE systems can be extended to multi-channel processing with spatial features or beamforming.
C. Speaker-open vs speaker-close methods
Speaker-open methods use target-speaker data only at test time, while speaker-close methods train on the target speaker’s data. The overview focuses on neural approaches that condition extraction on clues through a common framework.
- Speaker-open vs speaker-close: Speaker-open methods train on different speakers and receive target-speaker data only during testing, unlike speaker-close methods.
- Speaker-open vs speaker-close: Speaker-close methods include systems trained on clean utterances from the target speaker.
- Modeling approaches: TSE approaches may use generative models that estimate likelihoods or discriminative models that directly estimate target speech from observations and clues.
- Modeling approaches: Neural discriminative models became prevalent because supervised training on artificially generated data supports large-scale speech modeling and complex clue dependencies.
- Scope: The overview focuses on neural TSE methods that can operate with a single microphone and under speaker-open conditions.
- General framework: A common neural framework uses a clue encoder followed by a speech extraction module, with clue embeddings representing target-speaker information.
B. Speech extraction module
The speech extraction module transforms mixture features, conditions them on clue embeddings through a fusion layer, and estimates the target signal with a masking-based or alternative extractor.
- The speech extraction module comprises a mixture encoder, fusion layer, and target extractor.
- Mixture encoder: The mixture encoder extracts features and produces a target-agnostic mixture representation before clue conditioning.
- Fusion layer: The fusion layer combines mixture and clue embeddings to condition extraction, using options such as concatenation, FiLM, factorized, or attention-based fusion.
- Target extractor: The widely used masking-based extractor estimates a time-frequency mask and applies it element-wise to mixture features to obtain estimated target speech.
- Target extractor: Time-frequency masking exploits speech-energy concentration in a few spectral bins, although recent real-valued or complex masks do not require this assumption.
- Target extractor: Alternative extraction methods directly infer target features or replace masking with beamforming when a microphone array is available.
C. Integration with microphone array processing
Microphone-array processing adds spatial information to TSE and can replace masking with beamforming. Mask-derived spatial correlation matrices support distortionless target extraction, which is advantageous for ASR front ends.
- Microphone arrays provide spatial information that can identify the target speaker’s direction and support beamforming-based extraction.
- Beamforming: A beamformer performs linear spatial filtering of the observed microphone signals using frequency-dependent beamformer coefficients.
- Beamforming: The TSE-estimated mask can be used to compute target and interference spatial correlation matrices for deriving beamformer coefficients.
- Beamforming: Replacing time-frequency masking with spatial linear filtering enables distortionless extraction, often benefiting TSE front ends for ASR.
D. Training a TSE system
Neural TSE systems are trained on simulated mixture, target-speech, and clue triplets, with losses and architectures selected to condition extraction on the target speaker. Design choices affect performance, but their impact depends on task, clues, data generation, and training settings.
- Training data: Supervised TSE training uses mixture y, target speech xs, and clue Cs to learn the model parameters.Because clean target speech is required, training data are usually simulated by mixing clean speech and noise.
- Training data: Simulation can generate an audio clue from another target-speaker utterance and a visual clue from the associated target video.The example uses randomly selected target, interference, and noise signals to form the mixture.
- Training objectives: Training minimizes a loss between the estimated target speech and the target source signal, using signal-, mask-, or ASR-based objectives.Common choices include cross-entropy for time-frequency masks, magnitude-spectrum MSE, negative SNR, SI-SNR, SDR, and ASR losses.
- Training objectives: Time-domain SNR loss encourages correct estimation of both target-speech magnitude and phase, and has improved extraction performance.Scale-invariant SNR and SDR variants address arbitrary scaling or linear filtering of the estimate.
- Clue encoders: Pre-trained clue encoders can leverage large datasets, while jointly trained encoders optimize embeddings directly for TSE.The paper describes jointly trained, pre-trained, and combined encoder strategies.
- Architecture: Best reported performance used a shallow mixture encoder, a deeper extraction network, and fusion low in the extraction module, with multiplication or FiLM layers performing well.The choice of fusion layer itself had rather insignificant impact.
- Architecture: Specific network configurations are difficult to recommend because performance depends on task, clue type, data generation, architecture, and hyper-parameters.The overview therefore emphasizes general design considerations rather than a single optimal configuration.
V. AUDIO-BASED TSE
Audio-based TSE uses an enrollment utterance to derive a representation of the target speaker’s voice and condition extraction. The overview traces its embedding choices and early neural development, including comparisons with cascade systems.
- Audio clues: An audio clue is a target-speaker utterance whose voice characteristics identify the target in a mixture.It can be pre-recorded or taken from part of a longer recording, and is usually summarized by one vector.
- Clue encoders: Audio-clue TSE commonly uses i-vectors, NN-based speaker embeddings such as d-vectors and x-vectors, or embeddings trained directly for TSE.These alternatives are illustrated as distinct clue-encoder designs.
- Clue encoders: I-vectors capture speaker and channel variability, which can help when enrollment and mixed speech share recording conditions.They adapt a generic UBM speaker model to the specific recording.
- Clue encoders: NN-based speaker embeddings are trained for speaker classification, producing pooled vectors that are typically speaker-discriminative and robust to other variability.Large training corpora and publicly available models facilitate their reuse for TSE.
- Clue encoders: Jointly trained embeddings are optimized with the extraction module for TSE, whereas pre-trained encoders may be more robust because they use larger corpora.Fine-tuning a pre-trained encoder jointly with TSE is proposed as a middle ground but had not been reported in the overview’s account.
- Neural development: Early neural methods included SpeakerBeam’s single-channel masking and multi-channel beamforming, followed by VoiceFilter’s emphasis on ASR front ends and streaming.These methods helped establish neural audio-clue TSE around 2017.
C. Experiments
Experiments demonstrate audio-clue TSE across clean, noisy, and reverberant datasets, while visual clues and speaker modeling address difficult interference conditions. The overview also reports practical benefits and remaining identification challenges.
- Audio-clue experiments: Experiments evaluated time-domain SpeakerBeam on WSJ0-2mix, WHAM!, and WHAMR! using SI-SNR improvement over the observed mixture.These datasets represent clean, noisy, and reverberant conditions, respectively.
- Audio-clue experiments: Direct TSE outperformed a BSS-then-speaker-identification cascade, especially under difficult WHAMR! conditions.The proposed explanations are that TSE focuses capacity on the target task and receives additional speaker information.
- Audio-clue experiments: A WHAMR! spectrogram example shows TSE identifying the target speaker while removing the other speaker, noise, and reverberation.The example concerns a two-speaker recording with noise and reverberation.
- Audio-clue implications: Audio clues are practical because they require no additional cameras or multiple microphones, supporting broad applicability.The overview also describes rapidly improving accuracy in the literature.
- Challenges: Precise speaker modeling is needed because intra-speaker variability and similar voices can cause target-identification errors.Relevant variability includes emotional state, channel effects, and the Lombard effect.
- Visual clues: Visual TSE derives target-speech information from lip movements, including whether the speaker is talking or silent and potentially the phoneme being uttered.The visual clue encoder transforms synchronized video into representations used by the extraction module.
- Visual clues: Visual clues can better handle similar-voice mixtures than audio clues because they are not based on speaker voice characteristics.They are synchronized with target speech and are not corrupted by interfering speakers.
1) Pre-processing:
Visual TSE preprocesses video and encodes facial or lip information before combining it with mixture features. Audio and visual clues are complementary, and their combination improves robustness across mixture and clue conditions.
- Pre-processing: Visual preprocessing isolates the target’s face through detection, tracking, and video cropping when required by the application.These steps can use established video-processing algorithms.
- Pre-processing: Visual encoders may process raw video jointly with extraction or use pre-trained features to reduce high-dimensional video data.Joint optimization avoids information loss but can be complex for raw video.
- Visual features: Pre-trained visual representations include facial landmarks, face-recognition embeddings, lip-reading embeddings, and audio-visual synchronization embeddings.These alternatives capture facial structure, identity, acoustic content, or lip–sound timing associations.
- Visual features: Pre-trained visual features may not be optimal for TSE, and frame-level features may omit lip-motion dynamics, so a jointly trained network further transforms them.The additional network is trained with the speech extraction module.
- Multi-modal clues: Audio clues resist speaker movement and visual occlusion, whereas visual clues are less affected by voice characteristics in the mixture.Combining both clues exploits these complementary strengths.
- Multi-modal clues: Audio-visual TSE combines encoded enrollment audio and target video, using concatenation, summation, or reliability-dependent weighting such as attention.The combined embedding is supplied to the speech extraction module.
- Experimental comparison: Across mixture and clue conditions, audio-visual TSE achieved superior extraction performance and greater robustness to clue corruption.Visual TSE showed a smaller same- versus different-gender performance gap than audio TSE.
D. Discussions and outlook
Visual clues can reduce sensitivity to speaker-gender differences, while combining audio and visual clues improves robustness to corrupted inputs. Spatial-clue systems encode target direction or location through directional features, but depend on suitable visual or microphone-array information.
- Spatial clues: Spatial-clue extraction can use microphone-array signals and may obtain target location from known geometry, localization, or video-based face tracking.Depth cameras can additionally estimate elevation and speaker distance.
- Visual clues: Single-clue performance can degrade when the audio or visual clue is corrupted, whereas audio-visual TSE is more robust.The evaluated corruptions add white noise at 0 dB SNR to enrollment audio or mask the mouth region in video.
- Spatial clues: Spatial clues use target direction or a multi-channel enrollment utterance to form directional features for neural TSE.The spatial clue encoder typically extracts directional features and applies neural-network postprocessing.
- Spatial clues: For direction-of-arrival clues, angle features compare mixture inter-channel phase differences with target phase differences using a cosine operation.For bins dominated by the target direction, the angle feature should be close to 1 or -1.
C. Combination with other clues
Spatial clues are effective for TSE, but their performance declines when speakers are close together. Combining modalities outperforms individual clues, while related applications extend clue conditioning to ASR and diarization.
- Experimental results: Spatial clues are very effective, but performance declines when the angle separation between speakers is below 15°.The experiments used Mandarin audiovisual mixtures containing two or three speakers.
- Experimental results: Combining spatial, visual, and audio clues outperformed every individual clue type across all tested angle-separation conditions.
- Spatial clues: Spatial TSE requires microphone-array recordings and a way to determine the target location, limiting its application scope.Current neural systems also commonly assume that the target location is fixed, while dynamic settings remain relatively rare.
- Combination with ASR: Target-speech extraction can serve as an ASR front end, while joint optimization with an ASR criterion can improve target-speaker ASR performance.Integrated clue-conditioned ASR avoids explicit signal extraction and reduces computational cost, but may be less interpretable.
- VAD and diarization: Target-speaker VAD predicts whether a target is active, enabling simpler networks than speech-signal estimation; multi-target extensions achieved top CHiME 6 diarization performance.
IX. REMAINING ISSUES AND OUTLOOK
The overview identifies deployment, evaluation, inactive-target, and data-mismatch challenges for TSE. It points toward more realistic evaluation, robust or adaptive training, and lightweight low-latency systems.
- Deployment: Most TSE systems operate offline, are computationally expensive, and are evaluated mainly on controlled simulated mixtures.Deployment requires lower computational cost and robustness under less controlled recording conditions.
- Inactive target speaker: Most systems assume the target speaker is active, so inactive-target cases remain insufficiently addressed, especially with audio clues.Visual voice activity detection may alleviate this issue, whereas audio-clue handling is more challenging.
- Training and evaluation criteria: SNR and SDR may not correlate with human perception, intelligibility, or ASR performance, motivating perceptually relevant metrics and controlled listening tests.STOI and PESQ are suggested as alternative measures for human-listener applications.
- Training and evaluation criteria: TSE evaluation should distinguish extraction from target-identification performance because identification failures can produce interference estimates or mixture outputs.Signal-level metrics may also inadequately represent inactive-speaker cases.
- Robustness and adaptation: Training on simulated mixtures requires reasonable matching to deployment noise and reverberation, although large simulated datasets can generalize to real recordings.Real recordings without reference targets could augment training or support adaptation through unsupervised losses.
- Robustness and adaptation: Combining neural TSE with adaptive generative approaches is proposed to join strong matched-condition performance with adaptation to recording conditions.
- Lightweight systems: Low-latency TSE is increasingly relevant to teleconferencing, hearing aids, and hearables, which impose severe computation and latency constraints.Efficient systems must also preserve target identification over long mixture contexts.
5) Spatial rendering:
TSE research is expanding beyond audio, visual, and spatial clues toward semantic, brain-activity, and arbitrary-sound cues. Practical resources exist, but datasets and implementations remain centered largely on simulated mixtures and selected toolkits.
- Spatial rendering: TSE systems for hearing aids or hearables must estimate the target’s direction after extraction so the rendered speech preserves spatial location.
- Representation learning: Self-supervised and cross-modal learning could improve TSE representations, robustness, and shared embedding spaces across enrollment and mixture signals.
- Other clues: Future TSE systems may use semantic clues such as language or abstract concepts to model forms of selective attention beyond physical cues.
- Other clues: EEG signals correlated with a listener’s attended speech envelope have been explored as clues for brain-guided TSE.This direction could support hearing aids controlled by user brain activity.
- Beyond speech: The same clue-conditioned framework can extend from target speech to instruments and arbitrary sounds using class labels, enrollment audio, video, or text descriptions.
- Datasets and toolkits: TSE datasets generally use artificial mixtures of isolated speakers and background signals, with separate enrollment utterances or video clues for evaluation.
- Datasets and toolkits: Open implementations and visual feature extractors are available, though some visual TSE systems lack public implementations.