PaperlayerView exact revision on arXiv

Source-linked AI summary

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Daniel Michelsanti, Zheng-Hua Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, Jesper Jensen

arXiv:2008.09586v2eess.AScs.LGeess.IV

TL;DR

Deep-learning-based audio-visual speech enhancement and separation lack a focused overview despite extensive recent research. This paper systematically surveys their methods, related tasks, datasets, and evaluation, reporting benefits of visual information while noting that comparisons remain difficult without standardized audio-visual evaluation.

  • Problem

    No overview article focuses on deep-learning-based audio-visual speech enhancement and separation, despite substantial recent research and interest.

  • Method

    The paper presents an extensive systematic survey of audio-visual enhancement and separation methods, related tasks, datasets, and evaluation approaches.

  • Results

    Visual information benefits both enhancement and separation: enhancement gains occur at very low SNRs or match audio-only performance at high SNRs, while separation also avoids source permutation.

  • Takeaways & Limitations

    Audio-visual evaluation should reflect real-world listening conditions because existing comparisons and objective measures often omit the visual component.

  • Takeaways & Limitations

    Fair comparison is difficult because applications have different requirements and standardized audio-visual evaluation procedures are lacking.

Abstract

from arXiv · show

Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, visual information from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: acoustic features; visual features; deep learning methods; fusion techniques; training targets and objective functions. In addition, we review deep-learning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance.

I. INTRODUCTION

Speech enhancement and separation recover target speech from noisy or mixed acoustic signals, while visual speech cues provide information unaffected by acoustic noise. The paper surveys deep-learning-based audio-visual approaches and related tasks, emphasizing their system components and evaluation.

  • Task definitions: Speech enhancement recovers one target speech signal, whereas speech separation estimates multiple target speech signals from mixtures.Audio-visual enhancement uses noisy target speech and visual information; audio-visual separation uses mixtures and target-speaker visuals.
  • Motivation: High environmental noise and competing speakers impair speech communication, motivating enhancement systems for human listeners and automatic speech-recognition front ends.Enhancement commonly targets speech quality and intelligibility, while separation estimates multiple signals when several targets matter.
  • Methodological shift: Deep-learning approaches generally treat enhancement and separation as supervised learning problems rather than relying solely on statistically specified signal models.A footnote notes that some representation-learning approaches are more properly described as self-supervised or unsupervised.
  • Audio-visual motivation: Audio-visual methods exploit visible articulatory movements and facial expressions because visual information is immune to acoustic noise and competing speakers.This makes vision a reliable cue for focusing on a target stimulus in challenging acoustic conditions.
  • Survey scope: The survey addresses a gap in prior overviews by systematically covering deep-learning-based audio-visual enhancement and separation, related tasks, datasets, evaluation, and future directions.It organizes systems around acoustic and visual features, deep-learning methods, fusion techniques, training targets, and objective functions.

III. AUDIO-VISUAL SPEECH ENHANCEMENT AND SEPARATION SYSTEMS

Deep-learning-based audio-visual enhancement and separation systems learn from degraded and clean speech paired with speaker video. The survey analyzes their principal data representations, processing methods, multimodal fusion, training targets, and objective functions.

  • Training setup: Supervised deep-learning models learn enhancement or separation from paired degraded and clean speech signals together with speaker video.The training data should ideally represent the conditions in which systems will be deployed.
  • System elements: The survey organizes audio-visual systems around acoustic features, visual features, deep-learning methods, fusion techniques, training targets, and objective functions.Figure 2 presents the interconnections among these data and processing elements.

IV. ACOUSTIC FEATURES

AV-SE and AV-SS systems predominantly use single-microphone time-frequency representations, while some approaches incorporate speaker embeddings and fuller acoustic signal representations.

  • Magnitude spectrograms, sometimes in the log mel domain, are the predominant acoustic inputs in AV-SE and AV-SS systems.
  • Recent systems also process phase, complex spectrogram components, or raw waveforms to incorporate more complete acoustic information.
  • Speaker embeddings: Speaker embeddings, including i-vectors, condition AV-SE and AV-SS models on target-speaker information from enrolment or related representations.
  • Speaker embeddings: Visual lip information was more important than speaker embeddings in one reported system, while combining both generally improved performance.
  • Speaker embeddings: A first enhancement run can provide a speaker representation on the fly, bypassing the need for cumbersome or unavailable enrolment audio.

C. Multi-Microphone Features

Multi-microphone systems add spatial acoustic cues to spectral information, while visual processing depends on reliable face tracking and remains limited by single-camera input.

  • C. Multi-Microphone Features: Inter-channel phase, time, and level differences, directional statistics, and mixture STFT vectors provide spatial cues for multi-channel SE and SS.
  • C. Multi-Microphone Features: CosIPDs and sinIPDs are preferred over raw IPDs because they avoid the abrupt discontinuities caused by phase wrapping.
  • C. Multi-Microphone Features: Multi-microphone visual systems have used IPDs, cosIPDs, sinIPDs, and angle features alongside visual information.
  • C. Multi-Microphone Features: Further integration of multi-microphone features with visual information remains an open research direction.
  • Face Detection and Tracking: Face detection and tracking are critical because failures can propagate to later AV-SE and AV-SS modules.

D. Still Images as Visual Input

Still images can provide speaker-specific visual cues for speech extraction with lower system complexity, but omitting dynamic visual information considerably limits performance.

  • D. Still Images as Visual Input: Still-image conditioning outperformed an audio-only system because facial appearance has a cross-modal relationship with voice characteristics.
  • D. Still Images as Visual Input: Still images reduce overall system complexity compared with video input.
  • D. Still Images as Visual Input: The absence of dynamic information from video considerably limits still-image system performance.
  • Tracking limitations: Face detection may fail when people move or turn their heads, motivating more robust target-source tracking using body features.
  • Deep learning methods: Deep neural networks learn representations of acoustic and visual modalities at multiple abstraction levels and flexibly combine them.
  • Deep learning methods: Feedforward networks, CNNs, and gradient-based training are among the deep-learning methods used in these systems.

B. Recurrent Neural Networks

RNNs are suited to speech because they process sequential data, but deep-learning AV systems require substantial data and face realism, synchronization, and computational constraints.

  • B. Recurrent Neural Networks: RNNs are particularly suitable for speech because they process sequential data and capture its important temporal dimension.They are trained with backpropagation through time, but vanishing and exploding gradients remain common.
  • C. Shortcomings and Future Research: Deep-learning AV systems need large datasets to generalize across speakers, languages, noises, SNRs, illumination conditions, and face poses.Large-scale AV datasets marked an important step, while smaller-data training remains an open direction.
  • C. Shortcomings and Future Research: Training commonly assumes independent sound sources, although speakers alter their speech in noise through the Lombard effect.This assumption is convenient because realistic speech-in-noise data is costly to collect.
  • C. Shortcomings and Future Research: AV-SE performance is affected when training data does not match real acoustic and visual conditions.The cited mismatch concerns the conditions represented during training versus those encountered in deployment.
  • C. Shortcomings and Future Research: AV synchronization is necessary because audio and video may be temporally misaligned or sampled at different rates.Video is often upsampled to match acoustic-feature timing, while preprocessing for temporal alignment can fail at low SNRs.
  • C. Shortcomings and Future Research: Early fusion combines modalities at the feature level, whereas late fusion combines separately processed modalities at the decision level.Early fusion can exploit cross-modal correlation early; late fusion permits reuse of established unimodal models and is easier to implement.

C. Fusion Paradigms with Deep Learning

Deep-learning AV systems support several fusion strategies and training designs, with intermediate fusion widely used alongside attention and modality-balancing techniques.

  • Fusion paradigms: Intermediate fusion combines unimodal acoustic and visual representations into a single hidden layer and is used by most surveyed systems.Early and late fusion are also possible, but intermediate fusion is the predominant reported strategy.
  • Fusion techniques: Concatenation is the preferred fusion technique, but treating modalities identically can let one modality dominate and degrade performance.This dominance may occur even when the less reliable modality is not the best cue for a particular scenario.
  • Training strategies: Multi-style and multitask training can encourage joint use of modalities or combine speech enhancement with related objectives such as ASR.One strategy randomly zeros an input modality; another alternates AV-SE and ASR training, which was reported as most effective in that study.
  • Fusion techniques: Attention-based fusion softly selects more informative modalities or cues according to the situation being analyzed.For closely located speakers, attention can favor spectral audio and lip-motion cues over less discriminative spatial and directional features.
  • Hybrid approaches: Hybrid systems combine learned mappings with knowledge-based processing, including beamforming or Wiener filtering.Examples include jointly trained neural and MVDR beamforming modules and visual-to-filterbank mappings followed by Wiener filtering.
  • Shortcomings and future research: Fusion design remains expertise-intensive, and temporal misalignment can make multimodal integration difficult.The survey identifies automatic architecture exploration and misalignment-aware techniques as future directions.

A. Direct Mapping

Direct mapping trains a model to estimate clean-speech representations directly, while alternative mask-based targets and phase modeling address reconstruction limitations.

  • A. Direct Mapping: Direct mapping trains a system to minimize the difference between its estimated clean-speech magnitude spectrogram and the ground truth.MSE is common, while MAE has been reported to increase spectral detail and performance compared with MSE.
  • A. Direct Mapping: Time-domain reconstruction requires an estimate of the target short-time phase in addition to the target magnitude.Using target rather than noisy phase improved PESQ and STOI in one reported system.
  • A. Direct Mapping: Mask approximation estimates a mask that is multiplied element-wise with the noisy time-frequency representation during inference.Binary masks classify time-frequency units, whereas continuous masks provide soft decisions.
  • A. Direct Mapping: Binary masks such as IBM, TBM, and PBM have been used as AV training targets with cross-entropy objectives.These masks distinguish speech-dominated from noise-dominated time-frequency units.
  • A. Direct Mapping: The IRM is a continuous mask used by some AV models with MSE or MAE, while hybrid MAE-plus-cosine losses address limitations of single losses.The hybrid loss was intended to produce sharp results and avoid the statistical-independence assumption associated with MSE or MAE alone.
  • A. Direct Mapping: IRM cannot perfectly recover the target magnitude spectrogram from the noisy spectrogram, motivating IAM and phase-sensitive masks.PSM compensates for phase mismatch, while CRM can perfectly reconstruct the clean complex spectrogram when applied to the complex noisy spectrogram.

M PSM

Indirect mapping predicts a mask with a signal-domain objective, while related approaches incorporate phase, time-domain criteria, multitask learning, and permutation handling.

  • M PSM: CRM uniquely permits perfect reconstruction of the clean complex spectrogram when applied to the complex noisy spectrogram.IAM, PSM, and CRM have been used in AV systems with MSE objectives.
  • C. Indirect Mapping: Ratio-mask approaches are discouraged for dereverberation because reverberation is convolutive and ratio masking assumes uncorrelated target and interference.In cascaded separation and dereverberation, masking may also fail to reduce artifacts correlated with the target speech.
  • C. Indirect Mapping: Indirect mapping predicts a mask for easier neural estimation but defines the objective in the signal domain.It combines the mask output of mask approximation with the signal-domain objective associated with direct mapping.
  • C. Indirect Mapping: Indirect mapping did not achieve the best AV-SE performance in a direct comparison, although it sometimes helped fine-tune systems pretrained with mask objectives.Despite this comparison, AV systems trained from scratch with indirect mapping often obtained good results, likely aided by large datasets.
  • Related approaches: Other systems estimate clean magnitude spectrograms from visual features to construct masks, or extend indirect mapping to time-domain objectives.Multitask systems also combine enhancement objectives with ASR or speaker-representation objectives.
  • M PSM: Speaker-independent separation must address source permutation, for which permutation invariant training and deep clustering are established solutions.The problem is an inconsistent assignment of separated signals to sources over time.

G. Shortcomings and Future Research

The survey reviews silent-video speech reconstruction and related AV separation methods, while identifying limitations in speaker independence, real-world conditions, and future objective-function design.

  • Future research: Future AV-SE and AV-SS work should investigate perceptually motivated objectives, binaural-cue estimation, and end-to-end time-domain targets.These directions are presented as expected avenues for further improvement.
  • Cross-task connections: Speech reconstruction models can support AV enhancement by estimating time-frequency masks from reconstructed spectrograms.Non-speech AV separation models can likewise be adapted to speech by retraining them on AV speech datasets.
  • Speech reconstruction from silent videos: Silent-video speech reconstruction estimates speech from visual information, but frontal or side cameras miss excitation and most tongue movements.The missing tongue information is especially important because it contributes to generating several speech sounds.
  • Speech reconstruction from silent videos: Deep-learning approaches map visual inputs to speech features or signals, including spectrograms, vocoder parameters, LPC representations, and time-domain waveforms.Reported systems use regression, classification with codebooks, CNNs, RNNs, GANs, and speaker-conditioned sequence-to-sequence models.
  • Shortcomings: Speaker-independent reconstruction remains unsatisfactory, partly because training uses limited numbers of speakers.Most studies also use controlled environments, leaving lighting variation and occlusions insufficiently addressed.

B. Audio-Visual Sound Source Separation for Non-Speech Signals

AV separation for non-speech signals uses visual information alongside audio to localize and separate sounds, with deep learning enabling increasingly integrated and flexible systems.

  • Scope: Non-speech AV source separation extracts sounds such as musical instruments from mixtures and offers insights applicable to AV speech enhancement and separation.The reviewed task is complementary to AV-SS because its target signals are not speech.
  • Deep-learning approaches: Early deep systems combined visual and acoustic processing through MIML-guided NMF or jointly learned representations for source separation.The MIML approach linked frequency bases to visual object categories and operated on in-the-wild videos containing instruments, animals, and vehicles.
  • Deep-learning approaches: PixelPlayer simultaneously localised sound sources in video frames and acoustically separated them using audio and video information.Its results stimulated subsequent work aimed at improving the approach.
  • Subsequent extensions: Later systems addressed unsynchronised modalities, appearance and motion cues, curriculum learning, and conditioning with weak labels.These extensions used disentangled representations, trajectories, semantic features, U-Net variants, and additional presence or absence information.
  • Training limitations: Co-separation learns associations between consistent sounds and visual objects across paired videos, addressing correlations absent from randomly mixed independent clips.The survey identifies training on real-world recordings rather than only synthetic mixtures as a future direction.

X. AUDIO-VISUAL SPEECH CORPORA

AV speech datasets range from controlled recordings suited to targeted experiments to in-the-wild collections intended for robust real-world modeling, but unconstrained benchmarking remains unsettled.

  • Dataset selection: Dataset choice is critical because AV corpora support different research purposes and determine the data available for deep-learning development.The survey provides a non-exhaustive overview of resources for selecting datasets by application.
  • Dataset gaps: Most datasets provide clean speaker signals that researchers mix synthetically, leaving a need for corpora combining speaker mixtures with real noise.Such datasets would provide a benchmark for AV-SS in noise.
  • Controlled corpora: GRID and TCD-TIMIT are the most commonly used AV-SE and AV-SS databases.GRID contains 34 speakers recording 1000 sentences each, while TCD-TIMIT contains 62 speakers with phonetically balanced sentences.
  • Controlled corpora: Controlled datasets support studies of viewing angle, video frame rate, Lombard effects, and related experimental factors.Examples include TCD-TIMIT, OuluVS2, MODALITY, Lombard GRID, and RAVDESS.
  • In-the-wild corpora: In-the-wild datasets provide greater variation in speakers, sentences, languages, and audiovisual environments for training models intended for real-world robustness.They differ from conventional corpora collected under carefully controlled laboratory conditions.

XI. PERFORMANCE ASSESSMENT

AV-SE and AV-SS evaluation commonly measures speech quality and intelligibility, but existing tests often inherit audio-only procedures and do not fully reflect audiovisual human reception.

  • Evaluation goals: Speech quality is subjective and multidimensional, while intelligibility concerns how successfully listeners understand speech.The survey reviews evaluation methods for both attributes.
  • Listening tests: Human listening tests are the most reliable assessment for receiver-facing systems because they evaluate signals on the actual human receiver.This is especially relevant to hearing-assistive devices and teleconferences.
  • Quality assessment: MOS uses a five-point quality scale, whereas MUSHRA uses a continuous 0–100 scale across five labelled quality intervals.MOS was designed for speech coders, whose distortions differ from those in speech enhancement.
  • Intelligibility: Intelligibility tests include nonsense-syllable, word, and sentence tests, but word tests may poorly predict real-world sentence understanding.Sentence tests ask listeners to identify keywords or recognize whole utterances.
  • Listening tests: AV listening tests should present both auditory and corresponding visual stimuli rather than only audio signals.Audio-only presentation ignores visual information that affects speech perception and differs from many practical use conditions.
  • AV-specific challenges: AV test interpretation is complicated by individual differences in lip-reading ability and unpredictable responses to auditory–visual discrepancies.The survey calls for ecologically valid paradigms that reduce result variability and better estimate real-world performance.

B. Objective Measures

Objective evaluation of AV-SE and AV-SS combines listening tests with algorithmic measures of speech quality and intelligibility. Common measures are efficient and often correlate with listening tests, but current approaches remain limited for in-the-wild and genuinely audio-visual evaluation.

  • Listening tests provide the ideal assessment of processed speech, but they are time-consuming, costly, and require representative end users.
  • PESQ is widely used for speech quality and generally ranges from 1 to 4.5 while approximating mean opinion scores.It correlates highly with listening tests across mobile, fixed, VoIP, and multiple-type network conditions.
  • POLQA succeeds PESQ for conditions such as high background noise, super-wideband speech, variable delay, and time scaling.
  • STOI is the most commonly used objective intelligibility measure and computes correlations between clean and degraded temporal-envelope segments.It has been shown to correlate well with intelligibility listening experiments.
  • Intrusive estimators require target speech references, making them unsuitable for in-the-wild tests where clean reference signals cannot be collected.
  • Current objective measures use audio-only signals, so they may misestimate performance when listeners can see the speaker.Developing audio-visual quality and intelligibility predictors requires substantial audio-visual listening-test data.

C. Beyond Speech Quality and Intelligibility

Beyond speech quality and intelligibility, AV-SE and AV-SS evaluation and design must consider computational efficiency, listening effort, preserved speaker information, multimodal fusion, and real-world robustness. The survey also connects related tasks, datasets, and future research directions while avoiding method rankings because comparisons are difficult to interpret.

  • Beyond speech quality and intelligibility: Real-time factor is relevant for systems such as hearing aids and teleconference devices that must deliver processed signals during conversation.
  • Beyond speech quality and intelligibility: Listening effort measures the energy a subject invests in understanding speech, which can remain substantial even when the signal is fully intelligible.
  • Beyond speech quality and intelligibility: AV-SE and AV-SS systems should ideally preserve speaker and scene information, including gender, age, emotion, mood, and location.
  • Conclusion: AV-SE systems outperform audio-only systems at very low SNRs or match them at high SNRs, while AV-SS avoids source permutation through visual guidance.
  • Conclusion: The survey avoids ranking methods because application requirements and non-standardized audio-visual evaluation procedures make comparisons difficult to interpret.Future directions include robustness, lower computational complexity, and new audio-visual fusion paradigms.
  • Conclusion: Short-time magnitude spectrograms are the usual acoustic input, while low-dimensional visual features reduce processing complexity and data requirements.
  • Conclusion: Most systems separately process acoustic and visual features before fusion, often by concatenation, although attention mechanisms can mitigate modality imbalance.
  • Conclusion: The survey covers related silent-video speech reconstruction, non-speech audio-visual separation, datasets, and evaluation methods as sources of insight for AV-SE and AV-SS.Controlled datasets support specific phenomena, whereas large-scale in-the-wild datasets target robustness to real-world conditions.
Loading 2008.09586v2…