Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou; Syu-Siang Wang; Ying-Hui Lai; Yu Tsao; Hsiu-Wen Chang; Hsin-Min Wang

Source-linked AI summary

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

arXiv:1709.00944v5Sep 1, 2017cs.SDcs.MMeess.ASstat.ML

TL;DR

Speech enhancement traditionally emphasizes audio, motivating a model that also uses visual information. The paper introduces an end-to-end AVDCNN encoder-decoder with joint audio-visual learning and reports superior enhancement against audio-only, conventional, and prior audio-visual systems.

Problem
Most speech-enhancement techniques focus on audio information, motivating investigation of multimodal audio-visual enhancement.
Method
AVDCNN separately processes audio and visual inputs with CNNs, fuses them, and jointly reconstructs enhanced speech and visual signals end to end.
Results
AVDCNN outperforms four baselines across PESQ, STOI, SDI, HASQI, and HASPI, and consistently outperforms ADCNN across evaluation metrics.
Takeaways & Limitations
The results support incorporating visual information into CNN-based multimodal speech enhancement and using the proposed architecture rather than early fusion.
Takeaways & Limitations
Real-world testing leaves room for improvement because lighting, lower SIR, or background-noise properties may reduce reconstruction quality.

Abstract

from arXiv · show

Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual deep CNNs (AVDCNN) SE model, which incorporates audio and visual streams into a unified network model. We also propose a multi-task learning framework for reconstructing audio and visual signals at the output layer. Precisely speaking, the proposed AVDCNN model is structured as an audio-visual encoder-decoder network, in which audio and visual data are first processed using individual CNNs, and then fused into a joint network to generate enhanced speech (the primary task) and reconstructed images (the secondary task) at the output layer. The model is trained in an end-to-end manner, and parameters are jointly learned through back-propagation. We evaluate enhanced speech using five instrumental criteria. Results show that the AVDCNN model yields a notably superior performance compared with an audio-only CNN-based SE model and two conventional SE approaches, confirming the effectiveness of integrating visual information into the SE process. In addition, the AVDCNN model also outperforms an existing audio-visual SE model, confirming its capability of effectively combining audio and visual information in SE.

I. INTRODUCTION

The paper proposes AVDCNN, an end-to-end audio-visual encoder-decoder that fuses CNN-processed audio and visual streams for speech enhancement. Its experiments report better performance than audio-only, conventional, and prior audio-visual approaches.

Proposed approach: AVDCNN processes audio and visual streams with individual CNNs, fuses their outputs, and reconstructs enhanced speech and visual signals jointly.The visual reconstruction task supplies constraints during training in a multi-task learning framework.
Proposed approach: The model is trained end-to-end, jointly learning parameters through back-propagation while reducing reliance on human-engineered multimodal fusion.The architecture combines separate modality-specific CNNs with a joint fusion network.
Results: AVDCNN outperforms four baselines, including three audio-only systems and a prior audio-visual model, across PESQ, STOI, SDI, HASQI, and HASPI.The reported comparison also includes a CNN-based audio-only model and conventional enhancement methods.
Results: The proposed architecture performs better than its early-fusion alternative, supporting the reported value of its multimodal fusion design.The comparison uses instrumental metrics on enhanced speech.

A. Data Collection

The study uses synchronized speech and mouth-image data from Mandarin utterances, extracting normalized spectro-temporal audio features and aligned RGB mouth images. The AVDCNN architecture contains separate audio and visual networks whose outputs are fused.

Data Collection: The dataset contains 320 Mandarin utterances from a native speaker, recorded frontally in a quiet room with sufficient light.Each utterance lasts approximately 3–4 seconds and contains 10 Chinese characters.
Audio Feature Extraction: Audio is resampled to 16 kHz, converted into normalized logarithmic power spectra, and represented with ±2-frame context windows of size 257 × 5.Frames use 32-millisecond windows with 37.5% overlap, yielding 50 frames per second.
Visual Feature Extraction: Video is sampled at 50 fps, synchronized with speech frames, and processed by detecting, resizing, and normalizing the RGB mouth region.The cropped mouth region is resized to 16 pixels × 24 pixels.
Data Collection: Audio spectrogram and mouth-image sequences are aligned by truncating them to the same number of frames when necessary.
AVDCNN architecture: AVDCNN uses separate audio and visual networks, then fuses their outputs in a fusion network.The architecture is presented as the proposed model in Fig. 1.

A. Training the AVDCNN Model

AVDCNN training maps noisy speech and corresponding visual features through separate networks into fused multimodal representations. Back-propagation optimizes the model with a weighted objective over audio and visual reconstruction errors.

A. Training the AVDCNN Model: Training uses noisy-clean speech pairs and corresponding mouth images, represented by noisy spectra X, clean spectra Y, and visual features Z.
A. Training the AVDCNN Model: The audio and visual network outputs are flattened and concatenated to form the fusion-network input F_i.This creates a joint representation from the two modality-specific feature streams.
A. Training the AVDCNN Model: A feed-forward cascaded fully connected network processes the fused representation to produce the model outputs.
A. Training the AVDCNN Model: The model parameters are randomly initialized between -1 and 1 and trained by optimizing an objective function with back-propagation.
A. Training the AVDCNN Model: The mixing weight μ controls the relative contribution of audio and visual errors in the training objective.The paper states that the errors are considered equally harmful when μ is fixed to 1.
A. Training the AVDCNN Model: Dropout of 0.1 follows FC1 and FC2, while batch normalization is applied at each layer.A stride size of 1 × 1 is used in the CNNs.

B. Using the AVDCNN Model for Speech Enhancement

The AVDCNN model produces enhanced speech amplitudes and corresponding visual features from noisy speech amplitudes and visual inputs. Its audio-only counterpart disconnects visual-related components while retaining the remaining architecture.

Testing procedure: During testing, AVDCNN maps noisy speech amplitudes and corresponding visual features to enhanced speech amplitudes and visual-feature outputs.Noisy phases are reused for synthesis of the enhanced speech waveform.
Testing procedure: The enhanced waveform is synthesized by combining AVDCNN-enhanced amplitudes with the borrowed noisy-speech phase.
Audio-only comparison: The ADCNN architecture is obtained from AVDCNN by disconnecting its visual parts.

C. Baseline Models

The study compares AVDCNN with audio-only and conventional speech-enhancement baselines, as well as an existing audio-visual model. The experiments use diverse training noises, unseen test noises, and specified neural-network optimization settings.

Audio-only baselines: The audio-only ADCNN removes all visual-related parts from AVDCNN while preserving the remaining configurations.
Conventional baselines: KLT and logMMSE serve as the two conventional speech-enhancement baselines.
Audio-visual baseline: AVDNN is included as an audio-visual baseline and uses handcrafted Mel-filter-bank and lip-contour-distance-change features.Unlike AVDCNN, AVDNN uses DNNs without the described multi-task learning scheme.
Experimental setup: Training uses 91 interference-noise types, while testing uses 10 interference-noise types unseen during training.The test set therefore adopts a noise-mismatched condition.
Optimization: The neural network is trained with stochastic gradient descent and RMSprop using an initial learning rate of 0.0001.Training stops after 20 epochs show less than 0.1% improvement in training loss.
Qualitative comparison: Spectrogram comparisons include clean and noisy speech alongside outputs from logMMSE, KLT, AVDNN, ADCNN, and AVDCNN.

B. Comparison of Spectrograms

The AVDCNN model suppresses noise more effectively than the evaluated baseline methods, particularly when auxiliary visual information is available in challenging conditions. Across five instrumental metrics, it consistently outperforms the audio-only CNN and audio-visual comparison models.

Spectrogram comparison: AVDCNN effectively suppresses noise where the mouth is closed, whereas the audio-only methods leave noise components visible.This comparison uses 3T room noise at 5 dB SIR and −5 dB SAR.
Instrumental measures: AVDCNN outperforms ADCNN consistently across PESQ, STOI, SDI, HASQI, and HASPI when SAR is fixed at 0 dB.The comparison covers ten noise types and the corresponding enhanced speech outputs.
Instrumental measures: AVDCNN also outperforms AVDNN on all five metrics under the evaluated noise conditions.The reported differences were significant in most conditions according to ANOVA and Tukey post-hoc comparisons.
SAR and SIR comparisons: AVDCNN generally performs better at higher SAR levels and shows its clearest advantage over other methods at lower SIR levels.The figures average scores across ten noise types for each SIR and SAR condition.
SAR and SIR comparisons: Visual information provides important clues for AVDCNN speech enhancement in more challenging conditions.This conclusion follows the reported SAR and SIR comparisons.

D. Multi-style Training Strategy

The multi-style training strategy varies audio-visual, visual-only, and audio-only inputs during training to address modality dominance. Including visual features reduces the audio loss relative to audio-only training.

Training strategy: Multi-style training randomly selects audio-visual, visual-only, or audio-only inputs during training.The input type is changed every 45 epochs.
Training strategy: The strategy addresses the tendency of one modality to dominate the other in a multimodal network.The paper motivates the strategy using prior findings and preliminary observations.
Training results: The audio MSE drops to a lower level once visual features are included, indicating a strong correlation between the audio and visual streams.This pattern is reported for both Model-I and Model-II learning curves.

E. Mixing Weight

The mixing weight controls the trade-off between visual and audio losses during AVDCNN training. Increasing emphasis on visual information improves visual loss but worsens audio loss, while audio loss dominates enhancement results.

Loss weighting: Larger mixing weight μ improves visual loss but worsens audio loss during AVDCNN training.The paper therefore tends to select a smaller μ because audio loss dominates the enhancement results.

F. Multimodal Inputs with Mismatched Visual Features

AVDCNN enhancement depends strongly on correctly matched visual features. Correct lip features preserve more speech detail and produce substantially higher PESQ scores than mismatched lip sequences.

Mismatched visual features: Correct lip features preserve more detailed structures in AVDCNN-enhanced speech than incorrect lip-feature sequences.The experiment fixes one of eight selected mouth shapes as the visual feature for an entire utterance.
Mismatched visual features: PESQ was 2.54 with correct visual features, compared with 1.17–2.07 for eight fake lip-shape sequences.These scores are means over 40 testing utterances.
Figure comparison: Figure 17 compares noisy speech, clean speech, correctly conditioned enhancement, selected lip shapes, and enhancement conditioned on incorrect lip features.The noisy example uses 1T on-air noise at 0 dB SIR.
Model outputs: The AVDCNN system produces enhanced speech and corresponding mouth images through an audio-visual encoder-decoder with multi-task learning.Visual output is used as part of the training constraints.

H. Subjective Listening Tests

The AVDCNN was evaluated through subjective listening tests, fusion comparisons, and controlled versus real-world analyses. Visual cues aided target-speech identification and suppression, but performance degraded under real-world conditions and when mouth shape was misleading.

Subjective Listening Tests: Subjective listening tests compared logMMSE, ADCNN, and AVDCNN enhanced speech using five-point BAK and OVRL ratings.Each subject evaluated utterances enhanced under -5 dB SIR and -5 dB SAR across 10 testing noises.
Early Fusion Comparison: AVDCNN consistently outperformed AVDCNN-EF, showing that late fusion was better than combining heterogeneous features before convolution.The comparison used mean instrumental scores over 10 noises at different SIRs with 0 dB SAR.
Visual Cues and VAD: Visual information helped suppress non-target speech and noise when the mouth was closed, supporting its usefulness for voice activity detection.The lip shape could indicate whether to activate a voice-command system in the presence of background talkers or noise.
Real-World Testing: In real-world night-market recordings, lip shape helped identify target-speech segments, but target-speech reconstruction was worse than in the controlled environment.The authors suggest different light coverage, lower SIR, or background-noise properties as possible contributors.
Visual Cues and VAD: Noise was incompletely removed in some non-speech segments when the mouth remained open, revealing a limitation of relying on mouth shape for enhancement.The issue appears in the AVDCNN-enhanced spectrogram shown in Fig. 19.

VI. CONCLUSION

The paper concludes that AVDCNN combines separate audio and visual CNN streams through a fused encoder-decoder with multi-task learning for speech enhancement. It reports gains over audio-only and audio-visual baselines, while identifying late fusion, lip-shape accuracy, real-world data, synchronization, and architecture as important considerations.

VI. CONCLUSION: AVDCNN is an end-to-end CNN-based audio-visual encoder-decoder that fuses separately processed modalities and uses multi-task learning for speech enhancement.The model reconstructs audio and visual information through jointly learned multimodal representations.
VI. CONCLUSION: AVDCNN outperformed three audio-only baseline models and other audio-visual speech-enhancement models on five instrumental evaluation metrics.The conclusion attributes these results to integrating visual information with audio information in speech enhancement.
VI. CONCLUSION: Late fusion of audio and visual streams performed better than early fusion.The conclusion presents this as an experimentally demonstrated design advantage.
VI. CONCLUSION: Lip shapes correlated highly with speech and were effective as auxiliary features for voice activity detection, but correct lip shapes were important.The conclusion also points to potential problems when using audio-visual speech-enhancement models.
VI. CONCLUSION: Future work includes whole-face visual inputs, newer CNN architectures, improved audio-video synchronization, and training data from more complicated real conditions.These directions are intended to improve visual descriptors, model performance, synchronization, and practical real-world applicability.