PaperlayerView exact revision on arXiv

Source-linked AI summary

ADL-MVDR: All deep learning MVDR beamformer for target speech separation

Zhuohuang Zhang, Yong Xu, Meng Yu, Shi-Xiong Zhang, Lianwu Chen, Dong Yu

arXiv:2008.06994v3eess.AS

TL;DR

Neural speech separation can distort target speech in ways that harm ASR, while conventional mask-based MVDR can leave residual noise and involve unstable matrix operations during joint training. ADL-MVDR replaces matrix inversion and PCA with RNNs for frame-level beamforming; it outperforms prior methods across objective metrics and ASR accuracy. The evaluation uses simulated multichannel mixtures from a Mandarin corpus, without lip-movement features.

  • Problem

    Neural separation can introduce nonlinear distortion that harms ASR, while mask-based MVDR can leave residual noise and its matrix inversion can be unstable during joint neural-network training.

    Source: §1 · p. 1

  • Method

    ADL-MVDR replaces MVDR’s matrix inversion and PCA with two RNNs to estimate frame-level beamforming weights, using cRF to estimate speech and noise components.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    ADL-MVDR outperforms prior methods across objective metrics and ASR accuracy; versus NN with cRF, WER is 12.73% vs. 22.07%.

    Source: §6 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    The authors propose that ADL-MVDR may generalize to other speech-separation systems, with single-channel separation and dereverberation left for future verification.

    Source: §6 · p. 4

    Evaluation uses simulated multichannel mixtures from a Mandarin corpus, and lip-movement features are not fed into the model.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

Speech separation algorithms are often used to separate the target speech from other interfering sources. However, purely neural network based speech separation systems often cause nonlinear distortion that is harmful for automatic speech recognition (ASR) systems. The conventional mask-based minimum variance distortionless response (MVDR) beamformer can be used to minimize the distortion, but comes with high level of residual noise. Furthermore, the matrix operations (e.g., matrix inversion) involved in the conventional MVDR solution are sometimes numerically unstable when jointly trained with neural networks. In this paper, we propose a novel all deep learning MVDR framework, where the matrix inversion and eigenvalue decomposition are replaced by two recurrent neural networks (RNNs), to resolve both issues at the same time. The proposed method can greatly reduce the residual noise while keeping the target speech undistorted by leveraging on the RNN-predicted frame-wise beamforming weights. The system is evaluated on a Mandarin audio-visual corpus and compared against several state-of-the-art (SOTA) speech separation systems. Experimental results demonstrate the superiority of the proposed method across several objective metrics and ASR accuracy.

1. INTRODUCTION

The introduction frames speech separation as a trade-off between neural systems’ nonlinear distortion and conventional MVDR’s residual noise and training instability. It proposes ADL-MVDR and reports strong performance across objective metrics and ASR accuracy.

  • Neural separation can add nonlinear distortion harmful to ASR, while mask-based MVDR can leave residual noise and suffer unstable matrix inversion during joint neural-network training.Source: §1 · p. 1
  • ADL-MVDR uses two RNNs to replace MVDR’s matrix inversion and PCA, while cRF estimates speech and noise covariance matrices for frame-level beamforming.The framework is designed for stable joint training with a front-end filter estimator.Source: §1 · p. 1
  • The cRF-based ADL-MVDR achieves the best performance in many objective metrics and ASR accuracy.Source: §1 · p. 1
  • Unlike directly learned beamforming weights that lacked noise information, ADL-MVDR retains mask-based MVDR and explicitly uses speech and noise covariance matrices.Source: §1 · p. 2

2. SIGNAL MODEL FOR MVDR BEAMFORMER

The section formulates multichannel speech separation in the time-frequency domain and describes the mask-based MVDR objective and steering-vector solution. It also notes the limits of utterance-level weights and the use of diagonal loading and complex ratio masks.

  • A noisy mixture recorded by an M-microphone array is represented using clean speech and interfering noise, with signals indexed in time and frequency.Source: §2 · p. 2
  • The MVDR beamformer minimizes noise power while constraining the target speech to remain undistorted.Source: §2 · p. 2
  • The steering-vector MVDR solution derives the target steering vector by applying PCA to the speech covariance matrix; diagonal loading is used to alleviate matrix-inversion instability.Source: §2 · p. 2
  • Complex ratio masks estimate target speech with less phase distortion, while the resulting utterance-level covariance matrices yield weights that are not optimal for noise reduction on each frame.Source: §2 · p. 2

3. PROPOSED ADL-MVDR BEAMFORMER

ADL-MVDR combines complex ratio filtering with recurrent estimation of frame-level MVDR quantities. Two GRU networks replace matrix inversion and PCA, using speech and noise covariance information to produce frame-level beamforming weights.

3.1. cRF for covariance matrix estimation

  • Complex ratio filtering estimates speech and noise components from nearby time-frequency bins to stabilize the covariance estimates.The frame-level speech covariance retains temporal information by not summing over time.Source: §3.1 · p. 2

3.2. RNNs for replacing matrix inversion and PCA in MVDR

  • Two GRU-Nets replace matrix inversion and PCA to estimate frame-level steering vectors and inverse noise covariance from their respective covariance matrices.The recurrent networks use information from previous frames rather than heuristic updating factors between consecutive frames.Source: §3.2 · p. 2, §3 · p. 2
  • The GRU outputs are used to compute frame-wise MVDR weights, which are applied to the multichannel speech.The real and imaginary parts of the complex covariance matrices are concatenated as GRU-Net inputs.Source: §3.2 · p. 3

4. DATASET AND EXPERIMENTAL SETUP

The system uses audio and visual cues to guide multichannel target speech separation, evaluated on a simulated Mandarin audio-visual corpus. The setup specifies training conditions and compares the proposed approach with neural and MVDR baselines.

4.1. System overview and dataset

  • The system combines log-power interaural phase difference, spectra, and face-location-guided directional features from synchronized audio-visual recordings.The mixture is recorded with a 15-channel microphone array and a 180° camera.Source: §4.1 · p. 3
  • The corpus contains 205,500 audio clips (roughly 200 hours) with simulated speaker mixtures, indoor noise, and reverberation conditions from T60 0.05 s to 0.7 s.Although the platform synchronizes 15-channel recordings with a camera, lip movement features are not used in this study.Source: §4.1 · p. 3

4.2. Experimental setup

  • Training uses 4-second chunks, batch size 12, a 3×3 cRF, Adam with learning rate 1e−3, and 60 epochs; baselines include neural and conventional MVDR systems.The baseline set also includes multi-tap MVDR systems trained with cRM and cRF.Source: §4.2 · p. 3, Table 1 · p. 4, Figure 2 · p. 4

5. RESULTS AND DISCUSSIONS

ADL-MVDR outperforms the compared neural and conventional MVDR systems on reported objective metrics and ASR accuracy. The comparisons indicate gains in speech quality and recognition, including under close-interferer conditions.

ADL-MVDR vs. NN

  • Compared with NN with cRF, ADL-MVDR achieves around 42% WER improvement (12.73% vs. 22.07%) and better objective metrics: PESQ 3.42 vs. 3.10, Si-SNR 14.80 dB vs. 12.50 dB, SDR 15.45 dB vs. 13.01 dB.The authors attribute purely NN-based systems’ poorer ASR performance to a large amount of distortion.Source: §5 · p. 4

ADL-MVDR vs. MVDR

  • 3.42 vs. 2.92 PESQ: ADL-MVDR improves about 17% over MVDR with cRF, with WER 12.73% vs. 15.91%.ADL-MVDR also reports Si-SNR 14.80 dB vs. 12.66 dB and SDR 15.45 dB vs. 14.04 dB against MVDR with cRF.Source: §5 · p. 4

ADL-MVDR vs. MVDR; cRM vs. cRF

  • 3.04 vs. 1.88 PESQ: ADL-MVDR improves speech quality nearly 62% when interfering sources are 0–15° apart.For NN systems, cRF also outperforms cRM in all metrics and ASR accuracy, including Si-SNR 12.50 dB vs. 12.23 dB and WER 22.07% vs. 22.49%.Source: §5 · p. 4

6. CONCLUSIONS AND FUTURE WORK

The paper concludes that ADL-MVDR recursively learns spatio-temporal filtering for multichannel target speech separation and outperforms prior work on objective metrics and ASR accuracy.

6. CONCLUSIONS AND FUTURE WORK

  • ADL-MVDR recursively learns spatio-temporal filtering for multichannel target speech separation and outperforms prior work across objective metrics and ASR accuracy.The authors propose future verification on single-channel separation and dereverberation tasks.Source: §6 · p. 4
Loading 2008.06994v3…