Source-linked AI summary
DiffVQE2: An Efficient Low-delay Diffusion Model for Acoustic Echo and Noise Control
Haljan Lugo, Ernst Seidel, Pejman Mowlaee, Ziyue Zhao, Tim Fingscheidt
TL;DR
Diffusion-based acoustic echo control had been limited to non-causal, utterance-level processing. This work introduces causal DiffVQE2 models for joint echo and noise control, which outperform DeepVQE in subjective MOS while using less computational complexity.
Problem
Diffusion-based acoustic echo control was limited to non-causal, utterance-level processing, restricting its applicability in practice.
Source: §1 · p. 1Method
DiffVQE2 adapts a hybrid diffusion model for causal joint echo and noise control using causal convolutions and unidirectional GRUs, with a smaller variant and optional limited lookahead.
Source: §1 · p. 1, §2.2 · p. 2Results
DiffVQE2 exceeds DeepVQE across all four MOS metrics (3.58 vs. 3.44) at 13% of its computational complexity; two-frame-lookahead DiffVQE2-S reaches 3.62 vs. 3.58.
Source: §4 · p. 4Takeaways & Limitations
Causal diffusion-based echo and noise control can operate at low algorithmic delay and computational cost, while limited lookahead improves the small model to performance on par with the larger model.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Hands-free communication devices and speakerphones are inherently affected by acoustic echo and background noise. To mitigate these impairments, end-to-end discriminatively trained neural networks have emerged as the best-performing approach in research and deployment. While recent advancements in generative methods have provided remarkable results for various speech enhancement tasks, diffusion-based acoustic echo control (AEC) research is still restricted to non-causal, utterance-level processing, thereby not widely applicable in practice. With this work, we are the first to propose low-delay (i.e., causal) diffusion-based joint AEC and noise control models DiffVQE2 / DiffVQE2-S, excelling the so-far state of the art DeepVQE / DeepVQE-S models in multiple objective metrics, and, most importantly, in subjective MOS, respectively. In addition, our models are less complex. Furthermore, we show that a limited lookahead applied to the efficient DiffVQE2-S model allows for an even higher performance. These results have been obtained on the ICASSP 2023 AEC Challenge blind test set. We claim the first streaming-capable, diffusion-based acoustic echo and noise control that excels state-of-the-art discriminative approaches.
1. INTRODUCTION
Diffusion methods have shown strong results in speech enhancement, but prior generative AEC methods process whole utterances and do not meet low-delay causal requirements. DiffVQE2 introduces causal, lower-complexity diffusion models for joint echo and noise control and reports state-of-the-art performance in both DeepVQE model-size regimes.
1. INTRODUCTION
- Diffusion models have shown strong noise-reduction performance by learning clean-speech distributions and sampling from them, motivating their extension to AEC.The introduction also cites the URGENT Challenge as evidence of generative methods’ potential across distortion types.Source: §1 · p. 1
- Earlier generative AEC methods operate at utterance level, making them incompatible with low-delay causal requirements in hands-free communication.DiffVQE is described as a reproducible hybrid diffusion-based AEC and noise-reduction model.Source: §1 · p. 1
- DiffVQE2 and DiffVQE2-S achieve state-of-the-art performance in the DeepVQE and DeepVQE-S model regimes on the ICASSP 2023 AEC Challenge test set.The models use causal processing, and limited lookahead offers a trade-off between algorithmic delay and output quality.Source: §1 · p. 1
- DiffVQE2 adapts the larger, non-causal DiffVQE using causal convolutions and unidirectional GRUs, while DiffVQE2-S targets DeepVQE-S model size and lower computational complexity.The paper also introduces limited lookahead to partially compensate for the absence of future context.Source: §1 · p. 1
2. METHODS
DiffVQE2 estimates clean speech from the microphone mixture using a conditional estimate followed by a one-step score-based generative estimate. The paper makes the system causal, offers a smaller grouped-GRU variant and optional lookahead, and trains with a joint objective that supports the one-step sampler.
2.1. Score-based Diffusion for Voice Quality Enhancement
- The inference pipeline transforms signals to the frequency domain, estimates a discriminative mask and speech conditions, then applies one-step score-based generation before returning to the time domain.The score-based setup follows [10] and uses an EDM-style one-step sampler conditioned on the speech conditions and conditional estimate.Source: §2.1 · p. 2
2.2. Novel DiffVQE2
- DiffVQE2 replaces symmetric convolutions with left-padded causal convolutions and BGRUs with unidirectional GRUs to remove future-context dependencies.These changes are applied to both the Cond and Score DNNs.Source: §2.2 · p. 2
- DiffVQE2-S reduces encoder and decoder channel widths and replaces GRUs with grouped GRUs, processing latent-feature groups in parallel.Grouping reduces parameter count by approximately a factor of g, as the GRU count is dominated by quadratic terms.Source: §2.2 · p. 2
- DiffVQE2-S uses NL future STFT frames as Cond-DNN input, adding NL frame shifts of algorithmic delay to incorporate future context.The spectrogram and shifted versions are concatenated along the channel dimension before the first Cond-DNN convolution.Source: §2.2 · p. 2
2.3. Training
- The joint training objective combines auxiliary reconstruction for Cond, end-to-end reconstruction of the final output, and score matching for the Score DNN.The Score DNN is evaluated twice with shared weights: on the noised conditional estimate and on noised clean speech, using the same conditions and Gaussian noise.Source: §2.3 · p. 2
- Training inputs include NL additional future frames while outputs and clean-speech targets remain aligned to frame ℓ, preventing future context from entering the loss window.Training segments append the extra input frames while keeping the target window fixed at L frames.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
The evaluation uses synthesized training and validation data and the blind ICASSP 2023 AEC Challenge test set. It combines intrusive and nonintrusive metrics with subjective listening tests, and evaluates multiple lookaheads and retrained DeepVQE baselines.
3.1. Dataset and Framework
- Training uses roughly 623 hours of synthesized data, validation uses TIMIT, ETSI noise, and Aachen impulse responses, and testing uses the ICASSP 2023 AEC Challenge blind set.The test set receives causal delay compensation via GCC-PHAT before evaluation.Source: §3.1 · p. 3
3.2. Metrics
- Evaluation combines AECMOS and DNSMOS with PESQ, ESTOI, LPS, average nonintrusive-metric rank, and crowdsourced MOS listening tests.The intrusive metrics and LPS are reported on Dval; the subjective tests use a representative subset of 50 files per condition.Source: §3.2 · p. 3
3.3. Training Details
- The study tests lookaheads of NL ∈ {0, 1, 2, 3} frames and trains DiffVQE2 models for 500 k steps, selecting checkpoints by lowest Dval average rank.DeepVQE and DeepVQE-S are retrained on Dtrain for the same number of epochs.Source: §3.3 · p. 3
4. EXPERIMENTAL EVALUATION AND DISCUSSION
On the blind test set, DiffVQE2 improves subjective MOS over DeepVQE at the same delay and 13% of its computational complexity, while DiffVQE2-S improves MOS over DeepVQE-S at matched size and delay. Two-frame lookahead raises the small model’s performance toward the larger model, at the cost of additional delay.
Evaluation setup
- Figure 3 tracks AECMOS and DNSMOS alongside PESQ, LPS, and ESTOI as small-model lookahead varies across Dval conditions.Table 1 reports model footprint, algorithmic delay, and performance on blind test set Dtest.Source: §4 · pp. 3–4
Blind test results
- 3.58 vs. 3.44 average MOS: DiffVQE2 exceeds DeepVQE despite a lower instrumental average rank, at the same delay and 13% of the computational complexity.The models require 5.37 vs. 42.24 GFLOPS, respectively.Source: Table 1 · p. 4, §4 · p. 4
- 3.55 vs. 3.37 average MOS: DiffVQE2-S exceeds DeepVQE-S at the same 0.7M model size and algorithmic delay, using 83% of the computational complexity.The models require 2.57 vs. 3.12 GFLOPS, respectively.Source: Table 1 · p. 4, §4 · p. 4
- DiffVQE2 loses performance on most instrumental metrics versus non-causal DiffVQE, and its average MOS is 3.58 vs. 3.68.The isolated subjective MOS scores are mixed.Source: Table 1 · p. 4, §4 · p. 4
Validation results
- On Dval, DiffVQE2-S exceeds DeepVQE-S on double-talk PESQ, LPS, and ESTOI, but DeepVQE-S is slightly ahead in single-talk ESTOI.Across models, DT-Echo and STFE-Echo are at least 4.6; DiffVQE2 exceeds DeepVQE on intrusive metrics in both double-talk and single-talk conditions.Source: §4 · p. 4
Lookahead
- 3.62 vs. 3.58 average MOS: two-frame lookahead brings DiffVQE2-S to performance comparable with DiffVQE2, with 48 ms rather than 32 ms delay.The small model has 0.7M parameters and requires 2.59 GFLOPS, versus 5.1M parameters and 5.37 GFLOPS for DiffVQE2.Source: §4 · p. 4
5. CONCLUSIONS
The paper introduces causal, low-delay diffusion-based joint acoustic echo and noise control models at two model scales. On the blind test set, both outperform their DeepVQE counterparts in average MOS, while two-frame lookahead brings the small model to performance comparable with the larger one.
Blind test results
- DiffVQE2 exceeds DeepVQE on all four MOS metrics, with 3.58 vs. 3.44 overall MOS at 13% of the computational complexity.DiffVQE2-S reaches 3.55 vs. 3.37 overall MOS against DeepVQE-S at 17% less computational complexity.Source: §5 · p. 4
Contribution
- DiffVQE2 and DiffVQE2-S are the first proposed diffusion-based joint acoustic echo and noise control models operating at low algorithmic delay, or causally.The models target the 5.3M and 0.7M parameter scales of DeepVQE and DeepVQE-S, respectively, at lower computational complexity.Source: §5 · p. 4
Lookahead
- Two-frame lookahead improves DiffVQE2-S to performance on par with the larger DiffVQE2 model.Source: §5 · p. 4