Source-linked AI summary
Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention
Yuma Koizumi, Kohei Yatabe, Marc Delcroix, Yoshiki Masuyama, Daiki Takeuchi
TL;DR
Speech enhancement needs to work for unknown speakers, but speaker adaptation has generally relied on auxiliary guidance. This paper derives a speaker-aware feature from the test utterance and jointly trains enhancement and speaker identification. It reports state-of-the-art performance on a public dataset and better subjective quality than conventional methods.
Problem
The paper asks whether speech enhancement can adapt to unknown speakers without an auxiliary test-time guidance signal.
Source: §1 · p. 1Method
A jointly trained speaker-identification branch supplies an auxiliary feature from the test utterance, alongside BLSTM and multi-head self-attention modeling.
Source: §1 · p. 1, §2.2 · p. 2Results
Takeaways & Limitations
The reported experiments support speaker-aware self-adaptation as effective for DNN-based speech enhancement, including when the target speaker is unknown.
Source: §1 · p. 1, §5 · p. 4The MHSA path omits speaker-aware features and position information, as it is intended to focus on long-term noise dependencies.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
This paper investigates a self-adaptation method for speech enhancement using auxiliary speaker-aware features; we extract a speaker representation used for adaptation directly from the test utterance. Conventional studies of deep neural network (DNN)--based speech enhancement mainly focus on building a speaker independent model. Meanwhile, in speech applications including speech recognition and synthesis, it is known that model adaptation to the target speaker improves the accuracy. Our research question is whether a DNN for speech enhancement can be adopted to unknown speakers without any auxiliary guidance signal in test-phase. To achieve this, we adopt multi-task learning of speech enhancement and speaker identification, and use the output of the final hidden layer of speaker identification branch as an auxiliary feature. In addition, we use multi-head self-attention for capturing long-term dependencies in the speech and noise. Experimental results on a public dataset show that our strategy achieves the state-of-the-art performance and also outperform conventional methods in terms of subjective quality.
1. INTRODUCTION
DNN speech enhancement must generalize to unknown speakers, while related speech applications show gains from target-speaker adaptation. This paper tests whether speaker-aware self-adaptation can work without test-time guidance.
1. INTRODUCTION
- Conventional enhancement trains speaker-independent DNNs on many speakers, whereas target-speaker specialization has improved ASR and TTS performance.The cited adaptation approaches use auxiliary features such as i-vectors or speaker codes [16–19].Source: §1 · p. 1
- Speech separation systems using speaker information require a guidance signal, but enhancement's dominant signal is the target speech, motivating adaptation without test-phase guidance.Speech enhancement estimates a time-frequency mask from the noisy observation and applies it to recover the target speech.Source: §1 · p. 1
- The method extracts speaker-related information from the observation while enhancing speech, using a speaker-identification branch trained jointly with the enhancement model.Its final hidden-layer output serves as an auxiliary feature, and BLSTM and multi-head self-attention model speech and noise.Source: §1 · p. 1
- The authors report effectiveness for speakers absent from training, state-of-the-art performance on a public dataset, and better subjective quality than conventional methods.Source: §1 · p. 1
2. RELATED WORKS
The related work contrasts mask-based enhancement with separation and reviews speaker-guided extraction. The proposed study extends speaker-aware features to enhancement, where the target can be inferred from the noisy input.
2.1. DNN-based speech enhancement and separation
- Enhancement assumes the mixture contains target speech and environmental noise, whereas separation handles mixtures of multiple interfering speakers.A common DNN enhancement approach estimates a time-frequency mask in the STFT domain; the network mask is applied to the observation before inverse-STFT reconstruction.Source: §2.1 · pp. 1–2
2.2. Auxiliary speaker-aware feature for speech separation
- Speaker-aware separation methods use a clean target-speaker utterance as guidance, converting it into a summarized feature that adapts the enhancement network's hidden-layer outputs.In SpeakerBeam, a neural network maps the guidance signal to a sequence feature λ, which is multiplied with hidden-layer outputs.Source: §2.2 · p. 2
- This work applies speaker-aware auxiliary information to noise reduction rather than only target extraction, deriving speaker information directly from the noisy signal without guidance.The authors motivate this by the noisy signal containing the target speaker and noise, rather than competing speech.Source: §2.2 · p. 2
3. PROPOSED METHOD
The proposed network jointly enhances speech and identifies the speaker, using the speaker branch's representation as an auxiliary feature. It combines recurrent and attention paths, with the attention path designed to capture long-term noise dependencies.
3.1. Basic idea
- The enhancement and speaker-identification branches are trained together, and the speaker branch's final hidden-layer output supplies the auxiliary feature.The joint objective trains time-frequency mask estimation and speaker identification simultaneously.Source: §3.1 · p. 2
3.2. Implementation
- The base network combines CNN and BLSTM blocks, adds a parallel MHSA path, and outputs a complex-valued time-frequency mask for inverse-STFT resynthesis.The input is a normalized log-amplitude spectrogram; the CNN block uses two 2-D convolutions, a 1×1 convolution, and a linear layer.Source: §3.2 · p. 2
- The speaker branch uses a CNN and BLSTM to produce a time-dependent speaker feature, which is concatenated with CNN features for the enhancement BLSTM.Unlike sequence-summarized guidance, the network directly uses the speaker feature Λ, which the authors say could better represent phoneme dynamics.Source: §3.2 · p. 2
- A parallel MHSA path processes CNN features through two attention modules to capture long-term dependencies, while omitting the speaker feature and position information.The authors expect this path to focus on noise information, including long-time similarities in non-stationary noise such as intermittent noise.Source: §3.2 · pp. 2–3
- The network predicts a complex mask and speaker labels, trained with a multi-task loss combining SDR-based and cross-entropy losses.The speaker-label output is formed from the speaker features and a softmax; the loss also uses a mixing parameter and clipped SDR-based term.Source: §3.2 · p. 3
4. EXPERIMENTS
The experiments test self-adaptation on VoiceBank-DEMAND through verification, objective benchmark comparison, and subjective listening evaluation.
4. EXPERIMENTS
- Verification contrasted target-speaker training (Close), other-speaker training (Open), and other-speaker training with an SPK branch (Open+SPK), using 53 test samples.The broader experiments used VoiceBank-DEMAND, with 28 training speakers (11,572 utterances) and two test speakers (824 utterances).Source: §4.1 · p. 3, §4 · p. 3
4.1. Verification experiments
- Open+SPK beat Open on every verification measure except CSIG and approached Close’s upper-bound scores.Table 1 presents the verification results.Source: Table 1 · p. 3, §4.1 · p. 3
4.2. Objective evaluations
- Ours outperformed SEGAN, MMSE-GAN, DFL, and Metric-GAN across the evaluated objective metrics; attention maps showed longer-context focus around impulsive noise or consonants, possibly explaining MHSA’s best CBAK score.Table 2 reports the public-dataset objective evaluation.Source: §4.2 · p. 3, Table 2 · p. 4
- Compared with MHSA, SPK scored better on PESQ and COVL; the paper attributes this to speaker-aware information extracted by SPK.The paper also reports that MHSA achieved the best CBAK score.Source: §4.2 · pp. 3–4
- In one low-SNR female-speaker utterance, a hoarse male speaker was selected mainly during consonants despite a female utterance-level identification.Framewise speaker IDs varied across the utterance.Source: §4.2 · pp. 3–4, Figure 2 · p. 4
4.3. Subjective evaluations
- On 15 low-SNR samples, Ours scored highest on S-MOS, N-MOS, and G-MOS, with paired one-sided tests significant at p < 0.01.Ten participants rated the output signals.Source: Figure 3 · p. 4, §4.3 · p. 4
5. CONCLUSIONS
The paper extracts a speaker representation from the test utterance to adapt a DNN speech enhancer. Across its experiments, this strategy worked for unknown speakers, achieved state-of-the-art performance on a public dataset, and improved subjective quality over conventional methods.
5. CONCLUSIONS
- A multi-task cost function simultaneously trains a DNN time-frequency mask estimator and speaker identifier to extract a speaker representation for adaptation.The representation is extracted directly from the test utterance.Source: §5 · p. 4
- The strategy was effective for unknown speakers and achieved state-of-the-art performance on a public dataset.The conclusion also reports better subjective quality than conventional methods.Source: §5 · p. 4
- The authors conclude that self-adaptation using speaker-aware features is effective for DNN-based speech enhancement.Source: §5 · p. 4
Appendix A. Detail of MHSA module
The MHSA module computes scaled attention for each head, combines head context matrices, then applies normalization and a feed-forward layer. Its feed-forward linear layers output dimensions D/2 and 3D/2.
Appendix A. Detail of MHSA module
- Each head's attention matrix is computed by applying softmax to its scaled attention scores after layer normalization of Γ.The scaling parameter is d = D/(2H).Source: Appendix A · p. 4
- The h-th head’s context matrix is calculated; its value projection Wh,v has size D/(2H) × D/2, matching the stated sizes of Wh,q and Wh,k.Source: Appendix A · p. 4
- The head context matrices are concatenated, projected by Wp, and added to Γ before a second layer normalization.Source: Appendix A · p. 4
- The normalized output is passed to a point-wise feed-forward layer whose two linear layers output dimensions D/2 and 3D/2.Source: Appendix A · p. 4