Source-linked AI summary
DPT-FSNet: Dual-path Transformer Based Full-band and Sub-band Fusion Network for Speech Enhancement
Feng Dang, Hangting Chen, Pengyuan Zhang
TL;DR
Existing dual-path transformer studies did not fully investigate effective frequency-domain inputs or the structure of full-band and sub-band fusion for speech enhancement. DPT-FSNet addresses this with intra-transformers for sub-band information and inter-transformers for full-band information. Experiments on VoiceBank+DEMAND and DNS show better results than current state-of-the-art methods at relatively small model size.
Problem
Earlier dual-path transformer studies used simple time-domain features and did not further investigate how the input representation affects enhancement performance.
Method
DPT-FSNet uses a frequency-domain encoder, dual-path transformer, and decoder, with intra-transformers modeling sub-band information and inter-transformers modeling full-band information.
Results
The proposed model outperforms current state-of-the-art methods on the VoiceBank+DEMAND and DNS datasets at relatively small model size.
Takeaways & Limitations
The proposed frequency-domain full-band and sub-band fusion design provides more efficient features for dual-path structures within the evaluated speech-enhancement setting.
Abstract
from arXiv · showhide
Sub-band models have achieved promising results due to their ability to model local patterns in the spectrogram. Some studies further improve the performance by fusing sub-band and full-band information. However, the structure for the full-band and sub-band fusion model was not fully explored. This paper proposes a dual-path transformer-based full-band and sub-band fusion network (DPT-FSNet) for speech enhancement in the frequency domain. The intra and inter parts of the dual-path transformer model sub-band and full-band information, respectively. The features utilized by our proposed method are more interpretable than those utilized by the time-domain dual-path transformer. We conducted experiments on the Voice Bank + DEMAND and Interspeech 2020 Deep Noise Suppression (DNS) datasets to evaluate the proposed method. Experimental results show that the proposed method outperforms the current state-of-the-art.
1. INTRODUCTION
Speech enhancement removes noise to improve noisy-speech quality and intelligibility, while existing dual-path transformer studies had not fully explored frequency-domain inputs. DPT-FSNet addresses this gap by fusing sub-band and full-band information through intra- and inter-transformers and reports better results than other speech enhancement models.
- Speech enhancement improves noisy-speech quality and intelligibility by removing noise.
- Time-frequency methods transform noisy waveforms into Fourier magnitude spectra, modify them with masks, and reconstruct enhanced waveforms using inverse short-time Fourier transforms.
- Sub-band models focus on frequency-specific local spectral patterns, and fusing sub-band with full-band information can further improve enhancement performance.
- Earlier dual-path transformer studies modeled local and global features but relied on simple time-domain features and did not further investigate their input representation.
- DPT-FSNet uses intra-transformers for sub-band information and inter-transformers for full-band information, achieving better results than other speech enhancement models on two datasets.The evaluated datasets are VoiceBank+DEMAND and Interspeech 2020 DNS.
2. IMPROVED TRANSFORMER
The paper uses a transformer encoder as its basic block and replaces positional encoding with recurrent processing to learn order information in dual-path networks. Its multi-head self-attention combines multiple attention heads, while a modified feed-forward network uses a GRU layer.
- In this paper, “transformer” refers to the encoder part, containing multi-head self-attention and a modified position-wise feed-forward network.
- Multi-head self-attention: Multi-headed self-attention projects input sequences into queries, keys, and values, then concatenates the attention heads through an output transformation.
- Modified position-wise feed-forward network: A GRU layer replaces the first fully connected layer in the feed-forward network to learn location information because standard positional encoding is unsuitable for dual-path networks.
- Modified position-wise feed-forward network: The feed-forward network uses an expansion dimension dff = 4 × d, where d is the model dimension.
3. PROPOSED DPT-FSNET
DPT-FSNet is a frequency-domain speech-enhancement network that alternates sub-band and full-band modeling through a dual-path transformer. Its encoder-decoder pipeline produces a complex ratio mask, while training combines waveform- and spectrum-level losses.
- Architecture: DPT-FSNet comprises an encoder, dual-path transformer processing module, and decoder for frequency-domain speech enhancement.
- Encoder: The encoder transforms the STFT complex spectrum X ∈R2×T ×F into a high-dimensional representation U ∈RC×T ×F using 1x1 convolution and a four-layer dilated-dense block.
- Dual-path transformer: The dual-path module halves encoder channels before processing, then doubles the output channels and smooths them with a gated convolutional layer.
- Dual-path transformer: Each dual-path transformer alternates an intra-transformer for information across time steps within each sub-band and an inter-transformer for global information across sub-bands.
- Decoder: The decoder estimates a complex ratio mask; multiplying it element-wise with the input spectrum and applying ISTFT produces the enhanced waveform.
- Loss function: The training objective combines time-domain waveform MSE with a spectrum-domain L1 loss over real and imaginary complex-spectrogram components.
4. EXPERIMENTS
Experiments evaluate DPT-FSNet on small- and large-scale speech-enhancement datasets using perceptual, intelligibility, distortion, quality, and source-separation metrics. The setup includes specified spectral-processing and optimization configurations.
- Datasets: The experiments use VCTK+DEMAND as a small-scale dataset and DNS as a large-scale dataset.
- Datasets: VCTK+DEMAND contains 11,572 training utterances from 28 speakers and 872 test utterances from 2 speakers, with multiple noise conditions and SNRs.
- Datasets: DNS contains over 500 hours of clean clips from 2,150 speakers and over 180 hours of noise clips from 150 classes, with dynamic mixing during training.
- Implementation: STFT and ISTFT use 25 ms windows, 6.25 ms frame shifts, and a 512-point FFT, while the T-F representation has 64 feature maps.
- Implementation: The learning-rate schedule uses k1 = 0.2, k2 = 4e−4, dmodel = 32, and warmup = 4000.
- Metrics: Evaluation uses WB-PESQ and STOI on both datasets, plus CSIG, CBAK, and COVL on VCTK+DEMAND and SI-SDR on DNS.
- Results: Table 1 compares systems on the VCTK+DEMAND dataset.
5. EXPERIMENTAL RESULTS
Across VCTK+DEMAND and DNS evaluations, DPT-FSNet achieves strong enhancement performance, while ablations attribute gains to frequency-domain processing and full-band/sub-band fusion.
- 5.1. Results on the VCTK+DEMAND dataset: DPT-FSNet achieves state-of-the-art performance on WB-PESQ, STOI, CSIG, CBAK, and COVL with the fewest parameters among compared transformer-based models.The comparison includes TSTNN, T-GSA, and SE-Conformer.
- 5.2. Ablation analysis: The ablation analysis compares time-domain dual-path processing, BLSTM, sub-band transformers, and the proposed full-band/sub-band fusion model on VCTK+DEMAND.Table 2 reports these ablation configurations.
- 5.2. Ablation analysis: The improved transformer outperforms BLSTM, and the full-band/sub-band fusion model outperforms the sub-band-only variant.These comparisons support the improved transformer and the fusion of sub-band and full-band information.
- 5.2. Ablation analysis: The frequency-domain dual-path configuration substantially outperforms the time-domain configuration, supporting frequency-domain features for the dual-path transformer.The comparison is between exp.4 and exp.1.
- 5.3. Results on the DNS dataset: On DNS, the proposed method outperforms the baseline, with a more significant improvement on reverberation data than the full-band model.Table 3 compares systems on DNS test sets with and without reverberation.
6. CONCLUSIONS
The paper proposes DPT-FSNet, a frequency-domain dual-path transformer that fuses sub-band and full-band information for speech enhancement. Experiments on Voice Bank + DEMAND and DNS show state-of-the-art performance at relatively small model size.
- 6. CONCLUSIONS: DPT-FSNet uses the intra transformer to model sub-band information and the inter transformer to model full-band information.The network is designed for frequency-domain speech enhancement.
- 6. CONCLUSIONS: Experiments on Voice Bank + DEMAND and DNS show that DPT-FSNet outperforms the current state of the art at a relatively small model size.The conclusion reports this result across both datasets.