Source-linked AI summary
Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation
Jingjing Chen, Qirong Mao, Dong Liu
TL;DR
Conventional speech separation models condition on sequence context indirectly, motivating a more direct approach. The paper proposes DPTNet, which combines an improved transformer with recurrent order modeling and dual-path processing for long sequences. Experiments on benchmark datasets report state-of-the-art performance, including 20.6 dB SDR on WSJ0-2mix.
Problem
RNN- and CNN-based speech separation models condition on context indirectly, while transformers face difficulty modeling the extremely long sequences used in end-to-end systems.
Method
DPTNet uses an improved transformer with direct element interaction, an integrated RNN for order information without positional encodings, and dual-path processing for long sequences.
Results
20.6 dB SDR is reported on the public WSJ0-2mix data corpus, and experiments on benchmark datasets show performance exceeding current state-of-the-art methods.
Takeaways & Limitations
DPTNet provides direct context-aware modeling for end-to-end monaural speech separation while efficiently handling extremely long speech sequences.
Takeaways & Limitations
The dual-path structure slightly limits direct interaction because some sequence elements communicate through intermediate elements.
Abstract
from arXiv · showhide
The dominant speech separation models are based on complex recurrent or convolution neural network that model speech sequences indirectly conditioning on context, such as passing information through many intermediate states in recurrent neural network, leading to suboptimal separation performance. In this paper, we propose a dual-path transformer network (DPTNet) for end-to-end speech separation, which introduces direct context-awareness in the modeling for speech sequences. By introduces a improved transformer, elements in speech sequences can interact directly, which enables DPTNet can model for the speech sequences with direct context-awareness. The improved transformer in our approach learns the order information of the speech sequences without positional encodings by incorporating a recurrent neural network into the original transformer. In addition, the structure of dual paths makes our model efficient for extremely long speech sequence modeling. Extensive experiments on benchmark datasets show that our approach outperforms the current state-of-the-arts (20.6 dB SDR on the public WSj0-2mix data corpus).
1. Introduction
Monaural speech separation is difficult because conventional RNN- and CNN-based models condition on context indirectly, while transformers face sequence-length challenges. DPTNet addresses these issues through direct context-aware modeling, recurrent order learning, and dual-path processing for long sequences.
- Monaural speech separation remains difficult for automatic systems despite its importance in applications such as improving automatic speech recognition.The task is especially challenging when only a single mixed-speech recording is available.
- RNN- and CNN-based separation models condition on context indirectly, while transformers allow direct element interaction but typically handle only sequences of hundreds of elements.RNNs pass information through intermediate states, whereas CNNs are constrained by limited receptive fields.
- DPTNet introduces an improved transformer for direct context-aware modeling in end-to-end monaural speech separation.The method is presented as a response to limitations in indirect context modeling.
- Direct interaction among speech-sequence elements is proposed as a new form of context-aware modeling for speech separation.The paper identifies this as its first introduction of direct context-aware modeling into speech separation.
- The method integrates an RNN into the transformer to learn sequence order without positional encodings and embeds it in a dual-path network for extremely long sequences.The dual-path structure is intended to improve efficiency for long speech-sequence modeling.
- 20.6 dB SDR is reported on the public WSj0-2mix data corpus, with experiments described as outperforming current state-of-the-art methods.The reported result comes from extensive experiments on benchmark datasets.
2. Speech separation with dual-path transformer network
DPTNet separates monaural speech through an encoder–separation–decoder pipeline whose separation layer combines transformer processing with dual paths for local and global sequence modeling. Its improved transformer enables direct context-aware interactions and replaces positional encoding with an RNN-based order mechanism, while dual paths support efficient long-sequence modeling.
- System overview: The system encodes mixture-waveform segments, constructs source masks in the separation layer, and decodes masked features into separated waveforms.The encoder converts segments into intermediate features; the decoder reconstructs source signals after masking.
- System overview: The separation layer performs segmentation, dual-path transformer processing, and overlap-add reconstruction.Segmentation creates overlapping chunks of length K with hop size H, producing a 3-D tensor for processing.
- Improved transformer: Transformer self-attention lets speech-sequence elements interact directly, introducing direct context-aware modeling without intermediate transmission.The transformer uses attention, multi-head attention, feed-forward, residual, and normalization modules.
- Improved transformer: The improved transformer replaces the feed-forward network’s first fully connected layer with an RNN to learn sequence order without positional encodings.The paper reports that positional encodings are unsuitable for the dual-path network and can cause training divergence.
- Dual-path processing: Each dual-path transformer uses intra-transformer and inter-transformer blocks to model local chunk information and global dependencies, respectively.The intra-transformer operates along the chunk dimension, while the inter-transformer summarizes information across chunks.
- Dual-path processing: Dual paths make extremely long speech-sequence modeling efficient, but restrict some direct interactions because elements may communicate through intermediate elements.The paper characterizes this interaction restriction as a slight negative impact that is outweighed by the efficiency benefit.
3. Experiment
The experiments evaluate DPTNet on two-speaker mixtures from WSJ0-2mix and LS-2mix, using SI-SNR-optimized utterance-level permutation invariant training and standard separation metrics. The model is trained with specified dual-path transformer and optimization settings.
- Datasets: DPTNet is evaluated on two-speaker speech separation using the WSJ0-2mix and LS-2mix datasets.WSJ0-2mix uses mixtures at SNRs from -5 dB to 5 dB, while LS-2mix uses SNRs from 0 dB to 5 dB and contains 20,000/5,000/3,000 train/validation/test utterances.
- Model setup: The encoder and decoder use 2-sample windows with 50% stride and 64 filters; the separation layer uses 6 dual-path transformers with 4 parallel attention layers.
- Optimization: Training runs for up to 100 epochs on 4-second segments with Adam, gradient clipping at maximum L2-norm 5, and validation-based early stopping after 10 epochs without loss improvement.
- Optimization: The learning rate increases linearly during the first 4000 warmup steps and then decays by 0.98 every two epochs.The schedule uses empirically selected tunable scalars k1 = 0.2 and k2 = 4e−4.
- Objective: The model is trained with utterance-level permutation invariant training to maximize scale-invariant source-to-noise ratio.SI-SNR calculation uses zero-mean clean and estimated sources.
4. Performance evaluation
DPTNet achieves strong separation performance on WSJ0-2mix and remains superior to reproduced baselines on the more difficult LS-2mix dataset. The WSJ0-2mix result also combines high scores with a small model size.
- Metrics: The reported evaluation metrics are scale-invariant signal-to-noise ratio and signal-to-distortion ratio.
- WSJ0-2mix: 20.2 dB SI-SNR and 20.6 dB SDR on WSJ0-2mix establish new state-of-the-art performance for DPTNet.
- WSJ0-2mix: On WSJ0-2mix, DPTNet maintains a small model size alongside its reported separation performance.
- LS-2mix: DPTNet is significantly superior to DPRNN on LS-2mix, where the mixtures are described as more difficult to separate than WSJ0-2mix mixtures.
5. Conclusion and future work
The paper concludes that DPTNet effectively performs end-to-end multi-speaker monaural speech separation through direct context-aware sequence modeling. Future work will seek to model long speech feature sequences directly without the dual-path structure.
- Conclusion: DPTNet models speech sequences directly conditioning on context for end-to-end multi-speaker monaural speech separation.
- Conclusion: The model learns sequence order without positional encodings and effectively models extremely long speech-signal sequences.
- Conclusion: Experiments on two benchmark datasets demonstrate the proposed model’s effectiveness and yield a new state-of-the-art result on WSJ0-2mix.
- Future work: Future work will investigate directly modeling long speech feature sequences without the dual-path structure.The paper describes this direction as potentially improving separation performance.