Source-linked AI summary
Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
Yi Luo, Zhuo Chen, Takuya Yoshioka
TL;DR
Extremely long sequences make time-domain speech separation difficult for conventional RNNs and fixed-receptive-field CNNs. DPRNN splits inputs into chunks and alternates local and global recurrent processing, achieving new state-of-the-art WSJ0-2mix performance with a model 20 times smaller than the previous best system.
Problem
Time-domain speech separation requires modeling extremely long sequences, but conventional RNNs face optimization difficulties and 1-D CNNs lack utterance-level modeling beyond their fixed receptive fields.
Method
DPRNN splits sequential inputs into overlapping chunks and alternates intra-chunk local and inter-chunk global RNN processing, giving each operation a sublinear input length.
Results
A new state-of-the-art WSJ0-2mix result was achieved with a model 20 times smaller than the previous best system.
Takeaways & Limitations
DPRNN-TasNet indicates that strong WSJ0-2mix speech separation does not require enormous or complex models.
Takeaways & Limitations
For online processing, system latency is defined by chunk size, and the performance difference between online and offline settings is beyond the paper's scope.
Abstract
from arXiv · showhide
Recent studies in deep learning-based speech separation have proven the superiority of time-domain approaches to conventional time-frequency-based methods. Unlike the time-frequency domain approaches, the time-domain separation systems often receive input sequences consisting of a huge number of time steps, which introduces challenges for modeling extremely long sequences. Conventional recurrent neural networks (RNNs) are not effective for modeling such long sequences due to optimization difficulties, while one-dimensional convolutional neural networks (1-D CNNs) cannot perform utterance-level sequence modeling when its receptive field is smaller than the sequence length. In this paper, we propose dual-path recurrent neural network (DPRNN), a simple yet effective method for organizing RNN layers in a deep structure to model extremely long sequences. DPRNN splits the long sequential input into smaller chunks and applies intra- and inter-chunk operations iteratively, where the input length can be made proportional to the square root of the original sequence length in each operation. Experiments show that by replacing 1-D CNN with DPRNN and apply sample-level modeling in the time-domain audio separation network (TasNet), a new state-of-the-art performance on WSJ0-2mix is achieved with a 20 times smaller model than the previous best system.
1. INTRODUCTION
Time-domain speech separation must model extremely long sequences, exposing limitations in conventional RNNs and fixed-receptive-field CNNs. DPRNN addresses this by interleaving local and global recurrent processing over shorter chunks, improving TasNet performance while reducing model size.
- 1. INTRODUCTION: Time-domain separation systems often process tens of thousands of waveform samples, making long-term temporal modeling difficult.Adaptive front-end methods can also create longer representations when using smaller windows.
- 1. INTRODUCTION: Conventional RNNs face optimization difficulties on long sequences, while fixed-receptive-field 1-D CNNs cannot model utterance-level dependencies when sequences exceed their receptive fields.These limitations motivate an architecture that combines local and global sequence modeling.
- 1. INTRODUCTION: DPRNN splits sequences into shorter chunks and alternates intra-chunk RNNs for local modeling with inter-chunk RNNs for global utterance-level processing.When chunk size and chunk count are approximately equal, each recurrent operation receives a sublinear input length rather than the full sequence.
- 1. INTRODUCTION: Stacked DPRNN blocks repeatedly interleave local and global operations, unlike architectures whose first recurrent layer still processes the entire input.This keeps each intra- or inter-chunk RNN at the same sublinear input size across blocks.
- 1. INTRODUCTION: 0.7 dB (4.6%) relative SI-SNR improvement on WSJ0-2mix was achieved with a 49% smaller model by replacing TCN with DPRNN.Sample-level waveform separation further achieved new state-of-the-art performance with a model 20 times smaller than the previous best system.
2. DUAL-PATH RECURRENT NEURAL NETWORK
DPRNN models long sequences by repeatedly combining local intra-chunk processing with global inter-chunk processing. It segments inputs into a 3-D tensor, applies stacked DPRNN blocks, and reconstructs the sequence by overlap-add.
- Segmentation and overlap-add: DPRNN segments a sequential input into equal-sized chunks, concatenates them into a 3-D tensor, and later reconstructs the sequence with overlap-add.Chunks have length K and hop size P, with padding ensuring every input sample appears in K/P chunks.
- Block processing: Each DPRNN block first applies an intra-chunk RNN within individual chunks, then an inter-chunk RNN across aligned chunks.The intra-chunk RNN is bidirectional, while the inter-chunk RNN processes the aligned time steps across chunks.
- Block processing: Linear projections, layer normalization, and residual connections are applied around the RNN sub-modules before the next block receives the transformed tensor.The FC layer restores the feature dimension, and layer normalization is reported as important for generalization ability.
- Computational discussion: With 50% overlap, choosing K approximately proportional to the square root of L reduces each RNN’s input length from O(L) to O(sqrt(L)).The total intra- and inter-chunk input length is K + S, with S = ⌈2L/K⌉ + 1.
- Computational discussion: For online processing, the inter-chunk RNN can be unidirectional, with minimal system latency determined by the chunk size K.The paper does not evaluate the performance difference between online and offline settings.
3. EXPERIMENTAL PROCEDURES
The experiments apply DPRNN to TasNet for two-speaker speech separation and recognition. Evaluation uses WSJ0-2mix-style mixtures and measures both separation fidelity and recognition accuracy under specified training procedures.
- Systems and tasks: DPRNN is evaluated within TasNet, which uses a convolutional encoder, a separator that estimates source masks, and a transposed-convolutional decoder.The supplied passage identifies TasNet as an adaptive front-end system for speech separation.
- Dataset: The separation-only experiment uses WSJ0-2mix, with 30 hours of 8 kHz training data, 10 hours of validation data, and 5 hours of test data.The mixtures are generated from WSJ0 speech sets.
- Dataset: Evaluation mixtures use reverberation, randomly sampled speaker and microphone locations, SNRs from -5 to 5 dB, and additional noise at 10 to 20 dB SNR.The target is reverberant clean speech for both speakers.
- Training: Models are trained for 100 epochs on 4-second segments with Adam, gradient clipping, early stopping, and utterance-level permutation-invariant training.Training maximizes scale-invariant SNR, with a learning rate initialized at 1e−3 and decayed by 0.98 every two epochs.
- Evaluation: Performance is assessed using SDR improvement and SI-SNR improvement for fidelity, plus word error rate for recognition accuracy.WER is measured on both separated speakers.
4. RESULTS AND DISCUSSIONS
DPRNN improves TasNet-based speech separation on WSJ0-2mix over TCN and prior systems, while substantially reducing model size. It also outperforms TCN-TasNet in noisy reverberant separation and recognition.
- 4.6% improvement in separation performance with a 49% smaller model results from replacing TCN with DPRNN in TasNet.The comparison attributes this gain to DPRNN’s local-global modeling over CNN-based local-only modeling.
- A 2-sample filter length enables the best performance with DPRNN despite an encoder output exceeding 30000 frames.The corresponding sequence is described as extremely difficult or impossible for standard RNNs or CNNs to model.
- 20 times smaller model size accompanies a new record on SI-SNRi for DPRNN-TasNet versus FurcaNeXt on WSJ0-2mix.The authors state that this indicates strong WSJ0-2mix performance does not require enormous or complex models.
- DPRNN-TasNet significantly outperforms TCN-TasNet in both SI-SNRi and WER under noisy and reverberant conditions.The experiment uses a conventional hybrid recognition system trained on large-scale single-speaker noisy reverberant speech.
5. CONCLUSION
The paper presents DPRNN as an architecture for extremely long sequences that alternates local and global processing on shorter inputs. Applied to TasNet, it achieves state-of-the-art WSJ0-2mix performance with a much smaller model and remains effective in challenging acoustic conditions.
- DPRNN models extremely long sequences by splitting inputs into overlapping chunks and alternating intra-chunk and inter-chunk RNN processing.The intra-chunk operation is local, while the inter-chunk operation is global.
- Each RNN input can be proportional to the square root of the original sequence length, enabling sublinear processing and alleviating optimization challenges.
- Replacing 1-D CNN modules with deep DPRNN and using sample-level separation in TasNet yields state-of-the-art WSJ0-2mix performance with a 20 times smaller model.
- Experiments also report DPRNN effectiveness for noisy reverberant speech separation and recognition.