Source-linked AI summary
TasNet: time-domain audio separation network for real-time, single-channel speech separation
Yi Luo, Nima Mesgarani
TL;DR
Speech separation in multi-talker environments remains challenging for real-time systems because STFT representations introduce phase-related reconstruction limits and long latency. TasNet instead separates nonnegative encoder outputs in the time domain and reconstructs sources with a learned decoder. The authors report better separation performance, lower computational cost, and substantially lower latency than prior systems.
Problem
Speech separation remains difficult in multi-talker environments, while STFT-based approaches face phase reconstruction limitations and long windows that constrain short-latency operation.
Method
TasNet uses an encoder-decoder framework to represent mixture waveforms with nonnegative encoder outputs, estimate source masks, and synthesize source waveforms with a learned decoder.
Results
TasNet outperforms state-of-the-art causal and noncausal speech separation systems while reducing computational cost and output latency.
Takeaways & Limitations
Waveform-domain separation can support real-time, low-latency speech separation, including operation on waveform segments as small as 5 ms.
Abstract
from arXiv · showhide
Robust speech processing in multi-talker environments requires effective speech separation. Recent deep learning systems have made significant progress toward solving this problem, yet it remains challenging particularly in real-time, short latency applications. Most methods attempt to construct a mask for each source in time-frequency representation of the mixture signal which is not necessarily an optimal representation for speech separation. In addition, time-frequency decomposition results in inherent problems such as phase/magnitude decoupling and long time window which is required to achieve sufficient frequency resolution. We propose Time-domain Audio Separation Network (TasNet) to overcome these limitations. We directly model the signal in the time-domain using an encoder-decoder framework and perform the source separation on nonnegative encoder outputs. This method removes the frequency decomposition step and reduces the separation problem to estimation of source masks on encoder outputs which is then synthesized by the decoder. Our system outperforms the current state-of-the-art causal and noncausal speech separation algorithms, reduces the computational cost of speech separation, and significantly reduces the minimum required latency of the output. This makes TasNet suitable for applications where low-power, real-time implementation is desirable such as in hearable and telecommunication devices.
1. INTRODUCTION
TasNet addresses the limitations of STFT-based speech separation by operating directly on waveform representations. Its encoder-decoder design enables low-latency processing while improving separation performance over state-of-the-art systems.
- Speech separation is difficult in crowded multi-talker environments despite recent advances from deep learning.
- STFT-based systems may use a nonoptimal representation and face coupled magnitude-phase reconstruction issues that limit separation performance.Many methods modify only magnitude while retaining the mixture phase, imposing an upper bound because reconstruction is not exact.
- STFT systems also require long analysis windows for frequency resolution, limiting their minimum latency to typically more than 32 ms for speech.
- TasNet directly models the mixture waveform with an encoder-decoder and estimates source masks over nonnegative encoder outputs before decoder synthesis.The waveform is represented as a nonnegative weighted sum of learned basis signals, whose weights are separated by source-specific masks.
- Waveform segments as short as 5 ms allow real-time, very-low-latency operation, while noncausal processing can further improve performance when real-time constraints are absent.The system also outperforms the state-of-the-art STFT-based system.
2. MODEL DESCRIPTION
TasNet separates speech directly in a learned, nonnegative time-domain representation rather than a fixed time-frequency decomposition. An encoder, mask-estimating separation network, and decoder estimate and reconstruct each source waveform.
- 2.1. Problem formulation: TasNet segments the mixture and sources into K nonoverlapping vectors of L samples, representing each segment with nonnegative basis-signal weights.The mixture weight is factorized over N learned basis signals.
- 2.1. Problem formulation: Source separation is reformulated as estimating each source’s mask-like weight vector from the mixture weights using element-wise multiplication.The mask represents each source’s relative contribution to the mixture weight matrix.
- 2.2. MODEL DESCRIPTION: The model combines an encoder, a separation module, and a decoder that reconstructs source waveforms from separated weights.The encoder-decoder pair forms a nonnegative autoencoder, while the separation subnetwork estimates one mask per source.
- 2.2.1. Encoder for mixture weight calculation: A gated convolutional encoder estimates nonnegative mixture weights using ReLU and Sigmoid branches, with normalization supporting scale invariance and more efficient LSTM training.The gated operation empirically performs better than using only ReLU or Sigmoid in the described system.
- 2.2.2. Separation network: A deep LSTM models dependencies across segment weight vectors, then a Softmax fully connected layer generates source masks.Identity skip connections are added between later LSTM layers to enhance gradient flow and accelerate training.
- 2.2.3. Decoder for waveform reconstruction: The decoder applies masks to the original mixture weights and synthesizes each source through matrix multiplication or learned 1-D transposed convolution.Recovered segments are rescaled, concatenated, and trained directly against waveform-level SI-SNR with permutation-invariant training.
3. EXPERIMENTS
Experiments evaluate TasNet on WSJ0-2mix using SI-SNRi and SDRi, compare causal and noncausal configurations with prior systems, and analyze latency and learned basis signals.
- Experimental setup: The evaluation uses WSJ0-2mix, with 30 hours of training, 10 hours of validation, and five hours of evaluation mixtures from unseen speakers.Mixtures use randomly selected WSJ0 utterances at 0–5 dB SNR.
- Experimental setup: The system is evaluated with SI-SNR improvement and SDR improvement, alongside DPCL++, PIT, and DANet baselines.TasNet-LSTM is causal, whereas TasNet-BLSTM is noncausal and cannot operate in real time.
- Separation results: TasNet significantly outperforms the state-of-the-art causal time-frequency system and surpasses prior systems under the noncausal configuration.The comparison includes two-stage systems such as DPCL++ and uPIT-BLSTM-ST.
- Latency analysis: The causal system achieves 5.23 ms total latency, with less than 0.23 ms average processing time per segment.The corresponding STFT-based systems require at least a 32 ms interval before processing begins.
- Basis-signal analysis: Basis signals show a continuous low-to-high frequency transition, increasing bandwidth with center frequency and allocating 60% below 1 kHz.This representation has higher low-frequency resolution than Mel and STFT bases.
4. CONCLUSION
The proposed system performs speech separation directly on sound waveforms using an autoencoder and source masks applied to learned mixture weights. It was reported to run 6 times faster than state-of-the-art STFT-based systems.
- The system directly operates on sound waveforms rather than using a time-frequency representation.Its autoencoder represents each waveform as a nonnegative weighted sum of learned basis signals.
- Source separation is performed by estimating masks applied to the mixture weights.These weights correspond to the learned basis signals in the autoencoder representation.
- 6 times faster than state-of-the-art STFT-based systems was reported for the proposed system.