Source-linked AI summary
EConv-TasNet: Efficient Conv-TasNet for Effective Speech Separation
Pei-Chun Chang, Chuan-Yi Liu
TL;DR
Resource-intensive separator designs can limit speech separation on resource-constrained devices. eConv-TasNet adds GES and MGFA to Conv-TasNet, and experiments report improved accuracy and inference speed over the original alongside comparisons with state-of-the-art methods on three datasets.
Problem
Many advanced separator architectures introduce high computational cost and complexity, limiting deployment in resource-constrained scenarios.
Source: §I · pp. 1–2Method
eConv-TasNet extends Conv-TasNet with GES for intermediate group-level speaker representations and MGFA for their progressive aggregation and refinement.
Source: A. Overview · p. 2, §I · p. 2Results
Experiments report that eConv-TasNet outperforms original Conv-TasNet in separation accuracy and inference speed, with comparisons to state-of-the-art methods on WSJ0-2mix, WHAM!, and Libri2Mix.
Source: §I · p. 2Takeaways & Limitations
The proposed design targets improved effectiveness and efficiency while reducing computational cost and model complexity.
Source: §I · p. 2, A. Overview · p. 2Conv-TasNet's direct summation of X × R unit representations is described as computationally redundant and may affect information efficiency and diversity.
Source: B. The limitation of Conv-TasNet · pp. 2–3
Abstract
from arXiv · showhide
Conv-TasNet has served as a strong baseline for time-domain speech separation, and many studies have extended it with advanced architectures such as dual-path networks, U-Nets, and attention mechanisms. However, these methods often introduce high computational cost and complexity, limiting their deployment in resource-constrained scenarios. To address this issue, we propose eConv-TasNet, an efficient variant of Conv-TasNet that improves both effectiveness and efficiency without relying on resource-intensive modules. The proposed model consists of a group-wise early-splitting (GES) module and a multi-group feature aggregation (MGFA) module. GES generates discriminative speaker embeddings at intermediate stages, while MGFA progressively aggregates these group-level representations for refined mask estimation. Experimental results show that eConv-TasNet reduces model size by 22.4%, accelerates inference by 18.9%, and improves SI-SNRi by 14.0%-28.0% across three public benchmarks. Moreover, it achieves competitive performance compared with state-of-the-art methods while requiring significantly fewer parameters and lower inference cost. These results demonstrate a favorable efficiency-effectiveness trade-off for edge deployment.
I. INTRODUCTION
Speech separation extracts individual speakers from a mixed signal,, and prior work spans TF-domain processing, time-domain TasNet variants, and richer separator architectures. eConv-TasNet is proposed to improve separation effectiveness and efficiency while avoiding high computational costs.
I. INTRODUCTION
- TF-domain methods use STFT/iSTFT, whose fixed window and stride constrain resolution, while mixture-phase reuse can degrade separation and phase-aware methods add computational cost –.These limitations challenge real-time, low-latency, and edge-device use.Source: §I · p. 1
- TasNet separates raw waveforms with an encoder, separator, and decoder, while Conv-TasNet uses stacked TCN units and improves on original TasNet in accuracy, inference speed, and computational cost.Its separator estimates speaker masks from latent features.Source: §I · p. 1
- Prior separators use dual-path, multi-scale/U-Net, attention, recurrent, and hybrid designs [12]–, but generally incur memory and computational costs that challenge edge deployment.The paper motivates a fully convolutional Conv-TasNet-based design that avoids high computational costs and inference time.Source: §I · pp. 1–2
- GES extracts speaker-dependent group-level representations to reduce cost and increase discriminability, while MGFA progressively refines and aggregates them into compact embeddings.Together, the modules form eConv-TasNet, an efficient Conv-TasNet extension for speech separation.Source: §I · p. 2
- Experiments report that eConv-TasNet outperforms original Conv-TasNet in separation accuracy and inference speed, with comparisons to state-of-the-art methods on WSJ0-2mix, WHAM!, and Libri2Mix.The authors also report potential for practical application on edge devices.Source: §I · p. 2
A. Overview
eConv-TasNet retains Conv-TasNet’s encoder-decoder architecture while adding GES and MGFA to improve speaker representations and balance separation performance with computational cost. Its design addresses redundancy from aggregating all TCN-unit outputs.
A. Overview
- eConv-TasNet retains Conv-TasNet’s encoder-decoder framework and adds GES and MGFA to improve speaker representations while reducing computational cost and model complexity.The design uses progressively refined speaker embeddings from fewer representations.Source: A. Overview · p. 2
B. The limitation of Conv-TasNet
- Conv-TasNet sums all X × R unit-output representations for mask estimation, which the paper identifies as computationally redundant and inefficient in resource-constrained scenarios.The separator stacks X TCN-units and repeats them R times; each unit produces representations F and O.Source: B. The limitation of Conv-TasNet · p. 2
- Aggregating similar representations from all units may accumulate redundancy, motivating speaker-information extraction from each unit group followed by progressive refinement instead of direct summation.The paper notes that nearby units make similar contributions.Source: B. The limitation of Conv-TasNet · pp. 2–3
C. Efficient Conv-TasNet (eConv-TasNet)
eConv-TasNet groups eTCN-units and applies GES to each group output before MGFA aggregates the speaker representations for mask estimation. GES splits group-level features rather than every individual unit’s output.
C. Efficient Conv-TasNet (eConv-TasNet)
- eConv-TasNet groups X eTCN-units and repeats the group R times, then applies GES to group outputs and MGFA to aggregate speaker representations for mask estimation.Only the last eTCN-unit in each group uses a skip-connection path, reducing learnable parameters and computational cost relative to the original Conv-TasNet.Source: C. Efficient Conv-TasNet (eConv-TasNet) · p. 3
1) Group-wise early-splitting (GES):
- GES splits only the final output representation of each stacked eTCN-unit group, rather than splitting every unit, to reduce cost while preserving group-level features for subsequent processing.The module uses a GLU followed by a point-wise convolution and group normalization; its speaker-dependent representations are forwarded to MGFA.Source: 1) Group-wise early-splitting (GES) · p. 3, Figure 4 · p. 3
2) Multi-group feature aggregation (MGFA):
MGFA progressively refines speaker-dependent embeddings across groups using an exponentially weighted moving average, balancing current representations against previously accumulated embeddings.
2) Multi-group feature aggregation (MGFA):
- MGFA uses an exponentially weighted moving average to refine speaker embeddings across groups, balancing current group features with prior embeddings before the final embedding estimates masks.The coefficient β controls the contributions of the current and previous refined embeddings.Source: 2) Multi-group feature aggregation… · p. 3
III. EXPERIMENT SETUP
The evaluation covers two-speaker separation on WSJ0-2mix, WHAM!, and Libri2Mix, using held-out test speakers and full-waveform evaluation. The datasets differ in noise conditions and data construction.
A. Datasets
- Experiments evaluate two-speaker separation on WSJ0-2mix, WHAM!, and Libri2Mix, with test mixtures made from speakers absent from training.All datasets were resampled to 8 kHz; training used random 4-second segments, while validation and evaluation used full waveforms of varying lengths.Source: A. Datasets · p. 3
- WSJ0-2mix provides 30, 10, and 5 hours for training, validation, and testing, with two-speaker mixtures sampled at relative SNRs from [-5, 5] dB.The splits are drawn from the si tr s, si dt 05, and si et 05 WSJ0 sets, respectively.Source: A. Datasets · p. 3
- WHAM! adds environmental noise to WSJ0-2mix, with mixture SNRs uniformly sampled from [-6, 3] dB, making it more challenging than the original dataset.The noise recordings include scenes such as cafes, restaurants, and bars.Source: A. Datasets · pp. 3–4
- Libri2Mix has 58, 11, and 11 hours of training, validation, and testing data; this study uses its clean version without noise.Speech is sampled from LibriSpeech train-100, and mixtures use two speakers with loudness adjusted between -25 and -33 dB LUFS.Source: A. Datasets · p. 4
B. Baseline setup
The experiments use Conv-TasNet as the baseline and describe the training setup for eConv-TasNet, including its shared hyperparameters, optimizer, learning-rate schedule, objective, and augmentation.
B. Baseline setup
- The Conv-TasNet baseline uses 256 encoder filters with 16 coefficients and a separator of eight-by-three TCN units.Its speaker masks use sigmoid activation; the representation and embedding dimension is 128, with hidden size 512 and kernel size 3.Source: B. Baseline setup · p. 4
C. Implementation details
- eConv-TasNet uses the original Conv-TasNet settings for N, L, B/Sc, H, P, X, and R, with MGFA weight decay β = 0.2.The passage specifies N = 512, L = 16, B/Sc = 128, H = 512, P = 3, X = 8, and R = 3.Source: C. Implementation details · p. 4
- Training runs for up to 200 epochs with AdamW, a 1e−3 initial learning rate, warm-up, and validation-triggered decay.The learning rate decays by 0.8 after validation loss fails to improve for three consecutive epochs; warm-up lasts 1,000 steps in the first epoch.Source: C. Implementation details · p. 4
- Permutation invariant training and negative SI-SNR provide the training strategy and objective, while dynamic mixing increases training-data diversity.The passage identifies dynamic mixing as data augmentation and notes that the objective uses SI-SNRi.Source: C. Implementation details · p. 4
IV. EXPERIMENTAL RESULTS
On WSJ0-2mix, the ablations assess the effects of GES and MGFA, with the joint model improving separation while reducing model size and inference cost relative to Conv-TasNet.
IV. EXPERIMENTAL RESULTS
- SI-SNRi and SDRi assess separation quality, while single-thread CPU RTF and MACs measure inference speed and computation.MACs are calculated using a 16,000-sample mixture on a 13th Gen Intel Core i5-1335U CPU.Source: §IV · p. 4
A. Ablation studies
- Dynamic mixing raises reimplemented Conv-TasNet SI-SNRi from 15.3 to 16.0 dB without increasing model size or inference cost, while ES reaches 16.4 dB at higher cost.These ablations use the WSJ0-2mix dataset.Source: A. Ablation studies · p. 4
- Replacing ES with GES raises SI-SNRi to 17.7 dB, reduces model size to 3.8M parameters and RTF to 0.13, yielding a 1.3 dB SI-SNRi gain while reducing parameter count and inference cost.Source: A. Ablation studies · p. 4
- MGFA consistently improves separation across settings by aggregating complementary group-level speaker information and refining representations for mask estimation.The aggregation is described as progressively refining speaker embeddings.Source: A. Ablation studies · p. 5
- 12.2% higher SI-SNRi, 22.4% smaller model size, and 18.9% lower RTF are reported for eConv-TasNet versus Conv-TasNet.The best MGFA weight-decay setting is β = 0.2, yielding 18.4 dB SI-SNRi and a 0.7 dB gain over GES-only.Source: A. Ablation studies · p. 5
B. Performance gain analysis
On WSJ0-2mix, eConv-TasNet’s SI-SNRi gains are broadly distributed across utterances and vary with baseline difficulty, with the smallest median gain on the most difficult mixtures.
B. Performance gain analysis
- The SI-SNRi histogram shifts toward higher values, with its peak moving from approximately 17–18 dB for Conv-TasNet to 19–21 dB for eConv-TasNet.The passage reports that the shift reflects improvement across the test set.Source: B. Performance gain analysis · p. 5
- The largest median improvement is 5.67 dB for baseline SI-SNRi of 10–12 dB, while mixtures below 10 dB gain 0.66 dB.The 14–16 dB and 12–14 dB groups gain 3.99 dB and 3.92 dB, respectively; most extremely difficult cases still show positive gains.Source: B. Performance gain analysis · p. 5
- eConv-TasNet outperforms Conv-TasNet on 97% of test utterances, with a 2.58 dB median gain across 2,736 already well-separated instances.The reported improvements span a wide range of separation difficulties.Source: B. Performance gain analysis · p. 5
C. Performance-efficiency trade-off analysis
On WSJ0-2mix, larger eConv-TasNet configurations generally improve separation at greater model and inference cost, while lightweight versions can exceed the Conv-TasNet baseline.
C. Performance-efficiency trade-off analysis
- Increasing feature-channel and hidden-layer dimensions and the number of TCN blocks generally improves SI-SNRi at the expense of model size and inference cost.The compared configurations are evaluated on WSJ0-2mix.Source: C. Performance-efficiency trade-off… · pp. 5–6
- 18.8 dB SI-SNRi is achieved by the best configuration, 3.5 dB above baseline Conv-TasNet, with comparable model size and inference cost.The adopted configuration uses N = 512, H = 512, X = 8, and R = 3 to balance quality and cost.Source: C. Performance-efficiency trade-off… · p. 6
- A lightweight eConv-TasNet configuration with only 1.6M parameters can outperform baseline Conv-TasNet.This result highlights parameter efficiency across the tested configurations.Source: C. Performance-efficiency trade-off… · p. 6
D. Comparison with state-of-the-art methods
Existing separators often favor either efficiency or separation quality, while MACs alone do not capture real-time feasibility. eConv-TasNet offers a more favorable efficiency–effectiveness trade-off than Conv-TasNet, with competitive real-time inference speed and lower computational cost.
D. Comparison with state-of-the-art methods
- Conv-TasNet has significant runtime efficiency, yet its model size, MACs, and separation performance leave suboptimal trade-offs against newer specialized architectures.The passage cites Table III for this comparison.Source: D. Comparison with state-of-the-art… · p. 6
- Efficient models show uneven trade-offs: SuDoRM-RF has low MACs but limited performance gains, while A-FRCNN variants can require greater complexity for improved performance.DPRNN and SepFormer achieve strong separation performance through dual-path modeling and Transformers, but with increased computational overhead and inference latency.Source: D. Comparison with state-of-the-art… · p. 6
- MACs alone do not establish real-time feasibility: even compact TDANet and SepReformer-T variants have non-negligible runtime latency.Inference latency can dominate the edge-deployment constraint.Source: D. Comparison with state-of-the-art… · p. 6
- eConv-TasNet improves separation performance over Conv-TasNet while maintaining low computational cost and competitive real-time inference speed.The conclusion also reports a more favorable balance of performance and computational cost than state-of-the-art methods.Source: D. Comparison with state-of-the-art… · p. 6, §V · p. 6
V. CONCLUSION
- eConv-TasNet integrates GES and MGFA into Conv-TasNet to form speaker representations at intermediate separator stages and progressively refine them for mask estimation.The conclusion describes this stepwise fusion as improving feature consistency across stages while reducing redundant later-layer computation.Source: §V · p. 6