Source-linked AI summary
Sudo rm -rf: Efficient Networks for Universal Audio Source Separation
Efthymios Tzinis, Zhepei Wang, Paris Smaragdis
TL;DR
Audio source separation has advanced, but many effective methods remain costly to train and deploy on constrained devices. The paper introduces SuDoRM-RF, an end-to-end mask-based network built around multi-resolution temporal processing. Across speech and environmental sound separation, it matches or surpasses several state-of-the-art models while using substantially fewer computational resources.
Problem
Existing audio source-separation methods can require substantial computation, memory, and training cost, limiting use on embedded and other resource-constrained devices.
Method
SuDoRM-RF performs end-to-end mask-based separation using an encoder, a multi-resolution U-ConvBlock separator with successive downsampling and upsampling, and a decoder.
Results
SuDoRM-RF models match or outperform several state-of-the-art systems while requiring substantially fewer FLOPs, memory, and time across speech and environmental sound separation tasks.
Takeaways & Limitations
The architecture provides a lower-cost alternative for universal audio source separation and is intended for training and deployment under limited computational resources.
Abstract
from arXiv · showhide
In this paper, we present an efficient neural network for end-to-end general purpose audio source separation. Specifically, the backbone structure of this convolutional network is the SUccessive DOwnsampling and Resampling of Multi-Resolution Features (SuDoRMRF) as well as their aggregation which is performed through simple one-dimensional convolutions. In this way, we are able to obtain high quality audio source separation with limited number of floating point operations, memory requirements, number of parameters and latency. Our experiments on both speech and environmental sound separation datasets show that SuDoRMRF performs comparably and even surpasses various state-of-the-art approaches with significantly higher computational resource requirements.
1. INTRODUCTION
Efficient audio source-separation architectures remain underdeveloped despite advances in low-resource models elsewhere. The paper proposes SuDoRM-RF to reduce computational demands while maintaining competitive separation quality.
- Computationally intensive source-separation methods hinder deployment on embedded systems and other resource-constrained devices.Training these systems can also impose substantial computational costs.
- Efficient architectures have reduced requirements in image-domain tasks, but large-dilation convolutions may introduce artifacts.Depth-wise separable convolutions and lightweight variants such as MobileNets motivate analogous audio architectures.
- Efficient audio source separation has received relatively little attention compared with image-domain efficiency research.Prior audio work reduced FLOPs, latency, parameters, or model representation, but did not address the full resource profile together.
- SuDoRM-RF uses successive downsampling and resampling of multi-resolution features with depth-wise convolutions, avoiding multiple stacked dilated layers.The approach considers FLOPs, latency, and total memory requirements together.
- On speech and environmental sound separation tasks, the proposed model performs comparably to or better than recent state-of-the-art models with substantially lower computational requirements.
2. SUDO RM -RF NETWORK ARCHITECTURE
SuDoRM-RF performs end-to-end mask-based separation by encoding mixtures, processing multi-resolution features with U-ConvBlocks, and decoding source representations back to waveforms. Its U-ConvBlocks use successive downsampling and upsampling to expand receptive fields while preserving output resolution.
- Overall pipeline: The architecture encodes a raw mixture, estimates source masks in latent space, applies them element-wise, and decodes each masked representation into a time-domain source.The encoder, separator, and decoder form the end-to-end separation pipeline.
- Separator: The separator projects encoded features into a new channel space, repeatedly applies B U-ConvBlocks, aggregates channels, and estimates masks with a softmax across sources.The resulting masks lie in [0, 1] and sum to one across the source dimension.
- U-ConvBlock: U-ConvBlocks extract and aggregate multiple temporal resolutions through successive downsampling and upsampling, while preserving temporal resolution at the block output.The design replaces regular or dilated depth-wise processing with a U-Net-like multi-resolution structure.
- U-ConvBlock: Each U-ConvBlock increases the network’s effective receptive field multiplicatively through temporal subsampling, helping capture long-term dependencies with fewer layers.The block uses depth-wise convolutions, skip connections, and nearest-neighbor temporal interpolation.
- Decoder: The decoder uses a separate one-dimensional transposed convolution for each source to transform latent source representations back into the time domain.This source-specific decoder is the final reconstruction stage.
3. EXPERIMENTAL SETUP
The experiments evaluate speech and non-speech separation under standardized mixture generation, training, preprocessing, and resource-profiling procedures. Models are compared with recent state-of-the-art systems while tracking computational bottlenecks relevant to constrained deployment.
- Datasets and mixture generation: Speech experiments use WSJ0-2mix with two speakers mixed at random SNRs between −5 and 5dB.The mixtures are generated from Wall Street Journal speech utterances.
- Data preprocessing and generation: Each epoch generates 20,000 new training mixtures, while validation and test sets each contain 3,000 mixtures.Audio is randomly cropped into 4-second segments, mixed at random SNRs, downsampled to 8kHz, mean-subtracted, and standardized by mixture deviation.
- Training and evaluation: Training runs for 120 epochs with batch size 4 using negative permutation-invariant SI-SDR as the loss.The loss is defined between clean sources and estimated sources.
- Training and evaluation: SI-SDR improvement measures the gain obtained by using the estimated signal instead of the mixture signal.The source permutation is selected to maximize SI-SDR before evaluation.
- Baselines: Comparisons use best configurations and suggested training processes from recent speech, universal, and music separation approaches.Demucs is reduced to 80 channels so it can be trained on a single GPU.
- Resource evaluation: The study profiles FLOPs, trainable parameters, single-pass memory allocation, and process completion time as computational bottlenecks.Profiling uses an Intel Xeon CPU E5-2695 v3 and an Nvidia Tesla K80 GPU.
4. RESULTS & DISCUSSION
SuDoRM-RF models achieve competitive separation performance while using substantially fewer computational resources, parameters, and memory than several state-of-the-art alternatives. Their cost-efficiency extends to both inference and training, with performance scaling across model sizes.
- Overall performance: SuDoRM-RF models match or outperform several state-of-the-art models while requiring orders of magnitude fewer computational resources.Table 1 compares SI-SDRi performance with inference and backward-update requirements for one second of audio.
- Floating-point operations: SuDoRM-RF models become more Pareto efficient as U-ConvBlocks increase from 4 to 8 to 16, outperforming alternatives in SI-SDRi versus GFLOPs and time.DPRNN requires at least 45 times more FLOPs per pass than SuDoRM-RF 0.25x while performing worse after the same number of epochs.
- Cost-efficient training: SuDoRM-RF models achieve better speech-separation performance with significantly fewer training FLOPs than other models.SuDoRM-RF 1.0x reaches approximately 16 dB SI-SDRi, compared with approximately 10 dB for DPRNN under the same training-FLOP budget.
- Trainable parameters: SuDoRM-RF architectures use orders of magnitude fewer parameters than U-net architectures such as Demucs, while SuDoRM-RF 1.0x matches DPRNN's relatively low parameter count.The architecture's upsampling procedure does not require additional parameters.
- Memory requirements: SuDoRM-RF models are more Pareto efficient in memory than dilated-convolution architectures and require orders of magnitude less memory, especially during backward updates.The comparison includes ConvTasNet and Two-Step TDCN, whose receptive-field growth requires increased network depth.
5. CONCLUSIONS
SuDoRM-RF is an efficient architecture for universal sound source separation that extracts multi-resolution temporal features and aggregates them with nonparametric interpolation. It achieves similar or better performance than recent state-of-the-art models while using fewer computational resources.
- SuDoRM-RF extracts multi-resolution temporal features through successive depth-wise convolutional downsampling and aggregates them using nonparametric interpolation.This design reduces the number of layers needed to capture long-term temporal dependencies.
- SuDoRM-RF models perform similarly or better than recent state-of-the-art source-separation models.
- SuDoRM-RF requires significantly fewer FLOPs, memory, and time than recent state-of-the-art models.