Source-linked AI summary
Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation
Daniel Stoller, Sebastian Ewert, Simon Dixon
TL;DR
Magnitude-spectrogram separation fixes spectral front-end choices and commonly omits source phase, while high-rate audio makes long sample-level context difficult. The paper proposes Wave-U-Net, a multi-scale time-domain U-Net with context-aware prediction and artifact-reducing upsampling. It reports favorable singing-voice separation under comparable training conditions and recommends rank-based SDR summaries, while noting dataset-size and SDR limitations.
Problem
Spectrogram separation depends on fixed front-end parameters and commonly estimates source magnitudes without source phase, while long temporal context is difficult at audio sample rates.
Method
Wave-U-Net performs end-to-end time-domain separation with a one-dimensional U-Net that repeatedly resamples feature maps to combine features across time scales.
Results
Wave-U-Net achieves favorable singing-voice separation compared with a reimplemented spectrogram U-Net trained under comparable settings.
Takeaways & Limitations
The architecture provides a time-domain alternative that combines long temporal context with multi-scale features and reduces boundary and upsampling artifacts.
Takeaways & Limitations
The comparison is limited by smaller training data than the referenced state-of-the-art model, and SDR evaluation remains problematic for quiet source segments.
Abstract
from arXiv · showhide
Models for audio source separation usually operate on the magnitude spectrum, which ignores phase information and makes separation performance dependant on hyper-parameters for the spectral front-end. Therefore, we investigate end-to-end source separation in the time-domain, which allows modelling phase information and avoids fixed spectral transformations. Due to high sampling rates for audio, employing a long temporal input context on the sample level is difficult, but required for high quality separation results because of long-range temporal correlations. In this context, we propose the Wave-U-Net, an adaptation of the U-Net to the one-dimensional time domain, which repeatedly resamples feature maps to compute and combine features at different time scales. We introduce further architectural improvements, including an output layer that enforces source additivity, an upsampling technique and a context-aware prediction framework to reduce output artifacts. Experiments for singing voice separation indicate that our architecture yields a performance comparable to a state-of-the-art spectrogram-based U-Net architecture, given the same data. Finally, we reveal a problem with outliers in the currently used SDR evaluation metrics and suggest reporting rank-based statistics to alleviate this problem.
1. INTRODUCTION
The paper proposes Wave-U-Net, a one-dimensional time-domain U-Net for source separation that models phase and long temporal context while addressing boundary and upsampling artifacts. It reports favorable singing-voice separation against a comparably trained spectrogram U-Net and flags limitations in SDR evaluation and dataset comparability.
- Motivation: Spectrogram methods estimate source magnitudes and typically reuse mixture phase, while fixed STFT parameters constrain the separation pipeline.These choices can produce incorrect phase for overlapping partials, and Griffin-Lim reconstruction is slow and may have no exact signal solution.
- Proposed approach: Wave-U-Net separates sources directly in the time domain using a one-dimensional U-Net that handles large temporal contexts.The architecture repeatedly resamples feature maps to combine information at different time scales.
- Architectural improvements: Additional context is supplied to reduce artifacts near output-window boundaries, where missing temporal context makes predictions difficult.The paper contrasts this with prior models that provide no additional input context.
- Architectural improvements: Linear interpolation followed by normal convolution replaces strided transposed-convolution upsampling to avoid high-frequency artifacts.The paper also investigates learned interpolation weights for upsampling feature maps.
- Results: Wave-U-Net achieves good multi-instrument and singing-voice separation, with singing-voice performance comparing favorably to a comparably trained reimplementation of the state-of-the-art network.The comparison uses similar training conditions rather than the larger, better-designed dataset used for the referenced state-of-the-art model.
2. RELATED WORK
Related work largely uses spectrograms or time-domain models with restricted context, latency, or resolution. Wave-U-Net instead uses multi-scale, lower-resolution feature maps to capture longer dependencies efficiently while retaining high-resolution features for local structure.
- Spectrogram-based approaches: Adaptive spectrogram front-ends can be trained jointly with separation networks but still omit source phase and reconstruct sources using mixture phase.This leaves phase-related limitations despite improved performance over fixed representations.
- Time-domain approaches: TasNet and MRCAE are prior general time-domain systems, but TasNet trades conceptual flexibility for low latency and MRCAE uses only one time resolution.MRCAE also processes short 23 ms input-output segments, limiting usable context.
- Speech enhancement models: SEGAN uses encoder-decoder pathways with skip connections, but prior work reports upsampling artifacts, boundary prediction problems, and uncertain transfer to harder separation tasks.Wave-U-Net addresses the context and upsampling issues with padding and alternative upsampling choices.
- Efficient temporal context: WaveNet-based denoising is parameter-efficient but memory-intensive because dilated-convolution feature maps retain the original sample-level resolution.Wave-U-Net computes longer-term dependencies on increasingly lower-resolution feature maps, saving memory and enabling more high-level features.
3. THE WAVE-U-NET MODEL
Wave-U-Net separates audio directly in the time domain by combining features across multiple temporal scales. Its architectural improvements address source additivity, boundary context, and upsampling artifacts.
- Architecture: The model predicts K source waveforms from a mixture waveform with C audio channels, optionally using extra context to predict only the input’s centre region.For context-aware variants, the mixture length exceeds the predicted source length.
- Architecture: Wave-U-Net downsamples feature maps to compute higher-level features at coarser time scales, then combines them with local high-resolution features through upsampling blocks.Each successive level operates at half the previous time resolution, producing multi-scale features for source prediction.
- Architectural improvements: Linear interpolation replaces strided transposed convolution for upsampling, avoiding zero insertion and border extrapolation that can create high-frequency artifacts.The method retains boundary entries and interpolates only between known neighbouring feature values.
- Architectural improvements: A difference output layer enforces source additivity by estimating K−1 sources and computing the final source as the mixture minus their sum.This constrains the predicted sources to satisfy the mixture decomposition assumed by the task.
- Architectural improvements: Unpadded convolutions with additional input context avoid artificial silence at excerpt boundaries and reduce errors caused by incorrect temporal context.The output is smaller than the input because predictions are made where the convolution has valid audio context.
- Architectural improvements: Learned upsampling generalizes linear interpolation by using sigmoid-constrained feature-specific weights, allowing convex combinations beyond equal weighting.The operation can be implemented as a one-dimensional convolution with filters of size two and no padding.
4. EXPERIMENTS
The experiments evaluate Wave-U-Net on singing voice and multi-instrument separation using MUSDB-based training and validation splits. They also test successive architectural improvements and compare a Wave-U-Net variant with a spectrogram-based U-Net under matched conditions.
- Tasks: Wave-U-Net is evaluated on singing voice separation and music separation for bass, drums, guitar, vocals, and other instruments.The categories follow the SiSec separation campaign definition.
- Data: Training uses 75 MUSDB tracks, validation uses 25 tracks for early stopping, and final evaluation uses the 50-song MUSDB test partition.The singing-voice training set additionally includes the full CCMixter database.
- Data: The training pipeline applies source-level amplitude augmentation, forms mixtures by summing sources, and downsamples audio to 22050 Hz without further preprocessing.Signals are converted to mono except for stereo models.
- Training procedure: Models are trained with batch MSE loss and ADAM, followed by validation-based early stopping and fine-tuning at a lower learning rate.The best-validation-loss model is selected after the fine-tuning stage.
- Model comparisons: A baseline and models M2–M5 isolate the effects of the difference output layer, context and resampling, stereo channels, and learned upsampling.The best model among these variants is then applied to multi-instrument separation.
- Model comparisons: A spectrogram-based U-Net and Wave-U-Net comparison model are trained under the same conditions using mono signals, 8192 Hz sampling, and audio-based MSE loss.The comparison model is sized to have an output size similar to the spectrogram U-Net.
5. RESULTS
Results expose both evaluation and boundary-artifact issues while showing strong separation performance. Rank-based SDR statistics better handle outliers, and context-aware prediction reduces border problems.
- Evaluation metrics: Segment-wise SDR outliers, especially for silent or near-silent vocals, can greatly distort mean-based evaluation results.Silent segments may be excluded, while near-silent segments can receive very low SDR despite quiet output being perceptually acceptable.
- Evaluation metrics: Median and median absolute deviation provide more robust summaries than mean and standard deviation for the non-normal SDR distribution.The median is robust to outliers, while MAD describes the distribution's spread using rank-based statistics.
- Evaluation metrics: Increasing segment duration beyond one second removes many, but not all, SDR outliers, while requiring more memory and still penalising silent-section errors.Longer segments alleviate the issue only partially.
- Model comparison: Context noticeably improves singing voice separation, likely because additional input enables better predictions near output borders.Model M3 shows this improvement in the singing voice experiments.
- Model comparison: Stereo modelling improves accompaniment separation, while learned upsampling slightly raises median vocal SDR but slightly lowers mean vocal SDR.Unconstrained convolutions improved performance further but introduced high-frequency artifacts.
- Model comparison: The Wave-U-Net compares favourably with the state-of-the-art comparison architecture under matched experimental conditions, with higher mean and median SDR scores than U7a except for mean vocal SDR.The mean vocal exception reflects more outlier segments despite better output for the majority of segments.
- Model comparison: In the SiSec campaign, M4 performs better or as well as almost all other vocal-separation systems, although several better systems used 800 additional training songs.M4 also separates accompaniment well, though slightly less effectively than vocals.
- Qualitative results: Without additional input context, concatenated segment predictions create border inconsistencies, including abrupt loudness changes and cut-off vocal content.Figure 4 visualises these artifacts in a vocal estimate spectrogram.
6. DISCUSSION AND CONCLUSION
The paper presents Wave-U-Net as an end-to-end time-domain separator and reports strong separation results, while identifying architectural and evaluation limitations that remain open.
- Wave-U-Net performs end-to-end audio source separation without pre- or postprocessing for singing voice and multi-instrument tasks.
- Repeated downsampling and convolution combine high- and low-level features across time scales to process long temporal context.
- Under comparable training settings, Wave-U-Net outperforms the state-of-the-art spectrogram-based U-Net architecture.
- Future work includes larger comparable datasets, improved raw-audio losses, and signal decompositions inspired by filterbanks or TasNet.
- Proper temporal context and linear upsampling address boundary and high-frequency artifacts associated with earlier architectures.
- SDR evaluation produces outliers for quiet source segments, motivating rank-based metrics, but perceptual evaluation problems remain unresolved.