Source-linked AI summary
Adapting offline models to a streaming context for music source separation
Dylan Sechet, Marc Evrard, Matthieu Kowalski
TL;DR
Real-time music source separation must meet algorithmic-latency and computational-cost constraints, while offline separators are often omitted from comparisons or assigned their full input length as latency. This paper streams unmodified offline models by reading one output hop per step, with latency set by readout position. At equal algorithmic latency, streamed HT-Demucs and SCNet match published dedicated-model quality, but streamed models remain less computationally efficient.
Problem
Offline separators are often excluded from real-time comparisons or assigned their full 2–15 s input length as algorithmic latency, despite real-time systems facing separate algorithmic- and computational-latency constraints.
Source: §1 · p. 1Method
Results
At 23 and 93 ms, streamed HT-Demucs and SCNet at least match the SDR of the best dedicated architectures, with leads of 0.5 to 0.7 dB comparable to confidence-interval width.
Source: §5.3 · p. 4Takeaways & Limitations
Offline models can stream without changing architecture or weights, and lookahead offers a latency–quality trade-off; the model’s STFT window predicts where quality starts to drop sharply.
Source: §6 · p. 4The comparison uses models not necessarily trained on the same datasets, so checkpoint training-data differences qualify the quality comparison.
Source: §5.3 · p. 4
Abstract
from arXiv · showhide
Real-time music source separation must satisfy two constraints: a bound on algorithmic latency and a bound on computational cost. Offline separators are usually omitted from real-time comparisons or credited with a latency equal to their full input length. We show that this latency is set by where the output is read, not by the length of the separator's input. An unmodified offline model can therefore run in a streaming setting, without retraining. At each step, the input slides by one STFT hop, and one output hop is read out. The resulting latency can be as low as one STFT hop (23 ms), and the computational cost does not increase as latency shrinks. We identify a theoretical model-dependent latency boundary below which separation quality should drop steeply, and confirm this experimentally across three architectures. At equal algorithmic latency, streamed off-the-shelf checkpoints for HT-Demucs and SCNet match the published results of dedicated real-time models in terms of separation quality. Streamed models remain far less computationally efficient: only HT-Demucs runs faster than real time on a GPU.
1. INTRODUCTION
Real-time music source separation must balance algorithmic latency with computation time, while dedicated real-time models have tended to trail offline models in separation quality. This paper streams unmodified offline models by changing where their outputs are read, then evaluates latency, quality, and computational cost.
1. INTRODUCTION
- Reading one output hop reduces an offline model’s algorithmic latency to 23.2 ms without modifying the model or its STFT pipeline.Latency is determined by output readout configuration, not input length; lookahead trades latency for quality at fixed computational cost, with quality predicted to drop steeply below the model’s STFT-window length.Source: §1 · p. 1
- Offline models are often excluded from real-time comparisons or assigned latency equal to their 2–15 s input length, complicating comparison with dedicated real-time models.Dedicated real-time models target both algorithmic and computational latency, but still trail standard offline models in separation quality.Source: §1 · p. 1
- The study streams off-the-shelf offline models from 23.2 ms to 3 s and measures how the latency choice affects separation quality.Latencies around 100 ms are described as sufficient for less latency-critical uses such as karaoke or broadcast-music remixing.Source: §1 · p. 1
- Streamed off-the-shelf HT-Demucs and SCNet checkpoints match the best published dedicated real-time models at 23 and 93 ms, while only HT-Demucs runs faster than real time on a GPU.The paper also examines how streaming-specific fine-tuning affects the dependence of quality on future context.Source: §1 · p. 1
- A streaming-specific loss does not remove the dependence of separation quality on future context across architectures and sources.The paper supports this claim with an ablation.Source: §1 · p. 1
2. RELATED WORK
Prior music-separation work includes dedicated streaming architectures adapted from offline models, while speech research has studied latency and lookahead more extensively. This paper contrasts its single-hop streaming approach with earlier offline speech streaming.
2. RELATED WORK
- Dedicated music separators adapt offline architectures: Deep Latent Masking, HS-TasNet, RT-STT, Online SCNet, and Band-SCNet target algorithmic latencies of 23 or 93 ms.The approaches use latent masking, waveform and spectrogram branches, quantization, or causal SCNet variants with added cross-band and narrow-band blocks.Source: §2 · pp. 1–2
- Speech research has studied streaming latency and lookahead, including causal Demucs for enhancement and systems that tune or select future context.Real-time methods remain marginal in music source separation, while they are well studied in speech processing.Source: §2 · p. 2
- Earlier offline speech-separator streaming advances input by 0.5 s per pass and has a 0.5 s latency floor, whereas this paper advances by one STFT hop to reach 23 ms.The cited speech setup reports substantial performance degradation below its floor; for music, the paper reports less than 1 dB cost at one window (93 ms) and a steep quality drop only below that threshold.Source: §2 · p. 2
3. STREAMING OFFLINE MODELS
The streaming procedure slides an offline model’s context window by one STFT hop and emits one hop from each output. The output readout sets algorithmic latency, while the model’s STFT window marks a quality boundary and the streaming procedure discards most computed output.
3. STREAMING OFFLINE MODELS
- Each streaming step advances the fixed-length context window by one STFT hop over the incoming mixture.The offline separator receives its most recent C mixture samples and returns C samples per source; streaming emits only an H-sample buffer.Source: §3 · p. 2
3.1. Output readout and latency
- The readout position determines latency: the emitted H-sample buffer has latency W + kH, where k is lookahead measured in STFT hops.Each additional unit of k adds one hop of latency, and the emitted buffer must remain within the model output.Source: §3.1 · p. 2, Figure 1 · p. 2
- At 44.1 kHz with H = 1024, the construction reaches 23.2 ms minimum latency; k = 0 corresponds to W = 92.9 ms for HT-Demucs and SCNet and 139.3 ms for DTTNet.The minimum follows from the H-sample output buffer and is not a structural lower bound for streaming.Source: §3.1 · p. 2, Figure 2 · p. 2
- Below latency W, part of the emitted hop’s synthesis relies on STFT padding rather than observed input, predicting a quality knee at the model-dependent window length.For latency at least W, the synthesis frames lie within observed input; below W, padding grows as latency falls.Source: §3.1 · p. 2
3.2. Streaming procedure
- For nonnegative lookahead, the emitted hop uses the same STFT frames as offline synthesis, but the model context ends L(k) samples after that hop.For negative lookahead, the estimate is equivalent to the offline model deprived of future context; each pass produces one hop, so streaming cannot use offline overlap-add cross-fading.Source: §3.2 · pp. 2–3
- Changing lookahead leaves model input and forward-pass count unchanged, so computational cost does not depend on algorithmic latency, although most computed output is discarded at each step.Each pass computes C output samples but keeps only H.Source: §3.2 · p. 3
4. EXPERIMENTAL SETUP
The experiments evaluate three mid-sized separators on MUSDB18-HQ using SDR and compare offline processing with streamed inference and dedicated real-time architectures. They use released pretrained weights without retraining or structural changes, and sweep lookahead across a broad latency range.
4. EXPERIMENTAL SETUP
- Evaluation uses MUSDB18-HQ, museval SDR, and three mid-sized models: DTTNet, HT-Demucs, and SCNet.The study focuses on mid-sized models because recomputing full context at every hop already brings these models to an RTF near 1.Source: §4 · p. 3
- The study compares offline processing, including overlap-add where needed, with streamed models and then compares streamed models at each latency against dedicated real-time architectures.Reported results use authors’ released pretrained weights without retraining or structural modification; two models are fine-tuned only for the Sec. 5.2 ablation.Source: §4 · p. 3
- The lookahead sweep spans k = 1−W/H through k = 128, yielding latencies from 23.2 ms to just over 3 s.All three models advance their context window by 1024 samples per step.Source: §4 · p. 3
- RTF is measured on a single NVIDIA RTX 5080 GPU at batch size 1, using median execution times from 50 forward passes after 25 warm-up iterations.For streamed models, processing time is normalized by hop size; offline baselines use stride, and DTTNet timing assumes its four models run in parallel.Source: §4 · p. 3
5. RESULTS
The results show that streamed separation quality depends on lookahead, with a steep drop below each model’s STFT-window length. At equal algorithmic latency, streamed HT-Demucs and SCNet match or exceed dedicated models, while compute efficiency and comparisons across training datasets remain important qualifications.
5.1. Impact of streaming on model performance
- At 3 s latency, all models come within 0.12 dB of their offline equivalents, but quality drops steeply below 93 ms for HT-Demucs and SCNet and 139 ms for DTTNet.The quality knee follows each model’s STFT window; from 93 to 23 ms, all three lose about 1.5 dB.Source: §5.1 · p. 3, Figure 3 · p. 3
- Future context benefits vary by model and source: drums gain least for all three models, while bass gains most with HT-Demucs and SCNet.The authors suggest low-frequency components need longer context to be resolved.Source: §5.1 · pp. 3–4, Figure 4 · p. 3
5.2. Effect of fine-tuning models for a streaming setting
- After correcting for training data, Online SCNet trails offline SCNet by 1.86 dB in median SDR, while streaming SCNet costs 0.74 dB.The authors attribute at least 1.1 dB of the gap to architectural changes; the 0.74 dB streaming cost is an upper bound because it also includes lost overlap-add cross-fading.Source: §5.2 · p. 4
- Restricting fine-tuning loss to the readout buffer gains at most 0.1 dB over the control for both k = 0 and k = −3, and does not reduce streaming loss.Both fine-tuning runs used MUSDB18-HQ, 20 epochs, and a learning rate of 5 × 10−5.Source: §5.2 · p. 4
5.3. Comparison to dedicated real-time architectures
- At 93 ms, streamed SCNet leads Band-SCNet by 0.72 dB; at 23 ms, HT-Demucs and SCNet each lead Deep Latent Masking by about 0.5 dB.HT-Demucs and SCNet at least match the best dedicated architectures at both latencies; DTTNet trails Deep Latent Masking by 0.41 dB at 23 ms and Band-SCNet by 0.28 dB at 93 ms. Deep Latent Masking was evaluated on MUSDB18 rather than MUSDB18-HQ.Source: §5.3 · p. 4, Table 1 · p. 4
- The dedicated-model comparisons use best available checkpoints that are not necessarily trained on the same datasets.The extra 800 HT-Demucs training songs are worth about 1.5 dB offline, while additional MoisesDB training data adds 0.25 dB to SCNet.Source: §5.3 · p. 4
- Only HT-Demucs runs faster than real time on the tested GPU; at 23 ms algorithmic latency, its 20.6 ms pass yields separated samples about 44 ms after arrival.The comparison with dedicated models is therefore about quality at equal algorithmic latency, not equal real-time deployability.Source: §5.3 · p. 4
6. CONCLUSION
Streamed public checkpoints of HT-Demucs and SCNet match purpose-built real-time models in median SDR at equal algorithmic latency. Readout choice enables single-hop latency at constant computational cost, while model efficiency remains a limitation.
6. CONCLUSION
- At equal algorithmic latency, streamed public checkpoints of HT-Demucs and SCNet reach median SDR on par with published purpose-built real-time models, and HT-Demucs runs faster than real time on a GPU.Streamed models remain far less computationally efficient overall, a concern for embedded or edge hardware.Source: §6 · p. 4
- An offline separator can reach single-STFT-hop latency by changing its output readout, without changing its architecture or weights, while computational cost stays constant across the latency range.The model’s STFT window predicts where separation quality starts to drop sharply.Source: §6 · p. 4
- Above one STFT window, lookahead gains remain below 1 dB for up to 3 s and depend mainly on architecture; below that threshold, performance falls steeply as synthesis frames overlap STFT padding.Lookahead therefore offers a tunable latency–quality trade-off.Source: §6 · p. 4
- Emitting several output hops per forward pass, at the cost of added latency, and quantizing weights are proposed directions for reducing streamed-model cost.The efficiency gap matters particularly on embedded or edge hardware.Source: §6 · p. 4
Compliance with Ethical Standards
The work uses numerical simulations on publicly available, open-access datasets and required no ethical approval.
Compliance with Ethical Standards
- The study relies exclusively on numerical simulations using publicly available, open-access datasets, so no ethical approval was required.Source: Compliance with Ethical Standards · p. 5