Real-Time Music Source Separation on a Low-Power Audio DSP

1. INTRODUCTION

• No prior music source separator was evaluated against or demonstrated on a low-power audio DSP, where fixed MAC rates, hard frame deadlines, and limited SRAM jointly constrain deployment.Desktop CPU and GPU processing times do not transfer directly to this hardware setting.Source: §1 · p. 1
• The targeted embedded setting is harder for music separation than speech enhancement because it requires four correlated outputs and a decoder that reconstructs every stem.The cited passage contrasts the DSP setting with efficiency-oriented separation on mobile GPUs and embedded speech enhancement.Source: §1 · p. 1
• The paper evaluates published systems against a commercial DSP budget and designs, trains, and deploys a separator that fits.The cited work also reports a streaming failure of independent interest.Source: §1 · p. 1
• Block-padded convolution training makes continuous inference out-of-distribution: a model scoring 3.93 dB block-wise collapses to silence within about 2 seconds frame-by-frame.The paper reports this streaming failure as an additional finding alongside its deployable separator.Source: §1 · p. 1
• The separator adds a causal, per-source-gated complex FIR deep filter whose past/future tap split provides an explicit latency control.It replaces a point-wise complex mask with a short complex FIR filter convolved along time at each time-frequency bin.Source: §1 · p. 1

2. THE EMBEDDED BUDGET

• The target SHARC-FX DSP sustains 2.07 GMAC s−1, giving an 11.6 ms per-frame deadline at the stated operating point.The analysis treats weight memory as an on-chip design constraint and compares required compute with measured achievable rate, not datasheet peak.Source: §2 · p. 1
• No published system meets both DSP budgets: memory excludes X-UMX, TasNet, and both HS-TasNet variants by 8.0–25.5×, while RT-STT needs 11.4 GMAC s−1, 5.5× the sustained rate.HS-TasNet-S meets the frame deadline but overruns L2; RT-STT passes memory at 19% of L2 but exceeds compute.Source: Table 1 · p. 2, Figure 1 · p. 2, §2 · p. 2
• The memory test assumes int8 weights and excludes activations and streaming state, making its estimated memory gaps generous to the published systems.Deployability is defined by satisfying both weight memory and MAC-per-frame constraints.Source: §2 · pp. 1–2, Table 1 · p. 2
• Weight reuse spans ρ ≈1 for TasNet/X-UMX to ρ = 345 for RT-STT, so parameter count cannot predict per-frame compute.RT-STT’s 383 k weights are evaluated across 384 frequency bins per frame.Source: §2 · p. 2
• The verdict applies to audio DSPs of this class, not application processors with a vector backend.The tested part is described as favorable within the class; the passage says the verdict would not hold on an application processor.Source: §2 · p. 2

3.1. Backbone

• The causal TFC-TDF U-Net uses a 1024-point, 512-hop real STFT and bounded-state GRU; its complex ratio mask maps silence to silence.Source: §3.1 · p. 2

3.2. Deep filter

• The deep-filter head predicts gated complex FIR taps from latent features, blends filtered-mixture and mask estimates, and sets added latency through look-ahead order Q.At Q=0, left-only padding makes the tap predictor causal as well.Source: §3.2 · pp. 2–3, Figure 2 · p. 3

3.3. Training on continuous context

• Within 0.1 dB of block-wise performance, continuous-segment training restores frame-by-frame inference; block-padded training otherwise collapses to silence within ~2 s.With about eight stacked layers, roughly 16 of 65 chunk frames depend on padding and never occur in deployment.Source: §3.3 · p. 3

• On the full 50-song MUSDB18-HQ test split, cSDR is the median over 1 s windows then songs, while uSDR is whole-song SDR averaged over songs.Source: §4 · p. 3
• cSDR differs from BSSEval-v4 by −1.33 to +1.21 dB per stem, so cross-paper comparisons are indicative.Source: §4 · p. 3
• The MAC counter covers convolutional, linear, and recurrent layers and reproduces the independently timed 15.08 M MAC/frame on the DSP.Source: §4 · p. 3

5.1. Deployable operating points

• 4.70 dB cSDR at 21.9 M MAC/frame fits in 10.43 ms of the 11.6 ms hop and 1963 of 2040 kB of L2.The F=192 variant trades 0.36 dB for 77% of compute and a wide memory margin.Source: §5.1 · p. 3

5.2. Streaming stability

• Continuous training holds 3.17 dB during frame-by-frame inference, while chunk-trained weights fall to 0.04 dB within ∼2 s despite both scoring 3.93 dB block-wise.The comparison uses medians over 8 tracks; the deployed model’s continuous cSDR matches its block-wise value.Source: §5.2 · p. 4, Figure 3 · p. 4

5.3. Look-ahead as a latency knob

• A strictly causal deep filter (Q=0) gains 0.38 dB cSDR and 0.23 dB uSDR over the matched no-DF baseline on every stem.One look-ahead frame adds 0.19 dB on the full-band model; most look-ahead is unnecessary there.Source: §5.3 · p. 4, Figure 3 · p. 4

5.4. On-device validation

• The deployed full-band model runs end to end in 10.43 ms mean and 10.44 ms worst case, using 1963 of 2040 kB of L2 and 447 of 512 kB of L1.The F=192 no-DF variant runs in 7.56 ms (65%) in 1362 kB; its convolution ring caches use 16-bit float, while weights and carried GRU state remain 32-bit.Source: §5.4 · p. 4

Quality comparison

• 4.70 dB cSDR is 0.47 dB below RT-STT on the authors’ metric, or 0.69 dB below comparing BSSEval-v4 like for like, and is indistinguishable from HS-TasNet.The bootstrap 95% interval is [4.21, 5.09] over the 50 tracks.Source: §6 · p. 4

Compute and latency boundaries

• The unconstrained model reaches 5.49 dB but needs 6.4× the available compute; MAC alone also omits bandwidth and serialisation costs.At Q=2 the system adds 23.2 ms to the 23 ms window, while Q=0 costs 0.24 dB.Source: §6 · p. 4

Measurement scope

• Compute verdicts for systems not implemented are lower bounds, and the RT-STT MAC estimate comes from a reproduction scoring 4.93 dB paired with the authors’ quality result.The two F=192 deep-filter rows are not measured on hardware.Source: §6 · p. 4

Unmeasured quantities

• The paper reports no fixed-point accuracy or energy use, and MUSDB18-HQ leaves domain robustness untested.The recurrent state remains 32-bit because it is an accumulator carried indefinitely.Source: §6 · p. 4

7. CONCLUSION

• No published real-time music source separator runs on the embedded hardware it targets.Memory and compute constraints eliminate different model families, and parameter count predicts neither.Source: §7 · p. 4
• Building an embedded real-time music source separator was a training problem, not an architectural one: ours runs on-device at 4.70 dB cSDR in 10.43 ms of an 11.6 ms hop.Source: §7 · p. 4
• Although no published real-time music source separator runs on its target embedded hardware, this system runs on-device at 4.70 dB cSDR in 10.43 ms of an 11.6 ms hop.Source: §7 · p. 4