PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Where to Split: Dynamic Band Splitting for Audio Source Separation

Bingzhe Zhang, Kai Li, Xiaolin Hu

arXiv:2610.14993v1cs.SD

TL;DR

Fixed band layouts reduce separator cost, but they use the same frequency boundaries for mixtures with different spectral content. DBS adapts boundaries within a fixed band-token budget, and its variants improved separation over corresponding static models across three datasets.

  • Problem

    Fixed band layouts are shared across mixtures even though informative spectral components vary, creating a need to allocate frequency resolution per mixture without increasing the number of band representations.

    Source: §1 · p. 2

  • Method

    DBS predicts mixture-conditioned boundaries within fixed frequency blocks, selects a prescribed number with Hard Top-K, and pools the resulting bands into tokens under a fixed budget.

    Source: §1 · p. 2

  • Results

    DBS variants improved separation over corresponding static models on DnR v2, MUSDB18-HQ, and EchoSet; DBS-BSRNN gained 29.0% in average SI-SDR on DnR v2.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On DnR v2, matched-boundary controls and ablations support that mixture-based allocation of a fixed frequency-resolution budget can improve separation without lengthening the separator token sequence.

    Source: §5 · p. 9

    DBS fixes the number of bands and selects one partition per inference window; it does not adapt representation size or boundary positions within a window.

    Source: §5 · p. 9

Abstract

from arXiv · show

Audio source separation models must preserve fine spectral structure while keeping the cost of the separator tractable. Band-split architectures reduce this cost by manually partitioning the spectrum into predefined frequency bands and encoding each band as a token, but the same boundaries are used for every mixture. Because informative spectral structure varies with source content, fixed boundaries can merge discriminative components or spend tokens where little resolution is needed. We introduce Dynamic Band Splitting (DBS), an end-to-end framework that adapts boundary locations and band widths to each mixture under a fixed band-token budget. DBS predicts content-conditioned boundary scores within frequency blocks, selects a prescribed number of boundaries using hard Top-$K$, and applies content-dependent pooling to obtain compact band tokens. After separation, it smooths and expands these tokens to the original STFT grid, refines them along frequency, and decodes complex masks with a frequency-shared head. DBS improved separation over the corresponding static models on DnR v2, MUSDB18-HQ, and EchoSet, with DBS-BSRNN achieving relative gain of 29.0% in average SI-SDR on DnR v2. These results and boundary analyses on DnR v2 show that content-adaptive frequency partitioning benefits multiple separator backbones under a fixed band-token budget. Code and audio demos are available at https://cslikai.cn/Dynamic-Band-Splitting

1 INTRODUCTION

Fixed band layouts reduce separator cost but allocate frequency resolution identically across mixtures, despite content-dependent spectral structure. DBS adapts boundaries under a fixed token budget and improves separation across the evaluated datasets and backbones.

1 INTRODUCTION

  • Fixed layouts can merge discriminative components in wide bands or spend fine resolution on less informative content, while adding bands raises separator cost.The challenge is to adapt resolution to each mixture while keeping the number of band representations fixed.Source: §1 · p. 2
  • DBS predicts mixture-conditioned boundaries within fixed frequency blocks, selects a prescribed number with Hard Top-K, and reconstructs on the original STFT grid.The framework supports different separator backbones while keeping the band-token budget fixed.Source: §1 · p. 2
  • DBS improved separation over static models across DnR v2, MUSDB18-HQ, and EchoSet; DBS-BSRNN gained 29.0% in average SI-SDR on DnR v2.Gains also extended to other band-split backbones on MUSDB18-HQ and EchoSet.Source: §1 · p. 2

2 RELATED WORK

Band-split methods reduce the frequency-axis sequence by assigning specialized modules to spectral subbands, but typically treat the frequency layout as a fixed model-level prior. Other approaches relax fixed spectral representations or computation in different ways; DBS instead learns input-dependent boundaries while retaining a fixed band count.

2 RELATED WORK

  • Band-split architectures reduce high-resolution spectral processing by dividing the spectrum into subbands for within-band structure and cross-band interactions.BSRNN, BS-RoFormer, Mel-Band RoFormer, TIGER, and SCNet instantiate this design with differing structures.Source: §2 · p. 3
  • Related approaches optimize a learnable filterbank, replace the fixed encoder, or route experts, adapting representation or computation within a separation model.These approaches include learnable filterbanks, SFC, and mixture-of-experts models.Source: §2 · p. 3
  • DBS learns input-dependent boundaries between contiguous bands jointly with the separator, while keeping the band count fixed and retaining the STFT grid for reconstruction.Unlike the cited approaches, its adaptation is specifically in the locations of contiguous frequency-band boundaries.Source: §2 · p. 3

3 METHOD

DBS forms a mixture-dependent partition, pools its bands into fixed-length tokens for a separation backbone, and uses the same partition to reconstruct full-resolution complex masks. Its training combines separation loss with boundary-density regularization while using a hard partition in the forward pass.

3 METHOD

  • The method maps a mixture spectrogram to a mixture-dependent partition, fixed-length band tokens, separated features, and complex masks on the original STFT grid.The same partition is used for tokenization and reconstruction.Source: §3 · p. 3

3.1 DYNAMIC BAND SPLITTING

  • Within fixed contiguous frequency blocks, DBS selects Jr−1 internal boundaries with Hard Top-K, producing Jr contiguous bands per block under a fixed total budget.The blocks distribute a prescribed share of the band budget across the spectrum, while within-block locations adapt to the input.Source: §3.1 · pp. 3–4

3.2 BAND TOKENIZATION AND SEPARATION

  • A predictor combines learned spectral features with magnitude statistics to score candidate transitions, and the selected bands are pooled into one fixed-dimensional token per band.The magnitude statistics include temporal mean, peak-to-mean ratio, and standard-deviation-to-mean ratio; pooling weights are shared across frames.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.3 TOKEN EXPANSION & MASK DECODING

  • After separation, DBS smooths tokens using within-band boundary probabilities, expands them by the hard assignment, refines neighboring frequency features, and predicts complex masks.A frequency-shared decoder maps refined features to mask parameters; the inverse STFT transforms estimated spectrograms to waveforms.Source: §3.3 · p. 5

3.4 END-TO-END TRAINING

  • Straight-through gradients connect the separation loss to hard boundary decisions, while a ratio regularizer controls soft boundary density and Hard Top-K fixes the realized band count.The joint objective is L = Lsep + λratioLratio.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Across DnR v2, MUSDB18-HQ, and EchoSet, DBS improved separation over static counterparts across multiple backbones. Experiments also tested boundary adaptation and design choices under fixed band-token budgets.

4.1 EXPERIMENTAL SETUP

  • Experiments covered DnR v2 music, speech, and sound effects; MUSDB18-HQ four-stem music separation; and two-speaker EchoSet mixtures, using fixed budgets of 128, 128, and 67 tokens, respectively.MUSDB18-HQ models were trained as single models jointly estimating all four stems, without external audio.Source: §4.1 · p. 6

4.2 SEPARATION PERFORMANCE ACROSS DATASETS AND BACKBONES

  • DBS raised DnR v2 average SI-SDR by 1.38 dB for BSRNN and 1.39 dB for TIGER, while improving MUSDB18-HQ four-stem SDR across three backbones and EchoSet speech-separation metrics.On EchoSet, DBS-BSRNN gains over the published BSRNN baseline were 0.46 dB SDRi and 0.43 dB SI-SDRi; DBS-TIGER gains were 0.33 and 0.31 dB, respectively.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

4.3 ISOLATING THE EFFECT OF DYNAMIC BOUNDARIES

  • With the same architecture and training schedule, DBS-BSRNN improved average SI-SDR by 1.07 dB over a separately trained uniform control using 128 bands.Replacing learned boundaries with the BSRNN layout at inference reduced average SI-SDR from 9.07 to 8.62 dB with model weights unchanged.Source: §4.3 · p. 7, Table 3 · p. 7
  • Across 64 DnR v2 test mixtures, central windows had distinct partitions, adjacent windows also differed, and repeated inference on identical inputs produced identical boundaries.Figure 5 reports boundary replacement under the fixed 128-band budget; crosses denote repeated inference on identical inputs.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 DESIGN ABLATIONS

  • At a fixed 128-band total, average SI-SDR fell from 9.07 dB with four blocks to 8.32 dB with two and 6.30 dB with one; the one-block example concentrated boundaries at low frequencies.The upper frequency range was consequently sparsely partitioned in that example.Source: §4.4 · p. 8, Table 4 · p. 8
  • Average SI-SDR fell 2.14 dB with magnitude-only boundary prediction, 0.57 dB without energy descriptors, 4.98 dB without the acoustic transition prior, 1.54 dB without the confidence path, and 5.15 dB without frequency refinement.The largest listed loss was from removing frequency refinement after token expansion; the transition prior was particularly important among the tested boundary-prediction components.Source: §4.4 · pp. 8–9, Table 5 · p. 9

5 CONCLUSION

DBS adapts frequency boundaries to each mixture under a fixed band-token budget, and controls on DnR v2 indicate that boundary placement affects separation quality. The appendix details the predictor, tokenization, reconstruction, and training mechanism behind this approach.

5 CONCLUSION

  • DBS improved DnR v2 separation, with matched boundary-substitution and fixed-boundary training controls showing that boundary placement affected quality.Block and component ablations also identified roles for per-block budgets, boundary scoring, and frequency refinement; the conclusion reports gains without lengthening the separator’s token sequence.Source: §5 · p. 9

A BOUNDARY PREDICTOR

  • The boundary predictor fuses convolutional spectro-temporal features, per-bin magnitude statistics, and BiLSTM context to score frequency-bin boundaries.Its convolutional stages preserve frequency while downsampling time by a factor of four.Source: A BOUNDARY PREDICTOR · pp. 11–12

B.1 CONTENT-DEPENDENT BAND POOLING

  • Content-dependent pooling scores bins and normalizes weights within each selected band to produce fixed-dimension band tokens.The BSRNN bin encoder applies channel-wise normalization and a 1×1 convolution before a two-layer scoring network.Source: §B.1 · pp. 12–13

B.2 SMOOTHING, REFINEMENT, AND MASK PREDICTION

  • After separation, probability-guided smoothing blends each token with its preceding band, then expansion and frequency refinement support frequency-shared mask prediction.Larger mean boundary probability retains more of the current token; smaller values propagate more from the preceding band.Source: §B.2 · p. 13

C STRAIGHT-THROUGH GRADIENT FOR DISCRETE ASSIGNMENTS

  • Training keeps the Hard Top-K partition in the forward pass while surrogate gradients use neighboring boundary-present and boundary-absent assignments.When an alternative band index is invalid, its representation falls back to the selected band and contributes zero assignment gradient.Source: C STRAIGHT-THROUGH GRADIENT FOR… · p. 14

D SOURCE-SEPARATION OBJECTIVES

The source-separation objectives pair named outputs with references where sources are named and use permutation-invariant training for interchangeable EchoSet speakers. The appendix describes distinct objective choices across datasets and backbones.

DnR v2

  • For DnR v2, fixed and DBS BSRNN models minimized mean negative SI-SDR over the three named sources.DnR v2 and MUSDB18-HQ losses matched each output to its corresponding named reference.Source: D SOURCE-SEPARATION OBJECTIVES · p. 14

MUSDB18-HQ

  • On MUSDB18-HQ, fixed and DBS BSRNN, BS-RoFormer, and MelRoFormer jointly estimated four stems using waveform MAE plus multi-resolution complex-STFT MAE.The spectral term used window lengths 4096, 2048, 1024, 512, and 256, with weight 1.0 in the public four-stem configurations.Source: D SOURCE-SEPARATION OBJECTIVES · p. 15

EchoSet

  • EchoSet used permutation-invariant training for its interchangeable speakers, with DBS-BSRNN minimizing PIT negative SI-SDR and DBS-TIGER minimizing PIT negative SNR.The TIGER model followed its released configuration, which specified negative SNR rather than the negative SI-SDR described in the cited paper.Source: D SOURCE-SEPARATION OBJECTIVES · p. 15

E DATASET DETAILS

The evaluation uses three benchmarks spanning music, speech, sound effects, and mixtures with noise and reverberation. The appendix specifies dataset composition and the partitions used.

DnR v2

  • DnR v2 provides 3,406 training, 487 validation, and 973 test mixtures of music, speech, and sound effects at 44.1 kHz.The evaluation used the complete training and test partitions and all 973 full-length test recordings.Source: E DATASET DETAILS · p. 15, Table 6 · p. 16

MUSDB18-HQ

  • MUSDB18-HQ contains 150 full-length stereo songs with vocals, drums, bass, and other stems; the study used the official 100-track training and 50-track test partition.The uncompressed WAV release retains bandwidth up to approximately 22 kHz.Source: E DATASET DETAILS · pp. 15–16

EchoSet

  • EchoSet contains two-speaker mixtures with noise and reverberation, with 20,268 training, 4,604 validation, and 2,650 test mixtures at 16 kHz.All 2,650 complete six-second test mixtures were evaluated.Source: E DATASET DETAILS · p. 16

F MODEL AND TRAINING CONFIGURATIONS

The configurations describe fixed-band controls, dataset-specific frequency layouts, backbone integration, and optimization. DBS retains each separator’s temporal and inter-band modeling core while replacing its band interface and mask reconstruction.

F.1 CONSTRUCTION OF 128-BAND FIXED PARTITIONS

  • For DnR v2, each fixed BSRNN and TIGER control was expanded independently to 128 bands over 1025 STFT bins, preserving its backbone-specific frequency prior.The construction clamps boundaries to preserve strict ordering and reserve at least one STFT bin for every remaining band.Source: §F.1 · p. 16

F.2 FREQUENCY LAYOUTS AND BACKBONE INTEGRATION

  • DnR v2 and MUSDB18-HQ used 128 tokens, while EchoSet used 67; DBS retained each backbone’s separator core but replaced its band interface and mask reconstruction.Each inference window used one mixture-conditioned partition for band encoding and full-resolution reconstruction.Source: §F.2 · p. 17

F.3 OPTIMIZATION

  • The boundary predictor, band encoder, retained separator, and full-resolution decoder were jointly optimized, with Hard Top-K partitions and training signals from the straight-through path and boundary-density regularizer.The DBS decoder used one frequency-direction refinement block; on DnR v2 it used kernel size 3, no direct full-resolution mixture-spectrum features, and ratio-loss weight 0.01.Source: §F.3 · p. 18

G EVALUATION METRICS AND PROTOCOLS

The evaluation defines waveform scores and specifies how estimates are matched and averaged across the three benchmarks. Inference uses benchmark-specific windows or chunks, with one DBS partition shared by encoding and reconstruction within each window.

G.1 WAVEFORM SCORES

  • SI-SDR compares the projected target with the residual, while SNR compares the estimate with the reference at its original amplitude scale.Scores are measured in decibels, with larger values indicating better reconstruction; EchoSet’s public evaluator uses uncentered SI-SDR.Source: §G.1 · p. 18
  • SDR permits a finite impulse response filter to account for linear distortion, and EchoSet uses fast bss eval with filter length 512.The EchoSet evaluator selects the source permutation that maximizes mean SDR.Source: §G.1 · p. 18

G.2 SOURCE MATCHING, AVERAGING, AND IMPROVEMENT SCORES

  • DnR v2 and MUSDB18-HQ use named-stem matching, while EchoSet matches its two unlabeled outputs by permutation and reports improvement over the mixture.DnR v2 reports source-wise SI-SDR and an unweighted average over 973 recordings; EchoSet averages source scores within recordings and then across 2,650 recordings.Source: §G.2 · pp. 18–19

G.3 INFERENCE PROTOCOL

  • DnR v2 inference processes 973 recordings in overlapping three-second windows, MUSDB18-HQ combines chunk estimates by overlap-add, and EchoSet processes each six-second mixture in one pass.For each DnR v2 window, the predicted hard partition is used for both band encoding and full-resolution reconstruction.Source: §G.3 · p. 19

H ECHOSET SEPARATION RESULTS

DBS improves reported separation scores on EchoSet and under matched inference-time boundary substitutions on DnR v2. The accompanying front-end tests identify reconstruction distortion and filter-ordering issues in the tested learnable-filterbank configuration.

H ECHOSET SEPARATION RESULTS

  • 0.46 dB SDRi and 0.43 dB SI-SDRi gains over published BSRNN scores, and 0.33 dB and 0.31 dB gains over published TIGER scores, were reported on EchoSet.Both DBS backbones improved within EchoSet’s analyzed 0–8 kHz range.Source: H ECHOSET SEPARATION RESULTS · p. 19, Table 9 · p. 20

I INFERENCE-TIME BOUNDARY SUBSTITUTION

  • 1.27 dB and 0.45 dB average SI-SDR gains over uniform and BSRNN-layout boundaries, respectively, followed boundary substitution in the trained DnR v2 model.Weights, input windows, and the 128-band budget were held fixed, isolating boundary placement’s effect on separation quality.Source: I INFERENCE-TIME BOUNDARY SUBSTITUTION · p. 19, Table 10 · p. 20

J.1 IDENTITY RECONSTRUCTION TEST

  • The learned Conv1d/ConvTranspose1d analysis–synthesis path distorted a six-second mixed probe when the mask estimator was bypassed.The STFT/inverse-STFT Mel-band control reconstructed the same probe accurately; the learned path had low input correlation and strongly amplified RMS.Source: §J.1 · p. 20, Table 11 · p. 20

J.2 LEARNED FILTER DISTRIBUTION

  • Among 1,025 paired learned analysis filters, 923 peaked below 1 kHz; the highest peak was 11.35 kHz, below the 22.05 kHz Nyquist frequency, and mean pairwise cosine similarity was 0.732.Source: §J.2 · p. 20

J.3 COMPATIBILITY WITH BAND-SPLIT ARCHITECTURE

The tested learnable-filterbank pairing conflicts with the frequency topology assumed by band-split modeling: its latent ordering does not consistently track physical frequency order.

J.3 COMPATIBILITY WITH BAND-SPLIT ARCHITECTURE

  • 80 adjacent filter indices reversed peak-frequency order by more than 1 kHz, despite BSRNN modeling neighboring bands as adjacent frequency intervals.StaticBandSplit grouped latent-channel pairs as frequency bins, but the learned waveform filters had no ordering constraint.Source: §J.3 · p. 20

K.1 ANALYSIS PROTOCOL AND PER-BLOCK STATISTICS

The DnR v2 analysis measures boundary changes across mixtures and adjacent windows while checking that routing is deterministic and uses a fixed token budget. It finds stronger boundary movement in B4 than B2, with input-dependent layouts that vary over time and alongside mixture spectra.

K.1 ANALYSIS PROTOCOL AND PER-BLOCK STATISTICS

  • The analysis sampled 64 test mixtures, examined seven overlapping windows per mixture, and compared central windows across mixtures and adjacent windows within recordings.Repeated inference on central windows tested whether identical inputs produced identical boundaries.Source: §K.1 · p. 21
  • B4 had the largest mean cross-mixture boundary replacement rate at 46.13%, while B2 had the smallest at 7.20%.Among 64 central windows, the four blocks produced 64, 47, 64, and 64 unique partitions, respectively.Source: §K.1 · p. 21, Table 12 · p. 22, Figure 8 · p. 22
  • Adjacent windows showed replacement rates of 32.61% in B4 and 5.69% in B2, indicating window-level adaptation rather than frame-level changes.The partition responded to local content within a recording while retaining a single partition for each analysis window.Source: §K.1 · p. 21
  • Every analyzed window used the same 128-band token budget, and repeated inference on identical inputs produced identical boundaries and Boundary Probabilities.The observed variation reflects deterministic input conditioning, not stochastic inference or a larger budget for selected inputs.Source: §K.1 · pp. 21–22

K.2 ADDITIONAL BOUNDARY VISUALIZATIONS

  • Selection frequencies were comparatively stable in B2 and spread across a wider set of candidate bins in B4.The plotted frequency is the fraction of central windows selecting each non-mandatory bin as an internal boundary.Source: §K.2 · p. 22, Figure 9 · p. 23
  • Boundary layouts changed across successive windows and differed across examples when overlaid on mixture power, despite fixed per-block band counts.The spectral overlays provide mixture-level context but do not identify a selected bin with a particular source or spectral peak.Source: §K.2 · p. 23, Figure 10 · p. 24, Figure 11 · p. 25

Related papers

Loading 2610.14993v1…