PaperlayerView exact revision on arXiv

Source-linked AI summary

Music Source Separation with Band-split RNN

Yi Luo, Jianwei Yu

arXiv:2209.15174v1eess.AScs.LGcs.SDeess.SP

TL;DR

Existing MSS models do not fully exploit music-specific signal characteristics, motivating a frequency-domain architecture designed around explicit subbands. BSRNN interleaves band- and sequence-level modeling and uses semi-supervised finetuning, outperforming existing systems on most evaluated tracks and improving all four tracks after finetuning.

  • Problem

    Existing MSS model designs were mainly motivated by other fields and do not fully discover the intrinsic characteristics and patterns of music signals.

  • Method

    BSRNN explicitly splits mixture spectrograms into subbands, interleaves band-level and sequence-level RNN modeling, and finetunes with unlabeled data using pseudo-labels and clean-segment detection.

  • Results

    BSRNN trained only on MUSDB18-HQ outperforms existing systems on vocal, drum, and other tracks, while semi-supervised finetuning improves performance on all four tracks.

  • Takeaways & Limitations

    Explicit frequency-band modeling and semi-supervised finetuning provide a supported approach for improving music source separation across instrument tracks.

Abstract

from arXiv · show

The performance of music source separation (MSS) models has been greatly improved in recent years thanks to the development of novel neural network architectures and training pipelines. However, recent model designs for MSS were mainly motivated by other audio processing tasks or other research fields, while the intrinsic characteristics and patterns of the music signals were not fully discovered. In this paper, we propose band-split RNN (BSRNN), a frequency-domain model that explictly splits the spectrogram of the mixture into subbands and perform interleaved band-level and sequence-level modeling. The choices of the bandwidths of the subbands can be determined by a priori knowledge or expert knowledge on the characteristics of the target source in order to optimize the performance on a certain type of target musical instrument. To better make use of unlabeled data, we also describe a semi-supervised model finetuning pipeline that can further improve the performance of the model. Experiment results show that BSRNN trained only on MUSDB18-HQ dataset significantly outperforms several top-ranking models in Music Demixing (MDX) Challenge 2021, and the semi-supervised finetuning stage further improves the performance on all four instrument tracks.

I. INTRODUCTION

MSS is important but challenging, and existing models do not fully exploit music-specific signal characteristics. BSRNN addresses this gap while adding a semi-supervised finetuning pipeline for unlabeled data.

  • Music source separation supports music remixing, music information retrieval, and music education.
  • MSS is more challenging than speech separation because music uses higher sample rates and is professionally edited or manipulated.
  • Many MSS architectures were motivated by models from speech, image segmentation, and human pose estimation rather than music-specific characteristics.
  • BSRNN explicitly models frequency bands in high-sample-rate signals and adjusts bandwidths for different instrument types.
  • The semi-supervised finetuning pipeline uses a strong pretrained model to generate pseudo-labels and detect clean segments, improving all four instrument tracks.

II. BAND-SPLIT RNN

The BSRNN band-split architecture partitions the mixture spectrogram into predefined subbands, transforms each into a common feature dimension, and merges the features for subsequent modeling.

  • The overall BSRNN pipeline contains band split, band-and-sequence modeling, and mask estimation modules.
  • The band split module divides the complex-valued STFT spectrogram into K subband spectrograms with predefined bandwidths.The subband bandwidths can differ across bands.
  • Each subband’s real and imaginary components are concatenated, layer-normalized, and mapped by an FC layer to a real-valued feature with dimension N.
  • Because subband bandwidths may differ, each subband has its own normalization module and FC layer.
  • The K subband features are merged into a transformed fullband feature tensor Z ∈ R^N×K×T.

B. Band and Sequence Modeling Module

BSRNN alternates sequence-level and band-level residual RNN processing, then converts the resulting subband features into complex-valued masks for source extraction.

  • The sequence-level RNN models each subband feature across time, sharing one RNN across subbands to reduce model size and enable parallel processing.
  • The band-level RNN models features across subbands to capture intra-band feature dependencies.
  • Both modeling paths use residual RNN blocks built from group normalization, BLSTM, FC processing, and residual connections.Multiple such blocks can be stacked, producing Q ∈ R^N×K×T.
  • The mask estimation module splits Q by subband and uses layer normalization plus an MLP to generate each mask’s real and imaginary parts.

D. Discussion

BSRNN applies frequency-aware subband modeling rather than coarse frequency grouping, connecting band splitting with interleaved multi-axis sequence processing. Its design targets detailed harmonic structure in music signals.

  • BSRNN relates to dual-path, multi-path, group-splitting, and super-wideband architectures through interleaved processing across dimensions.
  • Unlike coarse group splitting, BSRNN explicitly partitions frequency components and uses sequential-order-sensitive modeling for intra-band dependencies.
  • Sharing the sequence-modeling layer across subbands enables parallel processing, reduces model size, and empirically performs better than separate layers per subband.
  • Compared with simple low- and high-frequency splits in super-wideband speech enhancement, BSRNN uses fine-grained bands to cover detailed harmonic patterns in music.
  • Collecting clean sources for every possible musical instrument and arrangement is difficult, motivating training strategies that use unlabeled data.

A. Semi-supervised Data Sampling

The semi-supervised pipeline uses a pretrained model to generate and filter target and residual signals from unlabeled mixtures, then finetunes an initialized model on the resulting samples.

  • Pipeline: The pipeline samples clean target and residual segments from labeled data L alongside mixture segments from U.
  • Pipeline: A pretrained model P generates separated target and residual signals from mixture segments in the unlabeled dataset U.The residual is obtained by subtracting the separated target from the mixture.
  • Filtering: Energy-based filtering identifies mixture segments as clean target or residual segments when the relevant energy difference exceeds 30 dB.Separated signals are not used when the mixture itself is treated as a clean target or residual segment.
  • Finetuning: The finetuning model Q is initialized by P and can replace P when it achieves better validation performance.

B. Discussion

The pipeline combines detected clean segments with pseudo-label signals from a strong pretrained model, reducing reliance on an external source activity detector while using unlabeled data.

  • Discussion: The pipeline combines clean-segment detection, pseudo-label generation and filtering, and self-boosting approaches through a strong pretrained model.
  • Discussion: Using the pretrained model as both pseudo-label generator and source activity detector removes the need for a separately trained external detector.
  • Data preprocessing: The preprocessing stage removes silent regions and retains salient segments for data mixing.Its energy-based implementation uses 6-second segments and a 50% overlap ratio.

2) On-the-fly Data Simulation:

On-the-fly simulation creates training mixtures by sampling salient source segments, selecting shorter chunks, and rescaling their energies before mixing.

  • On-the-fly Data Simulation: Training samples are formed by randomly mixing tracks from different songs for a target source and a duration T ≤ L.The default settings are T = 3 seconds and source segments of length L.
  • On-the-fly Data Simulation: Each sampled chunk is randomly rescaled between −10 and 10 dB relative to its original energy.
  • Training Objective: The training objective sums frequency-domain and time-domain mean-absolute-error losses.The time-domain term compares inverse-STFT waveforms, while the frequency-domain terms compare real and imaginary spectrogram parts.

4) Hyperparameter Configuration:

The configuration uses stacked residual BLSTM band-and-sequence modules, individual target-track models, and a separate lower-learning-rate finetuning setup.

  • Hyperparameter Configuration: BSRNN uses 12 band-and-sequence modules containing 24 residual BLSTM layers with feature dimension N = 128.The BLSTM hidden size is 256 and the mask-estimation MLP hidden size is 512.
  • Hyperparameter Configuration: The model uses STFT window and hop sizes of 2048 and 512 with a Hanning window.
  • Hyperparameter Configuration: The band-split bandwidth is treated as a separate hyperparameter discussed in the experiments.
  • Training: Individual models are trained for each target track for 100 epochs using Adam with an initial learning rate of 1e−3.Each epoch contains 10000 batches with batch size 2 across 8 GPUs.
  • Finetuning: Semi-supervised finetuning uses a private dataset of 1750 songs and an initial learning rate of 1e−4.Other training configurations remain identical to those of the main setup.

C. Evaluation

The evaluation examines metrics, band-split configurations, and instrument-specific bandwidth choices for BSRNN on MUSDB18 datasets.

  • Evaluation metrics and datasets: BSRNN performance is evaluated with uSDR and cSDR on MUSDB18-HQ and MUSDB18.uSDR is the mean across song-level SDR scores, while cSDR is chunk-level SDR.
  • Band-split configurations: Seven band-split bandwidth options are compared for vocal extraction, including configurations producing 14, 22, and 23 subbands.The options vary bandwidth granularity across frequency ranges.
  • Band-split configurations: Finer subbands below 1k Hz improve vocal-extraction performance across the evaluated configurations.The comparison attributes this pattern to better capture of vocal fundamental-frequency information and higher-frequency components.
  • Instrument-specific settings: Instrument-specific bandwidths are selected for bass, drum, and other tracks after a small-scale grid search.The bass configuration uses 30 subbands, with especially fine splitting below 500 Hz.

B. Effect of Evaluation Segment Hop Size

The evaluation studies how segment hop size affects overlap-add separation, finding a practical trade-off between performance and processing speed.

  • Performance effects: Any hop size P ≤ T can improve performance when separated chunks are combined with overlap-add.The authors suggest that overlap-add smooths outputs and mitigates introduced noise or distortion.
  • Performance effects: Reducing P from 1.5 to 0.5 seconds provides no significant gain while linearly increasing processing time.The additional segments required by smaller hops account for the increased processing time.
  • Practical setting: A hop size of P = 1.5 seconds is suggested as a practical balance between processing speed and performance.The experiments use P = 0.5 seconds by default.

C. Comparison with State-of-the-art Systems

BSRNN is compared with top-ranking MDX Challenge 2021 systems on MUSDB18 and MUSDB18-HQ, both before and after semi-supervised finetuning.

  • Comparison results: BSRNN trained only on MUSDB18-HQ outperforms existing systems on vocal, drum, and other tracks across both datasets and metrics.The comparison uses uSDR and cSDR on MUSDB18-HQ and MUSDB18.
  • Comparison results: BSRNN performs slightly worse than existing systems on the bass track.The authors associate this with energy rescaling and insufficient capture of bass-region frequency information.
  • Semi-supervised finetuning: Semi-supervised finetuning improves performance on all four instrument tracks.The paper reports this improvement after finetuning the BSRNN model.
  • Overall conclusion: Experiments on MUSDB18 and MUSDB18-HQ report that BSRNN surpasses existing state-of-the-art music source separation systems.The conclusion also states that the finetuning pipeline improves performance and robustness across songs with various genres.
Loading 2209.15174v1…