Source-linked AI summary
Restore, Separate, Restore: A Modular Framework for Music Source Restoration
Tobias Morocutti, Emmanouil Karystinaios, Gerhard Widmer
TL;DR
MSR seeks to recover unprocessed stems from recordings affected by production and transmission processing, which exceeds the clean-source assumption of conventional separation. The paper proposes mixture restoration, eight-stem separation, and stem-specific restoration experts. Mixture restoration improves all three reported metrics, while stem-restoration test gains are small and the metrics disagree.
Problem
MSR aims to recover original, unprocessed sources from recordings altered by production and transmission effects, beyond conventional separation’s clean-source linear-mixture assumption.
Source: §1 · p. 1Method
A three-stage framework restores mixture degradation, separates the restored mixture into eight stems, and applies source-specific experts fine-tuned on separator artifacts.
Source: §1 · p. 1Results
Mixture restoration improves all three metrics, with MMSNR rising 3.980→4.047 dB; stem restoration raises test MMSNR 4.047→4.050 dB while FAD-CLAP worsens 0.274→0.279.
Source: §4 · p. 4Takeaways & Limitations
The results support separating restoration from separation, while validation–test distribution shift and metric disagreement caution against treating a single-metric gain as comprehensive.
Source: §5 · p. 4, §4 · p. 4The separator is trained on clean, unmastered mixtures but receives Stage 1’s mastered-mixture approximation at inference, creating a mismatch left for future work.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Music Source Restoration (MSR) seeks to recover original, unprocessed instrument stems from mixed, mastered, and possibly degraded recordings. Unlike conventional source separation, which treats the mixture as a linear sum of clean sources, MSR must additionally invert nonlinear production effects, such as equalization and compression, and transmission-related degradations, such as codec artifacts. We propose a three-stage framework built around this distinction: (1) a mixture restoration model that addresses degradation before separation, (2) a single model that separates the restored mixture into eight target stems (vocals, guitars, keyboards, synthesizers, bass, drums, percussion, and orchestra), and (3) stem-specific restoration experts fine-tuned on the separator's own residual artifacts. Each stage improves restoration quality over the previous one on the MSR Challenge test set. We release code and models to support future research in MSR at https://github.com/theMoro/music_source_restoration.
1. INTRODUCTION
Music Source Restoration targets unprocessed stems from recordings altered by production and transmission effects, beyond the clean-source linear-mixture assumption. The paper motivates a modular framework that assigns separation and restoration distinct roles.
1. INTRODUCTION
- MSR seeks original, unprocessed sources from recordings affected by production and transmission degradations, unlike separation systems that assume clean sources sum linearly.Examples include equalization, compression, reverberation, noise, and codec artifacts.Source: §1 · p. 1
- Prior MSR systems largely relied on separation models, while xlancelab used a BS-RoFormer cascade and DTT-BSR+ reported higher fidelity than xlancelab on five of eight stems on a private split.The first MSR benchmark used professionally mixed stem-mixture pairs evaluated under different degradations.Source: §1 · p. 1
- Masking-based separation can assign mixture content to sources but cannot directly reconstruct detail altered or lost through nonlinear processing and degradation.The cited MSRBench baselines found complex-domain separation models insufficient for restoration, motivating restoration-specific and instrument-specific strategies.Source: §1 · p. 1
- The proposed framework restores mixture-level degradation, separates the restored mixture into eight target stems, then applies source-specific experts fine-tuned on separator artifacts.Mixture restoration and separation act globally; stem restoration acts locally on source-specific artifact statistics.Source: §1 · p. 1
- The paper introduces a three-stage architecture and training regime, measures each stage’s contribution to restoration quality, and formalizes the stem-processing and degradation structure described in prior work.The formalization consolidates material described across and.Source: §1 · p. 2
2. METHOD
The method models a released recording as individually degraded stems summed together, then mastered and potentially degraded in transmission. Its framework restores the mixture, separates eight stems, and refines them with source-specific experts, with a noted training–inference mismatch.
2.1. Structured Problem Formulation
- Each clean stem receives a stem-level degradation before summation; the resulting mixture is then mastered and may undergo transmission degradation.The composite recording-level processing is represented as mastering followed by degradation.Source: §2.1 · p. 2
- MSR is modeled as recovering clean stems by inverting recording-level processing, attributing sources, and undoing residual stem-level degradation.The model treats stems as a sum but does not explicitly model practical gain-staging or bus-level processing.Source: §2.1 · p. 2
2.2. Proposed Framework
- The three-stage pipeline restores the possibly degraded recording toward the mastered mixture, separates eight target stems, then restores each stem individually.The figure presents the proposed three-stage pipeline.Source: §2.2 · p. 2, Figure 1 · p. 2
- The separator is trained on clean, unmastered mixtures, whereas Stage 1 produces an approximation of the mastered mixture, creating a train–inference mismatch left for future work.The separation model is trained independently of Stage 1.Source: §2.2 · p. 2
- Stem-restoration experts are fine-tuned using actual separator outputs rather than synthetically degraded stems, targeting the artifacts the separator produces.Each expert refines its separated stem individually.Source: §2.2 · p. 2
- Unlike prior modular systems with separate per-stem models, the framework fine-tunes one separator across all eight stems, potentially trading per-stem performance for lower total parameter count and training cost.The single-model design is the framework’s stated efficiency trade-off.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
The experiments use multiple music datasets, train mixture restoration, separation, and stem restoration models, and evaluate restoration with three objective metrics.
3.1. Data
- MSRBench is used for validation and the independent MSR Challenge test set for testing, with both resampled from 48 kHz to 44.1 kHz.Training combines RawStems, MoisesDB, Slakh2100, and MUSDB25-style extensions; RawStems is the only source of unprocessed stems.Source: §3.1 · p. 3
3.2. Models and Training
- Apollo mixture restoration is fine-tuned on degraded inputs toward undegraded mixtures, while a pretrained BS-RoFormer is extended to separate eight target stems plus an auxiliary stem.Separate Apollo experts are fine-tuned on the separator’s estimates paired with clean stems, targeting its actual residual artifacts; the separation model uses multi-resolution STFT and L1 losses, while restoration models use mel, multi-mel-SNR, and adversarial losses.Source: §3.2 · p. 3
3.3. Evaluation Metrics
- Evaluation reports MMSNR as the primary metric alongside Zimtohrli and FAD-CLAP, and includes no subjective evaluation.Challenge objective-metric rankings correlated with blind-test Mean Opinion Scores (Spearman ρ = 0.9); Table 1 covers Stage 1’s effect on Stage 2 separation performance.Source: §3.3 · pp. 3–4, Table 1 · p. 4
4. RESULT AND ANALYSIS
The final system improves substantially over the BSRNN baseline across stems and metrics, while trailing xlancelab on most stems. Mixture restoration improves all three metrics, whereas stem-restoration gains vary by stem, metric, and evaluation split.
Effect of random mixing
- 3.81 to 3.94 dB test-set MMSNR: linearly annealing random mixing from 0.9 to 0.0 over the first 85% of training outperformed every fixed probability evaluated.The result is averaged over three seeds.Source: §4 · p. 4
Separation performance
- 3.98 dB MMSNR: one model jointly separating all eight stems substantially improves over the BSRNN baseline reported on MSRBench.Table 1 reports test-set separation performance.Source: §4 · p. 4
Effect of mixture restoration
- Preconditioning the mixture with Stage 1 before separation improves all three metrics, with MMSNR rising from 3.980 to 4.047 dB, although the separator was never trained on degraded mixtures.Source: §4 · p. 4
Effect of stem restoration
- 4.047→4.050 dB overall test MMSNR: stem restoration improves MMSNR for only three of five validation-selected stems, while FAD-CLAP degrades (0.274→0.279) and Zimtohrli is essentially unchanged.The retained stems are vocals, guitars, and synthesizers; Table 2 reports per-source effects.Source: §4 · p. 4, Table 2 · p. 4
Comparison to prior systems
- The final system improves over the BSRNN baseline across all stems and metrics but trails xlancelab on most stems; it exceeds xlancelab on synthesizers (3.131 vs. 2.244 dB).It remains close to xlancelab on vocals and drums.Source: §4 · p. 4, Table 3 · p. 4
5. CONCLUSION
The results support a modular three-stage approach to MSR, with mixture restoration improving separation across all three metrics. Stem-restoration gains do not always transfer from validation to test, highlighting benchmarking and metric-evaluation challenges.
Conclusion
- Mixture restoration improves separation across all three metrics, supporting distinct model classes for separation and restoration.Source: §5 · p. 4
- Stem-restoration gains do not always transfer from validation to test because MSRBench and the challenge test set differ in distribution.Source: §5 · p. 4
- Disagreement among MMSNR, Zimtohrli, and FAD-CLAP cautions against treating improvement in one metric as comprehensive.Source: §5 · p. 4
- Future benchmarks should define target degradation regimes, separate matched-distribution evaluation from cross-condition robustness, and validate MSR-specific metrics against human listening.Source: §5 · p. 4