Source-linked AI summary
RESTORE: REal-time Steerable Music resTORation and bandwidth Extension via stem disentanglement
Meiying Chen, Benjamin R. Thompson, Michael C. Heilemann
TL;DR
Historical audio restoration has a subjective target, while neural enhancement typically maps degraded inputs to a single output. RESTORE produces user-steerable semantic stems and an isolated high-frequency extension in one pass; it reports strong restoration results, stem control, and real-time processing.
Problem
Neural restoration typically produces a single clean output despite the subjective choice of how much noise to remove from degraded historical recordings.
Source: §1 · p. 1Method
RESTORE expands HTDemucs into six semantically targeted stems and uses mixture consistency to account for degradation while keeping generated high-frequency content isolated.
Source: §1 · p. 1Results
On historical recordings, RESTORE achieves the lowest FAD across song and instrumental categories, and its stems respond monotonically to gain with Spearman correlations of 0.91–0.97.
Source: §4 · p. 4Takeaways & Limitations
The single-pass system provides real-time, post-hoc control over artifact classes while its exact-additivity target factorization mitigates hallucination risks of cascaded pipelines.
Source: §4 · p. 4Listening studies remain future work to validate perceptual quality, archival fidelity, and steerability on real recordings.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Neural methods for audio restoration are typically framed as rigid mappings from degraded inputs to single clean outputs, enforcing decisions about what audio content is removed, and potentially adding unwanted content to the restored signal. Because what constitutes a restored audio signal is subjective, we introduce RESTORE, a framework that formulates audio restoration as a six-source semantic decomposition to allow for real-time interactive user control over the process. By expanding a pretrained HTDemucs backbone, a single forward pass disentangles a degraded mixture into vocals, music, broadband hiss, impulsive transients, and an unmodeled residual, while jointly synthesizing a high-frequency extension. Users may steer the restoration by adjusting stem gains, ensuring generative content remains isolated and auditable. RESTORE improves audio quality on diverse historical recordings compared to baselines,lowering Frechet Audio Distance (FAD) (12.13 VGGish; 0.92 CLAP) and delivering aesthetic steerability (Spearman rho greather than 0.91) at 50x real-time on a single GPU. Code and audio samples are available at https://melissachen15.github.io/restore-audio-demo.
1. INTRODUCTION
Historical recordings combine severe degradation with a subjective restoration target, while conventional neural enhancement often makes fixed decisions about what to remove. RESTORE recasts restoration as a steerable, single-pass decomposition of musical content, artifacts, and generated high frequencies.
1. INTRODUCTION
- Historical recordings suffer band-limiting, hiss, impulsive transients, and wear, while the appropriate degree of noise removal is subjective.Complete noise removal may sound inauthentic compared with less complete removal.Source: §1 · p. 1
- RESTORE expands HTDemucs into six source-specific heads: vocals, music, hiss, transients, residual, and high-frequency extension, with a mixture-consistency penalty.The high-frequency extension receives targeted adversarial training, while the additive decomposition accounts for degradation in noise stems rather than discarding it.Source: §1 · p. 1
- RESTORE repurposes source separation as a semantic noise decomposer, giving users aesthetic control over restoration.Its isolated extension stem enables targeted adversarial training without corrupting the original signal.Source: §1 · p. 1
- RESTORE is designed to generalize to complex 44.1 kHz stereo polyphonic mixtures, unlike prior models that overfit to solo sources.The introduction notes that models trained on solo singing and piano struggle with complex multi-voice mixtures.Source: §1 · p. 1
2. METHOD
RESTORE adapts HTDemucs to produce six semantically targeted outputs in one network, separating existing content and degradations from synthesized high frequencies. Its training combines simulated historical degradations with objectives that enforce additive accounting and isolate adversarial learning to the extension stem.
2. METHOD
- Six specialized outputs replace HTDemucs's four fixed stems: vocals, music, broadband noise, transient noise, residual, and high-frequency extension.The HFE head isolates generated frequencies from existing signals; noise heads reuse pretrained drums weights, with transient and extension outputs attenuated at initialization.Source: §2 · p. 2
- Training simulates historical band-limiting, hiss, and clicks on clean audio, with online corruption and empirically sampled noise and click conditions.The filtering includes a randomized Butterworth low-pass response; distortion and groove echo are omitted because they did not affect preliminary experiments.Source: §2 · p. 2
- A two-stage adversarial schedule first establishes semantic separation, then trains with Multi-Period and Multi-Resolution discriminators applied exclusively to HFE.The discriminator is restricted to the extension stem to avoid corrupting stems containing surviving original content.Source: §2 · pp. 2–3
2.1. Training Objectives and Mixture Consistency
- The training objective combines waveform and multi-resolution STFT losses with mixture consistency, requiring the five band-limited stems to sum to the input without HFE.Energy-dependent spectral penalties discourage hallucinated noise in silent regions; a light residual-energy penalty encourages the catch-all residual to remain silent.Source: §2.1 · p. 3
- Adversarial penalties apply only to HFE in phase two, while its loss weight rises from 0.1 to 1.0 when the discriminator engages.The mixture-consistency penalty uses λc = 0.1; the residual-energy penalty uses λr = 10^-3.Source: §2.1 · p. 3
3. EXPERIMENTAL SETUP
RESTORE is trained on clean vocal and instrumental sources mixed into simulated degradations, then evaluated on paired synthetic mixtures and historical recordings. The evaluation compares it with commercial processing and a cascaded deep-learning baseline using objective quality, controllability, and runtime measures.
3. EXPERIMENTAL SETUP
- Training uses 3,700 clean vocal stems and 1,790 instrumental stems at 44.1 kHz stereo, with random six-second mixtures and on-the-fly degradation.Vocal sources are split between MUSDB18-HQ and VocalSet; instrumental sources come from MUSDB18-HQ, MAESTRO, MusicNet, and URMP.Source: §3 · p. 3
- The historical test set includes 30 in-domain 78 RPM recordings and 18 additional diverse recordings spanning ten categories.The in-domain subset contains Enrico Caruso, Nellie Melba, and solo piano recordings; clips are trimmed to 60 seconds.Source: §3 · p. 3
3.1. Baselines and Evaluation Metrics
- Baselines are iZotope RX 12's archival-processing sequence and BABE-2 behind a reproduced denoising cascade, with HT-Demucs also used as a backbone comparison.The BABE-2 cascade adds U-Net denoising and, for vocal recordings, HT-Demucs voice isolation; per-singer fine-tuning is omitted.Source: §3.1 · pp. 3–4
- Evaluation measures synthetic-set SI-SDR and ViSQOL, reference-free VGGish and CLAP FAD, stem-gain steerability, vocal noise leak, music bleed, and A100 runtime.Steerability is the Spearman correlation between applied stem gain and realized RMS energy; runtime is processing time divided by audio duration.Source: §3.1 · p. 4
4. RESULTS AND CONCLUSION
RESTORE outperforms baselines on synthetic and historical restoration, while its single-pass outputs permit stem-level control. The paper presents it as a semantic decomposer and bandwidth-extension engine, with independent HFE heads and listening studies left for future work.
4. RESULTS AND CONCLUSION
- 0.76 dB SI-SDR and 2.86 ViSQOL were the highest scores, and FAD-CLAP was half the strongest baseline (0.40 vs. 0.81).RESTORE outperformed all baselines on every metric for full-band restoration on synthetic data.Source: §4 · p. 4
- On historic recordings, RESTORE achieved the lowest FAD across song and instrumental categories, even with generative HFE disabled.The authors attribute the gains to separation, with HFE remaining a controllable feature for archivists.Source: §4 · p. 4, Table 3 · p. 4
- At matched bandwidth, the isolated voice stem achieved 7.65 dB SI-SDR versus HTDemucs’s 4.21 dB and left less noise in voiced frames.Voiced-frame noise was −8.36 dB for RESTORE versus −5.38 dB and −2.86 dB for the compared systems.Source: §4 · p. 4
- HT-Demucs improved only slightly, with FAD-CLAP 0.83 vs. 0.86, which the authors say points to task formulation and training rather than model capacity.RESTORE and HT-Demucs share architecture and size.Source: §4 · p. 4
- A single forward pass runs at RTF 0.0195—about 50× faster than the U-Net and over 1,300× faster than BABE-2—with 3.7× fewer parameters.Stem gains correlate monotonically with output changes (Spearman 0.91–0.97), and music bleed into transient and broadband stems is near zero.Source: §4 · p. 4
- The paper concludes that one multi-task network can combine semantic decomposition and generative bandwidth extension; future work includes independent HFE heads and listening studies.The proposed exact-additivity target factorization is described as mitigating hallucination risks from cascaded pipelines.Source: §4 · p. 4