Source-linked AI summary
End-to-End Historical Music Restoration in Latent Space
Steven Cho, Junghyun Koo, Raphael Lafargue, Tushar Dhyani, Eloi Moliner, Yuki Mitsufuji
TL;DR
Orchestral historical music restoration is less explored than super-resolution and solo-piece restoration, in part because historical recordings lack clean pre-degradation references. The paper trains a latent CFM restoration model using synthetic degradation pairs; it reports the strongest overall perceptual and paired-reconstruction performance, with a quality–preservation trade-off.
Problem
Historical music restoration has focused largely on super-resolution and solo pieces, while orchestral restoration lacks clean pre-degradation reference pairs.
Source: §1 · p. 1Method
The paper synthesizes paired training recordings with a five-stage historical degradation function and trains an end-to-end latent conditional flow-matching restoration model.
Source: §1 · p. 1Results
Across objective and human evaluations, the latent CFM model achieves the strongest overall perceptual and paired-reconstruction performance while maintaining strong preservation of the historical input.
Source: §5 · p. 4Takeaways & Limitations
The study frames orchestral historical music restoration as a blind inverse problem extending beyond conventional super-resolution and single-instrument restoration.
Source: §5 · p. 4The formulation assumes a same-duration clean high-fidelity source and represents degradation as additive gramophone noise plus non-additive processes.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Historical music restoration (HMR) has almost exclusively focused on constrained problems such as Super-Resolution or the restoration of solo pieces, under-exploring the general task of restoring orchestral historical music, which has multiple instruments. This under-exploration is largely because the HMR domain, early-20th-century recordings, has no pre-degradation ground-truth pairs, making the restoration task unsupervised and more challenging. This paper presents a supervised end-to-end orchestral HMR benchmark by exploring both the synthetic degradation functions and the end-to-end generative deep-learning restoration methods. We simulate the historical recording degradation chain more faithfully than prior work, which makes orchestral restoration into a tractable supervised problem. A latent flow-matching model trained on the resulting synthetic pairs outperforms existing HMR baselines on intrusive, non-intrusive, and subjective evaluations. We also curate and release a 9.3-hour license-free, unpaired, historical classical-music test set, along with code and audio demos.
1. INTRODUCTION
Historical music restoration remains concentrated on super-resolution and solo pieces, while orchestral recordings pose a broader restoration problem. The paper introduces synthetic supervision and latent flow matching for orchestral restoration, alongside a real historical test set.
1. INTRODUCTION
- The paper defines historical music as music published before 1926, when electrical recording was popularized.Source: §1 · p. 1
- Prior work addressed synthetic-noise denoising and super-resolution, while unpaired blind restoration had been demonstrated only for piano and vocals.Historical recordings also contain recording-dependent nonlinear distortions and spectral attenuations, beyond ideal low-pass filtering.Source: §1 · p. 1
- The proposed latent-space CFM model trained with synthetic supervision outperforms previous historical music restoration models.The approach uses a degradation function that includes bandwidth limitation, spectral coloration, gramophone noise, and nonlinear behavior.Source: §1 · p. 1
- The paper introduces an unpaired test set of 149 real historical orchestral recordings for evaluating restoration.Source: §1 · p. 1
- A five-stage synthetic degradation function based on a Wiener–Hammerstein structure creates paired training data.Source: §1 · p. 1
2. METHODS
The methods formulate orchestral historical music restoration as a blind inverse problem, then create synthetic supervision with a stochastic historical degradation chain and a latent-space conditional flow-matching model.
2.1. Problem Definition
- Historical restoration estimates a clean recording from an observed one, although both the non-additive operator A and additive degradation n are unknown.The resulting blind inverse problem is severely ill-posed, so synthetic supervision aims to provide priors on plausible music and degradation.Source: §2.1 · p. 2
2.2. Synthetic Degradation Function
- A five-stage Wiener–Hammerstein degradation combines pre/post spectral filters, a static nonlinearity, bandwidth limiting, and additive gramophone noise.Source: §2.2 · p. 2
- Parameters are resampled per example from a historically informed distribution; MAD validation favored the proposed degradation over common alternatives except Gramophone Only.The parameter distribution is designed to balance robustness to unknown historical conditions against realism.Source: §2.2 · p. 2
2.3. Restoration Model
- The restoration model encodes paired clean and degraded audio into latents, then uses conditional flow matching to predict a clean latent from the degraded condition.At inference, the learned velocity field is integrated with an Euler ODE solver.Source: §2.3 · pp. 2–3
- Table 2 evaluates autoencoder encode/decode reconstruction using Mel-MSE with equal-window aggregation.Source: Table 2 · p. 3
3. EXPERIMENTAL SETUP
The setup pairs a latent conditional flow-matching model with synthetic classical-music training data and evaluates it on paired and historical unpaired test sets. Objective metrics and listener ratings assess restoration quality and preservation across orchestral recordings.
3.1. Autoencoder and Backbone Architecture
- The latent conditional flow-matching backbone uses the SAME-L autoencoder and a transformer with approximately 40M parameters, trained for 400k steps and sampled with 10 Euler steps at inference.The transformer uses 576-dimensional tokens, 8 AdaLN-Zero blocks, 9 attention heads, and an MLP width of 1152.Source: §3.1 · p. 3
3.2. Training and Validation Data
- Training uses a 6.82-hour instrumental-classical corpus, including 5.17 hours of Full-Orchestra music and 1.65 hours of brass, woodwind, and string sections.Section mixtures expose instrument-family timbre and high-frequency structure; the study restricts restoration to classical music and does not address vocals.Source: §3.2 · p. 3
3.3. Test Data
- The 9.3-hour unpaired historical test set contains instrumental orchestral classical recordings published in the United States from 1911 to 1925, divided into Full- and Light-Orchestra groups.A paired test set uses 53 minutes of Full-Orchestra recordings from Spheres.Source: §3.3 · p. 3
3.4. Objective Evaluation
- Four objective metrics jointly assess production quality, tonal and timbral reconstruction, musical-semantic similarity, and divergence from ground-truth audio distributions.The metrics are AA-PQ, Mel-MSE, CLAP Cos., and MAD; reference-free metrics are used on the historical unpaired set.Source: §3.4 · p. 3
3.5. Subjective Evaluation
- MOS-Q rates audio quality and MOS-P rates preservation of the historical input, using ten 5-second excerpts, 22 valid listeners, and 88 ratings per method for each test.The excerpts were split evenly between Full-Orchestra and Light-Orchestra recordings; scores are reported with 95% confidence intervals.Source: §3.5 · pp. 3–4
4. RESULTS
SAMECFM achieved the best unpaired metric and subjective scores, while the experiments also found an advantage for the broader degradation function and for degradation matching over sub-genre matching. The results indicate a quality–preservation trade-off not fully captured by a single objective metric.
4. RESULTS
- The comparisons included pretrained and FOS-domain-adapted BEHM-GAN, BABE2, and Denoiser against SAMECFM.Domain adaptation used the FOS dataset with the proposed historical degradation function.Source: §4 · p. 4
- The WH + Gram. Noise degradation function supervised better restoration than the Gram. Noise Only ablation across objective and subjective metrics.Gram. Noise Only had lower MAD in Table 1; the authors suggest that several embedding metrics were not intended to represent historical degradation, while SAME-L may encode more degradation-related information.Source: §4 · p. 4
- The shift from unpaired to paired testing produced greater AA-PQ improvements than the shift from Light-Orchestra to Full-Orchestra for pretrained baselines and FOS-adapted methods.The authors interpret this pattern as indicating that accurate target-degradation simulation matters more than sub-genre alignment.Source: §4 · p. 4
- SAMECFM achieved the best unpaired metric, MOS-Quality, and MOS-Preservation scores despite having the lowest parameter count.The authors also report that generative methods slightly sacrificed preservation for quality, and that SAMECFM and SAMECFM-G had third- and seventh-ranked MAD scores on the paired test set, respectively.Source: §4 · p. 4
- Among objective metrics, AA-PQ best aligned with MOS-Q, with Pearson correlations of r = 0.920 for Full-Orchestra and r = 0.867 for Light-Orchestra, supporting its use as a reference-free quality metric.SAMECFM and SAMECFM-G received the highest MOS-Preservation scores, while their paired-test MAD rankings were third and seventh, respectively.Source: §4 · p. 4
5. CONCLUSION
The paper frames orchestral historical music restoration as a blind inverse problem beyond conventional super-resolution and single-instrument restoration. Its latent CFM model has the strongest overall perceptual and paired reconstruction performance in the reported evaluations, while the results expose a quality–preservation trade-off not fully captured by one objective metric.
5. CONCLUSION
- The paper frames orchestral historical music restoration as a blind inverse problem extending beyond conventional super-resolution and single-instrument restoration.Source: §5 · p. 4
- Across objective and human evaluations, domain adaptation with Full-Orchestra and section-level data and a broader nonlinear degradation function improves restoration quality.Source: §5 · p. 4
- The latent CFM model achieves the strongest overall perceptual and paired reconstruction performance while maintaining strong input preservation, but one objective metric does not fully capture the quality–preservation trade-off.Future work aims to minimize hallucinations and extend restoration beyond orchestral classical music.Source: §5 · p. 4