PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Destination-Aware synthesizer Modulation Recovery

David Liu, Giulio Cengarle, David Cooper, Mark Vinton, Haici Yang

arXiv:2610.00642v1eess.AS

TL;DR

Prior modulation recovery is limited to fixed destinations and clean audio, and does not jointly address destination-specific LFO and waveform recovery. This paper builds an interpretable pipeline for destination detection, LFO and waveform recovery, and perceptual training; it reports accurate recovery for noisy single-oscillator audio and improved synthesis quality.

  • Problem

    Prior modulation extraction targets fixed destinations and clean audio, while holistic recovery of destination-specific LFOs and waveforms with perceptual losses has not been attempted.

    Source: §1 · p. 1

  • Method

    The pipeline detects active destinations, regresses destination-conditioned LFO curves, models oscillator waveforms, and co-trains regression with perceptual losses.

    Source: §1 · p. 1, §2.2 · p. 2, §2.3 · p. 2

  • Results

    The system accurately identifies destinations and reconstructs LFO and waveform shapes for noisy single-oscillator audio, with greatly improved synthesis quality.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Correct destination prediction can yield more performance gain than more accurate curve prediction that risks cross-destination interference.

    Source: §4.1 · p. 3

    Perceptual-loss-only training did not work; the authors say auxiliary LFO or waveform loss appears necessary to ground optimization.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Synthesizer programming is a challenging task, particularly in configuring modulation: the time-varying control of parameters such as pitch, filter cutoff, and oscillator level by modulators such as Low-Frequency Oscillators (LFOs). Prior work has shown that modulation can be reconstructed from the clean audio through curve matching, yet the recovered parameters are often untransferable to modern synthesizers. We present an interpretable destination-aware modulation and waveform recovery pipeline that mirrors how musicians often recreate sounds on a synthesizer. First, the system identifies the modulated destinations; then it recovers the LFO shape associated with each destination, and estimates the oscillator waveform to match the source timbre. The training is supported by a differentiable synthesizer that incorporates a noise oscillator, and more modulation options than previous work. We carefully train the models for optimal perceptual quality, with extensive exploration of perceptual losses and adversarial training. Through objective and subjective evaluation, we show that predicting the destinations correctly is essential, our model excels in the modulation focused inverse synthesis tasks, and our Gammatone loss and CQT discriminator significantly outperform the traditionally-used MSS loss in improving perceptual quality. We provide audio samples from the subjective test.

1. INTRODUCTION

Modern synthesizers expose many parameters and modulation destinations, while distinct settings can produce perceptually similar audio. Prior modulation-recovery work leaves gaps in destination flexibility and holistic recovery of modulation and waveform.

1. INTRODUCTION

  • Hundreds of parameters, dynamic LFO control, and many-to-one mappings make recreating synthesizer sounds challenging.Multiple distinct parameter configurations can produce perceptually similar audio.Source: §1 · p. 1
  • Modern differentiable synthesizers broaden controllable parameters, but prior inversion largely uses end-to-end regression or generative approaches that trade interpretability for generative power.Source: §1 · p. 1
  • Existing modulation extraction is limited to fixed destinations and clean audio, and no prior method holistically recovers destination-specific LFOs and waveforms with perceptual losses.Source: §1 · p. 1
  • The proposed interpretable pipeline detects active destinations, recovers each destination’s modulation shape, and learns the oscillator waveform to match source timbre.Source: §1 · p. 1
  • Modulation changes spectral shape over time, complicating waveform inference; the authors address the many-to-one issue with perceptual losses and phase-invariant waveform loss.Source: §1 · pp. 1–2

2. METHODS

The methods combine a differentiable subtractive synthesizer with destination classification, destination-conditioned LFO regression, waveform modeling, and perceptual and adversarial training.

2.1. Custom Synthesizer Design and Setup

  • A differentiable subtractive synthesizer routes LFOs to five destinations and supports end-to-end training through its oscillator, noise, filters, envelope, and output processing.The destinations are oscillator level, coarse pitch, noise level, highpass cutoff, and lowpass cutoff.Source: §2.1 · p. 2

2.2. LFO and Waveform Recovery Pipeline

  • A multi-label classifier predicts active destinations, then a destination- and f0-conditioned regressor recovers LFO curves while suppressing concurrent-destination interference.Chunking the input into small windows did not improve modeling accuracy.Source: §2.2 · p. 2
  • Waveform modeling uses unconditioned shape matching and phase-invariant spectral loss; larger model capacity helps recover timbre, while pitch curves are mean-aligned to CREPE-derived f0.Recovered LFOs are also smoothed and reconstructed through keypoints using linear interpolation or cubic splines.Source: §2.2 · p. 2

2.3. Perceptual Quality Improvement

  • Perceptual losses co-train LFO and waveform regression; explored losses include harmonic, multiscale mel-spectrogram, and Gammatone losses alongside MSS.The Gammatone loss measures reconstruction error in a fixed ERB-spaced filterbank domain.Source: §2.3 · p. 2
  • Adversarial training augments reconstruction losses with MPD, MSD, MRD, or a CQT discriminator.Source: §2.3 · p. 3

3. EXPERIMENT

Experiments use synthetic and Serum 2 audio to train and test destination, LFO, and waveform models, then compare a destination-specific baseline and an unconditioned ablation using objective and listening evaluations.

3.1. Dataset and Training

  • Synthetic data varies active destinations and oscillator waveforms, while Serum 2 test audio uses one waveform oscillator, noise, filters, and similarly generated LFOs.The Serum 2 data is treated as gray-box because it sounds more refined for unknown software reasons.Source: §3.1 · p. 3
  • The classifier trains on 5Dest, while LFO and waveform models train concurrently on 2Dest using ground-truth f0 and destinations; inference uses CREPE f0 and predicted destinations.Models use AdamW at 1e-4 until convergence, with batch sizes of 64 or 32 for adversarial training.Source: §3.1 · p. 3

3.2. Baseline and Evaluation

  • The baseline trains separate destination-specific models with f0 conditioning, while the NC ablation removes both classification and f0 conditioning.Evaluation combines objective metrics with a MUSHRA-style study using synthetic and Serum 2 samples; 12 listening experts participated.Source: §3.2 · p. 3

4. RESULTS

Destination conditioning outperformed the baseline, and the NC result underscores the importance of identifying active destinations. Perceptual-loss results varied by metric and test setting, while Gammatone loss and a CQT discriminator showed specific benefits; perceptual loss alone did not work in the tested setup.

4.1. Class Conditioning Analysis

  • Even though NC cannot model every LFO curve, it outperformed baseline, indicating correct destination prediction can matter more than curve accuracy with interference.The baseline also occasionally confused curves across destinations or produced curves for inactive destinations.Source: §4.1 · p. 3

4.2. Perceptual Loss Analysis

  • MSE + Gammatone performed best on MSS and MFCC-L1, closely followed by MSE-only; subjective preferences differed across test settings.MSE losses had an overall edge in the subjective test; Gammatone was rated best on Serum 2, and perceptual losses showed benefit in the Serum 2 gray-box setup.Source: §4.2 · pp. 3–4, Table 1 · p. 4
  • Scaling the model from Small to Large yielded a substantial improvement on 2Dest, with waveform-model size matched to LFO-model size.The baseline used the same waveform model size as the Large setup.Source: Table 2 · p. 4
  • Gammatone was the only perceptual loss reported to improve metrics, while MSS and mel-spectrogram losses had gradients poorly correlated with perceptual distance between patches.The authors suggest Gammatone's modeling of human auditory perception may explain its benefits.Source: §4.2 · p. 4
  • Adding the CQT discriminator clearly improved on MPD, MSD, and MRD, which were designed for speech synthesis rather than musical audio.Its log-frequency axis and constant relative bandwidth make harmonic relationships consistent across fundamental frequencies, providing a stronger training gradient.Source: §4.2 · p. 4
  • Perceptual loss alone did not work in the tested setup; auxiliary LFO or waveform loss appeared necessary to ground optimization in parameter-level supervision.The authors suggest that more active modulation destinations could potentially make perceptual loss more useful.Source: §4.2 · p. 4

5. DISCUSSION

The system identifies modulation destinations and reconstructs LFO and waveform shapes for single-oscillator audio, including under noisy conditions. Its current synthesizer omits several features of professional synthesizers, though the architecture allows extension.

5. DISCUSSION

  • The system accurately identifies modulation destinations and reconstructs LFO and waveform shapes for single-oscillator audio, demonstrating tractable reverse engineering under noisy conditions.It also delivers greatly improved synthesis quality with better perceptual treatment.Source: §5 · p. 4
  • The system delivers greatly improved synthesis quality with better perceptual treatment.Source: §5 · p. 4
  • The current system has restricted filter types, lacks wavetable oscillators, and omits reverb, distortion, and flanger because time-varying effect parameters are out of scope.Source: §5 · p. 4
  • The model and synthesizer capacity does not match modern synthesizers, but the architecture allows easy extension to additional modulation destinations.Source: §5 · p. 4
  • Future work will explore multiple oscillators, custom filter types, and effect modulation toward reverse engineering professional synthesizer patches.Source: §5 · p. 4
Loading 2610.00642v1…