Source-linked AI summary
Off-manifold robustness in synthesizer inversion with joint distribution flow matching
Ben Hayes
TL;DR
Synthesizer inversion models trained on paired synthesized audio and parameters can face a train-test mismatch on real, off-manifold recordings. This paper learns a joint audio-parameter flow with independent noise schedules, enabling unpaired real audio to train the audio marginal. On SURGE XT and DEXED, it reports substantially better real-audio inversion without sacrificing synthetic-audio performance.
Problem
Synthesizer-inversion models need paired audio and parameter labels, typically from the synthesizer, while real-world recordings can be off-manifold and lack those annotations.
Source: §1 · p. 1Method
The method learns a joint audio-parameter flow with independent noise schedules, allowing paired synthesizer data and unpaired real recordings to train joint and audio-marginal distributions.
Source: §1 · pp. 1–2Results
The method substantially improves real-audio inversion over a generative baseline on SURGE XT and DEXED without sacrificing synthetic-audio performance.
Source: §6 · p. 6Takeaways & Limitations
Partially noised audio conditioning at inference trades reference specificity against robustness to off-manifold detail while preserving higher-level structure.
Source: §1 · p. 2For off-manifold audio, the notion of a matching synthesizer patch is loosely defined, making human evaluation an important component of subsequent work.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Recent work on synthesizer inversion shows that generative models outperform deterministic approaches by explicitly modeling the ambiguity in mapping audio to parameters. Training such models, however, requires audio-parameter pairs, which are typically obtained by rendering sampled or preset parameters through the synthesizer itself. This creates a train-test mismatch that can degrade performance on off-manifold real-world recordings, for which ground-truth parameter annotations do not exist. To circumvent this obstacle, we propose to model the joint distribution of audio and parameters with a multi-modal continuous normalizing flow using independent noise schedules for each modality. This formulation allows us to train joint and conditional densities with paired synthesizer data, while unpaired real recordings can train the audio marginal alone, exposing the model to off-manifold signals without requiring parameter labels. Further, because the model learns to map from audio to parameters at all noise levels, we find that partially noising the audio reference at inference improves real-audio reconstruction, consistent with reducing sensitivity to distribution-specific detail while preserving coarse structure. Evaluating on Surge XT and Dexed, we find that modelling the joint distribution substantially improves both inversion of real-world and in-domain audio.
1. INTRODUCTION
Synthesizer inversion is an ill-posed task, and models trained on synthesized audio can struggle with real recordings that lie off the synthesizer’s audio manifold. The paper uses joint multi-modal flow matching to incorporate unpaired real audio and reports improved real-audio inversion while maintaining competitive in-domain performance.
1. INTRODUCTION
- Off-manifold real recordings create a train-test mismatch because paired audio-parameter training data comes from the synthesizer, and performance degrades on real audio.Figure 1 depicts synthesized audio on a manifold within the ambient audio space, with real inputs potentially off that manifold.Source: §1 · p. 1, Figure 1 · p. 1
- Differentiable synthesis and reinforcement-learning fine-tuning offer routes to real-audio supervision, but each brings costs that limit its use as a primary training approach.The paper cites implementation dependence and optimization challenges for differentiable synthesis, and scaling and design costs for reinforcement learning.Source: §1 · p. 1
- A joint flow over audio and parameters uses independent noise schedules to learn more than the conditional parameter distribution given audio.This formulation also admits the audio marginal, so unpaired real recordings can contribute training data.Source: §1 · pp. 1–2
- Training on partially noised audio states reduces extrapolation from the synthesized-audio manifold, while partially noised inference conditioning can suppress recording-specific detail and preserve higher-level structure.The paper presents both as ways the factorized noise schedule can support off-manifold robustness.Source: §1 · p. 2
- The approach substantially improves inversion of real-world audio on SURGE XT and DEXED while maintaining competitive in-domain performance.The comparison is against a prior generative inversion baseline.Source: §1 · p. 2
2. BACKGROUND
Synthesizer inversion is ambiguous because different parameter configurations can produce the same audio, while training labels are typically available only for synthesized signals. Flow matching provides a training objective for continuous normalizing flows, and multi-modal flow matching extends it to joint, conditional, and marginal tasks.
2.1 Synthesizer Inversion
- Multiple parameter configurations can produce the same signal, making deterministic regression prone to averaging equivalent solutions; generative models can represent multiple plausible parameter solutions but are trained on synthesized audio and can degrade on real recordings.DDSP is also deterministic and requires a differentiable synthesizer implementation.Source: §2.1 · p. 2
2.2 Flow Matching
- Flow matching trains a continuous normalizing flow by regressing a model’s velocity field using a tractable conditional objective, then generates samples by integrating the learned field from noise to data.The conditional objective has the same minimizer as the objective based on the marginal velocity field.Source: §2.2 · p. 2
2.3 Multi-modal Flow Matching
- Multi-modal flow matching assigns each modality its own time variable, with paths through the time cube corresponding to different generation tasks and supporting training on heterogeneous data with missing modalities.For audio and parameters, the time-square interior fits the joint distribution, edges fit conditional and marginal distributions, and sampling paths select inference tasks.Source: §2.3 · p. 3, Figure 3 · p. 3
3. JOINT DISTRIBUTION FLOW MATCHING FOR SYNTHESIZER INVERSION
The method learns a joint flow over synthesizer parameters and audio using independent noise schedules, so paired synthesizer data supports conditional inversion while unpaired recordings train the audio marginal. Different paths through the time square provide joint, marginal, or partially noised-audio inference.
3. JOINT DISTRIBUTION FLOW MATCHING FOR SYNTHESIZER INVERSION
- Independent noise schedules define per-modality interpolants, and a joint flow-matching objective trains the model to predict parameter and audio velocities over the time square.The training pairs combine synthesizer parameters and their rendered audio with independent noise samples.Source: §3 · p. 3
- The time-square edges recover conditional and marginal interpolants, while full interior support fits the joint distribution and its conditional and marginal paths.The top edge conditions parameter generation on clean audio; the left edge models the audio marginal.Source: §3 · p. 3
- Unpaired real-audio examples train only the audio-velocity term on the audio-marginal edge, allowing recordings without parameter labels to be used directly.The joint objective’s sampling distribution determines which regions of the time square constrain the learned velocity field.Source: §3 · p. 3
- Inference paths select the task: the top edge samples parameters given reference audio, the diagonal samples the joint distribution, and interior paths condition on partially noised audio.Interior paths yield a smoothed posterior that trades reference specificity against robustness to off-manifold detail.Source: §3 · p. 3
4. EXPERIMENTS
The experiments evaluate inversion on synthesized and real-world audio using SURGE XT and DEXED, with controls intended to isolate joint modeling and unpaired-audio supervision. The supplied passages describe the datasets, model variants, training and inference procedures, and reconstruction metrics, but not comparative outcomes.
4. EXPERIMENTS
- The evaluation tests in-domain synthesized and off-manifold real-world audio while controlling architecture and training budget to isolate joint modeling and unpaired-audio inclusion.The paper reports full ablations only on SURGE XT to use available computational resources efficiently.Source: §4 · p. 3, §4.1 · p. 3
4.1 Data
- SURGE XT is a hybrid subtractive/wavetable testbed, while DEXED jointly predicts its algorithm and continuous parameters across all 32 routings.DEXED’s parameter semantics depend on the algorithm, making this a harder setting than fixing or limiting the routing.Source: §4.1 · p. 3
- Paired examples are generated by uniformly sampling synthesizer parameters and rendering audio; low-RMS samples are rejected, and validation and test sets are frozen at 10k examples per synthesizer.Audio effects and known nondeterministic modulator settings are disabled.Source: §4.1 · pp. 3–4
4.1.1 Off-manifold audio
- The off-manifold corpus contains 617,016 unlabeled training files of isolated sounds from Freesound and a proprietary one-shot library, used only on the audio-marginal edge.Held-out validation and test sets each contain 10k files.Source: §4.1.1 · p. 4
4.2 Model variants
- Four variants compare joint-density flow (JDF), JDF with real-audio marginal training, a unified architecture trained only on the inversion edge, and an encoder-conditioned baseline.JDF uses the full time square; JDF-A adds synthetic and unpaired real audio on the audio-marginal edge.Source: §4.2 · p. 4
4.2.1 Architecture and training
- All models train for 1.5M steps with rectified flow and linear interpolants; classifier-free guidance uses different unconditional branches depending on each variant’s training edges.Time variables use a logit-normal schedule except when fixed, and guidance weights are swept.Source: §4.2.1 · p. 4
4.2.2 Inference
- Inference integrates the learned velocity field with a 20-step Euler solver, sweeping conditioning-noise level and guidance weight for inversion.The reference audio is fixed along the top edge, possibly after partial noising; guidance is applied over [0.15, 0.95].Source: §4.2.2 · p. 4
4.3 Metrics
- Reconstruction is assessed with MSS, wMFCC, SOT, and RMS on separate 10k-example in-distribution and off-manifold test conditions.Each metric compares input audio with its resynthesis from inferred parameters.Source: §4.3 · p. 4
5. RESULTS
Across two synthesizers, JDF variants improve real-audio inversion over Conditional without sacrificing synthetic-audio performance. The ablations associate these gains with joint training and show that noisy conditioning can further help real-audio reconstruction.
5. RESULTS
- MSS falls from 15.50 to 7.68 on SURGE XT and 13.03 to 7.98 on DEXED for JDF/JDF-A versus Conditional, while synthetic-audio performance also improves across metrics.The synthetic-audio comparison covers all metrics and both synthesizers.Source: §5 · p. 4, Table 1 · p. 5, Table 2 · p. 5
5.1 Training on off-manifold audio
- On SURGE XT off-manifold, JDF-A and JDF tie at MSS 7.68 vs. 7.69, while JDF is slightly better in-distribution.JDF-A adds an unpaired-real-audio marginal to JDF, which trains only on synthesizer audio.Source: §5.1 · p. 5
5.2 Isolating the effect of joint distribution training
- Unif-PARAMONLY’s slight gain over Conditional but worse off-manifold performance than JDF/JDF-A suggests shared-token architecture contributes modestly, while joint training is central.JDF uses direct self-attention between audio and parameter tokens in a shared-token DiT, unlike Conditional’s audio encoder and parameter decoder.Source: §5.2 · p. 5
5.3 Classifier-free guidance for free
- CFG modestly improves in-domain performance; off-manifold, Conditional worsens monotonically on SURGE XT and is nearly flat on DEXED, while other models improve.For Conditional on DEXED, the shallow off-manifold optimum is near guidance weight 3.Source: §5.3 · pp. 5–6
5.4 Noisy conditioning as posterior smoothing
- For JDF/JDF-A, real-audio MSS improves sharply near conditioning noise τ = 0.1 after slight low-noise degradation; on-manifold performance stays stable except at the highest noise.Unif-PARAMONLY degrades under noise, while Conditional improves slightly off-manifold when its input is noised.Source: §5.4 · p. 6
5.5 Joint sampling from a partially noised reference
- Joint sampling is possible for JDF variants and becomes robust once start level t(0)y ≳ 0.25, but does not beat fixed-noise conditioning; Unif-PARAMONLY works only at the inversion endpoint.JDF variants remain relatively insensitive to path choice, including the reversed path; Unif-PARAMONLY performs poorly away from its training edge.Source: §5.5 · p. 6, §5.4 · p. 6
6. CONCLUSION
Joint audio-parameter modelling improves off-manifold robustness while preserving in-domain performance. The authors identify open evaluation and design questions, alongside possible extensions to training data and fine-tuning.
6. CONCLUSION
- Experiments on SURGE XT and DEXED show substantial improvements over a generative baseline on real-audio inversion without sacrificing synthetic-audio performance.The conclusion attributes this to reframing inversion as joint modelling rather than purely conditional estimation.Source: §6 · p. 6
- A single multi-modal rectified flow can incorporate paired synthesizer data and unpaired real recordings in one objective, with partially noised conditioning offering an inference-time specificity–robustness trade-off.The authors also suggest adding further training-data views and exploring reinforcement-learning fine-tuning.Source: §6 · p. 6
- For off-manifold audio, a matching synthesizer patch is loosely defined, so the authors recommend human evaluation and failure-mode analysis; some design choices also warrant ablation.They specifically identify exploration of the time-square sampling distribution as a future avenue.Source: §6 · p. 6
8. AI USAGE STATEMENT
The authors used AI assistance for research-code tooling and manuscript drafting, with human review of code outputs and scientific claims.
8. AI USAGE STATEMENT
- An AI coding assistant supported experiment-management tooling and result collation and plotting, while an LLM aided manuscript drafting and editing; authors verified code functionality and approved scientific claims, results, and wording.Source: §8 · p. 7
9. ETHICS STATEMENT
The authors describe their data sources and note that their synthesizer choice reflects western popular music production conventions; generalization to other synthesis paradigms remains unverified.
9. ETHICS STATEMENT
- Training used synthetically generated audio, public datasets, and legally obtained private datasets.Source: §9 · p. 7
- The synthesizer choice reflects western popular music production conventions, and generalization to other synthesis paradigms has not been empirically verified.The authors frame inversion as augmenting rather than replacing creative workflows.Source: §9 · p. 7