Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

1. INTRODUCTION

• Many interacting synthesizer controls and multiple parameter configurations producing similar signals make sound matching challenging and the inverse mapping generally ill-posed.Existing approaches either optimize parameters for each target or learn an amortized audio-to-parameter mapping.Source: §1 · p. 1
• Candidate-based search typically renders each parameter set, while learned proxies depend on proxy accuracy and representation choice.Generic audio representations are not learned from synthesizer parameter–sound relationships.Source: §1 · p. 1
• Synth-JEPA learns mutually predictive audio and parameter representations from paired synthesizer data, then scores candidate parameters without synthesizer calls.The learned audio representation depends directly on parameter–audio correspondences.Source: §1 · p. 1
• Unlike prior joint DX7 audio–parameter embeddings for preset retrieval [14], Synth-JEPA uses joint representations for iterative parameter search and compares SLAP-style EMA with SIGReg.The related setup adapts the SLAP EMA-teacher objective for retrieval.Source: §1 · pp. 1–2

2. SYNTH-JEPA

• A uniform prior samples continuous and discrete synthesizer controls, but its induced sound distribution is generally non-uniform and may affect search geometry.The paper identifies audio-aware parameter sampling as future work.Source: §2 · p. 2

2.1. Cross-domain predictive learning

• Separate audio and parameter encoders feed cross-domain predictors trained with a predictive objective using stop-gradient targets.SIGReg regularizes each encoder output toward an isotropic Gaussian; an EMA-teacher variant explores an alternative anti-collapse mechanism.Source: §2.1 · p. 2

2.2. Renderer-free parameter search

• At inference, the target audio is encoded once and candidate parameters are scored by predicting their audio-space representation, without synthesizer calls.The differentiable objective supports gradient-free or gradient-based optimization; the method combines JADE population search with gradient refinement of leading candidates.Source: §2.2 · p. 2

3.1. Data and tasks

• Training uses online-rendered Surge XT audio with 139 active parameters; evaluation includes held-out in-domain targets and NSynth and FSD50K clips.The external clips are trimmed to three seconds, resampled to 44.1 kHz, and upmixed to stereo.Source: §3.1 · p. 2

3.2. Models and baselines

• Comparisons include amortized regression and flow-matching models, renderer-in-the-loop objectives, and neural proxies predicting log-Mel spectrograms or pretrained audio embeddings.Renderer-in-the-loop candidates are rendered, whereas Synth-JEPA and proxy objectives also permit gradient-based refinement.Source: §3.2 · p. 3

3.3. Evaluation

• Evaluation reports multi-scale spectral distance, warped MFCC distance, and cosine similarity between fixed CLAP embeddings; Fig. 2 scales search budget or ODE steps and reports per-target wall-clock time.Wall-clock measurements use one RTX 3090.Source: §3.3 · p. 3, Figure 2 · p. 3

4.1. Sound-matching performance

• Synth-JEPA (SIGReg) leads all in-domain metrics and is best or second-best out-of-domain except wMFCC, where log-Mel-based objectives lead.The comparison uses a budget of 2,048 objective evaluations for search-based methods.Source: §4.1 · p. 3
• The learned Ea representation gives the strongest renderer-in-the-loop MSS results across datasets; Synth-JEPA broadly beats proxy baselines in-domain and on NSynth, but results are more mixed on Freesound.The paper attributes the renderer-in-the-loop result to the parameter-induced embedding geometry benefiting search.Source: §4.1 · pp. 3–4

4.2. Inference-time scaling

• Synth-JEPA remains strongest at every tested best-of-k value, although both it and flow matching improve when rendered candidates are selected by log-Mel L1 distance.This optional selection renders candidates, so it is not strictly renderer-free; flow matching improves more steeply from its k = 1 baseline.Source: Figure 3 · p. 4, §4.2 · p. 4
• Flow matching improves rapidly with more ODE steps but largely saturates beyond 16, while Synth-JEPA continues improving with search budget and other search methods do not match it.Synth-JEPA takes longer to reach flow matching’s performance.Source: §4.2 · p. 4

4.3. Perceptual evaluation

• 85.1% of trials favored Synth-JEPA overall, with preference rates of 89.1% in-domain and 81.0% out-of-domain.Preference was strongest against RITL and weakest against flow matching.Source: Figure 4 · p. 4, §4.3 · p. 4

• Synth-JEPA learns mutually predictive audio and parameter representations from paired synthesizer data and uses them as a renderer-free objective for parameter search.Source: §5 · p. 4
• The authors report that Synth-JEPA consistently outperforms parameter-prediction baselines, and that its embedding space is more effective for search than generic audio representations in the compared settings.Source: §5 · p. 4
• Listeners preferred Synth-JEPA-produced matches in 85% of pairwise trials.Source: §5 · p. 4
• The presented model was not trained for nonsynthesizer audio; exploring additional self-supervised objectives and extending the framework to further synthesizers are future work.Source: §5 · p. 4

• All participants gave informed consent, and data collection followed the principles of the Declaration of Helsinki.Source: §7 · p. 5