AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text

• Audio generation still trades compression against fine-detail fidelity, and waveform outputs are difficult to edit at the note, filter, or modulation level (Zeghidour et al., 2021; Défossez et al., 2022; Kumar et al., 2023).Source: §1 · p. 1
• Jointly generating performance events, timbral parameters, and modulation structures remains comparatively underexplored because these controls must evolve together.Source: §1 · p. 1
• Synthesizer parameters are interpretable, editable controls whose sound-design aesthetics involve more than waveform fitting.Source: §1 · p. 2
• Synthesizer control spaces complicate learning because interfaces differ, parameters depend on one another, modulation routes vary in length, and native plugins generally lack direct backpropagation.Source: §1 · p. 2
• AutoSynth autoregressively generates MIDI events, synthesizer parameters, and modulation routes as a complete executable program, using a jointly trained audio encoder for conditioning.Source: §1 · p. 2
• A unified sequence represents performance events, fixed parameters, and variable-length routes, while automatically constructed supervised data and mixed-reward GRPO support training without paired text–target-program annotations.Source: §1 · p. 2
• The same inversion model supports audio- and text-driven generation, with text first converted to reference audio and outputs remaining directly editable.Source: §1 · p. 2

• Audio-generation research improves compression and reconstruction or supports post-generation transformations, but fine-detail fidelity still trades off with compression and audio edits do not directly expose production controls (Zeghidour et al., 2021; Défossez et al., 2022; Kumar et al., 2023).Source: §2 · p. 3
• Graphics work generates editable underlying assets, a structural approach that parallels controlling sound through synthesizer programs (Poole et al., 2022; Xiang et al., 2025; Meshy).Source: §2 · p. 3
• Text-driven audio work generates editable notes or searches synthesizer parameters, but jointly generating performance, timbre, and variable-length modulation remains open (Lu et al., 2023).Source: §2 · p. 3
• Synthesizer inversion has progressed from black-box search to neural prediction and rendering-based reinforcement learning, yet jointly optimizing rendered sound across MIDI and variable-length routes remains challenging (Horner et al., 1993; Heise et al., 2009; Le Vaillant et al., 2021; Wu et al., 2026).Source: §2 · p. 3
• Helm is selected for controls resembling commonly used software synthesizers, an open-source implementation, and 274 factory presets that facilitate data construction and native rendering (Tytel).Source: §2 · p. 3

3.1 A UNIFIED AUDIO- AND TEXT-DRIVEN FRAMEWORK

• The same encoder, decoder, and renderer support inversion from reference audio and text-driven generation via audio generated by Stable Audio 3.Text is not directly fed to the program decoder; the native synthesizer renders programs but does not participate in backpropagation.Source: §3.1 · p. 4

3.2 STRUCTURED AUTOREGRESSIVE SYNTHESIZER MODELING

• MIDI events, fixed synthesizer parameters, and modulation routes are serialized and generated in sequence, conditioning each later component on the earlier components.A shared decoder predicts the complete program, with legal values constrained by the current grammatical position.Source: §3.2 · pp. 4–5
• The audio frontend extracts conditioning features from separate left- and right-channel log-mel spectrograms, and the decoder uses causal self-attention and audio cross-attention.Both the frontend encoder and program decoder have 12 Transformer layers.Source: §3.2 · pp. 4–5

3.3 SUPERVISED LEARNING FROM SYNTHESIZED PAIRS

• Supervised learning uses rendered audio–program pairs from constrained preset, parameter, route, and MIDI augmentation, then trains the decoder with teacher forcing and field-balanced cross-entropy.The loss averages cross-entropy within each semantic field, then across fields present in the batch; ordered numerical fields use Gaussian-smoothed labels, while other fields use hard labels.Source: §3.3 · p. 5

3.4 MIXED-REWARD OPTIMIZATION FROM RENDERING FEEDBACK

• Because parameter supervision does not directly measure sound match, post-training renders candidate programs and scores them with eight weighted reward components.Rewards include CLAP distance, Basic Pitch hidden-feature distance, AudioBox enjoyment and usefulness, and acoustic similarity measures; Stable Audio 3 data use text in the reward, not as decoder input.Source: §3.4 · p. 5
• GRPO standardizes rewards within each candidate group and updates valid-position log probabilities; epsilon-greedy top-k sampling restricts exploration to plausible legal tokens.Each fresh candidate group is used once, without PPO clipping or reference-policy KL regularization.Source: §3.4 · p. 6

4 EXPERIMENTS

• The evaluation pairs sound-design-oriented synthetic tests with broader real-world FSD50K recordings, using objective metrics and listening evaluations for short sounds rather than complete songs.The Gemini–Woosh test uses 128 five-second Woosh clips and corresponding descriptions; FSD50K contributes 256 five-second recordings.Source: §4 · p. 6, §4.1 · p. 6

4.1 DATA AND EXPERIMENTAL SETUP

• Supervised training expands 274 Helm factory presets and an initialized state into two million audio–program pairs; post-training uses 500,000 generated clips mixed with in-domain audio at 5:3.Descriptions cover notes, chords, short arpeggios, and textures, without paired target synthesizer programs.Source: §4.1 · p. 6

4.2 SYNTHESIZER INVERSION

• On Gemini–Woosh, AutoSynth has a substantial audio CLAP-distance advantage and slightly outperforms DDSynth-RL on wMFCC, SOT, and MSS, but has lower RMS-envelope similarity.On FSD50K, its CLAP gap narrows while its lead on acoustic metrics other than RMS grows; it also leads on LanguageBind in both test sets.Source: §4.2 · p. 7

4.3 TEXT-DRIVEN GENERATION

• On Gemini, AutoSynth beats the other inversion cascades on text CLAP and CE, CU, and PQ; on FSD50K, it leads program systems in CU and CE but not LanguageBind.On both text-generation test sets, AutoSynth also scores higher in PQ than Stable Audio 3 and Woosh, end-to-end waveform references.Source: §4.3 · pp. 7–8
• A demonstration shows GPT-6 editing generated synthesizer programs from user instructions, followed by synthesizer re-rendering.The video is available on the project page.Source: §4.3 · p. 8

4.4 SUBJECTIVE EVALUATION

• Listeners prefer AutoSynth for inversion on both sets; for text generation, it leads both Gemini dimensions and has highest FSD50K content quality, tying CTAG on alignment.In inversion, listeners prefer AutoSynth despite DDSynth-RL’s higher RMS similarity and similar Gemini–Woosh acoustic distances; both methods receive lower FSD50K alignment ratings.Source: §4.4 · p. 8, Figure 3 · p. 9, Figure 4 · p. 9

5 DISCUSSION

• Cross-system comparisons cannot isolate generation methods from sound engines because synthesizers differ in expressive capabilities and control structures.A unified benchmark for complete synthesizer-program generation is still needed.Source: §5 · p. 9
• Training across synthesizers is constrained by varying program formats and scarce legally usable data, making augmentation rules and post-training reward design important.Reward components and weights translate human sound-design preferences into objectives for out-of-domain matching with black-box synthesizers.Source: §5 · p. 9
• Acoustic fidelity and semantic distance cannot fully capture creative intent or aesthetic preferences, which vary by listener, musical style, and intended use.The discussion calls for broader prior knowledge of sound, music, and creative contexts.Source: §5 · p. 9

6 CONCLUSION

• AutoSynth jointly models MIDI, synthesizer parameters, and variable-length modulation routes, using supervised learning and mixed-reward GRPO to produce executable, editable sound controls without paired text–target-program annotations.The paper presents this as a practical path from audio generation to editable music creation.Source: §6 · p. 9

A CHOICE OF AUDIO ENCODER

• The jointly trained encoder outperforms both frozen SAME branches on validation measures, whose post-training curves also show occasional validation-reward collapses.The comparison uses in-domain SFT validation loss and Stable Audio 3 validation reward.Source: A CHOICE OF AUDIO ENCODER · p. 14, Figure A1 · p. 14
• One possible explanation is that joint training adapts audio representations to predict MIDI, timbre parameters, and modulation structure, unlike frozen general-purpose compression features.The authors note that the waveform-mediated cascade can extract task-specific inversion features despite adding decoding and encoding steps.Source: A CHOICE OF AUDIO ENCODER · p. 14

B AN EXPERIMENT IN CREATIVE AUDIO SUPER-RESOLUTION

• AutoSynth inverts corrupted Woosh audio and produces new renders, with uncorrupted clips, corrupted clips, and resyntheses of uncorrupted inputs serving as controls.The 128 clips receive Gaussian noise at 15 dB SNR, 4 kHz low-pass filtering, or hard clipping 12 dB below the original peak.Source: B AN EXPERIMENT IN CREATIVE AUDIO… · p. 15
• PQ rises by 2.33, 0.23, and 0.53 for noise, low-pass filtering, and clipping, respectively, while CLAP distance worsens in all three cases.PQ estimates production quality; CLAP distance to the uncorrupted reference assesses content preservation.Source: B AN EXPERIMENT IN CREATIVE AUDIO… · p. 15, Table B1 · p. 15
• The authors interpret the results as creative resynthesis: projection into the synthesizer’s sound space may raise estimated production quality while changing original timbral and performance details.The results are not characterized as preserving the corrupted sound unchanged.Source: B AN EXPERIMENT IN CREATIVE AUDIO… · p. 15

C SFT AND GRPO ABLATIONS

• The comparison uses the same test examples and metric implementations for SFT and GRPO, with text-driven generation reusing the same intermediate Stable Audio 3 audio.It covers inversion of Gemini–Woosh, FSD50K, and in-domain Helm audio, plus generation from Gemini and FSD50K descriptions.Source: C SFT AND GRPO ABLATIONS · p. 15
• Mixed-reward GRPO improves all six metrics on out-of-domain and in-domain inversion test sets, including LanguageBind, which is not part of the reward.The evaluation reports no in-domain reconstruction trade-off alongside improved out-of-domain inversion.Source: C SFT AND GRPO ABLATIONS · pp. 15–16
• On Gemini text generation, GRPO reduces text CLAP and LanguageBind distances while improving CE, CU, and PQ; on FSD50K, AudioBox scores and FAD improve while text CLAP distance, LanguageBind distance, and KL worsen.The authors describe the broader test-set pattern as primarily improving output quality and usefulness scores.Source: C SFT AND GRPO ABLATIONS · p. 16

D PROGRAM REPRESENTATION AND AUTOREGRESSIVE ORDERING

• The sequence places MIDI events before fixed parameters and modulation-route triplets, with end markers allowing variable numbers of events and routes.The task predicts 133 fixed parameters; continuous parameters and modulation amounts use 128 quantization levels.Source: D PROGRAM REPRESENTATION AND… · p. 16
• The ordering specifies performance first, then timbre, then modulation; structural choices and switches precede dependent values within fixed parameters.For example, oscillator waveform precedes tuning and volume, and effect switches precede their control parameters.Source: D PROGRAM REPRESENTATION AND… · p. 16
• The current order generally achieves lower in-domain validation loss than the tested alternatives, indicating that autoregressive ordering affects training performance.Figure D1 shows validation loss over the first 100,000 steps; alternatives include a shared random parameter permutation and fixed parameters before MIDI.Source: D PROGRAM REPRESENTATION AND… · pp. 16–17, Figure D1 · p. 17

E.1 MODEL AND TRAINING HYPERPARAMETERS

• Sampling applies top-k filtering and probability thresholding before categorical selection, with a highest-probability fallback when no token passes the threshold.Scoring reconstructs q at the same prefix and reads the sampled token’s probability without resampling.Source: §E.1 · pp. 18–19

E.2 MIXED-REWARD WEIGHTS

• Basic Pitch branches each receive weight 1/3 before their mean is multiplied by 12; reward coefficients are positive, with metric direction determining their sign.Table E2 lists the coefficients used in Equation (3).Source: §E.2 · p. 19, Table E2 · p. 19

E.3 EVALUATION METRICS •

• Evaluation specifies encoder distances, acoustic and pitch-related metrics, AudioBox scores, and pooled-feature FAD rather than per-clip FAD.The described implementations include dynamic-time-warping MFCC distance, spectral measures, RMS-envelope similarity, and 128-dimensional VGGish features for FAD.Source: §E.3 · p. 19
• KL is computed as D_KL(preference∥p_generated) from softmaxed logits of a frozen 527-dimensional PaSST classifier, then averaged across pairs.The implementation uses natural logarithms.Source: §E.3 · p. 20

Stable Audio 3

• Stable Audio 3 provides identical intermediate audio for AutoSynth, Synth Permutations, and DDSynth-RL, and serves as a waveform-generation reference rather than a program-generation baseline.The setup uses Medium weights, eight generation steps, and CFG 1.Source: §E.4 · p. 20

Woosh

• Woosh generates waveform references and text-set audio with Woosh-Flow-Public, while its outputs remain waveforms rather than synthesizer programs.It also constructs reference audio for the Gemini–Woosh inversion test set.Source: §E.4 · p. 20

Synth Permutations

• Synth Permutations is retrained for Surge XT inversion and uses 100 RK4 steps with CFG 2 for reference or Stable Audio 3-generated input audio.The method uses approximately permutation-equivariant flow matching to represent multiple valid parameter configurations for a sound.Source: §E.4 · p. 20

DDSynth-RL

• DDSynth-RL inverts reference or Stable Audio 3 intermediate audio into Dexed controls, using a fine-tuned checkpoint with joint CREPE and CLAP reward.It uses greedy value selection and normalized-entropy-based position selection.Source: §E.4 · p. 20

CTAG

• CTAG searches directly from each text description without intermediate Stable Audio 3 audio, using SynthAX Voice, LES, a population of 50, and 300 iterations.It returns the best candidate from one independent LAION-CLAP-guided run per description.Source: §E.4 · p. 20

Study design

• Ten participants familiar with synthesizers answered 20 randomly sampled questions each, comparing three systems for inversion and six for text-driven generation.Method names were hidden and candidate order independently randomized for each question.Source: F LISTENING EVALUATION PROTOCOL · p. 20

Rating dimensions

• Participants rated alignment and content quality separately on integer scales from 1 to 5, with higher scores indicating better performance.Alignment questions assess fidelity to reference audio or text descriptions across timbre, pitch, articulation, and temporal changes.Source: F LISTENING EVALUATION PROTOCOL · pp. 20–21
• Content-quality ratings consider audio quality, unintended artifacts, and creative usefulness, while treating intentional distortion as not a defect.Quality is rated independently of alignment.Source: F LISTENING EVALUATION PROTOCOL · p. 21

Rating collection

• 1,800 valid scores were collected, with 50 ratings per method, test set, and dimension; audio was not loudness-normalized.Participants could adjust playback volume to avoid bias from loudness differences.Source: F LISTENING EVALUATION PROTOCOL · p. 21