PaperlayerView exact revision on arXiv

Source-linked AI summary

Melody Guidance for Audio Diffusion Models Without Retraining

Cristobal Andrade, Sebastian J. Schlecht, Ben Maman, Meinard Müller

arXiv:2610.32211v1eess.AS

TL;DR

Melody control is usually learned by training generative models with melody conditioning. This work guides pretrained unconditional diffusion at inference using differentiable F0 estimation and melody-informed warm initialization, achieving melody adherence competitive with dedicated conditional baselines across piano, guitar, and singing voice.

  • Problem

    Existing melody-control approaches learn conditioning during training and require a model trained or fine-tuned with melody-specific conditioning.

    Source: §1 · p. 1

  • Method

    The method treats target F0 as an inverse-problem measurement and guides unconditional diffusion with a differentiable pitch estimator and melody-informed warm initialization.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across four models spanning piano, guitar, and singing voice, melody adherence is competitive with dedicated conditional baselines.

    Source: §1 · p. 1, §5 · p. 4

  • Takeaways & Limitations

    The method applies to waveform- and latent-domain diffusion models without melody-specific training or modifications to the underlying generative model.

    Source: §1 · p. 1, §5 · p. 4

    Naive latent posterior sampling requires back-propagating guidance through the decoder, which can be unstable.

    Source: §2.3 · p. 2

Abstract

from arXiv · show

Controlling melody in audio generation normally requires training a conditional generative model on audio paired with pitch curves or MIDI notes. We instead guide an existing unconditional diffusion model, originally trained without melody inputs, to follow a target melody without retraining or modifying it. During inference, a differentiable pitch detector steers the sampled audio toward the desired pitches, while the diffusion model keeps it close to the distribution of realistic audio learned during training. Our main contribution is a sampling method that balances these two directions, providing melody control while preserving audio quality. To improve stability, we initialize sampling with a simple synthesized version of the target melody rather than pure noise. Experiments on piano, guitar, and singing voice show melody accuracy comparable to conditional models trained specifically for melody control. Our method therefore adds melody control to existing audio diffusion models without requiring paired melody data or task-specific retraining.

1. INTRODUCTION

Melody control is commonly learned through melody-conditioned training, while this work guides an unconditional diffusion model at inference using a differentiable pitch estimator and warm initialization.

  • Existing melody-control approaches learn conditioning during training, using signals such as F0 curves, MIDI, or spectrograms.Melody and F0 conditioning have been used in singing synthesis, speech synthesis, sound generation, music synthesis, and timbre transfer.Source: §1 · p. 1
  • The method treats a target F0 trajectory as an inverse-problem measurement and seeks plausible audio whose estimated pitch matches it.A differentiable pitch estimator serves as the measurement operator.Source: §1 · p. 1
  • Diffusion posterior sampling combines the model prior for realistic audio with a likelihood that steers generation toward the target melody.F0 estimation is nonlinear and highly underdetermined, and the Gaussian likelihood approximation used by DPS is not valid in this setting; warm initialization can stabilize sampling.Source: §1 · p. 1
  • The method achieves melody adherence competitive with dedicated conditional baselines while retaining promising perceptual quality.It requires neither melody-specific training nor modifications to the underlying generative model.Source: §1 · p. 1

2. BACKGROUND

The background develops diffusion sampling and posterior guidance as a basis for melody control. It also covers latent-domain guidance and differentiable pitch estimation.

2.1. Diffusion Models

  • Reverse diffusion uses a learned score to transform samples from a simple noise distribution toward the data distribution, with a denoised estimate available at each step.The denoised estimate is used to evaluate guidance objectives during sampling.Source: §2.1 · pp. 1–2

2.2. Diffusion Posterior Sampling

  • Diffusion posterior sampling combines the pretrained prior score with a likelihood score to seek samples that are plausible under the data distribution and consistent with an observation.Because the exact likelihood is intractable, DPS approximates it using the denoised estimate; its formulation accommodates differentiable linear or nonlinear operators.Source: §2.2 · p. 2
  • For melody guidance, the observation is a target F0 trajectory, and the likelihood score steers generation toward audio with a matching estimated pitch contour.The F0-based representation is nonlinear and highly underdetermined.Source: §2.2 · p. 2

2.3. Latent Posterior Sampling

  • Latent posterior sampling evaluates guidance through a decoded denoised latent estimate, retaining a training-free approach but requiring decoder backpropagation that can be unstable.Latent diffusion uses an encoder and decoder around a compact representation.Source: §2.3 · p. 2

2.4. Differentiable F0 Estimation

  • dSWIPE makes pitch estimation differentiable by replacing SWIPE’s hard argmax with a softmax distribution over pitch bins and an expected pitch.Its output is used to define the melody likelihood score for waveform and latent diffusion models.Source: §2.4 · p. 2

3. MELODY GUIDED SAMPLING

The method treats a target F0 trajectory as a degraded measurement and guides diffusion sampling toward audio whose estimated pitch matches it. A melody-informed triangular-wave initialization complements that guidance, helping balance pitch adherence and audio realism.

3. MELODY GUIDED SAMPLING

  • A synthesized triangular wave from the target F0 warm-starts reverse diffusion, supplying harmonic structure that reduces the corrective guidance required.For latent models it is encoded; for waveform models it is used directly, and sampling starts from it at tinit.Source: §3 · pp. 2–3
  • At each denoising step, the discrepancy between estimated and target F0 provides a gradient that steers the next reverse step toward the prescribed melody.The differentiable pitch estimator dSWIPE serves as the measurement operator in the inverse-problem formulation.Source: §3 · p. 3
  • The method frames melody conditioning as an inverse problem: recover audio from a target F0 trajectory, which is nonlinear and highly underdetermined.Many waveforms with different timbres, articulations, and harmonic structures can share the same F0 trajectory, so matching pitch alone does not determine realistic audio.Source: §3 · p. 3
  • Gaussian blurring of the target pitch representation helps retain expressive characteristics such as vibrato, with standard deviation ς controlling the amount of blurring.ς is expressed in cents.Source: §3 · p. 3
  • Guidance alone can sound unnatural, while high λ can move outputs away from the reference distribution; warm initialization permits smaller λ while maintaining F0 adherence.The authors report that guidance and warm initialization are both needed for melodically accurate, natural-sounding results.Source: §3 · p. 3

4. EXPERIMENTS

The experiments evaluate melody guidance across piano, guitar, and singing voice, varying guidance strength and initialization time and comparing against available baselines. The reported results indicate competitive melody adherence, alongside quality trade-offs that vary by setting.

4. EXPERIMENTS

  • The evaluation spans four diffusion models: two waveform-domain piano models and two latent models for singing voice and guitar, using publicly available baselines with released implementations and checkpoints.All audio examples are on the companion website.Source: §4 · p. 3

4.1. Effect of Guidance Strength λ and Initialization Time tinit

  • With pure-noise initialization, stronger guidance improves OA but increases KAD; warm initialization alone raises OA, while the quality trade-off persists, so both parameters are needed jointly.The authors suggest starting from a late warm initialization, increasing tinit until melody hallucinations emerge, then raising λ to suppress them.Source: §4.1 · p. 3

4.2. Baseline Comparison

  • Singing comparisons use LOUDAR on OpenSinger against DiffSVC, DiffSinger, and WORLD, with 100 POP909 melodies as guidance targets.The reported metrics include OA and RPA from CREPE and SWIPE, KAD, and FAD.Source: §4.2 · p. 4
  • Guitar comparisons use LOUDAR on GOAT against RAVE, DDSP-Guitar, and MSD; piano comparisons apply guidance to CQT-Diff and MRCQT-Diff against MSD and PianoKontext.The guitar results include insufficiently articulated note onsets and decays; the piano results attribute weaker perceptual quality to mismatch between the polyphonic training prior and monophonic melody lines.Source: §4.2 · p. 4
  • Across instruments, melody adherence is competitive with dedicated conditional models; informal listening reports plausible singing and guitar, while piano quality is degraded relative to those cases.Table 1 reports the per-instrument comparisons; KAD and FAD are directly comparable across piano systems trained on MAESTRO, but should be interpreted cautiously for singing and guitar systems trained on different datasets.Source: Table 1 · p. 4, §4.2 · p. 4

5. CONCLUSION

The training-free method applies without modification to waveform- and latent-domain diffusion models and achieves melody adherence competitive with dedicated conditional baselines across four models spanning piano, guitar, and singing voice.

5. CONCLUSION

  • Across four models spanning piano, guitar, and singing voice, melody adherence is competitive with dedicated conditional baselines.The method applies without modification to both waveform- and latent-domain diffusion models.Source: §5 · p. 4
  • The method guides unconditional diffusion models toward a target melody at inference time using diffusion posterior sampling, a differentiable pitch estimator, and melody-informed warm initialization.It formulates melody conditioning as an inverse problem and requires no melody-specific training.Source: §5 · p. 4
  • Uncontrolled aspects of generated audio, including phoneme content, articulation, and note decay, emerge as artifacts of the unconditional prior.The authors identify controlling these aspects as future work.Source: §5 · p. 4
  • Future directions include learned latent degradation models for stability, warm initialization for symbolic constraints beyond pitch, and long-gap inpainting guided by a target pitch trajectory.A preliminary motivating result for long-gap inpainting is provided on the companion website.Source: §5 · p. 4
Loading 2610.32211v1…