Source-linked AI summary
DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Zhongjie Duan, Shengchuan Gao, Hong Zhang, Yingda Chen
TL;DR
Text and lyrics leave musical timing, melody, and reference-based style only partly specified. DiffSynth-Music adds composable audio conditioning through layer-wise key-value injection, and evaluations report stronger control adherence and vocal-conditioned lyric fidelity while broadly retaining quality and instruction-following scores.
Problem
Text and lyrics specify broad musical characteristics and sung content but do not directly determine beat timing, vocal melody, or performance phrasing; audio can convey these details and reference style.
Source: §1 · p. 1Method
DiffSynth-Music adds composable audio conditioning to a pretrained music-generation DiT through three template models that inject layer-wise key-value representations for five control types.
Source: §1 · pp. 1–2Results
Mandarin and English song evaluations show improved adherence across all five control types and vocal-conditioned lyric fidelity, with automatic music-quality and instruction-following scores broadly comparable to evaluated base models despite some decreases relative to the backbone.
Source: §1 · p. 2, §5 · p. 8Takeaways & Limitations
The released template models support research and creative applications in controllable music generation.
Source: §5 · p. 8Pitch- and envelope-based prosody resynthesis reduces, but does not guarantee removal of, linguistic content or vocal identity.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
1 Introduction
Text and lyrics convey broad musical character and sung content but do not directly specify beat timing, vocal melody, or performance phrasing. DiffSynth-Music adds composable audio controls to a pretrained music-generation DiT through layer-wise attention keys and values.
1 Introduction
- DiffSynth-Music adds audio-based guidance for musical details that text prompts and lyrics do not directly determine, including beat timing, vocal melody, and performance phrasing.Audio references can convey temporal information and acoustic characteristics that are difficult to specify verbally.Source: §1 · p. 1
- The framework injects control-audio representations as layer-wise attention keys and values while preserving the backbone’s text-conditioning and waveform-decoding pathways.The adapter is initialized from a pretrained music-generation DiT.Source: §1 · p. 1
- Five controls cover beats, vocals, accompaniment, prosody, and reference audio, with joint conditioning supported through a shared waveform-to-latent representation.Accompaniment and prosody can constrain instrumental structure and vocal melody without requiring preservation of the original vocal timbre.Source: §1 · pp. 1–2
- Experiments on Mandarin and English songs report improved adherence across all five control types while maintaining broadly comparable automatic music-quality and instruction-following scores.The work trains and releases Control, Prosody, and Reference template models that can be combined at inference.Source: §1 · p. 2
2 Related Work
Related work spans latent diffusion and flow matching, music-generation architectures, audio-controllable generation, pretrained-model adaptation, and music-analysis tools. DiffSynth-Music extends a frozen pretrained DiT with composable audio-conditioned templates rather than introducing a new generative objective or audio conditioning itself.
2.1 Diffusion and Flow Matching
- DiffSynth-Music uses a linear noise-to-data path with conditional flow matching, augmenting the velocity model with audio-derived attention memory rather than proposing a new generative objective.The approach extends an existing generator, ACE-Step-1.5-XL-SFT, while keeping its backbone frozen.Source: §2.1 · p. 2, §2.2 · p. 2
2.2 Music Generation Architectures
- Music-generation systems vary in representation and generation order; DiffSynth-Music instead adds composable audio-conditioned templates to the frozen ACE-Step-1.5-XL-SFT backbone.The cited systems include autoregressive token models, latent diffusion, semiautoregressive flow matching, and interleaved sketching and diffusion refinement.Source: §2.2 · p. 2
2.3 Controllable Music Generation
- Prior controllable-generation methods use melody, symbolic, audio, and reference conditions; DiffSynth-Music combines five control types through a shared waveform-to-latent interface and layer-wise attention memories.Its prosody condition uses pitch- and envelope-based vocal resynthesis to reduce pronunciation cues.Source: §2.3 · p. 2
2.4 Adapting Pretrained Generators
- Unlike fixed learned prefixes or decoupled image cross-attention, DiffSynth-Music uses DiT-initialized audio templates to concatenate layer-wise keys and values within joint attention.This distinguishes its conditioning interface from residual feature injection as well.Source: §2.4 · p. 3
2.5 Music Analysis
- Beat tracking, source separation, and pitch estimation supply audio-derived control signals, while Qwen3-Omni provides descriptive prompts and lyric annotations for evaluation recordings.The analysis outputs are estimated annotations and signals, so extraction errors can affect the conditioning data.Source: §2.5 · p. 3
3 Methodology
DiffSynth-Music formulates generation as matching text with selected audio controls, then injects audio-derived key-value memories into a pretrained DiT. Its method shares a latent interface, reuses fixed control caches, fuses them through attention, and constructs controls from recordings.
3.1 Problem Formulation
- The task is to generate a waveform consistent with text and selected musical attributes, encoding target and control waveforms through a shared VAE into a common latent space.Source: §3.1 · p. 3
3.2 DiT-Based Template Modules
- Three DiT-initialized templates—Control, Prosody, and Reference—process five control modes, with controls assigned to each module sharing that module’s projections.Source: §3.2 · pp. 3–4
- Because control and auxiliary template inputs remain fixed at the clean-data endpoint, each active control’s layer-wise KV cache is computed once and reused throughout sampling.Source: §3.2 · p. 4
3.3 KV Injection
- At selected blocks, control and generation keys and values are concatenated for joint attention; only generation queries produce outputs, while Reference preserves excerpt order rather than output-position alignment.Source: §3.3 · pp. 4–5
3.4 Audio Conditioning
- From approximately 60k recordings, beats become synthetic timing pulses, Demucs stems supply vocals and accompaniment, and same-recording excerpts provide global reference context.Source: §3.4 · p. 5
- Prosody resynthesizes pitch-tracked vocals with an envelope to retain pitch and timing while reducing phonetic and timbral detail, but linguistic content and singer identity may remain.The pitch trajectory is interpolated to the sample grid; no voicing mask is applied, so the envelope controls amplitude during intervals with imputed pitch.Source: §3.4 · p. 5
3.5 Training Formulation and Inference
The templates are trained by conditional flow matching on a linear noise-to-data path, with the generation backbone components frozen during template-only optimization. At inference, fixed template inputs yield caches computed once and reused through sampling.
3.5 Training Formulation and Inference
- Conditional flow matching trains a neural vector field to predict velocity along a prescribed conditional probability path, using linear interpolation between noise and clean audio latents.The text encoder, VAE, and generation DiT remain frozen during template-only optimization; gradients update the selected template parameters through the injected key-value tensors.Source: §3.5 · p. 6
- Template-only optimization freezes the text encoder, VAE, and generation DiT while updating selected template parameters through the injected key-value tensors.Source: §3.5 · p. 6
- At inference, fixed template inputs and timestep 1 allow control caches to be computed once before integration and reused throughout sampling.Only the generation DiT is evaluated iteratively.Source: §3.5 · p. 6
4 Experiments
Experiments evaluate audio-control adherence and general generation quality on Mandarin and English songs, using separate single-control configurations. DiffSynth-Music improves all reported control metrics while retaining broadly comparable automatic quality and instruction-following scores, with metric-specific trade-offs.
4.1 Experimental Setup
- Control adherence was evaluated on 100 Mandarin and English songs with sample-specific prompts, lyrics, and five extracted control signals, against the backbone and four other systems.The test set included 50 English samples from MUSDB18-HQ and 50 Mandarin samples from a private dataset.Source: §4.1 · p. 6
4.2 Evaluation Metrics
- Control metrics compare generated beats, vocals, accompaniment, prosody, and reference audio with their targets using timing, signal-error, pitch, and embedding measures.Source: §4.2 · pp. 6–7
4.3 Experimental Results
- Across the five single-control settings, DiffSynth-Music achieved the best reported value on all seven control metrics.Beat-F1 rose from 0.3031 to 0.8496 and Cemgil from 0.1983 to 0.8088; V-MSE fell from 0.0158 to 0.0006, PER from 0.1928 to 0.1451, and A-MSE from 0.0221 to 0.0023. Pitch50 increased from 0.0366 to 0.4671, above MiniMax-Music3's 0.0402; MuLan-A rose from 0.6836 to 0.8088.Source: Table 1 · p. 7, §4.3 · p. 8
- Automatic music-quality and instruction-following scores were broadly comparable to evaluated base models, although some aesthetics scores and all MuLan-T scores were below the backbone.Source: Table 2 · p. 7, §4.3 · p. 8
4.2 Evaluation Metrics
- General generation quality was assessed separately using AudioBox-Aesthetics proxies and MuLan-T prompt similarity rather than treating control adherence alone as evidence of quality.Source: §4.2 · pp. 7–8
5 Conclusion
DiffSynth-Music extends a frozen ACE-Step-1.5-XL-SFT backbone with three audio-conditioned templates and composable controls. Experiments show improved control adherence and vocal-conditioned lyric fidelity, while automatic quality and instruction-following scores remain broadly comparable to evaluated base models despite some decreases from the backbone.
5 Conclusion
- DiffSynth-Music extends the frozen ACE-Step-1.5-XL-SFT backbone with Control, Prosody, and Reference templates trained using conditional flow matching and layer-wise KV injection.The five control types are composable, and their memories are computed once and reused during sampling.Source: §5 · p. 8
- Mandarin and English song experiments demonstrate improved control adherence and vocal-conditioned lyric fidelity, with automatic quality and instruction-following scores broadly comparable to evaluated base models despite some backbone-relative decreases.Source: §5 · p. 8
- The template models are released for research and creative applications; future work will evaluate joint conditioning and musical quality through human listening studies.Source: §5 · p. 8