Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators

Zachary Novack; Stephen Brade; Haven Kim; Hugo Flores García; Nithya Shikarpur; Chinmay Talegaonkar; Suwan Kim; Valerie K. Chen; Julian McAuley; Taylor Berg-Kirkpatrick; Cheng-Zhi Anna Huang

Source-linked AI summary

Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators

Zachary Novack, Stephen Brade, Haven Kim, Hugo Flores García, Nithya Shikarpur, Chinmay Talegaonkar, Suwan Kim, Valerie K. Chen, Julian McAuley, Taylor Berg-Kirkpatrick, Cheng-Zhi Anna Huang

arXiv:2605.22717v1May 21, 2026cs.SDcs.AIcs.LGcs.MM

TL;DR

Interactive diffusion music generation remains inefficient for streaming use, limiting practical deployment on consumer hardware. The paper introduces LMDMs with KV caching and ARC-Forcing, achieving competitive quality with reduced latency while enabling generative musical instruments.

Problem
Diffusion-based interactive music generation remains largely unexplored, while bidirectional block-wise outpainting limits efficiency and leading autoregressive models demand impractical consumer-hardware resources.
Method
LMDMs repurpose open-source audio diffusion models using routing, attention masking, KV caching, and ARC-Forcing rollout post-training through standard finetuning.
Results
LMDMs achieve competitive quality with drastically reduced latency, running on consumer hardware at a fraction of discrete-AR models’ parameter count and training cost.
Takeaways & Limitations
LMDMs can function as responsive generative musical instruments and partners, including a real-time generative delay for live musician-AI collaboration.
Takeaways & Limitations
LMDMs favor training-data genres and respond more strongly to past clean content than injected text conditions.

Abstract

from arXiv · show

Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist in the discrete-AR regime, requiring industrial levels of compute for both training and inference. In this work, we investigate whether audio diffusion models, with their wide support in the open-source community but non-streaming bidirectional nature, can be repurposed efficiently into interactive models accessible on consumer hardware. By taking a critical look at the modern pipeline for block-wise outpainting diffusion, we identify critical inefficiencies during inference that result in strictly worse computational efficiency than their discrete-AR counterparts. We propose Live Music Diffusion Models (LMDMs), a simple modification of the generative diffusion process that recovers, and then outperforms, the inference complexity of the discrete Live Music Models (LMMs) through block-wise KV Caching. Unlike LMMs, LMDMs further enable stable post-training alignment through our novel ARC-Forcing paradigm, reducing error accumulation without any explicit RL or reward models. We demonstrate the application of LMDMs in a number of creative domains, including text-conditioned generation, sketch-based music synthesis, and jamming. We finally show how LMDMs can be used as a generative instrument in a real artist-AI collaboration, utilizing LMDMs as a "generative delay" to transform musicians' improvisation live for variable timbral effects while running locally on a consumer gaming laptop.

1 Introduction

The paper repurposes smaller open-source audio diffusion models into interactive Live Music Diffusion Models for consumer hardware, addressing the size bottleneck of discrete autoregressive systems. LMDMs enable efficient KV-cached streaming, ARC-Forcing post-training without RL or reward models, broad control, and live musician collaboration.

Motivation: Discrete autoregressive live music models can require over 40 GB of VRAM, making local inference on consumer hardware impractical.This size bottleneck motivates seeking smaller interactive generation models.
Motivation: Open-source diffusion music models offer better data efficiency and substantially smaller model sizes than strong discrete-autoregressive methods.The paper repurposes these models into interactive streaming systems on consumer hardware.
LMDM architecture: LMDMs use separate routing for clean history and noisy present blocks plus dedicated attention masks to enable noise-wise KV caching over diffusion steps and time.This design recovers the exact inference complexity of encoder-decoder LMMs, while a block-causal variant is strictly faster.
Post-training: ARC-Forcing combines ARC with Self-Forcing to provide global adversarial supervision on multi-block rollouts, reduce error accumulation, and accelerate sampling without RL or pretrained reward models.The recipe is enabled by diffusion’s differentiability and is presented as an RL-free adversarial post-training method.
Applications: LMDMs bring text, sketch, and accompaniment controls into near-real-time streaming and support deployment as a generative instrument in live collaborations on consumer hardware.The paper reports use with real musicians in collaborative sessions and live performances.

2 Related Work

Related work has largely emphasized fixed-length, offline music generation, while newer systems pursue real-time interaction and more efficient inference. However, diffusion-based music generation remains largely unexplored for autoregressive interaction and rollout-based post-training.

Offline music generation: Most deep generative music systems prioritize one-shot generation, mapping text or other controls to fixed-length compositions.Examples include text-to-music models and offline systems controlled by dynamics, melody, stems, or gestural sketches.
Interactive music systems: RAVE and VampNet exemplify more interactive music systems through real-time timbre transfer and generative loop creation.RAVE runs on consumer hardware, while VampNet provides a loop-pedal-like generative paradigm for musicians.
Diffusion and autoregression: Diffusion-based music generation grew from static image diffusion, while video research expanded toward autoregressive generation for efficient inference and interactive world models.Early diffusion video approaches used bidirectional attention with time-dependent noise schedules.
Research gap: Diffusion-based music generation remains largely unexplored for the autoregressive direction pursued in video generation.The passage identifies this as an unresolved gap in adapting such approaches to music.
Continuous-AR music models: Recent continuous-AR music approaches generally omit rollout-based post-training, often require multi-billion-parameter models for strong performance, and differ architecturally from standard diffusion systems.These limitations distinguish continuous-AR approaches from the diffusion-based models studied in this work.

3 Background

The paper focuses on flow matching with an Optimal Transport path, which interpolates data with Gaussian noise and learns the reverse process through a noise-conditioned velocity field. It contrasts this full-sequence, bidirectional generation with block-based autoregressive and diffusion outpainting approaches for interactive music.

Flow Matching: Flow matching interpolates each data sample with Gaussian noise using x(k) = (1 − k) · x + k · ε, where ε ∼ N(0, I).The noise level k ranges from the clean sample at k = 0 toward Gaussian noise at k = 1.
Flow Matching: Its objective is to learn a noise-conditioned velocity network that reverses the corruption process from Gaussian noise to the data distribution by solving an ODE.The forward velocity is defined as dx(k)/dk = ε − x := v, and the learned field vθ can be integrated backward with standard solvers such as Euler.
Flow Matching: Because flow matching iterates over noise levels rather than time, most flow models use full bidirectional temporal attention to generate the entire sequence together.Additional conditions such as text prompts can be incorporated, with sampling supported by classifier-free guidance.
Block-Based Autoregressive Generation: Block-based autoregressive generation predicts the next o frames from s frames of past latent-audio context, then slides the context forward by one block.The conditional distribution is pθ(x_s:s+o | x_1:s, c), and the newly generated block becomes context for continued generation.
Block-Wise Diffusion Outpainting: Diffusion-based block-wise outpainting conditions flow models on clean audio context through channel concatenation, resets those context frames at each noise level, and continues after sliding the context window.The procedure generates one target block at a time while using freshly generated audio as subsequent context.

4 Live Music Diffusion Models

LMDMs convert offline diffusion models into interactive generators by separating clean context from noisy targets, enabling block-wise KV caching and LMM-level or better inference complexity. ARC-Forcing then supports RL-free post-training on long autoregressive rollouts to reduce error accumulation while improving inference speed.

Efficient inference: LMDMs use routing and attention masks to prevent clean context representations from mixing with noisy targets or attending to future frames.This makes the context encoding independent of diffusion noise and enables caching clean-frame representations.
Efficient inference: Standard block-wise diffusion repeatedly processes all T frames at every diffusion step, yielding worse efficiency than LMM-style separate context encoding and iterative decoding.Its latency includes dependence on the number of diffusion steps K and lacks a single-pass encoding of context frames.
Efficient inference: Encoder-Decoder LMDMs cache Key/Value states from clean context and reuse them during target denoising, achieving LMM-equivalent complexity O(ELMDM_s:T · K).The cached representation acts as an implicit encoder, requiring one clean-context encoding pass and iterative decoding for the next block.
Efficient inference: Block-wise LMDMs achieve strictly better complexity than LMMs by removing the need to encode the whole context for each new block.The improvement follows from reusing encoded context rather than recomputing it for every generated block.
Efficient inference: 110-170ms round trip latency is enabled by bidirectional encoding regions, optimized flash-attention kernels, and torch.compile.The latency is reported for inference after removing custom masked-attention implementations from the encoding passes.
ARC-Forcing post-training: ARC-Forcing addresses error accumulation by post-training a few-step generator on B-block rollouts with global supervision, without explicit RL or a reward model.Diffusion sampling is differentiable, while long rollout pairs provide learning signal beyond single-block supervision.

5 The Live Music Design Space

LMDMs unify text-conditioned generation, sketch-based control, and accompanying-stem jamming within one flexible design framework. The framework supports global prompts, local musical sketches, and latency-aware accompaniment from another audio stream.

Unified Design Space: LMDMs unify text control, sketch-based control, and audio jamming under one flexible design framework rather than bespoke isolated architectures.The section applies the framework to tasks previously studied separately.
Global Text-Conditioning: Global text prompts act as non-temporal, instrument-like conditions that can transition between prompts in real time.The prompt has no temporal access, but prompt transitions provide real-time modulation.
Instrument-like Sketch Controls: Instrument-like sketch controls provide local future-generation information through top-k CQT and loudness conditions.These conditions follow approaches similar to Tsai et al. (2025), García et al. (2025), and Novack et al. (2025b).
Accompanying Stem Generation: Accompanying-stem generation infers a musically compatible continuation from causal history and prior outputs without explicit target-block features.The model receives the other stem only up to a cutoff frame, with tf < 0, to compensate for system latency.

6 Experiments and Results

The experiments evaluate LMDMs across text-conditioned generation, accompaniment, sketch-conditioned synthesis, and real-time musician interaction. Results show competitive text-conditioned performance with faster inference, ARC-Forcing mitigates long-form error accumulation, and sketch-conditioned models support subsecond generative-delay interaction while exposing live-use limitations.

Global evaluation: LMDMs achieve competitive quality and text adherence with drastically faster inference despite using half the parameters and nearly 100x less training data than LMMs.Evaluation uses FD/KL for quality, CLAP for text adherence, and D-NFE/TTFF for latency, with and without ground-truth audio priming.
Per-window evaluation: ARC-Forcing significantly mitigates the gradual degradation and error accumulation of global metrics over two-minute text-conditioned rollouts.Without ARC-Forcing, nearly every metric degrades over time in both audio-primed and unprimed settings, across Enc-Dec and Block-Causal LMDMs.
Accompaniment evaluation: Reducing future stem visibility predictably reduces alignment, but ARC-Forcing prevents accompaniment models from collapsing toward random performance when tf < 0.The tf < 0 setting removes direct correspondence between conditioning stem audio and the target block, limiting available signal.
Real-time deployment: Subsecond generative-delay operation is achieved by distilling a sketch-based LMDM to 8 steps and deploying ONNX-exported DiT and VAE models.The block-processing effect induces delay ∆ = S + τθ, with τθ < S required for gapless playback; the reported system achieves ∆ < 1 second.
Musical interaction: Musicians experienced LMDMs as responsive musical partners that followed their playing while introducing new ideas and enabled timbral exploration beyond their instruments.Participants described dialogic interaction, constrained unpredictability, and transformations ranging from synth-like and bass tones to foley-like sounds.
Challenges: Live deployment revealed prompt-following regression toward generic EDM for the Jamendo model and weak CQT control for foley models.The authors suspect the Jamendo issue is related to the ONNX pipeline, while weak fundamental frequencies in FSD50k may make top-k CQT controls uninformative.

7 Limitations and Discussion

LMDMs still face genre-bias limitations from imbalanced training data, while interactive streaming models occupy a distinct space from scaling-focused offline Text2Song systems. Their latency requirements and consumer-hardware accessibility constrain capacity, making parity with large offline generators potentially incompatible with musicians’ needs.

Limitations: LMDMs trained on MTG-Jamendo exhibit a strong EDM quality bias because the dataset over-represents electronic dance music.They perform considerably better on EDM-like prompts than on less-represented genres such as country or jazz.
Discussion: Interactive streaming music models offer a future orthogonal to scaling offline Text2Song systems.The paper frames interactive models as pursuing a different direction rather than simply matching offline-system scale.
Discussion: Hard latency requirements and consumer-hardware accessibility impose a fundamental capacity cap relative to large offline song generators.The paper argues that striving for parity may be a lost cause and actively contradictory to interactive use by musicians.

8 Conclusion

The paper introduces Live Music Diffusion Models by adapting open-source audio diffusion models into interactive streaming generators with routing, attention masking, and KV caching. It also presents ARC-Forcing for stable, reinforcement-learning-free post-training supervision that mitigates error accumulation while achieving competitive quality with drastically reduced latency.

Model adaptation: LMDMs repurpose open-source audio diffusion models into interactive streaming generators through routing and attention-masking modifications.These modifications enable KV caching across both diffusion steps and time.
Model adaptation: KV caching operates over both diffusion steps and time, supporting LMDM inference efficiency.
Post-training: ARC-Forcing provides stable, RL-free global supervision on multi-block rollouts, significantly mitigating error accumulation.
Results: LMDMs achieve competitive quality with drastically reduced latency.

Contributions and Acknowledgments

The project credits contributors across algorithmic methodology, model development, data and evaluation, live API work, artist collaboration, ideation, and advising. It also acknowledges collaborators who supported debugging, paper feedback, and system demonstrations.

Contributions: Zachary Novack led the project, algorithmic methodology, and text- and stem-conditioned model development.
Contributions: Stephen Brade co-led the project and contributed to sketch-conditioned models, on-device model wrangling, live API development, and artist collaboration.
Contributions: Haven Kim handled data collection and pre-processing, evaluation design and development, and project ideation.
Contributions: Hugo Flores García, Nithya Shikarpur, Chinmay Talegaonkar, Suwan Kim, and Valerie K. Chen contributed to live API work, ideation, algorithmic methodology, and artist collaboration.Hugo Flores García led live API creation and development; Nithya Shikarpur supported live API development; Chinmay Talegaonkar contributed algorithmic methodology; Suwan Kim and Valerie K. Chen supported artist collaboration.
Acknowledgments: Julian McAuley, Taylor Berg-Kirkpatrick, and Cheng-Zhi Anna Huang provided project support and advising, while others assisted with debugging, feedback, and sketch-system demonstrations.The acknowledgments thank Petros Karypis for debugging a RoPE implementation, Shih-Lun Wu for paper feedback, and Sebastian Franjou and Matthew Michalek for helping demo the sketch-conditioned system.

A Experimental and Evaluation Protocol · A.1 Evaluation Metrics

The evaluation protocol reports three toolkit-computed metrics: FD-OpenL3 and KL-PaSST jointly assess quality and distributional fit, while CLAP measures audio–text similarity.

A.1 Evaluation Metrics: The protocol reports FD-OpenL3, KL-PaSST, and CLAP score as its three evaluation metrics.All three metrics are computed using the toolkit released with Saito et al. (2025).
A.1 Evaluation Metrics: FD-OpenL3 and KL-PaSST jointly measure quality and distributional fit.FD-OpenL3 is computed over OpenL3 embeddings, while KL-PaSST is computed over PaSST logits.
A.1 Evaluation Metrics: CLAP score measures audio–text similarity.The metric is reported following prior works and computed with the released evaluation toolkit.

A.2 Training and Inference Setup … A.2.3 Sketch-Conditioned Generation

The experiments fine-tune Stable Audio Open Small for text-conditioned, accompaniment, and sketch-conditioned generation, using task-specific datasets, conditioning setups, and evaluation protocols. The sketch models span multiple datasets and configurations for offline evaluation, user studies, and performance.

A.2 Training and Inference Setup: All models are fine-tuned from Stable Audio Open Small, a 340M-parameter DiT originally trained on approximately 12 seconds of Freesound latent audio.The base model is Stable Audio Open Small (Novack et al., 2025a).
A.2.1 Text-Conditioned Generation: LMDMs use MTG-Jamendo data excluding Song Describer, with 240 latent-frame sequences and 48-frame target generation blocks.Initial fine-tuning uses context routing and attention masking for 10k iterations; the supplied passage truncates the remaining batch-size detail.
A.2.1 Text-Conditioned Generation: Text-conditioned evaluation compares audio-primed and text-only generation on 47-second clips, with primed inputs containing captions and ground-truth audio prefixes.Audio-primed generation provides the caption and the first s frames of the corresponding ground-truth track; text-only generation provides only the caption.
A.2.1 Text-Conditioned Generation: Text-conditioned drift is measured over sliding windows using FD-OpenL3, KL-PaSST, and CLAP score, with 8 sampling steps for both ARC-Forced models.FD-OpenL3 uses 1-second windows and hops, while KL-PaSST and CLAP use 10-second windows with 1-second hops.
A.2.2 Accompaniment Generation: Accompaniment models are trained and post-trained on Slakh MIDI synthesized stems, with randomly sampled same-piece stems serving as context and target.ARC-Forcing runs for 8k steps because of observed faster convergence, and five models vary future visibility from 2.2 to -2.2 in 24-latent intervals.
A.2.3 Sketch-Conditioned Generation: The sketch-conditioned section presents models trained specifically for the sketch task, including models for offline evaluation, user studies, and performance.The supplied passage points to Table A.2.3 for the model overview.
A.2.3 Sketch-Conditioned Generation: Sketch-based encoder-decoder models span configurations across FSD50k, approximately 48 minutes of humpback whale song, and MTG-Jamendo.These datasets are listed in Table 3.

B Derivation of Ping-Pong++ (P4) Solver · C Interface Design · D Prompt Transition Pairs

The paper derives a P4 solver by recasting ping-pong sampling into CFG++-style conditional denoising and unconditional renoising, improving stability in few-step inference. It also describes a JUCE interface and a 128-pair prompt-transition evaluation drawn from 256 SDD captions.

B Derivation of Ping-Pong++ (P4) Solver: The standard ping-pong sampler is formulated for consistency-style models trained to output x(0) from any x(k).The formulation follows v-prediction, rather than arbitrary x(s) prediction, to ease convergence in adversarial post-training.
B Derivation of Ping-Pong++ (P4) Solver: P4 reframes ping-pong sampling as CFG++-style conditional denoising followed by unconditional renoising.The renoising process uses the unconditional velocity, while denoising maximizes adherence to the text prompt.
C Interface Design: The system interface is implemented in JUCE and provides the application’s main interaction components.Figure 7 shows the interface used in the user studies and performances.
D Prompt Transition Pairs: The prompt-transition evaluation uses 128 prompt pairs (A, B), with both endpoints sampled from 256 distinct Song Describer Dataset captions.Table 4 lists the source and target prompts used for cross-prompt continuity evaluation.
D Prompt Transition Pairs: The transition-pair prompts span diverse musical styles, instrumentation, vocals, tempos, moods, and production characteristics.Examples include electronic, rock, pop, folk, jazz, orchestral, Latin, blues, and experimental descriptions.