PaperlayerView exact revision on arXiv

Source-linked AI summary

DEMON: Diffusion Engine for Musical Orchestrated Noise

Ryan Fosdick

arXiv:2605.28657v1cs.SD

TL;DR

Real-time music generation needs controls finer and faster than batch-mode or chunk-boundary interaction provides. DEMON adapts streaming diffusion into a playable control surface with per-slot scheduling and mutable per-step controls, reaching 11.3 generations per second at its production depth while organizing parameters by propagation latency.

  • Problem

    Existing music-generation tools often provide batch or chunk-boundary interaction, while musicians need broad controls with finer temporal responsiveness.

  • Method

    DEMON combines ACE-Step 1.5 with a StreamDiffusion ring buffer, per-slot heterogeneous schedules, shared mutable per-step state, per-frame source blending, and windowed VAE decoding.

  • Results

    11.3 generations per second at depth 4 is achieved with 471 ms per-request convergence, while shared-mutable curves provide 1-tick onset.

  • Takeaways & Limitations

    Streaming diffusion parameters can function as live musical controls, with responsiveness determined by each parameter’s position in the denoising loop.

  • Takeaways & Limitations

    DEMON inherits ACE-Step 1.5’s limitations in music quality and genre coverage, and its controls cannot place notes, enforce chords, or guarantee downbeats.

Abstract

from arXiv · show

We present DEMON, a real-time diffusion engine that makes the denoising process playable as a live musical instrument: a control surface both broad (many parameters shaped per-frame across the output) and responsive (each control taking effect as fast as its place in the denoising loop allows). Built on ACE-Step 1.5 and StreamDiffusion's ring-buffer architecture with TensorRT acceleration, it sustains up to 12.3 decoder completions per second for 60-second music on a single consumer GPU (RTX 5090), or 11.3 generations per second at our production ring-depth of 4. At these rates denoising parameters become viable as live performance controls, but the ring buffer propagates per-request changes only at its drain rate, a floor of S denoising steps. We contribute four mechanisms. (1) Per-slot heterogeneous denoise scheduling: each ring-buffer slot owns its timestep schedule, so a moving denoise slider is tracked without wiping the in-flight queue, where the upstream global-schedule design must rebuild and discard it. (2) Shared mutable per-step state, giving any parameter consulted at every solver step next-tick effect, bypassing ring-buffer drain. (3) Per-frame source blending: a sampling-time control on the standard SDE re-noise step, giving a framewise transformation-strength axis that complements scalar denoise scheduling. (4) Windowed VAE decode exploiting receptive-field analysis for an 8.0x decode speedup. Together these separate streaming-diffusion parameters into four propagation classes, by onset and convergence latency.

1. INTRODUCTION

DEMON adapts ring-buffer streaming diffusion to music generation, combining high throughput with controls designed for live performance. Its mechanisms address the responsiveness limits created when per-request parameters wait for ring-buffer slots to complete their denoising schedules.

  • Motivation: 12.3 generations per second for 60-second music on an RTX 5090 makes denoising parameters viable as live performance controls.The system combines StreamDiffusion’s ring-buffer architecture, ACE-Step 1.5, TensorRT acceleration, and windowed VAE decoding.
  • Motivation: ∼650ms at depth 8 with S=8 is the first-effect latency for per-request parameter changes, falling to ∼470ms at depth 4 with an ∼8% throughput cost.Per-request parameters are baked into slots at submission and propagate when a new-conditioning slot completes its denoising schedule.
  • Implementation: The system also addresses long-form audio bottlenecks involving ∼56ms VAE decoding and fp16 precision accumulation through 24 DiT layers.These engineering challenges arise when adapting streaming diffusion to long-form music.
  • Contributions: Per-slot heterogeneous denoise scheduling tracks a moving slider without wiping in-flight work or repaying warmup.Each slot owns a timestep schedule, and finished outputs remain coherent single-schedule trajectories.
  • Contributions: 100% completion rate under a continuous slider sweep contrasts with 1.7% for a StreamDiffusion-style global-reset baseline.The comparison tests the queue-preserving heterogeneous schedule against a design that rebuilds and discards in-flight state.
  • Contributions: Shared mutable per-step state gives parameters consulted at every denoising step next-tick effect, bypassing ring-buffer drain.The mechanism applies to per-frame curves and can also support in-flight structural schedule migration.

2. RELATED WORK

Related systems span autoregressive, masked-parallel, diffusion, and waveform-level approaches to real-time or controllable audio generation. DEMON is most directly comparable to Lyria RealTime, while adapting StreamDiffusion’s ring-buffer insight to long-form audio.

  • Autoregressive and parallel generation: MusicGen generates 30 seconds in 41.3 seconds on an A100, while MAGNeT reports a 7x speedup over MusicGen.MAGNeT uses non-autoregressive masked parallel decoding but produces fixed-length clips without real-time streaming or fine-grained temporal control.
  • Real-time music generation: Lyria RealTime streams 2-second audio chunks with controls including tempo, brightness, density, musical key, and per-stem toggles.Its chunked autoregressive design applies updates only at chunk boundaries, with a minimum 2-second latency and no per-frame modulation within a chunk.
  • Streaming diffusion: StreamDiffusion uses a ring buffer of generations at staggered denoising stages to achieve one output per forward pass after warmup.DEMON adapts this architecture to audio, where 60–240-second temporal extent and VAE decoding introduce additional challenges.
  • Audio control: Audio control systems commonly rely on text prompts, timing metadata, melody conditioning, and audio-reference conditioning.These approaches include global timbre or style references and per-request source-structure conditioning.

3. ARCHITECTURE

DEMON combines a ring-buffer streaming pipeline with per-slot scheduling, shared mutable state, and per-frame denoising controls to make diffusion parameters playable during long-form music generation. Its architecture separates control breadth from responsiveness, while decode optimizations reduce the cost of producing streamed audio.

  • Streaming Pipeline: The StreamPipeline uses staggered in-flight generations, with each tick running a batched diffusion pass before filtering and decoding the finished latent.Slots carry per-slot curve and source state, allowing in-flight generations to use distinct modulation settings.
  • Propagation mechanisms: Per-slot heterogeneous schedules track a moving denoise slider without discarding in-flight work, while preserving coherent single-schedule trajectories.A changed value still reaches output at the natural S-tick drain floor.
  • Propagation mechanisms: Shared mutable per-step state gives parameters consulted at every denoising step next-tick onset and progressive convergence below the per-request drain floor.In-flight schedule migration applies the same shared-state pattern to structural timestep schedules, but produces trajectory-incoherent hybrids.
  • Diffusion Engine: The engine exposes per-frame curves at 25 Hz, including SDE source blending that varies transformation strength across frames rather than globally truncating the timestep schedule.Scalar denoise and the SDE curve operate on different axes: temporal schedule tracking versus framewise transformation strength.
  • Diffusion Engine: Composable multi-condition generation blends condition velocity fields with per-frame weights and supports prompt crossfades, negative conditioning, and multi-LoRA composition.Residual CFG variants reduce the per-step unconditional forward pass.
  • Decode optimizations: A deterministic latent-MSE gate skips VAE decoding for nearly identical consecutive latents, while windowed decoding processes only the playback window instead of the full long-form latent.The similarity gate reuses previous audio without perceptible quality change during stable regions; windowed decoding addresses the full-decode bottleneck.

4. EXPERIMENTS

The experiments evaluate latency, quality, propagation behavior, and cross-GPU consistency for DEMON's streaming-diffusion design. They include a direct heterogeneous-scheduling ablation and measure propagation classes as deterministic onset and convergence patterns.

  • Evaluation scope: The experiments cover component and end-to-end latency, output quality, parameter propagation, and cross-GPU validation.Quality tests include windowed VAE decoding, per-frame controls, and stream-versus-batch parity.
  • Evaluation scope: The heterogeneous-versus-global-reset ablation directly evaluates DEMON's primary scheduling contribution.Propagation measurements characterize per-request, per-step shared-mutable, and model-weight classes.
  • Cross-GPU validation: 0.02 RMS is the maximum reported cross-GPU difference across three GPU generations, supporting the propagation patterns as streaming-architecture properties.Primary benchmarks use an RTX 5090, with validation on RTX 4090 and RTX 3090.

4.2 System Comparison

Table 3 compares DEMON with real-time and near-real-time music systems, highlighting architectural differences in control timing and temporal resolution. DEMON places controls inside per-step denoising and across the temporal axis, separating per-frame resolution from control onset latency.

  • DEMON applies controls inside the per-step denoising loop and along the 1500-frame temporal axis, unlike systems with no mid-generation control.
  • Autoregressive streamers commit tokens sequentially, so control changes do not revise emitted audio or modulate the current chunk.
  • 40 ms (25 Hz) resolution lets DEMON shape controls per frame across the output.
  • ∼470 ms at depth 4 and 649 ms at depth 8 characterize onset for per-request controls such as scalar denoise or prompt changes.Per-step parameters such as SDE and guidance curves instead take effect in one tick, ∼43 ms at depth 4.

4.3 Component Latency

Component profiling shows that the larger 240s engine matches the 60s engine within measurement noise, while batching improves efficiency sub-linearly. Refit support enables runtime LoRA hot-swapping with limited overhead.

  • <3% latency difference separates 240s-profile and 60s engines at shared operating points.The larger profile is described as more capable without practical latency cost.
  • ∼8% inference overhead occurs at B=8 with a LoRA applied, while refit-enabled engines show no base-weight latency penalty.LoRA refit itself takes ∼1.2s as a one-time cost per style change.
  • ∼6.1x is the B=8 latency cost relative to B=1 at the 60s operating point, below the 8x linear ceiling.Intermediate batch sizes follow the same sub-linear curve.

4.4 End-to-End Streaming Latency

Windowed VAE decoding keeps its cost independent of generation duration, leaving decoder forward time as the component that scales with song length. Consequently, throughput falls for longer generations while audio production remains hundreds of times faster than real time.

  • 88ms per tick combines 81ms decoder and 7ms windowed VAE costs at 60s generation with a 3s VAE window.
  • The 60s engine is rejected beyond its 1500-frame profile, whereas the 240s engine scales linearly to 240s.The figure reports refit engines only; the table averages refit and non-refit variants of an earlier build.
  • ∼388ms per tick at 240s consists of 381ms decoder time plus windowed VAE decoding.The measurement uses the 240s engine on an RTX 5090.
  • 12.3 gens/sec at 60s and ∼2.6 at 240s reflect longer generation duration, while audio production remains ∼740s and ∼620s per wall-clock second.The tick grows from 88ms to ∼388ms entirely through the decoder term.

4.5 Windowed VAE Quality

Windowed VAE decoding with overlap preserves the rendered audio interior at 16-bit precision, while omitting overlap introduces boundary artifacts. The reported spectral discontinuity for windowed methods is attributed to hard concatenation in the evaluation script.

  • Zero per-sample difference and infinite SNR occur with 0.5s overlap margins versus full decode at 16-bit PCM precision.This is identity at rendered-audio precision, not a float-level claim.
  • Without overlap, boundary artifacts appear, including 29.3 dB SNR and 67% elevated spectral discontinuity.

4.6 Per-Frame Control: SDE Source Blending and x0-Target Morph

DEMON’s SDE source-blending and x0-target morph controls create genuine per-frame transformations, with measurable source-preservation or target-directed gradients across audio segments. These controls trade prompt adherence or linearity against framewise influence and temporal coupling.

  • SDE Source Blending: SDE curves produce genuine per-frame source-preservation gradients, unlike uniformly applied ODE denoise.Ramp 0→1 yields a monotonic similarity decline from 0.977 to 0.934, while ODE variation is at most 0.027.
  • SDE Source Blending: Ramp gradients remain negative across all tested sources, but their magnitude varies with the cross-genre transformation.The gradient ranges from −0.043 for techno to −0.399 for lo-fi, with consistent direction across sources.
  • Control Limitation: The ramp 1→0 saturates after segment 2 because global temporal attention couples neighboring frames.The control remains per-frame, but attention smooths the model’s response across time.
  • SDE Source Blending: SDE source blending lowers CLAP alignment in proportion to source anchoring, reflecting the intended source-preservation tradeoff.Shaped curves remain within the flat-curve CLAP range, while source anchoring produces a negative CLAP delta.
  • x0-Target Morph: An α ramp toward an independent target latent B produces monotonic segment-wise movement toward B across 3 source-target pairs and 3 seeds.The mean gradient is +0.217 ± 0.051, with every sample monotone toward B.

4.7 Quality Evaluation

DEMON evaluates whether its streaming additions preserve output quality while enabling responsive control. The pipeline matches batch audio at render precision, preserves intended source-versus-prompt tradeoffs, and maintains throughput during parameter changes, though baseline comparisons have a defined scope.

  • Stream-versus-Batch Parity: Streaming output is sample-for-sample identical to batch generation at 16-bit PCM render precision across six evaluation sources.The paths have infinite SNR with identical CLAP and RMS; no audible quality cost is reported.
  • Quality Tradeoff: SDE curves produce negative CLAP deltas because source preservation reduces prompt adherence, as intended.The evaluation measures matched-baseline CLAP deltas at denoise=1.0 for the cover task.
  • Parameter-Change Evaluation: Shared mutable curves affect the next completion and converge progressively, bypassing the S-tick drain required by per-request changes.A curve change takes effect at tick 1 and plateaus by tick 5; x0-target morphing shows a similar ramp.
  • Parameter-Change Evaluation: 100% completion under a continuous denoise sweep contrasts with 1.7% for the queue-wiping global-reset baseline.Per-slot scheduling completes 60/60 ticks at 81.1 ms mean latency, while global reset completes 1/60 useful outputs.
  • Baseline Scope: The reported 100%-versus-1.7% sweep gap applies specifically to the queue-wiping baseline, not the stronger in-place global-edit design.The in-place design is expected to maintain near-100% completion, leaving trajectory coherence as the per-slot distinction.
  • Operating Point: 11.3 generations per second at depth 4 trades near-peak throughput for lower first-effect latency than depth 8.Depth 4 reaches 11.3 gens/sec with 471 ms measured first-effect, versus 12.3 gens/sec and 649 ms at depth 8.

4.9 Cross-GPU Generalization

Cross-GPU tests show that DEMON’s throughput, decode optimization, and parameter-propagation signatures generalize across RTX 5090, 4090, and 3090 hardware. The propagation behavior is architectural rather than specific to the 5090.

  • Throughput: The 4090 reaches 8.9 generations per second at depth 8, while the 3090 reaches 4.2 generations per second.The 4090 is above real-time for 60-second outputs; the 3090 is sub-real-time but functional.
  • Windowed VAE: Windowed VAE decoding achieves 2.80x, 3.10x, and 3.22x speedups on the 5090, 4090, and 3090, respectively.The consistent gains indicate a receptive-field optimization tied to the Oobleck VAE rather than one GPU.
  • Per-Request Propagation: Per-request changes show zero latent RMS through ticks 0–7 and approximately 0.93 RMS from tick 8 on both the 4090 and 3090.These values are within 0.02 of the 5090 result and reproduce the S-tick step pattern.
  • Shared-State Propagation: Shared SDE curves take effect at tick 1 and converge progressively across all three GPUs.Per-GPU measurements remain within 0.01 at every tick, contrasting with per-request zero-then-step behavior.
  • Shared-State Propagation: The shared-state write costs under 0.1 ms on each tested GPU, with runtime dominated by the first tick consuming the new value.Measured write times are 0.005 ms on the 5090, 0.062 ms on the 4090, and 0.059 ms on the 3090.

5. DISCUSSION

DEMON separates streaming-diffusion controls by onset and convergence latency, balancing live responsiveness against throughput and trajectory constraints. The discussion also identifies scope, perceptual-evaluation, and future-development boundaries.

  • Performance and operating point: 11.3 generations per second at depth 4 provides the production operating point, while depth 8 reaches 12.3 gens/sec but increases convergence latency to 649 ms.Depth 4 measures 471 ms convergence and depth 8 measures 649 ms; depth 1 measures 112 ms at 8.9 gens/sec.
  • Responsiveness mechanisms: Shared-mutable per-step curves take effect after one tick at every depth, whereas per-request controls remain governed by the pipeline’s drain floor.At depth 4, one-tick onset is 42.8 ms; at depth 8 it is 81 ms.
  • Scope and limitations: Streaming diffusion controls denoising dynamics rather than musical content, so they cannot place notes, enforce chords, or guarantee downbeats.These musical structures remain determined by the base model’s learned distribution.
  • Scope and limitations: Higher pipeline depth improves throughput but delays per-request changes, while globally attentive DiT processing smooths sharp per-frame control transitions.At 240 seconds, the depth-8 tick rises to approximately 381 ms, and truly independent per-frame control is unattainable in a globally attentive model.
  • Evaluation and future work: The paper reports objective quality and latency evidence but no listening test, MOS, or other human perceptual study.The authors identify perceptual evaluation with musicians as important future validation, especially for schedule migration’s transient muffling.
  • Future work: ACE-Step XL support is described as an extension, but no XL benchmarks are reported.Further work also includes a distilled VAE decoder and characterization of latent-channel semantics.

6. CONCLUSION

DEMON makes diffusion denoising playable as a realtime musical instrument by combining broad per-frame controls with responsiveness determined by each control’s position in the denoising loop. Its mechanisms classify controls by propagation latency while its streaming infrastructure reaches high decoder throughput.

  • Core conclusion: DEMON exposes per-frame curves at 25 Hz and makes denoising parameters live controls subject to their denoising-loop position.The control surface is broad in parameter coverage and responsive within each control’s propagation limits.
  • Propagation classes: The system separates parameters into four propagation classes: per-request, migrated schedule, per-step shared-mutable, and model weights.Their stated onset and convergence patterns are S-tick, 1-tick/S-tick, 1-tick/sub-S, and immediate, respectively.
  • Throughput: 12.3 decoder completions per second at depth 8 and approximately 11.4 end-to-end generations per second with windowed VAE establish the streaming operating range.Depth 4 is the operating point used in practice.
Loading 2605.28657v1…