Source-linked AI summary
SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving
Ayush Barik, Sofia Stoica, Nikhil Sarda, Arnav Kethana, Abhinav Khanduja, Muchen Xu, Fan Lai
TL;DR
Text-to-audio diffusion serving remains costly because high-fidelity generation requires tens of denoising evaluations and can take multiple seconds. SoundWeaver warm-starts from semantically similar cached audio using retrieval, adaptive skipping, and cache management, achieving 1.8–3.0× latency speedups while preserving or improving perceptual quality.
Problem
Text-to-audio diffusion serving requires tens of function evaluations, causing multi-second latency and substantial infrastructure cost at production scale.
Method
SoundWeaver uses semantic and duration-aware reference selection, adaptive NFE skipping, and quality-aware cache eviction and refinement for audio-guided warm-starting.
Results
1.8–3.0× end-to-end latency speedup is achieved on realistic workloads while preserving or improving perceptual quality with approximately 1K cached audio entries.
Takeaways & Limitations
SoundWeaver provides a training-free, model-agnostic serving approach that converts semantic similarity in cached audio into computation savings.
Takeaways & Limitations
Remaining limitations include phase vocoder distortion on long audio requests, missing dedicated request schedulers, and untested compatibility with complex samplers.
Abstract
from arXiv · showhide
Text-to-audio diffusion models produce high-fidelity audio but require tens of function evaluations (NFEs), incurring multi-second latency and limited throughput. We present SoundWeaver, the first training-free, model-agnostic serving system that accelerates text-to-audio diffusion by warm-starting from semantically similar cached audio. SoundWeaver introduces three components: a Reference Selector that retrieves and temporally aligns cached candidates via semantic and duration-aware gating; a Skip Gater that dynamically determines the percentage of NFEs to skip; and a lightweight Cache Manager that maintains cache utility through quality-aware eviction and refinement. On real-world audio traces, SoundWeaver achieves 1.8--3.0$ \times $ latency reduction with a cache of only ${\sim}$1K entries while preserving or improving perceptual quality.
1. Introduction
Text-to-audio diffusion offers high-fidelity generation but incurs multi-second latency because it iteratively denoises over tens of steps. SoundWeaver exploits semantic similarity in audio distributions to warm-start generation, combining cache-based selection, adaptive NFE skipping, and cache management for faster serving.
- Motivation: Text-to-audio diffusion models require tens of denoising steps, creating multi-second latency and substantial infrastructure cost at production scale.Prior work mainly reduces NFEs algorithmically or improves serving through parallelism and hardware–software co-design.
- Motivation: Most user-uploaded audios have close semantic neighbors, enabling cached audio to provide a prior for skipping NFEs that construct coarse structure.Early NFEs establish coarse structure, while later steps refine high-frequency perceptual detail.
- Approach: SoundWeaver warm-starts diffusion from a semantically aligned cached audio candidate rather than pure noise, skipping initial NFEs.The system selects a cached reference for a new request and begins generation from an intermediate state.
- Approach: Its Reference Selector performs semantic and duration-aware retrieval and alignment, while the Skip Gater adapts the percentage of NFEs skipped.The Cache Manager maintains reuse utility through quality-aware eviction and refinement while bounding memory and compute overhead.
- Results: 1.8–3.0× end-to-end latency speedup is achieved on realistic workloads while preserving or improving perceptual quality with approximately 1K cached audio entries.These results position SoundWeaver as a serving approach that converts audio similarity into computation savings.
2. Methods
SoundWeaver warm-starts text-to-audio diffusion from semantically aligned cached audio, combining duration-aware retrieval, adaptive NFE skipping, and cache maintenance to balance latency, quality, and reuse.
- Warm-starting: SoundWeaver initializes diffusion from a semantically aligned cached reference instead of pure noise, enabling initial NFEs to be skipped.Greater semantic alignment permits more aggressive skipping, while duration alignment and adaptive skip selection address reuse challenges.
- Reference Selector: The Reference Selector retrieves top-K semantically similar candidates, applies quality and duration gates, samples a candidate, and phase-vocoder-aligns it to the requested duration.FAISS and pyramid indexing support retrieval of semantically aligned portions without increasing full-audio storage.
- Reference Selector: The quality gate combines normalized positive similarity and negative dissimilarity, while admitting candidates whose durations fall within a compatible range around the request.A phase vocoder performs precise frequency-domain time-scaling while preserving pitch; the gate limits stretch ratios to reduce artifacts.
- Skip Gater: Prompt-variance weighting emphasizes offline learning examples where skip choices substantially affect quality and downweights skip-insensitive prompts.The method addresses changing request distributions and the lack of golden labels that limit supervised skip prediction.
- Skip Gater: The Skip Gater uses a contextual multi-arm bandit to select among predefined skip percentages from prompt, cache, and total-NFE context.Its reward trades normalized efficiency gain against perceptual quality, with ranking feedback available when absolute quality scores are noisy.
- Cache Manager: The Cache Manager evicts entries using reuse-derived importance with exponential decay and selectively refines frequently reused, low-quality entries during idle periods.Importance tracks skipped NFEs and audio duration, while decay prioritizes recently beneficial entries.
3. Evaluation
SoundWeaver is evaluated on AudioLDM backbones using real and synthetic caches, standard quality metrics, pairwise preference scores, online traces, ablations, and cache-size studies. It reduces latency substantially with small caches while generally preserving quality and adding little overhead.
- Experimental Setup: Experiments use AudioLDM and AudioLDM2 on an A100 GPU, with caches built from Clotho v2 and prompts sampled from AudioCaps.The primary backbone has 652M parameters, while AudioLDM2 has 1.1B; both use 200 NFEs.
- Metrics: The evaluation reports CLAP, KL Divergence, Fréchet Distance, Inception Score, and pairwise Preference Score against vanilla generation.The baseline uses no caching or NFE skipping, while Preference Score compares outputs under identical prompts.
- Main Results: 1.81× and 2.25× latency speedups are achieved for AudioLDM and AudioLDM2, respectively, with a cache of approximately 1K entries.For AudioLDM, both cache variants improve all reported quality metrics; AudioLDM2 shows slight degradation but remains competitive.
- Online Deployment: 2.7× and 3.0× end-to-end latency speedups are observed for AudioLDM and AudioLDM2 in one-hour online deployments.The traces are derived from DiffusionDB and scaled to the available hardware capacity.
- Ablations and Overhead: Reference Selector ablation causes quality degradation, while the Reference Selector and Skip Gater together add 0.04 seconds per request and cache management contributes approximately 1% overhead.The Cache Manager runs asynchronously during off-peak hours, and refined entries are reused hundreds of times.
- Cache Size: Approximately 2K cache entries allow SoundWeaver to match or exceed full-denoising quality while latency is held constant.Larger cache pools consistently improve generation quality by providing more semantically aligned starting points.
4. Conclusion
SoundWeaver is presented as a model-agnostic approach to serving text-to-audio diffusion models. It reports 1.8×–3.0× speedups with maintained or improved perceptual quality and identifies remaining deployment and sampler limitations.
- Contribution: SoundWeaver uses audio-guided warm-starting to accelerate text-to-audio diffusion while maintaining or improving perceptual quality.The reported speedup range is 1.8× to 3.0× with minimal overhead.
- Limitations: Remaining limitations include phase vocoder distortion on long audio requests, no dedicated request schedulers, and untested compatibility with complex samplers.The paper identifies these issues as directions for future work.
6. Generative AI Use Disclosure
The paper uses generative AI to refine writing and presentation and to evaluate generated audio outputs with Gemini 3 Flash. The evaluator compares paired clips using specified perceptual criteria and a −3 to +3 score.
- Uses: Generative AI is used for writing refinement and as an automated evaluator of generated outputs.The automated evaluator is Gemini 3 Flash, following LLM-as-a-judge practice.
- Evaluation Criteria: The audio evaluator compares two clips from the same prompt using consistency, intelligibility, naturalness, artifact absence, timbral richness, fidelity, and overall quality.The prompt instructs the evaluator not to let clip duration or presentation order influence the decision.
- Scoring: Outputs are scored from −3 to +3, ranging from Audio 2 much better to Audio 1 much better.The required response includes a short rationale and an integer score.