Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation

• Existing TTA systems commonly use latent diffusion but still require extensive resources and large datasets while emphasizing coarse-grained performance over fine-grained alignment.Source: §1 · p. 1
• Auffusion adapts a pretrained T2I latent diffusion model to TTA to transfer generative strength and text-audio alignment.Source: §1 · pp. 1–2
• Auffusion achieves better quality and alignment than prior TTA systems and performs comparably to models trained on datasets over 60 times larger.Source: §1 · pp. 1–2
• The study systematically investigates text encoders because prior TTA findings about CLIP, CLAP, and instruction-tuned LLMs are contradictory and comprehensive ablations are lacking.Source: §1 · p. 2
• Cross-attention map visualizations provide an intuitive assessment of text-audio alignment, and the study reports better fine-grained alignment and versatile audio manipulations.Source: §1 · p. 2

2.1. Text-to-Image Synthesis

• T2I systems progressively use diffusion and latent representations to generate diverse, high-quality images from text prompts.Source: §2.1 · p. 2

2.2. Text-to-Audio Synthesis

• TTA methods include discrete-token diffusion, waveform autoregression, and latent diffusion models such as AudioLDM, AudioLDM2, Make-An-Audio, and Tango.Source: §2.2 · p. 2
• Auffusion’s architecture transforms between audio, spectrogram, pixel, and latent spaces, using a pretrained text-to-image LDM’s U-Net.Source: §2.2 · p. 2, Figure 1 · p. 3

3.1. Overview

• Auffusion converts audio to normalized mel-spectrograms, represents them as RGB image-like data, and uses a pixel VAE plus HiFi-GAN to reconstruct audio.Training maps audio through spectrogram and pixel spaces into latent representations; inference reverses this process from Gaussian noise.Source: §3.1 · p. 3, §3.2 · p. 3

3.3. Latent Diffusion Model

• The latent diffusion model learns to recover pixel-space audio representations by denoising latent variables with a text-guided U-Net trained using noise-space MSE.The forward process adds scheduled Gaussian noise from z0 to zT, while the learned reverse process reconstructs z0.Source: §3.3 · pp. 3–4

3.4. Conditioning Processes

• Cross-attention projects U-Net features and conditional text embeddings into attention representations, enabling visualized maps that assess text-audio alignment.The authors use these maps to examine how pretrained T2I cross-modal understanding transfers to TTA.Source: §3.4 · p. 4

3.5. Text Encoder

• Auffusion combines CLAP and FlanT5 embeddings as text conditions and independently drops each during training to support multi-source conditioning and unconditional training.Dropping both embeddings enables classifier-free guidance.Source: §3.5 · p. 4

3.6. Classifier-Free Guidance

• Classifier-free guidance controls text influence during reverse diffusion through a guidance scale w, with text conditions randomly discarded at a 10% training rate.The guided noise estimate combines conditional and unconditional denoising predictions.Source: §3.6 · p. 4

4.2. Results

• 21.99 FD, 1.63 FAD, 1.36 KL, 10.57 IS, and 55.3% CLAP: Auffusion-with-pretrained outperforms Tango on AudioCaps while maintaining comparable zero-shot Clotho performance despite far less training data.The pretrained model avoids the large Clotho-test drop observed for Tango and Auffusion-no-pretrained, indicating better generalization from limited data.Source: Table 1 · p. 5, §4.2 · p. 6
• Auffusion-Full-with-pretrained achieves state-of-the-art Clotho performance, preserves AudioCaps performance, and significantly exceeds baselines in IS and CLAP scores.The results indicate robustness across different data distributions; the authors also report that Riffusion performs much worse after its quantization-based transformation.Source: §4.2 · p. 6
• 69.36 OVL and 70.25 REL: human evaluation rates Auffusion-w-clip highly for overall quality and text relevance, with especially strong text-audio alignment.The evaluation uses five raters and reports results on AudioCaps and Clotho samples.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Analysis

• FlanT5-large surpasses FlanT5-base across all metrics, while combining CLAP and FlanT5-large yields the best overall objective performance.CLIP is strongest in IS, FlanT5-large in FAD, and CLAP benefits from audio-domain expertise but may capture less fine-grained information.Source: §4.3 · pp. 6, 8
• Pretrained LDMs produce more distinguishable cross-attention across tokens than randomly initialized LDMs, supporting transfer of text-image alignment to text-audio alignment.The maps highlight separate events such as “dogs” and “gunshot,” whereas CLAP-based conditioning produces similar token attention and less event separation.Source: §4.3 · pp. 6–7, Figure 2 · p. 7
• Cross-attention analysis finds Auffusion more accurately reflects captions than Tango, whose maps can omit “dog” sounds and introduce unintended “bird” noises.The comparison uses 50 diffusion steps and standardized square attention maps; the authors connect the result to pretrained LDM alignment capabilities.Source: §4.3 · pp. 6–7
• Human REL and ACC evaluations expose fine-grained multi-event alignment differences that global CLAP scores do not reliably capture.The analysis attributes weaker baseline alignment to globally pooled conditioning in AudioLDM and error accumulation through AudioLDM2’s autoregressive feature generation.Source: §4.3 · p. 8
• Auffusion supports audio style transfer, inpainting, word swapping, and text-attention re-weighting with more controllable and fine-grained manipulation than prior methods.These applications are enabled by the model’s text comprehension and text-audio alignment capabilities.Source: §4.3 · p. 8

• Auffusion harnesses pretrained text-to-image models for robust generation and precise cross-modal alignment in text-to-audio synthesis.Source: §5 · p. 8
• Objective and subjective evaluations report superior performance with limited data and computational resources, alongside accurate alignment and controllable audio manipulations.The demonstrated manipulations include style transfer, inpainting, word swapping, and attention re-weighting.Source: §5 · p. 8

• The collected datasets contain approximately 0.47M audio samples spanning about 7.7K hours, while training uses roughly 0.4M samples totaling about 2K hours.Source: A. Dataset Details · p. 12, Table 4 · p. 12

B.1. Vocoder

• 256-band mel-spectrograms are converted through a custom HiFi-GAN vocoder and VAE-compatible image and latent representations for 10-second audio.The vocoder uses 256 mel filter bands and produces spectrograms that are repeated by channel before VAE encoding.Source: §B.1 · p. 12

B.2. Configuration

• Auffusion fine-tunes an 860M-parameter Stable Diffusion U-Net for 100K steps on one A6000 GPU, requiring 48 hours.The text encoder is frozen, and the setup uses AdamW, mixed precision, and memory-efficient xformers; other baselines use substantially more GPU resources or training steps.Source: §B.2 · p. 12

B.3. Riffusion Re-implementation

• Riffusion is re-implemented after per-spectrogram min-max normalization and image quantization, with Griffin-Lim used to reconstruct audio despite information loss.The re-implementation changes the sample rate and hop size to match the audio-generation setup and uses the same whole datasets.Source: §B.3 · p. 13

C. Effect of Guidance Scale and Inference Steps

• Table 5 varies inference steps and classifier-free guidance scale to measure their effects on objective evaluation metrics.Auffusion uses 100 inference steps and guidance scale 7.5 as a balanced choice; guidance 5 has the best FD and FAD, while guidance 10 excels in IS and CLAP.Source: C. Effect of Guidance Scale and… · p. 13, Table 5 · p. 13

D. Evaluation

• Human evaluation measures overall quality, caption relevance, and the number of events to assess global and fine-grained text-audio alignment.Overall quality and relevance are rated from 0 to 100, while event counting targets fine-grained alignment.Source: D. Evaluation · p. 13
• Figure 3 shows the interface used for the subjective evaluation of generated audio.The figure is presented as a screenshot of the subjective evaluation procedure.Source: Figure 3 · p. 13

E.1. Text-to-Audio Generation

• Figure 4 demonstrates audio generation with the Auffusion-Full model.The supplied caption identifies the figure as a text-to-audio generation demonstration without reporting numerical results.Source: Figure 4 · p. 14

E.2. Audio Style Transfer Examples

• Style transfer gradually changes an original audio sample toward a text-described target as the reverse-process starting point increases.More added noise reduces similarity to the original audio while increasing attention to text guidance; the paper uses n = 0.7 for the last sample.Source: §E.2 · p. 15
• The demonstrated transfers include baby crying to cat meowing, cat screaming to car racing, and bird chirping to an ambulance siren.These transitions are shown in Figures 5–7.Source: §E.2 · p. 15, Figure 5 · p. 15, Figure 6 · p. 15
• The manipulation uses a shallow reverse process adapted from image-to-image techniques for the audio domain.Different starting points control how much information from the original audio is retained.Source: §E.2 · p. 15

E.3. Audio Inpainting Examples

• Auffusion fills a masked 2.5–7.5-second segment of a 10-second audio sample using the original text prompt and surrounding audio context.The authors observe that the model generates appropriate content for the masked segment.Source: §E.3 · p. 16, Figure 8 · p. 16

E.4. Word Swap Examples

• Replacing prompt tokens changes corresponding sounds, such as making a sound less echoic by swapping huge for small or replacing gunshots with speech.The examples are presented as attention-map replacement demonstrations.Source: §E.4 · p. 17, Figure 9 · p. 17

E.5. Attention Re-weighting Examples

• Increasing cross-attention for selected words selectively changes their effects, with chopping enhancing action-sound frequency and huge altering the sound effect.The manipulation changes targeted attributes without significant changes elsewhere in the example.Source: §E.5 · p. 17, Figure 10 · p. 17