Source-linked AI summary
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Soujanya Poria
TL;DR
Text-to-audio systems previously relied on joint text-audio encoders or non-instruction-tuned text models. TANGO uses frozen instruction-tuned FLAN-T5 embeddings in a latent diffusion pipeline and pressure-level-based audio mixing, achieving stronger AudioCaps performance than AudioLDM on most metrics despite 63 times less training data. Its control over fine-grained textual concepts remains limited by the small AudioCaps training set.
Problem
Prior text-to-audio systems used joint text-audio encoders or non-instruction-tuned models, motivating evaluation of an instruction-tuned LLM text encoder.
Method
TANGO feeds frozen FLAN-T5 text embeddings into a latent diffusion model, with pressure-level-based audio mixing for training augmentation.
Results
TANGO outperforms AudioLDM on most objective and subjective AudioCaps metrics despite using a 63 times smaller LDM training dataset.
Takeaways & Limitations
Instruction-based tuning and pressure-level-aware mixing support sample-efficient text-to-audio generation without fine-tuning the text encoder.
Takeaways & Limitations
TANGO has limited fine-grained textual control because it is trained only on the small AudioCaps dataset.
Abstract
from arXiv · showhide
The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.
1 Introduction
TANGO applies an instruction-tuned FLAN-T5 encoder to latent diffusion for text-to-audio generation, replacing joint text-audio guidance and non-instruction-tuned encoders. It reports stronger AudioCaps performance despite using much less training data, while augmenting mixtures according to audio pressure levels.
- Results: TANGO outperforms AudioLDM and other baselines on most objective and subjective AudioCaps test metrics despite training on a 63 times smaller dataset.The paper reports this comparison across both evaluation types.
- Approach: TANGO uses an instruction-tuned FLAN-T5 language model for text guidance in latent diffusion text-to-audio generation.The text encoder remains frozen during latent diffusion model training.
- Approach: The method replaces joint text-audio guidance such as CLAP with frozen FLAN-T5 text guidance during training and inference.This differs from prior approaches that use joint encoders or fine-tune text encoders.
- Approach: The latent diffusion model learns text-to-audio concept mapping and composition from the smaller training set when paired with an instruction-tuned LLM.The authors contrast this with AudioLDM’s text-encoder fine-tuning and larger training setup.
- Data augmentation: TANGO mixes audio pairs using their pressure levels rather than random combinations to preserve useful representations of both source audios.The captions of the mixed audio pairs are concatenated for training augmentation.
2 Method
TANGO combines an instruction-tuned LLM text encoder with latent diffusion and audio decoding components to generate audio from textual descriptions. Its method also uses text-guided reverse diffusion, classifier-free guidance, and pressure-level-aware audio mixing for augmentation.
- TANGO architecture: TANGO encodes the input description, generates an audio latent from Gaussian noise through reverse diffusion, decodes it into a mel-spectrogram, and vocodes the result.Its three major components are a textual-prompt encoder, latent diffusion model, and mel-spectrogram/audio VAE.
- Text encoder: FLAN-T5-LARGE serves as the frozen text encoder, producing token-level text encodings for conditioning the latent diffusion model.The model has 780M parameters and is instruction- and chain-of-thought-pretrained.
- Latent diffusion: The latent diffusion model approximates the text-conditioned audio prior by applying forward Gaussian noising and text-guided reverse denoising.The forward process progressively noises the audio latent toward standard Gaussian noise, while the reverse process reconstructs it using noise estimation.
- Latent diffusion: A U-Net with cross-attention estimates noise from the noisy latent and text encoding, replacing AudioLDM's audio-guided training with text guidance.During inference, the estimated noise reconstructs the audio prior for subsequent decoding.
- Data augmentation: TANGO augments training data by superimposing audio pairs and concatenating captions, weighting samples by relative audio pressure levels rather than mixing them randomly.The weighting is intended to prevent high-pressure audio from overwhelming low-pressure audio and preserve both samples after mixing.
- Classifier-free guidance: Classifier-free guidance controls the contribution of text conditioning during reverse diffusion, and training with text dropped on 10% of samples performed equivalently to always using text.The inference guidance scale determines the contribution of text-guided versus unguided noise estimates.
3 Experiments
TANGO is evaluated on AudioCaps using objective and human judgments, with comparisons against established text-to-audio baselines and analyses of data, guidance, event structure, and label complexity. It achieves strong performance despite smaller-scale training, while evaluation scope and checkpoint availability constrain some comparisons.
- Experimental Setup: 45,438 AudioCaps training clips and 2,240 validation instances support TANGO’s main text-to-audio experiments.AudioCaps clips are ten seconds long and paired with human-written captions.
- Experimental Setup: TANGO freezes FLAN-T5-LARGE and trains an 866M-parameter latent diffusion U-Net for text-guided audio generation.The model uses cross-attention with 1024 dimensions and eight channels.
- Main Results: TANGO outperforms AudioLDM and other baselines despite training the latent diffusion model on a much smaller dataset.The comparison includes objective and subjective metrics, while AudioLDM-L-Full was unavailable for subjective evaluation.
- Main Results: 24.52 FD, 1.37 KL, and 1.59 FAD are TANGO’s reported objective scores when trained only on AudioCaps.The paper reports these as new state-of-the-art objective results and describes TANGO as more sample efficient than AudioLDM fine-tuned models.
- Main Results: 85.94 overall audio quality and 80.36 relevance are TANGO’s subjective evaluation scores.Six evaluators rated 30 randomly selected baseline- and TANGO-generated samples on scales from 1 to 100.
- Additional Analyses: TANGO achieves better FAD scores in all but one analyzed label-category group, with large improvements for several sound combinations.FD and KL performance is more balanced between TANGO and AudioLDM across the reported category groups.
4 Related Works
Related work develops diffusion-based and other text-to-audio systems, using text encoders and audio synthesis pipelines to generate audio from descriptions. AudioLDM adapts latent diffusion to audio and uses CLAP embeddings, while earlier augmentation methods mix or transform audio-text pairs with different limitations.
- Diffusion Models: Diffusion models transform white noise into structured waveforms through a fixed sequence of Markov-chain steps.FastDiff and DiffWave apply diffusion to high-quality speech and broader waveform-generation tasks.
- Text-to-Audio Generation: Earlier text-to-audio systems encode text, generate spectrogram tokens or mel spectrograms, and use a vocoder to produce audio.One approach uses a non-autoregressive diffusion decoder, VQ-VAE, and vocoder; another augments data by masking non-event text.
- Text-to-Audio Generation: MBTG lacks diversity because it does not mix different audio samples, while later work mixes signals at random signal-to-noise ratios and concatenates captions.The later strategy creates new text-audio pairs from mixed audio and corresponding descriptions.
- Text-to-Audio Generation: AudioLDM uses a VAE-based latent representation and diffusion model, with CLAP audio and text embeddings supporting audio generation.Audio embeddings guide the backward diffusion process during training, while text embeddings are used for text-to-audio inference.
5 Limitations
TANGO’s control over textual prompts is limited when trained only on the small AudioCaps dataset. Larger training datasets are presented as necessary for learning more varied text-audio mappings and concept composition.
- 5 Limitations: TANGO is not always able to finely control generations over textual prompts because it is trained only on the small AudioCaps dataset.Prompts involving different vegetables, surfaces, or generalized chopping produce very similar audio samples.
- 5 Limitations: Larger datasets are required for TANGO to learn textual concept composition and varied text-audio mappings.The authors identify larger-data training and improved compositional control as future directions.
6 Conclusion
The paper investigates FLAN-T5 as an instruction-tuned text encoder for latent-diffusion text-to-audio generation. TANGO outperforms AudioLDM under objective and subjective evaluations while using 63 times less training data, which the authors primarily attribute to FLAN-T5’s representational power.
- 6 Conclusion: TANGO uses FLAN-T5 text embeddings in a latent diffusion model to generate mel-spectrogram tokens, which a VAE and vocoder convert into audio.The pipeline proceeds from textual embeddings to mel spectrograms and then synthesized audio.
- 6 Conclusion: 63 times less training data still yielded superior objective and subjective performance than the state-of-the-art text-to-audio model AudioLDM.The comparison covers both evaluation types and identifies AudioLDM as the baseline.
- 6 Conclusion: The authors primarily attribute TANGO’s performance improvement to FLAN-T5’s representational power from instruction-based pre-training.They also propose investigating FLAN-T5 for audio super-resolution and inpainting.