Source-linked AI summary
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization
Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, Soujanya Poria
TL;DR
Existing text-to-audio models do not explicitly control whether prompted concepts and events appear in the correct temporal order. The paper constructs a synthetic preference dataset and fine-tunes Tango with diffusion-DPO, producing Tango 2, which improves over Tango and AudioLDM2 on automatic and human evaluations.
Problem
Recent diffusion-based text-to-audio models do not explicitly focus on including prompted concepts and events in their correct temporal order.
Method
The authors synthesize Audio-alpaca preference pairs and fine-tune Tango using DPO-diffusion loss, with undesirable outputs containing missing concepts, incorrect order, or high noise.
Results
Tango 2 significantly surpasses Tango and AudioLDM2 in objective and human evaluations, including scores of 2.69 FAD, 1.12 KL, 9.09 IS, and 0.57 CLAP.
Takeaways & Limitations
The results support diffusion-DPO and Audio-alpaca as approaches for improving alignment in text-to-audio generation.
Abstract
from arXiv · showhide
Generative multimodal content is increasingly prevalent in much of the content creation arena, as it has the potential to allow artists and media personnel to create pre-production mockups by quickly bringing their ideas to life. The generation of audio from text prompts is an important aspect of such processes in the music and film industry. Many of the recent diffusion-based text-to-audio models focus on training increasingly sophisticated diffusion models on a large set of datasets of prompt-audio pairs. These models do not explicitly focus on the presence of concepts or events and their temporal ordering in the output audio with respect to the input prompt. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. As such, in this work, using an existing text-to-audio model Tango, we synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from. The loser outputs, in theory, have some concepts from the prompt missing or in an incorrect order. We fine-tune the publicly available Tango text-to-audio model using diffusion-DPO (direct preference optimization) loss on our preference dataset and show that it leads to improved audio output over Tango and AudioLDM2, in terms of both automatic- and manual-evaluation metrics.
3. Aligned
The paper header identifies Tango 2 as a work on aligning diffusion-based text-to-audio generations through direct preference optimization, presented at ACM MM 2024 in Melbourne.
- The work was presented at ACM MM 2024 in Melbourne, Australia.
CCS CONCEPTS
The paper is categorized under natural language processing and multimedia information systems.
- The work falls under natural language processing and multimedia information systems.
KEYWORDS
The paper concerns multimodal AI, text-to-audio generation, diffusion models, large language models, and preference optimization.
- The paper combines multimodal AI, text-to-audio generation, diffusion models, large language models, and preference optimization.
1 INTRODUCTION
The paper targets faithful text-to-audio generation by aligning outputs with prompt semantics, including concepts and their temporal ordering. It introduces a synthetic preference dataset and diffusion-DPO fine-tuning, reporting gains over existing models in objective and human evaluations.
- Improving fidelity to input prompts is important for quickly creating audio-visual content and prototypes across multimedia industries.
- The method fine-tunes Tango with DPO-diffusion on preferred and undesirable audios designed to expose missing concepts, incorrect temporal order, or high noise.
- Tango 2 significantly surpasses Tango and AudioLDM2 in both objective and human evaluations after fine-tuning on pruned Audio-alpaca.
- The authors propose automatically creating a preference dataset for text-to-audio generation using heuristics and sharing Audio-alpaca.
- The paper presents diffusion-DPO as applicable to text-to-audio generation without sourcing additional out-of-distribution text-audio pairs over Tango.
2 RELATED WORK
Text-to-audio systems use diffusion-based architectures, while Tango replaces AudioLDM’s CLAP conditioning with a fine-tuned language model. Prior preference-alignment work uses human annotations and reward models, whereas this paper introduces automatically constructed pairwise preferences.
- Diffusion-based systems such as AudioLDM, Make-an-Audio, Tango, and Audiogen generate audio from textual prompts.
- AudioLDM combines CLAP text-audio representations with a latent diffusion model to generate mel-spectrogram representations.
- Tango uses AudioLDM’s pretrained VAE but replaces CLAP with the fine-tuned FLAN-T5 language model, enabling training with a significantly smaller dataset.
- BATON gathers audio-text pairs, obtains human preference annotations, trains a reward model, and incorporates its reward into diffusion training.
- This work differs from BATON by automatically constructing a pairwise preference dataset for text-to-audio generation.
3 BACKGROUND
Tango is a text-to-audio pipeline built from a text encoder, latent diffusion model, and audio VAE-vocoder. The background also introduces preference optimization, contrasting RLHF’s reward-model-and-RL pipeline with DPO’s preference-based alignment framework.
- Tango architecture: Tango encodes a text prompt, uses reverse diffusion to construct an audio latent, decodes a mel-spectrogram, and vocodes it into audio.
- Tango architecture: Tango uses FLAN-T5-Large as its textual-prompt encoder to produce text embeddings for conditioning.
- Latent diffusion model: Its latent diffusion model approximates the text-conditioned audio prior through forward noise addition and reverse text-guided denoising.
- Latent diffusion model: The forward process gradually adds scheduled Gaussian noise until the final latent follows a standard normal distribution, while the reverse process reconstructs the audio prior.
- Audio representation: A U-Net with cross-attention estimates noise using textual guidance, and the audio VAE compresses and reconstructs mel-spectrograms in latent space.
- Audio representation: Tango uses HiFi-GAN to convert the VAE decoder’s mel-spectrogram into audio and applies pressure-aware audio mixing with concatenated textual prompts for augmentation.
- Preference optimization: RLHF typically fine-tunes a supervised model, trains a reward model from ranked responses, and optimizes the policy while constraining divergence from a reference model.
- Preference optimization: The paper’s alignment pipeline creates the Audio-alpaca preference dataset and applies diffusion-DPO loss to align Tango.
4 METHODOLOGY
The methodology constructs Audio-alpaca by generating candidate audios from diverse and perturbed prompts, ranking them with CLAP-based criteria, and applying DPO to learn from preferred and undesirable outputs.
- Audio Generation: Audio-alpaca combines audio samples generated from diverse captions, semantically perturbed captions, and temporally perturbed captions.The three strategies produce either four or two generated audio samples per prompt.
- Audio Generation: The diverse seed set contains 14,000 captions selected by clustering sentence embeddings into 200 clusters and sampling 70 captions per cluster.
- Audio Generation: Perturbed-prompt strategies use GPT-4 to create semantically close or temporally altered captions, while perturbed captions are used only to generate comparison audios.
- Ranking and Preference-Data Selection: For each candidate set, the audio with the highest prompt-audio CLAP matching score is selected as the winner and the remaining samples as losers.
- Ranking and Preference-Data Selection: Selection requires strong winner alignment, semantically close losers, and a sufficiently large winner-loser CLAP-score difference, using two CLAP models for robustness.
- DPO for Preference Modeling: DPO trains on preferred and undesirable outputs rather than optimizing only desirable outputs, avoiding the need for a high-quality reward model required by RLHF.
5 EXPERIMENTS
Experiments compare Tango 2 with diffusion-based text-to-audio baselines using holistic, temporal, and subjective metrics. Tango 2 improves semantic alignment, several objective and temporal measures, subjective quality and relevance, especially as prompts contain more concepts or events.
- Baselines: Tango 2 is compared primarily with AudioLDM, AudioLDM2, and Tango, alongside discussion of preference-based Baton.AudioLDM uses CLAP-guided latent diffusion, AudioLDM2 uses multimodal LOA conditioning, and Tango uses FLAN-T5 conditioning.
- Evaluation Metrics: The evaluation covers holistic FAD, KL, IS, and CLAP metrics; STEAM temporal metrics; and subjective overall-quality and relevance ratings.STEAM measures ordering, duration, frequency, and timestamp alignment, while subjective evaluation separates overall quality from prompt relevance.
- Objective Results: 2.69 FAD, 1.12 KL, 9.09 IS, and 0.57 CLAP are reported for Tango 2, with KL and IS improvements over various AudioLDM and AudioLDM2 versions.CLAP measures semantic alignment between the input prompt and generated audio, while FAD, KL, and IS assess broader generation properties.
- Temporal Results: Tango 2 shows superiority over Tango across temporal measurements except Duration, and removing Strategy 3 reduces temporal performance.Strategy 3 supplies temporal data augmentation, supporting the importance of temporal perturbations in the preference data.
- Subjective Results: 3.99 OVL and 4.07 REL are Tango 2’s subjective ratings, surpassing Tango and AudioLDM2, whose corresponding ratings are lower.AudioLDM2 receives 3.56 OVL and 3.19 REL in the reported comparison.
- Complexity and Ablation: Tango 2’s advantage grows with prompt complexity: Tango is close only for single-concept prompts, while the gap widens for multiple concepts.The analysis examines CLAP performance as the number of events or concepts in the textual prompt increases.
- Complexity and Ablation: Excluding samples from Strategies 2 and 3 notably diminishes Tango 2’s performance, underscoring the importance of event and temporal prompt perturbations.The DPO objective penalizes missing concepts and incorrect event order while promoting preferred outputs with correct concepts and ordering.
- Complexity and Ablation: Filtering performs best with Δ2 ≥0.08 and α2 ≥0.6, where Δ2 compares winner and loser CLAP scores and α2 thresholds the winning audio.These filtering parameters define which preference pairs enter the Audio-alpaca dataset.
6 CONCLUSION
The paper proposes aligning text-to-audio diffusion models with automatically generated pairwise preferences and DPO-diffusion training. Tango 2 uses Audio-alpaca to distinguish preferred audio from outputs with missing concepts, incorrect temporal order, or noise, and reports gains over prior models.
- Contributions: The paper proposes direct preference optimization for text-to-audio generation using an automatically generated preference dataset called Audio-alpaca.Audio-alpaca pairs prompts with preferred and undesirable audios whose flaws include missing concepts, incorrect temporal order, or high noise.
- Dataset Construction: Undesirable audios are created by removing or rearranging concepts in descriptions, generating multiple outputs, and applying adversarial CLAP-based filtering.Filtering selects outputs whose CLAP scores fall below a specified threshold.
- Model Alignment: The authors fine-tune Tango on Audio-alpaca with DPO-diffusion loss to obtain Tango 2.The approach uses preference pairs to align diffusion-based text-to-audio generation.
- Conclusion: Tango 2 reportedly achieves a significant performance leap over previous models on both objective and subjective metrics.The paper presents Audio-alpaca and Tango 2 as resources for further text-to-audio alignment research.