Source-linked AI summary
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization
Chia-Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, Soujanya Poria
TL;DR
Text-to-audio alignment lacks practical tools for creating preference pairs, and existing models can struggle to capture prompt details. TangoFlux uses rectified-flow generation with CRPO, which iteratively creates and optimizes CLAP-ranked preference data. The paper reports state-of-the-art performance across benchmarks and human evaluations, with fast generation and robustness at fewer sampling steps.
Problem
Text-to-audio alignment lacks suitable tools for creating preference pairs, including off-the-shelf reward models and scalable human feedback.
Source: §1 · p. 2Method
CRPO iteratively uses CLAP-ranked generated audio preferences to optimize a rectified-flow text-to-audio model.
Source: §1 · p. 2, §2.4 · p. 4Results
TangoFlux reports state-of-the-art objective and human-evaluation performance, with greater robustness when sampling with fewer time steps.
Source: §6 · p. 13Takeaways & Limitations
The authors characterize TangoFlux as a practical and scalable solution for widespread adoption.
Source: §6 · p. 13CRPO relies on the assumption that CLAP cosine similarity is a reasonable proxy reward for evaluating audio against text descriptions.
Source: §2.5.1 · p. 5
Abstract
from arXiv · showhide
We introduce TangoFlux, an efficient Text-to-Audio (TTA) generative model with 515M parameters, capable of generating up to 30 seconds of 44.1kHz audio in just 3.7 seconds on a single A40 GPU. A key challenge in aligning TTA models lies in the difficulty of creating preference pairs, as TTA lacks structured mechanisms like verifiable rewards or gold-standard answers available for Large Language Models (LLMs). To address this, we propose CLAP-Ranked Preference Optimization (CRPO), a novel framework that iteratively generates and optimizes preference data to enhance TTA alignment. We demonstrate that the audio preference dataset generated using CRPO outperforms existing alternatives. With this framework, TangoFlux achieves state-of-the-art performance across both objective and subjective benchmarks. We open source all code and models to support further research in TTA generation.
1 INTRODUCTION
TangoFlux addresses gaps in text-to-audio alignment and generation with a fast model and an iterative preference-optimization framework. The paper reports strong benchmark and human-evaluation performance, while CRPO-generated preference data outperforms static alternatives.
1 INTRODUCTION
- TTA alignment had not been a mainstay, while generated audio can omit prompt details or include unmentioned events that diverge from user intent.These issues motivate alignment for audio outputs.Source: §1 · p. 2
- TTA lacks off-the-shelf reward models, human-feedback data, and verifiable gold answers for creating preference pairs; manual audio labeling can also be impractical at scale.Audio-language-model feedback is described as too noisy for preference-pair creation.Source: §1 · p. 2
- CRPO-generated audio preference data yields better post-alignment performance than static datasets such as BATON and Audio-Alpaca.CRPO iteratively generates preference data and optimizes rectified flows.Source: §1 · p. 2
- Many TTA models use proprietary data and closed weights, while diffusion-based models may require many denoising steps, consuming compute and time.The paper positions TangoFlux as a smaller, fast model based on rectified flow.Source: §1 · pp. 2–3
- 3.7 seconds on an A40 GPU; TangoFlux claims state-of-the-art benchmark and out-of-distribution human-evaluation performance despite smaller size, with variable-duration generation up to 30 seconds.The model is trained on non-proprietary data.Source: §1 · p. 3
- The authors plan to release the code and model weights to support research in text-to-audio generation.Source: §1 · p. 3
2 METHOD
TANGOFLUX generates audio with a rectified-flow model and aligns it through CRPO, which iteratively creates preference pairs from its own outputs. Its design combines text and duration conditioning with a preference objective that also constrains the winning audio’s flow-matching loss.
2 METHOD
- TANGOFLUX combines pretraining and fine-tuning with CRPO alignment, which iteratively generates preference data and optimizes the model.Each iteration generates data online, estimates rewards and creates preference pairs, then fine-tunes the current checkpoint.Source: §2 · p. 3, §2.5 · p. 5
2.1 AUDIO ENCODING
- A frozen Stable Audio Open VAE encodes 44.1kHz stereo audio into latents and decodes those latents back to stereo audio.The VAE remains frozen during TANGOFLUX training.Source: §2.1 · p. 3
2.2 MODEL CONDITIONING
- Text embeddings and a duration embedding condition generation, with duration conditioning controlling output length up to 30 seconds.A small neural network encodes duration, and its embedding is concatenated with the text encoding.Source: §2.2 · p. 4
2.3 MODEL ARCHITECTURE
- The 515M-parameter backbone uses six MMDiT blocks followed by 18 DiT blocks, each with eight attention heads of width 128.The architecture has total width 1024.Source: §2.3 · p. 4
2.4 FLOW MATCHING
- Rectified flow learns a time-dependent vector field along a straight, shortest path from noise to the target audio distribution.The model regresses the path’s ground-truth velocity using a flow-matching loss.Source: §2.4 · p. 4
2.5 CLAP-RANKED PREFERENCE OPTIMIZATION (CRPO)
- CRPO uses CLAP as a proxy reward model to rank generated audio against its prompt, then iteratively aligns successive TANGOFLUX checkpoints.The approach generates online data and constructs preference pairs at each iteration.Source: §2.5 · p. 5
2.5.1 CLAP AS A REWARD MODEL
- CLAP ranks each prompt’s generated audios by text-audio similarity, and the highest- and lowest-ranked samples become the preference pair.The method assumes CLAP is a reasonable proxy for evaluating audio against the description.Source: §2.5.1 · p. 5, §2.5.2 · p. 5
2.5.3 PREFERENCE OPTIMIZATION
- LDPO-FM can be minimized even as both winning and losing losses increase; CRPO adds the winning-audio flow-matching loss to prevent the winning loss from rising.The added loss is computed on winning audio, and the authors report LCRPO outperforms LDPO-FM in Section 4.5.Source: §2.5.3 · p. 6
3 EXPERIMENTS
The experiments describe TANGOFLUX training, data preparation, and evaluation protocols for objective and human assessments. Objective evaluation uses an AudioCaps test split and established metrics, while subjective ratings assess audio quality and text relevance on challenging prompts.
3.1 MODEL TRAINING
- TANGOFLUX was pretrained on WavCaps for 80 epochs, fine-tuned on AudioCaps for 65 epochs, then aligned for five CRPO iterations after performance saturation appeared.Pretraining used five A40 GPUs with an overall batch size of 80; alignment used eight epochs per iteration and selected the last checkpoint for online generation.Source: §3.1 · p. 6
3.2 DATASETS
- Training data comprised about 400k WavCaps audios and 45k AudioCaps training audios; CRPO sampled 20k prompts and generated five audios per prompt per iteration.Audio was padded or center-cropped to 30 seconds, and mono files were duplicated into pseudostereo.Source: §3.2 · pp. 6–7
3.3 OBJECTIVE EVALUATION
- Objective evaluation used the same 886-sample AudioCaps test split as prior work, comparing Tango 2, AudioLDM 2, and Stable Audio Open against TANGOFLUX.For TANGOFLUX and Stable Audio Open, evaluation used the first 10 seconds of audio generated with 10-second duration conditioning.Source: §3.2 · p. 7, §3.3 · p. 7
3.4 HUMAN EVALUATION
- Human evaluation rated overall audio quality and text relevance on a 0–100 scale using at least four annotators per sample across 50 challenging prompts.Results were reported using z-score, Ranking, and Elo score.Source: §3.2 · p. 7, §3.4 · p. 7
3.3 OBJECTIVE EVALUATION
- Objective metrics were FDopenl3, KLpasst, CLAPscore, and Inception Score, measuring audio similarity, semantic-tag similarity, text alignment, and sample diversity, respectively.Lower FDopenl3 and KLpasst are favorable; higher CLAPscore and Inception Score are favorable.Source: §3.3 · p. 7
4 RESULTS
TangoFlux’s objective results are stronger than prior work on nearly all AudioCaps metrics, especially for multi-event prompts. The preference-optimization experiments favor online data generation and CRPO, while LCRPO shows stronger CLAPscore results than LDPO-FM.
4.1 MAIN RESULTS
- TANGOFLUX outperforms prior works on nearly all AudioCaps objective metrics, with a larger CLAPscore margin on multi-event prompts.Tango 2 is the exception on KLpasst.Source: §4.1 · p. 8
4.2 BATCHED ONLINE DATA GENERATION IS NECESSARY
- Offline CRPO degrades after iteration 2, while online CRPO improves through iteration 4 and outperforms offline CRPO on CLAP score and KL.Offline training’s final CLAP score and KL are worse than at its first iteration; online training’s KL trends downward.Source: Figure 2 · p. 8, §4.2 · p. 9
4.3 CLAP AS REWARD MODEL
- Increasing Best-of-N improves CLAPscore and KLpasst while FDopenl3 remains about the same.The results support CLAP as a proxy reward for selecting audio that better represents text without compromising diversity or quality.Source: §4.3 · p. 9
4.4 CRPO SURPASSES STATIC AUDIO PREFERENCE DATASETS
- Preference optimization with the CRPO dataset outperforms Audio-Alpaca and BATON across FDopenl3, KLpasst, and CLAPscore.The comparison applies one iteration of optimization to TANGOFLUX-base because the alternative datasets are fixed.Source: Table 3 · p. 9, §4.4 · p. 10
4.5 LCRPO VS LDPO-FM
- LCRPO achieves superior CLAPscore while maintaining similar KLpasst and FDopenl3 to LDPO-FM across iterations.Both methods’ winning and losing losses and their margin increase; LCRPO’s loss growth is more gradual, with a smaller margin and less performance degradation than LDPO-FM.Source: Figure 3 · p. 9, §4.5 · p. 10, Figure 4 · p. 10
4. Preference Optimization:
The preference-optimization results favor CRPO data and LCRPO’s more stable optimization, while TangoFlux performs strongly in human evaluations and offers favorable inference-time performance.
4. Preference Optimization:
- CRPO’s preference dataset beats BATON and Audio-Alpaca across metrics, and LCRPO is more stable and effective than LDPO-FM.The reported optimization behavior includes reduced performance saturation and better benchmark results for LCRPO.Source: §4 · p. 11
4.6 HUMAN EVALUATION RESULTS
- Tango 2 outperforms StableAudio on REL mean rank (1.9 vs. 3.3), while StableAudio’s OVL ranks have modes 1 and 3.The authors describe the bimodal OVL ranks as suggesting polarized annotator perceptions.Source: §4.6 · p. 12
- TANGOFLUX records the highest human-evaluation z-scores for OVL (0.2486) and REL (0.6919), and the highest Elo scores for OVL (1,501) and REL (1,628).The table reports OVL as overall audio quality and REL as relevance to the text input.Source: §4.6 · pp. 11–12, Table 5 · p. 12
4.7 INFERENCE TIME VS PERFORMANCE
- TANGOFLUX has better performance per inference time than other models on CLAP and FD, and shows less performance decline when sampling with fewer steps.The paper also reports approximately 2x faster generation alongside better audio quality measured by CLAP and FD.Source: §4 · p. 11, §4.7 · p. 12
5 RELATED WORKS
Prior text-to-audio systems use diffusion or flow-matching frameworks. Preference alignment is established in language models and has been applied to diffusion and flow-based models, but audio preference-pair construction remains challenging.
Text-To-Audio Generation
- Text-to-audio generation commonly uses latent diffusion conditioned on text embeddings or flow matching.The passage names AudioLDM, Tango, and Stable Audio among diffusion-based models, and VoiceBox, AudioBox, and FlashAudio among flow-matching models.Source: §5 · p. 12
Alignment Method
- Preference alignment uses reward models, model-based judgments, human-created pairs, or verifiable answers, while TTA lacks gold audio and scalable preference annotation.BATON relies on human annotations, which the passage describes as not scalable.Source: §5 · p. 13
6 CONCLUSION
TangoFlux is a fast flow-based text-to-audio model aligned with synthetic preference data generated online. The paper reports stronger text representation than existing diffusion-based models, state-of-the-art performance with fewer parameters, and robustness at fewer sampling steps.
6 CONCLUSION
- TANGOFLUX is a flow-based text-to-audio model aligned using synthetic preference data generated online during training.Source: §6 · p. 13
- Objective and human evaluations show that TANGOFLUX produces audio more representative of user prompts than existing diffusion-based models, with state-of-the-art performance using significantly fewer parameters.Source: §6 · p. 13
- TANGOFLUX maintains performance when sampled with fewer time steps, which the authors describe as supporting a practical and scalable solution.Source: §6 · p. 13
A.1 EFFECT OF CFG SCALE
The CFG-scale ablation shows a trade-off between audio fidelity and semantic alignment, with CFG=3.5 providing the best balance.
A.1 EFFECT OF CFG SCALE
- CFG=3.5 gives the optimal balance: higher CFG improves FD (lower is better) but slightly reduces CLAP, which peaks at 3.5.Source: §A.1 · p. 18
A.2 INFERENCE TIME VS PERFORMANCE COMPARISON
TANGOFLUX achieves stronger CLAP and FD scores with less inference time than competing models, and retains much of its performance at low step counts.
A.2 INFERENCE TIME VS PERFORMANCE COMPARISON
- At 50 steps, TANGOFLUX scores CLAP 0.480 and FD 75.1 in 3.7 seconds, versus Stable Audio Open's 0.284 and 87.8 in 4.5 seconds.Source: Figure 5 · p. 18, §A.2 · p. 18
- At 10 steps, TANGOFLUX scores CLAP 0.465 and FD 77.2 in 1.1 seconds, while AudioLDM2 scores 0.357 and 131.7 in 1.5 seconds.Source: §A.2 · p. 18
- Reducing TANGOFLUX from 200 to 10 steps changes CLAP by only -3.2% and FD by +4.5%, compared with Tango 2 changes of -16.0% and +37.8%.Source: §A.2 · p. 18
- The authors characterize TANGOFLUX as an efficient choice when inference time is critical.Source: §A.2 · p. 19
A.3 HUMAN EVALUATION
The human evaluation compares four models on shared prompts, with annotators rating audio quality and prompt relevance separately.
A.3 HUMAN EVALUATION
- Each annotator evaluates 20 prompts, each paired with four shuffled audio samples from distinct text-to-audio models.Source: §A.3 · p. 19
- Annotators rate overall quality from 0 to 100 based on fidelity, clarity, and noisiness, independently of prompt alignment.Source: §A.3 · p. 19
- Relevance rates audio-prompt alignment from 0 to 100, including whether prompt concepts appear in the described temporal order.Source: §A.3 · p. 19
A.3.1 EVALUATION DATASET
The appendix describes the human-evaluation dataset and scoring, its complex prompts, and additional preference and relation-aware evaluation resources.
A.3.1 EVALUATION DATASET
- The evaluation dataset contains 50 out-of-distribution captions with 3–6 events, designed to test complex instruction following and temporal ordering beyond common sound examples.Source: §A.3.1 · p. 19
A.3.2 METRICS
- Scoring includes annotator-level z-score normalization, mean and mode model ranks, and Elo scores derived from pairwise comparisons to capture relative strength.Source: §A.3.2 · pp. 20–21
A.3.3 PROMPTS USED IN THE EVALUATION
- The prompt examples combine multiple sound events, such as a robotic arm, plasma arc, metallic voice, and falling glass vials.Source: §A.3.3 · p. 21
A.4 BATON AS A PREFERENCE DATASET
- BATON preference pairs match audio labeled aligned with the prompt as winners against misaligned audio labeled as losers for that same prompt.Source: §A.4 · p. 23
A.5 MULTI-STAGED RELATION-AWARE EVALUATION
- The appendix identifies an Audio Events Relation Corpus and an Audio Events Category Corpus for multi-staged relation-aware evaluation.Source: Table 8 · p. 24, Table 9 · p. 24