DriftAudio: Marginal Drifting for Distributional Post-Training of One-Step Text-to-Audio Generators

Motivation

• Low-NFE methods reduce sampling cost, and MeanAudio enables strict 1-NFE generation, but generators can still benefit from quality refinement.Consistency-based methods enable one- or few-step synthesis, while trajectory rectification improves few-step generation.Source: §1 · p. 1

Related Work

• Distributional post-training refines a pretrained generator’s induced distribution without introducing a new inference-time sampler.FdAudio [13] matches real and generated distributions across frozen audio feature spaces using Fréchet objectives [14] based on first- and second-order moments.Source: §1 · p. 1

Motivation

• Free-form text conditions usually have only one or a few real samples, making condition-wise Drifting difficult to estimate.DriftAudio instead refines the marginal output distribution after conditional generation has been learned.Source: §1 · p. 1

Approach

• DriftAudio applies marginal Drifting to pretrained one-step TTA generators while retaining text-conditioned generation.It uses resampled real features and a FIFO bank of generated features; the estimated field becomes a detached target that updates only the generator and preserves one-step inference.Source: §1 · p. 1

2.1. Pretrained One-Step TTA Generator

• The DriftAudio overview keeps the generator text-conditioned while performing Drifting on the marginal audio distribution in feature space.Real features are resampled from training data, while generated references are kept in a FIFO bank initialized with pre-generated samples and updated with current samples.Source: Figure 1 · p. 2
• MeanAudio models average velocity over a finite interval, and taking the full noise-to-data interval yields a one-step latent generator.The latent is converted to a waveform by an audio decoding pipeline comprising a VAE and vocoder.Source: §2.1 · p. 2

2.2. Distribution Matching with Drifting

• Drifting evolves the generator-induced distribution toward the data distribution through a distribution-dependent field.The field uses real and generated reference samples, and its expectations can be approximated with finite samples in a feature space.Source: §2.2 · p. 2

3. DRIFTAUDIO

• DriftAudio preserves text-conditioned generation while constructing the marginal drifting field in a frozen audio feature space and updating only the generator.Source: §3 · p. 2

3.1. Marginal Drifting under Free-Form Conditioning

• Because free-form text conditions may have only one or a few real samples, DriftAudio applies Drifting to marginal distributions collected across text conditions rather than estimating each condition’s distribution separately.Source: §3.1 · p. 2
• The estimator uses current generated features, real features resampled from the training set, and a FIFO bank of detached recent generated features refreshed with current samples.Source: §3.1 · pp. 2–3

3.2. Drifting-Based Post-Training

• The generator objective regresses current generated features toward detached drifting targets, with a distance-based normalization, and backpropagates only through the generator pathway.Source: §3.2 · p. 3

4.1. Experimental Setup

• AudioCaps evaluation used 45,173 training, 445 validation, and 877 test clips, with frozen PANNs features and per-step 2,048-real/1,024-generated references.Source: §4.1 · p. 3

4.2. Main Results and Comparisons

• Fig. 2 decomposes FAD into mean and covariance components; DriftAudio reduces both from either starting checkpoint, including MeanAudio covariance from 1.548 to 1.021 and mean from 0.129 to 0.090.The reductions are also observed in VGGish features, which DriftAudio does not use for post-training.Source: Figure 2 · p. 3, §4.2 · p. 4
• Both DriftAudio and FdAudio preserve MeanAudio’s inference procedure and add no inference-time computation because post-training changes only model weights.Source: §4.2 · p. 3
• Starting from MeanAudio, DriftAudio reduces FAD from 1.68 to 1.11 and FD from 15.43 to 12.71, while achieving KL of 1.21; from FdAudio, it reduces FAD from 1.22 to 0.99 and FD from 13.73 to 12.61.Starting from MeanAudio, CLAP improves slightly while IS decreases; from FdAudio, both IS and CLAP decrease.Source: §4.2 · p. 3, Table 1 · p. 4

4.3. Analysis and Ablation Study

• FdAudio leads all three text–audio alignment metrics, with R@1 of 38.7%; MeanAudio-start DriftAudio raises R@1 from 32.5% to 34.5%, while FdAudio-start DriftAudio remains above MeanAudio.Starting from FdAudio, DriftAudio shows a modest decrease relative to FdAudio.Source: Table 2 · p. 4, §4.3 · p. 4
• Continued FdAudio training worsens FAD from 1.22 to 1.40, while DriftAudio reaches 0.99; fixed real references raise FAD from 0.99 to 1.04.The continued-training control supports that DriftAudio’s gains cannot be explained by additional optimization alone; real-reference resampling improves FAD and FD.Source: Table 3 · p. 4, §4.3 · p. 4
• AudioMAE remains competitive but PANNs performs better; doubling references yields FAD 0.98 while median step time rises from 23.7 s to 49.5 s (2.09×).The default reference sizes provide a practical balance between distributional performance and training cost.Source: Table 3 · p. 4, §4.3 · p. 4

• DriftAudio applies Drifting to the marginal audio-feature distribution of pretrained one-step TTA generators.Source: §5 · p. 4
• On AudioCaps, DriftAudio improves FAD, FD, and KL from both MeanAudio and FdAudio checkpoints without substantially weakening text–audio alignment.Source: §5 · p. 4
• The results suggest marginal Drifting may provide a general distributional post-training principle for conditional generative models; future work will add complementary audio representations and subjective evaluation.Source: §5 · p. 4