PaperlayerView exact revision on arXiv

Source-linked AI summary

WavFlow: Audio Generation in Waveform Space

Feiyan Zhou, Luyuan Wang, Shoufa Chen, Zhe Wang, Zhiheng Liu, Yuren Cong, Xiaohui Zhang, Fanny Yang, Belinda Zeng

arXiv:2605.18749v1cs.SDcs.CV

TL;DR

Latent-space compression is widely used in audio generation, but WavFlow asks whether it is necessary. It generates audio directly in raw waveform space and matches or exceeds latent-based systems across video-to-audio and text-to-audio benchmarks.

  • Problem

    Whether latent-space compression is necessary remains open because raw-waveform modeling is high-dimensional, difficult to optimize, and constrained by scarce paired video-audio data.

  • Method

    WavFlow uses waveform patchification, x-prediction, signal scaling, and approximately 5 M curated video-text-audio pairs for raw-waveform flow matching.

  • Results

    WavFlow matches or exceeds established latent-based systems across VGGSound and AudioCaps, including FDPANNs (10.63) on AudioCaps.

  • Takeaways & Limitations

    The findings support direct raw-waveform generation as an alternative to latent-space audio synthesis without intermediate representations.

  • Takeaways & Limitations

    WavFlow does not yet produce meaningful speech or singing, which requires finer linguistic granularity and larger speech datasets.

Abstract

from arXiv · show

Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.

1 Introduction

WavFlow questions the necessity of latent-space compression by generating Foley-style audio directly from raw waveforms. It addresses raw-space modeling challenges with waveform patchification, stabilized flow matching, amplitude scaling, and large-scale data curation, achieving competitive benchmark performance.

  • Motivation: Latent-space pipelines add tokenizer and decoder stages, making synthesis quality depend on reconstruction fidelity and motivating direct raw-waveform generation.The paper frames compression as a source of pipeline complexity and a potential quality constraint.
  • Challenges: Raw-waveform modeling is difficult because sequences are high-dimensional, amplitudes have poor training signal-to-noise ratios, and available paired data is limited.VGGSound contains only ∼200K samples (500 hours), insufficient for learning complex acoustic structures and cross-modal alignments end-to-end.
  • Method: WavFlow reshapes 1D waveforms into 2D token grids with waveform patchify and uses x-prediction under conditional flow matching for more stable raw-signal training.Its signal preprocessing combines RMS normalization and amplitude scaling to address the mismatch with the unit-variance Gaussian prior.
  • Data Curation: An automated curation pipeline harvests high-quality, large-scale multimodal VT2A supervision to address data scarcity and sensitivity in direct waveform learning.The pipeline is designed to improve audio quality and event diversity in the training data.
  • Empirical Validation: 55.82 at 44.1 kHz and 59.98 at 16 kHz FDPaSST on VGGSound, alongside DeSync (0.44) and ISPANNs (17.40), demonstrate competitive raw-waveform synthesis against latent-based models.On AudioCaps, WavFlow attains the best FDPANNs (10.63) and ISPANNs (12.62).

2 Related Work

Prior audio generation largely operates in continuous or discrete latent spaces, whose reconstruction bottlenecks can irreversibly compromise waveform detail. Earlier raw-waveform methods established feasibility, while WavFlow extends direct waveform modeling to multimodal synthesis with an MMDiT architecture.

  • Latent-space audio generation: Continuous latent models use audio VAEs and often vocoder discriminators, whereas discrete systems rely on neural audio codecs.Examples include AudioLDM, TANGO, and MMAudio for continuous modeling, and AudioGen and V-AURA for discrete synthesis.
  • Latent-space audio generation: Latent bottlenecking can irreversibly lose high-frequency transients and fine-grained phase information because output quality is bounded by pretrained-backbone reconstruction fidelity.This limitation motivates modeling audio directly in its native, uncompressed space.
  • Raw-waveform modeling: WaveNet, WaveRNN, WaveGrad, and DiffWave showed that high-fidelity raw-waveform synthesis is feasible, but primarily served as vocoders reconstructing waveforms from local spectral features.These approaches represent earlier autoregressive and diffusion-based waveform modeling before latent-space paradigms dominated.
  • Multimodal audio generation: VT2A systems increasingly use MMDiT for joint audio-video-text modeling, progressing from efficient latent synthesis in Frieren to MMAudio’s unified joint-attention paradigm.This architectural evolution targets precise temporal synchronization and semantic consistency through improved cross-modal alignment.
  • Multimodal audio generation: WavFlow eliminates the latent stage within an MMDiT-based architecture, enabling high-fidelity synthesis directly on raw waveforms.This addresses the compression constraint that remains in existing multimodal systems.

3 Method

WavFlow generates raw audio directly in waveform space using conditional flow matching with an MMDiT backbone. Waveform patchify, amplitude lifting, multimodal conditioning, and decoder-free ODE inference address high-dimensional modeling and semantic-temporal alignment.

  • Core formulation: WavFlow uses an MMDiT backbone and conditional flow matching to transport Gaussian noise directly into raw waveforms.The model reshapes audio with waveform patchify and uses x-prediction for stable training.
  • Prediction parameterization: The model predicts the clean signal and recovers velocity through vθ = (x̂1 − xt)/(1 − t), while optimizing a velocity-based loss.This keeps data-manifold recovery anchored to the flow-matching velocity field.
  • Waveform patchify: Waveform patchify converts audio into a C × D token grid, trading finer acoustic detail at smaller D against greater computational complexity.The method identifies D = 200 as the performance saturation point, producing 640 tokens and 12.5 ms granularity for 8-second, 16 kHz clips.
  • Multimodal conditioning and inference: Joint and audio-only transformer blocks fuse waveform tokens with visual and text features, using global and frame-aligned conditions for semantic and temporal control.Classifier-free guidance supports both video-to-audio and text-to-audio generation, while Euler ODE integration reconstructs raw waveforms without a learned decoder.

4 Experiments

Experiments establish that carefully curated, large-scale data and waveform-specific design choices enable WavFlow to achieve strong video-to-audio and text-to-audio performance without latent representations. Ablations further identify patch granularity, x-prediction with v-loss, and amplitude preprocessing as important factors for quality and synchronization.

  • Data Curation Pipeline: A unified automated pipeline produces balanced training mixtures from VGGSound, AudioCaps, and Freesound after multi-stage quality filtering.The pipeline extracts 8 s segments and removes samples with over 80% silence, PQ below 6.0, or bottom-10% PANNs classification confidence.
  • Data Curation Pipeline: 5 M category-aligned media samples form the balanced pool, while temporal augmentation expands VGGSound and public text-to-audio data to 200 K and 300 K samples.Two overlapping 8 s chunks begin at 0 s and 1 s, respectively.
  • Video-to-Audio Generation: 59.98 FDPaSST, 17.40 ISPANNs, and 0.44 DeSync show WavFlow-L-16kHz surpassing MMAudio-L-44.1kHz in distributional fidelity while matching perceptual and alignment metrics.WavFlow-L-44.1kHz further achieves the best reported FDPaSST of 55.82 with DeSync of 0.46.
  • Text-to-Audio Generation: 10.63 FDPANNs and 12.62 ISPANNs make WavFlow-M-16kHz competitive on AudioCaps, attaining the lowest FDPANNs and highest ISPANNs among compared methods.The unified model outperforms dedicated latent-space systems and the previous state-of-the-art MMAudio on these reported measures.
  • Patchify Granularity Analysis: 136.45 → 90.24 FDPaSST and 0.66 → 0.59 DeSync result when reducing D from 512 to 200 in the 200 K-data regime, showing the value of finer temporal tokenization.At fixed D=512, expanding data from 200 K to 1 M improves FDPaSST from 136.45 to 81.81, but 3 M samples degrade performance to 89.51.
  • Prediction Target and Loss Formulation: 63.05 FDPaSST and 15.58 ISPANNs are achieved by x-prediction with v-loss, which becomes the default because it balances diversity and high-frequency fidelity.x-prediction consistently outperforms v-prediction across metrics, while x-prediction with x-loss has a slight advantage in FDPANNs at 4.86.
  • Raw-waveform Preprocessing: 15.58 ISPANNs is obtained when RMS normalization and amplitude scaling are combined, whereas omitting RMS normalization at 1.0× worsens FDPaSST from 65.83 to 81.26 and DeSync from 0.49 to 0.57.The ablation finds both preprocessing techniques essential for overall signal quality.

5 Conclusion and Limitations · Appendix

WavFlow generates high-fidelity audio directly in raw-waveform space, using waveform patchify, x-prediction, signal scaling, and extensive data to stabilize training. It achieves competitive benchmark performance but currently lacks meaningful speech and singing synthesis.

  • 5 Conclusion and Limitations: WavFlow is a flow-matching framework for high-fidelity audio generation directly in raw-waveform space.
  • 5 Conclusion and Limitations: Waveform patchify, x-prediction, specific signal scaling, and extensive data stabilize training and enable modeling of complex raw signals.
  • 5 Conclusion and Limitations: WavFlow achieves highly competitive performance on VT2A and T2A benchmarks.
  • 5 Conclusion and Limitations: The framework matches or exceeds established latent-based systems.
  • 5 Conclusion and Limitations: WavFlow currently lacks explicit speech or singing synthesis because its generated vocalizations do not constitute meaningful language.
  • 5 Conclusion and Limitations: Extending WavFlow to speech and singing requires finer linguistic granularity and larger speech datasets.
  • 5 Conclusion and Limitations: Larger-scale corpora and fine-grained linguistic captions could support joint modeling of environmental sounds and human speech.
  • 5 Conclusion and Limitations: Such extensions could offer a robust and efficient alternative for future generative research.

A Training Details

WavFlow variants share a standardized H100 training setup, while main 16 kHz models use large-batch training from scratch on a 5 M-media mixture. Convergence studies establish 400 epochs as the default at data scales of at least 1 M, whereas smaller-batch and 44.1 kHz settings require about 650 epochs.

  • All variants use NVIDIA H100 GPUs, AdamW with β1=0.9 and β2=0.95, EMA decay 0.9999, gradient clipping 1.0, and BF16 mixed precision.
  • Main 16 kHz VT2A models train from scratch with learning rate 1 × 10−4 and global batch size 10,752 on 5 M media data plus 200 K augmented VGGSound.
  • The 200 K VGGSound-only run and the 44.1 kHz fine-tuning setting each use approximately 650 epochs to reach convergence.The 200 K run uses batch size 1,536, while the 44.1 kHz setting uses a reduced learning rate.
  • Models trained with ≥1 M samples converge around 400 epochs, based on VGGSound validation metrics across four data scales and corresponding global batch sizes.The evaluated scales are 200 K, 1 M, 3 M, and 5 M, with batch sizes 1,536, 5,632, 8,192, and 10,752, respectively.

B Audio Amplitude Distribution

Raw waveform amplitudes span a wide dynamic range but are concentrated near zero, with most RMS levels below 0.2. RMS normalization to 0.33 followed by ×3.0 scaling broadens the distributions toward [−3, 3] and better aligns them with N(0, 1) noise.

  • Raw amplitude distribution: Most waveform RMS levels remain below 0.2, producing sharply zero-centered distributions and low-energy signals that are easily submerged by noise.Histograms from 15 randomly sampled clips show that quiet samples occupy little of the dynamic range, while higher-energy samples remain concentrated near zero.
  • Amplitude preprocessing: RMS normalization to 0.33 followed by ×3.0 amplitude scaling spreads amplitudes more broadly across the [−3, 3] range.The preprocessing effectively lifts the signals, although the resulting distributions remain non-uniform.
  • Amplitude preprocessing: Preprocessing better aligns amplitude statistics with the standard N(0, 1) noise prior, keeping signals discernible at high noise levels.This reduces the risk that low-energy waveform signals are submerged during flow matching denoising.

C Data Mixture and Rationale for VT2A

The VT2A data-mixture study finds that visual supervision stabilizes training across mismatched caption styles, while sparse-label VGGSound with Media data provides the best distributional fidelity and synchronization. Dense captions improve semantic diversity but can reduce consistency with sparse-label evaluation.

  • Data sources: The study compares VGGSound (200K), Open-source T2A data (300K), and a 5 M Media collection, using a 1 M Media subset for ablations.Open-source T2A combines FreeSound and AudioCaps audio-only samples with fine-grained captions; Media consists of proprietary video-audio pairs with detailed captions.
  • Caption and modality alignment: Directly mixing VGGSound sparse labels with Open-source T2A fine-grained captions diverged because their text embeddings had a severe semantic mismatch without visual grounding.The loss initially decreased but eventually spiked.
  • Caption and modality alignment: VGGSound sparse labels plus 1 M Media data converged reliably despite differing text granularity, confirming visual features can anchor cross-style semantic alignment.Both datasets provide a visual modality, unlike the T2A samples.
  • Caption quality: 17.05 vs. 16.42 ISPANNs shows Dense VGGSound captions improve performance over sparse captions in the stabilized VGGSound-plus-Media mixture.The result indicates that fine-grained descriptions support more diverse and complex audio-visual semantic mappings.
  • Final mixture selection: 121.09 vs. 125.52 FDPaSST and 0.52 →0.57 DeSync show sparse-label VGGSound plus Media is preferred for distributional fidelity and temporal synchronization.Dense VGGSound captions stabilized the T2A mixture but underperformed VGGSound-only training, with DeSync 0.52 →0.57 and IB 0.29 →0.26.

D Waveform Patchify Configuration

WavFlow reshapes 8-second, 16 kHz waveforms into C × D token grids, using padding and truncation to support arbitrary lengths. Ablations show that finer patch granularity stabilizes quality and synchronization, motivating 640 × 200 as the default configuration.

  • Configuration: 8-second, 16 kHz waveforms are reshaped into C × D grids, with zero-padding before patching and truncation after unpatchifying to restore T = 128,000 samples.Padding handles waveforms whose lengths are not divisible by D.
  • Ablation: The patch-dimension sweep ranges from D = 512 to D = 160 and tests C = 576 or 768 configurations aligned to 192 Synchformer tokens.These settings evaluate whether matching token counts improves temporal alignment.
  • Ablation: D = 512, corresponding to 32 ms per token, performs significantly worse, while D < 256, corresponding to 16 ms, stabilizes generative and synchronization performance at a high level.The results indicate that finer granularity is needed to capture waveform details.
  • Configuration: 640 × 200 is selected as the default 16 kHz configuration because it balances generative quality, computational efficiency, and synchronization.Sync-aligned settings such as C = 576 or 768 do not substantially outperform other sufficiently fine-grained configurations.

E Effect of Noise-Level Shift

Noise-level shifting is parameterized to bias training toward higher noise levels, but it consistently harms waveform-space audio generation as the shift increases. WavFlow therefore uses no shift, s = 1.0, because audio’s weak signal is harder to recover under increased noise.

  • Noise-shift formulation: The shift ts = t/(t + s · (1 − t)) biases training toward higher noise levels and reduces effective signal-to-noise ratio by s^2.The shift factor is s.
  • Results: FDPaSST rises from 63.05 (s = 1.0) to 73.17 (s = 3.0) and 92.21 (s = 5.0), with all metrics progressively degrading as s increases.On VGGSound-Test, noise shift provides little benefit for waveform-space modeling.
  • Results: Increasing noise harms audio because waveforms have low information density, making their already weak signals harder to recover.This contrasts with images, whose broad value range allows high-noise training to capture low-frequency global structure.
  • Default setting: WavFlow adopts s = 1.0 (no shift) as the default configuration.The evaluation uses WavFlow-M-16k with 1M training data on 8 s waveforms.

F Inference Hyperparameters

Inference quality depends strongly on classifier-free guidance and ODE integration steps. The adopted default, CFG = 4.5 with 50 ODE steps, balances peak quality against diminishing returns.

  • Default configuration: The default inference configuration is CFG = 4.5 with 50 ODE steps, offering the best generation quality before diminishing returns.The ablation evaluates these hyperparameters on the VGGSound validation set.
  • Classifier-free guidance: CFG = 1.0 produces low-fidelity outputs, with IB: 0.26 and severe temporal artifacts measured by DeSync: 0.71.Insufficient guidance also harms semantic alignment.
  • Classifier-free guidance: CFG = 2.5 achieves the best distributional fidelity at FDPaSST: 108.19, whereas CFG = 4.5 reaches the highest per-sample quality with IS: 16.42 and IB: 0.33.Increasing guidance to 7.0 sharply degrades FDPaSST without meaningful quality gains, indicating reduced output diversity.
  • ODE integration steps: Increasing ODE integration from 10 to 25 steps significantly improves quality, while 25 to 50 steps further raises IS (16.19 →16.42) and IB (0.32 →0.33).Metrics plateau beyond 50 steps, and 100 steps provides no additional benefit.

G Evaluation on MovieGen-Audio-Bench

WavFlow generalizes effectively to the challenging MovieGen-Audio-Bench of AI-generated videos, using reference-free metrics because ground-truth audio is unavailable. It achieves strong audio quality and temporal synchronization while remaining competitive in semantic alignment.

  • Benchmark and protocol: MovieGen-Audio-Bench evaluates synthesis on AI-generated videos, requiring models to understand audio-visual correlations without ground-truth audio.The evaluation therefore uses reference-free metrics: IS for audio quality, CLAP and IB-score for semantic alignment, and DeSync for synchronization.
  • Quantitative results: 8.95 IS and 0.77 DeSync show WavFlow’s strong audio quality and temporal synchronization on synthetic visual content.WavFlow outperforms or matches existing latent-based models on these dimensions.
  • Quantitative results: MovieGen retains an advantage in IB-score, likely because of its substantially larger model capacity and training scale, while WavFlow remains highly competitive.This comparison concerns semantic richness rather than the reported audio-quality or synchronization metrics.
  • Qualitative synchronization: For Penguin Walking and Horse Trotting, WavFlow’s spectrograms show sharp vertical energy pulses that preserve fine-grained transients and correspond closely to footfalls.The supplied figure descriptions characterize WavFlow’s pulses as sharper or more defined than competing models in these scenarios.
  • Qualitative synchronization: In the Boxing scenario, WavFlow and MMAudio synchronize precisely, but WavFlow alone omits impact sound during air punches without bag contact.MovieGen exhibits desynchronization, while MMAudio and MovieGen incorrectly synthesize strike energy.
Loading 2605.18749v1…