PaperlayerView exact revision on arXiv

Source-linked AI summary

WavFlow: Audio Generation in Waveform Space

Feiyan Zhou, Luyuan Wang, Shoufa Chen, Zhe Wang, Zhiheng Liu, Yuren Cong, Xiaohui Zhang, Fanny Yang, Belinda Zeng

arXiv:2605.18749v1cs.SDcs.CV

TL;DR

Latent-space compression is widely used in audio generation, but WavFlow asks whether it is necessary. It generates audio directly in raw waveform space and matches or exceeds latent-based systems across video-to-audio and text-to-audio benchmarks.

  • Problem

    Whether latent-space compression is necessary remains open because raw-waveform modeling is high-dimensional, difficult to optimize, and constrained by scarce paired video-audio data.

    Source: §1 · pp. 1–2

  • Method

    WavFlow uses waveform patchification, x-prediction, signal scaling, and approximately 5 M curated video-text-audio pairs for raw-waveform flow matching.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    WavFlow matches or exceeds established latent-based systems across VGGSound and AudioCaps, including FDPANNs (10.63) on AudioCaps.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The findings support direct raw-waveform generation as an alternative to latent-space audio synthesis without intermediate representations.

    Source: §5 · p. 10

    WavFlow does not yet produce meaningful speech or singing, which requires finer linguistic granularity and larger speech datasets.

    Source: §5 · p. 10

Abstract

from arXiv · show

Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.

1 Introduction

WavFlow questions the necessity of latent-space compression by generating Foley-style audio directly from raw waveforms. It addresses raw-space modeling challenges with waveform patchification, stabilized flow matching, amplitude scaling, and large-scale data curation, achieving competitive benchmark performance.

Motivation

  • Latent-space pipelines add tokenizer and decoder stages, making synthesis quality depend on reconstruction fidelity and motivating direct raw-waveform generation.The paper frames compression as a source of pipeline complexity and a potential quality constraint.Source: §1 · p. 1

Challenges

  • Raw-waveform modeling is difficult because sequences are high-dimensional, amplitudes have poor training signal-to-noise ratios, and available paired data is limited.VGGSound contains only ∼200K samples (500 hours), insufficient for learning complex acoustic structures and cross-modal alignments end-to-end.Source: §1 · pp. 1–2

Method

  • WavFlow reshapes 1D waveforms into 2D token grids with waveform patchify and uses x-prediction under conditional flow matching for more stable raw-signal training.Its signal preprocessing combines RMS normalization and amplitude scaling to address the mismatch with the unit-variance Gaussian prior.Source: §1 · p. 2

Data Curation

  • An automated curation pipeline harvests high-quality, large-scale multimodal VT2A supervision to address data scarcity and sensitivity in direct waveform learning.The pipeline is designed to improve audio quality and event diversity in the training data.Source: §1 · p. 2

Empirical Validation

  • 55.82 at 44.1 kHz and 59.98 at 16 kHz FDPaSST on VGGSound, alongside DeSync (0.44) and ISPANNs (17.40), demonstrate competitive raw-waveform synthesis against latent-based models.On AudioCaps, WavFlow attains the best FDPANNs (10.63) and ISPANNs (12.62).Source: §1 · p. 2

2 Related Work

Prior audio generation largely operates in continuous or discrete latent spaces, whose reconstruction bottlenecks can irreversibly compromise waveform detail. Earlier raw-waveform methods established feasibility, while WavFlow extends direct waveform modeling to multimodal synthesis with an MMDiT architecture.

Latent-space audio generation

  • Continuous latent models use audio VAEs and often vocoder discriminators, whereas discrete systems rely on neural audio codecs.Examples include AudioLDM, TANGO, and MMAudio for continuous modeling, and AudioGen and V-AURA for discrete synthesis.Source: §2.1 · p. 3
  • Latent bottlenecking can irreversibly lose high-frequency transients and fine-grained phase information because output quality is bounded by pretrained-backbone reconstruction fidelity.This limitation motivates modeling audio directly in its native, uncompressed space.Source: §2.1 · p. 3

Raw-waveform modeling

  • WaveNet, WaveRNN, WaveGrad, and DiffWave showed that high-fidelity raw-waveform synthesis is feasible, but primarily served as vocoders reconstructing waveforms from local spectral features.These approaches represent earlier autoregressive and diffusion-based waveform modeling before latent-space paradigms dominated.Source: §2.2 · p. 3

Multimodal audio generation

  • VT2A systems increasingly use MMDiT for joint audio-video-text modeling, progressing from efficient latent synthesis in Frieren to MMAudio’s unified joint-attention paradigm.This architectural evolution targets precise temporal synchronization and semantic consistency through improved cross-modal alignment.Source: §2.3 · p. 3
  • WavFlow eliminates the latent stage within an MMDiT-based architecture, enabling high-fidelity synthesis directly on raw waveforms.This addresses the compression constraint that remains in existing multimodal systems.Source: §2.3 · p. 3

3 Method

WavFlow generates raw audio directly in waveform space using conditional flow matching with an MMDiT backbone. Waveform patchify, amplitude lifting, multimodal conditioning, and decoder-free ODE inference address high-dimensional modeling and semantic-temporal alignment.

Core formulation

  • WavFlow uses an MMDiT backbone and conditional flow matching to transport Gaussian noise directly into raw waveforms.The model reshapes audio with waveform patchify and uses x-prediction for stable training.Source: §3 · p. 4, §3.1 · p. 4

Prediction parameterization

  • The model predicts the clean signal and recovers velocity through vθ = (x̂1 − xt)/(1 − t), while optimizing a velocity-based loss.This keeps data-manifold recovery anchored to the flow-matching velocity field.Source: §3.1 · p. 4

Waveform patchify

  • Waveform patchify converts audio into a C × D token grid, trading finer acoustic detail at smaller D against greater computational complexity.The method identifies D = 200 as the performance saturation point, producing 640 tokens and 12.5 ms granularity for 8-second, 16 kHz clips.Source: §3.2 · pp. 4–5

Multimodal conditioning and inference

  • Joint and audio-only transformer blocks fuse waveform tokens with visual and text features, using global and frame-aligned conditions for semantic and temporal control.Classifier-free guidance supports both video-to-audio and text-to-audio generation, while Euler ODE integration reconstructs raw waveforms without a learned decoder.Source: §3.2 · p. 5, §3.3 · p. 6

4 Experiments

Experiments establish that carefully curated, large-scale data and waveform-specific design choices enable WavFlow to achieve strong video-to-audio and text-to-audio performance without latent representations. Ablations further identify patch granularity, x-prediction with v-loss, and amplitude preprocessing as important factors for quality and synchronization.

Data Curation Pipeline

  • A unified automated pipeline produces balanced training mixtures from VGGSound, AudioCaps, and Freesound after multi-stage quality filtering.The pipeline extracts 8 s segments and removes samples with over 80% silence, PQ below 6.0, or bottom-10% PANNs classification confidence.Source: §4.1 · p. 6
  • 5 M category-aligned media samples form the balanced pool, while temporal augmentation expands VGGSound and public text-to-audio data to 200 K and 300 K samples.Two overlapping 8 s chunks begin at 0 s and 1 s, respectively.Source: §4.1 · p. 6

Video-to-Audio Generation

  • 59.98 FDPaSST, 17.40 ISPANNs, and 0.44 DeSync show WavFlow-L-16kHz surpassing MMAudio-L-44.1kHz in distributional fidelity while matching perceptual and alignment metrics.WavFlow-L-44.1kHz further achieves the best reported FDPaSST of 55.82 with DeSync of 0.46.Source: §4.3 · p. 8

Text-to-Audio Generation

  • 10.63 FDPANNs and 12.62 ISPANNs make WavFlow-M-16kHz competitive on AudioCaps, attaining the lowest FDPANNs and highest ISPANNs among compared methods.The unified model outperforms dedicated latent-space systems and the previous state-of-the-art MMAudio on these reported measures.Source: §4.3 · p. 8

Patchify Granularity Analysis

  • 136.45 → 90.24 FDPaSST and 0.66 → 0.59 DeSync result when reducing D from 512 to 200 in the 200 K-data regime, showing the value of finer temporal tokenization.At fixed D=512, expanding data from 200 K to 1 M improves FDPaSST from 136.45 to 81.81, but 3 M samples degrade performance to 89.51.Source: §4.4 · p. 9

Prediction Target and Loss Formulation

  • 63.05 FDPaSST and 15.58 ISPANNs are achieved by x-prediction with v-loss, which becomes the default because it balances diversity and high-frequency fidelity.x-prediction consistently outperforms v-prediction across metrics, while x-prediction with x-loss has a slight advantage in FDPANNs at 4.86.Source: §4.4 · pp. 9–10

Raw-waveform Preprocessing

  • 15.58 ISPANNs is obtained when RMS normalization and amplitude scaling are combined, whereas omitting RMS normalization at 1.0× worsens FDPaSST from 65.83 to 81.26 and DeSync from 0.49 to 0.57.The ablation finds both preprocessing techniques essential for overall signal quality.Source: §4.4 · p. 10

5 Conclusion and Limitations · Appendix

WavFlow generates high-fidelity audio directly in raw-waveform space, using waveform patchify, x-prediction, signal scaling, and extensive data to stabilize training. It achieves competitive benchmark performance but currently lacks meaningful speech and singing synthesis.

5 Conclusion and Limitations

  • WavFlow is a flow-matching framework for high-fidelity audio generation directly in raw-waveform space.Source: §5 · p. 10
  • Waveform patchify, x-prediction, specific signal scaling, and extensive data stabilize training and enable modeling of complex raw signals.Source: §5 · p. 10
  • WavFlow achieves highly competitive performance on VT2A and T2A benchmarks.Source: §5 · p. 10
  • The framework matches or exceeds established latent-based systems.Source: §5 · p. 10
  • WavFlow currently lacks explicit speech or singing synthesis because its generated vocalizations do not constitute meaningful language.Source: §5 · p. 10
  • Extending WavFlow to speech and singing requires finer linguistic granularity and larger speech datasets.Source: §5 · p. 10
  • Larger-scale corpora and fine-grained linguistic captions could support joint modeling of environmental sounds and human speech.Source: §5 · p. 10
  • Such extensions could offer a robust and efficient alternative for future generative research.Source: §5 · p. 10

A Training Details

WavFlow variants share a standardized H100 training setup, while main 16 kHz models use large-batch training from scratch on a 5 M-media mixture. Convergence studies establish 400 epochs as the default at data scales of at least 1 M, whereas smaller-batch and 44.1 kHz settings require about 650 epochs.

  • All variants use NVIDIA H100 GPUs, AdamW with β1=0.9 and β2=0.95, EMA decay 0.9999, gradient clipping 1.0, and BF16 mixed precision.Source: A Training Details · p. 15
  • Main 16 kHz VT2A models train from scratch with learning rate 1 × 10−4 and global batch size 10,752 on 5 M media data plus 200 K augmented VGGSound.Source: A Training Details · p. 15
  • The 200 K VGGSound-only run and the 44.1 kHz fine-tuning setting each use approximately 650 epochs to reach convergence.The 200 K run uses batch size 1,536, while the 44.1 kHz setting uses a reduced learning rate.Source: A Training Details · p. 16
  • Models trained with ≥1 M samples converge around 400 epochs, based on VGGSound validation metrics across four data scales and corresponding global batch sizes.The evaluated scales are 200 K, 1 M, 3 M, and 5 M, with batch sizes 1,536, 5,632, 8,192, and 10,752, respectively.Source: A Training Details · p. 15, Figure 5 · p. 16

B Audio Amplitude Distribution

Raw waveform amplitudes span a wide dynamic range but are concentrated near zero, with most RMS levels below 0.2. RMS normalization to 0.33 followed by ×3.0 scaling broadens the distributions toward [−3, 3] and better aligns them with N(0, 1) noise.

Raw amplitude distribution

  • Most waveform RMS levels remain below 0.2, producing sharply zero-centered distributions and low-energy signals that are easily submerged by noise.Histograms from 15 randomly sampled clips show that quiet samples occupy little of the dynamic range, while higher-energy samples remain concentrated near zero.Source: B Audio Amplitude Distribution · p. 16

Amplitude preprocessing

  • RMS normalization to 0.33 followed by ×3.0 amplitude scaling spreads amplitudes more broadly across the [−3, 3] range.The preprocessing effectively lifts the signals, although the resulting distributions remain non-uniform.Source: B Audio Amplitude Distribution · p. 16, Figure 6 · p. 17
  • Preprocessing better aligns amplitude statistics with the standard N(0, 1) noise prior, keeping signals discernible at high noise levels.This reduces the risk that low-energy waveform signals are submerged during flow matching denoising.Source: B Audio Amplitude Distribution · p. 16, Figure 6 · p. 17

C Data Mixture and Rationale for VT2A

The VT2A data-mixture study finds that visual supervision stabilizes training across mismatched caption styles, while sparse-label VGGSound with Media data provides the best distributional fidelity and synchronization. Dense captions improve semantic diversity but can reduce consistency with sparse-label evaluation.

Data sources

  • The study compares VGGSound (200K), Open-source T2A data (300K), and a 5 M Media collection, using a 1 M Media subset for ablations.Open-source T2A combines FreeSound and AudioCaps audio-only samples with fine-grained captions; Media consists of proprietary video-audio pairs with detailed captions.Source: C Data Mixture and Rationale for VT2A · p. 17

Caption and modality alignment

  • Directly mixing VGGSound sparse labels with Open-source T2A fine-grained captions diverged because their text embeddings had a severe semantic mismatch without visual grounding.The loss initially decreased but eventually spiked.Source: C Data Mixture and Rationale for VT2A · p. 17
  • VGGSound sparse labels plus 1 M Media data converged reliably despite differing text granularity, confirming visual features can anchor cross-style semantic alignment.Both datasets provide a visual modality, unlike the T2A samples.Source: C Data Mixture and Rationale for VT2A · p. 17

Caption quality

  • 17.05 vs. 16.42 ISPANNs shows Dense VGGSound captions improve performance over sparse captions in the stabilized VGGSound-plus-Media mixture.The result indicates that fine-grained descriptions support more diverse and complex audio-visual semantic mappings.Source: C Data Mixture and Rationale for VT2A · p. 17

Final mixture selection

  • 121.09 vs. 125.52 FDPaSST and 0.52 →0.57 DeSync show sparse-label VGGSound plus Media is preferred for distributional fidelity and temporal synchronization.Dense VGGSound captions stabilized the T2A mixture but underperformed VGGSound-only training, with DeSync 0.52 →0.57 and IB 0.29 →0.26.Source: C Data Mixture and Rationale for VT2A · pp. 17–18

D Waveform Patchify Configuration

WavFlow reshapes 8-second, 16 kHz waveforms into C × D token grids, using padding and truncation to support arbitrary lengths. Ablations show that finer patch granularity stabilizes quality and synchronization, motivating 640 × 200 as the default configuration.

Configuration

Ablation

  • The patch-dimension sweep ranges from D = 512 to D = 160 and tests C = 576 or 768 configurations aligned to 192 Synchformer tokens.These settings evaluate whether matching token counts improves temporal alignment.Source: D Waveform Patchify Configuration · p. 18
  • D = 512, corresponding to 32 ms per token, performs significantly worse, while D < 256, corresponding to 16 ms, stabilizes generative and synchronization performance at a high level.The results indicate that finer granularity is needed to capture waveform details.Source: D Waveform Patchify Configuration · p. 19

Configuration

  • 640 × 200 is selected as the default 16 kHz configuration because it balances generative quality, computational efficiency, and synchronization.Sync-aligned settings such as C = 576 or 768 do not substantially outperform other sufficiently fine-grained configurations.Source: D Waveform Patchify Configuration · p. 19

E Effect of Noise-Level Shift

Noise-level shifting is parameterized to bias training toward higher noise levels, but it consistently harms waveform-space audio generation as the shift increases. WavFlow therefore uses no shift, s = 1.0, because audio’s weak signal is harder to recover under increased noise.

Noise-shift formulation

  • The shift ts = t/(t + s · (1 − t)) biases training toward higher noise levels and reduces effective signal-to-noise ratio by s^2.The shift factor is s.Source: E Effect of Noise-Level Shift · p. 19

Results

  • FDPaSST rises from 63.05 (s = 1.0) to 73.17 (s = 3.0) and 92.21 (s = 5.0), with all metrics progressively degrading as s increases.On VGGSound-Test, noise shift provides little benefit for waveform-space modeling.Source: E Effect of Noise-Level Shift · p. 19
  • Increasing noise harms audio because waveforms have low information density, making their already weak signals harder to recover.This contrasts with images, whose broad value range allows high-noise training to capture low-frequency global structure.Source: E Effect of Noise-Level Shift · p. 19

Default setting

F Inference Hyperparameters

Inference quality depends strongly on classifier-free guidance and ODE integration steps. The adopted default, CFG = 4.5 with 50 ODE steps, balances peak quality against diminishing returns.

Default configuration

  • The default inference configuration is CFG = 4.5 with 50 ODE steps, offering the best generation quality before diminishing returns.The ablation evaluates these hyperparameters on the VGGSound validation set.Source: F Inference Hyperparameters · p. 19

Classifier-free guidance

  • CFG = 1.0 produces low-fidelity outputs, with IB: 0.26 and severe temporal artifacts measured by DeSync: 0.71.Insufficient guidance also harms semantic alignment.Source: F Inference Hyperparameters · p. 19
  • CFG = 2.5 achieves the best distributional fidelity at FDPaSST: 108.19, whereas CFG = 4.5 reaches the highest per-sample quality with IS: 16.42 and IB: 0.33.Increasing guidance to 7.0 sharply degrades FDPaSST without meaningful quality gains, indicating reduced output diversity.Source: F Inference Hyperparameters · p. 19

ODE integration steps

  • Increasing ODE integration from 10 to 25 steps significantly improves quality, while 25 to 50 steps further raises IS (16.19 →16.42) and IB (0.32 →0.33).Metrics plateau beyond 50 steps, and 100 steps provides no additional benefit.Source: F Inference Hyperparameters · p. 19

G Evaluation on MovieGen-Audio-Bench

WavFlow generalizes effectively to the challenging MovieGen-Audio-Bench of AI-generated videos, using reference-free metrics because ground-truth audio is unavailable. It achieves strong audio quality and temporal synchronization while remaining competitive in semantic alignment.

Benchmark and protocol

  • MovieGen-Audio-Bench evaluates synthesis on AI-generated videos, requiring models to understand audio-visual correlations without ground-truth audio.The evaluation therefore uses reference-free metrics: IS for audio quality, CLAP and IB-score for semantic alignment, and DeSync for synchronization.Source: G Evaluation on MovieGen-Audio-Bench · p. 20

Quantitative results

  • 8.95 IS and 0.77 DeSync show WavFlow’s strong audio quality and temporal synchronization on synthetic visual content.WavFlow outperforms or matches existing latent-based models on these dimensions.Source: G Evaluation on MovieGen-Audio-Bench · p. 20
  • MovieGen retains an advantage in IB-score, likely because of its substantially larger model capacity and training scale, while WavFlow remains highly competitive.This comparison concerns semantic richness rather than the reported audio-quality or synchronization metrics.Source: G Evaluation on MovieGen-Audio-Bench · p. 20

Qualitative synchronization

  • For Penguin Walking and Horse Trotting, WavFlow’s spectrograms show sharp vertical energy pulses that preserve fine-grained transients and correspond closely to footfalls.The supplied figure descriptions characterize WavFlow’s pulses as sharper or more defined than competing models in these scenarios.Source: Figure 8 · p. 21, Figure 10 · p. 22
  • In the Boxing scenario, WavFlow and MMAudio synchronize precisely, but WavFlow alone omits impact sound during air punches without bag contact.MovieGen exhibits desynchronization, while MMAudio and MovieGen incorrectly synthesize strike energy.Source: Figure 9 · p. 21

Related papers

Loading 2605.18749v1…