Woosh: A Sound Effects Foundation Model

Gaëtan Hadjeres; Marc Ferras; Khaled Koutini; Benno Weck; Alexandre Bittar; Thomas Hummel; Zineb Lahrichi; Hakim Missoum; Joan Serrà; Yuki Mitsufuji

Source-linked AI summary

Woosh: A Sound Effects Foundation Model

Gaëtan Hadjeres, Marc Ferras, Khaled Koutini, Benno Weck, Alexandre Bittar, Thomas Hummel, Zineb Lahrichi, Hakim Missoum, Joan Serrà, Yuki Mitsufuji

arXiv:2604.01929v3Apr 2, 2026cs.SDcs.AIcs.LG

TL;DR

Open generative audio research needs reusable models with accessible weights and strong sound-effect quality. Woosh addresses this with a sound-effect foundation-model pipeline covering encoding, text-audio alignment, text-to-audio, and video-to-audio generation, including distilled variants. Across public and private evaluations, the authors report competitive or better performance against open alternatives, with professional-library training particularly advantageous on private data.

Problem
Many generative audio approaches lack open weights, while some open alternatives use audio sampling rates up to 16 kHz rather than targeting high-quality sound effects.
Method
Woosh releases modular audio encoding/decoding, text-audio alignment, text-to-audio, and video-to-audio models, including distilled diffusion variants for faster inference.
Results
Woosh achieves competitive or better performance than StableAudio-Open and TangoFlux across evaluated modules, with private professional-library models outperforming public-data models on private sound-effect data.
Takeaways & Limitations
The release provides open research infrastructure for developing sound-effect generation methods and downstream audio applications.
Takeaways & Limitations
Woosh-AE is monaural, and comparisons must account for its larger model and substantially lower compression ratio than SAO-VAE.

Abstract

from arXiv · show

The audio research community depends on open generative models as foundational tools for building novel approaches and establishing baselines. In this report, we present Woosh, Sony AI's publicly released sound effect foundation model, detailing its architecture, training process, and an evaluation against other popular open models. Being optimized for sound effects, we provide (1) a high-quality audio encoder/decoder model and (2) a text-audio alignment model for conditioning, together with (3) text-to-audio and (4) video-to-audio generative models. Distilled text-to-audio and video-to-audio models are also included in the release, allowing for low-resource operation and fast inference. Our evaluation on both public and private data shows competitive or better performance for each module when compared to existing open alternatives like StableAudio-Open and TangoFlux. Inference code and model weights are available at https://github.com/SonyResearch/Woosh. Demo samples can be found at https://sonyresearch.github.io/Woosh/.

1 Introduction

Woosh is Sony AI’s publicly released foundation model specialized for high-quality sound-effect generation, addressing gaps in open, high-quality audio models. Its release provides a modular pipeline spanning audio representation, text conditioning, and text-to-audio and video-to-audio generation.

Motivation and contribution: Unlike many existing approaches, Woosh emphasizes open research access while operating beyond low audio sampling rates and specializing in sound effects.The report contrasts limited open-weight availability and 16 kHz ceilings with higher-quality alternatives covering general audio and music.
Motivation and contribution: Woosh targets instantaneous, high-quality sound-effect generation and provides inference code and open weights for non-commercial research.The full released pipeline includes encoder/decoder, text-conditioning, and diffusion models.
Release components: The public release includes four modules covering audio encoding/decoding, text-audio alignment, text-to-audio generation, and video-to-audio generation.These components are released as Woosh-AE, Woosh-CLAP, Woosh-Flow/Woosh-DFlow, and Woosh-VFlow/Woosh-DVFlow.
Release components: Woosh-AE provides high-quality latent representations for generative modeling and decodes generated latents back into audio.Woosh-CLAP provides token latents for diffusion conditioning or CLAP scoring.
Generation models: Woosh-Flow and its distilled counterpart generate audio unconditionally or from text prompts, while Woosh-VFlow generates audio from video with optional text prompts.Figure 1 presents their inference-time layouts for text-to-audio and video-to-audio generation.

2 Woosh-AE: Audio Encoder/Decoder

Woosh-AE is a monaural, VOCOS-based audio encoder/decoder that represents audio in continuous latents and reconstructs it through STFT-domain processing. Trained with multi-scale spectral, adversarial, and feature-matching objectives, it outperforms the compared baselines on AudioCaps and InternalSFX reconstruction metrics.

Architecture: Woosh-AE uses a GAN-based VOCOS vocoder over STFT complex coefficients without quantization, with efficient one-step down/up sampling through STFT/iSTFT.The architecture is monaural, and iSTFT is used to avoid aliasing artifacts typically associated with transposed-convolution upsampling.
Architecture: The decoder predicts magnitude and real/imaginary components before forming complex STFT coefficients, an alternative that improved reconstruction and discrimination metrics in preliminary experiments.This approach avoids phase-wrapping issues by using predicted real and imaginary components rather than phase directly.
Training: Woosh-AE training combines multi-scale mel-spectral loss, adversarial loss, and discriminator feature-matching loss, with final weighting L = 15Lspec + Ladv + 2LFM.Spectral losses use multiple STFT window sizes and mel resolutions, while adversarial training uses multi-period and multi-band multi-scale complex-STFT discriminators.
Evaluation: 85% lower MelDist (0.032 vs. 0.217) and 23% lower STFTDist (1.18 vs. 1.55) are reported for Woosh-AE-Public versus SAO-VAE on AudioCaps.On AudioCaps, Woosh-AE-Public reaches 20.79 dB SI-SDR and Woosh-AE-Private reaches 22.52 dB, compared with 9.69 dB for Descript.
Evaluation: On InternalSFX, Woosh-AE-Public and Woosh-AE-Private remain comparable, with SI-SDR values of 12.7 dB and 16.03 dB, respectively.The lower scores than on AudioCaps suggest that InternalSFX is more demanding, particularly for transients and high-frequency content.

3 Woosh-CLAP: Text Conditioning

Woosh-CLAP aligns text and audio embeddings in a shared space for text conditioning, using pretrained RoBERTa-Large and PaSST encoders with projected embeddings. Its retrieval performance depends strongly on the evaluation domain and training data.

Architecture: Woosh-CLAP uses text and audio encoders trained contrastively so matching pairs approach each other while mismatched pairs separate in a shared embedding space.The text encoder supplies conditioning during generation.
Architecture: RoBERTa-Large encodes text and PaSST encodes audio, with linear heads projecting pooled embeddings into a shared 1024-dimensional space.RoBERTa-Large has 355M parameters, while PaSST has 86M parameters.
Training: The symmetric contrastive loss uses positive pairs on the similarity-matrix diagonal and all text-audio pairs in the denominators.Training used an effective batch size of N = 448 and a fixed temperature of τ = 0.2.
Evaluation: On the public test set, LAION-CLAP outperformed Woosh-CLAP-Public by up to 13% in T2A-R@10 (0.546 vs. 0.618).The authors attribute this gap to differences in training datasets and data volume.
Evaluation: On the independent internal studio-quality test set, Woosh-CLAP-Private exceeded LAION-CLAP by up to 248% in T2A-R@10 (0.655 vs. 0.188) and over two times in A2T-R@10 (0.766 vs. 0.328).The results emphasize the importance of commercial audio libraries for professional applications.

4 Woosh-Flow: Text-to-Audio Generation

Woosh-Flow is a text-conditioned latent diffusion model for audio, using multimodal transformer blocks to condition noisy audio latents on text. It is trained with flow matching and distilled with MeanFlow to reduce inference cost while preserving audio quality, then evaluated for fidelity and text-audio alignment.

Architecture: Woosh-Flow applies latent diffusion to audio latents encoded by Woosh-AE and uses multimodal transformer blocks for text conditioning.The architecture handles text and noise-latent modalities with MultiStream and SingleStream blocks.
Architecture: MultiStream blocks process each modality independently, whereas SingleStream blocks attend across time-concatenated modality sequences for implicit cross-modal conditioning.Both block types use rotary positional embeddings.
Training: Training uses a flow-matching objective in which the network estimates the velocity from noisy intermediate latents toward the data-to-noise path.The model is conditioned on the noisy sample and its timestep and learns to approximate the target velocity x1 − x0.
Distillation: MeanFlow distillation reduces inference from 100 to 4 neural function evaluations with negligible audio quality loss.The distilled model is called Woosh-DFlow.
Evaluation: Evaluation compares audio fidelity using FD and KL with cross-modal alignment using LAION-CLAP against StableAudio-Open and TangoFlux.FD measures perceptual distance, KL compares acoustic and semantic content, and CLAP measures text-audio relevance.
Evaluation: On AudioCaps, Woosh-Flow-Public improves FD over TangoFlux and SAO by 17% and 27%, while improving CLAP by 6% and 150%, respectively.On InternalSFX, Woosh-Flow-Private has 27% lower FD than SAO and 34% lower FD than TangoFlux.

5 Woosh-VFlow: Video-to-Audio Generation

Woosh-VFlow extends text-to-audio generation to video conditioning through a dedicated video encoder interface and multimodal transformer, trained on audio-visual data with synthetic captions. Distillation reduces video-to-audio inference from 100 to 4 NFEs, while evaluation reports strong quality but cautions that synchronization metrics may not reflect perception.

Architecture: Video features become condition tokens through SynchFormer extraction at 24 Hz and linear projection to the diffusion transformer’s hidden dimension.Variable-duration videos use validity masking for unconditioned positions.
Architecture: Woosh-VFlow adds video-specific query, key, value projections and a video FFN while jointly attending over text, video, and audio/noise sequences.The modality-specific representations are concatenated for joint self-attention before separate feed-forward processing.
Data and training: Training uses VGGSound and OGameData250k, with Qwen3-Omni-generated captions addressing weak or visually focused original annotations.FoleyBench supplies 5000 curated non-speech, non-music audiovisual instances as the primary fidelity and alignment benchmark.
Distillation: 100 to 4 NFEs: Woosh-DVFlow distillation accelerates video-to-audio inference while maintaining high-fidelity audio generation.The distillation uses teacher predictions and classifier-free-guidance distillation, with training adapted to three-modality data.
Evaluation: Synchronization metrics are unreliable in this setting: SynchFormer can score generated audio above ground truth despite manual inspection finding no perceptual sync issues.The report therefore presents the metric with a caveat and encourages qualitative assessment.
Evaluation: Woosh-VFlow achieves lower FD and KL divergence than the smaller-parameter-comparable baseline on FoleyBench in text-conditioned and video-only settings.The evaluation compares Woosh models with MMAudio-M and reports audio quality, semantic alignment, cross-modal alignment, and synchronization metrics.

6 Future Work and Applications

The initial Woosh foundation models are presented as a basis for downstream sound-generation applications. Proposed extensions target finer control, editing, personalization, morphing, and looping.

Future Work and Applications: Future creative controls could condition generation on precise time-varying acoustic attributes such as loudness or spectral descriptions.Suggested mechanisms include cross-attention and adaptive layer normalization.
Future Work and Applications: Existing sounds could be varied, inpainted, personalized, morphed, or looped for applications such as game-footstep generation.The proposed techniques include latent corruption, masked reconstruction, few-shot fine-tuning, target-prompt sampling, and looping noise latents.

7 Conclusion

Woosh is a publicly released foundation-model suite for high-quality sound-effect generation, mainly covering text-to-audio and video-to-audio. Its results show that both model choice and domain-matched data matter for professional sound-effect applications.

Conclusion: Woosh models trained on public and private datasets achieve competitive or better performance than other popular open-source models on a public dataset.The conclusion frames this result across the released sound-effect generation models.
Conclusion: On private data, Woosh models trained on professional sound-effect libraries outperform models trained on public datasets in Fréchet Distance.The conclusion identifies domain shift in audio properties and textual annotations between public and commercial datasets.

Appendix A Distillation Pseudocode

The distillation pseudocode alternates discriminator and generator training around noisy data, teacher targets, and MeanFlow loss. The appendix presents this procedure as the training process for Woosh-DFlow and Woosh-DVFlow.

Appendix A Distillation Pseudocode: Algorithm 1 distills Woosh-DFlow and Woosh-DVFlow using the MeanFlow criterion together with latent adversarial diffusion distillation.The procedure is explicitly titled “Distillation Training with Adversarial Loss.”
Appendix A Distillation Pseudocode: Each iteration samples clean data and Gaussian noise, then constructs a noisy input by interpolating between them.The pseudocode samples x0 from pdata and x1 from N(0, I).
Appendix A Distillation Pseudocode: After 5000 steps, the discriminator is unfrozen to distinguish detached student predictions from ground-truth noisy samples.The discriminator is later frozen during generator training.
Appendix A Distillation Pseudocode: The student predicts a noisy-state update, while the ground-truth path is formed from the interpolation endpoint and scheduler variable.The pseudocode separately defines xfake_r from the student and xtrue_r from x0 and x1.
Appendix A Distillation Pseudocode: Generator training uses a detached teacher target and student prediction before applying the MeanFlow loss.This alternation combines teacher-guided distillation with adversarial training.

Appendix B Additional Results

Woosh-VFlow and Woosh-DVFlow are evaluated on VGGSound under three conditioning setups, with text quality substantially affecting generated-audio performance.

Text Conditioning Quality: Higher-quality Qwen3-Omni captions better reflect complex auditory scenes than the dataset’s noisy original labels.The synthetic captions help mitigate inaccurate or isolated-event descriptions in the original labels.
Text Conditioning Quality: Improved text conditioning guides outputs toward greater semantic and temporal alignment with the video.The comparison across conditioning scenarios demonstrates the role of caption quality in final audio generation.