PaperlayerView exact revision on arXiv

Source-linked AI summary

Audiobox: Unified Audio Generation with Natural Language Prompts

Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, Carleigh Wood, Joshua Lane, Mary Williamson, Wei-Ning Hsu

arXiv:2312.15821v1cs.SDcs.LGeess.AS

TL;DR

Audio generation remains difficult to control across modalities, styles, and transcripts while creation is time-consuming and expertise-intensive. Audiobox unifies speech and sound generation through flow-matching, description- and example-based prompts, and self-supervised pre-training. It reports broader controllability and benchmark-leading performance, while identifying limits in labeled data for description-based control.

  • Problem

    Existing audio generators are often modality-specific, lack fine-grained or transcript control, and require expertise-intensive, time-consuming audio creation.

  • Method

    Audiobox unifies speech and sound generation with flow-matching, description- and example-based prompting, and self-supervised infilling pre-training on unlabeled audio.

  • Results

    Audiobox outperforms prior in-context speech and sound-generation models on established benchmarks while generating speech and sound from descriptions, examples, or both.

  • Takeaways & Limitations

    Large quantities of data can support a unified audio model with unified controls that outperforms modality-specific models and enables diverse vocal and acoustic styles.

  • Takeaways & Limitations

    Description-based control is harder to generalize because labeled audio-description data covering varied concepts and granularities is scarce.

Abstract

from arXiv · show

Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like "a person speaking" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/

1 Introduction

Audiobox addresses limited controllability and modality coverage in audio generation by unifying speech and sound generation with multiple prompting methods and large-scale, variably supervised training. It reports broader capabilities, benchmark improvements, improved evaluation, faster inference, and responsible-generation studies.

  • Existing audio generators are mostly modality-specific, while real-world audio often mixes speech, music, and sound effects.
  • Speech and sound models use different control paradigms: precise example-based speech control versus novel-style description-based sound control.
  • Audiobox unifies sound and speech generation while accepting reference audio, text descriptions, or both to improve style controllability and generalization.
  • Audiobox supports holistic natural-language style control and disentangled vocal control, while outperforming domain-specific models on multiple tasks.
  • Joint-CLAP improves description-based speech retrieval over CLAP and correlates more strongly with human text-audio similarity judgments.
  • Bespoke Solver speeds flow-matching inference by 25x over the adaptive-step dopri5 solver without performance loss on several tasks.

2 Related Work

Prior audio generators expanded modality-specific capabilities but remained constrained in controllability, transcript control, and unified modeling. Related work spans large-scale speech models, natural-language style prompting, general-domain sound and music generation, and emerging unified systems.

  • Large-scale in-context text-to-speech generative models: Large-scale speech models use transcript-guided continuation or infilling and audio context to control voice, prosody, emotion, environment, channel, and noise.
  • Large-scale in-context text-to-speech generative models: In-context speech control cannot use descriptive text or independently control voice and attributes such as emotion and acoustic condition.
  • Natural language style prompting for controllable speech generation: Natural-language speech prompting studies sought flexible control but often used small datasets, few speakers, or limited categorical attribute combinations.
  • Large scale general-domain models for sound and music generation: General-domain sound and music models generate complex audio from holistic descriptions but typically cannot control transcripts or produce intelligible speech.
  • Unified model for audio generation: Unified systems explored text-to-music, text-to-sound, in-context TTS, and natural-language style-prompted TTS using shared models.
  • Unified model for audio generation: VoiceLDM combines transcript conditioning with CLAP-based style control but inherits coarse speech attributes, fixed-length audio, and weaker sound-generation performance.

3 Background

Audiobox builds on flow-matching models and self-supervised infilling to model continuous audio, condition generation on transcript and masked context, and learn from unlabeled speech. These foundations support scalable speech generation and versatile infilling-based control.

  • Conditional flow-matching: Conditional flow-matching models continuous paths transforming samples from a prior distribution into samples from the data distribution.
  • Conditional flow-matching: During flow-matching training, the model predicts a noisy data derivative from a sampled flow step and noisy data point.
  • Conditional flow-matching: Optimal-transport paths use constant directions and speeds, enabling more accurate ODE estimation with fewer steps than general paths.
  • Voicebox: Voicebox conditions audio prediction on frame-aligned phonetic transcripts and masked audio, representing audio as Mel spectrograms converted to waveforms by HiFi-GAN.
  • Voicebox: Voicebox uses 60K hours of audiobooks and supports in-context TTS, noise removal, style generation, speech editing, and cross-lingual style transfer.
  • SpeechFlow: SpeechFlow removes transcript conditioning and learns speech infilling from unlabeled audio, addressing label scarcity while modeling temporal correlations within utterances.

4 Audiobox SSL: Self-supervised Generative Audio Pre-training

Audiobox SSL is a self-supervised foundation model designed to generalize across speech, music, and sound using large-scale unlabeled audio. It adapts infilling-based pre-training and predicts audio latent features for broader non-speech coverage.

  • Audiobox SSL is a foundation model that can be fine-tuned for downstream audio generation tasks.
  • The model is pre-trained with an infilling objective on unlabeled speech, music, and sound effects, avoiding reliance on transcripts, captions, or attribute labels.
  • Audiobox predicts dense Encodec latent features rather than spectrograms to improve resynthesis for sound and music.
  • The 330M-parameter Transformer is trained for 1 million updates with masked audio conditioning and flow-matching loss computed on masked frames.
  • The training dataset contains over 160K hours of speech, 20K hours of music, and 6K hours of sound samples spanning diverse domains and speakers.

5 Audiobox Speech: Scaling In-context Text-to-speech Synthesis

Audiobox Speech fine-tunes Audiobox SSL for transcript-guided in-context TTS, combining audio-style prompting with transcript control and scalable speech training. It improves style transfer across domains while maintaining or improving content correctness and quality.

  • Audiobox Speech fine-tunes Audiobox SSL with transcript-guided speech infilling for multiple downstream speech-generation tasks.
  • In-context TTS uses an audio example to control style while the target transcript specifies generated content.
  • Audiobox Speech evaluates style similarity, content correctness, and quality using embedding similarity, WER, and subjective judgments.
  • 0.745 style similarity is achieved on the Librispeech audiobook test set versus 0.710 from UniAudio.
  • Audiobox Speech improves Voicebox across other domains, with similarity gains ranging from 0.096 to 0.156, while subjective tests report better style transfer and audio quality.
  • Scaling pre-training data improves style similarity especially on domains absent from fine-tuning, while full fine-tuning reduces WER by 23% to 43% on most domains.

6 Audiobox Sound: Simple Text-to-sound Generation and Infilling

Audiobox Sound applies flow matching to text-guided sound generation and infilling, using scalable pre-training and staged fine-tuning to improve quality, efficiency, and controllability.

  • Model design: Audiobox Sound directly models audio in an autoencoding-based latent representation rather than relying on an off-the-shelf variational auto-encoder.The approach streamlines the architecture and reduces trainable parameters during fine-tuning.
  • Training: The model is trained in two stages using noisy tag data followed by scarce high-quality caption data.The first-stage model initializes the subsequent model, allowing larger-scale tagged data to support text-conditional learning.
  • Evaluation: Audiobox Sound evaluates text-to-sound generation and text-guided audio infilling on AudioCaps using objective and subjective quality metrics.Infilling conditions on 30% of the ground-truth audio and fills the middle 70%.
  • Results: Audiobox Sound consistently outperforms prior works in text-to-audio generation by a large margin while using fewer parameters and less sound training data.The result highlights the contribution of general-domain pre-training to sound generation.
  • Results: 0.61→0.77 CLAP-aa similarity improves when context is supplied for text-to-audio infilling, indicating greater coherence with the reference audio.FAD and KLD show mixed trends across the compared infilling conditions.
  • Efficiency: Audiobox Sound achieves consistently lower FAD than AudioLDM2 and Tango at the same number of inference steps.This quality-speed comparison supports more efficient inference for the flow-matching approach.
  • Ablations: ∼20% FAD improvement follows from general pre-training, while CLAP reranking changes FAD from 0.91 to 0.78 and CLAP score from 0.60 to 0.71.The ablation also reports that two-stage fine-tuning consistently outperforms using caption data alone.
  • Ablations: LoRA fine-tuning is 15% to 30% worse relative to full-model fine-tuning.The authors associate this gap with the architectural change introduced by cross-attention layers.

7 Audiobox: Toward Universal and Controllable Audio Generation

Audiobox unifies speech and sound generation under text and audio-example conditioning, enabling richer control over speech attributes and acoustic scenes. Joint-CLAP supports evaluation of description matching, while the unified model improves performance across universal audio tasks.

  • Unified model: Audiobox is a single model that generates speech and audio conditioned on text descriptions or audio examples.The unified model combines capabilities previously discussed separately for speech and sound generation.
  • Controllability: Description-based control bridges speech and sound datasets, complementing example-based control with natural-language prompts.The paper uses both human annotations and automatically created captions to support speech descriptions.
  • Controllability: Speech captions represent age, gender, audio quality, pitch, speaking rate, accent, emotion, and environment.The captions are generated from extracted or pseudo-labeled attributes using an LLM.
  • Speech generation: Audiobox supports description-guided TTS and description-guided TTS with additional voice conditioning.These tasks combine transcript control with textual style descriptions and, when available, a voice prompt.
  • Evaluation: Joint-CLAP improves speech retrieval over public CLAP models, with T2A@10 reaching 22.01 versus 12.01 for an in-domain CLAP variant.The model uses a speech encoder and is designed for text-audio similarity in speech settings.
  • Speech results: Audiobox outperforms VoiceLDM and AudioLDM2 across description-guided TTS datasets and metrics, including complicated expressive and accented scenarios.The reported datasets include background events, expressive speech, and accented speech with diverse backgrounds.
  • Speech results: Voice prompts greatly improve Audiobox speaker similarity while preserving description matching.Without a voice prompt, speaker similarity is greatly reduced because descriptions do not capture all aspects of voice.
  • Unified-model comparison: The unified Audiobox model gives higher speaker similarity but marginally worse word error rate than Audiobox Speech on zero-shot TTS.Subjective evaluations report only minor differences between the two models.

8 Inference Optimization with Bespoke Solver

Bespoke Solvers optimize flow-matching inference by learning a transformed trajectory that can be accurately estimated with very few ODE steps. Experiments show substantial step reduction while retaining speech-generation performance.

  • Efficiency objective: Inference efficiency is measured by NFE, the number of times an ODE solver evaluates the derivative.The method targets the extreme low-NFE regime, particularly four evaluations.
  • Solver design: Bespoke Solvers learn a transformed sample trajectory that improves accuracy at a desired low number of ODE steps.The transformation uses time reparameterization and an invertible function parameterized by extra solver parameters.
  • Zero-shot TTS evaluation: 4 ODE steps retain similar style similarity and WER to standard solver settings for zero-shot TTS.Ground-truth paths were generated with dopri5 using 200 steps and guidance weight 0.7.
  • Zero-shot TTS evaluation: Bespoke Solvers generalize to a different guidance setup, including evaluation with guidance weight 0.0.The comparison used a standard midpoint solver and the Bespoke Solver.

9 Responsible AI

The paper evaluates responsible deployment through demographic-group performance analysis and watermark robustness testing. Results indicate similar speech performance across gender and accent groups, with watermark detection remaining robust under augmentation.

  • Fairness: The responsible-AI evaluation measures gender and accent differences using WER and style similarity.Bias is assessed by comparing group performance with the overall population.
  • Fairness: The zero-shot TTS fairness experiment evaluates 64 accent groups using sampled transcripts and utterances from male and female speakers.The experiment uses country and gender labels and samples 20 transcripts.
  • Fairness: WER and speaker similarity remain similar across gender and accent groups, with group speaker-similarity means within ±1 standard deviation.The reported figures cover both gender-group and accent-group comparisons.
  • Watermarking: Watermark detection is evaluated across zero-shot TTS, description-based TTS, voice-plus-description TTS, and sound generation under audio augmentations.Performance is measured using false-positive and false-negative rates.
  • Watermarking: FPR and FNR are close to 0%, while the watermarked audio has SI-SNR of -20.6 dB.The results indicate robust detection and an imperceptible watermark residual from a human perspective.

10 Discussion

The discussion links controllability to training-data coverage and supervision, especially for fine-grained description-based control. It also describes applications, responsible-use risks, and evidence of similar performance across demographic groups.

  • Fine-grained Control: Controllability depends on training-data domain coverage and quantity, with self-supervised pre-training improving style similarity for in-context TTS.In-context TTS uses example-based control and audio infilling from context.
  • Fine-grained Control: Description-based control is harder to generalize because paired labeled data covering diverse concepts and granularities is scarce.Fine-grained text descriptions must align language concepts with variations in audio.
  • Fine-grained Control: If training captions collapse distinct species into “a dog barking,” prompting “a chihuahua barking” may not produce the correct species-specific sound.The same limitation applies to regional accents without paired accent examples.
  • Data creation: Fine-grained audio description datasets are difficult to create because some attributes are hard to identify and annotators may disagree.The passage specifically mentions species, accents, emotion, perceived age, and audio quality.
  • Applications: Audiobox supports applications requiring vocal and acoustic-style control, including immersive audiobooks, advertising, film scripts, and varied environments.The discussion highlights preserving voice while changing emotion or acoustic scenes for long-form content.
  • Broader impact: Speech synthesis carries misinformation and impersonation risks, while the paper reports watermarking as a mitigation approach and similar performance across demographic groups.The demographic result is attributed to evaluation across groups and data scaling.

11 Conclusion

Audiobox is presented as a unified system for speech and sound generation with controls from text, audio examples, or both. The paper reports fine-grained style control and performance exceeding prior in-context speech and sound-generation models.

  • Conclusion: Audiobox generates speech and sound from text descriptions, audio examples, or a combination of vocal references and descriptions.Speech controls include transcript, vocal style, and other audio styles.
  • Conclusion: The model controls fine-grained speech styles such as accent, emotion, and timbre while simulating diverse environments.These capabilities extend the range of vocal and acoustic conditions represented by the model.
  • Conclusion: Audiobox outperforms prior in-context speech-generation and sound-generation models on established benchmarks.The conclusion frames the result as applying to evaluations of existing capabilities.
  • Conclusion: The paper argues that large quantities of data can enable a unified audio model to outperform modality-specific models.This conclusion is presented as a direction for future audio-generative-model research.

Contribution

The project contribution spans speech and sound data preparation, annotation, model implementation, experiments, evaluation, and demo development. It also includes speech-attribute extraction and a two-stage process for filtering annotators and captions.

  • The team prepared speech and sound data and developed Audiobox, including audio infilling for baselines and duration-model experiments.
  • The project included human annotation, subjective evaluation, Joint-CLAP experiments, two-stage fine-tuning, and studies on evaluation.
  • Contributors also implemented responsible-AI studies, demo infrastructure, backend and frontend systems, deployment, content filtering, and mitigation verification.
  • Speech descriptions use extracted or metadata-based attributes including age, gender, audio quality, pitch, speaking rate, accent, emotion, and environment.
  • Annotator selection combines gold-sample accuracy above 73% with LLM-based scoring of submitted captions.

C Unified Audiobox Task Description

Unified Audiobox supports multiple audio-generation tasks by combining transcripts or pseudo-transcripts with examples, descriptions, masked contexts, and voice prompts. These inputs enable controllable synthesis, sampling, and editing for speech and sound.

  • Zero-shot TTS uses a transcript and an audio example to generate speech resembling the example’s audio style.
  • Description-TTS and description-TTA generate speech or audio from a transcript or pseudo-transcript plus a text description.
  • Voice restylization combines a transcript, voice prompt, and description to match vocal style while changing specified attributes such as emotion or environment.
  • Sampling generates diverse voices from a transcript alone.
  • Speech and audio infilling or editing reconstruct masked portions using context, transcript or pseudo-transcript, and an optional or required description.

D Subjective Evaluation Interface

The subjective evaluation interface includes separate evaluations for sound and speech, covering OVL, REL, quality MOS, and similarity MOS.

  • Sound: Sound evaluations include OVL and REL interfaces.
  • Speech: Speech evaluations include quality MOS, similarity MOS, and REL interfaces.
Loading 2312.15821v1…