PaperlayerView exact revision on arXiv

Source-linked AI summary

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, Wei-Ning Hsu

arXiv:2306.15687v2eess.AScs.CLcs.LGcs.SD

TL;DR

Speech generative models have remained limited in scale and task generalization compared with large-scale generative models. Voicebox trains a non-autoregressive flow-matching model for text-guided speech infilling with audio context, enabling multiple speech-generation tasks. It reports state-of-the-art results across mono- and cross-lingual TTS and related tasks, with inference up to 20 times faster than autoregressive models.

  • Problem

    Speech generative models are generally trained at smaller scale and have limited task generalization, while in-the-wild speech data has posed quality challenges.

  • Method

    Voicebox is a non-autoregressive continuous normalizing flow trained with flow-matching to infill masked speech from surrounding audio and transcript.

  • Results

    Voicebox achieves state-of-the-art performance on mono- and cross-lingual zero-shot TTS, speech denoising, speech inpainting, and diverse speech sampling, with inference up to 20 times faster than autoregressive models.

  • Takeaways & Limitations

    A single text-guided infilling approach can support diverse speech-generation tasks through in-context learning without audio style labels.

  • Takeaways & Limitations

    The models are trained on read audiobook speech in up to six written languages and may not transfer well to conversational speech.

Abstract

from arXiv · show

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

1 Introduction

Speech generative models have lagged behind large-scale general-purpose models in scale and task generalization. Voicebox addresses this gap with scalable text-guided speech infilling and reports broad task coverage with strong benchmark results.

  • Motivation: Large-scale generative models generalize to tasks not explicitly trained on, whereas speech models remain limited in scale and task generalization.Prior speech systems commonly use datasets ranging from tens to hundreds of hours.
  • Motivation: Curated speech datasets provide limited variation, and models trained on in-the-wild data have shown substantial quality degradation.The limitations include variation in emotion, voice, background noise, acoustic condition, and text.
  • Approach: Voicebox learns text-guided speech infilling from surrounding audio and transcript, using context to infer style without audio style labels.The approach is presented as guided in-context learning and is intended to subsume common speech generation tasks.
  • Approach: A non-autoregressive continuous normalizing flow trained with flow-matching can condition on both past and future audio context and trade inference quality against runtime.The number of flow steps is controllable at inference time.
  • Results: 5.9% → 1.9% WER and 0.580 → 0.681 audio similarity mark Voicebox’s English zero-shot TTS result against VALL-E.The paper reports Voicebox as outperforming VALL-E on both metrics.
  • Results: Voicebox reports cross-lingual TTS, text-guided denoising, and synthetic-speech data creation results, alongside diverse speech generation.Compared with YourTTS, average WER changes from 10.9% to 5.2% and audio similarity from 0.335 to 0.481; denoising gains include -8.8% WER, +0.450 similarity, and +0.80 mean opinion score.

2 Related Work

Prior speech generators are often task-specific, style-constrained, or limited in their ability to model varied speech. Voicebox’s infilling formulation and flow-based design target broader in-context generation than these approaches.

  • Task-specific generation: Many speech generative models are task-specific, including voice conversion, emotion conversion, and speech enhancement systems.These tasks commonly convert a specific audio attribute while preserving others.
  • Style control: Earlier controllable TTS systems use labels or embeddings for style, but low-dimensional style representations can limit realistic noisy-speech generation.Performance can degrade when such models are conditioned on noisy references.
  • Infilling: Infilling models use surrounding speech as context for in-context learning, but prior text-guided systems often assume deterministic mappings and typically infill segments up to 1 second.Voicebox instead uses a continuous normalizing flow that can parameterize any distribution and infill speech of any length.
  • Large-scale in-context learning: Textless speech language models do not use text conditioning, and generated speech may fail to preserve the prompt’s voice.AudioLM adds a cascaded codec-based approach to preserve style, but these models are evaluated on spoken language modeling tasks.
  • Related zero-shot TTS: VALL-E performs zero-shot TTS through text-conditioned autoregressive and non-autoregressive modeling of Encodec tokens and uses a speech prompt for target style.Its autoregressive generation contrasts with Voicebox’s ability to use future context and its faster flow-matching inference.
  • Concurrent work: NaturalSpeech2 uses latent diffusion with predicted pitch and a speech prompt, whereas Voicebox predicts Mel spectrograms and supports diverse speech sampling.Voicebox uses 16 ODE steps, while NaturalSpeech2 uses 150 diffusion steps, according to the comparison passage.

3 Method

Voicebox formulates text-guided speech infilling as conditional in-context generation and implements it with flow-matching continuous normalizing flows. Its audio and duration models support flexible conditioning and efficient ODE-based inference for multiple speech-generation tasks.

  • 3.1 Background: Flow Matching with an optimal transport path: CNFs transform a simple prior distribution into a data distribution through a learned time-dependent vector field.Inference samples x0 from the prior and solves an initial-value ODE to obtain the generated sample.
  • 3.1 Background: Flow Matching with an optimal transport path: Flow Matching trains the vector field by regressing toward a target field, while Conditional Flow Matching provides an unbiased gradient estimator using tractable conditional paths.The paper compares paths and objectives and reports superior performance and efficiency for CFM with an optimal-transport path.
  • 3.2 Problem formulation: Voicebox learns p(xmis | y, xctx), predicting masked speech from the transcript and surrounding unmasked audio.The binary temporal mask separates missing speech xmis from context xctx, turning many tasks into text-guided in-context learning.
  • 3.2 Problem formulation: The model decouples generation into an audio model and a duration model to provide fine-grained alignment control between speech and text.Phone durations are expanded into a frame-level phone transcript, while masked audio and duration segments are modeled conditionally.
  • 3.3 Model and Training: The audio model represents speech as 80-dimensional log-Mel spectrograms and uses a Transformer-conditioned CNF over flow samples, audio context, and phone embeddings.The Transformer parameterizes the conditional vector field from frame-level inputs and the flow step.
  • 3.4 Inference: Less than 10 function evaluations empirically produce very high-quality speech, with a user-controlled speed–accuracy trade-off.More evaluations can improve ODE accuracy but increase runtime; Voicebox also demonstrates tasks not explicitly trained on through constructed context.

4 Metrics

The paper uses reproducible model-based metrics to evaluate correctness, coherence, diversity, and quality in speech generation. It supplements these automatic measures with subjective ratings while noting that WER is not a complete quality measure.

  • Metrics: Model-based perceptual metrics are proposed to make speech-generation comparisons more reproducible than subjective metrics alone.The paper motivates this approach because MOS ratings are difficult to compare across studies.
  • Correctness and intelligibility: WER measures correctness and intelligibility by comparing synthesized-speech transcriptions with the input text using public ASR models.The setup uses HuBERT-L for English-only evaluation.
  • Correctness and intelligibility: Lower WER indicates greater ASR-recognized intelligibility and content correctness, but does not necessarily indicate better overall speech quality.Expressive, noisy, or diverse speech may receive higher WER despite not being poor samples.
  • Coherence: Coherence is measured by similarity between generated-speech and audio-context embeddings, including WavLM-TDCNN speaker embeddings for comparison with VALL-E.Different embedding models reflect coherence of different attributes.
  • Diversity and quality: FID-style distance evaluates diversity and quality by comparing generated and real-sample distributions in a feature space.A shorter distance indicates more similar distributions and generally reflects higher sample quality.
  • Diversity and quality: QMOS and SMOS provide supplementary subjective ratings of audio quality and similarity on a 1-to-5 scale.Fifty samples per system receive ten ratings each, with averaged ratings and 95% confidence intervals reported.

5 Experiment

Voicebox is evaluated across multilingual TTS, denoising, prompt conditioning, diversity, speed, and flow-matching design choices. It generally outperforms prior systems while supporting flexible context conditioning and efficient inference.

  • Experimental setup: Voicebox trains English and multilingual models on 60K and 50K audiobook hours, respectively, covering six languages.The multilingual languages are English, French, German, Spanish, Polish, and Portuguese.
  • Cross-lingual zero-shot TTS: Voicebox outperforms YourTTS across supported cross-lingual directions, with lower WERs and higher audio similarity.On English, French, and Portuguese, WER is 3.1%, 5.9%, and 8.1% lower, while similarity is 0.136, 0.141, and 0.160 higher.
  • Cross-lingual zero-shot TTS: 0.59 higher audio-similarity MOS and 0.27 higher quality MOS distinguish Voicebox from YourTTS in multilingual subjective evaluations.The reported scores are 3.89 vs 3.30 for similarity and 3.50 vs 3.23 for quality.
  • Transient noise removal: 2.0% WER, 0.612 SIM-o, and 3.87 MOS show Voicebox produces intelligible, similar, and high-quality speech for transient-noise removal.The comparison includes A3T and Demucs; Voicebox uses transcript and noise-location information.
  • Diverse speech sampling and ASR data generation: Voicebox generates diverse speech whose distribution is close to the training data, with the FM duration model producing more varied speaking styles.The FM duration model also leads to a better ASR system when its synthetic speech is used for training.
  • Diverse speech sampling and ASR data generation: 0.4% and 1.7% absolute WER gaps behind real data show that Voicebox-generated speech can train ASR systems effectively.These gaps correspond to Librispeech test-other and test-clean, respectively.
  • Efficiency and controllability: 0.31 seconds for a 10-second sample makes Voicebox about 20 times faster than VALL-E at NFE=2 without classifier-free guidance.At NFE=64, Voicebox is only 4% slower than VALL-E.
  • Ablation on generative modeling approaches: 8 NFEs suffice for good results with flow matching using optimal transport, outperforming alternatives in training and inference efficiency.FM w/ OT achieves its best performance with 100K training steps and outperforms SM w/ diff with 50K updates.

6 Ethical Statement

The ethical statement highlights risks from generating speech in arbitrary people’s styles and evaluates whether generated speech can be detected. Detection is easiest against original audio and harder against vocoder-resynthesized audio.

  • Risk: Voicebox’s ability to generate speech in arbitrary people’s styles creates potential risks recognized by the authors.The paper addresses this risk through a real-versus-generated speech detection study.
  • Detection study: A convolutional binary classifier is trained to distinguish real and generated speech using controlled Librispeech-based splits and masked-phoneme synthesis.The study uses a 100-hour subset of English training data and masks 90%, 50%, or 30% of phonemes.
  • Detection results: Detection is trivial for original versus Voicebox-generated audio because vocoder artifacts also separate original from resynthesized audio.Distinguishing Voicebox-generated audio from resynthesized audio is substantially harder.

7 Conclusion and Discussion

Voicebox generalizes across speech-generation tasks through large-scale text-guided infilling, while its current scope and controllability remain limited by training data, preprocessing, and prompting.

  • Voicebox achieves state-of-the-art performance across monolingual and cross-lingual zero-shot TTS, speech inpainting, and diverse speech sampling, with generation up to 20 times faster than autoregressive models.
  • Voicebox models are trained on read audiobook speech in up to six written languages, so they may not transfer well to conversational speech.Conversational speech contains more casual behavior and non-verbal sounds such as laughter and back-channeling.
  • Voicebox depends on a phonemizer and forced aligner to produce frame-level phonetic transcripts.
  • Word-based phonemizers may mispredict pronunciation because pronunciation is context-dependent in languages such as French.
  • Voicebox transfers voice, speaking style, emotion, and acoustic condition but does not independently control each attribute.The paper leaves disentangled control through prompting or text description for future work.
  • Phone representation: The system uses ghost silence and word-position-dependent phones to represent silence and word boundaries for duration and audio modeling.Ghost silences have zero duration, while phone postfixes mark beginning, end, intermediate, or singleton positions.

A.4 Cross-lingual zero-shot TTS test data filtering

The cross-lingual TTS test sets are filtered using transcription quality and alignment success before evaluation and are paired with standardized model-training procedures.

  • Test data filtering: Cross-lingual test samples are retained when Whisper transcription WER is below 20%, or below 30% for Polish and Portuguese.The higher threshold is used because those splits contain fewer than 1K samples.
  • Test data filtering: A small number of utterances are excluded because of MFA alignment failure.
  • ASR setup: The ASR model uses 80-dimensional log Mel features, a Conformer-based encoder, SpecAugment, and 120 training epochs unless otherwise noted.
  • Model configuration: The symmetric bidirectional ALiBi bias assigns distance-based representations and improves convergence and extrapolation to longer sequences in the experiments.

B.1 Comparing audio model training objectives

A controlled ablation compares regression and flow-matching objectives and tests whether computing loss only on masked frames improves speech generation.

  • Experimental setup: The ablation uses 1K hours of English audiobook speech, a reduced model, 150k training steps, and evaluation on cross-sentence zero-shot TTS and diverse sampling.
  • Objective comparison: Regression audio models produce comparable WER but significantly worse audio similarity and diversity than the flow-matching objective.Subjective listening also finds substantially worse audio quality and audio similarity for regression models.
  • Masked loss: Masked loss improves audio similarity and diversity while having little impact on intelligibility.

B.2 Effectiveness on data scaling

Scaling the English training data consistently improves zero-shot TTS and diverse-sampling WER, although FSD worsens at the largest scale under the reported reference distribution.

  • Scaling results: Scaling data consistently improves zero-shot TTS WER and SIM-r, as well as WER on diverse speech sampling.
  • Scaling results: FSD regresses when scaling from 6K hours to 60K hours, possibly because its reference distribution comes from less diverse 1K-hour audiobook data.
  • Experimental setup: The study evaluates models trained on 0.1%, 1%, 10%, and 100% subsets of the 60K-hour English dataset.

B.3 Complete results on comparing generative modeling approaches

This section evaluates flow-matching choices, noise removal, reconstruction components, and FSD-based measures of speech diversity and quality. It finds advantages for OT-path flow matching, Voicebox under varied noise conditions, and wav2vec 2.0 features for diversity measurement.

  • Generative modeling approaches: Flow Matching with OT paths performs strictly better than diffusion-path flow matching and score matching across all evaluated settings.The comparison covers training steps, inference steps, monolingual zero-shot TTS, and diverse speech sampling.
  • Transient noise removal: Voicebox produces the most intelligible audio across all tested noise levels, overlap ratios, and noise types.For audio similarity, it is better than Demucs in high noise, with gains of 0.265–0.324, and comparable in low noise.
  • Audio reconstruction: Audio reconstruction depends jointly on information retained by the acoustic encoding and the vocoder’s ability to translate features back into waveform.The study compares Mel spectrogram with HiFi-GAN or Parallel WaveGAN, and Encodec dense features with its decoder.
  • Speech diversity and quality: All wav2vec 2.0 layers show the desired FSD pattern: utterance sampling stays low, while speaker sampling rises sharply as sample size and speaker diversity decrease.Layer 6 is selected as the default FSD feature.
  • Speech diversity and quality: WavLM-TDCNN and wav2vec 2.0 layer 6 show similar low-variance diversity trends, whereas FAD fails to track decreasing speaker diversity reliably.Wav2vec 2.0 is selected because its self-supervised features may capture holistic speech information, including prosody and emotion.
  • Duration metrics: MS-MAE measures masked duration error per utterance, while FDD compares the means and variances of sampled and training duration distributions.These metrics evaluate duration correctness and duration diversity or quality, respectively.

C.3 Duration model evaluation with standalone metrics

The standalone evaluation compares flow-matching and regression duration models on unconditional TTS and second-half duration infilling. Conditioning on duration context improves correlation, while flow matching better captures distributional behavior.

  • Models: Three duration variants are evaluated: conditional flow matching, conditional regression, and unconditional regression.The unconditional model ignores duration context and uses only the phonetic transcript.
  • Tasks: The evaluation covers unconditional TTS and infilling the second half of each utterance’s durations on Librispeech test-other.The infilling task distinguishes models that use duration context from the unconditional regression model.
  • Prediction quality: Conditional regression performs slightly better overall on MS-MAE, while unconditional regression has substantially lower Phn-MS-Corr than the other models.The ground-truth Phn-MS-Corr is 0.47, and the result indicates that conditioning on duration context is beneficial.
  • Distributional quality and diversity: Flow matching shows the largest distinction from regression on distributional comparisons of duration samples.The standalone metrics include MS-MAE, MS-Corr, and distribution-level duration comparisons.

C.4 Duration model evaluation with end-to-end metrics

End-to-end evaluation compares the duration variants on zero-shot TTS and diverse speech generation. Duration-conditional regression lowers WER substantially, while flow matching preserves greater duration diversity.

  • Evaluation: End-to-end tests cover zero-shot TTS in cross-sentence and continuation settings, plus diverse speech generation.The appendix results use the VB-En-1K flow-matching audio model and are not directly comparable with the main text.
  • Results: FSD and SIM are similar across duration variants, but WER is substantially lower for duration-conditional regression.Subjective listening indicates that regression produces more regular speech patterns that may be easier for ASR to recognize.
  • Trade-off: Duration-conditional regression improves ASR-oriented regularity while sacrificing some duration diversity relative to flow matching.The reported trade-off concerns speech regularity and duration diversity, not FSD or SIM.

D Detailed Configurations for Acoustic and Duration model training

The appendix provides detailed configurations for the acoustic models and conditional flow-matching duration models used in the experiments.

  • Acoustic model training: Table D13 lists detailed configurations for the acoustic models used in the experiments.
  • Duration model training: Table D14 lists detailed configurations for conditional flow-matching duration models; regression models use the same configurations with regression loss.
Loading 2306.15687v2…