PaperlayerView exact revision on arXiv

Source-linked AI summary

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

Xueyao Zhang, Xiaohui Zhang, Kainan Peng, Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, Yuan Huang, Zhizheng Wu, Mingbo Ma

arXiv:2502.07243v1cs.SDcs.AIeess.AS

TL;DR

Existing voice-imitation methods depend heavily on annotated data and do not reliably disentangle timbre and style for controllable zero-shot generation. Vevo combines self-supervised disentangled tokens with separate style-conditioned content-style modeling and timbre-conditioned acoustic modeling, matching or surpassing existing methods in accent and emotion conversion while generalizing to voice conversion and TTS.

  • Problem

    Existing methods rely heavily on annotated data and struggle to disentangle timbre and style for controllable generation, especially in zero-shot scenarios.

  • Method

    Vevo uses self-supervised VQ-VAE tokenizers over HuBERT features with codebook-size information bottlenecks, followed by autoregressive content-style modeling and flow-matching acoustic modeling.

  • Results

    Pre-trained on 60K hours of audiobook speech without style-specific fine-tuning, Vevo matches or surpasses existing methods in accent and emotion conversion and performs effectively in voice conversion and TTS.

  • Takeaways & Limitations

    Vevo provides a unified framework for versatile zero-shot voice imitation with controllable timbre and style across conversion and synthesis tasks.

Abstract

from arXiv · show

The imitation of voice, targeted on specific speech attributes such as timbre and speaking style, is crucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, we propose Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (1) Content-Style Modeling: Given either text or speech's content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Modeling: Given the content-style tokens as input, we employ a flow-matching transformer to produce acoustic representations, which is prompted by a timbre reference. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE as the tokenizer for the continuous hidden features of HuBERT. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained on 60K hours of audiobook speech data, without any fine-tuning on style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion conversion tasks. Additionally, Vevo's effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility. Audio samples are available at https://versavoice.github.io.

1 INTRODUCTION

Vevo frames speech imitation as separately controllable linguistic content, style, and timbre, then addresses zero-shot generation with a self-supervised two-stage framework. Trained on 60K hours of audiobook speech without style-specific fine-tuning, it matches or surpasses existing methods in accent and emotion conversion and generalizes to voice conversion and TTS.

  • Motivation: Vevo decomposes speech imitation into linguistic content, style, and timbre, defining timbre, style, and voice imitation tasks around which attributes are preserved or transferred.Timbre imitation preserves content and style; style imitation preserves content and timbre; voice imitation transfers timbre and style while preserving content.
  • Motivation: Existing style-imitation methods rely heavily on parallel corpora, style labels, and textual transcriptions, making supervision difficult to collect and scale.The stated limitation is especially relevant to zero-shot imitation of accents, emotions, and other speaking styles.
  • Vevo Framework: Vevo uses Content-Style Modeling with an autoregressive transformer and Acoustic Modeling with a flow-matching transformer, prompted separately by style and timbre references.The first stage generates content-style tokens from content tokens or text; the second produces acoustic representations from those tokens.
  • Vevo Framework: A fully self-supervised tokenizer progressively decouples timbre, style, and linguistic content by quantizing HuBERT features with VQ-VAE and adjusting codebook vocabulary size.The codebook vocabulary is treated as an information bottleneck for filtering speech information.
  • Results: Pre-trained on 60K hours of audiobook speech without style-specific fine-tuning, Vevo matches or surpasses existing methods in accent and emotion conversion and works for voice conversion and TTS.Figure 1 describes speech or text input with content controlled by the source and style and timbre controlled by separate references in one forward pass.

2 RELATED WORK

Related work on controllable voice imitation and disentangled speech representations uses conversion corpora, auxiliary supervision, architecture-based bottlenecks, and representation-learning methods.

  • Controllable Voice Imitation: Voice-conversion research commonly imitates timbre through speaker-agnostic representations such as PPG or self-supervised features combined with generative models.The cited approaches include GANs and autoencoders, among other model families.
  • Disentangled Speech Representation: Disentangled speech representations separate linguistic content, timbre, and style using auxiliary-task distillation, information-bottleneck architectures, and vector quantization.These approaches target different speech attributes through supervision or constrained representation capacity.
  • Disentangled Speech Representation: Vevo’s representation-learning strategy is situated within prior work that seeks to decouple the same speech factors for controllable generation.Its stated motivation is to use VQ-VAE to disentangle high-level information within speech.

3 METHODOLOGY

Vevo progressively disentangles speech attributes with a VQ-VAE bottleneck, then uses separate transformers to control style and timbre during zero-shot generation.

  • 3.1 VQ-VAE TOKENIZER FOR HUBERT: VQ-VAE tokenizes HuBERT features while using codebook vocabulary size K as an information bottleneck to progressively filter speech information.The method first removes timbre, then style-related information, while retaining linguistic content at an intermediate bottleneck.
  • 3.2 CONTENT-STYLE MODELING (CONTENT TO CONTENT-STYLE): The content-style stage uses an autoregressive transformer to convert content tokens or text into content-style tokens prompted by a style reference.Inference can combine a global style embedding with the reference content-style tokens, strengthening style guidance.
  • 3.2 CONTENT-STYLE MODELING (CONTENT TO CONTENT-STYLE): Duration Reduction merges consecutive duplicate content-token units before style modeling, shortening sequences and filtering unit-level duration information.The reduced sequence also helps the transformer handle the length change between content and content-style representations.
  • 3.3 ACOUSTIC MODELING (CONTENT-STYLE TO ACOUSTIC): The acoustic stage uses a flow-matching transformer to transform content-style tokens into Mel spectrograms under a timbre reference.Training masks temporal spans of a Mel spectrogram, while inference conditions generation on the reference Mel spectrogram and concatenated content-style tokens.
  • 3.4 ZERO-SHOT SPEECH GENERATION TASKS: Vevo combines its pretrained stages into separate timbre, style, and voice variants by changing only the inference pipeline and reference conditions.The variants use different combinations of source content-style tokens, reduced content tokens, style references, and timbre references.

4 EXPERIMENTS

Vevo’s experiments examine self-supervised token disentanglement and evaluate zero-shot timbre, voice, style, and synthesis imitation across audiobook and other speech data.

  • Experimental setup: Vevo trains English-only models on 60K hours of ASR-transcribed audiobook speech, with some tokenizer training using a 100-hour subset.Macoustic and Mstyle use speech data; the text-input variant also uses transcriptions.
  • Experimental setup: The evaluation includes clean audiobook speech and noisier Common Voice recordings spanning broader accents and recording conditions.Objective metrics include WER, speaker similarity, accent similarity, emotion similarity, and F0 correlation.
  • Self-supervised disentanglement: HuBERT representations retain timbre, style, and linguistic content, while ASR fine-tuning reduces timbre and style information.Continuous HuBERT features show high source speaker similarity, high F0 correlation, and low WER; ASR-derived representations retain linguistic content but reduce the other attributes.
  • Self-supervised disentanglement: VQ-VAE vocabulary size controls disentanglement: K=4096 filters much timbre while retaining style, whereas K=32 filters most style as well.The selected tokenizers use Kc=32 for content and Ks=4096 for content-style representations.
  • Zero-shot conversion: Vevo-Timbre outperforms four baselines across intelligibility, speaker similarity, quality, and style-consistency metrics in zero-shot timbre imitation.Vevo-Voice additionally excels at speaker, accent, and emotion imitation, while Vevo-Timbre better preserves source style and intelligibility.
  • Zero-shot style imitation: Without fine-tuning on accented or emotional corpora, Vevo-Style surpasses baselines on intelligibility, quality, and accent- and emotion-imitation metrics.Vevo-Style (ASR) further improves intelligibility and some accent-imitation aspects using text as additional supervision.
  • Zero-shot synthesis: Vevo-TTS exceeds Voicebox on metrics other than WER and reaches ES-MOS 4.03, surpassing CosyVoice and MaskGCT on some emotion tasks.The comparison with Voicebox uses identical training data; the authors attribute the style performance to content-style tokens.
  • Efficiency ablations: Reference-global-guided continuation reduces the sequence length to 42% of Vevo-Voice with only a slight decline in performance metrics.The ablation identifies potential savings in inference memory and speed.

5 CONCLUSION

Vevo frames speech as linguistic content, timbre, and style, and combines controllable zero-shot imitation with self-supervised disentanglement and scalable modeling.

  • The framework uses content-style modeling with an autoregressive transformer and acoustic modeling with a flow-matching transformer.
  • Vevo’s content and content-style tokens are produced by HuBERT-based VQ-VAE tokenizers with carefully adjusted vocabulary sizes.
  • Trained on 60K hours of audiobook speech without style-specific fine-tuning, Vevo outperforms state-of-the-art models in zero-shot accent and emotion conversion.
  • Vevo also performs robustly in zero-shot voice conversion and text-to-speech, supporting its versatility across voice imitation tasks.
  • Vevo decomposes speech into linguistic content, timbre, and style, corresponding respectively to what is spoken, who speaks, and how it is spoken.

B.1 VQ-VAE ARCHITECTURE

The implementation uses VQ-VAE tokenizers and transformer-based modeling stages, with reference-guided continuation and flow-matching acoustic generation.

  • The content and content-style VQ-VAE tokenizers use vocabulary sizes of 32 and 4096, with 59M and 63M parameters, respectively.
  • The tokenizer architecture is an encoder-decoder VQ-VAE whose symbols include kernel size, stride, channel counts, vocabulary size, and hidden dimension.
  • The content-style model uses a LLaMA-like 12-layer autoregressive transformer and a WavLM- and TDNN-based global style encoder.
  • During inference, content-style generation supports default reference-style-enhanced continuation or reference-global-guided continuation.
  • The text-input variant replaces duration-reduced content tokens with text tokens while retaining Mstyle’s training and inference hyperparameters.
  • Acoustic modeling randomly masks 70%-100% of frames and uses a 24-layer flow-matching transformer with midpoint ODE sampling.
  • BigVGAN converts 24-kHz, 100-band Mel spectrograms into waveforms and is fine-tuned using the 60K-hour training data.
  • Classifier-free guidance drops conditioning during training and uses puncond=0.2 and α=0.7 during inference.

C.1 ZERO-SHOT TIMBRE IMITATION AND VOICE IMITATION (CONVERSION TASK) •

The conversion-task comparison covers diverse systems and highlights Vevo’s zero-shot style-imitation evaluation setup against methods trained with varying supervision and data.

  • HierSpeech++ uses MMS content features, a VITS-based architecture, and 2.8K hours of Libri-light and LibriTTS data.
  • LM-VC predicts SoundStream codecs from HuBERT-like soft units with an autoregressive hierarchical transformer trained on Libri-light.
  • UniAudio predicts acoustic codecs from 500-cluster HuBERT-base K-means tokens using approximately 80K hours of speech and 20K hours of other audio.
  • FACodec factorizes waveform attributes with an autoencoder and residual vector quantization, using ASR, F0 prediction, and speaker classification tasks.
  • ASR-AC reconstructs waveforms from ASR text predictions, accent labels, and F0, trained on about 700 hours of accented speech.
  • VoiceShop learns BNF mappings from a parallel conversion corpus of about 300 hours after ASR-based feature extraction.
  • Conv-Speak reduces parallel-data requirements through self-supervised pretraining and trains conversion on about 600 hours, including about 1 hour of parallel data.
  • Emovox uses emotion-agnostic recognition features and emotion labels to reconstruct Mel spectrograms, with training data including 80 hours from VCTK and ESD.

C.3 ZERO-SHOT VOICE IMITATION (SYNTHESIS TASK) •

The synthesis-task comparison spans autoregressive, flow-matching, and diffusion-based systems trained on datasets ranging from 10K to 171K hours and beyond.

  • VALL-E is an autoregressive zero-shot TTS model that predicts EnCodec codecs from 45K hours of MLS English speech.
  • Voicebox applies flow-matching transformers to both duration modeling and acoustic modeling.
  • VoiceCraft predicts EnCodec codecs with an autoregressive transformer and uses token rearrangement and delayed stacking strategies.
  • CosyVoice uses an ASR-supervised semantic tokenizer, an autoregressive text-to-semantic model, and a flow-matching Mel-spectrogram model.
  • MaskGCT uses two-stage discrete diffusion models based on semantic tokens extracted from w2v-bert 2.0 features.

D ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments examine how tokenizer vocabulary size affects phonetic discriminability and how content/style codebook sizes affect Vevo-Voice. Results indicate that linguistic-content representation stabilizes for VQ-VAE beyond vocabulary size 1024, while reducing content or style capacity harms performance.

  • D.1.1 Effect on Phonetic Discriminability: Beyond vocabulary size 1024, additional VQ-VAE capacity is likely allocated to style information such as F0 rather than linguistic content.Style information continues increasing from vocabulary sizes 1024 to 4096 to 16384, as indicated by rising FPC scores.
  • D.1.1 Effect on Phonetic Discriminability: K-means phonetic discriminability continuously improves as vocabulary size increases from 32 to 1024.This contrasts with the convergence observed for VQ-VAE tokens beyond vocabulary size 1024.
  • D.1.1 Effect on Phonetic Discriminability: VQ-VAE tokens are less sensitive than K-means tokens to vocabulary-size changes in their linguistic-content representation.For example, VQ-VAE (32) and K-means (128) have nearly identical ABX error rates.
  • D.1.2 Effect of Different (Kc, Ks): Reducing Kc from 32 to 16 causes a significant drop in intelligibility.The result indicates that a smaller content-tokenizer vocabulary loses linguistic-content information.

D.2 ZERO-SHOT VOICE IMITATION (SYNTHESIS TASK)

Vevo-TTS is evaluated on zero-shot imitation synthesis across four evaluation sets. It outperforms several autoregressive baselines on many metrics, shows advantages over CosyVoice on selected metrics, but trails non-autoregressive baselines in WER.

  • D.2 Zero-Shot Voice Imitation (Synthesis Task): Vevo-TTS clearly outperforms VALL-E and VoiceCraft across various metrics on all four zero-shot imitation datasets.The evaluation sets are AB, CV, ACCENT, and EMOTION.
  • D.2 Zero-Shot Voice Imitation (Synthesis Task): Vevo-TTS exceeds CosyVoice on selected Naturalness CMOS and Speaker S-MOS metrics despite training solely on 60K hours of audiobook data.The reported advantages include Naturalness CMOS on AB, ACCENT, and EMOTION, and Speaker S-MOS on EMOTION.
  • D.2 Zero-Shot Voice Imitation (Synthesis Task): Vevo-TTS notably performs better on style-imitation metrics including Accent S-MOS and Emotion S-MOS than CosyVoice.The paper attributes this result to the autoregressive TTS model using the proposed content-style tokens.
  • D.2 Zero-Shot Voice Imitation (Synthesis Task): Vevo-TTS has higher WER than Voicebox and MaskGCT across all datasets.The comparison indicates remaining stability challenges for autoregressive models.

E SUBJECTIVE EVALUATION

Subjective evaluations were conducted by experienced paid assessors familiar with common English accents. Each audio sample received at least ten ratings.

  • E Subjective Evaluation: Dozens of paid subjects with extensive AI-audio assessment experience conducted the subjective evaluations.Assessors had lived in English-speaking countries for extended periods.
  • E Subjective Evaluation: The assessors were familiar with American, British, Hindi, and Mandarin English accents.This familiarity supported evaluation across the common accent types used in the study.
  • E Subjective Evaluation: Each audio sample was rated at least ten times.The repeated ratings provide the basis for the reported subjective evaluations.

E.2 METRICS AND QUESTIONNAIRES

The subjective-evaluation appendix defines interfaces, questionnaires, and scoring criteria for naturalness, speaker similarity, accent, emotion, prosody, and comparative quality measures. It also documents the study’s stated misuse risks and planned mitigation.

  • E.2 Metrics and Questionnaires: The automated evaluation interface presents the audio, questionnaire, and scoring criteria for each item.These components structure the subjective assessment workflow.
  • E.2.1 Naturalness MOS: Naturalness MOS asks whether the clip sounds human-like and uses a 1–5 scale from very poor artificial speech to perfect human-speech imitation.The evaluated system audio is presented with target text.
  • E.2.2 Speaker Similarity MOS: Speaker Similarity MOS compares the evaluated voice with a reference while instructing raters to ignore content and audio quality.Scores range from 1 for a completely different person to 5 for exactly the same person.
  • E.2.3 Accent Similarity MOS: Accent Similarity MOS asks raters to ignore vocal characteristics and judge whether the evaluated accent matches the reference.The scoring scale runs from 1 for mostly different accent to 5 for exactly the same accent.
  • E.2.4 Emotion Similarity MOS: Emotion Similarity MOS similarly isolates emotion from vocal characteristics when comparing the evaluated audio with a reference.The scale ranges from 1 for mostly different emotion to 5 for exactly the same emotion.
  • E.2.5 Prosody Similarity MOS: Prosody Similarity MOS evaluates whether pace, tone, stress, intonation, pitch, and emotion match the reference speaking style.Raters use a 1–5 scale from completely different style to completely identical style.
  • E.2.6 Naturalness CMOS: Naturalness CMOS asks whether the evaluated audio is better or worse than the reference in quality and human likeness.The comparative scale spans −3, much worse, to 3, much better, with 0 meaning no preference.
  • E.2.7 Accentedness CMOS / E.2.8 Emotiveness CMOS: Accentedness CMOS and Emotiveness CMOS compare generated accentedness or emotional expressiveness against a reference using labeled prompts and the same −3 to 3 scale.The prompts specify a target accent or emotion label before the comparison.
Loading 2502.07243v1…