Source-linked AI summary
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, David Harwath
TL;DR
Speech editing and zero-shot TTS require models to generate speech in realistic settings while preserving or matching voice and context. VoiceCraft addresses both tasks with a token-infilling neural codec language model using Transformer decoding and token rearrangement. It achieves state-of-the-art performance across both tasks, while REALEDIT broadens speech-editing evaluation; limitations include occasional generation artifacts and unresolved safety risks.
Problem
Speech editing and zero-shot TTS must handle diverse real-world speech while preserving contextual naturalness and synthesizing unseen voices from limited reference audio.
Method
VoiceCraft uses a Transformer decoder and a token rearrangement procedure combining causal masking with delayed stacking for autoregressive codec-token infilling.
Results
VoiceCraft achieves state-of-the-art performance on speech editing and zero-shot TTS, with edited speech nearly indistinguishable from original recordings in human naturalness evaluations.
Takeaways & Limitations
REALEDIT provides a challenging, realistic benchmark for assessing the practicality of speech editing models on diverse in-the-wild recordings.
Takeaways & Limitations
VoiceCraft occasionally produces long silences and scratching sounds during generation, and voice cloning raises impersonation, fraud, and misinformation risks.
Abstract
from arXiv · showhide
We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALLE and the popular commercial model XTTS-v2. Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named RealEdit. We encourage readers to listen to the demos at https://jasonppy.github.io/VoiceCraft_web.
1 Introduction
VOICECRAFT is a Transformer-based neural codec language model for speech editing and zero-shot TTS, using token rearrangement to generate codec tokens with bidirectional context. It is evaluated on realistic, diverse speech data and achieves strong human-evaluated performance while introducing REALEDIT.
- 1 Introduction: VOICECRAFT uses causal masking and delayed stacking to enable efficient autoregressive neural codec generation with bidirectional context.The model combines these steps in a Transformer-based neural codec language model.
- 1 Introduction: REALEDIT contains 310 realistic speech-editing examples sourced from audiobooks, YouTube videos, and podcasts.Its examples span insertion, deletion, substitution, and multi-span editing, with edited text ranging from 1 to 16 words.
- 1 Introduction: REALEDIT covers diverse accents, speaking styles, recording conditions, and background sounds to assess real-world practicality.The dataset is designed to be more challenging and realistic than audiobook-only evaluation datasets.
- 1 Introduction: VOICECRAFT achieves state-of-the-art performance on both speech editing and zero-shot TTS in realistic, in-the-wild settings.The evaluations include audiobooks, YouTube videos, and podcasts, with comparisons against prior systems and real recordings.
- 1 Introduction: Human listeners found VOICECRAFT-edited speech nearly indistinguishable from original unedited recordings in naturalness.Listeners preferred VOICECRAFT-edited speech over the original recording 48% of the time in side-by-side comparison.
- 1 Introduction: VOICECRAFT generalizes to zero-shot TTS without finetuning and outperforms prior state-of-the-art systems including VALL-E and XTTS v2.The paper also releases VOICECRAFT code and model weights.
2 Related Work
Prior work treats speech editing and zero-shot TTS as related but often separate problems, while recent systems explore unified models with varying architectures and evaluation scope. VOICECRAFT positions itself as an end-to-end model evaluated on broader editing scenarios and with human assessment.
- 2 Related Work: Zero-shot TTS synthesizes speech in an unseen target voice from a target transcript and a short reference recording.Neural codec language models such as VALL-E and Spear-TTS frame the task as transcript-conditioned speech continuation.
- 2 Related Work: Speech editing changes words or phrases to match a target transcript while preserving non-targeted regions of the original recording.Earlier methods often concatenate generated segments, causing prosody mismatch and boundary artifacts when unedited context is not modeled.
- 2 Related Work: Recent research investigates unified systems for speech editing and zero-shot TTS, including modular models and end-to-end approaches.VOICECRAFT is described as end-to-end, whereas some concurrent systems use modular architectures or prompt tuning.
- 2 Related Work: VOICECRAFT differs from concurrent systems through broader evaluation, including human evaluation and editing spans of up to 16 words.The cited concurrent models either lack human evaluation, evaluate spans shorter than 2 seconds, or do not evaluate speech editing in the paper.
3 Method
VoiceCraft performs speech editing and zero-shot TTS by rearranging neural codec tokens for autoregressive generation with bidirectional context. Its Transformer decoder models the rearranged sequence across time and codebooks.
- Token rearrangement: VoiceCraft casts speech editing as sequence infilling and zero-shot TTS as continuation through neural codec token rearrangement.The procedure combines causal masking for bidirectional context with delayed stacking for efficient multi-codebook modeling.
- Token rearrangement: Causal masking moves masked spans to the sequence end, allowing autoregressive infilling to condition on both past and future unmasked tokens.Multiple non-overlapping masked spans can be moved to the end and marked with mask tokens.
- Token rearrangement: Delayed stacking shifts tokens according to codebook index so each timestep contains K tokens and later codebooks can use earlier-codebook predictions.Special [empty] tokens fill invalid positions, while mask tokens remain unchanged.
- Modeling: A Transformer decoder autoregressively predicts all K codebook tokens at each timestep using K MLP heads, conditioned on transcript W and prior rearranged tokens Hs,t.The model factorizes probability across time and codebooks, assuming the K RVQ codes at a timestep are conditionally independent.
- Modeling: The training objective is a weighted sum of codebook losses, with empirically greater weight assigned to earlier residual codebooks.Prediction loss is calculated on all tokens except mask and [empty] tokens.
- Inference: During speech-editing inference, forced alignment locates codec spans corresponding to transcript differences, while neighboring audio is included to model co-articulation.Zero-shot TTS is implemented as an insertion edit at the end of a voice-prompt utterance.
4 REALEDIT: a realistic and challenging speech editing dataset
REALEDIT is a manually constructed speech-editing benchmark designed to evaluate realistic edits across varied sources and editing conditions. It combines multiple edit types, span configurations, and lengths in 310 examples.
- Dataset construction: REALEDIT contains 310 manually crafted speech-editing examples drawn from LibriTTS, YouTube, and Spotify podcasts.Each example pairs original audio and transcript with a revised transcript.
- Dataset motivation: The benchmark is intended to support realistic evaluation of speech-editing models across diverse real-world source recordings and editing scenarios.
- Dataset design: The dataset covers one- or two-span edits involving insertion, deletion, and substitution, with short, medium, or long edited spans.For two-span editing, each example uses two of the three edit types.
- Dataset construction: REALEDIT revisions are manually checked and rewritten by native English speakers to remain accurate, grammatical, and semantically coherent.
5 Experiments
VoiceCraft is evaluated on realistic speech editing and zero-shot TTS settings using diverse data, human judgments, objective metrics, and multiple baselines. It performs strongly across both tasks, while results also expose metric disagreement and reduced naturalness for longer edits and noisy TTS inputs.
- 5.1 Setup: The zero-shot TTS evaluation uses 250 prompt-transcript pairs from LibriTTS and YouTube test data, with prompts kept close to 3 seconds.Target transcripts range from 8 to 40 words, and selected source utterances have Whisper WER below 15%.
- 5.1 Setup: VoiceCraft is compared with FluentSpeech for speech editing and with VALL-E, XTTS v2, YourTTS, and FluentSpeech for zero-shot TTS.The reproduced VALL-E and commercial XTTS v2 models are included among the zero-shot TTS baselines.
- 5.1 Setup: Human evaluations collect 10 ratings per item on 5-point scales for naturalness, intelligibility, and speaker similarity, with 64 and 59 Turkers for editing and TTS.Side-by-side comparisons report model preference percentages, while other evaluations report MOS with 95% confidence intervals.
- 5.3 Speech Editing Results: 56.4% of side-by-side judgments rate VoiceCraft edits equally or more natural than the original, while 75.9% rate them equal or more natural than FluentSpeech.VoiceCraft is preferred over FluentSpeech 56.1% of the time, with 19.7% ties; longer edit spans receive slightly lower naturalness judgments.
- 5.4 Zero-Shot TTS Results: VoiceCraft achieves the best zero-shot TTS automatic speaker-similarity metric and all human evaluation metrics, with intelligibility MOS 4.23 versus 4.39 for ground truth.The naturalness gap is larger on noisy YouTube utterances, and audiobook recordings receive lower naturalness than YouTube and Spotify originals.
6 Conclusion
VOICECRAFT achieves state-of-the-art performance on speech editing and zero-shot TTS using an innovative token rearrangement procedure. The paper also introduces REALEDIT as a realistic benchmark for speech editing practicality.
- VOICECRAFT achieves state-of-the-art performance on speech editing and zero-shot TTS using in-the-wild data.
- Its token rearrangement procedure enables efficient and effective autoregressive codec generation with bidirectional context.
- REALEDIT is introduced as a high-quality, challenging dataset intended to reliably measure speech-editing practicality.
7 Limitations
VOICECRAFT still has generation-quality and safety limitations. Occasional long silence and scratching sounds require sampling multiple utterances, while watermarking and detection remain important open challenges.
- Long silence and scratching sounds occasionally occur during generation, and the current workaround samples multiple utterances before selecting shorter ones.
- More elegant and efficient methods are needed to address these generation artifacts.
- Watermarking and detecting synthesized speech remain important safety challenges for models such as VOICECRAFT.
8 Ethical Implications
VOICECRAFT may support communication and content creation, but its deployment raises risks involving bias, voice-cloning misuse, impersonation, fraud, and misinformation. The authors advocate open collaboration and safeguards to mitigate these risks.
- VOICECRAFT could help people with speech impairments communicate and help content creators streamline speech editing.
- The model may exacerbate ethnic biases, causing unequal performance across groups and potentially worsening existing disparities.
- Cloning voices from only a few seconds of reference audio creates risks of impersonation, fraud, and misinformation.
- Open-sourcing the methods can support research into vulnerabilities, countermeasures, detection, and watermarking for synthetic speech.
A Additional Experiments
Additional experiments examine optimization and model-comparison behavior. Larger models improve all reported metrics, while VOICECRAFT outperforms FluentSpeech especially on substitution edits and longer edit spans, though the AdamW comparison is computationally limited.
- Optimizer comparison: ScaledAdam achieves better performance across all metrics while using less compute than the tested AdamW settings.
- Optimizer comparison: The AdamW comparison is limited because computational resources prevented an exhaustive hyperparameter search.
- Comparison breakdown: VOICECRAFT outperforms FluentSpeech across the board, especially for substitution edits and longer edit spans.
A.3 Spectrograms Comparison
Spectrogram comparisons show VoiceCraft producing more detailed frequency patterns than FluentSpeech across examples with increasing accent and recording-condition difficulty. The comparison also documents high-frequency noise in FluentSpeech’s unedited speech under challenging recording conditions.
- Spectrogram comparison: VoiceCraft generated more detailed frequency patterns than FluentSpeech in three examples spanning increasing accent and recording-condition difficulty.The examples included a low-bandwidth transmission condition.
- Optimization comparison: ScaledAdam consistently outperformed AdamW across all reported metrics while taking 10% less training time.This optimizer comparison is reported in Table 7.
- Human preference comparison: Figure 4 organizes side-by-side naturalness preferences between VoiceCraft and FluentSpeech by edit type and edit-span length.The figure provides two breakdowns of the human comparison.
- Spectrogram comparison: FluentSpeech’s full-mel-spectrogram resynthesis introduced high-frequency noise into unedited speech in a challenging recording condition.The figure description attributes this artifact to passing the entire mel-spectrogram to HiFi-GAN.
B Examples of the Speech Editing Dataset REALEDIT
Table 8 presents examples from the REALEDIT speech-editing dataset. The examples illustrate the dataset’s concrete editing cases.
- Dataset examples: Table 8 provides examples of the speech editing dataset REALEDIT.The table is presented as an example collection for the dataset.
C Implementational Details
The implementation details cover codec construction, optimization schedules, ablation data and metrics, baseline scaling, evaluation procedures, and human-test protocols. They also document assumptions and scope boundaries relevant to interpreting these experiments.
- Codec and model setup: The Encodec codec uses a 320-sample stride, yielding a 50Hz framerate at a 16kHz recording sample rate.Its base dimension is 64 and doubles across five encoder convolutional layers.
- Optimization: The Eden scheduler linearly warms its factor from αstart to 1 over twarmup steps before keeping it at 1.The schedule also uses αstep and αepoch to control when the learning rate is significantly reduced.
- Optimization: One pseudoepoch equals 3000 training steps because the dataset is large.The hyperparameter choices were inspired by prior work, and a grid search might improve them if resources permitted.
- Ablation studies: The ablation task masks a randomly selected span of 1–15 words and asks VoiceCraft to reconstruct the speech from the transcript and unmasked speech.Evaluation uses 1000 Gigaspeech validation utterances whose Whisper medium.en WER is below 15%.
- Ablation studies: Ablation metrics include WER, MCD, F0 distance, and energy distance, with WER and MCD more associated with intelligibility and F0 and Energy with prosody similarity.MCD compares MFCCs between generated and ground-truth recordings; F0 and energy use specified signal-processing procedures with dynamic time warping for alignment.
- Baseline setup: FluentSpeech was scaled to a 330M-parameter model trained on Gigaspeech for comparison.The scaled configuration increased batch size, hidden dimension, residual layers, and residual channels, and retrained HiFi-GAN.
- Baseline setup: Zero-shot TTS comparisons include VALL-E, XTTS v2, YourTTS, and FluentSpeech.The compared systems use different training data and architectures, and XTTS v2’s exact data sources are unknown.