PaperlayerView exact revision on arXiv

Source-linked AI summary

EditVoice: Variable-Length Non-Autoregressive Zero-Shot TTS and Speech Editing with Edit Flows

Hongyao Deng, Wenhao Guan, Xuetao Lin, Peijie Chen, Weijie Wu, Lin Li, Qingyang Hong

arXiv:2609.29889v1cs.SD

TL;DR

Non-autoregressive TTS typically requires a target length before generation. EDITVOICE uses Edit Flows and speech-infilling training for variable-length generation, TTS, and editing, and reports competitive performance across both tasks.

  • Problem

    Non-autoregressive TTS generates in parallel but typically requires the target sequence length to be specified or predicted before generation.

    Source: §1 · p. 1

  • Method

    EDITVOICE combines variable-length Edit Flow generation with speech-infilling training and complementary prefix and suffix prompts for TTS and speech editing.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2

  • Results

    EDITVOICE reports competitive zero-shot TTS and speech-editing performance; versus CosyVoice 2, WER falls from 2.58 to 1.48 on Seed-TTS Eval EN and from 1.80 to 1.63 on LibriSpeech-PC.

    Source: §4.2 · p. 3, §5 · p. 4

  • Takeaways & Limitations

    Generalization to plausible speech tokens supports end-to-end editing without external edit-region localization and training-free post-generation refinement.

    Source: §1 · p. 1, §5 · p. 4

Abstract

from arXiv · show

Recent non-autoregressive (NAR) zero-shot text-to-speech (TTS) models generate in parallel but typically require the target sequence length to be specified before generation. We introduce EditVoice, to our knowledge the first variable-length NAR zero-shot TTS model, which uses Edit Flows to jointly update speech content and sequence length through insertions, deletions, and substitutions. EditVoice adopts speech-infilling training, which unifies zero-shot TTS and text-based speech editing and allows both prefix and suffix speech prompt placements at inference. We introduce Complementary Prompt Sampling (CPS) to leverage the complementary Edit Flow predictions induced by the two prompt placements. We further find that EditVoice can edit source and model-generated speech beyond its training sources. We use this generalization for end-to-end editing and training-free post-generation refinement. With the Edit Flow model trained on 10K h of GigaSpeech, EditVoice demonstrates competitive zero-shot TTS performance on Seed-TTS Eval EN and LibriSpeech-PC and speech editing performance on RealEdit.

1. INTRODUCTION

EDITVOICE addresses the fixed-length constraint of non-autoregressive TTS with variable-length Edit Flow generation. Speech-infilling training supports both TTS and localized editing, while complementary prompts and editing generalization add sampling and refinement capabilities.

1. INTRODUCTION

  • EDITVOICE applies Edit Flows to update speech content and sequence length through insertions, deletions, and substitutions, unlike NAR generation with a length specified or predicted beforehand.The introduction contrasts this with autoregressive decoding, which naturally produces variable-length sequences but commits to prior predictions that can propagate local errors.Source: §1 · p. 1
  • Random-span speech infilling lets one model perform zero-shot TTS and localized text-based speech editing by regenerating masked speech regions.Speech infilling is presented as a common formulation for TTS and editing using text and surrounding audio context.Source: §1 · p. 1
  • Complementary Prompt Sampling combines complementary predictions from prefix and suffix prompts, with local prompt consistency to reduce artifacts from switching between placements.The two placements are available for both zero-shot TTS and speech editing; their joint use was unexplored in prior infilling-based TTS.Source: §1 · p. 1
  • EDITVOICE can edit plausible speech tokens beyond its training sources, supporting end-to-end editing without external edit-region localization and training-free post-generation refinement.The capability is not explicitly trained, creating a training–inference mismatch that continued Edit Flow sampling is used to mitigate.Source: §1 · p. 1

2. BACKGROUND: EDIT FLOWS

Edit Flows model non-autoregressive generation on variable-length sequences using edits. Auxiliary alignments make conditional paths tractable by representing sequence changes in aligned coordinates.

2.1. Variable-Length Edit Process

  • Edit Flows model non-autoregressive generation over variable-length sequences, with a CTMC applying insertions, deletions, and substitutions through operation rates and token distributions.A rate over a sampling step maps to a finite-step event probability of p = 1 − exp(−hλ).Source: §2.1 · pp. 1–2

2.2. Training with Auxiliary Alignments

  • Because multiple edit sequences can connect the same endpoints, Edit Flows use an auxiliary blank-augmented aligned space and a mapping that removes blanks.This provides a representation for conditional paths between variable-length sequences.Source: §2.2 · p. 2
  • For aligned endpoints, the conditional path factorizes across coordinates; unresolved coordinates correspond to insertion, deletion, or substitution edits under the training objective.The objective uses β_t = κ̇_t/(1 − κ_t), with Ω(X_t) denoting valid edits and ω_j the edit corresponding to replacing an aligned coordinate.Source: §2.2 · p. 2

3. EDITVOICE

EditVoice uses speech infilling and Edit Flows for zero-shot TTS and speech editing with variable-length targets. It combines complementary prefix and suffix predictions and uses speech-token generalization for end-to-end editing and post-generation refinement.

3.1. Speech Infilling with Edit Flows

  • Random-span infilling aligns a 50-token initialization with target spans of varying lengths, using auxiliary blanks to mark insertions, deletions, and substitutions.Random span placement exposes context on either side of the generated region.Source: §3.1 · p. 2
  • For zero-shot TTS, EditVoice initializes the target with samples from p0, places the speech prompt before or after it, and allows insertions and deletions to change its length.The resulting semantic tokens are decoded by a frozen CosyVoice 2 acoustic decoder.Source: §3.1 · p. 2
  • Localized cascade editing replaces the transcript-diff region with initialized samples, keeps other speech fixed, and copies retained source Mel frames during decoding.A text diff and word-level forced alignment identify the edited region.Source: §3.1 · p. 2

3.2. Complementary Prompt Sampling

  • CPS selects the prefix or suffix placement with the higher operation rate and uses that placement’s token distribution for insertion and substitution.The same target state is evaluated with both prompt placements.Source: §3.2 · p. 2, Figure 2 · p. 3
  • Prompt switching across nearby edits can create inconsistent predictions and audible artifacts; after t > τ, conflicting tokens and immediate neighbors inherit the recorded placement.The delayed consistency rule leaves early sampling free to exploit complementary predictions.Source: §3.2 · pp. 2–3

3.3. End-to-End Editing and Post-Generation Refinement

  • Despite training sources initialized with random tokens, EditVoice can edit plausible speech tokens, enabling end-to-end editing without external edit-region localization.The source speech also serves as a fixed prompt for speaker conditioning while the model decides which source tokens to edit or retain.Source: §3.3 · p. 3
  • For post-generation refinement, EditVoice continues Edit Flow sampling for several steps at fixed final sampling time, using only the prefix prompt to correct residual content errors.The method revises model-generated tokens, which are not used as training sources.Source: §3.3 · p. 3

4. EXPERIMENTS

Experiments evaluate EditVoice on zero-shot TTS and speech editing benchmarks, then examine CPS and post-generation refinement. The results show competitive benchmark performance and improvements from both analysis techniques.

4.1. Experimental Setup

  • The Edit Flow model is trained for 230K steps on 10K h of GigaSpeech and evaluated for TTS on Seed-TTS Eval EN and LibriSpeech-PC, and editing on RealEdit.Speech editing uses 24 CPS generation steps followed by 8 refinement steps; zero-shot TTS includes 64/32/16-NFE settings.Source: §4.1 · p. 3

4.2. Zero-Shot TTS and Speech Editing

  • EDITVOICE achieves the lowest WER on Seed-TTS Eval EN and second-lowest on LibriSpeech-PC; it leads cascade WER and MOSNet and end-to-end SIM-o, MOSNet, and UTMOS.Against CosyVoice 2, WER falls from 2.58 to 1.48 on Seed-TTS Eval EN and from 1.80 to 1.63 on LibriSpeech-PC; end-to-end WER is second-best.Source: §4.2 · pp. 3–4, Table 1 · p. 4, Table 2 · p. 4
  • End-to-end editing without external localization preserves 31.5% of source Mel frames, compared with 76.7% for MFA-localized cascade editing.The end-to-end setting also achieves the highest SIM-o; compared systems regenerate the complete edited utterance.Source: §4.2 · p. 4

4.3. Ablation and Analysis

  • On Seed-TTS Eval EN, local prompt consistency lowers WER from 1.61% to 1.54% and raises UTMOS from 4.08 to 4.11.Combining complementary prompt predictions without local consistency reduces WER from 1.65–1.66% to 1.61%.Source: §4.3 · p. 4, Table 3 · p. 4
  • Post-generation refinement lowers WER from 2.66% to 1.55%; 8 generation steps plus 2 refinement steps outperform 16 generation steps without refinement.Refinement continues deleting or substituting model-generated tokens, which are not used as training sources; Fig. 3 measures WER and edits over 8 refinement steps.Source: §4.3 · p. 4, Figure 3 · p. 4

5. CONCLUSION

The conclusion presents EditVoice as a variable-length non-autoregressive zero-shot TTS model based on Edit Flows. It highlights speech-infilling, CPS, generalized editing and refinement, and competitive performance.

5. CONCLUSION

  • EditVoice is a variable-length non-autoregressive zero-shot TTS model based on Edit Flows.Experiments show competitive zero-shot TTS and speech-editing performance with efficient non-autoregressive generation.Source: §5 · p. 4
  • Speech-infilling training supports localized editing and two prompt placements whose complementary predictions are combined by CPS.Source: §5 · p. 4
  • Generalization of the learned editing capability supports end-to-end editing and training-free post-generation refinement.Source: §5 · p. 4

COMPLIANCE WITH ETHICAL STANDARDS

The study reports ethics approval, informed consent for subjective listening tests, and collection of anonymized ratings.

COMPLIANCE WITH ETHICAL STANDARDS

Related papers

Loading 2609.29889v1…