PaperlayerView exact revision on arXiv

Source-linked AI summary

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

Chenpeng Du, Yiwei Guo, Feiyu Shen, Zhijun Liu, Zheng Liang, Xie Chen, Shuai Wang, Hui Zhang, Kai Yu

arXiv:2306.07547v6cs.SDeess.AS

TL;DR

Discrete-token TTS systems face limits in speech editing, acoustic-token prediction complexity, and codec-dependent audio quality. UniCATS combines contextual semantic-token prediction with acoustic-context-aware vocoding; the authors report state-of-the-art performance in continuation and editing, and improved semantic-token resynthesis over HifiGAN and AudioLM.

  • Problem

    Existing discrete-token TTS models are often left-to-right autoregressive systems unsuited to editing with preceding and following context, while acoustic tokens add RVQ prediction complexity and codec-limited audio quality.

    Source: §1 · p. 2

  • Method

    UniCATS combines contextual VQ-diffusion in CTX-txt2vec to predict semantic tokens with acoustic-context-aware vocoding in CTX-vec2wav.

    Source: §1 · p. 2, CTX-vec2wav with Contextual Vocoding · p. 4

  • Results

    UniCATS achieves state-of-the-art performance in speech continuation and editing, while CTX-vec2wav outperforms HifiGAN and AudioLM in semantic-token resynthesis.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    UniCATS eliminates acoustic tokens and speaker embeddings, using contextual VQ-diffusion and vocoding to incorporate semantic and acoustic context.

    Source: §4 · p. 7

    Longer mel-spectrogram prompts may improve speaker similarity, but this possibility is left for future work.

    Source: CTX-vec2wav with Contextual Vocoding · p. 5

Abstract

from arXiv · show

The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.

1 Introduction

Discrete-token TTS offers an alternative to mel-spectrogram pipelines, but existing systems face limits in bidirectional editing, acoustic-token prediction complexity, and codec-dependent audio quality. UniCATS uses contextual semantic-token prediction and vocoding for both continuation and editing, with state-of-the-art performance reported on both tasks.

1 Introduction

  • Semantic tokens primarily capture articulation but provide limited acoustic detail, whereas codec-based acoustic tokens capture details such as speaker identity.Discrete tokens have been reported to provide greater naturalness and robustness than mel-spectrograms for TTS.Source: §1 · p. 1
  • Speech editing synthesizes input text while maintaining smooth concatenation with surrounding speech, using both preceding context A and following context B.Speech continuation differs by not taking both contexts into account.Source: Figure 1 · p. 1, §1 · p. 2
  • Most discrete-token TTS models generate left to right, making them unsuitable for editing with preceding and following context; acoustic tokens also add RVQ prediction complexity and codec-limited quality.VALL-E uses a non-autoregressive module for residual indices, while SPEAR-TTS pads RVQ indices into a longer sequence.Source: §1 · p. 2
  • UniCATS achieves state-of-the-art performance in both speech continuation and speech editing.CTX-vec2wav also outperforms HifiGAN and AudioLM for resynthesis from semantic tokens.Source: §1 · p. 2
  • CTX-txt2vec predicts semantic tokens with contextual VQ-diffusion, and CTX-vec2wav vocodes them using acoustic context, especially speaker identity.The two tasks share the same model; speech editing additionally uses context B.Source: §1 · p. 2

2 UniCATS

UniCATS is designed for both speech continuation and speech editing, with an acoustic model and a vocoder as its two components.

2 UniCATS

  • UniCATS is designed to address both speech continuation and speech editing.It comprises an acoustic model, CTX-txt2vec, and a vocoder, CTX-vec2wav.Source: §2 · p. 3
  • CTX-txt2vec is the framework's acoustic model, and CTX-vec2wav is its vocoder.Source: §2 · p. 3
  • The paper describes the two UniCATS components in the following sections.Source: §2 · p. 3

CTX-txt2vec with Contextual VQ-Diffusion

CTX-txt2vec predicts semantic tokens with VQ-diffusion, conditioning generation on text and surrounding semantic context. Its training and inference support context on both sides, context on only the left, or no context.

Contextual VQ-Diffusion.

VQ-Diffusion.

  • VQ-diffusion corrupts discrete token sequences through masking, substitution, or no change, then estimates the original sequence from corrupted tokens conditioned on text.The forward process composes per-step transition matrices; a Transformer estimates the original-token distribution for reverse sampling.Source: VQ-Diffusion · p. 3

Model Architecture.

  • The CTX-txt2vec architecture combines a text encoder, duration predictor, length regulator, and VQ-diffusion decoder.The decoder adds aligned text encodings to self-attention outputs and retains only the output segment corresponding to the corrupted tokens.Source: Model Architecture · p. 4

Training Scheme.

  • Training randomly uses both contexts, context A alone, or no context, with respective proportions of 0.6, 0.3, and 0.1.The loss combines duration-prediction mean squared error and VQ-diffusion loss using a weighting hyperparameter.Source: Training Scheme · p. 4

Inference Algorithm.

  • For editing inference, the model predicts durations from the text and context phonemes, rescales the generated segment's duration, and iteratively refines fully corrupted tokens.It returns edited semantic tokens [cA, x0, cB].Source: Inference Algorithm · p. 4, Training Scheme · p. 4

CTX-vec2wav with Contextual Vocoding

CTX-vec2wav converts semantic tokens into waveforms while conditioning on acoustic context from mel-spectrograms. Its training assumes speaker identity is consistent within each utterance.

Model Architecture.

  • CTX-vec2wav encodes semantic tokens and uses convolutional upsampling layers based on the HifiGAN generator to produce waveforms.An auxiliary adaptor conditions generation on pitch, energy, and probability of voice; the paper reports preliminary audio-quality improvement from using it.Source: CTX-vec2wav with Contextual Vocoding · p. 5
  • Mel-spectrogram context prompts the semantic encoders through cross-attention, providing acoustic information such as speaker identity without speaker embeddings or acoustic tokens.The mel encoder processes consecutive frames, and the mel features have no positional encoding, allowing variable-length context prompts at inference.Source: CTX-vec2wav with Contextual Vocoding · pp. 4–5
  • The paper leaves the effect of longer mel-spectrogram prompts on speaker similarity for future work.Although variable-length prompts are possible at inference, only a short segment is used during training.Source: CTX-vec2wav with Contextual Vocoding · p. 5

Training Scheme.

  • Training assumes speaker identity stays consistent within each utterance and uses an initial 2–3-second segment for mel context, with the remainder for semantic tokens and vocoding.The training criterion follows HifiGAN with an additional L1 loss for auxiliary-feature prediction.Source: CTX-vec2wav with Contextual Vocoding · p. 5

Unified Framework for Context-Aware TTS

UniCATS uses semantic tokens and semantic context for token generation, then vocodes those tokens using acoustic context. The same editing algorithms are generalized to continuation by omitting following context B.

Unified Framework for Context-Aware TTS

  • UniCATS obtains edited semantic tokens using context, then vocodes them into waveforms using context mel-spectrograms to indicate speaker information.The semantic and acoustic contexts are represented by semantic tokens and mel-spectrograms, respectively.Source: Unified Framework for Context-Aware TTS · pp. 5–6
  • Table 1 reports speech-resynthesis performance from semantic tokens.Its caption identifies the evaluation scope but does not provide specific comparison values.Source: Table 1 · p. 6
  • Speech editing algorithms generalize to continuation by excluding context B, the following context.The paper identifies presence or absence of B as the distinction between the two tasks.Source: Unified Framework for Context-Aware TTS · p. 6

3 Experiments and Results

Experiments evaluate UniCATS on LibriTTS resynthesis, zero-shot continuation, and speech editing. CTX-vec2wav leads semantic-token resynthesis, while UniCATS outperforms the reported baselines in continuation and editing.

Dataset

  • Evaluation uses 500 LibriTTS training-set utterances as test set A and 500 test-clean utterances as test set B for zero-shot adaptation, while editing uses test-clean set C.A has 369 speakers from the 2,306 training speakers; B has 37 unseen speakers. Each speaker in A and B has an approximately 3-second prompt.Source: Dataset · p. 6

Training Setup

  • CTX-txt2vec uses 12 Transformer-based VQ-diffusion blocks and 100 diffusion steps; CTX-vec2wav is trained for 800,000 steps.CTX-txt2vec is trained for 50 epochs, and the vocoder’s learning rate is halved every 200,000 steps.Source: Training Setup · p. 6

Speech Resynthesis from Semantic Tokens

  • CTX-vec2wav is best on semantic-token resynthesis for naturalness and speaker similarity, surpassing Encodec subjectively with SECS comparable to ground truth.HifiGAN has the lowest SECS; the authors attribute this to information compression in its x-vector speaker feature. Encodec occasionally introduces artifacts that lower subjective scores.Source: Speech Resynthesis from Semantic Tokens · pp. 6–7

Speech Continuation for Zero-Shot Speaker Adaptation

  • UniCATS leads FastSpeech 2 and VALL-E in naturalness and speaker similarity for seen and unseen speakers, though unseen-speaker subjective scores dip across systems.UniCATS also receives higher naturalness and speaker-similarity MOS scores than Encodec resynthesis.Source: Speech Continuation for Zero-Shot… · p. 7, Table 2 · p. 7

Speech Editing

  • UniCATS outperforms RetrieverTTS in both speech-editing scenarios; as editing-segment length increases, RetrieverTTS performance declines, while UniCATS exhibits consistent performance across varying segment lengths.Source: Speech Editing · p. 7

4 Conclusion

UniCATS is a two-component context-aware TTS framework for continuation and editing. Its reported evaluations show strong results in semantic-token resynthesis, zero-shot continuation, and speech editing.

Conclusion

  • UniCATS combines the acoustic model CTX-txt2vec and vocoder CTX-vec2wav for speech continuation and editing.The framework uses contextual VQ-diffusion and contextual vocoding to incorporate semantic and acoustic context, respectively.Source: §4 · p. 7
  • CTX-vec2wav outperforms HifiGAN and AudioLM in speech resynthesis from semantic tokens on LibriTTS.Source: §4 · p. 7
  • The authors report state-of-the-art performance for UniCATS in zero-shot speech continuation and speech editing.Source: §4 · p. 7

A Word Error Rate of Resynthesis

The section compares speech reconstruction from HuBERT and vq-wav2vec semantic tokens. The authors report a tradeoff between word error rate and prosody information.

A Word Error Rate of Resynthesis

Loading 2306.07547v6…