Source-linked AI summary
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen, Kai Yu
TL;DR
Decoder-only TTS models need monotonic alignment constraints to address hallucination robustness. VALL-T adds shifting relative positions within a decoder-only Transducer, and achieves a 28.3% relative WER reduction compared with VALL-E.
Problem
VALL-T introduces monotonic alignment constraints into decoder-only TTS to improve robustness against hallucinations.
Source: §I · p. 1Method
VALL-T integrates Transducer modules into one decoder-only Transformer and uses shifted relative positions to indicate phoneme positions during generation.
Source: §I · p. 1, §III · p. 2Results
28.3% relative reduction in WER versus VALL-E; VALL-T’s WER is only 0.41 higher than NAR resynthesis.
Source: C. Evaluation on zero-shot TTS · p. 4Takeaways & Limitations
VALL-T implicitly models phoneme durations without requiring them before training and supports forced alignment from input phonemes and output speech.
Source: §V · p. 4The alignment analysis is qualitative because ground-truth alignment labels are unavailable.
Source: B. Alignment analysis · p. 4
Abstract
from arXiv · showhide
Recent TTS models with decoder-only Transformer architecture, such as SPEAR-TTS and VALL-E, achieve impressive naturalness and demonstrate the ability for zero-shot adaptation given a speech prompt. However, such decoder-only TTS models lack monotonic alignment constraints, sometimes leading to hallucination issues such as mispronunciation, word skipping and repeating. To address this limitation, we propose VALL-T, a generative Transducer model that introduces shifting relative position embeddings for input phoneme sequence, explicitly indicating the monotonic generation process while maintaining the architecture of decoder-only Transformer. Consequently, VALL-T retains the capability of prompt-based zero-shot adaptation and demonstrates better robustness against hallucinations with a relative reduction of 28.3% in the word error rate.
I. INTRODUCTION
TTS requires monotonic phoneme-to-speech generation, while decoder-only models offer naturalness and prompt-based zero-shot adaptation without explicit phoneme durations. VALL-T combines a decoder-only architecture with monotonic alignment constraints to address hallucinations and support alignment-aware inference.
I. INTRODUCTION
- TTS is a monotonic sequence-to-sequence task whose frame-level speech output is longer than its phoneme input.Typical non-autoregressive TTS models use explicit duration prediction, conventionally derived with Viterbi forced alignment.Source: §I · p. 1
- SPEAR-TTS and VALL-E combine decoder-only TTS with naturalness and prompt-based zero-shot speaker adaptation, but lack monotonic alignment constraints.These models also avoid explicit phoneme-duration modeling.Source: §I · p. 1
- VALL-T combines decoder-only Transformer architecture with a generative Transducer that avoids obtaining phoneme durations beforehand while maintaining monotonic alignment.The method integrates Transducer modules into one decoder-only Transformer using shifted relative positions.Source: §I · p. 1
- VALL-T introduces monotonic alignment constraints into decoder-only TTS to improve robustness against hallucinations.Decoder-only models can mispronounce, skip, or repeat words when implicit duration modeling lacks monotonic alignment constraints.Source: §I · p. 1
- VALL-T supports forced alignment from paired phoneme and speech sequences.The alignment is found by searching the best path on the posterior probability map.Source: §I · p. 1
- During inference, shifting relative positions lets the model track alignment between phonemes and speech as generation proceeds.The model shifts positions from left to right to guide monotonic generation.Source: §I · p. 1
II. RELATED WORK
Transducer models address monotonic sequence-to-sequence tasks through an alignment grid and a training objective that sums over possible paths. Their inference uses blank tokens to mark when the input position advances.
II. RELATED WORK
- A Transducer uses an encoder, prediction network, and joint network to model monotonic sequence-to-sequence tasks.The prediction network is autoregressive, and a blank token marks an alignment boundary between input and output.Source: §II · p. 1
- During training, the joint network combines encoded input and output sequences at possible positions to construct an alignment grid.Each path through the grid represents an alignment, and the objective maximizes Pr(y|x) by summing probabilities over possible paths.Source: §II · pp. 1–2
- During inference, the prediction network generates tokens autoregressively and advances through input hidden vectors when a blank token appears.The input vectors are sliced from f0 to fT−1 as blank tokens emerge.Source: §II · p. 2
III. VALL-T: DECODER-ONLY GENERATIVE TRANSDUCER
Modularized Transducers have limitations for conditional generation because input conditioning arrives at the joint network, which may be too small to integrate it effectively. VALL-T instead combines the Transducer components in a decoder-only Transformer using relative position embeddings.
III. VALL-T: DECODER-ONLY GENERATIVE TRANSDUCER
- In modularized Transducers, input conditioning reaches generation at the joint network, which may be too small to integrate the condition effectively.This architecture can make it difficult to achieve satisfactory conditional-generation performance.Source: §III · p. 2
- VALL-T integrates the encoder, prediction network, and joint network into one decoder-only Transformer and uses relative position embeddings to denote corresponding positions.This design replaces the modularized architecture's slicing approach for denoting positions.Source: §III · p. 2
A. Training
VALL-T concatenates phoneme and speech tokens in a decoder-only Transformer, then adds relative phoneme positions to make the current phoneme explicit. It builds an alignment grid for Transducer loss using outputs associated with relative position 0.
A. Training
- Phoneme and speech tokens are concatenated along the time axis as one sequence, and a triangular attention mask lets each speech token attend only to previously generated speech tokens.The output sequence includes an initial <sos> token and uses absolute position indices.Source: A. Training · pp. 2–3
- VALL-T adds sinusoidal relative-position embeddings to input phoneme embeddings, marking the current phoneme as 0 and its left and right context with negative and positive indices.The absolute and relative embeddings are both presented to the decoder-only Transformer.Source: A. Training · p. 3
- The output projection maps hidden states to an extended vocabulary that includes the blank token ∅, and Softmax produces output distributions.The projected sequence has length U + 1.Source: A. Training · p. 3
- Transducer loss is computed on a T × (U + 1) alignment grid built by iterating over relative positions, with the decoder output serving as the column for position 0.The loss maximizes the probability of p(y|x).Source: A. Training · p. 3
B. Monotonic auto-regressive inference
During autoregressive inference, VALL-T generates speech for the phoneme at relative position 0 until a blank token marks completion, then shifts positions. With a speech prompt, generation starts after the prompt transcript and speech.
B. Monotonic auto-regressive inference
- Generation starts from <sos> with the first phoneme at relative position 0; speech tokens are produced until ∅ marks that phoneme's completion, triggering a position shift.The process repeats until ∅ appears for the final phoneme.Source: B. Monotonic auto-regressive inference · p. 3
- For prompted continuation, the prompt transcript and speech precede the input phonemes and output speech, and relative position 0 is assigned to the first phoneme after the prompt transcript.Positions shift whenever ∅ appears, until generation for the final phoneme is complete.Source: B. Monotonic auto-regressive inference · p. 3
A. Setup
The experiments use Encodec speech tokens and a VALL-E-matched model architecture, trained on LibriTTS. The supplied table caption identifies a zero-shot TTS performance table but gives no result values.
A. Setup
- VALL-T uses Encodec tokens at 50 Hz and 16 kHz output waveforms, with eight RVQ indices per frame; an autoregressive model predicts the first, then a separate NAR model predicts the remaining seven conditioned on it.Both input and output sequences are BPE-encoded to shorten sequence lengths.Source: A. Setup · p. 3
- Table I is labeled as reporting zero-shot TTS performance, but the supplied caption provides no metrics or comparisons.No result values can be reported from the caption alone.Source: Table I · p. 4
- The model matches VALL-E's architecture with 12 Transformer layers, 12 attention heads per block, and a hidden dimension of 1024.The authors select Encodec following VALL-E to support a fair comparison and reduce other factors affecting analysis.Source: A. Setup · pp. 3–4
- Training uses approximately 580 hours of LibriTTS speech from 2,306 speakers for 40 epochs, with ScaledAdam and eight A800 GPUs.The scheduler is Eden, with a base learning rate of 0.05.Source: A. Setup · p. 4
B. Alignment analysis
The alignment analysis qualitatively examines VALL-T's forward and backward variables and posterior probabilities. The posterior map shows a clearer diagonal, and its most probable path provides a forced alignment, though ground-truth alignment labels are unavailable.
B. Alignment analysis
- Figure 2 plots forward variables, backward variables, and posterior probabilities on a logarithmic scale, with darker colors representing lower values.The forward and backward plots show a faint bright diagonal line.Source: B. Alignment analysis · p. 4
- The posterior probability map has a clearer diagonal, indicating modeled input-output alignment; its most probable bottom-left-to-top-right path serves as a forced alignment.The analysis is qualitative because ground-truth alignment labels are unavailable.Source: B. Alignment analysis · p. 4
C. Evaluation on zero-shot TTS
The zero-shot TTS evaluation compares VALL-T with VALL-E and Transduce and Speak, using WER as its main metric. VALL-T reports a lower WER than the baselines while performing comparably to or better than them on most naturalness and speaker-similarity metrics.
C. Evaluation on zero-shot TTS
- The evaluation uses 500 utterances from 37 LibriTTS test-set speakers, each assigned a speech prompt, for zero-shot TTS.Zero-shot TTS here means synthesizing speech in unseen speakers’ voices given speech prompts and corresponding transcripts.Source: C. Evaluation on zero-shot TTS · p. 4
- NAR resynthesis measures performance degradation introduced by the NAR model to help analyze the full pipeline, with AR models as the primary focus.It predicts the remaining seven RVQ indices after receiving the ground-truth first index.Source: C. Evaluation on zero-shot TTS · p. 4
- VALL-T achieves a 28.3% relative WER reduction versus VALL-E and has WER only 0.41 higher than NAR resynthesis.The evaluation compares synthesized-speech ASR transcripts with the ground-truth input text.Source: C. Evaluation on zero-shot TTS · p. 4
- VALL-T performs comparably to or better than the baselines on most MCD, SECS, and MOS metrics.SECS compares generated speech with the provided prompt; 15 listeners rated MOS on a 1-to-5 scale.Source: C. Evaluation on zero-shot TTS · p. 4
V. CONCLUSION
VALL-T incorporates monotonic alignment constraints into a decoder-only TTS framework, allowing implicit phoneme-duration modeling without acquiring durations before training. It also supports forced alignment by searching for the best path on the posterior probability map.
V. CONCLUSION
- VALL-T incorporates monotonic alignment constraints into decoder-only TTS, enabling implicit phoneme-duration modeling without acquiring durations before training.Source: §V · p. 4
- VALL-T supports forced alignment of input phonemes and corresponding output speech by searching for the best path on the posterior probability map.Source: §V · p. 4
- VALL-T is a decoder-only generative Transducer model designed to improve TTS robustness and controllability.Source: §V · p. 4