Source-linked AI summary
Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks
Soumi Maiti, Yifan Peng, Shukjae Choi, Jee-weon Jung, Xuankai Chang, Shinji Watanabe
TL;DR
Prior speech language models often focus on individual tasks, and task- and modality-specific encoder-decoder components complicate combining speech tasks. VoxtLM unifies four speech and text tasks in a decoder-only model with a shared vocabulary. It reports improvements across tasks, with substantial TTS gains in intelligibility and neural-predicted quality.
Problem
Prior speech language models focus on individual tasks, while task- and modality-specific encoder-decoder components complicate integration of multiple tasks.
Source: §1 · p. 1Method
Results
VoxtLM improves speech generation, ASR, and TTS; TTS intelligibility changes from 28.9 to 5.6 and neural-predicted quality from 2.68 to 3.90.
Source: §1 · p. 1, §4.1 · p. 4Takeaways & Limitations
The results support modeling ASR and TTS as language-modeling tasks and combining them with joint speech-text language modeling.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
We propose a decoder-only language model, VoxtLM, that can perform four tasks: speech recognition, speech synthesis, text generation, and speech continuation. VoxtLM integrates text vocabulary with discrete speech tokens from self-supervised speech features and uses special tokens to enable multitask learning. Compared to a single-task model, VoxtLM exhibits a significant improvement in speech synthesis, with improvements in both speech intelligibility from 28.9 to 5.6 and objective quality from 2.68 to 3.90. VoxtLM also improves speech generation and speech recognition performance over the single-task counterpart. Further, VoxtLM is trained with publicly available data and training recipes and model checkpoints are open-sourced to make fully reproducible work.
1. INTRODUCTION
Prior speech language models often target individual tasks, while encoder-decoder designs use modality- and task-specific components that complicate multitask integration. VoxtLM combines four speech and text tasks in one decoder-only model, using a shared vocabulary and reporting improvements across tasks, especially TTS.
1. INTRODUCTION
- Prior speech language models focus on individual tasks, such as speech continuation or text-to-speech.The paper hypothesizes that unifying diverse speech tasks may support improved generalization through multitask learning.Source: §1 · p. 1
- Encoder-decoder architectures use task- and modality-specific components, complicating integration of multiple speech tasks.VoxtLM instead shares parameters across speech and text modalities in a decoder-only design.Source: §1 · p. 1, Figure 1 · p. 1
- VoxtLM combines speech recognition, speech synthesis, text generation, and speech generation in one autoregressive decoder model.Its Voxt vocabulary merges self-supervised discrete speech tokens with text vocabulary.Source: §1 · p. 1
- VoxtLM improves speech generation, ASR, and TTS; TTS intelligibility changes from 28.9 to 5.6 and neural-predicted quality from 2.68 to 3.90.The introduction also reports that pretrained textLM initialization and larger model parameters help ASR.Source: §1 · p. 1
2. RELATED WORK
Prior work explored discrete speech representations and shared speech-text modeling, while several concurrent systems combined speech and text tasks in single models. VoxtLM is presented as a decoder-only model with an integrated speech-text vocabulary and modules for conversion between continuous speech and discrete tokens.
2. RELATED WORK
- Earlier shared speech-text representation methods use separate modality-specific encoders and decoders and additional alignment losses for cross-modal transfer.Semantic tokens from HuBERT or w2v-BERT capture linguistic content; codec-based acoustic tokens capture richer acoustic information but use multiple code streams.Source: §2 · p. 1
- VoxtLM integrates speech and text in an autoregressive decoder-only model, with a tokenizer and decoder converting between continuous speech and discrete tokens.The related-work materials include a data-format table covering training and inference conditions for the tasks.Source: Figure 2 · p. 2, Table 1 · p. 2
- Concurrent systems combine speech and text conversion tasks, but VioLA omits continuation tasks and adds sequence modeling for speech representations.The paper also distinguishes SpeechGPT's three-stage adaptation and PolyVoice's three decoder-only LMs.Source: §2 · p. 2
3. METHOD
VoxtLM models text and discrete speech tokens autoregressively in a shared vocabulary, using task tokens and speech-conversion modules to support multiple task formats. Training uses a transformer decoder, and evaluation covers language modeling, recognition, and synthesis metrics.
3. METHOD
- Text utterances and tokenizer-produced speech tokens are modeled autoregressively in the union vocabulary Vvoxt, which can represent either modality or mixtures.The method defines the text sequence Y over Vtxt and the discrete speech sequence D over Vdst.Source: §3 · p. 2
3.1. VoxtLM
- The speech tokenizer maps continuous speech X to discrete tokens D, while the speech token decoder maps generated tokens back to speech.The tokenizer derives discrete features from pretrained HuBERT using k-means; subword modeling replaces frequent patterns with metatokens and can shorten speech sequences.Source: §3.1 · p. 2
3.1.1. Data format
- Four task tokens mark text or speech conditioning and direct generation of text or speech.The paper notes that additional task-specific tokens could extend the format to more tasks.Source: §3.1.1 · p. 2
3.1.2. Training
- VoxtLM is an autoregressive transformer decoder trained with teacher forcing and cross-entropy; pretrained OPT initializes its weights while the embedding table is learned from scratch.The experiments compare multitask training setups against four single-task models, with utterance counts and setup cases described in Tables 2 and 3.Source: §3.1.2 · p. 2, Table 2 · p. 2, Table 3 · p. 3
3.1.3. Inference
- At inference, task-specific conditions determine whether VoxtLM predicts speech tokens or text; the speech token decoder also uses a speaker embedding to produce speech.The decoder uses HiFiGAN and an x-vector speaker embedding.Source: §3.1.3 · p. 3
3.2. Evaluation Metrics
- Evaluation uses PPL or sWUGGY and sBLIMP for generation, WER for ASR, and CER plus neural-predicted MOS for TTS.The neural MOS model is chosen to scale to many evaluations and is reported to correlate highly with English TTS evaluations.Source: §3.2 · p. 3
4. EXPERIMENTS
Experiments evaluate VoxtLM on public speech-only, text-only, and paired corpora, then test multitask performance, initialization, token vocabulary size, and scaling. Combining tasks improves performance, with TTS gains also reported against VITS; comparisons with other state-of-the-art models have training and architecture differences.
4. EXPERIMENTS
- Experiments draw on public speech-only LibriLight, text-only Librispeech, and paired Librispeech, LibriTTS, and VCTK data.The reported corpus sizes include 60K hours of LibriLight speech, 40M text utterances, and 580 hours of LibriTTS.Source: §4 · p. 3
- Paired ASR and TTS data are 100× smaller than text-only data and 40× smaller than speech-only data, so experiments compare balanced and unbalanced training sets.The study also varies token vocabulary size and model configuration, and uses separate test and development sets for each task.Source: §4 · p. 3
4.1. Results
- Multitask models improve on single-task TTS, and ASR improves in DBal; ASR is most affected in the unbalanced case, where its data ratio is lower.VoxtLM is competitive across all four tasks, although the best model differs by task.Source: §4.1 · p. 3
- Pretrained textLM initialization improves textLM, speechLM, and ASR; in unbalanced ASR it reduces test-clean WER from 21.0 to 13.1.For TTS, CER degrades slightly while objective quality improves.Source: Table 4 · p. 3, §4.1 · p. 4
- For speechLM on DSet, k=200 yields the best sWUGGY and sBLIMP scores; k=50 performs poorly for ASR and TTS.TextLM shows no significant pattern as k varies.Source: §4.1 · p. 4, Table 5 · p. 4
- With larger models, all TextLM, speechLM, and ASR metrics improve, while TTS intelligibility degrades by 0.4 and quality by 0.03.Adding supervised ASR data in DSet+ improves ASR performance.Source: §4.1 · p. 4, Table 6 · p. 4
- Combining four tasks leads to improvement; against VITS, VoxtLM reports better TTS intelligibility and quality, while ASR WER is lower than same-tokenizer dst-ASR-Hubert but higher than ASR-Fbank.The state-of-the-art comparisons are not fully comparable because training data, strategies, and architectures differ.Source: §4.1 · p. 4, Table 7 · p. 4, Table 8 · p. 4
5. CONCLUSION
The conclusion presents joint speech-text language modeling as a promising direction for speech processing. It reports gains from combining speech and text tasks in one generative model, particularly for TTS, and proposes adding more speech tasks in future work.
5. CONCLUSION
- A joint language-modeling framework is presented as a promising avenue for speech processing.Source: §5 · p. 4
- The special-token approach combines speech recognition, speech synthesis, text generation, and speech generation in one generative model, with reported performance improvements.Source: §5 · p. 4
- VoxtLM shows impressive TTS performance compared with VITS, and the authors plan to expand the work to more speech tasks.Source: §5 · p. 4