Source-linked AI summary
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
TL;DR
The paper asks how one pre-trained model can use unlabeled speech and text while supporting both understanding and sequence-to-sequence generation. SpeechT5 uses a shared encoder-decoder with modality-specific networks and cross-modal vector quantization, and it significantly outperforms baselines across several spoken language processing tasks.
Problem
Prior speech pre-training commonly uses only unlabeled speech and does not pre-train decoders for sequence-to-sequence generation, leaving unified speech-text encoder-decoder modeling underexplored.
Method
SpeechT5 uses a shared encoder-decoder with six modality-specific pre/post-nets, large-scale unlabeled speech and text, and cross-modal vector quantization with mixed representations.
Results
SpeechT5 significantly outperforms state-of-the-art baselines across several spoken language processing tasks, including recognition, synthesis, translation, voice conversion, enhancement, and speaker identification.
Takeaways & Limitations
A unified encoder-decoder can support both generation tasks and diverse spoken language processing tasks while using cross-modal information from unlabeled speech and text.
Abstract
from arXiv · showhide
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification. We release our code and model at https://github.com/microsoft/SpeechT5.
1 Introduction
SpeechT5 addresses limitations in prior speech pre-training by unifying speech and text within a shared encoder-decoder framework. Its cross-modal pre-training and broad evaluations support a single model for diverse spoken language processing tasks.
- Task coverage: The framework covers automatic speech recognition, speech translation, speaker identification, text-to-speech, voice conversion, and speech enhancement.These tasks are represented uniformly as speech/text-to-speech/text transformations.
- Motivation: Prior speech pre-training often uses only unlabeled speech and leaves decoders unpre-trained for sequence-to-sequence generation.The paper identifies these as limitations for tasks requiring modality transformation and generation.
- Framework: SpeechT5 formulates spoken language tasks as speech/text-to-speech/text problems handled by one encoder-decoder model.Modal-specific pre/post-nets connect speech and text inputs and outputs to a shared backbone.
- Evaluation: Extensive experiments demonstrate SpeechT5’s effectiveness across a wide variety of spoken language processing tasks.The model is reported to outperform wav2vec 2.0, HuBERT, and a state-of-the-art voice Transformer on selected tasks.
- Cross-modal pre-training: Cross-modal vector quantization aligns acoustic and textual representations using large-scale unlabeled speech and text data.The method maps both modalities into a shared vector-quantization space and mixes quantized latent representations with contextual states.
2 SpeechT5
SpeechT5 is a unified-modal encoder-decoder framework that learns shared contextual representations for speech and text through a common backbone with modality-specific processing. Its pre-training combines speech and text objectives with cross-modal vector quantization to align their representations.
- Model Architecture: SpeechT5 uses a shared encoder-decoder backbone with six modal-specific pre/post-nets for speech and text processing.Pre-nets map inputs into a unified hidden space, while post-nets generate outputs in the target modality.
- Model Architecture: Speech and text tasks are formulated uniformly as speech/text-to-speech/text sequence-to-sequence problems.This formulation supports a single model across diverse spoken language processing tasks.
- Speech Pre-Training: SpeechT5 pre-trains on unlabeled speech using bidirectional masked prediction and sequence-to-sequence generation objectives.Masked prediction uses acoustic frame-level targets, while generation reconstructs log Mel-filterbank features from masked input.
- Text Pre-Training: Text pre-training reconstructs original text from inputs corrupted by masking randomly selected spans.The method masks 30% of text spans, with span lengths sampled from a Poisson distribution with λ = 3.5.
- Joint Pre-Training: Cross-modal vector quantization aligns speech and text representations through a shared codebook and mixed contextual and quantized representations.Ten percent of contextual representations are randomly replaced with quantized latent representations before decoder cross-attention.
3 Experiments
Experiments evaluate SpeechT5 across diverse spoken language tasks using large-scale pre-training and task-specific fine-tuning, with improvements over strong baselines across the reported settings.
- Pre-Training Setup: SpeechT5 uses 12 Transformer encoder blocks, 6 decoder blocks, and 768-dimensional representations, with speech and text pre-training data drawn from LibriSpeech.Speech pre-training uses 960 hours of LibriSpeech audio, while text pre-training uses 400M sentences.
- ASR: Joint CTC/attention decoding is used for ASR, and the baseline already outperforms wav2vec 2.0 BASE and HuBERT BASE without language-model fusion.The comparison is reported on the 100-hour LibriSpeech setting.
- ASR: SpeechT5 achieves significant ASR improvements over wav2vec 2.0 BASE, HuBERT BASE, and strong baselines across all reported settings.With language-model fusion, SpeechT5 obtains lower WERs than wav2vec 2.0 BASE on all sets and achieves state-of-the-art performance.
- TTS: SpeechT5 improves speech generation quality in TTS, reaching 2.91 Naturalness and 3.65 MOS, with a +0.29 CMOS gain over the baseline.The reported TTS experiments use LibriTTS clean sets and evaluate naturalness with multiple metrics.
- Speech Translation: SpeechT5 improves speech translation by 1.75 BLEU in EN-DE and 1.54 BLEU in EN-FR relative to the baseline.It also outperforms SpeechT5 without decoder initialization and the strong HuBERT-initialized encoder baseline.
4 Related Work
Prior work largely separates speech and text modeling or focuses on encoder-based systems, while SpeechT5 uses a shared cross-modal encoder-decoder pre-trained with both modalities.
- Large-scale pre-training models generally target either text or speech, limiting them to single-modal modeling.
- Speech-language pre-training methods typically use an encoder with task-specific layers rather than a unified encoder-decoder.
- SpeechT5 differs by using shared cross-modal encoder-decoder processing with speech/text pre- and post-nets and unlabeled bimodal pre-training.
- SUPERB instead evaluates frozen shared encoders with lightweight task-specific prediction modules, whereas SpeechT5 fine-tunes a unified-modal encoder-decoder.
5 Conclusion
The paper presents SpeechT5 as a pre-trained encoder-decoder framework that converts spoken language tasks into speech/text-to-speech/text problems. It reports improvements across several tasks and identifies larger-scale and multilingual pre-training as future directions.
- SpeechT5 converts spoken language processing tasks into a speech/text-to-speech/text format and uses joint pre-training on unlabeled speech and text.
- The unified encoder-decoder supports generation tasks including speech translation and voice conversion.
- SpeechT5 significantly outperforms all baselines in several spoken language processing tasks.
- Future work includes pre-training larger SpeechT5 models with more unlabeled data and extending the framework to multilingual processing.
A Comparisons of Text Mask Strategies
The appendix compares BART and T5 text-masking strategies for SpeechT5 pre-training under different ASR inference settings. BART performs comparably to or better than T5 in these comparisons.
- The comparison evaluates BART and T5 strategies for text masking under different inference settings.
- BART achieves comparable or better ASR performance than T5 across the reported inference settings.
B.1 ASR
The appendix details datasets, optimization, decoding, and evaluation procedures for SpeechT5 experiments, with the supplied ASR results emphasizing language-model quality and joint decoding settings.
- ASR: ASR uses LibriSpeech with either 960 hours or 100 hours of transcribed training data, plus LibriSpeech-LM text for language-model fusion.
- ASR: ASR fine-tuning combines CTC and cross-entropy losses with equal weights and uses multi-GPU optimization with warmup and linear decay.
- ASR: The character-level ASR language model has 20 Transformer-decoder blocks, dimension 1280, inner dimension 6144, and 16 attention heads.
- ASR: The SpeechT5 language model reaches perplexities of 56.5 on dev-clean and 59.3 on dev-other, higher than the comparison word-level language model.
- ASR: Decoding uses beam size 30, joint CTC/attention decoding, and a weighted combination of decoder, CTC, and language-model probabilities.
- Other tasks: The appendix also specifies evaluation setups for text-to-speech, speech translation, and voice conversion using LibriTTS, MUST-C, and CMU Arctic.
B.5 SE
The speech-enhancement evaluation uses WHAM! noisy speech to assess content recovery with speech-quality, intelligibility, and recognition metrics. NSNet2 provides a baseline, though its WER is affected by corpus mismatch.
- Dataset: WHAM! contains 20,000 training, 5,000 validation, and 3,000 test utterances for 16 kHz speech enhancement.Each input waveform mixes the first WSJ09 speaker with noise.
- Metrics: Speech enhancement is evaluated with PESQ, ESTOI, and WER to quantify noisy-speech difficulty.WER uses an open-source ASR model because encoder-decoder input and output lengths may differ.
- Baseline: NSNet2 records 45.8% WER, likely because WHAM! noise intensity differs from the DNS corpus.The passage identifies NSNet2 as the baseline from the 2020 Deep Noise Suppression challenge.
C Results for 960 Hours Set of LibriSpeech
SpeechT5 is fine-tuned on the 960-hour LibriSpeech set and shows significant improvement without LM fusion. With LM fusion, it performs comparably to or better than wav2vec 2.0 with LM fusion.
- Evaluation: SpeechT5 is fine-tuned on the 960-hour labeled LibriSpeech set.The results are reported in Table 11.
- Results: SpeechT5 achieves significant improvement on LibriSpeech even without LM fusion.
- Results: With LM fusion, SpeechT5 performs comparably to or better than wav2vec 2.0 with LM fusion.
D Results of the SpeechT5 without Ls
The study compares SpeechT5 with a variant that omits Ls mlm using automatic and human speech-naturalness evaluations. Omitting this speech-specific loss improves naturalness and motivates selecting the variant for MOS and CMOS tests.
- Evaluation: NISQA-TTS is used for convenient, low-cost automatic evaluation, while MOS and CMOS require human judgments.The evaluations use randomly selected generated sentences with consistent text content.
- Results: The SpeechT5 variant without Ls mlm achieves improved naturalness compared with SpeechT5.The result suggests that pretraining without the speech-specific loss brings a significant gain.
- Evaluation: The variant without Ls mlm is selected for MOS and CMOS evaluations.