Source-linked AI summary
MAESTRO: Matched Speech Text Representations through Modality Matching
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, Heiga Zen
TL;DR
Jointly learning speech and text representations is challenging because the modalities differ, and earlier approaches relied on multitask learning or modality conversion. Maestro aligns and matches their representations without explicit conversion, reporting state-of-the-art results on multilingual ASR and speech translation.
Problem
Jointly learning shared speech and text representations is challenging because the two modalities are very different.
Source: §1 · p. 1Method
Maestro aligns text representations with speech and matches embeddings in a shared space using aligned masked-language-model training.
Source: §1 · p. 1, §3.2.3 · p. 2Results
Takeaways & Limitations
Maestro provides aligned speech-text representations without explicit modality conversion, which the authors describe as computationally efficient.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We present Maestro, a self-supervised training method to unify representations learnt from speech and text modalities. Self-supervised learning from speech signals aims to learn the latent structure inherent in the signal, while self-supervised learning from text attempts to capture lexical information. Learning aligned representations from unpaired speech and text sequences is a challenging task. Previous work either implicitly enforced the representations learnt from these two modalities to be aligned in the latent space through multitasking and parameter sharing or explicitly through conversion of modalities via speech synthesis. While the former suffers from interference between the two modalities, the latter introduces additional complexity. In this paper, we propose Maestro, a novel algorithm to learn unified representations from both these modalities simultaneously that can transfer to diverse downstream tasks such as Automated Speech Recognition (ASR) and Speech Translation (ST). Maestro learns unified representations through sequence alignment, duration prediction and matching embeddings in the learned space through an aligned masked-language model loss. We establish a new state-of-the-art (SOTA) on VoxPopuli multilingual ASR with a 8% relative reduction in Word Error Rate (WER), multidomain SpeechStew ASR (3.7% relative) and 21 languages to English multilingual ST on CoVoST 2 with an improvement of 2.8 BLEU averaged over 21 languages.
1. Introduction
Jointly learning speech and text representations requires combining two very different modalities. Maestro aligns their representations without explicit modality conversion and reports state-of-the-art results on multilingual ASR and speech translation.
- Speech and text pretraining capture different information, while diverse downstream tasks call for representations of acoustic, prosodic, speaker, linguistic, and semantic characteristics.The introduction presents joint pretraining as a natural extension.Source: §1 · p. 1
- Maestro combines a common latent space and aligned representations without converting text to speech or speech to text.The paper describes this approach as computationally efficient.Source: §1 · p. 1
- Maestro uses small amounts of transcribed speech to unify representations learned from unlabeled speech and text.Source: §1 · p. 1
- The method also uses Machine Translation and Speech Translation supervision to improve multilingual joint representations during pretraining.Source: §1 · p. 1
- 8% relative WER reduction on VoxPopuli multilingual ASR and 2.8 BLEU absolute improvement on CoVoST-2 21-language-to-English speech translation establish new state of the art.These are the paper’s reported headline comparisons.Source: §1 · p. 1
2. Related Work
Prior joint speech-text approaches use multitask training or speech synthesis, each with a different trade-off. Maestro bridges these lines of work through latent-space alignment and extends joint representation learning to multilingual data and tasks.
- Multitask approaches can suffer interference and capacity limits, whereas TTS-based augmentation avoids inter-modal interference but requires a complete TTS model [5] [10] [12].Maestro bridges these approaches by implicitly aligning text and speech representations in latent space.Source: §2 · p. 1
- Prior work also improves representations with transcribed speech [25] or paired masked-language-model objectives on MT [26] and ST [27] data.Source: §2 · p. 2
- Maestro applies cross-lingual supervision from MT and ST data while training monolingual and massively multilingual speech-text models.Source: §2 · p. 2
3. Proposed method
Maestro jointly pretrains speech and text representations in a shared multimodal space. It aligns text embeddings to speech using paired-data alignments or predicted durations, then applies aligned language-model training to resampled text.
3.1. Architecture
- A shared multimodal encoder learns joint representations from separate speech and text encoders, with modality matching and aligned masked-language-model objectives.Task-dependent decoders predict grapheme, phoneme, word-piece, or word targets from the shared representations.Source: §3.1 · p. 2
3.2. Modality Matched Training
- Modality matching links speech and text at the phonetic level, mapping text units to speech representations independently of speaker and prosody.Source: §3.2 · p. 2
3.2.1. Learning Initial Embeddings
- The speech encoder learns local acoustic and phonetic information with a within-utterance contrastive loss, while the text extractor captures lexical information.Source: §3.2.1 · p. 2
3.2.2. Learning Aligned Embeddings
- Initial speech and text embeddings are not expected to align; Maestro iteratively self-aligns them, using RNN-T alignments for paired data and predicted token durations for unspoken text.For paired data, the alignments guide resampling and refining text embeddings to match speech-embedding duration; predicted durations are used to resample text embeddings for unspoken text.Source: §3.2.2 · p. 2
3.2.3. Aligned Masked Language Model training objective
- Aligned masked-language-model training applies RNN-T loss to masked, resampled text embeddings and also includes an A-MLM loss with paired speech and text.The same RNN-T loss can be used with unpaired text or paired speech data.Source: §3.2.3 · p. 2
3.3. Untranscribed Speech
- For untranscribed speech, Maestro uses contrastive loss on speech-encoder outputs and masked-language-model loss on shared-encoder outputs.Source: §3.3 · p. 3
4. Data and Experimental Setup
The experiments pretrain Maestro on monolingual and multilingual speech and text, then evaluate ASR and speech translation. The setup varies data sources, model components, training schedule, and downstream tasks.
4.1. Pre-training Data
- Pretraining covers LibriLight speech and TEDLIUM/Librispeech text for monolingual ASR, 429k hours of multilingual speech, and CoVoST translation data.Multilingual unspoken text uses VoxPopuli text, mC4, or both; paired MT text comes from CoVoST, WMT, and TED.Source: §4.1 · p. 3
4.2. Architecture Details
- The 24-layer, 600M-parameter Conformer XL assigns six lower layers to speech encoding and 18 upper layers to the shared encoder.The text encoder uses convolutional and Transformer layers, with a refiner and duration model; the RNN-T decoder can be replaced by other decoders.Source: §4.2 · p. 3
4.3. Pretraining hyper-parameters
- Training includes untranscribed speech, unspoken text, and paired data, with EMA stabilization and a curriculum that adds paired data after 500k steps and text 15k steps later.Joint training continues for another 300k steps; batch sizes vary across SpeechStew, VoxPopuli, and mC4 settings.Source: §4.3 · p. 3
4.4. Downstream Tasks
- Downstream evaluation covers SpeechStew and 14-language VoxPopuli ASR, plus CoVoST-2 translation from 21 languages into English.The SpeechStew table covers five domains and reports results for models with 0.6B parameters.Source: §4.4 · p. 3, Table 1 · p. 3
5. Experiments and Results
Across SpeechStew multi-domain ASR, VoxPopuli multilingual ASR, and CoVoST 2 speech translation, Maestro improves on prior systems. Results include an 8% relative WER reduction on VoxPopuli and a 2.8 BLEU gain over 0.6B mSLAM on CoVoST 2.
5.1. Monolingual Multi-domain Speech Recognition
- Up to 6% and 12% relative WER reductions over TTS4Pretrain 2.0 and SLAM, respectively, accompany Maestro’s gains on multi-domain ASR.On LibriSpeech, Maestro closes the gap with LM shallow fusion and gains further with LM fusion.Source: §5.1 · pp. 3–4
5.2. Multilingual Speech Recognition
- 8% relative: combining mC4 and VP-T gives Maestro a new VoxPopuli ASR state of the art over speech-only W2v-BERT.Adding massive out-of-domain unspoken text alone causes a slight degradation; the combined corpora yield the best performance.Source: §5.2 · p. 4, Table 2 · p. 4
- Most languages improve, including Slovenian with only 10 hours of supervised speech, while phoneme and grapheme pretraining yield similar overall performance.Graphemes are used for supervised fine-tuning regardless of the pretraining units.Source: §5.2 · p. 4, Figure 2 · p. 4
6. Conclusions
Maestro jointly learns speech and text representations by aligning text to speech and matching text representations to W2v-BERT speech representations. The method yields reported improvements of 8% on ASR and 2.8 BLEU on speech translation.
- Maestro aligns text to speech, then trains text representations to match W2v-BERT speech representations, improving reported ASR and speech translation results.The conclusion reports an 8% ASR improvement and a 2.8 BLEU speech translation improvement.Source: §6 · p. 4