Source-linked AI summary
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
TL;DR
Low-resource speech recognition is constrained by scarce labeled speech, despite the availability of unlabeled low-resource audio. UniSpeech unifies supervised phonetic CTC learning with phonetically aware contrastive learning, and reports better cross-lingual and domain-transfer results than prior approaches.
Problem
Many languages and specialized domains lack the thousands of hours of labeled speech used by successful recognition systems, while labeled low-resource speech is difficult to collect.
Source: §1 · p. 1Method
UniSpeech jointly pre-trains on labeled high-resource and unlabeled low-resource data using phonetic CTC and phonetically aware contrastive learning, with CTC also guiding quantized representations.
Source: §1 · pp. 1–2Results
UniSpeech outperforms supervised transfer and unsupervised contrastive learning across three CommonVoice cross-lingual settings and yields a 6% relative word error rate reduction on Librispeech-to-Tedlium3 transfer.
Source: §1 · p. 2Takeaways & Limitations
The paper presents UniSpeech as a way to use accessible labeled high-resource and unlabeled low-resource speech for representations intended to capture recognition-relevant content across languages and domains.
Source: §1 · p. 1, §5 · p. 9The CommonVoice evaluation excludes Turkish and Chinese because their test-set files were missing from the June 2020 release.
Source: §4.1.1 · p. 5
Abstract
from arXiv · showhide
In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture information more correlated with phonetic structures and improve the generalization across languages and domains. We evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The results show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech recognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all testing languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task, i.e., a relative word error rate reduction of 6% against the previous approach.
1. Introduction
UniSpeech combines labeled high-resource speech and unlabeled low-resource speech to learn phonetic-aware representations for recognition across languages and domains. It reports gains over supervised transfer and self-supervised learning on cross-lingual and domain-transfer tasks.
1. Introduction
- Low-resource recognition is difficult because many languages lack the thousands of hours of labeled speech used by successful systems, while domain-specific noise and distortion also limit available training data.The paper notes that this challenge extends to specific domains in rich-resource languages.Source: §1 · p. 1
- UniSpeech pre-trains on labeled high-resource and unlabeled low-resource data, then freezes the feature extractor and fine-tunes the Transformer on a small amount of labeled low-resource data.Its feature extractor, Transformer context network, and quantizer follow the wav2vec 2.0 structure.Source: §1 · pp. 1–2
- For labeled data, UniSpeech combines phoneme-label CTC and contrastive losses, then applies CTC to mixtures of contextual and quantized representations to guide the codebook toward speech-recognition-specific information.The paper reports that this replacement strategy activates more codewords and supports a phonetic-aware codebook; unlabeled low-resource data receives contrastive learning only.Source: §1 · p. 2
- UniSpeech outperforms supervised transfer learning and unsupervised contrastive learning in three CommonVoice cross-lingual settings, and achieves a 6% relative word error rate reduction on Librispeech-to-Tedlium3 transfer.The cross-lingual settings are one-to-one, many-to-one, and many-to-many; the domain shift is from audiobook reading to live presentation.Source: §1 · p. 2
- The paper presents joint supervised and self-supervised pre-training and explicit alignment of discrete representations to linguistic units as contributions, reporting state-of-the-art CommonVoice performance without extra data.It frames combining transfer learning and self-supervised learning as a way to use available data in low-resource settings.Source: §1 · pp. 1–2
2. Related Work
Prior low-resource speech-recognition work uses supervised transfer or multitask learning with labeled data, or self-supervised learning from audio. UniSpeech builds on contrastive quantization and multitask approaches while targeting speech recognition with both labeled and unlabeled data.
2. Related Work
- Transfer learning fine-tunes a model trained in a high-resource setting, while multitask learning trains shared components across languages; both rely on labeled data and leave large unlabeled datasets unused.The paper contrasts this with its goal of using all accessible data.Source: §2 · p. 2
- Self-supervised speech methods include reconstructive objectives, which recreate audio frames, and contrastive objectives, which distinguish true samples from negative examples.The cited examples span autoencoders, predictive coding, masked reconstruction, and CPC-style contrastive learning.Source: §2 · p. 2
- UniSpeech's framework masks inputs to the Transformer while feeding unmasked features to the quantizer, whose output maps encoder representations to the nearest codebook point.The caption identifies these as the framework and quantizer views in Figure 1.Source: Figure 1 · p. 3
- Wav2vec 2.0 masks latent speech input and contrasts contextual representations with quantized latent representations; multilingual XLSR extends this approach.Its quantizer's discrete latent representations are reported as related to phonemes.Source: §2 · p. 3
- Earlier multitask representation-learning work combines self-supervised tasks on unlabeled data, while Talnikar et al. combine masked CPC and supervised CTC; UniSpeech instead targets low-resource recognition using all available data.The paper says Talnikar et al. report word error rates equivalent to wav2vec 2.0, whereas UniSpeech aims to improve recognition performance.Source: §2 · p. 3
3. UniSpeech
UniSpeech combines labeled high-resource and unlabeled low-resource speech data to learn representations for low-resource speech recognition. Its multitask objectives align phonetic supervision with contrastive learning over discrete speech representations.
3.1. Problem Formulation
- The problem formulation uses accessible high-resource datasets and low-resource datasets, each with large unlabeled and small labeled sets, to support fine-tuning for low-resource recognition.The stated goal is to learn robust representations from these data and then fine-tune on the low-resource settings.Source: §3.1 · p. 3
3.2. Model Structure
- The architecture uses a convolutional feature encoder, Transformer context encoder, and vector quantizer; Base and Large differ in Transformer size.The feature encoder’s seven convolutional blocks produce representations at roughly 25 ms intervals, strided by 20 ms; Base has 12 Transformer blocks and Large has 24.Source: §3.2 · p. 3
3.3. Multitask Learning with Unified Representation
- Three pretraining objectives jointly use high-resource labeled data and low-resource speech: phonetic CTC, source-data contrastive learning, and target-data contrastive learning.The CTC objective aligns contextual representations with phonetic units; source contrastive learning brings contextual and discrete features closer, while target contrastive learning adapts the model to the target language or domain.Source: §3.3 · p. 4
- For masked time steps, the model contrasts each contextual representation with its true quantized target and K distractors sampled from other times in the same utterance.The paper states this setup focuses model capacity on phonetic rather than speaker-dependent features.Source: §3.3 · p. 4
- Predicting phoneme sequences from quantized representations guides the codebooks toward phonetic clustering, while unifying supervised and self-supervised representations in the same space.The model replaces continuous representations with quantized ones for the CTC loss with probability r; the paper reports that this addresses limited gains from its earlier multitask method.Source: §3.3 · pp. 4–5
- After pretraining, the model is fine-tuned with CTC on low-resource data, with the feature encoder frozen and only the Transformer updated.The pretrained CTC layer is replaced with a new layer for the target vocabulary.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate UniSpeech on multilingual CommonVoice recognition and transfer from reading to spoken English. Across the tested settings, it outperforms supervised and self-supervised baselines, while analyses examine its phonetic representations and hyperparameters.
4.1.1. SETUP
- CommonVoice evaluation uses eight languages, with English as a high-resource language; fine-tuning and testing use one hour of paired data each.The dataset is multilingual read speech, and the experiments define one-to-one, many-to-one, and many-to-many settings.Source: §4.1.1 · p. 5
4.1.2. RESULTS
- Across one-to-one, many-to-one, and many-to-many settings, UniSpeech reduces PER versus CTC-Transfer/XLSR by 9.6%/13.4%, 17.8%/9.8%, and 10.4%/5.1%, respectively.The comparisons use the corresponding CommonVoice evaluation settings.Source: Table 1 · p. 6, §4.1.2 · pp. 6–7, Table 2 · p. 6
- UniSpeech+ outperforms XLSR+ by 21.1% relative PER in many-to-one evaluation, and its Large model improves the baseline by 26.9% relative.With target-language unlabeled data, UniSpeech+ also outperforms XLSR-10+; the passage gives no numerical margin for that comparison.Source: §4.1.2 · p. 7
4.2. Domain Transfer
- UniSpeech outperforms wav2vec 2.0 and CTC transfer on the Librispeech-to-Tedlium3 domain-transfer task.The comparison uses English reading-domain pretraining and spoken-English-domain transfer.Source: §4.2 · pp. 7–8, Table 4 · p. 7
4.3. Discussion and Analysis
- On TIMIT, many learned discrete latents specialize in phonetic sounds, while silence phoneme bcl aligns with the most latents.The analysis uses a multilingual UniSpeech model without fine-tuning and attributes bcl’s broad alignment to CTC blanks and silence in TIMIT utterances.Source: §4.3 · p. 8, Figure 2 · p. 8
- UniSpeech has 25,743 active codewords and 0.83 alignment entropy, versus XLSR’s 3,645 and 1.34, indicating a more diverse codebook and better phoneme clustering.The authors offer this as a possible interpretation of UniSpeech’s CommonVoice performance.Source: §4.3 · p. 8
- Setting replacement probability r to 0 reduces performance by about 7.8% relatively; r values from 0.3 to 0.7 perform similarly, with 0.5 selected for experiments.Loss weight α that is too low also reduces performance, while increasing it from 0.5 to 0.7 brings no improvement; mask probability has little impact.Source: §4.3 · p. 9
5. Conclusion
UniSpeech unifies supervised CTC labeling and phonetically-aware contrastive learning to learn speech representations from labeled and unlabeled data. It outperforms supervised and self-supervised pre-training alone on multilingual speech recognition and domain transfer tasks.
5. Conclusion
- UniSpeech unifies supervised CTC labeling and phonetically-aware contrastive learning in a multitask framework for speech representations from labeled and unlabeled data.Its convolutional feature extractor, Transformer encoder, and quantizer are designed to learn SR-specific information.Source: §5 · p. 9
- UniSpeech outperforms supervised and self-supervised pre-training alone by a large margin on multilingual speech recognition and domain transfer tasks.Source: §5 · p. 9
- Future work will scale training with over one million hours of unlabeled data and explore more complex speech-recognition architectures and pre-trained language models.Source: §5 · p. 9