Source-linked AI summary
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
TL;DR
Speech self-supervised learning has made strong progress in speech recognition, while speaker-characteristic modeling has received limited exploration. UniSpeech-SAT adds utterance-wise contrastive learning and utterance mixing to HuBERT; it reports state-of-the-art overall SUPERB performance, especially on speaker-oriented tasks.
Problem
Speech self-supervised learning has succeeded in speech recognition, but its use for modeling speaker characteristics has seen limited exploration.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The paper presents speaker-aware pretraining as complementary to ASR-oriented pretraining and reports gains across speaker and emotion tasks.
Source: §1 · p. 2The contrastive-learning setup assumes one active speaker per training utterance and different speakers across utterances in each batch.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled data and avoids extensive human labeling. Recent years witness great successes in applying self-supervised learning in speech recognition, while limited exploration was attempted in applying SSL for modeling speaker characteristics. In this paper, we aim to improve the existing SSL framework for speaker representation learning. Two methods are introduced for enhancing the unsupervised speaker information extraction. First, we apply the multi-task learning to the current SSL framework, where we integrate the utterance-wise contrastive loss with the SSL objective function. Second, for better speaker discrimination, we propose an utterance mixing strategy for data augmentation, where additional overlapped utterances are created unsupervisely and incorporate during training. We integrate the proposed methods into the HuBERT framework. Experiment results on SUPERB benchmark show that the proposed system achieves state-of-the-art performance in universal representation learning, especially for speaker identification oriented tasks. An ablation study is performed verifying the efficacy of each proposed method. Finally, we scale up training dataset to 94 thousand hours public audio data and achieve further performance improvement in all SUPERB tasks.
1. INTRODUCTION
Speech self-supervised pretraining has advanced speech representation learning, but most work focuses on spoken content and speaker-characteristic modeling remains less explored. UniSpeech-SAT adds speaker-aware training to HuBERT and reports state-of-the-art overall SUPERB performance.
1. INTRODUCTION
- Speech SSL work includes generative, discriminative and multi-task approaches, with discriminative pretraining showing success in ASR and SUPERB evaluations.The passage describes future-frame prediction in APC and contrastive learning or weak-cluster-label classification among discriminative methods.Source: §1 · p. 1
- Most speech pretraining targets spoken content, while whether matched pretraining can further improve speaker-task performance remains an open question.The passage notes that optimizing one speech characteristic may yield suboptimal representations for other tasks.Source: §1 · p. 1
- The method combines utterance-wise contrastive learning with unsupervised representation learning and mixes partially overlapped speech into training samples.The contrastive method uses utterance-level aggregated embeddings and speaker-wise pseudo-labels; the mixing strategy retains the training objective.Source: §1 · pp. 1–2
- UniSpeech-SAT achieves state-of-the-art overall SUPERB performance, improving speaker identification, verification, diarization and emotion recognition while maintaining speech-recognition performance.The paper also reports releasing its model.Source: §1 · p. 2
- 94k hours of public English audio from LibriVox, GigaSpeech and VoxPopuli further improves speaker-task performance over prior work using 60k LibriVox hours.The paper presents speaker-aware pretraining as complementary to ASR-oriented pretraining.Source: §1 · p. 2
2. BACKGROUND
HuBERT learns speech representations through iterative clustering and masked prediction. As training iterations proceed, the model’s embeddings replace MFCC features as clustering input.
2. BACKGROUND
- HuBERT obtains frame-level pseudo-labels by offline clustering MFCC features in its first iteration.The cluster center for each frame is indexed as that frame’s pseudo-label.Source: §2 · p. 2
- The model trains a Transformer with a CNN feature extractor to predict cluster pseudo-labels for masked regions from the corrupted input sequence.The mask replaces selected frame features with a mask embedding; the loss is cross-entropy over masked positions.Source: §2 · p. 2
- HuBERT iterates between clustering and network training, using MFCC features initially and embeddings from the previous iteration for later clustering.The passage says pseudo-labels and embeddings are presumed to be refined through these iterations.Source: §2 · p. 2
3. UNISPEECH-SAT
UniSpeech-SAT adds utterance-wise contrastive learning and utterance mixing to HuBERT to strengthen speaker information modeling. It also uses large, diverse pre-training data.
3. UNISPEECH-SAT
- UniSpeech-SAT combines utterance-wise contrastive learning and utterance mixing with HuBERT, targeting single-speaker identity and multi-speaker tasks, respectively.The contrastive approach targets tasks such as speaker verification and identification; mixing is intended to benefit tasks such as diarization.Source: §3 · p. 2
3.1. Utterance-wise Contrastive Learning
- The integration assumes one active speaker per training utterance and different speakers across utterances in each batch.The authors believe these assumptions are mostly satisfied because the dataset is collected from various sources.Source: §3.1 · p. 2
- Intermediate Transformer representations are quantized, and the contrastive objective treats representations from the same utterance as positives and those from other utterances in the batch as negatives.The speaker loss combines contrastive and codebook-diversity losses, then combines with the content loss; the diversity term encourages equal use of codebook entries.Source: §3.1 · pp. 2–3
3.2. Utterance Mixing Augmentation
- Utterance mixing simulates multi-speaker speech by adding a randomly cropped chunk from another utterance, while training still targets the main speaker and its content.The mixed portion is kept below 50% to avoid potential label permutation.Source: §3.2 · p. 3
3.3. Large and Diverse Pre-training Data
- The expanded pre-training set combines LibriVox, GigaSpeech, and VoxPopuli for 94K hours of audio spanning diverse sources and speaking contexts.GigaSpeech includes audiobooks, podcasts, and YouTube; VoxPopuli consists of European Parliament event recordings.Source: §3.3 · p. 3
4. EXPERIMENT
SUPERB results show strong speaker-task gains alongside a task-dependent ASR trade-off. The experiments also examine training setup, utterance-mixing ratios, and which model layers contribute to different tasks.
4.1. Implementation Details
- Base is pretrained for 400k steps on LibriSpeech 960 hours; Base+ and Large use 400k steps on 94K hours of diverse data.The models use HuBERT-generated clustering labels and the same hyperparameters as HuBERT.Source: §4.1 · pp. 3–4
4.2. Universal Representation Evaluation
- SUPERB evaluates ten speech tasks across speaker, content, semantics, and paralinguistics, with a frozen Transformer and task-specific weighted sums of hidden states.Task-specific layers follow SUPERB policies.Source: §4.2 · p. 4
- DER is reduced by over 25% in both base and large settings, while ASR without LM is 9% worse than baseline for large and less than 2% worse for base.The reported results also show gains in speaker identification, speaker verification, and emotion recognition; the authors attribute the ASR degradation to limited capacity and a trade-off between speaker and content information.Source: §4.2 · p. 4, Table 1 · p. 4
4.3. Analysis
- Shallow layers contribute more to speaker verification and diarization, while top layers contribute more to ASR and intent classification.The authors interpret this pattern as shallow layers learning speaker information and top layers learning content and semantic information.Source: §4.3 · p. 4, Figure 2 · p. 4
- Utterance mixing remains effective with 94k hours of training data; 20% is selected for Base+ as a trade-off between speaker and content.The analysis tests mixing ratios of 0%, 20%, and 50%.Source: §4.3 · p. 4, Table 2 · p. 4
5. CONCLUSION
The paper combines contrastive loss and utterance mixing within unsupervised speech representation learning to improve speaker discrimination. On SUPERB, it reports state-of-the-art performance and large margins over other baselines.
5. CONCLUSION
- The work integrates contrastive loss and utterance mixing into an existing unsupervised speech representation learning framework.The integration aims to improve speaker discrimination in learned embeddings.Source: §5 · p. 4
- The integrated methods aim to improve speaker discrimination in learned embeddings.Source: §5 · p. 4
- On SUPERB, the model achieves state-of-the-art performance and outperforms other baselines by a large margin.Source: §5 · p. 4