PaperlayerView exact revision on arXiv

Source-linked AI summary

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed

arXiv:2106.07447v1cs.CLcs.AIcs.LGeess.AS

TL;DR

Speech representation learning lacks discrete sound-unit labels and explicit boundaries, complicating masked prediction. HuBERT uses clustered hidden-unit targets for masked continuous-speech prediction and matches or improves upon wav2vec 2.0 across fine-tuning subsets, with up to 19% relative WER improvement on dev-other.

  • Problem

    Self-supervised speech learning lacks a lexicon of discrete sound units and known boundaries in continuous utterances containing multiple sounds.

  • Method

    HuBERT clusters speech into frame-level hidden units offline, then predicts the assignments only for masked regions of continuous inputs with a BERT-like objective.

  • Results

    HuBERT matches or improves upon wav2vec 2.0 across Librispeech and Libri-light fine-tuning subsets, with up to 19% and 13% relative WER improvement on dev-other and test-other.

  • Takeaways & Limitations

    HuBERT provides a speech representation learning approach that performs competitively across labeled-data regimes using masked prediction of clustered continuous speech.

Abstract

from arXiv · show

Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.

I. INTRODUCTION

HuBERT addresses speech representation learning’s distinctive challenges—multiple sounds, unavailable discrete-unit lexicons, and continuous sequences—by combining offline clustering with masked prediction over continuous speech. It matches or improves wav2vec 2.0 across Librispeech and Libri-Light fine-tuning subsets, with gains from larger models.

  • Challenges: Speech self-supervised learning must handle multiple sounds per utterance, no pre-training lexicon of discrete units, and continuous-valued sequences.These properties distinguish speech from common computer-vision and natural-language pre-training assumptions.
  • Method: HuBERT uses offline clustering to generate noisy target labels, then applies BERT-like prediction to masked continuous speech features.The model predicts predetermined cluster assignments rather than relying on a prior lexicon.
  • Method: Masked-region-only loss forces HuBERT to learn high-level representations of unmasked inputs that support inference of masked targets.Restricting the predictive loss to masked regions encourages the model to combine acoustic and contextual information.
  • Results: HuBERT matches or improves wav2vec 2.0 on Librispeech 960h and Libri-Light 60k hours across 10mins, 1h, 10h, 100h, and 960h fine-tuning subsets.The reported HuBERT model sizes are BASE (90M parameters), LARGE (300M), and X-LARGE (1B).
  • Results: Up to 19% and 13% relative WER improvement from LARGE models is achieved by X-LARGE on dev-other and test-other, respectively.The X-LARGE model contains 1B parameters, compared with 300M for LARGE.

II. METHOD · A. Learning the Hidden Units for HuBERT · B. Representation Learning via Masked Prediction

HuBERT learns frame-level hidden units through unsupervised clustering and trains a masked prediction model to infer those targets from corrupted speech. Applying the loss only to masked frames encourages learning both acoustic representations and longer-range temporal structure while reducing dependence on cluster quality.

  • A. Learning the Hidden Units for HuBERT: Simple k-means and GMMs produce hidden units that correlate non-trivially with underlying acoustic units.More advanced graphical or neural models can improve acoustic unit discovery.
  • A. Learning the Hidden Units for HuBERT: HuBERT predicts the hidden cluster assignments of masked frames generated by one or more k-means clustering iterations.The approach therefore uses clustering to provide aligned frame-level prediction targets without a speech lexicon.
  • A. Learning the Hidden Units for HuBERT: HuBERT uses an acoustic unit discovery model h to map each T-frame utterance X to frame-level categorical targets Z with C hidden-unit classes.The clustering model may be k-means.
  • B. Representation Learning via Masked Prediction: The model corrupts a length-T sequence by replacing selected positions M with a mask embedding, then predicts target distributions at every timestep.Masking uses SpanBERT and wav2vec 2.0 strategies: p% of timesteps start spans of l steps.
  • B. Representation Learning via Masked Prediction: The final objective combines masked and unmasked cross-entropy losses as L = αLm + (1 −α)Lu.Lm sums over masked timesteps, while Lu has the same form over t̸ ∈ M.
  • B. Representation Learning via Masked Prediction: When α = 0, training uses only unmasked timesteps and limits learning to mimicking the clustering model.This resembles acoustic modeling in hybrid speech recognition systems.
  • B. Representation Learning via Masked Prediction: When α = 1, masked-only prediction requires inferring unseen-frame targets from context, encouraging acoustic representations and long-range temporal structure.The paper hypothesizes that this setting is more resilient to cluster-target quality.

C. Learning with Cluster Ensembles · D. Iterative Refinement of Cluster Assignments

Cluster ensembles improve target construction by combining complementary unsupervised clusterings at different granularities, including through product quantization. Iterative refinement instead replaces initial targets with units discovered from learned representations.

  • C. Learning with Cluster Ensembles: Cluster ensembles combine multiple clustering models to provide complementary information for representation learning.The approach can use ensemble-generated target sequences within the proposed framework.
  • C. Learning with Cluster Ensembles: Different k-means codebook sizes produce targets spanning manner classes such as vowels and consonants to sub-phone states such as senones.This supplies targets at multiple granularities rather than relying on one clustering resolution.
  • C. Learning with Cluster Ensembles: The ensemble objective is analogous to multi-task learning, with tasks created by unsupervised clustering.The framework defines masked and unmasked losses for the ensemble-generated targets.
  • C. Learning with Cluster Ensembles: Product quantization partitions feature space into subspaces that are quantized separately, enabling k-means for high-dimensional and heterogeneous features.PQ is compatible with ensembling and accommodates subspaces whose scales differ substantially.
  • C. Learning with Cluster Ensembles: The theoretical target-space size under product quantization equals the product of all codebook sizes.This follows because each subspace contributes its own codebook choices.
  • D. Iterative Refinement of Cluster Assignments: Iterative refinement creates new cluster generations from learned latent representations rather than continuing to use clusters derived from raw acoustic features.A discrete latent model is trained over the learned representations, after which learning proceeds with the newly discovered units.

E. Implementation · III. RELATED WORK

HuBERT uses a wav2vec 2.0-style architecture with convolutional waveform encoding, masked BERT prediction, and codeword projection, then applies CTC fine-tuning for ASR. Related work spans generative, predictive, contrastive, discrete-target, and iterative pseudo-labeling approaches, with HuBERT positioned closest to DiscreteBERT and wav2vec 2.0.

  • E. Implementation: HuBERT follows wav2vec 2.0 with a convolutional waveform encoder, BERT encoder, projection layer, and code embedding layer.
  • E. Implementation: HuBERT has BASE, LARGE, and X-LARGE configurations; X-LARGE expands to about 1 billion parameters, while all use the same seven-layer waveform encoder.The encoder uses 512-channel layers with strides [5] [2] and kernel widths [10] [3] [2].
  • E. Implementation: The waveform encoder produces 20ms-framerate features at 16kHz, which are randomly masked before the BERT encoder outputs a feature sequence.The CNN encoder down-sampling factor is 320x.
  • E. Implementation: Codeword distributions use cosine similarity between projected features and codeword embeddings, with the logit scale τ set to 0.1.Cluster ensembles apply one projection matrix A(k) for each clustering model k.
  • E. Implementation: ASR fine-tuning uses CTC with the convolutional audio encoder frozen, replacing projection layers with a randomly initialized softmax layer.The target vocabulary contains 26 English characters, a space token, an apostrophe, and a CTC blank symbol.
  • III. RELATED WORK: Prior self-supervised speech work includes generative latent-variable models trained by likelihood maximization, using continuous, discrete, or sequential latent structures.
  • III. RELATED WORK: Prediction-based methods predict unseen content or contrast unseen frames with sampled negatives; some combine predictive and contrastive losses, often interpreted as mutual-information maximization.
  • III. RELATED WORK: HuBERT most closely relates to DiscreteBERT because both predict discrete targets in masked regions, but HuBERT uses raw waveforms instead of quantized inputs.HuBERT is also related to wav2vec 2.0, while extending iterative pseudo-label refinement to self-supervised masked prediction.

IV. EXPERIMENTAL DETAILS … C. Pre-Training

HuBERT pre-training uses large-scale LibriSpeech or Libri-light audio, iterative k-means targets, and specified training configurations across BASE, LARGE, and X-LARGE models. The setup includes low-resource fine-tuning partitions, masked-frame optimization, and detailed clustering and optimization schedules.

  • A. Data: Pre-training uses 960 hours of LibriSpeech or 60,000 hours of Libri-light, with fine-tuning on 10-minute, 1-hour, 10-hour, 100-hour, and 960-hour partitions.Both corpora derive from LibriVox English audiobook recordings.
  • B. Unsupervised Unit Discovery: HuBERT initially generates targets with k-means clustering using 100 clusters over 39-dimensional acoustic features from the 960-hour LibriSpeech set.The default k-means unit-discovery model is intentionally simple and represents each unit with an isotropic Gaussian sharing a scalar variance.
  • B. Unsupervised Unit Discovery: Subsequent iterations use 500-cluster k-means on intermediate HuBERT transformer features, fitted from a randomly sampled 10% of the data.HuBERT BASE transformer outputs are 768-dimensional, making full-dataset loading impractical.
  • B. Unsupervised Unit Discovery: Clustering uses scikit-learn MiniBatchKMeans with 10,000-frame minibatches and k-means++ initialization with 20 random starts.MiniBatchKMeans fits a minibatch of samples at a time.
  • C. Pre-Training: BASE trains for two iterations on 960 hours using 32 GPUs, with 250k steps initially and 400k steps subsequently using labels from the first model’s 6-th transformer layer.The maximum batch size is 87.5 seconds per GPU, and 100k steps take about 9.5 hours.
  • C. Pre-Training: LARGE and X-LARGE each train for one iteration on 60,000 hours using 128 and 256 GPUs, respectively, for 400k steps.They use labels clustered from the 9th transformer layer of the second-iteration BASE model, with per-GPU batches reduced to 56.25 and 22.5 seconds.
  • C. Pre-Training: All configurations use mask span l = 10 and select p = 8% of waveform-encoder frames as mask starts, with Adam optimization and linear warmup and decay.Peak learning rates are 5e-4/1.5e-3/3e-3 for BASE/LARGE/X-LARGE models.

D. Supervised Fine-Tuning and Decoding · E. Metrics of Target Quality

Supervised fine-tuning fixes the convolutional encoder, controls transformer freezing, and tunes optimization settings before language-model-fused decoding. Target quality is assessed by comparing frame-level k-means assignments with forced-aligned phonetic labels using purity and information-theoretic metrics.

  • D. Supervised Fine-Tuning and Decoding: Each model is fine-tuned on 8 GPUs using labeled splits, with batches capped at 200/80/40 seconds for BASE/LARGE/X-LARGE models.The convolutional waveform audio encoder remains fixed during fine-tuning.
  • D. Supervised Fine-Tuning and Decoding: Fine-tuning sweeps peak learning rate, schedule, training steps, freeze step, and waveform settings while controlling transformer freezing.The peak learning-rate sweep is [1e-5, 1e-4], and only the new softmax matrix is trained during frozen steps.
  • D. Supervised Fine-Tuning and Decoding: Language-model-fused decoding uses wav2letter++ beam search wrapped in Fairseq and searches decoding hyperparameters with Ax.Both n-gram and transformer language models trained on official Librispeech language-modeling data are considered.
  • E. Metrics of Target Quality: Frame-level forced-aligned phonetic transcripts from a hybrid ASR system are compared with k-means assignments through their estimated joint distribution.The joint distribution is estimated by counting aligned phonetic labels and k-means labels.
  • E. Metrics of Target Quality: Phone-normalized mutual information measures the percentage of phone-label uncertainty eliminated by observing k-means labels, with higher values indicating better clustering quality.The metrics section considers phone purity, cluster purity, and phone-normalized mutual information.
  • E. Metrics of Target Quality: Phone purity measures average phone purity within each k-means class and indicates better quality when comparing target-label sets with equal unit counts.It can be interpreted as frame-level phone accuracy when each k-means class receives its most likely phone label.
  • E. Metrics of Target Quality: Cluster purity measures the counterpart relationship from phone labels to k-means labels and generally decreases as the number of units increases.For equal unit counts, higher cluster purity indicates that frames from the same phone share k-means labels more often.

V. RESULTS · A. Main Results: Low- and High-Resource Setups

HuBERT performs strongly across low- and high-resource Librispeech fine-tuning settings, with performance improving as unlabeled data and model size increase. It surpasses several competing approaches, including DiscreteBERT and supervised or self-training methods, while matching the strongest wav2vec 2.0 pre-training results.

  • A. Main Results: Low- and High-Resource Setups: HuBERT is evaluated with 10 minutes, 1 hour, 10 hours, or 100 hours of labeled data in low-resource fine-tuning setups.The comparison includes semi-supervised and self-supervised methods from prior literature.
  • A. Main Results: Low- and High-Resource Setups: Increasing unlabeled data and model size improves HuBERT performance, demonstrating scalability across low-resource conditions.
  • A. Main Results: Low- and High-Resource Setups: HuBERT outperforms DiscreteBERT by a large margin in all reported setups despite using virtually identical masked-prediction objectives.The performance gap is attributed to waveform inputs preserving information lost during quantization.
  • A. Main Results: Low- and High-Resource Setups: HuBERT results are reported after fine-tuning on the full 960 hours of Librispeech data and compared with prior methods.Prior approaches using additional unpaired speech are organized as self-training, pre-training, or pre-training plus self-training.
  • A. Main Results: Low- and High-Resource Setups: HuBERT outperforms state-of-the-art supervised and self-training methods on the full Librispeech setup.
  • A. Main Results: Low- and High-Resource Setups: HuBERT is on par with the two best pre-training results, both based on wav2vec 2.0 contrastive learning, but trails methods combining pre-training and self-training.

B. Analysis: K-Means Stability · C. Analysis: Clustering Quality Across Layers and Iterations

The analyses assess k-means stability across features, cluster counts, and training-data sizes, then compare clustering quality across HuBERT layers and training iterations. HuBERT features substantially outperform MFCC, while clustering trends differ between iterations.

  • B. Analysis: K-Means Stability: K-means stability was evaluated with MFCC and 6th-layer HuBERT features, using each feature for target generation in the first and second HuBERT training iterations, respectively.The study varied cluster counts and training-data sizes to examine clustering behavior.
  • B. Analysis: K-Means Stability: 100 and 500 clusters were fitted on 1, 10, and 100 hours of LibriSpeech speech, with 10 trials per configuration and supervised PNMI reported on the combined development set.The reported results use the mean and standard deviation across trials.
  • B. Analysis: K-Means Stability: The k-means clustering was reasonably stable across the evaluated configurations, as indicated by the small standard deviations in supervised PNMI.Stability was assessed over different features, cluster numbers, and training-data sizes.
  • C. Analysis: Clustering Quality Across Layers and Iterations: 26 features from BASE-it1 and BASE-it2 were evaluated, covering Layer 0 and all 12 transformer layers, with K = 100, 500, and 1000 clusters fitted on 100 hours of speech.BASE-it1 and BASE-it2 denote the first- and second-iteration BASE HuBERT models.
  • C. Analysis: Clustering Quality Across Layers and Iterations: MFCC achieves (0.099, 0.335, 0.255) for (cluster purity, phone purity, PNMI) at K = 100 and (0.031, 0.356, 0.287) at K = 500.These values provide the baseline for Figure 2’s clustering-quality comparison.
  • C. Analysis: Clustering Quality Across Layers and Iterations: Both BASE-it1 and BASE-it2 features outperform MFCC on cluster purity, phone purity, and PNMI for the same number of clusters.The best BASE-it2 feature exceeds the best BASE-it1 on phone purity and PNMI but is slightly worse on cluster purity.
  • C. Analysis: Clustering Quality Across Layers and Iterations: BASE-it2 features generally improve across layers, whereas BASE-it1 performs best in middle layers around the 6th layer.The two iterations therefore exhibit different layer-wise clustering trends.

D. Ablation: The Importance of Predicting Masked Frames

The ablation compares predicting masked, all, or unmasked frames across clustering sources and qualities. Masked-only prediction is most important with poor cluster assignments, whereas better clusters reduce or reverse the cost of unmasked-frame losses.

  • Prediction target: The ablation compares masked-frame, all-frame, and unmasked-frame prediction, corresponding to α values of 1.0, 0.5, and 0.0.It evaluates MFCC k-means teachers with 50, 100, and 500 clusters, HuBERT-BASE-it1 layer-6 features, and supervised forced-alignment labels.
  • Prediction target: Masked-only loss achieves the best performance when learning from bad cluster assignments, while adding unmasked-frame loss produces significantly higher WERs.The results are reported for models pretrained for 100k steps and fine-tuned on the 10-hour Libri-light split.
  • Prediction target: As clustering quality improves, unmasked-frame losses become less harmful for BASE-it1-layer6 and can improve performance for chenone.This indicates that the masked-only advantage depends on the quality of the target cluster assignments.

E. Ablation: The Effect of Cluster Ensembles … WER (%)

HuBERT ablations examine cluster-ensemble targets and pre-training hyperparameters, while the conclusion reports broad benchmark competitiveness, gains from iterative cluster refinement, and scaling to a 1B-parameter model. The reported best hyperparameter configuration reaches 11.68% WER, and the largest model reduces test-other WER by up to 13% relatively.

  • E. Ablation: The Effect of Cluster Ensembles: HuBERT’s cluster-ensemble ablation compares k-means teachers with different cluster counts, denoted KM-{50,100,500}.A second setup trains k-means models on 117-dimensional spliced MFCC features and applies product quantization.
  • F. Ablation: Impact of Hyperparameters: p =8% is the optimal portion of frames selected as mask starts in the hyperparameter study.The study varies masking probability and effective batch size through the number of GPUs.
  • F. Ablation: Impact of Hyperparameters: Increasing batch size can significantly improve HuBERT performance.The finding is reported alongside experiments varying masking probability and effective batch size.
  • F. Ablation: Impact of Hyperparameters: 11.68% WER is achieved by the best model after longer pre-training consistently improves both k-means models with C={50, 100}.Table VII varies the number of HuBERT pre-training steps with p set to 6.5%.
  • WER (%): Iteratively refining k-means cluster assignments with learned latent representations from a previous iteration dramatically improves learned representation quality.This improvement is reported as part of the conclusion’s summary of HuBERT’s training approach and results.
  • WER (%): 13% relative WER reduction is achieved on the test-other subset when HuBERT scales to a 1B transformer model.The conclusion identifies scaling to the 1B model as a key result and proposes future work toward a single-phase training procedure.
Loading 2106.07447v1…