Source-linked AI summary
Self-supervised Learning with Random-projection Quantizer for Speech Recognition
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, Yonghui Wu
TL;DR
Speech self-supervised learning must connect continuous speech with discrete prediction targets while accommodating architectures such as low-latency streaming models. BEST-RQ uses a fixed random-projection quantizer to generate labels for masked speech and trains ASR encoders to predict them. It matches prior non-streaming LibriSpeech results and improves over wav2vec 2.0 and w2v-BERT on streaming and multilingual tasks.
Problem
Combining learned speech representations with self-supervised learning can constrain model architectures and increase design complexity.
Method
BEST-RQ masks speech and trains an ASR encoder to predict discrete labels from a fixed random-projection quantizer separated from the encoder.
Results
BEST-RQ matches prior non-streaming LibriSpeech results and outperforms wav2vec 2.0 and w2v-BERT on streaming LibriSpeech and multilingual tasks.
Takeaways & Limitations
The results support effective self-supervised learning without representation learning and show benefits for streaming and multilingual ASR.
Takeaways & Limitations
Random initialization causes quality variation across runs, especially with smaller pre-training and fine-tuning datasets.
Abstract
from arXiv · showhide
We present a simple and effective self-supervised learning approach for speech recognition. The approach learns a model to predict the masked speech signals, in the form of discrete labels generated with a random-projection quantizer. In particular the quantizer projects speech inputs with a randomly initialized matrix, and does a nearest-neighbor lookup in a randomly-initialized codebook. Neither the matrix nor the codebook is updated during self-supervised learning. Since the random-projection quantizer is not trained and is separated from the speech recognition model, the design makes the approach flexible and is compatible with universal speech recognition architecture. On LibriSpeech our approach achieves similar word-error-rates as previous work using self-supervised learning with non-streaming models, and provides lower word-error-rates and latency than wav2vec 2.0 and w2v-BERT with streaming models. On multilingual tasks the approach also provides significant improvement over wav2vec 2.0 and w2v-BERT.
1. Introduction
BEST-RQ addresses architectural and complexity limitations in speech self-supervised learning by replacing learned representation targets with fixed random-quantizer labels. It masks speech and trains an ASR encoder to predict labels for masked regions.
- Self-supervised learning can improve speech recognition when supervised data is limited, supporting low-resource languages and domains.
- Combining representation learning with self-supervised learning can constrain model architecture and increase design complexity.Representation models may require future context, whereas downstream low-latency models may prohibit it.
- BEST-RQ masks speech signals and trains the speech-recognition encoder to predict labels generated by a random-projection quantizer.
- The quantizer uses a random matrix and random codebook whose parameters remain fixed during self-supervised learning.
- BEST-RQ achieves similar LibriSpeech results to prior non-streaming work, improves over prior approaches with streaming models, and gains further on multilingual tasks.
- Representation-learning quality and self-supervised-learning quality are not inherently aligned, motivating self-supervised learning without representation learning.
2. Related Work
Prior speech self-supervised methods commonly learn representations or quantized representations to bridge continuous speech and discrete text-like targets. BEST-RQ instead uses a random-projection quantizer and trains the encoder to predict its labels.
- Earlier methods use representation learning, quantization, or both to connect continuous speech signals with discrete text tokens.
- BEST-RQ projects speech into a random codebook and assigns each input the nearest codebook vector's discrete label.
- The approach separates quantization from the speech-recognition model rather than learning the quantizer jointly with the model.
- Unlike VQ-VAE-based approaches, BEST-RQ uses a randomly initialized quantizer for BERT-style speech pre-training.
3. Self-supervised Learning with Random-projection Quantizer
BEST-RQ uses a fixed random-projection quantizer to create discrete speech targets, then trains ASR encoders to predict masked targets. The separated design supports both non-streaming and streaming architectures.
- 3.1. Random-projection Quantizer: BEST-RQ maps speech signals to discrete labels with a randomly initialized matrix and codebook, both fixed during pre-training.
- 3. Self-supervised Learning with Random-projection Quantizer: Masking samples frames with fixed probability, spans a fixed length, and replaces masked speech with Gaussian noise.
- 3.1. Random-projection Quantizer: For each speech vector, the quantizer projects the input and selects the nearest normalized codebook vector as its label.
- 3. Self-supervised Learning with Random-projection Quantizer: An ASR encoder predicts quantized labels for masked speech, with a softmax layer added during pre-training and removed during fine-tuning.
- 3.2.2. STREAMING MODELS: The quantizer's independence from the encoder permits different encoder architectures, including non-streaming and streaming models.
- 3.2.2. STREAMING MODELS: Streaming pre-training predicts masked labels from past context, while non-streaming pre-training can also benefit streaming models.
- 3.4. Understanding the Effectiveness of the Random-projection Quantizer: Compared with VQ-VAE quantization, the random-projection quantizer has poorer quantization quality but remains effective for self-supervised learning.
4. Experiments
The experiments include self-supervised pre-training on LibriSpeech with streaming and non-streaming models, plus multilingual evaluation and quantizer analysis.
- Experiments evaluate LibriSpeech with non-streaming and streaming models and assess multilingual tasks with non-streaming models.They also compare random-projection quantization with quantization learned using VQ-VAEs.
4.1. LibriSpeech
BEST-RQ is evaluated with matched 0.6B LibriSpeech architectures for non-streaming and streaming pre-training. It matches previous non-streaming WERs and outperforms wav2vec 2.0 and w2v-BERT on streaming WER and latency.
- Setup: The experiments use LibriLight for pre-training and LibriSpeech’s 960-hour training set for fine-tuning.Inputs are 80-dimensional log-mel filter-bank coefficients with 10 ms frame stride.
- Non-streaming models: The 0.6B non-streaming model uses two convolution layers followed by 24 Conformer layers.The convolution layers provide four-times temporal-dimension reduction.
- Non-streaming models: BEST-RQ achieves similar WERs to previous best LibriSpeech results with and without a language model.The comparison uses the same architecture and a similar optimization strategy as prior work.
- Streaming models: The streaming architecture uses three lower Conformer layers, two-times temporal reduction, and 20 upper Conformer layers.The model is scaled to 0.6B for consistency with the non-streaming experiments.
- Streaming models: Relative latency compares matched-word prediction times against a baseline, with negative values indicating lower latency for the compared model.The measurement averages timing differences across matched words and utterances.
- Streaming models: BEST-RQ outperforms wav2vec 2.0 and w2v-BERT on both WER and latency in streaming and non-streaming pre-training comparisons.Streaming pre-training provides the most significant latency reduction among the self-supervised setups.
4.2. Multilingual Tasks
BEST-RQ improves multilingual speech recognition across low- and high-resource evaluations. Gains are reported against w2v-BERT and wav2vec 2.0, with larger relative improvement in some languages and settings.
- MLS-10hrs: MLS-10hrs evaluates few-shot learning across eight languages using 10 hours of labeled training data.The experiments use XLS-R-U for pre-training and compare results on the MLS 10-hour and full supervised sets.
- MLS-10hrs: BEST-RQ reduces average MLS-10hrs WER by 3% relative compared with the w2v-BERT baseline.The baseline already outperforms the cited XLS-R(2B) result.
- MLS-10hrs: Fine-tuning on MLS-full provides more relative improvement than fine-tuning on MLS-10hrs.The passage reports that BEST-RQ performs especially better than w2v-BERT for Portuguese and Polish.
- Voice Search: BEST-RQ outperforms w2v-BERT and wav2vec 2.0 by 9% and 5% relatively on the Voice Search evaluation.BEST-RQ outperforms w2v-BERT on all evaluated languages.
4.3. Analyzing Quantization Quality
The analysis separates quantizer representation quality from self-supervised learning quality. Although Transformer-based VQ-VAE learns better representations, all tested quantizers produce similar self-supervised ASR WERs.
- Quantizer comparison: The analysis compares random-projection, projection-based VQ-VAE, and Transformer-based VQ-VAE quantizers.Trained quantizers use the full 960-hour LibriSpeech audio-only dataset.
- Representation quality: Transformer-based VQ-VAE provides significantly better direct ASR performance than the random-projection and projection-based VQ-VAE quantizers.This indicates that the Transformer-based quantizer learns a better representation.
- Self-supervised learning quality: All three quantizers lead to similar WERs when used for self-supervised learning.The result indicates that quantizer representation quality and self-supervised learning quality are not inherently aligned.
4.4. Analyzing the Effect of Pre-training Data Size
The study examines whether better quantization quality improves self-supervised learning when pre-training data is limited. The random-projection quantizer initially trails the trained Transformer VQ-VAE quantizer but catches up as pre-training data increases.
- Motivation: The comparison tests whether sufficient pre-training data can mitigate the performance gap associated with lower quantization quality.
- Quantizer quality: The table reports that random-projection and projection-based VQ-VAE quantizers produce similar self-supervised-learning WERs despite the Transformer-based quantizer performing better as a direct ASR input.The direct ASR task uses a 25M model, whereas the self-supervised setup uses a 0.6B model.
- Results: At low pre-training-data amounts, the random-projection quantizer performs worse than the trained Transformer VQ-VAE quantizer.
- Results: As pre-training data increases, the random-projection quantizer catches up with the Transformer-based VQ-VAE quantizer.
- Experimental setup: The experiment compares random-projection and Transformer-based VQ-VAE quantizers across different pre-training data sizes.The Transformer VQ-VAE quantizers were trained with 1/64, 4/64, 16/64, or 64/64 of LibriLight data, while the random quantizer remained untrained.
5. Conclusions and Discussions
BEST-RQ uses fixed random-projection quantization to support self-supervised speech recognition without requiring learned speech representations. It matches prior non-streaming LibriSpeech results and improves over wav2vec 2.0 and w2v-BERT in streaming and multilingual settings, while retaining initialization and convergence limitations.
- Conclusion: BEST-RQ masks speech signals and trains speech recognition models to predict discrete labels generated by a fixed random-projection quantizer.
- Results: BEST-RQ achieves similar WERs to existing state-of-the-art self-supervised results on LibriSpeech with non-streaming models.
- Results: BEST-RQ outperforms wav2vec 2.0 and w2v-BERT on LibriSpeech streaming models and multilingual non-streaming tasks.
- Design implications: Separating the quantizer from the speech recognition model removes the requirement for representation learning and supports a simpler framework.
- Codebook utilization: Higher codebook utilization correlates strongly with better pre-training quality, while random initialization can vary utilization across runs.
- Limitations: Random initialization causes greater pre-training variance when pre-training and fine-tuning datasets are smaller.
- Limitations: Non-streaming models usually require 50% more steps to reach final convergence, although this was not observed with streaming models.
- Limitations: Results may depend on the quantizer’s initialization algorithm because its randomly initialized parameters remain fixed during training.