Source-linked AI summary
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Alexei Baevski, Steffen Schneider, Michael Auli
TL;DR
The paper addresses how to learn discrete representations of speech that support methods requiring discrete inputs. It introduces vq-wav2vec, which combines wav2vec-style context prediction with vector quantization and BERT pre-training, achieving new state-of-the-art results on TIMIT and WSJ.
Problem
The paper seeks discrete speech representations learned through context prediction rather than reconstruction so NLP algorithms requiring discrete inputs can be applied to speech.
Method
vq-wav2vec quantizes wav2vec-style dense audio representations using Gumbel-Softmax or k-means, then BERT is trained on the resulting discrete speech sequence.
Results
BERT pre-training with vq-wav2vec achieves new state-of-the-art results of 2.34 WER on WSJ and 11.64 PER on TIMIT.
Takeaways & Limitations
Discretization enables direct use of NLP algorithms for speech, including BERT and a standard sequence-to-sequence model over discrete audio tokens.
Abstract
from arXiv · showhide
We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.
1 INTRODUCTION
vq-wav2vec combines self-supervised context prediction with discretization to learn speech units, enabling NLP methods designed for discrete inputs. BERT over these discretized representations improves results on TIMIT and WSJ benchmarks.
- 1 INTRODUCTION: The introduction frames discrete speech learning as complementary to autoencoding and continuous self-supervised context prediction.Prior work includes both discrete-unit discovery through autoencoding and continuous representations learned by predicting context information.
- 1 INTRODUCTION: The paper learns discrete speech representations through context prediction rather than input reconstruction, combining two research directions.This design enables direct application of well-performing NLP algorithms to speech data.
- 1 INTRODUCTION: vq-wav2vec learns discrete representations of fixed-length audio segments using the wav2vec loss and architecture.The quantization choices include Gumbel-Softmax and online k-means clustering.
- 1 INTRODUCTION: BERT is trained on discretized unlabeled speech, and its representations are provided to a standard acoustic model.The pipeline applies BERT before acoustic-model transcription.
- 1 INTRODUCTION: BERT representations perform better than log-mel filterbank inputs and dense wav2vec representations on both TIMIT and WSJ benchmarks.The experiments also use a standard NLP sequence-to-sequence model for speech recognition over discrete audio tokens.
2 BACKGROUND
wav2vec learns audio representations by predicting future latent representations from aggregated context while distinguishing true future samples from distractors. BERT provides a separate transformer-based pre-training framework for building sequence representations from text.
- 2 BACKGROUND: wav2vec uses a self-supervised context-prediction task to learn audio representations with two convolutional neural networks.The encoder produces representations at each time step, while the aggregator combines multiple encoder outputs into context representations.
- 2 BACKGROUND: The model distinguishes a future representation from distractor samples drawn from a noise distribution using a contrastive loss.Future steps k = 1, . . . , K are scored through step-specific affine transformations and a sigmoid probability.
- 2 BACKGROUND: After training, wav2vec context representations replace log-mel filterbank features as inputs to the acoustic model.The loss is summed across different future step sizes.
- 2 BACKGROUND: BERT is an NLP pre-training approach that uses a transformer encoder and self-attention to build text representations.The original BERT combines masked language modeling with next sentence prediction.
3 VQ-WAV2VEC
vq-wav2vec adds a quantization module to wav2vec, converting dense speech features into discrete codebook representations while retaining future-step context prediction. It supports differentiable Gumbel-Softmax or online k-means selection and uses grouped quantization to reduce mode collapse.
- Architecture: vq-wav2vec inserts a quantization module between wav2vec feature extraction and aggregation to produce discrete audio representations for future-step prediction.The encoder maps speech to dense features, the quantizer produces discrete indices and reconstructed codebook vectors, and the aggregator receives those vectors.
- Quantization methods: Gumbel-Softmax selects codebook variables differentiably during training and uses the largest logit index at inference.The forward pass uses the argmax codeword, while backward propagation uses the Gumbel-Softmax gradient through a straight-through estimator.
- Quantization methods: Online k-means selects the codebook vector nearest to the input feature by Euclidean distance and back-propagates wav2vec loss through the selected representation.Its objective adds terms that move codebook vectors toward encoder outputs and encoder outputs toward codeword centroids.
- Multiple variable groups: Grouped quantization addresses mode collapse, in which only some codewords are used, and yields larger dictionaries with increased downstream performance.Codebook variables may be shared across groups, and the paper reports that sharing generally remains competitive with non-shared representations.
- Multiple variable groups: Grouped quantization partitions the dense feature vector into G groups and represents each group with an integer codebook index.Each group independently applies one of the two vector-quantization approaches, producing indices in [V]^G.
4 BERT PRE-TRAINING ON QUANTIZED SPEECH
The paper applies BERT pre-training to vq-wav2vec’s discretized speech tokens by predicting masked tokens from surrounding context. Because individual 10 ms tokens may be too easy to predict, it masks consecutive spans instead.
- BERT pre-training: BERT is trained on discretized speech by predicting masked input tokens from an encoding of their surrounding context.The resulting BERT representations are fed into an acoustic model for speech recognition.
- Span masking: Consecutive spans of 10 discretized speech tokens are masked because predicting a single token may be too easy.Starting indices are sampled from 5% of all tokens, and spans may overlap.
5 EXPERIMENTAL SETUP
The experiments pre-train vq-wav2vec and BERT on Librispeech, then evaluate speech representations and acoustic models on TIMIT phoneme classification and WSJ speech recognition. The setup includes full-scale and smaller ablation configurations.
- Pre-training data: 960h of Librispeech is used for general pre-training, producing 345M discretized tokens after vq-wav2vec training.A clean 100h subset is used for ablations and produces 39.9M tokens.
- Datasets: TIMIT provides a 5h phoneme-classification benchmark with 39 phonemes, while WSJ provides an 81h speech-recognition benchmark using 31 graphemes.The WSJ output inventory includes alphabetic characters, apostrophe, silence, and repeating-character tokens.
- Model configuration: The main vq-wav2vec and wav2vec models use 34 × 10^6 parameters, with an encoder stride totaling 160.The encoder has eight convolutional layers, while the aggregator has 12 layers with skip connections between subsequent blocks.
- Training: vq-wav2vec training uses 400k updates, predicts K = 8 future steps, samples 10 negatives, and crops approximately 9.3 seconds per example.The learning rate is warmed up for 500 steps, then annealed with a cosine schedule.
- Quantizer configurations: Gumbel-Softmax and k-means models use G = 2 groups and V = 320 variables per group.The Gumbel-Softmax temperature is annealed from 2 to 0.5, while the k-means setup uses γ = 0.25 for its VQ auxiliary loss.
- BERT configuration: BERT base uses 12 layers, dimension 768, inner dimension 3072, and 12 attention heads, trained for 250k updates on 128 GPUs.Its total batch size is 393k tokens, and each token represents 10ms of audio.
- Acoustic modeling: Acoustic models use wav2letter and are trained for 1,000 epochs on eight GPUs for both TIMIT and WSJ.WSJ decoding uses a lexicon and either a 4-gram KenLM or character-based convolutional language model.
6 RESULTS
Across WSJ, TIMIT, sequence-to-sequence speech recognition, and bitrate experiments, vq-wav2vec representations support strong performance and enable comparisons across quantization and compression settings. BERT pre-training is especially effective, while quantization choices and bitrate shape results.
- 6.1 WSJ SPEECH RECOGNITION: The WSJ experiments compare No LM, 4-gram LM, and character convolutional LM settings using log-mel filterbanks, wav2vec, and vq-wav2vec representations.For vq-wav2vec, the experiments include Gumbel-Softmax models with and without BERT.
- 6.1 WSJ SPEECH RECOGNITION: 2.34 WER on nov92 is a new state of the art for vq-wav2vec with BERT training on WSJ.The evaluation includes No LM, 4-gram LM, and character convolutional LM settings across development and test sets.
- 6.1 WSJ SPEECH RECOGNITION: Gumbel-Softmax and k-means perform relatively comparably on WSJ, with their differences disappearing after BERT training.K-means is better in the 4-gram LM setup before BERT, while Gumbel-Softmax is more accurate without a language model and BERT removes these differences.
- 6.2 TIMIT PHONEME RECOGNITION: 11.64 PER on TIMIT is a new state of the art, corresponding to a 21% error reduction over the previous best wav2vec result.The models are evaluated on the TIMIT phoneme-recognition task using the CNN-8L-PReLU-do0.7 architecture.
- 6.3 SEQUENCE-TO-SEQUENCE SPEECH RECOGNITION: A standard Big Transformer trained on vq-wav2vec-discretized Librispeech produces promising sequence-to-sequence results without BERT pre-training, though below the state of the art.The model uses a 4k BPE output vocabulary and is evaluated on Librispeech dev/test sets without a language model.
- 6.4 ACCURACY VS. BITRATE: Acoustic models using vq-wav2vec achieve the best TIMIT results across most tested bitrate settings compared with Codec23, Opus, MP3, and Ogg Vorbis.The compression study varies codebook groups and variables over bitrates from 0.53 kbit/s to 33.03 kbit/s.
7 CONCLUSION
vq-wav2vec quantizes unlabeled audio through self-supervised learning, enabling discrete-input algorithms for speech. BERT pre-training improves benchmark performance, while future work extends the approach to other algorithms and transcription fine-tuning.
- vq-wav2vec quantizes unlabeled audio into discrete representations suitable for algorithms requiring discrete inputs.
- BERT pre-training improves the state of the art on the WSJ and TIMIT benchmarks.
- Future work will apply other discrete-input algorithms to audio and explore masking parts of the continuous input during self-supervised pre-training.
- Another proposed direction is fine-tuning the pre-trained model to output transcriptions instead of feeding its features to a custom ASR model.
APPENDIX A NUMBER OF VARIABLES VS. GROUPS
The appendix studies how the number of variables and groups affects vq-wav2vec codebooks and performance. Multiple groups outperform a single group with many variables, where only a small fraction of possible codewords survives.
- Multiple groups are beneficial compared to a single group with a large number of variables.
- With a single group and many variables, only a small number of codewords survive.
- The appendix evaluates phoneme error rate on the TIMIT development set using models trained on Libri100.
- The codeword analysis compares used-codeword fractions with the theoretically possible codeword count V^G.