Source-linked AI summary
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
Shaoshi Ling, Yuzong Liu
TL;DR
Speech representation learning offers a way to use unlabeled data for ASR by pretraining features before fitting on labeled data. DeCoAR 2.0 combines Transformer encoding, vector quantization, and reconstruction-based training; in LibriSpeech, 10 hours of labeled data outperformed 960-hour filterbank training.
Problem
Speech representation learning provides a way to use unlabeled data to learn features before training an ASR model on labeled data.
Source: §1 · p. 1Method
DeCoAR 2.0 uses a Transformer encoder, vector quantization between encoding and reconstruction, and an objective combining masked reconstruction with VQ diversity loss.
Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 2Results
6.7%/8.5% relative WER improvements on test-clean/test-other over 960-hour filterbank training were reported with 10 hours of labeled data; DeCoAR 2.0 was also comparable to wav2vec 2.0 across SSL conditions.
Source: §4.2 · p. 4Takeaways & Limitations
The findings support DeCoAR 2.0 features for ASR in the reported extreme data-limited semi-supervised settings.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recent success in speech representation learning enables a new way to leverage unlabeled data to train speech recognition model. In speech representation learning, a large amount of unlabeled data is used in a self-supervised manner to learn a feature representation. Then a smaller amount of labeled data is used to train a downstream ASR system using the new feature representations. Based on our previous work DeCoAR and inspirations from other speech representation learning, we propose DeCoAR 2.0, a Deep Contextualized Acoustic Representation with vector quantization. We introduce several modifications over the DeCoAR: first, we use Transformers in encoding module instead of LSTMs; second, we introduce a vector quantization layer between encoder and reconstruction modules; third, we propose an objective that combines the reconstructive loss with vector quantization diversity loss to train speech representations. Our experiments show consistent improvements over other speech representations in different data-sparse scenarios. Without fine-tuning, a light-weight ASR model trained on 10 hours of LibriSpeech labeled data with DeCoAR 2.0 features outperforms the model trained on the full 960-hour dataset with filterbank features.
1. INTRODUCTION
The introduction frames self-supervised speech representation learning as a way to use unlabeled data, then presents DeCoAR 2.0 with Transformer encoding, vector quantization, and a combined reconstruction and diversity objective.
1. INTRODUCTION
- Self-supervised representation learning uses proxy tasks to obtain labels from unlabeled data, then trains a downstream model using the learned representations and labeled data.The representation and downstream task blocks can optionally be fine-tuned together.Source: §1 · p. 1
- Speech representation learning uses contrastive objectives to distinguish future or masked frames from negatives, or reconstructive objectives to predict masked or future acoustic slices from context.APC uses autoregressive reconstruction; other speech methods mask acoustic slices for reconstruction.Source: §1 · p. 1
- Vector quantization is presented as orthogonal to contrastive and reconstructive learning, with uses including linguistic-unit discovery, limiting model capacity, and enabling NLP algorithms.Prior methods use VQ for different purposes, including controlling encoded information and supporting direct use of NLP algorithms.Source: §1 · p. 1
- DeCoAR 2.0 replaces DeCoAR’s LSTM encoder with a Transformer, adds vector quantization, and combines masked reconstruction with VQ diversity loss.These are the paper’s stated contributions.Source: §1 · p. 1
2. RELATED WORK
The related work describes DeCoAR’s contextual reconstruction approach and contrasts it with wav2vec 2.0 and VQ-APC, which combine quantization with different training objectives.
2.1. An Overview on DeCoAR
- DeCoAR uses bidirectional LSTMs to encode context, then reconstructs frames in each acoustic slice from concatenated forward and backward states using position-dependent feed-forward networks.Its loss is calculated over all possible slices in the sequence.Source: §2.1 · p. 2
2.2.1. wav2vec 2.0
- wav2vec 2.0 applies vector quantization and masked-LM-style training, while nuisance conditions can let contrastive codebooks encode noise or voice activity.Frame reconstruction is proposed as a way to use all input-feature information to guide robust representations.Source: §2.2.1 · p. 2
2.2.2. VQ-APC
- VQ-APC inserts quantization before frame prediction; its reported APC/reconstruction objective outperformed CPC/contrastive training under the same conditions, and its codes correlated with phoneme paths.The authors describe VQ as a way to control the information needed to encode speech.Source: §2.2.2 · p. 2
3. PROPOSED FRAMEWORK
DeCoAR 2.0 uses a Transformer encoder, vector quantization, and reconstruction to learn speech representations from masked acoustic frames. Its training combines reconstruction and codebook-diversity objectives, and downstream ASR uses the frozen encoder.
3. PROPOSED FRAMEWORK
- The framework maps masked acoustic frames through a Transformer encoder and vector quantizer, then reconstructs the original frames from the quantized representations.The paper describes these as the encoder, quantization, and reconstruction modules.Source: §3 · p. 2
3.1. Encoder Module
- The encoder replaces forward/backward LSTMs with Transformer blocks, preceded by a convolutional layer that provides implicit relative positional encoding.The masking strategy replaces nonoverlapping spans of K consecutive frames, masking about 40% of frames overall.Source: §3.1 · p. 2
3.2. Quantization Module
- The quantization module selects codebook entries for encoder representations and uses a straight-through Gumbel-Softmax estimator to make selection trainable.At inference, it selects the highest-logit entry from each codebook.Source: §3.2 · pp. 2–3
3.3. Training Objective
- Training combines masked-frame ℓ1 reconstruction loss with a diversity loss that encourages equal use of codebook entries, weighted by α.The diversity loss maximizes the entropy of the averaged softmax distribution over entries in each codebook.Source: §3.3 · p. 3
3.4. Semi-supervised Speech Recognition with DeCoAR 2.0
- For downstream ASR, the quantization and reconstruction modules are removed, the encoder is frozen, and only the ASR model parameters are trained.The frozen representations can feed a hybrid acoustic model or an end-to-end ASR system.Source: §3.4 · p. 3, Figure 2 · p. 3
4. EXPERIMENTAL SETUP AND RESULTS
On LibriSpeech, DeCoAR 2.0 is evaluated across labeled-data conditions using learned speech representations and supervised filterbank baselines. With 10 hours of labels, it outperforms the full-960-hour filterbank system and is comparable to wav2vec 2.0 across semi-supervised conditions.
4.2. Semi-supervised Speech Recognition Experimental Results
- The evaluation varies labeled LibriSpeech data from 1 to 100 hours, with the 1- and 10-hour subsets randomly drawn from train-clean-100.The downstream acoustic models use CTC; learned-representation systems use two-layer bidirectional LSTMs, while filterbank systems use six-layer bidirectional LSTMs.Source: §4 · p. 3, §4.2 · p. 4
4.1. Pretrain DeCoAR 2.0 Model using Unlabeled Data
- Pretraining uses all 960 hours of LibriSpeech as unlabeled data, with 80-dimensional log-mel features and a 12-block Transformer encoder.The model also uses 8 attention heads, 16 GPUs, and 150 training epochs.Source: §4.1 · pp. 3–4
4.2. Semi-supervised Speech Recognition Experimental Results
- 6.7%/8.5% relative WER improvements on test-clean/test-other: with 10 hours of labels, DeCoAR 2.0 outperforms the full-960-hour filterbank system.The corresponding 10-hour DeCoAR 2.0 WERs are 5.43%/13.27% on test-clean/test-other.Source: Table 1 · p. 4, §4.2 · p. 4
- DeCoAR 2.0 is comparable to wav2vec 2.0 across SSL conditions, and both perform favorably compared with VQ-APC and DeCoAR.The representation comparison uses pretrained features directly, without fine-tuning representation-learning layers.Source: §4.2 · p. 4
- The VQ ablation confirms that adding the VQ module benefits ASR; the authors hypothesize it limits capacity and prioritizes linguistic or phonetic information.They suggest it may deemphasize speaker traits and acoustic conditions.Source: §4.2 · p. 4, Table 2 · p. 4
5. CONCLUSION
DeCoAR 2.0 combines a Transformer encoder with vector quantization before reconstruction. In data-limited semi-supervised settings, 10 hours of labels achieve performance on par with a 960-hour filterbank system, and results are comparable to wav2vec 2.0; future work includes noisy conditions and other downstream ASR systems.
5. CONCLUSION
- DeCoAR 2.0 combines a deep Transformer encoder with a vector quantization module before reconstruction.It is presented as an improved speech representation approach based on DeCoAR and vector quantization.Source: §5 · p. 4
- With 10 hours of labeled data in extreme data-limited semi-supervised conditions, DeCoAR 2.0 performs on par with a 960-hour conventional filterbank system.It also performs comparably to wav2vec 2.0 across the semi-supervised scenarios.Source: §5 · p. 4
- Future work will explore noisy and adverse real-world data and extend evaluation to neural transducers and other end-to-end ASR systems.Source: §5 · p. 4