PaperlayerView exact revision on arXiv

Source-linked AI summary

Wav2vec-C: A Self-supervised Model for Speech Representation Learning

Samik Sadhu, Di He, Che-Wei Huang, Sri Harish Mallidi, Minhua Wu, Ariya Rastrow, Andreas Stolcke, Jasha Droppo, Roland Maas

arXiv:2103.08393v2eess.AScs.LGcs.SD

TL;DR

Self-supervised speech pre-training faces codebook-learning difficulties, including locally optimal codebooks that may not generalize across datasets or architectures. Wav2vec-C adds a consistency network to reconstruct input features from quantized representations. After RNN-T fine-tuning, it achieves a 1.4% relative WER reduction over baseline, compared with 0.7% for wav2vec 2.0.

  • Problem

    Wav2vec 2.0 can yield locally optimal codebooks, and its codebook-learning methodology might not generalize across datasets and model architectures.

    Source: §2.2 · pp. 1–2

  • Method

    Wav2vec-C combines contrastive learning with a consistency network that reconstructs input features from quantized encodings.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    1.4% relative WER reduction over baseline for wav2vec-C after RNN-T fine-tuning, compared with 0.7% for wav2vec 2.0.

    Source: §5 · p. 4

  • Takeaways & Limitations

    ASR robustness is correlated with codebook diversity, and wav2vec-C achieved full codebook utilization in the reported experiments.

    Source: §5 · p. 4, §4.1 · p. 4

Abstract

from arXiv · show

Wav2vec-C introduces a novel representation learning technique combining elements from wav2vec 2.0 and VQ-VAE. Our model learns to reproduce quantized representations from partially masked speech encoding using a contrastive loss in a way similar to Wav2vec 2.0. However, the quantization process is regularized by an additional consistency network that learns to reconstruct the input features to the wav2vec 2.0 network from the quantized representations in a way similar to a VQ-VAE model. The proposed self-supervised model is trained on 10k hours of unlabeled data and subsequently used as the speech encoder in a RNN-T ASR model and fine-tuned with 1k hours of labeled data. This work is one of only a few studies of self-supervised learning on speech tasks with a large volume of real far-field labeled data. The Wav2vec-C encoded representations achieves, on average, twice the error reduction over baseline and a higher codebook utilization in comparison to wav2vec 2.0

1. Introduction

The paper introduces wav2vec-C to regularize codebook learning and maintain consistency between learned representations and input features. It studies self-supervision with real-world far-field speech and substantial labeled data, while limiting model size for low-latency ASR.

1. Introduction

  • Wav2vec-C adds regularization that reconstructs input features from discrete speech representations, maintaining consistency between learned representations and network inputs.The paper contrasts this with wav2vec 2.0, where contrastive loss on discretized codes drives learning, including the codebook.Source: §1 · p. 1
  • The study uses real-world far-field voice-query speech with SNRs from -40 to 50 dB, unlike much prior work using clean read or simulated noisy speech.The paper cites clean read speech and simulated noisy speech as common alternatives.Source: §1 · p. 1
  • The authors explore self-supervised pre-training with 1k hours of labeled data, where prior work had observed decreasing effectiveness as labeled-data volume increases.The paper notes prior findings of usefulness with little labeled data and declining effectiveness as labeled data increases [10, 7].Source: §1 · p. 1
  • The model is kept small to facilitate low-latency production ASR, and the study compares quantization variants for robustness and codebook utilization.The paper frames its size choice against the trend toward exceedingly large self-supervised models.Source: §1 · p. 1

2. The Wav2vec-C Model

The model builds on wav2vec 2.0’s masked-encoding contrastive learning but addresses codebooks that can settle into locally optimal patterns. Wav2vec-C adds a reconstruction network that encourages quantized codes to retain input-feature information.

2.1. Summary of wav2vec 2.0

  • Wav2vec 2.0 encodes log-STFT inputs, masks some embeddings, and trains contextual representations to contrast with quantized embeddings.Its contrastive objective depends on diverse codes associated with underlying speech units, but the formulation can yield locally optimal codebooks.Source: §2.1 · p. 1, §2.2 · p. 1

2.2. Wav2vec-C

  • The paper identifies speech-versus-nonspeech and fixed-temporal-location codebooks as locally optimal patterns, and notes that repeated query terms at fixed times contributed to the latter.The authors warn that wav2vec 2.0’s codebook-learning method might not generalize to other datasets or architectures.Source: §2.2 · pp. 1–2
  • Wav2vec-C adds a consistency network that reconstructs input features from quantized encodings, encouraging the latent space to preserve meaningful information.The authors describe the approach as integrating ideas from wav2vec 2.0 and VQ-VAE.Source: §2.2 · p. 2

2.3. Encoder network (f)

The encoder is a three-layer LSTM with 768 hidden dimensions; its gradients are scaled to help stabilize the codebook during training.

2.3. Encoder network (f)

  • The encoder uses three LSTM layers with 768 hidden dimensions, and scales encoder gradients by 0.1 to help stabilize the codebook.The paper states this gradient scaling follows wav2vec 2.0.Source: §2.3 · p. 2

2.4. Vector quantization (q)

The model uses product quantization to split encoded representations across codebooks, then concatenates their selected codes. It describes Gumbel-softmax and k-means as quantization variants, with k-means training using a straight-through estimator and two loss terms.

2.4. Vector quantization (q)

  • Product quantization splits each 768-dimensional encoding across two codebooks of 320 codes, each with dimension 384, then concatenates the selected codes.Each split selects one code from its corresponding codebook.Source: §2.4 · p. 2

2.4.1. Gumbel-softmax [14]

  • The paper identifies Gumbel-softmax as one of its two vector-quantization techniques.The supplied description specifies that the experiments use two VQ techniques but does not describe this variant’s operation.Source: §2.4 · p. 2

2.4.2. K-means [12]

  • For k-means quantization, a straight-through estimator copies gradients to the continuous encoding, while two loss terms pull quantized and continuous representations together and encourage commitment.The second term is the commitment loss; β is set to 0.25 as reported in.Source: §2.4.2 · p. 2

2.5. Masking

Masking removes portions of the continuous speech encodings before they enter the context network. That network compares context representations with quantized encodings using a contrastive score and sampled negatives.

2.5. Masking

  • Five masks cover an average of 40% of encoded frames, with each mask at most 16% of the utterance length.The masks are applied to continuous encodings before the context network.Source: §2.5 · p. 2

2.6. Context network (g)

  • The context network uses five transformer layers with dimension 1024, feed-forward dimension 4096, 16 attention heads, and sinusoidal positional embeddings.Source: §2.6 · p. 2
  • The contrastive score compares context representations with quantized encodings using cosine similarity and 50 uniformly sampled utterance-level negatives.The temperature is updated as proposed in.Source: §2.6 · p. 2

2.7. Consistency network (r)

The consistency network maps quantized embeddings to consistency vectors and trains them to match the input features using an L2 distance.

2.7. Consistency network (r)

  • A three-layer LSTM maps quantized embeddings Ẑ to consistency vectors S.Source: §2.7 · p. 2
  • The consistency vectors are trained to match input features X by minimizing their L2-normed distance.Source: §2.7 · p. 2
  • The network’s input is the sequence of quantized embeddings, and its output is a sequence of consistency vectors.Source: §2.7 · p. 2

2.8. Loss

Training combines the primary contrastive loss, a codebook loss, and a weighted consistency loss. The codebook loss differs by quantization type, while the consistency weight distinguishes wav2vec 2.0 from full wav2vec-C.

2.8. Loss

  • The total loss is the primary contrastive loss plus the codebook loss and γ-weighted consistency loss.Source: §2.8 · p. 3
  • γ = 0 ignores consistency loss and yields wav2vec 2.0, while γ = 1 gives wav2vec-C in full effect.Source: §2.8 · p. 3
  • For Gumbel-softmax VQ, the codebook loss is α times the diversity loss on the assignment distribution.Source: §2.8 · p. 3
  • The diversity-loss weight α helps avoid codebook collapse; α = 1.5 was suitable in the experiments, while k-means VQ uses the k-means loss.Source: §2.8 · p. 3

3. Experimental Setup

The experiments use in-house far-field English speech, with self-supervised encoders evaluated in RNN-T models against an otherwise matched un-pretrained baseline. The setup includes 10k hours of unlabeled training data and 1k hours of transcribed training data.

3.1. Data sets

  • The study evaluates pre-training on real-world far-field English voice commands and queries collected in home environments, with SNRs from -40 to 50 dB.Source: §3.1 · p. 3

3.1.1. Training data

  • Training data comprise 10k hours of unlabeled speech and 1k hours of transcribed English from native and non-native speakers.Source: §3.1.1 · p. 3

3.1.2. Test data

3.2. Recurrent Neural Network Transducer (RNN-T) Model

  • Pre-trained wav2vec-C and wav2vec 2.0 initialize the speech encoder of an RNN-T model.Source: §3.2 · p. 3

3.2.1. Pre-trained RNN-T

  • The pre-trained encoder uses the context network’s output, retains its five transformer layers after three LSTM layers, and is fine-tuned during RNN-T training.The masking module is removed; the RNN-T uses a two-layer, 1024-unit LSTM prediction network and a single-layer feed-forward joint network.Source: §3.2.1 · p. 3

3.2.2. Baseline RNN-T

  • The baseline uses the same RNN-T architecture but does not pre-train the speech encoder.Source: §3.2.2 · p. 3

3.3. Training details

  • Four self-supervised models are trained, then used to replace the speech encoder in RNN-T models.Source: §3.3 · p. 3
  • Self-supervised models train for 100k steps and RNN-T models for 60k steps, using 30 minutes and an average of one hour of speech per step, respectively.Source: §3.3 · p. 3

4. Results

Wav2vec-C (GS) improves relative word error rate on clean test sets but trails wav2vec 2.0 on noisy sets. Its codebook is fully utilized, while k-means codebooks trade lower utilization for greater noisy-set robustness.

4. Results

  • 1.6% rWERR on SNR20 and 1.2% on SNR16 for wav2vec-C (GS), but it underperforms wav2vec 2.0 on noisy sets.Table 2 compares rWERR for pre-trained RNN-T models across test sets.Source: Table 2 · p. 4, §4 · p. 4

4.1. Discussions on codebook utilization

  • The k-means codebook uses only a small fraction of codes but is more robust than Gumbel-softmax models on noisy sets, particularly N2.The authors note that lower codebook diversity may suit applications where robustness is important.Source: §4.1 · p. 4

5. Conclusions

Wav2vec-C combines wav2vec 2.0 and VQ-VAE ideas and is evaluated with real-world far-field noisy data and 1k hours of supervised ASR training. After RNN-T fine-tuning, it achieves greater average relative WER reduction than wav2vec 2.0, and ASR robustness is reported to correlate with codebook diversity.

5. Conclusions

  • Wav2vec-C combines wav2vec 2.0 and VQ-VAE ideas to address codebook utilization difficulties observed for wav2vec 2.0.The model is evaluated using real-world far-field noisy data.Source: §5 · p. 4
  • The authors report that ASR robustness is correlated with codebook diversity.They describe this observation as validating the motivation for the wav2vec-C architecture.Source: §5 · p. 4
Loading 2103.08393v2…