PaperlayerView exact revision on arXiv

Source-linked AI summary

Vector-Quantized Autoregressive Predictive Coding

Yu-An Chung, Hao Tang, James Glass

arXiv:2005.08392v1eess.AScs.CLcs.LGcs.SD

TL;DR

Why low APC self-supervised loss yields accessible phonetic and speaker information remains unclear. VQ-APC uses quantization to control capacity; experiments find that constrained models retain speaker information more than phonetic information, while some codes correspond strongly to phones.

  • Problem

    Why APC’s future-prediction objective produces accessible phonetic and speaker information remains unclear.

    Source: §1 · p. 1

  • Method

    VQ-APC uses vector quantization as a capacity bottleneck to study what information self-supervised future prediction preserves.

    Source: §1 · p. 1, §2.1 · p. 1

  • Results

    As codebook size shrinks, phone error worsens below its minimum at 512 while speaker error changes little; configuration {3} has normalized mutual information 0.406.

    Source: §3.3 · p. 3, §3.4 · p. 4

  • Takeaways & Limitations

    Mutual information estimates complement probing by supporting conclusions about absent phonetic information, not only information presence.

    Source: §3.4 · p. 4

    Models with multiple VQ layers struggled to fit the training set and performed worse than regular APC on phonetic classification; scheduled VQ was outside the paper’s scope.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Autoregressive Predictive Coding (APC), as a self-supervised objective, has enjoyed success in learning representations from large amounts of unlabeled data, and the learned representations are rich for many downstream tasks. However, the connection between low self-supervised loss and strong performance in downstream tasks remains unclear. In this work, we propose Vector-Quantized Autoregressive Predictive Coding (VQ-APC), a novel model that produces quantized representations, allowing us to explicitly control the amount of information encoded in the representations. By studying a sequence of increasingly limited models, we reveal the constituents of the learned representations. In particular, we confirm the presence of information with probing tasks, while showing the absence of information with mutual information, uncovering the model's preference in preserving speech information as its capacity becomes constrained. We find that there exists a point where phonetic and speaker information are amplified to maximize a self-supervised objective. As a byproduct, the learned codes for a particular model capacity correspond well to English phones.

1. Introduction

APC learns speech representations, but why low future-prediction loss yields accessible phonetic and speaker information remains unclear. The paper uses quantization to control capacity and study retained information, rather than to discover discrete speech units.

1. Introduction

  • Self-supervised speech representations are learned without additional labels and have outperformed surface features across several downstream tasks.The cited tasks include speech recognition, translation, speaker identification, and emotion recognition.Source: §1 · p. 1
  • APC representations contain accessible phonetic and speaker information, but the reason future-frame prediction produces them remains unclear.The study uses linear separability, or separability with a shallow network, to define information accessibility for downstream tasks.Source: §1 · p. 1
  • Vector quantization provides an explicitly quantifiable way to limit model capacity and investigate which information learned representations retain.The authors quantify capacity by the bits needed to transmit the codebook and code sequence.Source: §1 · p. 1
  • Unlike prior VQ work targeting acoustic-unit discovery or speech recognition, this study uses quantization to examine information in learned representations.Prior work explores VQ for acoustic unit discovery and automatic speech recognition.Source: §1 · p. 1

2. Proposed Models

APC predicts future acoustic frames from autoregressive context; VQ-APC inserts codebook quantization into that model, passing quantized vectors forward while retaining APC’s L1 prediction objective.

2.1. A review on APC

  • APC summarizes acoustic context with an autoregressive model to predict a frame n steps ahead, minimizing frame-wise L1 loss against future targets.The model may use a unidirectional RNN or Transformer decoder; trained hidden representations can later serve as downstream features.Source: §2.1 · pp. 1–2

2.2. VQ-APC

  • VQ-APC inserts quantization layers into APC, replacing a hidden vector with a codebook vector that is passed to the next layer.The illustrated example inserts a VQ layer between the first and second autoregressive layers; the training objective remains APC’s future-frame L1 loss.Source: §2.1 · p. 2, §2.2 · p. 2
  • The model uses Gumbel-Softmax with a straight-through estimator to select discrete codebook variables during training.The forward pass chooses the argmax code, while the backward pass uses gradients from the Gumbel-Softmax outputs.Source: §2.2 · p. 2
  • Codebook size and code dimension control how much information passes between layers, enabling a sequence of models with progressively constrained capacity.The authors use these increasingly limited models to study the constituents of the prediction task and learned representations.Source: §2.2 · p. 2

3. Experiments

The experiments examine VQ-APC under constrained capacity, probe its learned speech representations, visualize code–phone correspondence, and compare it with other self-supervised models. They test how VQ placement and codebook size relate to prediction loss and downstream phonetic and speaker classification.

3. Experiments

  • The experiments probe phone and speaker information, vary VQ placement and codebook size, visualize learned codes, and compare VQ-APC with other self-supervised models.The study uses phonetic and speaker classification as probes of the representations.Source: §3 · p. 2

3.1. Setup

  • Models are trained on 360 hours of LibriSpeech, then evaluated with frozen representations on WSJ phone and speaker classification using linear logistic regression.Phone error is reported on dev93; speaker classification is a 259-class task and serves as a sanity check for speaker and potentially correlated channel information.Source: §3.1 · p. 2

3.2. Preliminary VQ experiments

  • Third-layer VQ lowers phone error from 33.3 to 30.5 versus APC; quantized z(3) scores 30.8, close to h(3) at 30.5.Quantizing multiple layers makes fitting the training set difficult and yields worse phonetic classification than regular APC.Source: §3.2 · p. 3, Table 1 · p. 3

3.3. The constituents of the learned representations

  • Phone error reaches its minimum at codebook size 512, while speaker error changes little as the codebook shrinks; even size 64 retains gains over APC.Training loss rises as the codebook shrinks; the authors interpret the differing probe trends as a preference for retaining speaker over phonetic information for future prediction.Source: §3.3 · p. 3, Figure 2 · p. 3

3.4. Visualizations of learned codes

  • Phone–code correspondence strengthens at higher VQ layers and is strongest for {3}; normalized mutual information is 0.167, 0.285, and 0.406 for {1}, {2}, and {3}.The phone labels are used only for analysis and are not seen during training.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5. A comparison with other self-supervised models

  • VQ-APC reaches a phonetic classification error of 28.4, improving over APC (33.3) and MT-APC (30.5); its speaker classification error of 5.5 is on par with Mockingjay (5.1).It also outperforms the other compared self-supervised models on phonetic classification, despite using the same or smaller network.Source: §3.5 · p. 4, Table 2 · p. 4
  • Across the compared self-supervised models, all outperform log Mel on both tasks, regardless of classifier type.Table 2 reports phonetic and speaker classification results and notes each model’s neural architecture.Source: §3.5 · p. 4, Table 2 · p. 4

4. Conclusions

The conclusions report that VQ layers impose a bottleneck for studying learned representations, including their speech information and correspondence to English phones.

  • VQ layers impose a bottleneck in APC, and the paper uses it to study how learned representations change with model capacity.The experiments compare VQ configurations and codebook sizes.Source: §4 · p. 4
  • Probing and mutual-information analyses provide evidence about the presence and absence of phonetic and speaker information in the learned representations.Source: §4 · p. 4
  • As capacity becomes constrained, the model’s information-retention preference is examined, and learned codes for one capacity correspond well with English phones.Source: §4 · p. 4
Loading 2005.08392v1…