PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised speech representation learning using WaveNet autoencoders

Jan Chorowski, Ron J. Weiss, Samy Bengio, Aäron van den Oord

arXiv:1901.08810v2cs.LGeess.ASstat.ML

TL;DR

The paper addresses how to learn speech representations that preserve phonetic content while discarding speaker and low-level variation without labeled data. It compares bottlenecked autoencoders with WaveNet decoding and introduces time-jitter regularization; VQ-VAE representations preserve the most phonetic information while being the most speaker-invariant, with competitive ZeroSpeech 2017 performance.

  • Problem

    The paper asks how unsupervised speech representations can capture phonetic content while remaining invariant to speaker and other confounding details.

  • Method

    The authors compare dimensionality-reduction, Gaussian VAE, and discrete VQ-VAE bottlenecks in autoencoders paired with a WaveNet decoder and optional time-jitter regularization.

  • Results

    VQ-VAE representations preserved the most phonetic information, were the most speaker-invariant, mapped accurately to phonemes, and achieved competitive ZeroSpeech 2017 performance.

  • Takeaways & Limitations

    An information bottleneck and time-jitter regularization support speech representations focused on phonetic content rather than speaker characteristics.

Abstract

from arXiv · show

We consider the task of unsupervised extraction of meaningful latent representations of speech by applying autoencoding neural networks to speech waveforms. The goal is to learn a representation able to capture high level semantic content from the signal, e.g.\ phoneme identities, while being invariant to confounding low level details in the signal such as the underlying pitch contour or background noise. Since the learned representation is tuned to contain only phonetic content, we resort to using a high capacity WaveNet decoder to infer information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the kind of constraint that is applied to the latent representation. We compare three variants: a simple dimensionality reduction bottleneck, a Gaussian Variational Autoencoder (VAE), and a discrete Vector Quantized VAE (VQ-VAE). We analyze the quality of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately reconstruct individual spectrogram frames. Moreover, for discrete encodings extracted using the VQ-VAE, we measure the ease of mapping them to phonemes. We introduce a regularization scheme that forces the representations to focus on the phonetic content of the utterance and report performance comparable with the top entries in the ZeroSpeech 2017 unsupervised acoustic unit discovery task.

I. INTRODUCTION

The paper studies unsupervised speech representations that retain phonetic content while discarding speaker and other low-level variation. It compares autoencoder bottlenecks and uses WaveNet decoding to reconstruct discarded signal detail.

  • Motivation: Unsupervised representations can reduce reliance on labeled datasets while making learned speech features easier to analyze and reuse.The motivation includes low-resource ASR, where limited labels may suffice to train an acoustic model on an unsupervised representation.
  • Motivation: The target representation separates phonetic content from speaker traits such as gender and identity.These properties are associated with internal representations learned by speech recognizers and are useful for low-resource ASR.
  • Approach: A WaveNet decoder infers low-level details rejected by the encoder, allowing the latent representation to focus on high-level semantic features.The decoder uses autoregressive context to reconstruct waveform samples from representations that are tuned toward phonetic content.
  • Approach: The study compares dimensionality reduction, Gaussian VAE, and discrete VQ-VAE bottlenecks for unsupervised speech representation learning.VQ-VAE replaces continuous encoder outputs with prototype-based discrete token IDs, while VAE uses a probabilistic latent representation.
  • Bottleneck principles: The VAE bottleneck penalizes information about each data sample, with β controlling the trade-off between reconstruction quality and representation simplicity.A free-information formulation penalizes transmission only beyond B nats over the prior.

C. Autoencoders for sequential data

The paper combines periodic latent features with an autoregressive WaveNet decoder for sequential speech reconstruction. This design introduces latent-space collapse, which requires specific VAE or VQ-VAE treatments.

  • Architecture: Autoregressive models predict each next observation from recent history but do not themselves produce latent representations.The paper therefore combines sequence generation with an encoder that periodically emits latent feature vectors.
  • Latent collapse: Mixed latent-variable and autoregressive models are prone to latent space collapse when the decoder ignores constrained latent representations.The unconstrained autoregressive path can provide enough information for decoding without using the latent code.
  • Latent collapse: VAE collapse can be mitigated by KL-weight annealing and a free-information objective, whereas VQ-VAE is naturally resilient because its KL term is a fixed hyperparameter.The VQ-VAE’s relevant hyperparameter is tied to the prototype inventory rather than optimized by gradient training.
  • Architecture: The proposed model uses an encoder, bottleneck, WaveNet decoder, autoregressive past-sample stream, and speaker-identity conditioning.The encoder produces latent vectors from waveform input, while the decoder reconstructs the waveform using the additional information streams.

III. MODEL DESCRIPTION

The model encodes waveform or acoustic-feature sequences into bottlenecked latent vectors and reconstructs them with a WaveNet decoder conditioned separately on speaker identity. The study varies bottleneck, regularization, and input/output representations.

  • Encoder and decoder: The encoder extracts hidden-vector sequences from raw audio or acoustic features, with strided convolutions determining latent-vector frequency.These hidden vectors pass through a bottleneck to form the latent representations.
  • Encoder and decoder: The decoder conditions WaveNet on encoder latents and a separate speaker embedding to reconstruct the utterance.Explicit speaker conditioning frees the encoder from representing speaker-dependent information.
  • Encoder and decoder: The decoder combines autoregressive past-sample information, global speaker information, and latent information from past and future encoder steps.Latent vectors are optionally regularized, combined across neighboring time steps, and upsampled before waveform reconstruction.
  • Model variants: The experiments compare dimensionality reduction, Gaussian VAE, and VQ-VAE bottlenecks with optional time-jitter regularization.VQ-VAE capacity is varied through the number of quantization prototypes.
  • Representations: Inputs and outputs include raw waveforms, log-mel filterbanks, and MFCCs, with MFCCs discarding pitch information present in spectrograms.The feature extractor can be viewed as a fixed signal-processing layer in the encoder.

A. Time-jitter regularization

The paper seeks slowly changing phonetic representations, but directly penalizing temporal differences collapses the latent space. It instead introduces time-jitter to prevent neighboring latent vectors from co-adapting.

  • Motivation: The desired representation is mostly constant within phonetic units and changes abruptly at phoneme boundaries.This target follows the slowly changing phonetic content of an utterance.
  • Failure mode: Penalizing time differences before or after the bottleneck caused the model to output a constant encoding.This is identified as a latent-space collapse problem in sequential VAEs with latent regularization.
  • Regularization: Time-jitter prevents consecutive latent vectors from serving as separate units by replacing each vector with a neighboring token during training.The regularizer is intended to prevent latent-vector co-adaptation while preserving meaningful phonetic units.
  • Implementation: The regularization layer follows the bottleneck and independently replaces each training-time latent vector with the token immediately before or after it.It is enabled only during training and never copies a token more than one timestep.

IV. EXPERIMENTS

The experiments evaluate unsupervised speech representations across datasets, bottleneck types, probe tasks, and VQ-VAE token interpretability, using MFCC-based encoders and WaveNet decoders.

  • Experimental setup: Models were evaluated on LibriSpeech and ZeroSpeech 2017 data, both consisting of clean, read speech from multiple speakers.
  • Experimental setup: The experiments compare bottleneck variants through phoneme, speaker, gender, and filterbank-reconstruction probes at multiple network locations.
  • Token interpretability: VQ-VAE tokens were mapped to phonemes using forced alignments on LibriSpeech development data and evaluated on a separate test set.
  • Acoustic unit discovery: The VQ-VAE was also applied to ZeroSpeech 2017 acoustic unit discovery, which evaluates discrimination with respect to phonetic class.
  • Model configuration: The encoder used MFCC features, while the decoder reconstructed raw waveforms with a speaker-conditioned autoregressive WaveNet.
  • Model configuration: Training applied randomized time-jitter regularization, which replaced each latent vector with a neighbor with probability 0.12.
  • Training: Training used minibatches of 64 sequences containing 5120 waveform samples, corresponding to 320 ms.

B. Bottleneck comparison

The bottleneck comparison shows that VQ-VAE most selectively retains phonetic information while discarding speaker-related information, whereas AE retains more mixed signal information and VAE performs intermediately.

  • Evaluation: The study probes speaker gender, speaker identity, framewise phonemes, and log-mel filterbank reconstruction at four locations surrounding the bottleneck.
  • Bottleneck effects: Each bottleneck reduces information between the encoder output and bottleneck, while bottleneck width also affects information retained in preceding encoder layers.
  • AE: AE discards the least information, retaining strong speaker and phoneme predictability and achieving the best filterbank reconstructions at pcond.
  • VQ-VAE: VQ-VAE retains highly predictive phonetic content while effectively discarding speaker identity and gender, with speaker information primarily removed during quantization.
  • VQ-VAE: VQ-VAE phoneme prediction improves after combining latent vectors across a wider receptive field, but this added context does not recover speaker information.
  • VAE: VAE separates speaker and phonetic information better than dimensionality reduction but less effectively than VQ-VAE.
  • VAE: The VAE discards phonetic and speaker information more uniformly than VQ-VAE, and its stochastic noise may mask information at wider bottleneck dimensions.
  • Conclusion: The authors conclude that VQ-VAE is the most appropriate bottleneck for phonetic representations invariant to speaker identity.

C. VQ-VAE token interpretability

VQ-VAE tokens show measurable but incomplete phonetic interpretability, with mapping accuracy improving as the token inventory grows. In ZeroSpeech, the speaker-independent system performs strongly across languages, while Mandarin remains a clear limitation.

  • Token-to-phoneme mapping: The mapping task evaluates whether discrete VQ-VAE tokens can be associated with phonemes using forced alignments and held-out test data.A token is assigned its most frequent phoneme from labeled development data, then evaluated on a separate test set.
  • Token-to-phoneme mapping: 64.5% mapping accuracy is reached with 32768 VQ-VAE tokens, although gains diminish beyond 4096 tokens.The mapping measures generalization rather than cluster purity, so larger inventories do not trivially guarantee better accuracy.
  • ZeroSpeech evaluation: The ZeroSpeech ABX task complements phoneme mapping by testing discrimination between three-phoneme segments rather than correspondence to predefined units.The evaluation reports across-speaker and within-speaker performance, with lower ABX scores being better.
  • ZeroSpeech evaluation: On English and French, the speaker-independent VQ-VAE outperforms the top ZeroSpeech contestant.The best across-speaker configuration uses pcond, which combines neighboring bottleneck frames; quantization can discard speaker information while neighboring context restores phonetic detail.
  • Limitations: Mandarin performance is worse because of limited training data, discarded pitch in MFCCs, and the VQ-VAE’s lack of prosody encoding.Multilingual training and alternative filterbank features partially improve performance, with multilingual prequantized features reaching accuracies comparable to the challenge reference.

E. Hyperparameter impact

Hyperparameter choices were validated on LibriSpeech and then checked on English ZeroSpeech data. Time-jitter improved ABX scores across input representations, while MFCCs or filterbanks and larger token inventories performed better than waveforms or smaller inventories.

  • Validation: Time-jitter regularization improved ZeroSpeech ABX scores for all tested input representations.The design choices were tuned on LibriSpeech and validated on the English ZeroSpeech task.
  • Input representation: MFCC and filterbank inputs yielded better ZeroSpeech ABX scores than waveform inputs.The comparison used the English part of the challenge.
  • Token inventory: Larger token inventories consistently produced better ZeroSpeech ABX scores.This trend was observed alongside the input-representation and regularization comparisons.

1) Time-jitter regularization:

Time-jitter regularization discourages latent co-adaptation and shifts representations toward phonetic information. It improves phoneme recognition and token mapping while reducing reconstruction quality and speaker-sensitive information, with benefits extending to higher token rates.

  • Comparison with dropout: Step-wise dropout performs slightly worse than time-jitter because it promotes timestep-independent encodings without forcing the same neighboring-information separation.Regular dropout applied to individual dimensions does not impose the same constraint across neighboring timesteps.
  • Time-jitter regularization: Time-jitter greatly improves token mapping accuracy and makes 50 Hz token rates perform well.The 50 Hz model is better on ZeroSpeech because 25 Hz representations tend to omit short phones.
  • Information trade-off: Regularization limits filterbank reconstruction quality but increases phoneme recognition accuracy in the constrained representation.This pattern holds across VQ-VAE, VAE, and dimensionality-reduction autoencoder bottlenecks.
  • Information trade-off: VQ-VAE and VAE regularization decreases gender prediction accuracy and makes representations slightly less speaker-sensitive.The benefit becomes smaller after neighboring timesteps are combined at pcond.

3) Output representation:

Representation quality depends on decoder context, encoder receptive field, token rate, and embedding size. The strongest settings balance speaker invariance, phonetic accuracy, and low bitrate, while excessive token rates or poorly matched embedding dimensions hurt performance.

  • Encoder receptive field: A 0.3 s encoder receptive field produced 56.5% framewise phoneme recognition accuracy under the reported long-segment training condition.The effective receptive field can be controlled through architecture or by limiting the signal duration seen during training.
  • Token rate: At 200k steps, phoneme recognition accuracy was 56.3% at 25 Hz and 56.0% at 50 Hz, but fell to 49.3% at 100 Hz.The 12.5 Hz setting reached 53% because lower token rates miss very short phones.
  • Embedding size: Embedding dimensionality has a secondary effect: 64 dimensions worked well, while very small or large dimensions hurt models with small or large token inventories, respectively.The interaction depends on the number of quantization tokens.
  • Bit rate: 14 bits at 50 Hz gives a 700 bps encoder bitrate even for the largest token inventory.This bitrate is reported as comparable to the lowest bitrate of classical speech codecs.

7) Training corpus size:

Phoneme recognition improved with more training data up to roughly 23 hours, while ZeroSpeech performance varied strongly with language-resource size. The authors also identify future capacity improvements as important for exploiting larger unlabeled datasets.

  • Training corpus size: 54% phoneme recognition accuracy was reached after about 23 hours of training data, with no further improvement on the full 460-hour set.Accuracy peaked at 50.5% with 4.6 hours, 52.5% with 9 hours, and 54% beyond 23 hours.
  • Future work: Increasing model capacity is proposed as a future direction for making better use of larger amounts of unlabeled data.This is presented as an open research area rather than a demonstrated result.
  • Training corpus size: VQ-VAE models performed well on English and French but poorly on Mandarin, which had substantially less training data.The reported training amounts were 45 hours for English, 24 hours for French, and 2.5 hours for Mandarin.
  • Training corpus size: Monolingual training produced the best English and French ZeroSpeech results, whereas joint multilingual training was slightly better for Mandarin.The Mandarin result is reported as a slight improvement from training jointly across all languages.

VI. CONCLUSIONS

The study finds that discrete VQ-VAE representations retain the most phonetic information while being the most speaker-invariant, and that an information bottleneck is needed to separate content from speaker characteristics. Time-jitter regularization further limits latent capacity without collapsing the latent space, although the role of quantization remains unresolved.

  • Conclusions: VQ-VAE representations preserved the most phonetic information while also being the most speaker-invariant among the compared bottlenecks.The comparison included sequence autoencoders with dimensionality, VAE, and VQ-VAE bottlenecks.
  • Conclusions: An information bottleneck is required for the model to learn representations that separate content from speaker characteristics.This conclusion is stated as an empirical finding of the study.
  • Conclusions: Latent collapse from overly strong bottlenecks can be avoided by tuning bottleneck strength, removing it as in VQ-VAE or using the free-information VAE objective.The conclusion identifies two alternative ways to control the bottleneck.
  • Conclusions: Time-jitter regularization limits latent-code capacity without causing collapse of the latent space.The scheme was introduced to further improve representation quality.
  • Limitations: The study does not establish whether VQ-VAE’s better information separation comes from quantization or deterministic operation.The authors identify this distinction as requiring further experiments.
  • Limitations: The VQ-VAE token set was so large that assigning each token a separate meaning was impractical in the ZeroSpeech experiments.The experiments instead used dense token embeddings for a more nuanced similarity measure.
  • Future work: Optimizing encoder and decoder receptive fields was important for performance, while multiscale modeling and an HMM prior are proposed for future improvement.These proposals target prosodic separation and temporal phonetic structure, respectively.
Loading 1901.08810v2…