PaperlayerView exact revision on arXiv

Source-linked AI summary

Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge

Benjamin van Niekerk, Leanne Nortje, Herman Kamper

arXiv:2005.09409v2eess.AScs.CL

TL;DR

Low-resource languages often lack transcribed speech, motivating methods that learn useful speech representations without explicit supervision. This paper combines vector quantization with waveform reconstruction or future-unit prediction, and reports leading ABX phone discrimination alongside competitive voice conversion results.

  • Problem

    Low-resource languages often lack transcribed speech corpora, while creating them can be prohibitively expensive, limiting the availability of speech technologies.

    Source: §1 · p. 1

  • Method

    The paper proposes a VQ-VAE that reconstructs waveforms from discrete speech units and a VQ-CPC model that uses contrastive prediction of future acoustic units.

    Source: §1 · p. 1

  • Results

    More than 30% on English and 50% on Indonesian ABX scores: both models outperform all ZeroSpeech 2019 and 2020 submissions, and VQ-CPC achieves top English naturalness and speaker-similarity scores.

    Source: §4 · p. 3

  • Takeaways & Limitations

    Probing indicates that vector quantization acts as an information bottleneck that discards speaker details, while the models achieve competitive phone discrimination and voice conversion.

    Source: §4 · p. 4, §5 · p. 4

    On Indonesian voice conversion, a 9.4 LUFS volume disparity between synthesized speech and source utterances may have negatively affected results.

    Source: §4 · p. 4

Abstract

from arXiv · show

In this paper, we explore vector quantization for acoustic unit discovery. Leveraging unlabelled data, we aim to learn discrete representations of speech that separate phonetic content from speaker-specific details. We propose two neural models to tackle this challenge - both use vector quantization to map continuous features to a finite set of codes. The first model is a type of vector-quantized variational autoencoder (VQ-VAE). The VQ-VAE encodes speech into a sequence of discrete units before reconstructing the audio waveform. Our second model combines vector quantization with contrastive predictive coding (VQ-CPC). The idea is to learn a representation of speech by predicting future acoustic units. We evaluate the models on English and Indonesian data for the ZeroSpeech 2020 challenge. In ABX phone discrimination tests, both models outperform all submissions to the 2019 and 2020 challenges, with a relative improvement of more than 30%. The models also perform competitively on a downstream voice conversion task. Of the two, VQ-CPC performs slightly better in general and is simpler and faster to train. Finally, probing experiments show that vector quantization is an effective bottleneck, forcing the models to discard speaker information.

1. Introduction

The paper addresses acoustic unit discovery from unlabelled speech, aiming for discrete representations that capture phonetic contrasts while discarding speaker-specific details. It proposes two vector-quantized models and reports strong ABX phone discrimination and competitive voice conversion results.

1. Introduction

  • Low-resource languages often lack transcribed speech corpora, and building new corpora can be prohibitively expensive.Consequently, technologies such as automatic speech recognition and text-to-speech are unavailable for many languages.Source: §1 · p. 1
  • Zero-resource speech processing seeks to learn from unlabelled speech representations that capture phonetic contrasts while remaining invariant to noise and speaker details.Such representations can bootstrap downstream speech systems and reduce the need for labelled data.Source: §1 · p. 1
  • ZeroSpeech 2019 requires discrete acoustic units from unlabelled data for target-speaker synthesis, with phone discrimination and low bitrate as explicit goals.The challenge evaluates synthesized speech for intelligibility, speaker similarity, and naturalness.Source: §1 · p. 1
  • The paper proposes a VQ-VAE that reconstructs waveforms from discrete units and a VQ-CPC model trained to distinguish future acoustic units from negative examples.Within-speaker negative sampling is reported as important for speaker invariance.Source: §1 · p. 1
  • On English and Indonesian ABX phone discrimination, both models outperform all submissions to the ZeroSpeech 2019 and 2020 challenges.The models also perform competitively on voice conversion; VQ-CPC achieves the best English naturalness and speaker-similarity scores.Source: §1 · p. 1

2. Vector-quantized neural networks

The section describes vector quantization as a discrete bottleneck, then develops VQ-VAE and VQ-CPC architectures for acoustic unit discovery. The models reconstruct speech or predict future codes, with speaker conditioning and negative-sampling choices shaping their training and use.

2.1. Vector quantization

  • The VQ layer maps each continuous feature vector to its nearest codebook entry, using a straight-through estimator for gradients and an exponential moving average to train the codebook.A commitment cost encourages each feature vector to commit to its selected code.Source: §2.1 · p. 1

2.2. Vector-quantized variational autoencoder

  • The VQ-VAE uses a 512-code bottleneck between a convolutional speech encoder and a decoder that reconstructs the waveform.Its encoder processes 16 kHz speech through five convolutional layers and downsamples by a factor of two; the decoder is an autoregressive model conditioned on bottleneck output, speaker identity, and past waveform samples.Source: §2.2 · p. 2, Figure 1 · p. 2
  • The VQ-VAE balances waveform reconstruction against compression, and conditions its decoder on speaker identity to encourage the bottleneck to discard speaker information.The model replaces a WaveNet decoder with a lightweight RNN-based vocoder, enabling single-GPU training, and uses a smaller codebook for a low-bitrate representation.Source: §2.2 · p. 2
  • Time-jitter regularization replaces each frame's code with a neighboring code with probability 0.5, while batch normalization and large batches improve codebook utilization.These choices target consistency across frames and codebook collapse, respectively.Source: §2.2 · p. 2

2.3. Vector-quantized contrastive predictive coding

  • VQ-CPC adapts contrastive predictive coding for acoustic unit discovery by combining it with vector quantization; product-codebook choices useful for ASR are reported as unsuitable for this task.The encoder maps log-Mel speech features through a strided convolution and four linear layers before a 512-code VQ bottleneck.Source: §2.3 · p. 2
  • VQ-CPC summarizes quantized codes into an autoregressive context vector and trains that context to predict future codes against negatives drawn from other utterances.The model uses InfoNCE with a six-step prediction horizon and 17 negative examples per step.Source: §2.3 · p. 3
  • The authors compare across-speaker negatives with within-speaker negatives, hypothesizing that same-speaker sampling encourages speaker invariance because speaker identity cannot identify the positive example.For voice conversion, VQ-CPC requires a separately trained vocoder because it is not a generative model.Source: §2.3 · p. 3

3. Experimental setup

The experiments use English and Indonesian ZeroSpeech 2019 data, with English for development and finalized procedures applied to Indonesian. Evaluation covers ABX phone discrimination and voice conversion, against challenge systems and other learned approaches.

3. Experimental setup

  • Training uses about 15 hours of speech from over 100 speakers per language; voice conversion adds an hour per target speaker, and test sets contain about 30 minutes from unseen speakers.English is the development language; after tuning there, the finalized training procedure is applied to Indonesian.Source: §3 · p. 3
  • ABX tests compare triphones differing in the middle phone, using same-speaker A and B and a different-speaker X to measure speaker invariance.Similarity is average cosine distance along a dynamic time-warping path, with error reported over all test-set triphone pairs.Source: §3 · p. 3
  • Voice conversion is judged by intelligibility, speaker similarity, and naturalness; intelligibility uses character error rate, while the latter scores use a 1–5 scale.Naturalness is reported as mean opinion score, with higher scores better.Source: §3 · p. 3
  • The challenge baseline combines DPGMM acoustic-unit discovery with Merlin synthesis, while the topline uses supervised recognition and text-to-speech trained on transcriptions.The topline feeds supervised speech-recognition output into a text-to-speech system.Source: §3 · p. 3
  • Comparators include the prior VQ-VAE(spec), Chen and Hain’s system, and vq-wav2vec, which was trained on 960 hours of Librispeech.VQ-VAE(spec) reconstructs spectrograms for a separate FFTNet vocoder; Chen and Hain use instance normalization and a 216-code product codebook.Source: §3 · p. 3

4. Experimental results

The models lead challenge submissions on ABX phone discrimination, while voice-conversion results vary by language and metric. Probing indicates that quantization removes speaker information, and comparisons reveal trade-offs in intelligibility, bitrate, and training efficiency.

4. Experimental results

  • On English voice conversion, VQ-CPC marginally beats VQ-VAE in naturalness and speaker similarity; Indonesian results favor some other submissions, possibly due to a 9.4 LUFS volume disparity.The English volume difference is around 6.1 LUFS; the larger Indonesian disparity may have negatively affected results.Source: §4 · pp. 3–4
  • VQ-CPC produces similar code sequences for utterances from different speakers, and adjacent frames within a phone often map to the same code.Figure 3 overlays aligned transcriptions and acoustic units; common units are highlighted in yellow.Source: Figure 3 · p. 4, §4 · p. 4
  • More than 30% on English and 50% on Indonesian: our models improve ABX scores over all submissions to the 2019 and 2020 challenges.Table 1 reports human and machine evaluations on both language test sets.Source: §4 · p. 3, Table 1 · p. 4
  • Chen and Hain achieve the best CER across both languages; their larger codebook improves CER at the cost of speaker similarity, while our models beat the supervised topline at much higher bitrate.The authors describe a trade-off between intelligibility and voice-conversion quality, and another between bitrate and CER/ABX score.Source: §4 · p. 4
  • VQ-VAE and VQ-CPC perform similarly across metrics, but VQ-CPC trains an order of magnitude faster and is more robust to codebook collapse in these experiments.Comparison with VQ-VAE(spec) suggests jointly training an autoregressive decoder with the encoder is beneficial; the models also achieve comparable ABX scores to a visually grounded VQ model.Source: §4 · p. 4
  • Within-speaker VQ-CPC sampling improves speaker invariance and ABX scores (13.4% vs. 36.2%); probing shows speaker-classification accuracy drops across quantization.The table compares speaker classification before quantization and on the discrete codes.Source: Table 2 · p. 4, §4 · p. 4

5. Conclusions and future work

The two VQ models learn discrete representations from unlabelled speech that capture phonetic content while discarding speaker information, and perform competitively on phone discrimination and voice conversion. Their bitrates remain high relative to phonetic transcriptions and a supervised topline, motivating work on lower bitrates and units more consistent across phones.

  • Both models learn discrete speech representations that capture phonetic content while discarding speaker information.Source: §5 · p. 4
  • The models perform competitively on phone discrimination tests and a voice conversion task for ZeroSpeech 2020.Source: §5 · p. 4
  • Their bitrates remain high compared with phonetic transcriptions and a supervised topline; future work targets lower bitrates and units more consistent across phones.Source: §5 · p. 4
Loading 2005.09409v2…