Source-linked AI summary
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, Emmanuel Dupoux
TL;DR
CPC remains behind state-of-the-art unsupervised speech representation methods, while the role of speech-specific augmentation is not well established. The paper introduces WavAugment for time-domain augmentation and evaluates it across unsupervised, multilingual, and limited-supervision settings. Augmentation improves CPC across these settings, including 18-22% relative gains on Libri-light, parity with or improvement over leading baselines in some out-of-domain ZeroSpeech settings, and 12-15% relative gains in limited-supervision phoneme classification.
Problem
CPC has not reached state-of-the-art performance on purely unsupervised speech-learning metrics, and time-domain augmentation is not well established for unsupervised speech.
Method
The paper introduces WavAugment, a time-domain augmentation library, and applies pitch, additive noise, reverberation, masking, and related transformations within CPC.
Results
Augmentation improves CPC across unsupervised and limited-supervision evaluations, with 18-22% relative Libri-light gains and 12-15% relative phoneme-classification gains.
Takeaways & Limitations
Pitch and additive noise are the most effective techniques for this CPC implementation, and their gains extend to fine-tuning with very limited data.
Takeaways & Limitations
CPC with augmentation remains insufficiently competitive with very small datasets between 2.5 and 50 hours, and its data efficiency is below methods using speaker adaptation.
Abstract
from arXiv · showhide
Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech signal. However, it still under-performs other methods on unsupervised evaluation benchmarks. Here, we introduce WavAugment, a time-domain data augmentation library and find that applying augmentation in the past is generally more efficient and yields better performances than other methods. We find that a combination of pitch modification, additive noise and reverberation substantially increase the performance of CPC (relative improvement of 18-22%), beating the reference Libri-light results with 600 times less data. Using an out-of-domain dataset, time-domain data augmentation can push CPC to be on par with the state of the art on the Zero Speech Benchmark 2017. We also show that time-domain data augmentation consistently improves downstream limited-supervision phoneme classification tasks by a factor of 12-15% relative.
1. Introduction
Unsupervised speech representation learning aims to support downstream tasks with little labeled data, but CPC has not matched state-of-the-art unsupervised benchmarks. This work investigates whether speech-specific time-domain augmentation can improve CPC.
- CPC and related methods remain below state-of-the-art results on purely unsupervised speech-learning metrics.
- Time-domain augmentation is established in supervised learning and image-domain unsupervised learning, but its role in unsupervised speech is not well established.Speech’s sequential signal structure may create augmentation-specific considerations.
- The study evaluates additive noise, masking, reverberation, and augmentation applied to past, future, or both representations using English LibriSpeech.It then extends evaluation to French and Mandarin and examines semi-supervised learning with Libri-light.
2. Related work
Prior work established CPC and related predictive or contrastive approaches for speech representation learning, while augmentation had been less explored for learned representations. WavAugment extends time-domain augmentation into this setting.
- CPC learns speech representations by predicting future encoder embeddings and using a contrastive loss to separate positive from negative future embeddings [6].It has been used for speech pretraining and speaker identification [1] [14].
- Non-contrastive predictive-coding variants with fixed embeddings can learn generic multitask representations [7] [8].
- Earlier speech augmentations modify sampling rate, add noise with room responses, or mask spectral frequency and time regions [17].
- WavAugment implements these augmentations in the time domain and supports on-the-fly use during data loading.The library provides a PyTorch interface over interleaved PyTorch- and libsox-based effects.
- Related representation-learning work applied augmentation to autoencoders but evaluated downstream pretraining rather than the learned representation itself [18].
3. Method
The method combines an optimized CPC architecture with independently configurable time-domain augmentations for past and future speech representations. WavAugment applies these transformations to short audio sequences during training and evaluates representations with ABX testing.
- 3.1. The CPC2 architecture: The convolutional encoder maps raw waveforms to z_t, a recurrent context network produces c_t, and predictors estimate future z_{t+k} embeddings.The contrastive loss compares each prediction with the correct future embedding and 128 sampled negatives.
- 3.1. The CPC2 architecture: CPC2 modifies a state-of-the-art CPC architecture [3] with a deeper autoregressive network and a single multi-head transformer prediction layer.The two-layer LSTM balances feature quality and training speed, while the multi-head design reduces training time.
- 3.2. Data augmentation and CPC: The same speech sequence can receive different augmentations for past and future, including augmentation applied only to past, only to future, or independently to both.This design lets the contrastive objective learn while ignoring augmentation noise.
- 3.2. Data augmentation and CPC: Past representations form the context input, while future representations are positive or negative prediction targets in CPC.
- 3.3. WavAugment: WavAugment applies independent time-domain transformations to approximately 1-second audio sequences through a PyTorch interface combining PyTorch and libsox effects.
- 3.4. Datasets and evaluation measures: Models are primarily trained on 100 hours of clean, balanced-speaker LibriSpeech and evaluated with the Libri-light ABX metric.ABX measures whether a segment is closer to a same-transcription item than to a different-transcription item, within and across speakers.
4. Experiments
Experiments evaluate time-domain augmentations for CPC across English, multilingual zero-resource benchmarks, and limited-supervision phoneme classification. Past-only augmentation, especially pitch, additive noise, and reverberation, provides the strongest overall results.
- 4.1. Preliminary Experiment: Tuning data augmentation: The study evaluates pitch, additive noise, reverberation, band rejection, and time masking using LibriSpeech and multilingual datasets.Augmentation strengths are calibrated using ABX performance, including pitch shifts, reverberation, time dropout, and band rejection settings.
- 4.1. Preliminary Experiment: Tuning data augmentation: Past augmentation matches or outperforms past+future for pitch, reverberation, and band rejection, while additive noise is the exception.Average single-augmentation usefulness ranks as pitch, additive noise, reverberation, time dropout, then band rejection.
- 4.2. Experiment 1: Data augmentation combinations: Pitch+add+reverb performs best in 3 of 4 metrics among augmentation combinations, after band rejection is removed for poor results.Combination experiments augment only the past because this requires less computation while producing roughly the same representation quality.
- 4.2. Experiment 1: Data augmentation combinations: 18-22% relative improvement across all Libri-light test metrics over no augmentation, exceeding original CPC trained on 60k hours with the augmented model.The evaluated configuration uses pitch+add+reverb applied to the past sequence.
- 4.3. Experiment 2: Extending to other languages: Noise augmentation improves scores in English, French, and Mandarin, but reaches parity with or exceeds the best baseline only with larger out-of-domain training sets.With the small challenge-provided datasets, the method does not reach state of the art and remains less data-efficient than methods using speaker adaptation [27].
- 4.4. Experiment 3: Pretraining and limited supervision: 12-15% relative improvement appears across limited-supervision phoneme-classification training sets when augmentation is used in pretraining and fine-tuning.Pitch alone performs best during supervised fine-tuning; augmentation also beats the 60k-hour-pretraining, 10-hour-label reference with 100 hours plus 1 hour.
5. Discussion
The paper introduces WavAugment for time-domain augmentation in CPC and limited-supervision learning. It reports strong gains from pitch and additive noise, while leaving open whether time-domain advantages depend on CPC or waveform-based input.
- WavAugment applies time-domain augmentation to unsupervised contrastive representation learning and limited-supervision training.
- Pitch and additive noise are the most powerful augmentations for this CPC implementation across English, Mandarin, and French, while spectral-style methods perform poorly for CPC training.
- Augmentation gains extend to fine-tuning with very limited labelled data, improving phone error rate.
- Whether time-domain noise is superior because of CPC loss or direct waveform processing remains unresolved.The comparison is left for future work against architectures using spectral features such as Mel Filterbanks or MFCCs.
6. Conclusion
Data augmentation helps CPC use relatively short, clean, well-segmented speech, but it remains insufficient for competitive learning with very small datasets.
- Around 100 hours of clean, well-segmented speech lets augmented CPC achieve strong performance.
- Augmented CPC is currently insufficient to learn competitively from 2.5-50 hours of data.
- The differing effects of augmentations motivate more systematic study across tasks and architectures.
S1. Supplementary Results
The supplementary results examine architecture depth across data sizes and additive-noise frequency filtering, alongside implementation details and architecture ablations. With the best augmentation, deeper models become beneficial at 100 hours, while two LSTM layers remain best at 3 and 45 hours.
- Implementation: The evaluated CPC encoder uses five convolutional layers with specified kernels and strides, channel normalization between layers, a two-layer LSTM context network, and a single multihead transformer predictor.The encoder has hidden dimension 256 and ReLU activations; the architecture starts from the model described in [3].
- Implementation: Training uses Adam with learning rate 2e −4, β1 = 0.9, and β2 = 0.999, plus a ten-epoch learning-rate ramp without decay.
- Architecture ablations: Table S1 compares the original CPC architecture from [3] with deeper LSTM variants and a single multihead predictor for all time steps; CPC2 is the bottom model.
- Architecture and data-size ablations: Table S2 compares CPC architecture modifications across simulated 3-hour Mandarin, 45-hour French, and 100-hour English data conditions using pitch, additive-noise, and past reverberation augmentation.The 3-hour and 45-hour conditions are subsampled from LibriLight, while the 100-hour condition uses LibriSpeech.
- Architecture and data-size ablations: With the best data augmentation, increasing model depth benefits the 100-hour setting, whereas two LSTM layers perform best with 3-hour and 45-hour datasets.The authors attribute the ability to leverage a deeper architecture to the presence of data augmentation.
- Additive-noise filtering: The additive-noise study evaluates band-pass and low-pass filtering across frequency bands spanning 0–8000 Hz, including ranges associated with F0, F1, and F2.The frequency bands are applied to the MUSAN dataset; the supplied passage truncates the description of the extreme ranges.