Source-linked AI summary
WaveNet: A Generative Model for Raw Audio
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu
TL;DR
Generating wideband raw audio is challenging because of its high temporal resolution. WaveNet addresses this with an autoregressive waveform-level model using causal and dilated convolutions, achieving state-of-the-art subjective naturalness in text-to-speech and promising results for music modeling and speech recognition.
Problem
The paper asks whether generative models can produce wideband raw audio waveforms with very high temporal resolution.
Method
WaveNet is an autoregressive waveform-level audio model that combines causal and dilated convolutions to expand its receptive field.
Results
WaveNet outperformed existing TTS systems in subjective naturalness and showed promising results for music modeling, speaker-conditioned voice generation, and speech recognition.
Takeaways & Limitations
WaveNet provides a generic, flexible framework for audio-generation applications including TTS, music, speech enhancement, voice conversion, and source separation.
Takeaways & Limitations
Without text conditioning, WaveNet produces language-like but non-existent words and lacks long-range coherence because its receptive field is limited to about 300 milliseconds.
Abstract
from arXiv · showhide
This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.
1 INTRODUCTION
The introduction presents WaveNet as a PixelCNN-based model for generating wideband raw audio at very high temporal resolution. It highlights contributions in natural speech synthesis, dilated causal architectures, speaker conditioning, speech recognition, and music generation.
- Motivation: The paper asks whether neural autoregressive methods successful for images and text can generate wideband raw audio sampled at least 16,000 times per second.These methods model joint distributions as products of conditional distributions and can handle thousands of variables, such as 64×64 pixels.
- Model: WaveNet is introduced as an audio generative model based on the PixelCNN architecture.The model targets raw audio generation within a neural autoregressive framework.
- Speech synthesis: WaveNets generate raw speech with subjective naturalness never before reported in text-to-speech, according to human raters.This establishes the paper’s principal reported result for speech synthesis.
- Architecture: New dilated causal convolution architectures address long-range temporal dependencies in raw audio generation through very large receptive fields.The architectural design is intended to capture dependencies spanning long audio sequences.
- Audio applications: Conditioning on speaker identity lets one model generate different voices, while the same architecture performs strongly on speech recognition and shows promise for music generation.The paper frames WaveNets as a generic, flexible framework for audio applications including TTS, music, speech enhancement, voice conversion, and source separation.
2 WAVENET
WaveNet is a raw-audio generative model that factorizes waveform probability autoregressively, predicting each sample from all preceding samples. Its causal and dilated convolutional architecture preserves ordering while efficiently expanding context, with softmax quantization and conditioning supporting practical audio generation.
- Autoregressive modeling: WaveNet models raw audio by factorizing the waveform’s joint probability into conditional probabilities, with each sample conditioned on all previous timesteps.The model operates directly on the raw audio waveform and predicts each audio sample autoregressively.
- Audio representation: The network uses a softmax output over quantized audio values, applying µ-law companding and reducing 16-bit samples to 256 possible values.For speech, the reconstructed signal after quantization sounded very similar to the original.
- Causal convolutions: Causal convolutions prevent the prediction p(x_t+1 | x_1, ..., x_t) from depending on future samples.Training predictions can be computed in parallel from known ground truth, whereas generation proceeds sequentially by feeding each predicted sample back into the network.
- Dilated convolutions: Dilated causal convolutions expand receptive fields by skipping input values, enabling large context with few layers while preserving input resolution and computational efficiency.The dilation schedule doubles across layers up to 512 and repeats; each 1, 2, 4, ..., 512 block has receptive field size 1024.
- Conditional generation: WaveNet supports global and local conditioning, allowing generation to reflect inputs such as speaker identity, text, or time-varying linguistic features.Local conditioning uses learned upsampling to map a lower-frequency input timeseries to the audio resolution.
3 EXPERIMENTS
WaveNet was evaluated on multi-speaker speech generation, text-to-speech, music modelling, and speech recognition. It produced realistic but sometimes incoherent audio, achieved strong TTS results, generated often musical fragments, and adapted to discriminative recognition.
- Multi-speaker speech generation: WaveNet generated free-form, language-like speech for 109 speakers using speaker-only conditioning on 44 hours of VCTK audio.A one-hot speaker ID enabled one model to capture all speakers’ characteristics, including acoustics, recording quality, breathing, and mouth movements.
- Multi-speaker speech generation: A roughly 300-millisecond receptive field limited long-range coherence, despite smooth words and realistic intonations.The model generally remembered only the previous 2–3 phonemes, contributing to unnatural outputs on closer inspection.
- Text-to-speech: WaveNet outperformed statistical parametric and concatenative baselines in both English and Mandarin TTS evaluations.Five-point MOS values exceeded 4.0 and significantly surpassed baseline systems; the naturalness gap fell by 51% in US English and 69% in Mandarin Chinese.
- Music modelling: WaveNet music samples were often harmonic and aesthetically pleasing, but required enlarged receptive fields and still lacked long-range consistency.With receptive fields of several seconds, outputs could vary in genre, instrumentation, volume, and sound quality from second to second.
- Speech recognition: WaveNet was also adapted to TIMIT speech recognition with mean pooling, non-causal convolutions, and joint next-sample and frame-classification losses.Mean pooling aggregated activations into 10-millisecond frames with 160× downsampling, and the combined-loss model generalized better than a single-loss model.
4 CONCLUSION
WaveNet is a deep generative model that operates directly on raw audio waveforms, using autoregressive dilated convolutions to model long-range temporal dependencies and conditioning on global or local inputs.
- WaveNet operates directly at the waveform level as a deep generative model of audio data.
- Its autoregressive architecture combines causal filters with dilated convolutions, allowing receptive fields to grow exponentially with depth.This design is important for modeling long-range temporal dependencies in audio signals.
- WaveNet can be conditioned globally, such as on speaker identity, or locally, such as on linguistic features.
A TEXT-TO-SPEECH BACKGROUND
TTS maps discrete text symbols to real-valued speech signals through text analysis and synthesis. Conventional synthesis uses concatenative or statistical parametric approaches whose assumptions and vocoder-based optimization can reduce naturalness, whereas WaveNet avoids these assumptions and preserves audio detail.
- TTS pipeline: TTS is a sequence-to-sequence mapping from discrete text symbols to real-valued speech signals, typically combining text analysis with speech synthesis.Text analysis includes NLP steps such as segmentation, normalization, POS tagging, and grapheme-to-phoneme conversion.
- Synthesis approaches: Speech synthesis mainly uses concatenative systems that assemble recorded units or statistical parametric systems that generate speech with a model.Concatenative synthesis is example-based, whereas statistical parametric synthesis is model-based.
- Synthesis limitations: Statistical parametric synthesis offers a small footprint and flexible voice characteristics, but its subjective naturalness is often significantly worse than concatenative synthesis and may sound muffled or artifact-laden.The approach reconstructs speech waveforms from generated vocoder parameters.
- Synthesis limitations: Its training is sub-optimal because it first estimates vocoder parameters with a generative speech model and then separately models their trajectories as time series.This separates waveform modeling from trajectory modeling into two optimization steps.
- Conventional audio models: Conventional raw-audio models use fixed-length analysis windows and Gaussian-process assumptions, causing generated samples to lose important details needed for natural-sounding audio.Typical windows are 20 to 30 milliseconds long with 5 to 10 millisecond shifts.
- WaveNet motivation: WaveNet avoids these assumptions, using little prior audio knowledge beyond its receptive field and µ-law encoding while representing complex signals through a trainable non-linear causal filter.This design is intended to preserve audio details in quantized signals.
B DETAILS OF TTS EXPERIMENT
The TTS experiments compared WaveNet with LSTM-RNN statistical-parametric and HMM-driven unit-selection systems using blind, crowdsourced evaluations on held-out English and Mandarin sentences. The systems differed in sampling rates and input representations, while paired comparisons and MOS tests assessed speech quality.
- System setup: WaveNet and HMM-driven unit-selection systems used 16 kHz speech, whereas LSTM-RNN systems were trained at 22.05 kHz and synthesized at 16 kHz runtime.Runtime resampling used the Vocaine vocoder.
- System setup: The linguistic features covered phone, syllable, word, phrase, and utterance levels, plus frame-position and phone-duration features aligned every 5 milliseconds.Features were derived using phone-level forced alignment during training.
- Evaluation: The blind, crowdsourced tests evaluated 100 held-out sentences in paired comparisons and MOS tests, with stimuli randomly selected and presented either as pairs or in isolation.Each paired comparison used identical text synthesized by different models.
- Evaluation: The paired comparisons assessed LSTM, Concat, and proposed WaveNet speech synthesizers, with significant preferences defined at p < 0.01.The table distinguishes WaveNet (L) and WaveNet (L+F) conditions.