Source-linked AI summary
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, Yoshua Bengio
TL;DR
Audio generation must model extremely high-dimensional raw waveforms despite dependencies spanning very different temporal scales, and handcrafted feature pipelines can degrade quality. The paper introduces SampleRNN, a hierarchical end-to-end model combining recurrent modules at multiple clock-rates and frame-level conditioning. Across three datasets, the model generalizes well and its generated samples are preferred by human raters, while implementation choices may prevent exact reproduction of WaveNet results.
Problem
Raw audio has much higher dimensionality than semantic signals, and feature-based pipelines can degrade waveform quality while requiring extensive task-specific correction.
Method
SampleRNN is a tractable end-to-end unconditional raw-waveform model using recurrent modules at different temporal scales, with higher-level modules operating on frames.
Results
The model generalizes across three audio datasets, and human raters prefer its generated samples.
Takeaways & Limitations
A hierarchy of time scales and frequent updates can support learning the audio data manifold directly from raw samples.
Takeaways & Limitations
WaveNet reproduction results may differ because implementation and hyperparameter choices likely differ from the original authors' model.
Abstract
from arXiv · showhide
In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.
1 INTRODUCTION
Audio generation is difficult because raw waveforms have far higher dimensionality than semantic signals, while conventional feature-based pipelines can degrade quality and require task-specific correction. SampleRNN addresses this with a tractable end-to-end raw-waveform model using recurrent modules at multiple temporal scales, evaluated through model comparisons, component studies, and human ratings.
- Motivation: At 16kHz, speech generation averages 6,000 raw samples per word, illustrating the gap between waveform dimensionality and sentence-level semantics.The estimate is based on the average speaking rate of a set of TED talk speakers.
- Motivation: Feature-based systems compress raw audio into spectral or hand-engineered representations, but decompression can degrade quality and require extensive expert correction.These systems therefore involve complicated pipelines that must adapt to new tasks or domains.
- Motivation: RNNs can model audio dependencies, but generating one sample at a time creates temporal resolutions where conventional recurrent models do not scale well.At 16kHz, sample-level generation requires approximately 16,000 recurrent steps per second.
- Approach: SampleRNN presents a tractable end-to-end unconditional model for raw waveforms with modules operating at different clock-rates.This design allocates computation across different levels of abstraction rather than treating all temporal dependencies identically.
- Contributions: The model uses RNNs at different scales to capture longer-term waveform dependencies while training on short sequences for memory efficiency.The paper also compares model variants and evaluates component impacts across three audio datasets, including human evaluation of generated samples.
2 SAMPLERNN MODEL
SampleRNN models raw-audio waveforms with a hierarchy of modules operating at different temporal resolutions, combining recurrent frame-level modeling with sample-level autoregressive prediction. The design supports long-term dependencies while keeping training and generation tractable.
- SampleRNN models waveform-sequence probability as a product of each sample’s probability conditioned on all preceding samples.This defines the model as an autoregressive density model over waveform samples.
- FRAME-LEVEL MODULES: A hierarchy of modules operates at different temporal resolutions, with higher modules conditioning lower modules and the lowest module producing sample-level predictions.The hierarchy jointly models longer timescales and fine-grained waveform details.
- FRAME-LEVEL MODULES: Higher-level modules process non-overlapping frames and use recurrent hidden states to summarize input history into conditioning vectors for lower modules.Frame size FS(k) determines how many samples are processed at each higher tier.
- FRAME-LEVEL MODULES: Different-tier conditioning vectors are expanded through r(k) separate linear projections before being passed to the next lower-resolution module.The paper identifies this operation as perforated upsampling, equivalent to zero insertion followed by linear convolution.
- SAMPLE-LEVEL MODULE: The lowest module uses an MLP to predict the next quantized sample from recent samples and conditioning from the next higher module.The MLP is convolutionally applied over windows of FS(1) samples and repeatedly run one sample at a time during generation.
- SAMPLE-LEVEL MODULE: The hierarchical architecture substantially outperforms an RNN baseline, while replacing the sample-level autoregressive module with conditionally independent frame predictions sharply worsens likelihood and sample quality.The real-valued-output variant also beats an RNN baseline, but its generated samples are almost indistinguishable from random noise.
- TRUNCATED BPTT: Truncated backpropagation through time enables training on short subsequences despite modeling long-term dependencies, with length 512 corresponding to 32 milliseconds in the best models.Increasing subsequence length improves performance but also increases training memory usage and convergence time; generated samples exhibit longer word-like structures.
3 EXPERIMENTS AND RESULTS
The experiments evaluate SampleRNN across three audio datasets using likelihood, ablations, human preference tests, and a speaker-memory test. Results show benefits from hierarchical modeling, short training subsequences, and the three-tier architecture, while the WaveNet comparison is limited by reimplementation differences.
- Datasets and setup: The study evaluates raw-audio models on Blizzard speech, Onomatopoeia vocal sounds, and Beethoven piano sonatas using 16 kHz, 16-bit audio.Blizzard contributes 20.5 hours, Onomatopoeia 3.5 hours from 51 actors, and Music 10 hours.
- Likelihood evaluation: Table 1 reports test negative log-likelihood in bits across the three datasets, with the hierarchical SampleRNN architecture substantially outperforming the baseline RNN.The authors attribute the considerable gap to SampleRNN’s hierarchically structured architecture.
- Baseline comparison: The WaveNet comparison uses a reimplementation constrained by missing architectural details and limited compute, so it may differ from the original model.The implementation was designed to fit on a single GPU and had a receptive field of around 250 milliseconds.
- Human evaluation: In Blizzard human evaluation, SampleRNN (3-tier) wins by a large preference margin, followed by SampleRNN (2-tier) and the other compared models.The test compares random samples from an RNN, two SampleRNN variants, and the authors’ WaveNet implementation.
- Human evaluation: The Music evaluation compares RNN, SampleRNN (2-tier), and SampleRNN (3-tier) after excluding generated samples judged to be random noise.This filtering step is specific to the Music dataset.
- Information retention: SampleRNN preserves speaker-related information across separated segments in 83% of samples without speaker IDs or conditioning information.The authors classify the first and last two seconds using speakers’ mean fundamental frequencies.
4 RELATED WORK
The paper situates its raw-audio autoregressive modeling alongside earlier autoregressive neural networks, multi-scale RNNs, and prior raw-waveform approaches. It specifically contrasts SampleRNN with WaveNet through different module clock rates and hierarchical handling of long-term dependencies.
- Autoregressive modeling: Autoregressive neural networks tractably model joint distributions by decomposing them over data units, an approach SampleRNN applies to acoustic samples.The paper relates SampleRNN to earlier work including NADE and PixelRNN, and describes its acoustic-sample decomposition in Eq. 1.
- Multi-scale modeling: Multi-scale RNN research motivates running parts of a model at different clock rates.
- Raw-waveform modeling: Earlier raw-audio work contrasts with traditional systems that model spectral features instead of raw waveforms.
- Comparison with WaveNet: Like WaveNet, SampleRNN generates one acoustic sample at a time conditioned on previous samples and quantizes acoustics into bins, but its modules operate at different clock rates.The paper presents this difference as relevant to comparing higher-level low-resolution RNN stages and mitigating long-term dependency problems.
5 DISCUSSION AND CONCLUSION
The paper concludes that hierarchical time scales and frequent updates support unconditional raw-audio generation across diverse datasets. It also identifies broader applicability while acknowledging that domain knowledge could enable further improvement.
- Discussion and conclusion: A hierarchy of time scales and frequent updates helps overcome the challenge of modeling extremely high-resolution temporal data.The authors state that this permits learning the data manifold directly from audio samples.
- Discussion and conclusion: SampleRNN generalizes across three audio datasets that differ in nature, and its generated samples are preferred by human raters.
- Discussion and conclusion: The general-purpose solution leaves room for further improvement when specific domain knowledge is applied.
- Discussion and conclusion: The method can be adapted to tasks involving sequential data with high temporal resolution and long-range complex structure.
A MODEL VARIANT: SAMPLERNN-WAVENET HYBRID
The SampleRNN-WaveNet hybrid combines frame-level and sample-level modules operating at different clock rates. Two implementations were tested, but preliminary experiments found that this variant did not meet expectations and pointed toward future work.
- Architecture: The frame-level module processes frames while the sample-level component processes individual acoustic samples, with their clock-rate ratio equal to the frame size FS.The frame-level component therefore takes FS times fewer sequential steps.
- Variants: With FS fixed at 128, the study tested fully convolutional WaveNet and RNN-WaveNet variants.The RNN-WaveNet uses a high-capacity RNN at frame level, while its sample-level WaveNet has a 509-sample receptive field.
- Assessment: Preliminary experiments found that the hybrid variant was not meeting expectations, motivating it as possible future work.