Source-linked AI summary
Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, Mohammad Norouzi
TL;DR
Audio synthesis needs models and datasets that handle fine-grained waveforms and difficult long-term structure. This paper introduces a WaveNet autoencoder conditioned on temporal codes from raw audio and the large NSynth musical-note dataset, then demonstrates improved performance over a spectral autoencoder and meaningful timbre interpolation. The learned embedding space supports morphing between instruments to create realistic and expressive sounds.
Problem
Audio signals contain difficult multi-scale dependencies, while prior data-driven synthesis often uses constrained domains; large, high-quality audio datasets are also needed for consistent evaluation.
Method
The paper combines a WaveNet-style autoencoder that learns temporal hidden codes from raw audio with NSynth, a large-scale dataset of annotated musical notes.
Results
The WaveNet autoencoder shows improved qualitative and quantitative performance over a well-tuned spectral autoencoder baseline and learns embeddings that support meaningful instrument-timbre interpolation.
Takeaways & Limitations
The learned representation can serve as a high-level control signal for manipulating tone, timbre, and dynamics and for creating new expressive instrument sounds.
Takeaways & Limitations
The WaveNet autoencoder’s temporal context is limited to the chunk-size of the training audio, and larger context remains ongoing research.
Abstract
from arXiv · showhide
Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.
1. Introduction
The paper addresses difficult long-term audio modeling by introducing a WaveNet autoencoder with temporal hidden codes and NSynth, a large musical-note dataset. It evaluates reconstruction, interpolation, and embedding structure while targeting realistic, expressive instrument sounds.
- The paper presents a data-driven approach that generates realistic, expressive instrument sounds and uses hidden representations to control tone, timbre, and dynamics.
- The WaveNet-style autoencoder learns temporal hidden codes that capture longer-term structure without external conditioning.The encoder infers embeddings distributed in time, while the decoder uses them to reconstruct audio over longer input-dependent time scales.
- The work is motivated by the difficulty of modeling multi-scale dependencies in wild audio, which has led prior synthesis efforts toward more constrained domains.
- NSynth contains ∼300k four-second annotated notes sampled at 16kHz from ∼1k harmonic musical instruments and is an order of magnitude larger than comparable public datasets.
- The study evaluates WaveNet autoencoder reconstruction and interpolation against a convolutional autoencoder baseline trained on spectrograms.It also analyzes the learned embedding space using qualitative and quantitative evaluation.
2. Models
The models compare a spectral convolutional autoencoder with a WaveNet autoencoder that encodes raw audio into temporal embeddings and conditions an autoregressive decoder. The WaveNet design removes reliance on external conditioning but remains limited by the training-audio chunk size.
- WaveNet autoencoder: The WaveNet autoencoder uses a temporal encoder and decoder to capture long-term structure without external conditioning.Its learned encodings also support meaningful audio interpolation.
- WaveNet autoencoder: The encoder produces a temporal embedding Z from raw audio, and the decoder reconstructs the causally shifted input waveform using that encoding.
- WaveNet autoencoder: During inference, the decoder autoregressively generates samples from an embedding and an initial palette of zeros, with embeddings also usable for interpolation or analogy.
- WaveNet autoencoder: The temporal encoder is a 30-layer dilated-convolution residual network followed by 1x1 convolutions and average pooling to form time-indexed hidden codes.
- Limitation: The WaveNet autoencoder is limited in temporal context to the chunk-size of the training audio.The paper identifies larger context as an area for ongoing research.
- Spectral baseline: The baseline is a 10-layer convolutional autoencoder whose raw-waveform MSE training was difficult, while spectral representations improved training but produced low perceptual quality.The baseline ultimately used magnitude-oriented spectral processing with iterative phase reconstruction.
3. The NSynth Dataset
NSynth is a large musical-note dataset designed to support learning and examining embeddings for pitch, dynamics, and timbre. It contains annotated, parameterized audio notes spanning many instruments, pitches, velocities, and sonic qualities.
- Motivation: The dataset was created because deep networks benefit from abundant, high-quality data, while existing musical-note datasets were smaller.Musical notes provide a setting for exploring embeddings that may capture pitch, dynamics, and timbre.
- Dataset construction: NSynth contains 306 043 four-second, monophonic 16kHz musical notes from 1006 instruments, each with a unique pitch, timbre, and envelope.Notes cover MIDI pitches 21–108 and five velocities; the first three seconds are held and the final second decays.
- Dataset construction: Instrument coverage varies because some instruments cannot produce all 88 pitches, averaging 65.4 pitches per instrument and 4.75 unique velocities per pitch.Duplicate sounds across velocities in commercial sample packs reduce the average number of unique velocities.
- Annotations: Each note is annotated with source, family, and zero or more sonic qualities using human evaluation and heuristic algorithms.Source distinguishes acoustic, electronic, and synthetic production; family assigns each instrument to exactly one high-level family.
- Release format: The full dataset is distributed as TFRecord files split into training and holdout sets, with each note stored alongside its annotations.Each note is serialized as a TensorFlow Example protocol buffer.
4. Evaluation
The evaluation compares WaveNet and spectral autoencoders on note reconstruction, interpolation, and embedding analyses, using Rainbowgrams and audio-based qualitative assessment. WaveNet reconstructions generally preserve phase and harmonic structure more faithfully, while quantitative classification and interpolation studies support richer learned representations.
- Evaluation: The evaluation covers note reconstruction, instrument interpolation, and pitch interpolation, with qualitative analysis supported by audio examples.The authors emphasize listening because plots cannot fully convey audio quality.
- Evaluation: Rainbowgrams encode time horizontally, frequency vertically, log power-spectrum magnitude as line intensity, and instantaneous-frequency phase as color.The representation uses CQT spectrograms to display magnitude and phase together.
- 4.1. Reconstruction: WaveNet better preserves instrument-specific harmonic and phase structure than the baseline across Glockenspiel, electric piano, and flugelhorn reconstructions.Examples include attack noise and fundamentals, coherent piano harmonics, and synchronized flugelhorn vibrato; WaveNet also exhibits some distortion and occasional discontinuities.
- 4.1.1. Quantitative Comparison: ∼70% more successful pitch extraction and several points higher quality prediction distinguish WaveNet reconstructions from the baseline, with quality accuracy roughly matching original audio.A multitask classifier evaluated 4096 randomly selected held-out reconstructions.
- 4.2. Interpolation in Timbre and Dynamics: WaveNet embedding interpolations create dynamic overtone mixtures and new harmonic structures that combine expressive timbral and dynamic traits beyond simple audio superposition.Examples include bass–flute and flute–organ blends, while the baseline produces less distinctive or duller sounds.
Z No Pitch Pitch Relative Size Cond. Cond. Change
Pitch conditioning reduces pitch information in the latent code, while embedding size affects this trade-off and embeddings capture instrument-specific pitch structure. The model also generalizes across time and can reconstruct extended note sequences despite short-note training.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: 13-30% classification-accuracy decreases occur when pitch conditioning is added, indicating reduced pitch information and greater pitch–timbre decoupling in the latent code.Below a total embedding size of 512, the relative accuracy decrease reaches 75%.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: A baseline model with embedding size 128 balances reconstruction quality and conditioning response across two octaves of a C-major chord.The original harmonic structure is only partially preserved as pitch changes.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: Pitch shifts can produce octave-related errors, with pitches most commonly confused with notes one octave away and up to 20% absolute classification error.A sub-harmonic emerges as pitch shifts upward, while octave harmonics become more prominent.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: Embedding correlations partition each instrument’s 88-note range into two or more pitch registers with similar embeddings.The average across instruments also shows a broad distinction between high and low registers.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: Temporal embeddings act as driving functions for a nonlinear oscillator or infinite impulse response filter, following the sound’s magnitude contour.Their dimensions decay close to zero when no sound is present.
- Z No Pitch Pitch Relative Size Cond. Cond. Change: The model reconstructs longer note sequences and notes exceeding three seconds despite training only on single notes lasting up to three seconds.It follows the fundamental frequency through extended arpeggios, although it adds harmonics to the organ’s original timbre.
5. Conclusion and Future Directions
The paper concludes that its WaveNet autoencoder captures long-term audio structure without external conditioning and works effectively on NSynth. It also identifies global context as an unresolved limitation and positions NSynth as a foundation for broader audio datasets and tasks.
- 5. Conclusion and Future Directions: The WaveNet autoencoder captures long-term structure without external conditioning and demonstrates effectiveness for generative audio modeling on NSynth.
- 5. Conclusion and Future Directions: Memory constraints prevent the autoencoder from fully capturing global context, leaving this as an important open problem.
- 5. Conclusion and Future Directions: NSynth focuses on single notes, reflecting a factorization in which MIDI represents note sequences while synthesizers represent timbre.Note-to-note dependencies can be partly restored by passing sequence-level timbre and dynamics information to the note-level synthesizer.
- 5. Conclusion and Future Directions: The model can fairly reconstruct a ten-second scale despite never being trained on more than one note at a time or on clips longer than four seconds.
- 5. Conclusion and Future Directions: The authors encourage NSynth’s use as a benchmark and entry point for audio machine learning and envision high-quality multi-note datasets for generation and transcription.
A. Phase Representation for the Baseline Model
The baseline explores phase representation alongside magnitude-based spectrogram objectives, using a circular likelihood for phase angles. Peak-normalized log-magnitude power spectra correlated better with perceptual distortion.
- A. Phase Representation for the Baseline Model: Peak-normalized log magnitude of the power spectra correlated better with perceptual distortion during baseline training.
- A. Phase Representation for the Baseline Model: When phase is included, the baseline regresses phase angles using a circular normal likelihood proportional to cos(π ∗(x −ˆx)).
B. Description of Quality Tags
NSynth annotates notes with ten quality tags describing spectral content, onset character, decay, acoustics, rhythm, and envelope behavior. The tags are generally nonexclusive, except Bright and Dark, which are mutually exclusive by definition.
- B. Description of Quality Tags: NSynth provides ten quality annotations, with tags generally nonexclusive except for Bright and Dark.A note may be neither Bright nor Dark.
- B. Description of Quality Tags: Bright denotes abundant high-frequency content and strong upper harmonics, whereas Dark denotes reduced high-frequency content and a muted, bassy sound.
- B. Description of Quality Tags: Distortion denotes waveshaping with a crunchy sound, many harmonics, and sometimes non-harmonic noise.
- B. Description of Quality Tags: Fast Decay indicates substantial harmonic-amplitude decay before the three-second note-off point, whereas Long Release extends decay after note-off.
- B. Description of Quality Tags: Multiphonic denotes overtones related to more than one fundamental frequency, while Non-Linear Envelope denotes distinct nonmonotonic envelope behavior.The latter can include filter and dynamic envelopes.
- B. Description of Quality Tags: Percussive marks a loud non-harmonic onset, while Reverb marks room acoustics that remain in the original sample.
- B. Description of Quality Tags: Tempo-Synced identifies rhythmic modulation tied to a fixed tempo.
C. Details of Pitch and Quality Classifier
The paper trains a multi-task model on NSynth to classify pitch and quality tags using a baseline-style encoder without a bottleneck. Embedding-based pitch classification most often errs by one octave, while pitch conditioning reduces classifier accuracy.
- Classifier training: A multi-task classifier predicts mutually exclusive pitch labels and non-exclusive quality tags across the entire NSynth dataset.It uses softmax-crossentropy for pitch and sigmoid-crossentropy for quality annotations.
- Model architecture: The classifier reuses the baseline encoder structure but removes its bottleneck.Each convolution layer is followed by batch normalization and a Leaky-ReLU with a 0.1 off-slope.
- Dataset annotations: NSynth instruments are annotated by both generation source and high-level instrument family, while quality annotations include co-occurrence probabilities and marginal frequencies.The source categories are acoustic instrument, electronic instrument, and software synthesis; annotation frequencies are presented as percentages.
- Pitch classification: The dominant pitch-classification error is predicting the wrong octave, corresponding to an error of 12 tones.This confusion matrix evaluates a linear pitch classifier trained on WaveNet autoencoder embeddings.
- Pitch classification: Adding pitch conditioning reduces classifier accuracy.The supplied figure caption reports this comparison without specifying the accuracy values.