Source-linked AI summary
MelNet: A Generative Model for Audio in the Frequency Domain
Sean Vasquez, Mike Lewis
TL;DR
Audio waveforms pose a challenge for capturing long-range structure because they span tens of thousands of timesteps per second. MelNet models spectrograms using an expressive autoregressive model and multiscale generation, with reported results across audio-generation tasks. The paper describes longer-range dependencies and unconditional speech, music, and end-to-end text-to-speech generation.
Problem
Waveform models can capture local structure but typically backpropagate through only a fraction of a second, limiting their ability to capture structure emerging over several seconds.
Source: §1 · p. 1Method
MelNet models high-resolution spectrograms with a fine-grained autoregressive model and coarse-to-fine multiscale generation to capture local and global structure.
Source: §1 · p. 1Results
MelNet models longer-range dependencies than existing time-domain models and generates unconditional speech and music as well as end-to-end text-to-speech.
Source: §1 · p. 1Takeaways & Limitations
The paper concludes that MelNet is particularly well-suited to modeling long-range temporal dependencies and reports promising results across diverse audio-generation tasks.
Source: §9 · p. 11The cited GAN approaches are restricted to fixed-duration audio segments, precluding their use in many audio-generation tasks.
Source: §8 · p. 10
Abstract
from arXiv · showhide
Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.
1. Introduction
MelNet addresses the difficulty of capturing both local audio detail and long-range structure by modeling spectrograms with an expressive autoregressive model and multiscale generation. The paper reports longer-range modeling and generation across speech, music, and text-to-speech tasks.
1. Introduction
- Spectrograms compact dependencies spanning tens of thousands of waveform timesteps into hundreds of temporal steps, making long-range structure more tractable to model.For the same four-second signal, the waveform spans nearly 100,000 timesteps while the spectrogram spans roughly 400.Source: §1 · p. 1, Figure 1 · p. 2
- High-resolution spectrograms and an autoregressive model factorized over time and frequency aim to preserve local characteristics while limiting information loss and oversmoothing.The paper attributes fidelity challenges to lossy spectrograms and insufficiently expressive models.Source: §1 · p. 1
- A coarse-to-fine multiscale procedure first generates a low-resolution spectrogram for high-level structure, then iteratively adds high-resolution detail.Source: §1 · p. 1
- MelNet models longer-range dependencies than existing time-domain models and supports unconditional speech, music generation, and end-to-end text-to-speech.The introduction presents these as the model’s core results and task coverage.Source: §1 · p. 1
2. Preliminaries
The preliminaries define spectrograms as time-frequency representations derived from audio and describe their perceptual scaling and information loss. High-resolution spectrograms and inversion methods are used to reduce artifacts in generated audio.
2. Preliminaries
- A spectrogram is the squared magnitude of an audio signal’s short-time Fourier transform, with each value representing frequency-response energy at a timestep.A frame is one time slice of the spectrogram.Source: §2 · p. 2
- Mel scaling and logarithmic amplitude rescaling roughly align spectrogram frequency and amplitude axes with human perception of pitch and loudness.Source: §2 · p. 2
- Mel scaling discards frequency information and removing STFT phase discards temporal information, causing distortion when reconstructing waveforms from spectrograms.Source: §2 · p. 2
- Higher temporal resolution comes from a smaller STFT hop size, while higher frequency resolution comes from more mel channels; the paper uses gradient-based inversion, which generally produced fewer artifacts than Griffin-Lim.Source: §2 · p. 2
3. Probabilistic Model
The probabilistic model factorizes a spectrogram’s joint density into conditional distributions ordered across frames and frequencies. Neural networks use preceding spectrogram elements as context to parameterize Gaussian mixture factors.
3. Probabilistic Model
- The joint spectrogram density is factorized into conditional distributions, with row-major ordering traversing each frame from low to high frequency before moving to the next frame.Each conditional depends on elements preceding the current spectrogram value.Source: §3 · p. 2
- Each factor is a K-component Gaussian mixture whose parameters include component means, standard deviations, and mixture coefficients.Source: §3 · p. 2
- Four RNNs encode context: three in the time-delayed stack summarize preceding frames, and one in the frequency-delayed stack summarizes preceding elements in the current frame.The figure caption states that arrows of the same color share parameters.Source: Figure 2 · p. 3
- A neural network maps the preceding context to mixture parameters, and its parameters are trained by minimizing negative log-likelihood with gradient descent.Source: §3 · p. 3
- Transformations enforce positive standard deviations and mixture coefficients that sum to one.Source: §3 · p. 3
4. Network Architecture
MelNet uses recurrent stacks to model spectrogram context across time and frequency, with optional centralized and conditioning pathways. Its architecture avoids 2D convolution because spectrograms are not translation-invariant along frequency.
4. Network Architecture
- The architecture uses multidimensional recurrence rather than 2D convolution because spectrograms are not invariant to translation along frequency.The recurrent design draws inspiration from autoregressive image models and prior multidimensional recurrence for spectrograms.Source: §4 · p. 3
4.1. Time-Delayed Stack
- The time-delayed stack uses three one-dimensional RNNs along frequency and time, concatenating their hidden states to summarize preceding frames.Its inputs are shifted backward in time so outputs depend only on the permitted context.Source: §4.1 · p. 3
4.2. Frequency-Delayed Stack
- The frequency-delayed stack combines preceding elements within a frame with time-stack outputs to represent the full two-dimensional context.Source: §4.2 · p. 4
4.3. Centralized Stack
- An optional centralized stack reads each preceding frame into one vector, which is summed with the other layer inputs to the frequency-delayed stack.Source: §4.3 · p. 4
4.4. Conditioning
- Conditioning features are projected alongside spectrogram inputs and reshaped, upsampled, or broadcast to match the input’s time-frequency shape.A speaker-ID one-hot vector, for example, can be broadcast across both axes.Source: §4.4 · p. 4
5. Learned Alignment
For text-to-speech, MelNet must align spectrogram frames with character tokens. It builds character features and learns attention over them, then uses a threshold to decide when to stop sampling.
5. Learned Alignment
- Text-to-speech requires learning a latent alignment between spectrogram frames and discrete character tokens.Source: §5 · p. 4
- A bidirectional RNN over embedded characters produces features that support pronunciation and the semantics needed for intonation and prosody.Source: §5 · p. 4
- A centralized-stack RNN attends over character features using a learned distribution parameterized as a discretized mixture of logistics with mixture coefficients constrained to increase over time.In the example alignment, flat regions correspond to deliberate pauses in the spoken text.Source: §5 · p. 5, Figure 4 · p. 5
- Sampling stops at the first timestep when the estimated probability of having passed the final character exceeds threshold τ.The threshold is estimated after training.Source: §5 · p. 5
6. Multiscale Modelling
MelNet generates high-resolution spectrograms coarse-to-fine, using tiers to capture high-level structure before adding detail. Training recursively partitions spectrograms into tiers, and sampling generates and interleaves them in sequence.
6. Multiscale Modelling
- High-resolution spectrograms are generated coarse-to-fine, using multiscale ordering to address the challenge of capturing global structure in hundreds of thousands of dimensions.The initial tier captures high-level structure; later upsampling tiers add details, and the initial tier is upsampled by a factor of 32.Source: §6 · p. 5, Figure 5 · p. 6
- The spectrogram is partitioned into G tiers of progressively higher-resolution information, with each tier modeled by a separate network.The distribution is factorized over tiers, and each tier is further factorized element-by-element.Source: §6 · p. 5
6.1. Training
- Training recursively partitions alternating rows along the time or frequency axis to form the tiers.At the initial recursive step, even-numbered rows form xG and odd-numbered rows form x<G; subsequent tiers are defined recursively.Source: §6.1 · p. 6
- A feature-extraction network conditions each tier on preceding tiers using a multidimensional RNN with four one-dimensional RNNs running bidirectionally along slices of both axes.Because each tier doubles resolution, features from preceding tiers have the same time and frequency shape as the tier being generated.Source: §6.1 · p. 6
6.2. Sampling
- Sampling generates the initial tier unconditionally, samples each later tier conditioned on preceding tiers, and interleaves the tiers until a full spectrogram is formed.The interleaving alternates along frequency and time axes as shown in the tier-generation schematic.Source: §6.2 · p. 6, Figure 6 · p. 6
7. Experiments
Experiments assess MelNet across unconditional audio generation and text-to-speech using human judgments and density estimates. Human evaluators favored its long-range structure over WaveNet, and density results favored fine-grained autoregression over coarser assumptions.
7. Experiments
- Experiments span three unconditional tasks and single- and multi-speaker TTS across four public datasets; unconditional samples receive qualitative review and WaveNet comparison, while TTS is also evaluated by density estimation.The density-estimation comparison uses surrogate models because existing TTS systems generally do not train with log-likelihood objectives; models are constrained to roughly equal parameter counts.Source: §7 · pp. 6–7, §7.1 · p. 7, §7.2 · p. 8
7.1.1. SINGLE-SPEAKER SPEECH
- On 140 hours of animated audiobook narration by one professional speaker, MelNet often generates coherent words and phrases while maintaining consistent prosody and speaker characteristics, even when speech is incoherent.Biasing tends to lengthen comprehensible word strings but makes delivery less expressive; priming preserves speaking style and intonation.Source: §7.1.1 · p. 7
7.1.2. MULTI-SPEAKER SPEECH
- On noisy, multilingual VoxCeleb2 speech, MelNet varies speaker characteristics and acoustic conditions; samples are generally unintelligible, but often identifiable by language and can sound realistic to listeners unfamiliar with that language.The dataset includes more than 2,000 hours of speech from speakers of 145 nationalities.Source: §7.1.2 · pp. 7–8
7.1.3. MUSIC
- On MAESTRO solo piano, MelNet generates melody and harmony, often with consistent tempo and variation in volume, timbre, and rhythm.The dataset contains over 172 hours of piano performances.Source: §7.1.3 · p. 8
7.1.4. HUMAN EVALUATION
- Listeners overwhelmingly judged MelNet’s ten-second samples more coherent in long-range structure than WaveNet’s across three unconditional tasks; it also outperformed MIDI-conditioned WaveNet in music.The second comparison is against a two-stage model using a Music Transformer trained on labelled MIDI and a conditional WaveNet.Source: Table 2 · p. 8, §7.1.4 · p. 8
7.2.1. SINGLE-SPEAKER TTS
- For expressive single-speaker Blizzard TTS, MelNet produces diverse vocalizations for the same text and can preserve a primed speaker style when synthesizing unseen text.It also reads dialogue with varied character voices and narrative text relatively inexpressively.Source: §7.2.1 · p. 8
7.2.2. MULTI-SPEAKER TTS
- On noisy TED-LIUM 3, MelNet produces realistic multi-speaker TTS conditioned on speaker IDs, alongside disfluencies, repeated or skipped words, applause, and laughter.The dataset has unpunctuated, unnormalized, error-prone transcripts, noisy speaker IDs, and varied recording conditions.Source: §7.2.2 · p. 9
7.2.3. DENSITY ESTIMATION
- Fine-grained autoregression improves density estimates for unconditional speech and TTS; unimodal and independent frame assumptions hurt, latent conditioning helps, but VAEs remain below full autoregression and multimodal outputs help.Table 3 reports negative log-likelihood in nats/dim, with lower values better.Source: Table 3 · p. 9, §7.2.3 · p. 9
8. Related Work
MelNet addresses long-range audio structure through spectrogram modeling and end-to-end generation, complementing waveform models that produce higher-fidelity audio. The related work contrasts this approach with intermediate-feature pipelines, hierarchical models, and prior autoregressive image methods.
8. Related Work
- MelNet captures high-level structure more effectively, while WaveNet produces higher-fidelity audio; the approaches may be combined.Time-domain models have also been used to invert spectral representations to high-fidelity audio.Source: §8 · p. 10
- Unlike waveform pipelines using annotated linguistic features or MIDI, MelNet requires no intermediate supervision for TTS and learns high-level musical structure end to end.The cited pipelines depend on domain-specific representations and annotated features.Source: §8 · p. 10
- Hierarchical autoencoder approaches require sequentially trained model stages, whereas MelNet’s multiscale tiers can be parallelized and directly optimize data likelihood.The approaches are described as largely complementary because their modeling techniques can also apply to autoregressive models.Source: §8 · pp. 10–11
- GAN audio approaches leave mode coverage unclear and generate fixed-duration segments, limiting their use in many audio-generation tasks.Source: §8 · p. 10
- MelNet models the full conditional distribution of audio given text, unlike TTS systems designed to produce a single high-quality sample.The authors report that its flexible probabilistic model supports realistic textures and direct inversion to high-fidelity audio without a neural vocoder.Source: §8 · p. 10
- MelNet adapts autoregressive image-modeling ideas to spectrograms, using factorized multidimensional recurrence that processes an M × N grid in max(M, N) sequential steps.Tightly coupled recurrences require M + N steps; the factorized approach also permits optimized one-dimensional RNN implementations.Source: §8 · p. 10
9. Conclusion
MelNet combines an expressive autoregressive model with multiscale modeling to generate high-resolution spectrograms with local and global structure. The paper reports promising results across unconditional speech, music, and text-to-speech tasks.
9. Conclusion
- MelNet combines a highly expressive autoregressive model with multiscale modeling to generate high-resolution spectrograms with realistic local and global structure.Source: §9 · p. 11
- Compared with models of time-domain signals, MelNet is particularly well-suited to modeling long-range temporal dependencies.Source: §9 · p. 11
- Experiments report promising results on unconditional speech generation, music generation, and text-to-speech synthesis.Source: §9 · p. 11
A.1. Human Evaluation
The appendix describes the human-evaluation comparison and implementation choices for the WaveNet baseline and density-estimation experiments.
A.1. Human Evaluation
- Human evaluation compared 50 randomly drawn ten-second samples from each model per unconditional task, with 50 judgments per task on long-term coherence.Source: §A.1 · p. 14
A.2. WaveNet Baseline
- Blizzard and VoxCeleb2 human-evaluation baselines used a reimplemented 8-bit µ-law WaveNet trained for 150,000 steps; MAESTRO evaluations instead sampled from website audio.For MAESTRO, 50 random ten-second slices were drawn from roughly two minutes of WaveNet and Wave2Midi2Wave samples.Source: §A.2 · p. 14
A.3. Density Estimation
- Density estimation used Blizzard low-resolution spectrograms with 80 mel channels, a 512-sample STFT hop, and a 6 · 512-sample window.Baseline models used 1024 hidden units and MelNet models 512; all used eight-layer autoregressive decoders, with additional VAE inference-network and latent-vector settings.Source: §A.3 · p. 14