Source-linked AI summary
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
Antoine Caillon, Philippe Esling
TL;DR
Raw-waveform audio generation remains computationally demanding, often uses low sampling rates, or restricts signal types, while VAEs provide control but commonly have low synthesis quality. RAVE combines two-stage VAE and adversarial training with multiband decomposition and latent-space analysis. It generates 48kHz audio at 20 times faster than realtime on a standard laptop CPU and reports superior synthesis quality to existing models.
Problem
Raw-waveform audio models can be computationally intensive, operate at low sampling rates, be difficult to control, or restrict the signals they generate, while VAEs often provide low synthesis quality.
Method
RAVE uses two-stage representation learning and adversarial fine-tuning, multiband waveform decomposition, and post-training latent-space analysis.
Results
RAVE reports superior synthesis quality to existing models while generating 48kHz audio at 20 times faster than realtime on a standard laptop CPU.
Takeaways & Limitations
RAVE provides fast, high-quality neural audio synthesis while allowing control over the trade-off between reconstruction fidelity and representation compactness.
Abstract
from arXiv · showhide
Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks. However, as raw waveform modelling remains an inherently difficult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU. We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.
1 Introduction
RAVE addresses the difficulty of generating high-quality audio by combining variational representation learning with adversarial fine-tuning, multiband waveform decomposition, and latent-space analysis. The approach targets 48kHz synthesis while remaining suitable for real-time CPU use and balancing reconstruction fidelity against representation compactness.
- RAVE is evaluated as superior to several state-of-the-art models in unsupervised audio modeling, with accompanying audio samples provided online.
- RAVE uses a two-stage procedure: regular VAE representation learning followed by adversarial fine-tuning for high-quality synthesis.The procedure first learns the representation, then optimizes generation quality with an adversarial objective.
- A post-training latent-space analysis balances reconstruction fidelity against representation compactness.
- 48kHz audio synthesis is achieved through multiband waveform decomposition without a major increase in computational complexity.
- 20 times faster than realtime synthesis is achieved on a standard laptop CPU.
2 State-of-art
This section frames audio generation as difficult because waveform models are costly, autoregressive synthesis is slow, and some efficient methods restrict signal types or sacrifice quality. VAEs offer latent control, but their probabilistic formulation and representation design introduce reconstruction and dimensionality challenges.
- VAEs model data with latent variables and use an inference model qφ(z|x) to approximate the posterior p(z|x).
- The VAE ELBO combines reconstruction likelihood with KL regularization toward a predefined prior, with β controlling the reconstruction–regularization trade-off.
- Autoregressive waveform models require substantial data and parameters, synthesize prohibitively slowly, and can accumulate errors.
- Deterministic latent encoding in NSynth prevents prior-based sampling, restricting applications to simple reconstructions and interpolations.
- A feed-forward spectral-distance autoencoder is 2500 times faster than NSynth during synthesis, but has degraded sound quality.
- Adversarial parallel audio modeling trains a discriminator against generated samples and adds feature matching between real and synthesized audio.
3 Method
RAVE uses two-stage training to learn audio representations with a spectral reconstruction objective, then improve synthesis quality adversarially. Post-training latent analysis controls the trade-off between fidelity and compactness.
- 3.1.1 Stage 1: Representation learning: Stage 1 trains the encoder and decoder with a multiscale spectral distance derived from the ELBO for representation learning.The amplitude-spectrum distance avoids penalizing phase inaccuracies while capturing perceptually important signal features.
- 3.1.2 Stage 2: Adversarial fine-tuning: Stage 2 freezes the encoder and fine-tunes the decoder with an adversarial objective to improve synthesized audio quality and naturalness.The decoder continues minimizing spectral distance and also uses feature matching to keep outputs close to ground truth.
- 3.2 Latent representation compactness: The latent-space analysis identifies informative dimensions by centering posterior samples and applying SVD, separating them from dimensions collapsed to the prior.Collapsed dimensions become constant after variance removal and are set to zero by centering.
- 3.2 Latent representation compactness: A fidelity parameter f determines the retained rank rf, allowing latent dimensionality to vary with the dataset and desired reconstruction fidelity.The reduced representation keeps the first rf SVD dimensions, adds prior noise, and is projected back before decoding.
4 Experiments
RAVE targets 48kHz audio using multiband decomposition and combines convolutional encoding with a structured decoder. Experiments evaluate its speed, representation learning, and audio modelling against established baselines on strings and speech.
- 4.1 RAVE: A 16-band decomposition enables RAVE to model 48kHz audio while producing a compact latent representation.The decomposition reduces temporal dimensionality and supports a larger receptive field without a major computational increase.
- 4.1 RAVE: The encoder combines multiband decomposition with a convolutional neural network that maps raw waveforms to 128-dimensional latent representations.
- 4.1 RAVE: The decoder uses waveform, loudness, and noise sub-networks to synthesize multiband audio, apply an amplitude envelope, and add filtered noise.
- 4.1 RAVE: An explicit amplitude envelope reduces artifacts in silent signal regions, while the noise synthesizer slightly improves naturalness for noisy signals.
- 4.1 RAVE: Experiments compare RAVE with unsupervised NSynth and SING using official implementations, while excluding DDSP and NSF because they target different modelling assumptions.RAVE is described as modelling both monophonic and polyphonic signals.
- 4.2 Datasets: The strings dataset contains approximately 30 hours of 48kHz recordings with monophonic and polyphonic configurations, split 90/10 for training and testing.
- 4.2 Datasets: VCTK provides approximately 44 hours of 48kHz speech from 110 speakers with varied accents, also using a 90/10 train/test split.
5 Results
RAVE achieves high-quality, fast audio synthesis while exposing a post-training fidelity control that trades reconstruction quality against latent compactness. It also supports domain transfer, compression, and latent-signal generation.
- 5.1 Synthesis quality: RAVE outperforms NSynth and SING in audio quality without autoregressive generation and uses at least 3.5 times fewer parameters.The evaluation included 33 participants rating unseen strings-dataset samples on a 1–5 scale.
- 5.2 Synthesis speed: 48kHz synthesis runs 20× faster than realtime on CPU and up to 240× faster than realtime on GPU.Multiband decomposition speeds RAVE by a factor of 25 and lets it outperform SING on both CPU and GPU.
- 5.3 Balancing compactness and fidelity: Setting f = 0.99 reduces latent dimensionality from 128 to 24 for strings and 16 for VCTK.The resulting latent representation is sampled at ∼23Hz, while decreasing f increases spectral distance and further reduces representation size.
- 5.3 Balancing compactness and fidelity: As f decreases, reconstructions become less accurate and lose attributes such as phonemes or speaker identity.Figure 3 visualizes reconstructions of a VCTK sample across fidelity values.
- Applications: RAVE performs domain transfer on audio outside its training distribution, including violin reconstruction with a speech-trained model.The model can also provide a compression ratio of 2048, producing latent signals sampled at ∼23Hz.
- Applications: Combining RAVE’s decoder with a WaveNet-inspired latent generator yields synthesis at a 5 times faster than realtime factor.The slower latent sampling rate enables a 3-second receptive field with as few as 9M parameters.
6 Related work
RAVE builds on VAE-GAN and multiband audio modeling while using adversarial fine-tuning as a generation objective rather than merely a convergence warmup. Its multiband design addresses the temporal complexity of 48kHz modeling.
- VAE-GAN approaches: Prior VAE-GAN approaches use learned or perceptual metrics, whereas RAVE’s perceptual-loss variant produces a larger estimated latent space dimensionality.This motivates RAVE’s distinct training strategy for representation learning and generation quality.
- Adversarial and perceptual objectives: Unlike related methods that combine perceptual and auxiliary losses mainly as warmup, RAVE uses both stages as its training procedure.The comparison concerns the role of these objectives rather than only their presence.
- Multiband audio modeling: A 16-band decomposition enables RAVE to model 48kHz audio despite its temporal complexity and memory requirements.Earlier multiband work also reported improved quality with reduced training and synthesis time.
7 Conclusion
The paper concludes that RAVE combines two-stage representation learning, high-quality synthesis, and multiband acceleration. It also provides a post-training control over latent compactness and reconstruction quality.
- Conclusion: RAVE first learns an adequate latent representation, then applies adversarial fine-tuning to generate high-quality audio signals.The conclusion presents this sequencing as the paper’s central training procedure.
- Conclusion: RAVE outperforms previous approaches in both quantitative and qualitative analyses.The conclusion states this as an overall evaluation result.
- Conclusion: Multiband raw-waveform decomposition enables synthesis 20 times faster than realtime on a standard laptop CPU.The paper uses multiband decomposition to address high-rate audio synthesis.
- Conclusion: Post-training latent-space analysis controls the trade-off between reconstruction quality and representation compactness.The method uses informative latent dimensions and low-rank projections to obtain compact representations.
- B Multiband decomposition: Multiband decomposition represents a high-rate signal as several downsampled sub-signals covering different frequency ranges.The filter-bank can be reapplied in reverse for resynthesis.
C Model architecture
RAVE uses a convolutional encoder and a decoder that upsamples latent representations before producing waveform, loudness-envelope, and filtered-noise outputs.
- Encoder: The encoder is a convolutional neural network with leaky ReLU activation, batch normalization, four layers, and a 128-dimensional latent space.Its hidden sizes are [64, 128, 256, 512], with strides [4, 4, 4, 2].
- Decoder: The decoder alternates upsampling layers with a residual stack to expand the latent representation.
- Decoder: Three decoder sub-networks respectively produce the waveform, loudness envelope, and filtered-noise signals.
- Decoder: The decoder architecture is organized into specialized decoder blocks used throughout RAVE.
D Latent component collapse
RAVE’s regularization makes latent components differ in informativeness, with most components collapsing toward the prior and only a few retaining substantial divergence.
- Latent component collapse: The KL regularization pressures the posterior toward the prior, making some latent dimensions more informative than others.
- Latent component collapse: Most latent components have KL divergence below 0.01 on both strings and VCTK datasets.
- Latent component collapse: Only 16 strings components and 9 VCTK components have KL divergence above 0.1.
- Latent component collapse: The observed informative-component counts are consistent with the dimensionality estimated at fidelity parameter f = 0.95.
E Two stage training and latent space compactness
RAVE avoids adversarial encoder training to preserve compact latent representations, contrasting with feature-matching fine-tuning that increases estimated dimensionality.
- Training procedure: RAVE excludes adversarial losses from encoder training during the two-stage procedure.
- Latent space compactness: Training the encoder with feature-matching loss during the second stage dramatically increases estimated latent-space dimensionality.
- Latent space compactness: Freezing the encoder during adversarial fine-tuning is compared with continued encoder training on the Strings dataset.
F Out of domain latent representation
RAVE supports timbre transfer while preserving some high-level attributes, but its latent prior alignment weakens for out-of-domain inputs.
- Timbre transfer: RAVE is demonstrated on the timbre-transfer task.
- Timbre transfer: Overall loudness and the fundamental frequency of harmonic components are preserved after domain transfer.
- Timbre transfer: Formants absent from strings examples are added in the speech-transferred versions.
- Out-of-domain representation: The encoder has no guarantee of mapping out-of-domain signals to latent representations matching the prior.
- Out-of-domain representation: Out-of-domain signals roughly double KL divergence relative to signals sampled from the training distribution.