Source-linked AI summary
Empirical Variational Autoencoder
Kaede Shiohara
TL;DR
Continuous-valued sequence generation remains challenging because direct regression misses multimodality and conventional VAEs can have a gap between posterior dependencies and their factorized prior. EVA learns an empirical autoregressive latent prior, achieving competitive generation quality with efficient ancestral sampling.
Problem
For continuous-valued sequences, direct regression misses full next-token density, and conventional VAEs can mismatch latent posterior dependencies with a factorized standard-Gaussian prior.
Source: §1 · p. 1Method
EVA replaces the fixed standard-Gaussian constraint with a self-predicted autoregressive latent prior learned empirically from data.
Source: §3 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
EVA’s empirical prior addresses the prior–posterior distribution gap and supports high-fidelity sampling for continuous-valued sequence generation.
Source: §6 · p. 8EVA linearizes spatial latent maps into a 1D causal sequence and is evaluated on closed-set ImageNet and VGGSound benchmarks at moderate model scales.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present Empirical Variational Autoencoder, a general generative framework for continuous-valued (i.e., non-vector-quantized) sequences. EVA is based on the evidence lower bound of the Variational Autoencoder (VAE) but learns autoregressive latent priors empirically from training data, which can be implemented only by an additional single linear layer on top of VAEs. By replacing the conventional standard-Gaussian constraint with the self-predicted priors, EVA significantly alleviates the latent distribution gap between prior and posterior which is typically observed in conventional VAEs, and leads to high-fidelity ancestral sampling for sequential data generation. Extensive experiments on image and sound synthesis demonstrate that EVA achieves competitive generation quality with autoregressive diffusion baselines despite its much faster inference time.
1. Introduction
Continuous-valued sequence generation challenges autoregressive models because direct regression misses multimodality and uncertainty, while existing density heads impose fixed families or add inference cost. EVA replaces the VAE’s fixed latent prior with an empirical autoregressive prior, achieving competitive generation quality with more efficient ancestral inference.
1. Introduction
- Direct regression losses such as mean squared error do not model full next-token densities, while Gaussian mixture and diffusion heads impose fixed families or substantial inference costs.These limits complicate continuous-valued sequence generation with autoregressive models.Source: §1 · p. 1
- Conventional VAEs can mismatch posterior dependencies across latent positions with a factorized standard-Gaussian prior, degrading sampling quality and contributing to their use as compressors or tokenizers [27] [43].Source: §1 · pp. 1–2
- EVA learns an autoregressive latent prior empirically, aligning it with the posterior and achieving competitive results with autoregressive generators [18] [38] at more efficient inference than diffusion-based approaches.Experiments cover a toy benchmark, ImageNet [28], and VGGSound [6].Source: §1 · p. 2
2. Related Work
Prior work addresses sequence generation through discrete tokenization, continuous-valued autoregressive density models, and more expressive VAE priors or latent structures. EVA instead models continuous tokens and autoregressively conditions each latent position on all preceding positions.
2. Related Work
- Discrete autoregressive models have succeeded on sequential data, while discretizing continuous signals enables token prediction but can introduce quantization error and optimization difficulties [22].Examples include pixel, waveform, and latent-token prediction [30] [41] [42], [40], and [9] [36] [43].Source: §2 · p. 2
- Continuous autoregressive alternatives use Gaussian mixtures with a fixed number of components [11] [38], diffusion [18], or energy scores [31], trading off expressiveness, inference efficiency, or mode coverage.Source: §2 · p. 2
- Early VAEs [16] represent an observation with a single global latent, creating an inherent trade-off between reconstruction fidelity and prior regularization; later work introduces expressive priors [37], [3], and combines VAEs with AR decoders [4] [12] [26].Source: §2 · p. 2
- Hierarchical VAEs factorize within spatial latent maps, leaving dependencies unmodeled and prior–aggregate posterior mismatch reported [1]; EVA conditions each latent token on all preceding positions.Examples of hierarchical VAEs include NVAE [39] and VDVAE [7].Source: §2 · p. 2
3. Proposed Method
EVA retains the VAE ELBO while learning an autoregressive latent prior from data; in a multimodal toy experiment, its samples concentrate more tightly on true modes than the compared model families.
3.2. Empirical Variational Autoencoder (EVA)
- EVA keeps the VAE ELBO but replaces the fixed standard-Gaussian prior with an autoregressive latent prior learned empirically from data.The prior predictor is an additional network; it can share the causal Transformer backbone with the decoder using separate linear heads.Source: §3 · p. 2, §3.2 · p. 4
3.1. Revisiting Variational Autoencoder
- A fixed standard-normal prior creates a trade-off: reconstruction encourages data-dependent posteriors to diverge from the prior, while the KL term pulls them back.This makes accurate reconstruction and high-quality prior sampling difficult to obtain simultaneously.Source: §3.1 · p. 3
3.2. Empirical Variational Autoencoder (EVA)
- For sequences, EVA assumes a causal decoder, autoregressive prior, and mean-field posterior, then samples latents autoregressively and decodes each token causally.The predictor learns pω(zt | z1:t−1) rather than imposing the same standard-Gaussian prior across temporal positions.Source: §3.2 · pp. 3–4, Figure 2 · p. 3
3.3. Multimodal Sequence Structure
- The toy benchmark varies latent branching modes and observable modes, using settings (1, 3), (2, 2), and (2, 3) for (Mlat, Mobs).The synthetic process generates observations from latent states with branching transitions.Source: §3.3 · pp. 4–5
- EVA concentrates samples on true modes and suppresses inter-mode samples, while AR averages modes, GMM exceeds its finite capacity, and causal VAE retains inter-mode samples.The toy experiment compares generated samples in the joint (x0, x1) space.Source: §3.3 · pp. 5–6, Figure 3 · p. 5
4. Experiments
Experiments evaluate EVA on ImageNet image synthesis and VGGSound sound synthesis. EVA is competitive with autoregressive generators, while the best-metric diffusion variant is substantially slower.
4.1. Setup.
- Experiments cover ImageNet and VGGSound; EVA adds a single linear latent-prior prediction layer to Causal VAE, and baselines share a 170M-parameter Transformer backbone.The VGGSound dataset is third-party-provided and lacks part of the original dataset.Source: §4 · p. 6, §4.1 · p. 6
4.2. Comparison with Baselines
- Direct-regression AR fails, GMM improves generation quality but remains restrictive, and Causal VAE performs poorly; EVA is competitive at comparable inference cost.The results suggest that learning an autoregressive latent prior is critical for high-fidelity continuous autoregressive generation.Source: §4.2 · p. 7
- At matched inference depth, EVA achieves substantially lower FID with fewer inference parameters; AR-Diffusion (321M) has the best metrics but is 10 times slower than EVA at comparable FID.AR-Diffusion (236M), matched to EVA's inference depth, has much worse FID.Source: §4.2 · p. 7
4.3. Framework Analysis.
- EVA's latent space gives semantically similar patches similar distributions, while Causal VAE retains spatially structured posterior variation despite standard-Gaussian regularization.Figure 4 compares patches using bi-directional KL divergence; lighter patches are more similar to the source patch.Source: §4.3 · p. 7, Figure 4 · p. 7
- EVA improves generation FID while maintaining reconstruction fidelity, with Causal VAE and EVA achieving comparable reconstruction fidelity as measured by rFID.The authors conclude that generation fidelity can improve without sacrificing reconstruction fidelity when latent capacity is sufficient.Source: §4.3 · p. 8, Table 2 · p. 8
- EVA completes images from prefix latents with diverse samples; completed patches align with the supplied class conditions, including changed conditions.Figure 5 marks the regions to be regenerated with darker areas in the first row.Source: Figure 5 · p. 7, §4.3 · p. 8
- FID 7.68 is best for EVA-B at latent dimension 128; EVA-L reaches FID 6.04 with latent size 64 and CFG scale 7.Increasing latent size beyond 128 brings no further gains and can hurt performance; larger backbones improve FID.Source: Figure 6 · p. 8, §4.3 · p. 8
5. Limitations and Future Work
EVA’s stated limitations are its 1D causal ordering of spatial latent maps and evaluation at moderate scale on closed-set ImageNet and VGGSound benchmarks. The authors identify spatially specialized priors and broader downstream tasks as future work.
5. Limitations and Future Work
- EVA linearizes spatial latent maps into a 1D causal sequence, so it does not explicitly exploit their 2D spatial structure.The authors suggest spatially specialized autoregressive priors as a possible way to improve image generation.Source: §5 · p. 8
- The evaluation focuses on closed-set generation benchmarks on ImageNet and VGGSound and on moderate model scales.The authors propose validation on larger-scale, more diverse tasks, including text-to-image, text-to-speech, and multimodal generation.Source: §5 · p. 8
- Larger-scale and more diverse downstream tasks remain future validation targets.Examples named by the authors include text-to-image, text-to-speech, and multimodal data generation.Source: §5 · p. 8
6. Conclusion
EVA learns its latent prior empirically using an additional linear layer, rather than the traditional VAE prior. On ImageNet and VGGSound, it achieves competitive generation quality against autoregressive diffusion models while supporting efficient ancestral sampling.
6. Conclusion
- EVA learns its latent prior empirically from data using an additional single linear layer on top of a VAE.The authors say this addresses the prior-posterior distribution gap and leads to high-fidelity sampling.Source: §6 · p. 8
- On ImageNet and VGGSound, EVA achieves competitive generation quality against autoregressive diffusion models.The conclusion also reports efficient ancestral sampling.Source: §6 · p. 8
- The authors hope EVA will encourage renewed consideration of VAEs as standalone generative models.Source: §6 · p. 8
A. Additional Experiments
The additional experiments describe a system-level comparison on ImageNet-256 and provide uncurated generated examples. The comparison distinguishes bidirectional models from causal models by how they use image-token context.
A. Additional Experiments
- VAR and SiT exchange information across the full spatial extent at each generation stage, while LlamaGen and EVA condition on previously generated raster-scan tokens.Source: A. Additional Experiments · p. 9
- Figures 7 and 8 show uncurated EVA-L images, with each block sharing the same class condition.The authors also direct readers to the project page to listen to generated sounds.Source: A. Additional Experiments · p. 9, Figure 7 · p. 9, Figure 8 · p. 10
- Table 3 compares recent state-of-the-art models on ImageNet-256 using inference parameters, FID, and IS.The passage identifies VAR, SiT, JiT, and LlamaGen among the compared models, but does not provide their table values.Source: Table 3 · p. 9, A. Additional Experiments · p. 9
B.1. On the Toy Problem
The toy experiment uses a one-dimensional latent process and observations to examine sequential multimodality. It describes autoregressive baselines and VAE/EVA parameterizations, then samples unconditionally through each model’s generative pathway.
B.1. On the Toy Problem
- The toy benchmark uses a one-dimensional latent process and one-dimensional observations, with xBOS=0 as a fixed beginning-of-sequence token.Source: §B.1 · p. 11
- The construction distinguishes latent branching modes, controlled by Mlat, from visible observation multimodality, controlled by Mobs.Observation-level offsets are placed on a one-dimensional grid with spacing 2.4.Source: §B.1 · p. 11
- The autoregressive baselines use separate MLPs for successive conditionals; the GMM variant replaces scalar outputs with mixture logits, means, and log-standard-deviations.Source: §B.1 · p. 11
- EVA keeps the VAE encoder and decoder structure but replaces its unconditional latent prior with predictive latent priors.The models are trained for 20,000 gradient steps, and evaluation samples follow each model’s native pathway from the BOS token.Source: §B.1 · pp. 11–12
B.2. On the ImageNet and VGGSound
EVA and Causal VAE use matched 768-wide encoder and decoder Transformers, while autoregressive baselines use a 24-block causal Transformer. The implementation details specify mixture, diffusion, and normalizing-flow heads, sampling choices, and training settings.
Transformer Backbone for Causal VAE and EVA
- EVA and Causal VAE use 12-block, 768-wide encoder and decoder Transformers, with full encoder attention and strictly causal decoder attention.Both use 12 attention heads, MLP expansion ratio 4.0, and RoPE; the model prepends a class token and four register tokens.Source: §B.2 · p. 12
Transformer Backbone for AR Baselines; AR-GMM
- AR baselines use a 24-block causal Transformer, while the AR-GMM head predicts mixture weights, means, and scales for each next latent token.The scale parameters use a square-plus transform and a 10^-6 lower bound; the model minimizes mixture negative log-likelihood and follows GIVT [38] for its head implementation.Source: §B.2 · p. 12
AR-Diffusion
- AR-Diffusion uses a 12-residual-block AdaLN-conditioned MLP head and a cosine diffusion noise schedule.It jointly predicts mean-related noise and variance terms; inference uses DDIM rather than DDPM for efficiency and fidelity, following MAR [18] with this sampler change.Source: §B.2 · p. 12
Causal VAE w/ NF
- The Causal VAE normalizing-flow prior adds 88.1M parameters across six transformations and is trained jointly with the latent reconstruction objective.Following SimFlow [45], it minimizes change-of-variables negative log-likelihood for posterior samples under a standard Gaussian base distribution.Source: §B.2 · p. 12
Additional Training Details
- Training uses AdamW with 0.02 weight decay, 50-epoch linear warmup, and EMA decay 0.9999; ImageNet training takes about 3 days and VGGSound training 12 hours on eight RTX5090 GPUs.Class conditioning is dropped with probability 0.1, with a learnable unconditional token used for CFG.Source: §B.2 · p. 12