Dirichlet Variational Autoencoder

1 INTRODUCTION

• The introduction situates DirVAE among Gaussian-prior VAEs and continuous approximations to discrete priors, including approaches supporting multinomial posteriors without explicitly modeling a Dirichlet prior.Source: §1 · p. 1
• Gaussian-Softmax cannot generate some Dirichlet multi-modal distributions, and GEM sampling may favor a few components through the rich-get-richer phenomenon.The Dirichlet can distribute weights across components and capture multi-modality by controlling its prior hyper-parameter.Source: §1 · p. 2, Figure 1 · p. 2
• DirVAE uses a Dirichlet prior to model multi-modal distributions and serve as a prior for discrete latent distributions.The proposed model is intended to retain characteristics of the Dirichlet distribution.Source: §1 · p. 2
• The inverse Gamma CDF approximation for DirVAE yields the best log-likelihood among the compared Weibull and softmax-Gaussian approximations.The approximation is applied to Gamma variables composing the Dirichlet distribution.Source: §1 · p. 2
• DirVAE shows neither near-zero decoder weights nor near-zero latent values, while Gaussian VAE and SBVAE exhibit decoder-weight and latent-value collapsing, respectively.The paper expands component collapsing to include both failure modes.Source: §1 · p. 2
• Image generation and latent-value visualizations show more meaningful or disentangled DirVAE representations.Source: §1 · p. 3
• DirVAE achieves the best classification performance in supervised and semi-supervised tasks, while DirVAE-augmented topic models improve perplexity and topic coherence, with most best performers augmented.The classification tasks use MNIST, OMNIGLOT, and SVHN; the topic-model experiments use 20Newsgroup and RCV1-v2.Source: §1 · p. 3

2.1 VARIATIONAL AUTOENCODERS

• A VAE combines a probabilistic decoder that generates observations from latent variables with an encoder that infers latent variables, and GVAE and SBVAE use Gaussian and GEM priors, respectively.The encoder and decoder parameters are jointly learned by optimizing an ELBO using stochastic gradients.Source: §2.1 · p. 3

2.2 DIRICHLET DISTRIBUTION AS A COMPOSITION OF GAMMA RANDOM VARIABLES

• A Dirichlet variable is formed by normalizing independent Gamma variables that share a rate parameter.For component k, Y_k = X_k / sum_i X_i; the passage also notes a KL-divergence derivation for the composed Gamma variables.Source: §2.2 · p. 3

2.3 SGVB FOR GAMMA RANDOM VARIABLE AND APPROXIMATION ON DIRICHLET DISTRIBUTION

• SGVB approximations for Gamma variables composing a Dirichlet include Weibull, softmax Gaussian, and inverse-Gamma-CDF approaches.The Weibull approximation offers closed-form KL divergence and inverse-CDF reparameterization, but its exponential power k can make optimization sensitive to small changes in k.Source: §2.3 · pp. 3–4

• DirVAE uses a Dirichlet prior for latent z, rather than the standard Gaussian prior, and the Dirichlet is conjugate to the multinomial.Source: §3 · p. 4
• The encoder maps observation x through an MLP with softplus outputs to positive Dirichlet posterior parameters, then normalizes Gamma-component samples to obtain z.The posterior is not sampled directly from the Dirichlet; the model draws from a Gamma composition and divides by the sum of its components.Source: §3 · p. 4
• The inverse Gamma CDF method enables stochastic-gradient backpropagation to the input, and the stated comparison sets β = 1 and α_k = 1 − 1/K for the specified Gaussian parameters.Source: §3 · p. 5
• The inverse Gamma CDF approximation replaces a Gamma draw with a deterministic function of its parameters and an auxiliary uniform random variable.For X ∼ Gamma(α, β), the passage gives F^-1(u; α, β) ≈ β^-1(u^αΓ(α))^(1/α), with u ∼ Uniform(0, 1).Source: §3 · p. 5
• The paper frames combining Dirichlet decomposition with inverse Gamma CDF approximation as a way to examine learning properties, component collapsing, and topic-modeling performance.Source: §3 · p. 5

4.1 EXPERIMENTS FOR REPRESENTATION LEARNING OF VAES

• DirVAE outperforms all baselines on log-likelihood for both MNIST and OMNIGLOT.The comparison includes negative log-likelihood, negative ELBO, and reconstruction loss; lower values are better for all measures.Source: §4.1 · p. 5, Table 1 · p. 6
• DirVAE has interpretable latent dimensions in most dimensions, while GVAE-Softmax and SBVAE have components without significant semantic information.The section assesses latent-dimension reconstructions and t-SNE visualizations of learned latent values.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 DISCUSSION ON COMPONENT COLLAPSING

• GVAE and GVAE-Softmax exhibit decoder weight collapsing, whereas SBVAE has latent value collapsing and DirVAE does not.SBVAE has many near-zero latent outputs; DirVAE samples relatively dense latent values and uses the full spectrum of latent dimensions compared with SBVAE.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 APPLICATION 1. EXPERIMENTS OF (SEMI-)SUPERVISED CLASSIFICATION WITH VAES

• DirVAE has the best performance among the three VAEs in semi-supervised classification, with greater improvement on the most complex SVHN task.The reported error rates use 10%, 5%, and 1% labeled data for each dataset.Source: §4.3 · p. 7, Table 2 · p. 8
• DirVAE gives the lowest kNN classification error among the tested VAEs on MNIST and OMNIGLOT latent representations.The section notes OMNIGLOT's 50-category test setting, compared with 10 categories in MNIST.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 APPLICATION 2. EXPERIMENTS OF TOPIC MODEL AUGMENTATION WITH DIRVAE

• DirVAE augmentation improves topic-model performance in general, and produces the best result on both measures except RCV1-v2 perplexity, where it remains competent.The measures are perplexity and topic coherence via NPMI, evaluated on 20Newsgroups and RCV1-v2.Source: §4.4 · p. 8

5 CONCLUSION

• DirVAE models latent values as categorical probabilities and shows better representation learning across classification and topic-model experiments, with no component collapsing.The conclusion says DirVAE can serve as a building block for complex probabilistic models with neural networks, given the conjugate relation between multinomial and Dirichlet distributions.Source: §5 · p. 9

APPENDIX

• The appendix introduces derivations of key equations and experimental-setting details used in the paper.It says model names, parameter names, and experiment assumptions are based on the main paper.Source: APPENDIX · p. 11

A MONTE-CARLO ESTIMATION ON THE MARGINAL LIKELIHOOD

• Proposition A.1 states an approximation for the marginal log-likelihood.Source: A MONTE-CARLO ESTIMATION ON THE… · p. 11

B KL DIVERGENCE OF TWO MULTI-GAMMA DISTRIBUTIONS

• Appendix B derives KL divergence between two MultiGamma distributions with a shared rate parameter; ψ is the digamma function.Source: B KL DIVERGENCE OF TWO MULTI-GAMMA… · p. 11

C HYPER-PARAMETER α LEARNING STRATEGY

• The method-of-moments estimator updates Dirichlet prior parameter α from sampled proportions, alternating with neural-parameter updates after burn-in.The authors choose this estimator for its closed-form nature and consistency.Source: C HYPER-PARAMETER α LEARNING STRATEGY · p. 12

D EXPERIMENTAL SETTINGS

• The appendix lists the TensorFlow implementation as “TO BE RELEASED.”Source: D EXPERIMENTAL SETTINGS · p. 13

D.1 DATASET DESCRIPTION

• MNIST contains 60,000 training and 10,000 testing images of 28 × 28 handwritten digits across 10 labels.Source: §D.1 · p. 13
• OMNIGLOT contains 24,345 training and 8,070 testing 28 × 28 character images across 50 labels.Source: §D.1 · p. 13
• The benchmarks also include MNIST with rotations and SVHN reduced by PCA to 500 dimensions.Source: §D.1 · p. 13

D.2 REPRESENTATION LEARNING OF VAES

• The MNIST split is 45,000/5,000/10,000 for train/validation/test, while OMNIGLOT uses 22,095/2,250/8,070.Source: §D.2 · p. 13
• MNIST uses 50 latent dimensions and OMNIGLOT 100, each with two encoder hidden layers and one 500-dimensional decoder hidden layer.Source: §D.2 · p. 13
• Both datasets use gradient clipping, ReLU, Xavier initialization, and Adam; SBVAE uses a 3e-4 learning rate while other VAEs use 5e-4.The marginal log-likelihood estimate uses 100 samples for each of 1,000 randomly selected test examples.Source: §D.2 · p. 13
• The experiments add GVAE-NF20 as a baseline and provide latent decoder-weight norms and t-SNE visualizations for MNIST.Source: §D.2 · p. 13
• DirVAE-Learning improves marginal log-likelihood, ELBO, and reconstruction loss on both MNIST and OMNIGLOT.It updates α through burn-in, alternating neural-parameter and α updates, then neural-parameter updates with α fixed.Source: §D.2 · p. 13

D.3 SEMI-SUPERVISED CLASSIFICATION TASK WITH VAES

• The semi-supervised setup uses a VAE with separate latent variables z and y, following the SBVAE structure.Source: §D.3 · p. 13
• Table 5 covers negative log-likelihood, negative ELBO, and reconstruction loss for VAEs on MNIST and OMNIGLOT.Source: Table 5 · p. 14
• The inference model factors qφ(z, y|x) into qφ(z|x)qφ(y|x), with qφ(y|x) predicting unseen labels.The training objective is a weighted sum of the labeled and unlabeled objectives, with 0 < λ < 1.Source: §D.3 · p. 14
• MNIST, MNIST+rot, and SVHN use specified train/validation/test splits; SVHN is reduced to 500 dimensions by PCA.The VAEs use 50-dimensional latent variables, with λ set to 0.375 for MNIST variants and 0.45 for SVHN.Source: §D.3 · p. 14
• Figure 6 visualizes the optimized dimension-wise α values learned by DirVAE-Learning on MNIST.Source: Figure 6 · p. 15

D.4 SUPERVISED CLASSIFICATION TASK WITH LATENT VALUES OF VAES

• The supervised classification comparison uses the same experimental settings as the preceding section and reports kNN error rates for VAE latent representations.Source: §D.4 · p. 15, Table 6 · p. 15
• DLGMM is described as a more complex model than the VAE alternatives, so its published result from Nalisnick et al. (2016) is used for comparison.Source: §D.4 · p. 15
• VaDE is omitted because the authors characterize it as more customized for clustering than the ordinary VAE baselines selected here.Source: §D.4 · p. 15

D.5 TOPIC MODEL AUGMENTATION WITH DIRVAE

• The topic-model evaluation uses perplexity and NPMI topic coherence on 20Newsgroups and RCV1-v2; lower perplexity and higher NPMI are better.The RCV1-v2 experiment samples 20,000 training and 10,000 test documents; 20Newsgroups uses 11,258 training and 7,487 test documents.Source: §D.5 · p. 15
• The experiments replace each topic model's prior with DirVAE and select hyperparameters using 1,000 randomly selected test documents.The setups use 50 topics for 20Newsgroups and 100 topics for RCV1-v2.Source: §D.5 · p. 15, Table 7 · p. 16
• Figures 7–9 show t-SNE document embeddings for 20Newsgroups across DirVAE-augmented, Stick-Breaking-augmented, and original topic models.Source: §D.5 · p. 15, Figure 7 · p. 17, Figure 8 · p. 17
• Table 8 presents top-10 high-probability words for topics learned on 20Newsgroups by activating single latent dimensions.Examples include Turkish-Armenian terms in Topic 1, religious terms in Topic 3, and law-related terms in Topic 8.Source: §D.5 · pp. 15–16, Table 8 · p. 16
• The listed topic words also include computer hardware in Topic 5, software and space in Topics 6–7, prices in Topic 9, and internet communication in Topic 10.Source: §D.5 · p. 16