Source-linked AI summary
Improving Variational Auto-Encoders using Householder Flow
Jakub M. Tomczak, Max Welling
TL;DR
Diagonal-covariance posteriors are computationally efficient but may lack flexibility, motivating richer variational posteriors. The paper proposes a volume-preserving Householder flow and reports flexible posteriors and competitive results on MNIST and histopathology data.
Problem
Diagonal-covariance variational posteriors are efficient, but normalizing-flow research asks whether cheap, tractable transformations can provide more flexible posterior distributions.
Method
The paper models orthogonal transformations with a series of Householder transformations to construct a volume-preserving flow for variational posteriors.
Results
The proposed Householder flow produced competitive results against other normalizing flows and helped obtain more flexible posteriors on MNIST and histopathology data.
Takeaways & Limitations
Householder flows provide a computationally cheap way to enrich variational posteriors while requiring only T × M additional parameters relative to a vanilla VAE.
Takeaways & Limitations
The paper leaves investigating the flow in further variational-inference extensions and broader natural and medical-image modeling for future research.
Abstract
from arXiv · showhide
Variational auto-encoders (VAE) are scalable and powerful generative models. However, the choice of the variational posterior determines tractability and flexibility of the VAE. Commonly, latent variables are modeled using the normal distribution with a diagonal covariance matrix. This results in computational efficiency but typically it is not flexible enough to match the true posterior distribution. One fashion of enriching the variational posterior distribution is application of normalizing flows, i.e., a series of invertible transformations to latent variables with a simple posterior. In this paper, we follow this line of thinking and propose a volume-preserving flow that uses a series of Householder transformations. We show empirically on MNIST dataset and histopathology data that the proposed flow allows to obtain more flexible variational posterior and competitive results comparing to other normalizing flows.
1 Variational Auto-Encoder
VAEs introduce an inference model to optimize a tractable variational lower bound when marginal likelihood is intractable. Their usual diagonal-covariance posterior is efficient but may be too inflexible to match the true posterior.
- The marginal likelihood can be intractable when the model is parameterized by a neural network.
- An inference model q(z|x), called an encoder, is introduced to optimize a variational lower bound.
- For continuous latent variables, reparameterization enables efficient optimization of the lower bound and yields a variational auto-encoder.
- The decoder models p(x|z), while the prior is p(z) = N(z|0, I).
- The commonly assumed diagonal posterior covariance can be insufficiently flexible to match the true posterior.
2 Improving posterior flexibility using Normalizing Flows
Normalizing flows enrich a VAE posterior by transforming samples from a simple distribution through invertible mappings. Volume-preserving flows retain flexible posteriors while setting the Jacobian determinant to one, reducing computational complexity.
- A finite normalizing flow starts with a simple random variable and applies invertible transformations to obtain a more flexible final distribution.
- The flow objective is tractable when the Jacobian determinant of each transformation can be computed.
- Normalizing flows can enrich a VAE posterior with small or no architectural changes to the encoder and decoder.
- Volume-preserving flows use transformations with Jacobian determinant equal to 1, making them computationally appealing while retaining flexible posterior distributions.
3 Householder Flow
The Householder flow models flexible full-covariance posteriors by representing the orthogonal component through Householder transformations. Because each transformation is orthogonal and volume-preserving, the flow retains a tractable Jacobian while requiring only linear operations.
- Motivation: Σ = UDU⊤ decomposes covariance into eigenvalues D and an orthogonal eigenvector matrix U, motivating direct modeling of U.
- Orthogonal matrix representation: Any orthogonal matrix of degree K can be expressed as a product of exactly K Householder transformations.
- Volume preservation: Each Householder matrix is orthogonal, so its Jacobian determinant has absolute value 1 and simplifies the variational objective.
- Flow construction: Starting from a diagonal-covariance posterior, T encoder-produced vectors define a Householder flow that yields a more flexible posterior with an approximate full-covariance matrix.The encoder produces the vectors alongside posterior means and variances; T linear operations generate a sample.
- Architecture: Figure 1 depicts the encoder producing posterior parameters and the first Householder vector, followed by individual linear Householder-flow steps.
4 Related Work
Prior flows use either analytically tractable Jacobians or volume-preserving transformations, while Householder flow applies linear orthogonal transformations to enrich the posterior. It is similar in spirit to linIAF but uses Householder matrices instead of lower-triangular inverse Cholesky matrices.
- Normalizing flows: Invertible linear flows have analytically computable Jacobians, but many transformations may be needed to capture high-dimensional dependencies.
- Volume-preserving flows: Volume-preserving flows set the absolute Jacobian determinant to 1 while retaining flexible posterior distributions, reducing computational complexity.
- Comparison with linIAF: Householder flow resembles linIAF through linear transformations, but uses Householder matrices rather than lower-triangular inverse Cholesky matrices with unit diagonal.
- Other uses: Householder transformations were also used in recurrent neural networks to represent unitary weights and matrix eigenvectors, rather than variational posteriors.
5 Experiments
Experiments on MNIST and grayscale histopathology patches compared Householder flows with vanilla VAEs and other normalizing flows. The results indicate more flexible posteriors, competitive likelihood bounds, and favorable computational characteristics.
- 5.1 MNIST: MNIST experiments compared VAE+HF with T=1 and T=10 against VAE+NF, VAE+NICE, and VAE+HVI using test-set marginal log-likelihood bounds.The experiment used 60,000 training images, 10,000 test images, and a two-layer encoder and decoder with 40 stochastic hidden units.
- 5.1 MNIST: −85.1 was the reported ≤ln p(x) for VAE+NF (T=80), compared with −87.5 for VAE+NF (T=10), −88.6 for VAE+NICE (T=10), and −88.30 for VAE+HVI (T=8).Table 1 reports the lower bound in nats on the MNIST test set.
- 5.1 MNIST: Householder flows improved data modeling through lower reconstruction error and increased posterior flexibility through a smaller Kullback-Leibler penalty.The orthogonal component lets the initial posterior model eigenvalues rather than the full posterior information.
- 5.1 MNIST: Larger flow length did not improve MNIST performance, since taking larger values of T produced no better results.The tested Householder-flow lengths were T=1 and T=10.
- 5.2 Histopathology data: Histopathology experiments applied vanilla VAE and VAE+HF with T=1, 10, and 20 to grayscale image patches from bone marrow biopsies.The same architecture and training procedure were used, with a normal likelihood for grayscale data.
- 5.2 Histopathology data: The histopathology results confirmed greater posterior flexibility, showed slight improvement with 20 HF steps, and yielded less variable results than the vanilla VAE.The comparison used lower bounds of marginal log-likelihood on the histopathology test set, with three repetitions.
6 Conclusion
The paper proposes a volume-preserving Householder flow for variational inference. Experiments indicate competitive performance against other volume-preserving flows, while broader applications and image-modeling benefits remain future work.
- 6 Conclusion: The proposed method is a new volume-preserving flow constructed from Householder transformations.The conclusion identifies this flow as the paper’s central contribution.
- 6 Conclusion: Using the flow for importance-weighted VAE, Rényi Divergence for VAE, Ladder VAE, and natural or medical image modeling is left for future research.The paper specifically identifies these extensions and applications as future investigation.
Appendix
The appendix introduces a gating activation mechanism as an alternative to ReLU and illustrates its calculation for a single neural-network layer.
- A hidden layer is typically computed with the ReLU activation function.
- ReLU can also be expressed using an indicator function.
- The gating mechanism is presented as a newly proposed type of activation function.
- Figure 4 shows the calculation of the gating mechanism for a single layer.
- The notation sets h0 = x to keep the neural-network expressions uncluttered.