Variational Autoencoders Pursue PCA Directions (by Accident)

Michal Rolinek; Dominik Zietlow; Georg Martius

Source-linked AI summary

Variational Autoencoders Pursue PCA Directions (by Accident)

Michal Rolinek, Dominik Zietlow, Georg Martius

arXiv:1812.06775v2Dec 17, 2018cs.LGcs.CVstat.ML

TL;DR

VAE-based models achieve strong disentanglement despite lacking an explicit mechanism for choosing the required latent alignment. This paper analyzes the implemented VAE loss and shows that diagonal encoder covariance and stochasticity promote local decoder orthogonality, closely matching PCA’s alignment behavior. The results provide a theoretical, intuitive, and experimental explanation for why β-VAEs disentangle.

Problem
VAE-based architectures perform strongly on disentanglement, but the mechanism selecting the required axis-aligned latent representation remains unexplained.
Method
The paper analyzes the implemented VAE loss in a practically prevalent regime, including nonlinear behavior, to isolate the mechanism linking diagonal covariance, local orthogonality, and latent alignment.
Results
The analysis and experiments show that VAE mechanisms promote local orthogonalization and local PCA-like representations, explaining why β-VAEs disentangle.
Takeaways & Limitations
VAEs use variance differences to shape latent representations, while factorized representations require additional terms promoting independence.
Takeaways & Limitations
Orthogonality is promoted indirectly, motivating architectures that allow explicit control over this feature.

Abstract

from arXiv · show

The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.

1 Introduction

VAEs perform strongly in learning disentangled representations, yet the mechanism selecting the required latent alignment remains unexplained. This paper identifies an internal mechanism linking diagonal encoder covariance, local orthogonality, disentanglement, and PCA-like behavior.

Motivation: VAEs support generative modeling and representation learning across applications including image interpolation, one-shot generalization, language models, and speech transformation.They are described as foundational architectures in modern deep learning.
Motivation: Disentangled representations assign individual latent components to individual generative factors, and VAE-based architectures dominate current disentanglement benchmarks.A β-VAE traversal illustrates one latent coordinate isolating face azimuth.
Open problem: Disentanglement requires an axis-aligned latent manifold, but the idealized VAE objective is invariant to rotations and does not explicitly select such alignment.Rotated representations can preserve reconstruction while losing interpretable latent meanings.
Open problem: A mechanistic explanation for the VAE’s unexpected disentanglement ability remained missing before this work.The paper frames this gap as evidence that VAE inner workings were not sufficiently understood.
Contribution: The paper analyzes an internal VAE mechanism that promotes local orthogonality, explains its relation to disentanglement, and reveals a strong resemblance to PCA.The analysis covers nonlinear cases and is supported by theoretical and experimental results.
Contribution: The theoretical analysis uses the implemented loss and a practically prevalent regime whose simplification enables formalization.The same results are reported to explain why β-VAEs disentangle.

2 Background

The background introduces VAE objectives, disentanglement, and the rotation sensitivity of latent representations, then uses PCA and SVD to formalize axis-preserving embeddings. The key geometric condition is orthogonality of decoder columns, which preserves principal directions and supports axis alignment.

VAE basics: A VAE uses probabilistic encoder and decoder mappings between data space and a latent space, with a fixed prior and induced joint distributions.The encoder and decoder are denoted Encϕ and Decθ, respectively.
VAE basics: The ELBO combines expected reconstruction quality with KL divergence from the latent posterior to the prior, while β-VAE weights the KL term.The reconstruction term is implemented with square loss or crossentropy loss.
VAE basics: The VAE encoder uses a diagonal covariance matrix, a restriction that is highly significant for the paper’s main result.The prior is set to N(0, I).
Disentanglement: Disentanglement means that individual latent variables respond to individual generating factors while remaining relatively insensitive to other changes.Examples include face azimuth, skin brightness, and hair length.
PCA and latent representations: PCA isolates sources of variance, whereas ICA recovers a factorized representation; the paper examines how PCA chooses latent alignment.This contrast motivates separating variance-based alignment from factorization.
Disentanglement: Rotating a latent embedding can preserve reconstruction while destroying the interpretable meaning of individual latent variables.The background also states that different rotations of the same latent space are equally suitable for reconstruction.
PCA and latent representations: Linear autoencoder reconstruction is minimized by the PCA decomposition, although invertible rotations and scalings create other minimizers with different latent representations.PCA orders eigenvector directions by the magnitudes of their covariance eigenvalues.
SVD and axes preservation: For non-degenerate linear mappings, orthogonal decoder columns are equivalent to preserving coordinate axes in the SVD representation.Once the encoder preserves principal directions, accurate reconstruction makes the embedding axis-aligned when the decoder is axes-preserving.

3 Results

The paper formalizes how diagonal encoder posteriors and stochastic reconstruction promote local decoder orthogonality, linking VAE training to PCA-like latent organization. Its main theorem characterizes the resulting optima, while experiments support the predicted orthogonality across architectures and datasets.

Rotation and disentanglement: Rotationally symmetric idealized VAE objectives cannot claim to be designed to produce disentangled representations.The log-likelihood objective and its ELBO approximation are invariant to latent rotations when the prior is rotationally symmetric.
Rotation and disentanglement: Diagonalizing the encoder posterior disrupts rotational symmetry and yields beneficial effects regarding disentanglement, although this was likely introduced for tractability or computational convenience.This is the paper’s “by accident” explanation for why the diagonal approximation can help alignment.
Loss reformulation: The analysis starts from the implemented loss, isolates the stochastic reconstruction term, and assumes the practically prevalent polarized regime.In that regime, passive latent variables are filled with noise and ignored by the decoder, simplifying the KL loss.
The main result: Theorem 2 shows that every local minimum is global and that the columns of every decoder Jacobian Ji are orthogonal at global minima.The theorem analyzes an optimization over local decoder and encoder quantities under global-loss constraints.
The main result: The same local-minimum and orthogonality structure extends to the pure stochastic loss without the logarithm, while the β term does not affect the result provided polarization is ensured.The argument’s logarithmic formulation is indirect, but global optimality supports the corresponding local orthogonality claim for the non-logarithmic loss.

4 Proof outline

The proof outline shows that optimizing encoder precision and rotating the latent space reduces the objective when decoder columns become orthogonal, linking the resulting selection to PCA directions.

Proof setup: Fixing the mean encoder and using a linear decoder decomposes the analysis into per-example subproblems, followed by separate optimization over σ and V.The decoder is written as M = UΣV^T, removing the need for local linearization in the simplified setting.
Optimizing precision: The precision terms −log(σ_j) distribute a total precision budget across latent coordinates to minimize reconstruction deviation from uncorrupted values.The example predicts higher precision for the latent variable with greater reconstruction influence.
Optimizing orientation: After optimizing σ, the remaining objective depends on the product of decoder column norms, whose minimum under rotations is attained when the columns are orthogonal.Orthogonal rotations preserve volume, while the product-of-norms bound is tight precisely for orthogonal columns.
Rotated example: A 45° latent rotation can more than double the minimum objective because it removes unequal reconstruction impacts that otherwise permit more efficient precision allocation.The rotated example reports approximately 52.4e−C for the objective and describes the minimum as more than twice as high.
Connection to PCA: For higher-dimensional latent spaces, greedily selecting directions with maximal successive impact resembles the greedy algorithm for PCA.The proof connects the decoder-orientation optimization to choosing directions that maximize the stretch of latent-variable impacts.

5 Experiments

Experiments across architectures and datasets test polarization, local orthogonality, and disentanglement. The results support stronger orthogonal effects for diagonal-covariance VAE architectures and a correlation between alignment and disentanglement.

Setup: Experiments compare VAE, β-VAE, plain autoencoder, and β-VAEΣ across dSprites, MNIST, FashionMNIST, and synthetic linear and nonlinear embeddings.The synthetic tasks embed a unit square into higher-dimensional spaces, with disentanglement evaluated relative to its axes.
Metrics: The Disentanglement score ranges from 0 to 1, with higher values indicating stronger disentanglement, while DtO measures decoder non-orthogonality relative to a signed permutation.DtO is computed from decoder Jacobian SVDs and the closest signed permutation matrix.
Polarized regime: The polarized regime dominates training after a short initial phase across the reported examples, as assessed by the fraction of training time with relative KL error below 3%.Table 2 reports this fraction for β-VAE models with low and high latent dimensions.
Orthogonality: VAE-based architectures show local orthogonality, whereas β-VAEΣ is rotationally symmetric and plain autoencoders do not favor particular latent orientations.β-VAEΣ removes the diagonal approximation and therefore prioritizes no specific alignment.
Orthogonality and disentanglement: Independent β-VAE restarts on dSprites show a clear correlation between lower DtO and higher disentanglement scores across runs with 10, 30, or 50 epochs.The paper uses β = 4 for this experiment.

6 Discussion

The paper identifies VAE local orthogonalization as effectively performing local PCA and explains why orthogonality helps disentangled representations. It also emphasizes that this behavior is indirect and does not directly encourage factorized latent codes.

Discussion: The VAE mechanism leads to local orthogonalization and, in effect, local PCA.The paper supports this conclusion through intuitive, formal, and experimental analysis.
Discussion: Orthogonality is desirable because it helps enforce disentangled representations.The paper explains this connection as part of its interpretation of VAE behavior.
Discussion: VAEs use differences in variance to form latent representations, collapsing to PCA in the linear case.This behavior does not directly encourage factorized latent representations.
Discussion: Explicit architectural control over orthogonality could be beneficial because VAEs promote it only indirectly.The paper characterizes this indirect promotion as unsatisfying.

Supplementary Material

The supplementary material documents omitted dataset–architecture combinations and explains why some reported values are unavailable or meaningless.

Supplementary Material: Some dataset–architecture combinations in Table 1 are omitted for stated reasons.The omissions concern both disentanglement-score availability and overpruning behavior.
Supplementary Material: Disentanglement scores are not calculated for MNIST and FashionMNIST because their generating factors are unavailable.The single categorical label cannot serve as a replacement for the generating factors.
Supplementary Material: β-VAE numbers are not reported for MNIST and FashionMNIST because β values are chosen according to the unavailable score.For either synthetic task, regular VAE overprunes and produces meaningless values.

A.1 Proof of Theorem 2

The proof analyzes the VAE objective through bounds and perturbations, showing that global optima have locally orthogonal decoder Jacobians and that non-optimal configurations lack local minima.

A.1 Proof of Theorem 2: The proof derives a lower bound on objective (18) independent of optimization variables σ2.Equality is associated with pairwise orthogonality of the relevant column vectors.
A.1 Proof of Theorem 2: Global optima can be attained by choosing σ2 and orthogonal transformations so every decoder Jacobian Ji has orthogonal columns.The resulting construction yields the described global minima.
A.1 Proof of Theorem 2: Any configuration that does not minimize the objective admits a small perturbation that improves it, so local minima do not exist there.The argument tightens the inequalities through perturbations while keeping the relevant right-hand sides fixed.
A.1 Proof of Theorem 2: The encoder’s diagonal approximation and stochastic sampling are used in the objective analysis underlying the orthogonality result.The paper also contrasts this mechanism with rotationally symmetric formulations and defines disentanglement scores using prediction performance.
A.1 Proof of Theorem 2: Hadamard’s inequality supplies the key orthogonality condition: equality occurs exactly when the columns are pairwise orthogonal.A two-dimensional rotation provides the base case for the locally improving inequality argument.
A.1 Proof of Theorem 2: The proof uses invariance under left orthogonal transformations and preserves column norms and singular values during the matrix reduction.This reduction connects column orthogonality of the simplified matrix with that of the original matrix.

B.4 Network Details and Training

The network-details section specifies how latent dimensionality and evaluation data are selected for the reported experiments.

B.4 Network Details and Training: Latent dimensionality is set to the number of independent generating factors when applicable, or large enough to ensure decent reconstruction loss.This rule is applied across the evaluated architectures.
B.4 Network Details and Training: Reported numbers are computed on a previously unseen test dataset after random 80%/10%/10% training, evaluation, and test splits.The evaluation set is used during development, while final reports use the test set.

B.5 Synthetic Datasets

The synthetic datasets use linear and nonlinear transformations of two-dimensional inputs, and β controls the reconstruction–KL trade-off underlying disentanglement selection.

Dataset construction: The linear synthetic dataset maps a unit square from R2 to R3 through stretching, embedding, and a 45° rotation.The stretching factor is 2, followed by trivial embedding and rotation along the line containing (1, −1, 1).
β selection: β trades reconstruction error against KL loss to achieve the desired disentanglement level, with Disentanglement Score and DtO evaluated across datasets.Figure 5 reports these measures for dSprites and synthetic datasets, with dashed lines marking experimental choices.
Dataset construction: The nonlinear dataset maps R2 to R6 using a randomly initialized one-hidden-layer MLP with width 10, biases, and tanh nonlinearities.
Dataset construction: Both synthetic datasets contain 50000 samples.
β selection: Small β values make KL-driven orthogonalization negligible, whereas excessive β causes overpruning and fewer active latent coordinates than generating factors.

C.2 Degenerate case

When generating factors have equal scale, β-VAE representations can rotate arbitrarily because orthogonality alone does not determine alignment. Small rescaling removes this degeneracy, paralleling PCA behavior.

Degenerate case: Distinct singular values are required for local decoder orthogonality to preserve the generating axes.
Degenerate case: Equal scaling makes rotated linear encoders orthogonal without respecting the original square’s alignment.
Degenerate case: Four random β-VAE restarts produce random latent alignments, accompanied by high Disentanglement Score and DtO variances.
Degenerate case: PCA has the same degeneracy because every projection of a unit square onto a line has equal variance, leaving alignment implementation dependent.
Connection to PCA: β-VAE, like PCA, selects sources of variance rather than statistical independence.
Connection to PCA: Even small factor rescaling removes degeneracy by making singular values unequal, allowing β-VAE and PCA to find the correct alignment.β-VAE’s normalized latent representations contribute to unequal singular values after rescaling.
Nonlinear eigenfaces: On CelebA, nonlinear eigenfaces capture semantic features beyond the mostly photometric properties of classical eigenfaces.The first five latent coordinates are traversed and ordered by mean σj, emphasizing components with high reconstruction impact.