Source-linked AI summary
Learning the Irreducible Representations of Commutative Lie Groups
Taco Cohen, Max Welling
TL;DR
The paper addresses the lack of mathematically precise notions of invariant and disentangled representations. It introduces TSA, a probabilistic model of compact commutative Lie groups using Weyl’s principle. The analysis identifies invariant quantities and relates the learned representation to the discrete Fourier transform, while noting limits on stable additional invariants.
Problem
Invariance and disentangling are desired properties of learned representations, but their meanings have often lacked mathematical precision.
Source: §1 · p. 1Method
TSA learns a probabilistic representation of a compact commutative group from image pairs related by unobserved transformations, using Weyl’s principle to identify elementary components.
Source: §1 · pp. 1–2, §2.2 · p. 4Results
The posterior over transformations contains invariant information, and the change of basis in the analyzed case is a discrete Fourier transform.
Source: §3.1 · p. 6, §3.2 · p. 6Takeaways & Limitations
The analysis connects probabilistic Lie-group representation learning with pooling operations and gives a probabilistic interpretation of the DFT and its generalizations.
Source: §5 · p. 8The disentangling depends on the minimalist assumption that images are only equivalent as rotated copies or inequivalent; additional angle-based invariants are unstable for low-energy subspaces.
Source: §3.3 · p. 7, §5 · p. 8
Abstract
from arXiv · showhide
We present a new probabilistic model of compact commutative Lie groups that produces invariant-equivariant and disentangled representations of data. To define the notion of disentangling, we borrow a fundamental principle from physics that is used to derive the elementary particles of a system from its symmetries. Our model employs a newfound Bayesian conjugacy relation that enables fully tractable probabilistic inference over compact commutative Lie groups -- a class that includes the groups that describe the rotation and cyclic translation of images. We train the model on pairs of transformed image patches, and show that the learned invariant representation is highly effective for classification.
1. Introduction
The paper seeks a mathematically precise account of invariance and disentangling, using Weyl’s principle to relate elementary components of a system to its symmetries. It introduces Toroidal Subgroup Analysis (TSA), a tractable probabilistic model for compact commutative groups, and distinguishes it from prior approaches that do not represent full transformation uncertainty.
1. Introduction
- The paper frames representation learning as seeking multiple layers of representation while noting that what characterizes a good representation remains unclear.It situates this question alongside work on RBMs and autoencoders.Source: §1 · p. 1
- The paper focuses on invariance and disentangling because these representation desiderata have not generally been defined mathematically precisely.It leaves the search for meaning for future work.Source: §1 · p. 1
- Weyl’s principle treats irreducible representations of a data symmetry group as the system’s elementary components, adapting a physics principle to vision.The paper uses object identity under image rotation as an example of an invariant property.Source: §1 · p. 1
- TSA models compact commutative groups from image pairs related by arbitrary, unobserved transformations, with probabilistic quantities and their gradients available in closed form.The model uses a novel conjugate prior.Source: §1 · p. 2
1.1. Related work
- Prior methods do not account for full transformation-parameter uncertainty, whereas TSA represents multimodal posteriors and performs posterior inference with a fast feed-forward procedure.The cited comparison contrasts this with the more expensive iterative MAP optimization of Sohl-Dickstein et al. (2010).Source: §1.1 · p. 2
2. Preliminaries
The preliminaries define symmetry and reducibility as tools for identifying elementary representation components. They then restrict the analysis to compact commutative orthogonal groups, whose structure supports learning through block rotations and linear subspaces.
2.1. Equivalence, Invariance and Reducibility
- A representation induces an equivalence relation by mapping inputs to the same output, and its symmetry group contains the invertible transformations that leave that representation invariant.Under this group, two inputs are equivalent when some group transformation maps one to the other.Source: §2.1 · p. 2
- Changing basis can expose invariant subspaces and separate components that remain distinct under group transformations, instead of coordinates tangled by those transformations.The paper calls this process reduction of a group representation.Source: §2.1 · p. 2
- A representation is irreducible when it has no non-trivial invariant subspace remaining; Weyl’s principle treats such representations as a system’s elementary components.The principle applies wherever equivalence is well-defined, though the paper demonstrates it for compact commutative subgroups of SO(D).Source: §2.1 · p. 3
2.2. Maximal Tori in the Orthogonal Group
- The analysis restricts to compact commutative subgroups of SO(D), called toroidal subgroups, and represents their elements by jointly block-diagonalized rotations in a shared orthogonal basis.Orthogonality preserves Euclidean norm and gives Q^-1 = Q^T; these transformations can represent image rotation and cyclic translation, but not contrast scaling.Source: §2.2 · p. 3
- A toroidal group with I parameters can be learned by learning a maximal toroidal group and then an I-dimensional linear subspace of its angle parameters.Commutativity makes any such Lie-algebra subspace a subalgebra.Source: §2.2 · p. 4
- Compact one-parameter subgroups require commensurate frequencies; restricting frequencies to integers makes the parameter periodic over [0, 2π].Each 2D rotation block is an irreducible representation, and blocks with different weights transform differently.Source: §2.2 · p. 4
3. Toroidal Subgroup Analysis
TSA models transformations in compact commutative groups with tractable Bayesian posteriors, then derives invariant representations and metrics from those posteriors. The section connects this framework to the DFT, lower-dimensional subgroups, and maximum-marginal-likelihood learning.
3. Toroidal Subgroup Analysis
- TSA’s von-Mises prior is conjugate to the likelihood, yielding a product-form posterior over transformation angles; the update combines prior and data information.The data contribution grows with the projected vector magnitudes and declines with the square of the noise level.Source: §3 · p. 5
3.1. Invariant Representation and Metric
- For a maximal torus, the posterior precision parameters form an invariant representation, and the exact manifold distance is readily computable.The posterior precision corresponds to square pooling over paired filters; the proposed Hellinger distance on this representation equals the exact manifold distance up to a factor.Source: §3.1 · pp. 5–6
3.2. Relation to the Discrete Fourier Transform
- With sinusoidal filters, TSA’s change of basis is the DFT, and posterior precision and mean correspond to Fourier magnitude and phase.This gives a probabilistic interpretation of DFT coefficients and permits learning a generalized transform from data.Source: §3.2 · p. 6
3.3. Modeling a Lie subalgebra
- For a one-parameter subgroup, a generalized von-Mises prior yields a tractable posterior and pools same-weight subspaces to form invariant summaries.The paper notes that angle-based candidate invariants are unstable in low-energy subspaces, leaving stable maximal invariants as future work.Source: §3.3 · pp. 6–7
3.4. Maximum Marginal Likelihood Learning
- The model is trained by gradient descent on marginal likelihood, whose required integrations are available in closed form for coupled and decoupled models.For the uncoupled model, minibatch SGD is used and W is orthogonalized after each update by setting its singular values to 1.Source: §3.4 · pp. 7–8
4. Experiments
The experiments learn frequency-ordered filters and evaluate invariant classification on rotated MNIST. The TSA-derived and manifold metrics outperform tangent distance, with MD near non-rotated ED accuracy.
4. Experiments
- TSA was trained with 100 filters on 250,000 pairs of 16 × 16 patches, with each y patch produced by uniformly random rotation of x.The rotation angle was drawn uniformly from [0, 2π].Source: §4 · p. 8
- The learned filters are clean and mostly correctly weighted, though a few row-1/2 filters were assigned zero despite having higher frequencies.Figure 3 sorts the filters by absolute frequency |ω_j|; learned ω_j values range from −11 to 12.Source: §4 · p. 8, Figure 3 · p. 8
- On rotated MNIST, √κ and MD outperform tangent distance by a large margin, and MD is about as accurate as ED on the simpler non-rotated dataset.Figure 4 reports the classification experiment; the authors say this result indicates that MD has almost completely modded out rotation.Source: Figure 4 · p. 8, §4 · p. 8
5. Conclusions and outlook
The paper applies a principle for learning disentangled representations to a simple symmetry-group setting. It presents a tractable model with potential applications to invariant classification and Bayesian motion estimation, while noting its assumption about image equivalence.
5. Conclusions and outlook
- The paper presents a principle for learning disentangled representations and works out its consequences for a simple type of symmetry group.Source: §5 · p. 8
- The resulting tractable model has potential applications to invariant classification and Bayesian estimation of motion.Source: §5 · p. 8
- The model derives convolutional-network pooling operations from probabilistic and Lie-group principles and gives the DFT and its generalizations a probabilistic interpretation.Source: §5 · p. 8
- The disentangling result assumes images are described only as equivalent rotated copies or inequivalent.Source: §5 · p. 8