Source-linked AI summary
Transformation Properties of Learned Visual Representations
Taco S. Cohen, Max Welling
TL;DR
Invariance alone does not capture the transformation structure needed for general-purpose visual representations, and perspective projection can make scene motion non-linear in image space. The paper analyzes linear group representations and uses a latent SO(3) model for rotating objects. It establishes a conditional link between irreducibility and decorrelation and demonstrates generation of reasonable views at unseen angles.
Problem
Method
Results
Under uniform orbit sampling, irreducible representations have diagonal covariance, and the model generates reasonable images at unseen angles while retaining object identity.
Source: §3.2 · p. 4, §6 · pp. 7–8Takeaways & Limitations
Under the stated sampling condition, decorrelation can serve as a surrogate objective for the deeper structural objective of irreducibility.
Source: §3.2 · p. 4Orbit sampling is not uniform in real datasets, so the stated decorrelation result's sampling assumption may not hold exactly.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
When a three-dimensional object moves relative to an observer, a change occurs on the observer's image plane and in the visual representation computed by a learned model. Starting with the idea that a good visual representation is one that transforms linearly under scene motions, we show, using the theory of group representations, that any such representation is equivalent to a combination of the elementary irreducible representations. We derive a striking relationship between irreducibility and the statistical dependency structure of the representation, by showing that under restricted conditions, irreducible representations are decorrelated. Under partial observability, as induced by the perspective projection of a scene onto the image plane, the motion group does not have a linear action on the space of images, so that it becomes necessary to perform inference over a latent representation that does transform linearly. This idea is demonstrated in a model of rotating NORB objects that employs a latent representation of the non-commutative 3D rotation group SO(3).
1 INTRODUCTION
For general-purpose vision, invariance alone is insufficient: representations should transform predictably under scene motion. The paper uses group representation theory to study these transformations and introduces a latent SO(3) model for partially observed scenes.
1 INTRODUCTION
- A general-purpose visual representation must capture transformation properties, since invariance alone is insufficient for motion understanding and geometrical reasoning.Source: §1 · p. 1
- The paper studies distributed representations with group representation theory and relates their transformation properties to statistical dependency structure under complete observability.Source: §1 · p. 1
- Under partial observability, the model uses a shared latent vector for an object's views and an SO(3) transformation for each pose.Source: §1 · p. 1
- Unlike prior approaches that assume a linear group action on inputs, the model acts linearly on a latent scene representation and handles non-commutative transformations.Source: §1 · p. 2
2 SYMMETRIES AND REPRESENTATIONS
The section formalizes scene transformations as linear group representations, assuming the complete scene is observable. It motivates irreducible representations as basic components with useful computational and statistical properties.
2 SYMMETRIES AND REPRESENTATIONS
- The analysis assumes the complete scene is known and represents it as a vector in a Hilbert space of functions.Source: §2 · p. 2
- A scene representation changes with both scene content and the chosen reference frame, while SE(3) describes rigid motions that preserve Euclidean geometry.Source: §2 · p. 2
- A group representation maps group elements to invertible linear transformations that preserve the group's composition structure.Source: §2 · p. 2
- Every unitary representation can be transformed into a block-diagonal form whose blocks are irreducible representations.Source: §2 · pp. 2–3
- Irreducible representations offer computational efficiency, a precise account of disentanglement, and a simpler route to generators of polynomial invariants.Source: §2 · p. 3
3 IRREDUCIBILITY, INDEPENDENCE AND DECORRELATION
The section connects transformation-centered representations to statistical structure: under specified conditions, irreducibility corresponds to decorrelation or conditional independence. It develops this link from a circle-rotation example to compact-group and time-series results.
3 IRREDUCIBILITY, INDEPENDENCE AND DECORRELATION
- Irreducible representations, rather than independent factors alone, can organize visual representations; under certain conditions they are decorrelated or conditionally independent.The section contrasts this transformation-centered view with PCA and ICA, which seek independent latent variables.Source: §3 · p. 3
3.1 IRREDUCIBILITY AND DECORRELATION: AN ELEMENTARY EXAMPLE
- For uniformly sampled rotations of a circle template, the Fourier transform gives one-dimensional irreducible components and diagonalizes the data covariance.In the spectral domain, the rotation representation is diagonal.Source: §3.1 · p. 3
3.2 IRREDUCIBILITY AND DECORRELATION: GENERAL CASE
- For compact groups, a fully reduced unitary representation of a fixed template sampled uniformly over the group has diagonal covariance.Real orbits are not uniformly sampled; irreducible and decorrelated representations are expected to align only insofar as data density is group-invariant.Source: §3.2 · p. 4
3.3 IRREDUCIBILITY AND CONDITIONAL INDEPENDENCE
- In the time-series model, unitary-transform invariance yields a conjugate posterior, while the conditional marginal factorizes by irreducible representations, giving conditional independence.The model constructs an exponential family using matrix elements of irreducible unitary representations as sufficient statistics.Source: §3.3 · p. 4
4 PARTIAL OBSERVABILITY
Perspective projection exposes only an image, and 3D motion can bring new structures into view, preventing a linear group action on images. The paper proposes inferring latent scenes whose representations transform linearly instead.
4 PARTIAL OBSERVABILITY
- 3D motion can bring entirely new structures into a projected image, so no image-space action can make perspective projection equivariant to scene motion.The image is only a projection of the complete scene.Source: §4 · p. 4
- Planar-scene assumptions yield projective-group representations, but the paper notes that real scenes are not flat.Diffeomorphisms offer another model for variability outside a low-dimensional Lie-group action, but projected motions are not invertible and need not be Lipschitz-continuous to scene motions.Source: §4 · pp. 4–5
- The proposed alternative is to learn a scene prior and image-generative model, then infer scenes from images while requiring latent scenes to transform as a symmetry-group representation.Requiring irreducible latent transformations may also yield a simple dependency structure.Source: §4 · p. 5
5 A LATENT GROUP REPRESENTATION
The model represents multiple views of one object with a shared latent vector transformed by per-view SO(3) elements before image generation. It uses block-diagonal irreducible representations, computes rotation matrices in a real-spherical-harmonic basis, and learns by stochastic hard EM.
5 A LATENT GROUP REPRESENTATION
- A shared latent vector and one SO(3) element per view generate each view by transforming the latent scene and decoding it with a neural network.The image likelihood is normal, centered on the decoder output.Source: §5 · p. 5
5.1 REPRESENTATION THEORY OF SO(3)
- SO(3) representations are taken block-diagonal, with each block an irreducible representation; spherical-harmonic expansions provide the irreducible representations of SO(3).Each harmonic degree l spans a representation, and multiple copies can represent functions on 3D objects.Source: §5.1 · p. 6
5.2 COMPUTATION OF THE REPRESENTATION MATRICES
- Wigner D-functions are the irreducible-representation matrix elements, and the paper uses a fast computation method in the real-spherical-harmonic basis for ZYZ-Euler rotations.Figure 2b shows the matrix at weight l = 35 for three rotation values.Source: §5.2 · pp. 6–7, Figure 2 · p. 7
5.3 LEARNING
- Stochastic hard EM alternates partial per-instance optimization of latent variables and poses with parameter updates while holding those latent variables fixed.Each step uses one gradient-ascent step on the corresponding log joint probability.Source: §5.3 · p. 7
6 EXPERIMENTS
The model generates reasonable views at unseen azimuths, including finer angles than its 20-degree training spacing and extrapolated poses of known objects. Experiments use NORB images and a specified neural-network and representation setup.
6 EXPERIMENTS
- NORB training uses five instances from each of five object categories, across camera elevations, azimuths, and lighting conditions.The resulting training set contains 24,300 images.Source: §6 · p. 7
- The inputs are whitened to retain 95% of variance, and a 550-unit hidden layer is used with a specified SO(3) representation and regularization parameters.The regularization parameters are β = 0.1 and α = 0.1.Source: §6 · p. 7
- The model generates reasonable images at azimuths smaller than its 20-degree training spacing.Figure 3 shows interpolation over 40 degrees for various objects.Source: §6 · p. 7, Figure 3 · p. 8
- The model extrapolates to unseen angles of known objects while retaining object identity.It is trained only on azimuths more than 40 degrees from the reference, then produces mean figures at 0, 20, 40, and 60 degrees.Source: §6 · pp. 7–8, Figure 4 · p. 8
7 RELATED WORK
The paper relates its latent group representation to transforming auto-encoders and argues that a shared latent space can support generalization and represent ambiguous poses. The current model remains a proof of concept because its network and inference algorithm are unsuitable for large images.
7 RELATED WORK
- Unlike transforming auto-encoders, this model represents scene content in a shared latent space rather than as probabilities of preferred visual entities.Both approaches explicitly represent pose variables that act linearly on other pose variables.Source: §7 · p. 8
- A shared latent space may aid generalization, enable metric relations between visual entities, and represent symmetric objects without requiring an unambiguous pose estimate.For an edge-like structure, the model represents a vector with a lower-dimensional orbit rather than estimating motion anyway.Source: §7 · p. 8
- The fully connected generative network and hard-EM algorithm are unsuitable for large images, so the authors describe the current model as a proof of concept.They suggest a group-invariant convolutional network as a possible more scalable direction.Source: §7 · p. 8
8 CONCLUSION
The paper argues that modeling visual-world dynamics and generalizing from few examples require attention to transformation properties. It characterizes linear group representations through irreducible components and connects irreducibility to decorrelation under a stated sampling condition.
8 CONCLUSION
- Modeling visual-world dynamics and generalizing from few examples are identified as challenges that require examining transformation properties of learned representations.The paper contrasts this direction with treating images as independent draws from a distribution, as in current benchmark datasets.Source: §8 · p. 9
- Linear representations of symmetry groups are direct sums of irreducible representations, with the theory specialized to the 3D rotation group.The paper studies such representations in observed or latent spaces.Source: §8 · p. 9
- Under uniform sampling of orbits, learning a linear, unitary, irreducible representation leads to decorrelated representations.Source: §8 · p. 9
9 APPENDIX
The appendix excerpt gives a proof step based on uniform sampling over the group and orthogonality of irreducible-representation matrix elements, and identifies the representation dimension.
9 APPENDIX
- The proof considers a transformed vector with the group element distributed uniformly.Source: §9.1 · p. 11
- The derivation invokes orthogonality of matrix elements of irreducible representations.Source: §9.1 · p. 11
- The dimension notation refers to the dimension of the representation indexed by l.Source: §9.1 · p. 11