Source-linked AI summary
Towards a Definition of Disentangled Representations
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, Alexander Lerchner
TL;DR
Disentangled representation learning lacks a generally agreed-upon definition, while many machine-learning algorithms remain data-inefficient and less robust or generalisable than biological intelligence. This paper defines disentangling through symmetry transformations and group representation theory, showing that the definition aligns with prior intuitions and resolves several disagreements.
Problem
Disentangling lacks a clear definition beyond datasets with known ground-truth factors, while many machine-learning algorithms have poor data efficiency, robustness, and generalisability.
Method
The paper characterises disentanglement through symmetry transformations and defines independent subspaces using group actions, group decompositions, and representation theory.
Results
The definition agrees with prior views while resolving disagreements about whether disentangled subspaces must be single-dimensional, have unique bases, or use linear group actions.
Takeaways & Limitations
Invariant subspaces function as separate representations without mixing under world dynamics, making extension to continual-learning scenarios feasible.
Takeaways & Limitations
The framework assumes that world dynamics can be described by symmetry transformations and defines disentanglement relative to a particular decomposition of the symmetry group into subgroups.
Abstract
from arXiv · showhide
How can intelligent agents solve a diverse set of tasks in a data-efficient manner? The disentangled representation learning approach posits that such an agent would benefit from separating out (disentangling) the underlying structure of the world into disjoint parts of its representation. However, there is no generally agreed-upon definition of disentangling, not least because it is unclear how to formalise the notion of world structure beyond toy datasets with a known ground truth generative process. Here we propose that a principled solution to characterising disentangled representations can be found by focusing on the transformation properties of the world. In particular, we suggest that those transformations that change only some properties of the underlying world state, while leaving all other properties invariant, are what gives exploitable structure to any kind of data. Similar ideas have already been successfully applied in physics, where the study of symmetry transformations has revolutionised the understanding of the world structure. By connecting symmetry transformations to vector representations using the formalism of group and representation theory we arrive at the first formal definition of disentangled representations. Our new definition is in agreement with many of the current intuitions about disentangling, while also providing principled resolutions to a number of previous points of contention. While this work focuses on formally defining disentangling - as opposed to solving the learning problem - we believe that the shift in perspective to studying data transformations can stimulate the development of better representation learning algorithms.
1. Introduction
The paper addresses the lack of agreement over what disentangled representations are by connecting them to symmetry transformations and representation theory. It offers a formal definition while explicitly leaving general algorithmic learning solutions open.
- Motivation: Poor data efficiency, robustness, and generalisability motivate methods that better reflect underlying data structure.The paper frames inductive biases and learned representations as responses to these shortcomings.
- Motivation: Disentangled representation learning aims to align representation axes with the data’s generative factors.The field still disagrees about generative factors, dimensionality, and uniqueness of axis alignment.
- Approach: The paper defines disentangling by decomposing vector representations into subspaces transformed independently by distinct symmetry transformations.Group and representation theory provide constraints connecting symmetry groups to independent subspaces.
- Scope: The contribution is theoretical and does not provide a general algorithmic recipe for learning disentangled representations.Its stated purpose is to establish a formal framework and resolve conceptual points of contention.
- Implications: The framework addresses disputes about generative factors, disentangleable factors, latent dimensionality, and basis uniqueness.The authors argue that clarifying these issues may support more robust and scalable future approaches.
- Structure: The paper develops its perspective through physics, symmetry transformations, mathematical formalisation, and discussion of consequences for disentangling.The paper’s organisation moves from physical intuition to formal group-theoretic definitions and implications.
2. Our symmetrical world
The paper motivates its framework by treating transformations that preserve some properties while changing others as symmetries. It argues that studying such structure has supported explanation, generalisation, classification, and prediction across physics and machine perception.
- Core assumption: The framework assumes that world dynamics can be described by transformations changing some state aspects while leaving others invariant.This assumption is explicitly inspired by the role of symmetries across levels of abstraction in physics.
- Symmetry: Symmetries preserve selected properties of objects under transformations such as translation, rotation, and axis permutation.The examples include physical objects and mathematical subspaces.
- Physics: Noether’s Theorem connects conservation laws to continuous symmetries, including energy with time translation and momentum with space translation.The passage also links angular momentum to rotational symmetry.
- Physics: Symmetry-based organisation has supported physical prediction, including Gell-Mann’s prediction of the Ω− particle from flavour symmetries.The particle was observed two years after the 1962 prediction.
- Scientific role: Studying structure-preserving transformations has helped unify theories, classify objects, and predict new phenomena.Examples include electromagnetism, crystals, elementary particles, and gaps in the periodic table.
- Machine perception: In machine perception, identifying invariant properties under transformations is presented as a route to generalisation in new settings.Relevant transformations include translations, rotations, and object-colour changes in scene understanding.
3. A roadmap to defining disentangled representations
The paper defines disentangled representations through symmetry transformations that alter selected world-state properties while leaving others invariant. It formalizes this structure as independent representation subspaces associated with subgroup actions, while noting that useful decompositions may not always exist or be known.
- Concrete example: In the grid-world example, horizontal movement, vertical movement, and colour changes form separate symmetry actions affecting only their corresponding properties.These actions motivate a decomposition into horizontal, vertical, and colour subgroups.
- Definition: A disentangled representation decomposes into independent subspaces, each affected by one subgroup while all other subgroup actions leave it unaffected.The definition does not require a fixed dimensionality or basis for any subspace.
- Definition: A linear disentangled representation additionally requires each subgroup’s action on its corresponding subspace to be linear.The underlying subspaces and bases may otherwise remain flexible.
- Consequences and limitations: The definition is relative to a chosen subgroup decomposition, and different decompositions may split factors usefully, redundantly, or in undesirable mixtures.Abstract group structure alone may not identify decompositions that are useful for subsequent tasks.
- Consequences and limitations: The paper leaves discovery of a useful group decomposition and the empirical learning of disentangled representations for future work.Its contribution is theoretical rather than a general algorithmic solution.
4. Related work
Related work connects representation learning to invariance, equivariance, and group-theoretic descriptions of transformations. The paper distinguishes its definition from approaches that discard nuisance variables and reviews prior disentangling methods and evaluation practices.
- Invariance and perception: Psychology and artificial intelligence have long studied recovering independent factors of variation and invariants from perceptual inputs using transformation-based ideas.This history includes invariants related to illumination, viewpoint, overlapping samples, and local structural disturbances.
- Invariance and perception: Perception research often seeks invariance to pose or illumination, whereas equivariant approaches preserve information about transformations while allowing subsets of features to remain invariant.The paper places disentangled representation learning in the equivariant line of research.
- Group-theoretic perception: Group theory has been used to quantify natural symmetries, characterize visual recognition, and study transformations such as translation, rotation, illumination, viewpoint, and occlusion.Prior work includes hierarchical models based on group orbits and invariant signatures.
- Distinction from prior usage: The paper differentiates its use of disentangling from machine-perception work that removes nuisance variables while preserving task-relevant information.Here, disentangling concerns structuring representation subspaces according to subgroup actions rather than discarding transformations.
- Disentangled representation learning: Other group-theoretic machine-learning approaches address weight constraints, affine invariance, or abstraction discovery, but do not explicitly formalize disentangled representations.The paper positions its contribution as a formal connection between symmetry groups and representations.
- Disentangled representation learning: Earlier disentangling methods often required known generative factors or failed to scale beyond toy datasets, while evaluation commonly relied on human intuition or ground-truth-factor metrics.The lack of a clear definition motivates the present work.
5. A formal definition of disentangled representations
The paper defines disentangled representations by requiring symmetry actions to decompose across independent factors and to be preserved equivariantly in representation space. The framework clarifies how world symmetries, group decompositions, and invariant subspaces constrain representations, while allowing multiple decompositions and non-axis-aligned factors.
- 5.1. Disentangled group action: A disentangled group action decomposes G into subgroups and X into corresponding factors, with each subgroup acting only on its matching factor.For two factors, (g1, g2) · (v1, v2) = (g1 ·1 v1, g2 ·2 v2).
- 5.1. Disentangled group action: The framework permits linear or continuous subgroup actions when the ambient space has corresponding structure, and direct-sum factors need not align with a chosen basis.For vector spaces, the decomposition is written X = X1 ⊕ X2 even when the full group action is nonlinear.
- 5.1. Disentangled group action: The definition extends to n factors: each subspace Xi is affected only by Gi and remains invariant under every Gj for j ≠ i.The decomposition may be a Cartesian product or a direct sum.
- 5.2. Disentangled representation: A representation is built from world states through the composition f = h ◦ b, and its symmetry structure is required to correspond to an action on Z.The map f sends world states to representations, while b generates observations and h infers representations.
- 5.2. Disentangled representation: Equivariance requires the group action to commute with f, expressed as g · f(w) = f(g · w), so transformations in Z reflect transformations in W.For bijective f, the compatible action on Z can be defined directly from the action on W; for injective f, only the image f(W) matters.
- 5.2. Disentangled representation: The formal definition combines equivariance with a decomposition Z = Z1 × ... × Zn or Z = Z1 ⊕... ⊕Zn whose subspaces correspond to independent symmetry factors.The paper does not require the world action itself to be disentangled, although that assumption naturally expresses compositional world structure.
- 5.3. A worked example of a disentangled representation: In the gridworld example, G = Gx × Gy × Gc represents x translation, y translation, and colour transformations, with each subgroup isomorphic to CN.The CCI-VAE representation approximately preserves these independent actions and much of their group structure, but loses linearity and cyclicity.
- 5.3. A worked example of a disentangled representation: Disentanglement depends on the chosen group factorisation: positional factors may admit rotated invariant-subspace bases, while noncommuting subgroups cannot form a direct-product decomposition.The paper leaves learning a useful group decomposition beyond scope and notes that SO(3) rotations cannot be decomposed into the noncommuting subgroups considered.
6. A formal definition of linear disentangled representations
The paper formalizes linear disentangled representations by connecting symmetry-group decompositions to invariant subspaces of vector representations. It then illustrates the definition with a grid-world construction whose coordinates separately encode position and colour transformations.
- 6. A formal definition of linear disentangled representations: The framework adds linearity to disentangled group actions, requiring a representation that is equivariant between world-state and representation-space actions.The representation is defined relative to a direct-product decomposition of the symmetry group.
- 6.1. Disentangled group representation: A group representation is linearly disentangled when its vector space decomposes into subspaces carrying separate representations of the group factors.The direct-sum decomposition excludes irreducible factors in which multiple group components act non-trivially.
- 6.1. Disentangled group representation: For G = G1 × ... × Gn, disentanglement requires each irreducible tensor factor to involve at most one non-trivial subgroup representation.This generalizes the two-factor definition to arbitrary direct-product decompositions.
- 6.2. Linear disentangled representation: The linear agent representation is a map from world-states to representations that admits an equivariant linear action and a direct-sum decomposition by group factor.Factors of the form ρ1 ⊗ρ2 are excluded unless at most one component acts non-trivially.
- 6.3. A worked example of a linear disentangled representation: In the grid world, the symmetry group decomposes into independent x-translation, y-translation, and colour-transformation subgroups.World states combine two position coordinates with one colour coordinate, giving |W| = N3.
- 6.3. A worked example of a linear disentangled representation: The example maps grid-world states into Z = C3, identified with R6, using one complex coordinate for each position or colour factor.The construction supplies an explicit equivariant representation for the example.
- 6.3. A worked example of a linear disentangled representation: The constructed action is linear and equivariant, with invariant coordinate subspaces on which all but one subgroup acts trivially.Therefore the agent representation is disentangled under the proposed definition.
- 6.3. A worked example of a linear disentangled representation: Over R6, the same construction uses block-diagonal matrices of 2 × 2 rotation matrices and two-dimensional invariant subspaces.The argument for disentanglement remains unchanged under this real-coordinate view.
7. Backward compatibility of the new definition
The proposed definition aligns with established intuitions about modularity, compactness, and explicitness while resolving disagreements through the structure of symmetry-group decompositions. It permits multidimensional factors and makes linearity specific to the linear version of the definition.
- Dimensions of compatibility: The paper evaluates backward compatibility using modularity, compactness, and explicitness, dimensions on which prior models and metrics often disagree.The authors argue that the new definition resolves these disagreements in a principled manner.
- Modularity: Modularity is recovered by replacing data generative factors with disentangled actions of the symmetry group.A single latent dimension should encode no more than one such factor.
- Modularity: 3D rotations cannot be disentangled under this definition because their symmetry group does not decompose as a direct product.Thus intuitive axis-wise separation is not always compatible with the group structure.
- Compactness: The definition allows each disentangled subspace to be multidimensional, rejecting compactness as a necessary requirement.This matches approaches and metrics that permit one generative factor to occupy multiple latent dimensions.
- Explicitness: A representation can preserve information about all factors while varying in disentanglement across alternative group decompositions of the same symmetry group.In the grid world, separating colour from combined position differs from separately separating x, y, and colour.
- Explicitness: Linearity remains optional for disentanglement generally and is required only for the separately defined class of linear disentangled representations.This preserves compatibility with models that do not optimize linear decodability.
8. Conclusions
The paper replaces the ill-defined notion of data generative factors with symmetry transformations and uses group and representation theory to define disentangled representations. It reports compatibility with prior intuitions while allowing group-dependent subspace dimensionality and multiple suitable bases.
- 8. Conclusions: The paper’s central aim is to provide a formal definition of disentangled representations.It combines insights from modern physics with prior machine-learning work.
- 8. Conclusions: Symmetry transformations of world states replace the previously ill-defined notion of data generative factors.Group and representation theory then transfers this structure into representation vector spaces.
- 8. Conclusions: The definition fits prior views while resolving disagreements: disentangled subspaces may be single- or multi-dimensional, and suitable bases need not be unique.These properties depend on the structure of the symmetry group.
- 8. Conclusions: Invariant subspaces function as separate representations because world dynamics do not mix them.The framework therefore permits extending the set of representations as an agent encounters new aspects in continual learning.
- 8. Conclusions: The authors hope the definition will accelerate progress toward scalable algorithmic solutions and methods for measuring disentangled representation learning.This is presented as a prospective benefit rather than an algorithmic result of the paper.
A.1. Review of group theory
The appendix reviews groups as sets with binary operations satisfying identity and inverse properties, then introduces subgroups, direct products, group actions, and structure-preserving actions on spaces.
- Group basics: A group consists of a set with a binary operation satisfying standard group axioms, including identity and inverses.The binary operation need not be commutative; commutative groups are Abelian.
- Group basics: The operation may be written multiplicatively when clear, with the identity denoted by 1.This is a notational convention used throughout group theory.
- Examples: Examples include permutation groups, general linear groups of invertible linear transformations, and special orthogonal groups of distance-preserving transformations.SO(2) and SO(3) provide familiar examples involving rotations.
- Subgroups and direct products: A subgroup is a subset closed under multiplication and inverses, such as SO(n) within GL(n, R).Subgroups inherit the relevant group operation from the larger group.
- Subgroups and direct products: The direct product G × H forms ordered pairs with component-wise group operations and contains identifiable copies of G and H as subgroups.Its underlying set is the Cartesian product of the two groups.
- Group actions: A group action maps G × X to X while respecting the group structure, so group elements act as transformations of the space.The notation is often written by concatenating a group element and a point.
- Structure-preserving actions: When the acted-on space has additional structure, the action must preserve that structure, such as linearity for vector spaces or continuity for topological spaces.For vector spaces, linearity includes g(x + y) = gx + gy.
A.2. Review of group representation theory
Group representation theory studies abstract groups through structure-preserving linear actions on vector spaces. Representations can be decomposed into irreducible components, combined through direct sums and tensor products, and related to direct-product groups.
- A group representation maps group elements to invertible linear transformations while preserving the group structure.Relative to a basis, these transformations are represented by invertible matrices; the representation need not be injective.
- A representation can also be viewed as a linear group action on a vector space, with each group element acting through its associated transformation.The action sends (g, v) to ρ(g)(v).
- Direct sums combine representations on V and W, while tensor products construct a representation on V ⊗W with basis vectors formed by tensoring basis elements.If the basis dimensions are m and n, then V ⊗W has dimension m × n.
- Invariant subspaces define subrepresentations, and recursively decomposing them expresses reducible representations as direct sums of irreducible representations.With a suitable basis, the resulting matrices become block-diagonal.
- The decomposition into irreducible representations is unique up to isomorphism, ordering, and change of basis under the stated conditions.
- For a direct-product group G = G1 × G2, its irreducible representations are precisely tensor products of irreducible representations of G1 and G2.