Source-linked AI summary
Understanding image representations by measuring their equivariance and equivalence
Karel Lenc, Andrea Vedaldi
TL;DR
The paper addresses limited theoretical understanding of how image representations encode transformations and visual information. It empirically characterises equivariance, invariance, and equivalence using learned transformations and CNN transformation and stitching layers, finding predictable transformations across many representations and clarifying invariance with depth. It also demonstrates applications to structured-output regression.
Problem
Theoretical understanding remains limited regarding which invariances image representations contain, how they are obtained, and whether different CNN parametrisations capture the same information.
Method
The paper empirically studies equivariance, invariance, and equivalence using learned linear transformations, CNN transformation layers, and stitching layers between networks.
Results
The study finds predictable transformation behaviour in most representations, including HOG and many CNN layers, and uses it to quantify how invariance builds with CNN depth.
Takeaways & Limitations
The framework functionally characterises and quantifies representation properties systematically, including equivalence between different representations, and supports structured-output regression applications.
Takeaways & Limitations
The analysis focuses in practice on geometric transformations such as affine warps and flips, and most HOG implementations use nine orientation bins that break rotational symmetry.
Abstract
from arXiv · showhide
Despite the importance of image representations such as histograms of oriented gradients and deep Convolutional Neural Networks (CNN), our theoretical understanding of them remains limited. Aiming at filling this gap, we investigate three key mathematical properties of representations: equivariance, invariance, and equivalence. Equivariance studies how transformations of the input image are encoded by the representation, invariance being a special case where a transformation has no effect. Equivalence studies whether two representations, for example two different parametrisations of a CNN, capture the same visual information or not. A number of methods to establish these properties empirically are proposed, including introducing transformation and stitching layers in CNNs. These methods are then applied to popular representations to reveal insightful aspects of their structure, including clarifying at which layers in a CNN certain geometric invariances are achieved. While the focus of the paper is theoretical, direct applications to structured-output regression are demonstrated too.
1. Introduction
The paper develops a systematic empirical framework for characterising equivariance, invariance, and equivalence in image representations, addressing limited theoretical understanding of how representations encode visual information. It applies learned transformations and stitching layers to CNNs and related representations.
- Image representations are widely used, but their invariances and how those invariances arise remain theoretically unclear.
- The paper studies equivariance, invariance, and equivalence as mathematical properties of a representation mapping an image to a vector.Equivariance concerns predictable changes under input transformations; invariance is the special case where the output is unchanged.
- Most representations, including HOG and many deep-network layers, change predictably with image transformations, and these changes can be learned as linear output transformations.For CNNs, the paper introduces and learns transformation layers, then uses them to quantify how invariance accumulates with depth.
- The paper tests equivalence by learning stitching layers that swap parts of different CNNs and examining whether the resulting Franken-CNNs retain the original performance.
- Unlike approaches that design mechanisms intended to produce invariance, the paper systematically teases out and quantifies properties already present in representations, including equivalence between different representations.
2. Notable properties of representations
The paper formalizes equivariance, invariance, and equivalence as properties of image representations and develops empirical methods to measure them. It models equivariant transformations with learnable mappings, including structured sparse and convolutional transformations for CNN features.
- Equivariance and invariance: Equivariance transfers an input transformation g to representation space through a mapping M_g, while invariance is the identity-map special case.The analysis focuses on geometric transformations such as affine warps and flips, and uses a common mapping across images to capture intrinsic representation structure.
- Equivalence: Equivalence asks whether heterogeneous representations capture the same information through a mapping E_φ→φ′, including between differently trained CNNs.The paper proposes stitching layers to swap parts of different networks and assess whether their representations are functionally interchangeable.
- Examples: HOG feature transformations for flips and 180° rotations are exact permutations, whereas 90° rotations are only approximately represented because orientation components must be remapped.The permutation swaps spatial cells and symmetric gradient-orientation components within each cell.
- Learning mappings: Equivariant mappings are learned by regularised empirical risk minimisation, using affine predictors φ(gx) ≈ A_gφ(x)+b_g and analogous regression for equivalence.The objective uses a regression loss and regulariser; the equivalence version replaces transformed features φ(gx) with another representation φ′(x).
- Learning mappings: Sparsity-inducing regularisers are preferred to standard l2 regularisation because the dense mapping A_g has Ω(d^2) parameters and sparse structure better resembles a permutation.Structured sparsity limits each output feature to a small neighbourhood determined by the back-projected receptive-field location, while loss choice matters for deep CNN layers.
- CNN transformation layers: Affine transformations of convolutional representations are approximately local and translation invariant, enabling efficient CNN transformation layers with a permutation followed by learned m×m×D filters.These filters reduce parameters and can permute or interpolate feature channels; sampling artifacts are excluded from the stated locality and translation-invariance claim.
3. Experiments
Experiments learn and test equivariant mappings for HOG and CNN representations, examine invariance and equivalence across CNN layers, and apply learned mappings to structured-output regression.
- 3.1. Equivariance in shallow representations: HOG experiments show that sparse regression generalizes better than unconstrained or Frobenius-regularized regression for learning transformation mappings.Least-squares regression overfits, while forward selection outperforms ridge regression because the solution is highly sparse.
- 3.1. Equivariance in shallow representations: FS with k = 5 achieves zero prediction error for exact 180° rotations and maintains small errors across the transformation range.Least squares and ridge regression fail to recover the exact 180° rotation, while errors generally increase with distance from the identity transformation.
- 3.2. Equivariance in deep representations: For CNNs, learned equivariant mappings recover most classifier performance, but the best regression objective depends on layer depth.FS is better through Conv2, whereas task-oriented loss performs better from Conv3 onward; feature reconstruction error does not always predict classification performance.
- 3.2. Equivariance in deep representations: CNN testing finds that horizontal flips and scaling are largely represented as invariances, while vertical flips and rotations benefit substantially from learned mappings, especially in early layers.For expected transformations, learning a mapping is no better than leaving features unchanged; unexpected transformations remain more transformable in shallower layers.
- 3.3. Equivalence of deep representations: Equivalence experiments find CNN representations compatible through Conv4, with Conv1 and Conv2 interchangeable across models but Conv5 less compatible, especially for PLCS.The identity stitching map yields top-1 error > 99%, whereas learned stitching establishes strong equivalence between several independently parameterized networks.
- 3.4. Application to structured-output regression: In structured-output pose regression, equivariant regression is generally as accurate or nearly as accurate as direct regression while being up to 22 times faster.The application evaluates HOG and CNN Conv3, Conv4, and Conv5 features on pose estimation using rotations and affine transformations.
4. Summary
The paper studies representations through learned equivariance and equivalence properties, finding predictable transformations and interchangeability in shallow and early deep CNN layers. Deeper layers retain these properties less strongly and are more task-specific, while the methods also support structured-output regression applications.
- The paper introduces learning-based analyses of representation equivariance and equivalence.
- Shallow representations and the first several layers of deep state-of-the-art CNNs transform predictably under image warps.
- These early representations are interchangeable across different architectures and therefore equivalent.
- Deeper layers share these properties to a lesser degree because they are more task-specific.
- The learned equivariant mappings also have practical applications, including accelerating structured-output regressors.