PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Equivariant Convolutional Networks

Taco S. Cohen, Max Welling

arXiv:1602.07576v3cs.LGstat.ML

TL;DR

Standard convolutions exploit translation symmetry but do not directly capture broader transformations such as rotations and reflections. The paper introduces G-CNNs with G-convolutions, achieving state-of-the-art results on rotated MNIST and CIFAR10 while remaining compatible with standard architectures and discrete-group computation.

  • Problem

    Standard convolutional networks primarily exploit translation symmetry, motivating a generalization that can use larger symmetry groups such as rotations and reflections.

  • Method

    G-CNNs use G-convolutions and group-structured representations to share weights across transformations while preserving equivariance through compatible network layers.

  • Results

    G-CNNs achieve state-of-the-art results on rotated MNIST and CIFAR10, including 2.28% error on MNIST-rot and 4.19% and 6.46% on augmented and plain CIFAR10.

  • Takeaways & Limitations

    G-convolutions can replace spatial convolutions in modern architectures and improve performance without further tuning, with roughly planar-convolution computational cost.

  • Takeaways & Limitations

    The presented method works only for discrete groups, while continuous groups may be difficult to approximate equivariantly and large groups may be difficult to enumerate fully.

Abstract

from arXiv · show

We introduce Group equivariant Convolutional Neural Networks (G-CNNs), a natural generalization of convolutional neural networks that reduces sample complexity by exploiting symmetries. G-CNNs use G-convolutions, a new type of layer that enjoys a substantially higher degree of weight sharing than regular convolution layers. G-convolutions increase the expressive capacity of the network without increasing the number of parameters. Group convolution layers are easy to use and can be implemented with negligible computational overhead for discrete groups generated by translations, reflections and rotations. G-CNNs achieve state of the art results on CIFAR10 and rotated MNIST.

1. Introduction

The paper generalizes convolutional networks to exploit rotations and reflections while preserving the benefits of equivariance and weight sharing. G-CNNs achieve state-of-the-art results on rotated MNIST and CIFAR10 without additional tuning.

  • Convolutional weight sharing exploits approximate translation symmetry, reducing parameters while preserving capacity for useful transformations.
  • Translation equivariance lets every layer preserve symmetry, extending weight sharing from image inputs to higher-level feature maps.
  • G-CNNs generalize convolutional networks to larger symmetry groups, including rotations and reflections.
  • G-convolutions and compatible CNN components, including pooling, nonlinearities, batch normalization, and residual blocks, remain equivariant.
  • 2.28% error on MNIST-rot and 4.19% and 6.46% on augmented and plain CIFAR10 demonstrate state-of-the-art results without additional tuning.

2. Structured & Equivariant Representations

The paper frames equivariance as structure-preserving representation learning, using group actions to retain feature pose information across transformations. This structure supports efficient modeling and can aid generalization.

  • G-CNN representations are linear G-spaces in which each vector carries a pose transformable by elements of a chosen group G.
  • A layer preserves G-space structure when transforming its input before mapping matches mapping first and transforming the resulting representation.
  • Equivariance permits different input and output transformation operators, provided the input operator forms a linear representation of G.
  • Invariance is a special case of equivariance, but general equivariance retains spatial configuration needed to determine whether features are correctly arranged.
  • Equivariance constrains representations so relationships judged identical by the network remain identical after group transformations, which can aid generalization.

3. Related Work

Related work develops invariant and equivariant representations through pose normalization, group averaging, learned transformations, and specialized convolutional architectures. These studies motivate equivariance as an inductive bias for deep networks.

  • Invariant representations have been constructed through pose normalization with equivariant detectors or by averaging functions over a group.
  • Scattering networks combine wavelet convolutions, nonlinearities, and group averaging to produce stable invariants for recognition tasks.
  • Prior work includes transforming autoencoders, equivariant Boltzmann machines, equivariant descriptors, and equivariant filtering.
  • AlexNet and unsupervised convolutional models have been trained or observed to learn representations equivariant to transformations such as flips, scaling, rotation, and ego-motion.
  • Group-theoretic learning theories have been developed for sensory cortex and implemented for time and vocal-tract-length shifts in speech recognition.
  • Other convolutional approaches address high-dimensional transformation groups or exploit cyclic and rotation symmetries in galaxy morphology and computer vision.
  • Disentangling has been interpreted as reducing transformation operators in equivariant representations and related to statistical decorrelation.

4. Mathematical Framework

The mathematical framework models feature maps as functions on symmetry groups and defines their transformations, group structures, and visualization. It covers translation, rotation, and reflection groups used by G-CNNs.

  • The framework defines symmetry groups, functions on groups, and their transformation properties to support generic G-CNN analysis.
  • A symmetry group is a set of transformations closed under composition and inversion, including the identity transformation.
  • The integer translations Z2 form a group under addition, making group convolution applicable beyond this particular example.
  • 4.1. Symmetry Groups: The p4 group combines translations with 90-degree rotations, parameterized by r, u, and v, with composition represented by matrix multiplication.
  • 4.1. Symmetry Groups: The p4m group extends p4 with mirror reflections and uses an additional binary parameter m alongside rotations and translations.
  • 4.4. Functions on groups: Conventional CNN images and feature stacks are modeled as bounded-domain functions from Z2 to R^K, with K channels at each pixel.
  • 4.3. Transformations of functions: Feature-map transformations use inverse group elements to look up the original value mapped to a target point, preserving the transformation composition law.
  • 4.4. Functions on groups: G-CNN feature maps are functions on G, and their transformation rule extends by replacing spatial points with group elements and interpreting products as composition.

5. Equivariance properties of CNNs

Standard CNN convolutions preserve translation equivariance, but this property does not generally extend to rotations and other lattice isometries. Under rotations, equivariance can be recovered only by transforming the filters alongside the feature maps.

  • Layer operations: At each layer, regular CNNs correlate stacks of spatial feature maps with spatial filters, using correlation in the forward pass and convolution terminology generically.The forward and backward operations use correlation and convolution in opposite roles, but the paper refers to both generically as convolution.
  • Translation equivariance: Convolution and correlation commute with translations, making standard CNN layers equivariant to the translation group.The result follows by substituting the translated coordinates in the correlation operation; convolution has the analogous property.
  • Limits of standard CNNs: Standard convolutions are not generally equivariant to rotations or other non-translation isometries of the sampling lattice.Rotating before convolution differs from convolving first and then rotating the result when the filter remains fixed.
  • Rotated inputs: For a rotated input, correlation with a filter equals rotating the original response while applying the inverse-rotated filter.Thus, an ordinary CNN can achieve stack-level equivariance by learning rotated filter copies, although individual feature maps remain non-equivariant.

6. Group Equivariant Networks

G-CNNs generalize convolution to functions on transformation groups, allowing filters and feature maps to transform consistently under rotations, reflections, and translations. Equivariant nonlinearities and pooling preserve this structure, while subgroup subsampling trades full G-equivariance for H-equivariance unless coset pooling is used.

  • Deep G-CNNs: Because all layer types are equivariant, G-convolution, pooling, nonlinearities, batch normalization, and residual blocks can be stacked into deep G-CNNs.The paper therefore expects G-convolution parameter sharing to remain effective at arbitrary depth.
  • G-convolution: G-correlation replaces spatial shifts with transformations from a group G, producing first-layer feature maps defined on G rather than only on the plane.After the first layer, filters are also functions on G, so full G-correlation operates between group-valued feature maps and filters.
  • Equivariance: The same equivariance formula applies to first-layer and full G-correlation, but the left-action operator has different domains in the two cases.In the first layer, inputs are functions on Z2 and outputs on G; in later layers, both filters and feature maps are functions on G.
  • Group structure: G-correlation is generally noncommutative when G is noncommutative, although reversing the operands preserves information through an invertible involution.The ordering f ⋆ψ is computationally preferable because transforming a small filter is faster than transforming a large feature map.
  • Nonlinearities: Pointwise nonlinearities commute with group transformations, so rectified feature maps inherit the transformation properties of the preceding layer.The proof uses function composition: precomposition by the group action commutes with postcomposition by the nonlinearity.
  • Subgroup pooling and subsampling: Subsampling on a subgroup H preserves equivariance to H but not necessarily to the full group G.For p4-CNNs, H can contain all four rotations together with shifts by multiples of two pixels.
  • Coset pooling: Choosing the pooling region as a subgroup H yields coset pooling, whose output is invariant to H and can be viewed as a function on the quotient space G/H.In p4, pooling over all four rotations produces a feature map on Z2 ≅ p4/R that transforms like the input image.

7. Efficient Implementation

The implementation reduces G-convolution to filter transformations followed by a planar convolution, using indexing arithmetic and reshaping to exploit existing fast routines. For split groups such as p4 and p4m, the added transformation cost is negligible relative to the planar convolution.

  • Implementation strategy: G-convolution can be implemented with indexing arithmetic and inner products while leveraging fast planar-convolution routines.The implementation first transforms filters and then applies a planar correlation to the expanded filter bank.
  • Split groups: Split plane symmetry groups decompose each transformation into a translation and an origin-preserving transformation, enabling filter-bank expansion.For p4, the origin-preserving transformations are rotations; for p4m, they are rotation-flips.
  • Filter transformation: For p4 and p4m, transformed filters are computed for four rotations or eight rotation-flips before invoking a fast planar correlation routine.The first layer uses X = Z2, while later layers use X = G in the correlation formulation.
  • Computational cost: The computational cost is roughly equal to planar convolution with an equally sized augmented filter bank because filter transformation costs are negligible.This comparison includes the planar convolution over the expanded filters, not an unmodified planar filter bank.
  • Filter representation: Filter tensors encode output channels, input channels, stabilizer transformations, and spatial extent, with typical deeper-layer stabilizer sizes of four or eight.For 2D image inputs, S1 is typically 1, whereas later layers commonly use Sl = 4 or Sl = 8.
  • Tensor reshaping: Filter transformation permutes tensor entries into an expanded bank, which can be reshaped into conventional planar input and output channels.A new group can be supported by implementing the index-to-matrix map g(·).

8. Experiments

Experiments replace standard convolutions with p4 or p4m convolutions in rotated MNIST and CIFAR10 architectures, generally improving accuracy while approximately preserving parameter counts. The strongest reported results are 2.28% error on rotated MNIST and 4.19% error on augmented CIFAR10.

  • Rotated MNIST: The rotated MNIST evaluation uses 62,000 randomly rotated digits split into 10,000 training, 2,000 validation, and 50,000 test examples.The CNN baseline uses seven convolutional layers with batch normalization, dropout, and max-pooling.
  • Rotated MNIST: 2.28% vs 3.98% error: P4CNN almost halves the previous state-of-the-art error on rotated MNIST.The model replaces convolutions with p4-convolutions, approximately preserves parameter count, and adds max-pooling over rotations after the final convolution.
  • Rotated MNIST: Rotation pooling in every intermediate layer performs significantly worse than P4CNN, supporting the hypothesis that premature invariance is undesirable.P4CNNRotationPooling still outperforms the baseline and previous state of the art.
  • CIFAR10: The experiments compare conventional, p4, and p4m convolutions across All-CNN-C and residual-network architectures on CIFAR10 and augmented CIFAR10+.Augmented CIFAR10+ uses horizontal flips and small translations.
  • CIFAR10: Replacing baseline convolutions with p4 or p4m convolutions increases feature-map size while approximately preserving parameters by reducing the number of filters.Filters are halved for p4-convolution layers and divided by roughly 3 for p4m-convolution layers.
  • CIFAR10: 4.19% vs 5.27% error: p4m convolutions outperform planar convolutions in a 26-layer ResNet on moderately augmented CIFAR10.The p4m model uses 7.2M parameters, compared with 36.5M for the cited 4.17%-error model.

9. Discussion & Future work

G-CNN convolutions consistently improve results as drop-in replacements for standard convolutions and can benefit from augmentation drawn from a larger symmetry group. The method is limited to discrete groups as presented, while future work targets richer spatial groups.

  • Discussion: p4 and p4m convolution layers consistently improve results as drop-in replacements for standard convolutions.The paper reports this pattern across modern network architectures.
  • Discussion: G-CNNs benefit from augmentation when the augmentation group is larger than G; flips and small translations improve p4 and p4m results.This extends the augmentation behavior observed for ordinary convolutional networks.
  • Discussion: CIFAR objects are typically upright, yet G-convolutions still substantially increase accuracy, so full dataset symmetry is not required for benefit.The paper presents this as evidence from CIFAR10.
  • Future work: Future work targets G-CNNs on hexagonal lattices and 3D space groups, with the existing theory stated to apply directly to these groups.The implementation can add new groups by specifying the group operation and a bijection to index sets.
  • Limitations: As presented, the method works only for discrete groups; continuous groups may be difficult to approximate equivariantly, and large groups may be infeasible to enumerate fully.These constraints define the method’s current scope and computational boundary.
  • Future work: Structured representations may help neural networks recognize abstract similarities between superficially different concepts.This is presented as a broader philosophy motivating future applications of equivariant representations.

10. Conclusion

G-CNNs generalize convolutional networks by exploiting group symmetries, increasing expressive capacity without increasing parameters while achieving state-of-the-art results.

  • G-CNNs substantially increase network expressive capacity without increasing the number of parameters.
  • G-CNNs achieve state-of-the-art results on rotated MNIST and CIFAR10.
  • G-convolutions can replace spatial convolutions in modern architectures and improve performance without further tuning.

Appendix A: Equivariance Derivations

The appendix derives equivariance and gradient computations for G-CNN operations, showing how symmetry-preserving layers and convolution-like backward passes are implemented.

  • Ordinary planar correlation is not equivariant to rotations, motivating the generalized G-convolution framework.
  • Pooling commutes with the group action, so it remains compatible with the equivariant network construction.
  • For training, the fast algorithm requires implementing the gradient of filter transformation, while convolution and its gradient are already available.
  • The gradient of an indexing operation accumulates output-cell gradients into the corresponding input cells and can be implemented efficiently on GPUs.
  • The gradient with respect to a filter component is computed by G-convolving the corresponding input feature map with the output feature map.
  • Both forward and backward passes use convolution or correlation operations, paralleling standard convolutional networks.
Loading 1602.07576v3…