PaperlayerView exact revision on arXiv

Source-linked AI summary

Steerable CNNs

Taco S. Cohen, Max Welling

arXiv:1612.08498v1cs.LGstat.ML

TL;DR

Existing equivariant CNNs are costly to scale to larger transformation groups, limiting flexible symmetry-aware representations. This paper develops steerable CNNs through a type-based theory of representations and feature-dependent parameterization, achieving state-of-the-art results on CIFAR10 and CIFAR100 while outperforming recent residual architectures.

  • Problem

    Existing equivariant CNNs are limited by computational costs that scale with transformation-group size, restricting their extension to larger groups.

  • Method

    The paper develops steerable CNNs using elementary feature types, capsule-based representations, and feature-type-dependent parameter costs for equivariant filters.

  • Results

    Steerable CNNs achieve state-of-the-art results on CIFAR10 and CIFAR100, substantially outperforming the ResNet baseline and improving over wide and dense residual networks.

  • Takeaways & Limitations

    Steerability is a useful inductive bias that can improve model accuracy, particularly when little data is available.

  • Takeaways & Limitations

    The experiments focus on the small discrete group of flips and rotations by multiples of 90 degrees, leaving large continuous high-dimensional groups for future evaluation.

Abstract

from arXiv · show

It has long been recognized that the invariance and equivariance properties of a representation are critically important for success in many vision tasks. In this paper we present Steerable Convolutional Neural Networks, an efficient and flexible class of equivariant convolutional networks. We show that steerable CNNs achieve state of the art results on the CIFAR image classification benchmark. The mathematical theory of steerable representations reveals a type system in which any steerable representation is a composition of elementary feature types, each one associated with a particular kind of symmetry. We show how the parameter cost of a steerable filter bank depends on the types of the input and output features, and show how to use this knowledge to construct CNNs that utilize parameters effectively.

1 INTRODUCTION

The introduction motivates equivariant representations as a way to improve statistical efficiency while preserving local-feature pose, and presents steerable CNNs as a scalable framework based on elementary feature types. This type system constrains architectures and reduces the parameters required by equivariant filter banks.

  • Motivation: Large labelled datasets and information-intensive deep networks have driven computer-vision progress, but reducing the need for labelled data remains an important challenge.The introduction contrasts current practical success with human ability to learn concepts from very few labels.
  • Motivation: Equivariance preserves the relative pose of local features while allowing representations to transform predictably under input transformations.Intermediate layers should not be fully invariant because later layers need relative-pose information.
  • Problem: Existing equivariant CNNs achieve state-of-the-art classification results, but methods for larger transformation groups become impractical because computational cost scales with group size.Prior methods enforce equivariance only to small groups such as rotations by multiples of 90 degrees.
  • Contribution: Steerable representations decompose into independently steerable elementary feature types, each capturing a distinct characteristic with invariant or objective meaning.The paper presents these feature types as the basis for steerable CNNs and relates them to observer-independent quantities.
  • Contribution: The type system constrains network weights and architecture, reduces parameters in filters mapping specified input to output types, and prevents combining features of different types.The introduction illustrates the type constraint through the analogy that meters should not be added to seconds and mentions residual learning as an example.

2 STEERABLE CNNS … 2.4 INDUCTION

Steerable CNNs organize feature spaces into fibers carrying group representations, constrain convolutional filters to intertwine these representations, and use induction to propagate local H-steerability to global G-steerability. This construction supports efficient equivariant layers whose transformation laws specify how feature channels mix under geometric symmetries.

  • 2.1 FEATURE MAPS AND FIBERS: Feature representations are decomposed into K-dimensional fibers F_x over spatial positions x, equivalently describing signals as feature vectors rather than only channel-wise maps.Each feature space F_l is a linear space of signals, with one fiber at every position in Z2.
  • 2.2 STEERABLE REPRESENTATIONS: Steerability requires transformed outputs Φπ(g)f and Φf to be related by an input-independent linear representation π′(g), making the network equivariant.The representation law π′(gh) = π′(g)π′(h) follows from steerability and the representation structure of π.
  • 2.2 STEERABLE REPRESENTATIONS: The framework focuses on discrete transformation groups such as p4m, formed from translations Z2 and origin-fixing transformations H, with H = D4 for p4m.This decomposition separates translations from rotations and reflections around the origin.
  • 2.3 EQUIVARIANT FILTER BANKS: A convolutional filter bank must be H-equivariant, rather than fully G-equivariant, because translations can move patterns across a fiber’s receptive field.Admissible filters are intertwiners in Hom_H(π, ρ), mapping input representation π to output-fiber representation ρ.
  • 2.3 EQUIVARIANT FILTER BANKS: Because the equivariance constraint is linear, admissible filter banks form the vector space Hom_H(π, ρ), whose basis can be computed by solving a linear system.The basis is computed offline, after which filters are learned as linear combinations parameterized by coefficients α_i.
  • 2.4 INDUCTION: Induction converts H-steerable output fibers into a feature space that is steerable under the full group G, establishing the layer-by-layer construction.The induced representation satisfies Ψ ⋆π(g)f = π′(g)Ψ ⋆f and is denoted π′ = Ind_G.
  • 2.4 INDUCTION: The induced representation π′ acts on the whole feature space, whereas ρ acts within individual fibers, determining how channels mix under transformations.Compared with the input representation, induction adds the fiber transformation factor ρ(r); trivial ρ leaves color channels unmixed.
  • 2.4 INDUCTION: After induction produces a G-steerable feature space, the procedure can be iterated by computing intertwiners between the restricted representation π′ and a chosen output representation ρ′.This recursive construction supplies the next equivariant convolutional layer.

2.5 FEATURE TYPES AND CHARACTER THEORY

Steerable representations are classified by multiplicities of irreducible symmetry types, which can be recovered from characters. These types determine equivariant filter-bank parameter counts and enable parameter-efficiency evaluation.

  • Feature types: Irreducible representations encode distinct feature symmetries; for example, B1 and B2 basis filters change sign under a 90-degree rotation.The elementary filters in higher layers can differ because they depend on the representation being decomposed.
  • Feature types: Choosing representation multiplicities determines the number of feature channels, while the basis choice is later restricted so multiplicities become the relevant hyperparameters.The channel dimension is K = P_i m_i dim ϕ_i.
  • Character theory: A representation’s type can be computed by taking character inner products, because irreducible characters are orthogonal and direct sums add their characters.Thus, a simple dot product of characters determines each irrep’s multiplicity.
  • Parameter efficiency: An equivariant filter bank has dim Hom_H(π, ρ) parameters, so its cost depends only on the input and output feature types.Parameter utilization compares this equivariant cost with an ordinary filter bank; typical effective D4 layers achieve μ = 8.

2.6 EQUIVARIANT NONLINEARITIES & CAPSULES · 2.7 COMPUTATIONAL EFFICIENCY · 2.8 USING STEERABLE CNNS IN PRACTICE

Steerable CNNs require basis-aware equivariant nonlinearities implemented through capsules, while their block structure enables efficient filter-bank construction. In practice, users select capsule multiplicities and admissible activations from developer-provided libraries rather than designing representations from scratch.

  • 2.6 EQUIVARIANT NONLINEARITIES & CAPSULES: Equivariant nonlinearities must commute with the group action, and their admissibility depends on the chosen representation basis rather than feature type alone.An admissible fiber-wise nonlinearity maps an input representation ρ to an output representation ρ′ while preserving commutation with the group action.
  • 2.6 EQUIVARIANT NONLINEARITIES & CAPSULES: A ρ-capsule is a typically low-dimensional feature vector transforming under ρ; capsules sharing a type can nevertheless be inequivalent because their bases differ.Fibers are constructed by stacking catalogued capsules with specified multiplicities.
  • 2.6 EQUIVARIANT NONLINEARITIES & CAPSULES: Signed-permutation capsules support concatenated nonlinearities such as CReLU, while quotient capsules are permutation-realized and compatible with any nonlinearity.For many groups, irreducible capsules can use CReLU; scale-free concatenated nonlinearities also apply to monomial-matrix representations.
  • 2.7 COMPUTATIONAL EFFICIENCY: Although equivariant filter banks can reduce parameters by about 10×, their full basis can remain impractically large, motivating block-structured computation.With hundreds of channels and 3 × 3 filters, the ordinary filter bank can have about 10^6 dimensions, while its equivariant basis can still be about 10^6 × 10^5.
  • 2.7 COMPUTATIONAL EFFICIENCY: Block-diagonal input and output representations induce intertwiner blocks, with each block corresponding to an input-output capsule pair and parameterized by basis matrices.Repeated equivalent capsules can be grouped into superblocks that reuse the same intertwiner basis.
  • 2.7 COMPUTATIONAL EFFICIENCY: The computation graph precomputes induced representations and intertwiner bases offline, then forms superblocks by multiplying each basis matrix stack by a learned parameter matrix.After filling superblocks, the resulting matrix is reshaped into the convolutional filter-bank dimensions.
  • 2.8 USING STEERABLE CNNS IN PRACTICE: Using steerable CNNs is comparable to using ordinary CNNs: users choose multiplicities for capsules in a developer-provided library and select each capsule’s admissible activation.Preserving equivariance requires choosing the activation function from the admissible list associated with that capsule.

3 RELATED WORK

Prior work developed steerable filters, equivariant kernels, and invariant or equivariant neural networks, while connecting steerability to group representation theory. This paper situates its type system within the decomposition of group representations associated with neural-network features.

  • Steerable filters: Steerable filters originated in signal processing and low-level vision, with subsequent work connecting steerability to group representation theory.These connections were observed across several studies, including Lenz (1989), Koenderink & Van Doorn (1990), Teo (1998), and Krajsek & Mester (2007).
  • Equivariant learning: Machine-learning research studied equivariant kernels and representations through capsules, brain-inspired equivariance, and group-equivariant scattering networks.The cited work includes Reisert (2008), Skibbe (2013), Hinton et al. (2011), Tieleman (2014), Anselmi et al. (2014), and Mallat (2012).
  • Invariant and equivariant CNNs: Invariant and equivariant CNNs, along with scattering networks, commonly use feature maps interpreted as functions on G carrying the regular representation.The cited CNN work includes Gens & Domingos (2014), Kanazawa et al. (2014), Dieleman et al. (2015; 2016), Cohen & Welling (2016), and Marcos et al. (2016).
  • Type systems: A neural-network type system emerges naturally by decomposing a group representation associated with the representation learned by the network.This connects the paper’s type-system perspective to earlier discussions of adding type systems to neural networks by Olah (2015) and Balduzzi & Ghifary (2015).

4 EXPERIMENTS

Experiments on CIFAR10 show that steerable CNNs provide a useful inductive bias, especially when combining capsule types. The final architecture outperforms ResNet baselines and achieves state-of-the-art results on full CIFAR10 and CIFAR100.

  • Experimental setup: Experiments used CIFAR10 with only 2,000 training samples initially to test steerability as an inductive bias and compare capsule types in the small-data regime.The baseline was a tuned 20-layer wide ResNet.
  • Capsule comparisons: Single-capsule architectures generally perform poorly at roughly 30–40% error, but regular-representation capsules reach 26.75% error and outperform standard CNNs.The experiments suggest that many capsule types restrict the spatial patterns the network can express.
  • Mixed capsule architectures: With 2k CIFAR10 labels, the mixed-capsule architecture achieves 24.48% error, outperforming standard ResNets and regular capsules.Its residual blocks use quotient capsules at the input/output layers and irreducible capsules in the intermediate layer.
  • Final results: On full CIFAR10 and CIFAR100, steerable CNNs substantially outperform the ResNet baseline and achieve state-of-the-art results; on 4k-label CIFAR10, they approach semi-supervised state of the art.The 4k-label method also outperforms DCGAN transfer learning, which achieves 26.2% error.

5 CONCLUSION & FUTURE WORK

The paper presents steerable representations as a theoretical framework and useful inductive bias, with a simple steerable architecture achieving state-of-the-art CIFAR10 and CIFAR100 results. Future work includes extending the theory to large continuous groups, learning feature types, and applying steerable CNNs beyond classification.

  • A simple steerable architecture achieves state-of-the-art results on CIFAR10 and CIFAR100, outperforming recent wide and dense residual networks.Steerability particularly improves accuracy when little data is available.
  • The framework connects representation learning with representation theory, clarifying the inner workings of equivariant convolutional networks.The authors characterize CNNs as geometrical computation engines and expect representation theory to benefit future research.
  • Evaluating steerable CNNs for large, continuous, and high-dimensional groups is an important direction for future work.The paper’s running example uses flips and rotations by multiples of 90 degrees, a small discrete non-commutative group, while the theory extends to continuous settings.
  • Future work includes learning feature types and applying steerable CNNs to action recognition, pose and motion estimation, and continuous control.Learning feature types may be easier for non-compact groups because their irreducible representations occupy a continuous space suitable for optimization.

G. B. Folland. A Course in Abstract Harmonic Analysis. CRC Press, 1995.

The section lists prior work on steerable filters, steerable pyramids, group-theoretical representations, and symmetry-aware convolutional networks. It also includes foundational and adjacent work on residual, dense, invariant, scattering, and generative neural architectures.

  • Prior steerable-filter work includes steerable filters, overcomplete steerable pyramid filters, and multi-scale derivative computation.
  • Related neural-network research covers deep symmetry networks, equivariance, rotation and scale invariance, scattering, residual and dense connectivity, and generative or semi-supervised learning.
  • The bibliography includes foundational treatments of group-theoretical methods, induced representations, finite-group representations, and harmonic analysis.

APPENDIX A: INDUCTION

This appendix derives that convolution with an H-equivariant filter bank produces feature maps transforming according to the induced representation. It establishes the result using explicit group notation, matrix computations, and a geometric interpretation.

  • Induction: The appendix derives the transformation law showing that convolution with an H-equivariant filter bank yields feature maps transforming under the induced representation.This derivation establishes equation 5, whose final line is the transformation result stated in the paper.
  • Notation: The notation distinguishes points x ∈ Z2 from translations ¯x ∈ G and separates G’s action on itself from its action on Z2.The section map ¯· selects translations in G that project to points in Z2, while the two actions use matrix–matrix and matrix–vector multiplication, respectively.
  • Matrix formulation: An explicit matrix representation writes group elements using a transformation matrix R and translation vector T, making the induced-representation computations checkable.The convolution is defined using this notation and the chosen section.
  • Interpretation: The algebraic derivation is acknowledged as involved, while the induced representation is presented geometrically as a natural construction; broader treatments appear in Serre (1977), Mackey (1952), and Reeder (2014).The appendix characterizes the complexity as arising from limited abstraction in the presentation.

APPENDIX B: RELATION TO GROUP EQUIVARIANT CNNS

This section establishes that Group Equivariant Convolutional Networks (G-CNNs) are a special case of steerable CNNs, specifically those using regular capsules.

  • Relation to Group Equivariant CNNs: G-CNNs are steerable CNNs with regular capsules.Their feature maps transform according to the regular representation, which is naturally realized by permutation matrices.
  • Relation to Group Equivariant CNNs: Unlike the plane-based feature maps used here, G-CNN feature maps are functions from G to R^K, except for the input maps.The regular representation of G is induced by the regular representation of H, which the paper defines as regular capsules.

APPENDIX C: REGULAR AND QUOTIENT FEATURES

Quotient features arise from the action of a finite group H on cosets H/K, yielding permutation representations on functions over each quotient. The trivial-subgroup case gives regular representations, whose use in steerable CNNs is equivalent to group convolutions and strictly generalizes G-CNNs.

  • Quotient features: Each subgroup K of H defines a quotient H/K whose cosets partition H and carry a natural left action by H.The action is a · bK = (ab)K.
  • Quotient features: Functions on H/K inherit an H-representation because group actions permute cosets and therefore permute the attached function values.These representations can be realized by permutation matrices and computed by hand for small groups or automated for large groups.
  • Regular features: For K = {e}, H/K is isomorphic to H and the resulting function space is the regular representation.Using regular representations in steerable CNNs is equivalent to group convolutions introduced in Cohen & Welling (2016).
  • Regular features: Thus, steerable CNNs are a strict generalization of G-CNNs when regular representations are used.This follows from the equivalence between regular representations and the group convolutions introduced in Cohen & Welling (2016).
  • D4 example: For D4, the listed quotient features include r with dimensionality 2, r2m with dimensionality 2, and r2mr with dimensionality 2.These correspond respectively to the subgroups {e, r, r2, r3}, {e, r2, m, mr2}, and {e, r2, mr, mr3}.
Loading 1612.08498v1…