PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Equivariant Capsule Networks

Jan Eric Lenssen, Matthias Fey, Pascal Libuschewski

arXiv:1806.05086v2cs.CV

TL;DR

General capsule networks lack guaranteed equivariance and invariance, properties identified as important for disentangled representations and viewpoint invariance. The paper defines iterative routing over group-valued poses using a weighted mean and distance measure, and connects group capsules with group convolutions. On MNIST trained on non-rotated images and tested at rotations by multiples of π/2, accuracy remained 99.02%; experiments confirmed equivariance and invariance, and combined capsules-convolutions outperformed both alone.

  • Problem

    General capsule networks lack guaranteed equivariance and invariance, properties identified as important for disentangled representations and viewpoint invariance.

    Source: §1.2 · p. 2

  • Method

    The paper defines iterative routing over group-valued poses using a weighted mean and distance measure, and connects group capsules with group convolutions.

    Source: §2.1 · p. 3, §4.1 · p. 6

  • Results

    On MNIST trained on non-rotated images and tested at rotations by multiples of π/2, accuracy remained 99.02%; experiments confirmed equivariance and invariance, and combined capsules-convolutions outperformed both alone.

    Source: §6.2 · p. 8

  • Takeaways & Limitations

    The framework combines group capsules and group convolutions, supports sparse evaluation of group-convolution feature maps, and produces pose vectors described as disentangled representations.

    Source: §4.1 · p. 6, §8 · p. 9

    Pose vectors must belong to a group for which suitable averaging and distance operations are available, restricting the poses that can be extracted.

    Source: §7 · p. 9

Abstract

from arXiv · show

We present group equivariant capsule networks, a framework to introduce guaranteed equivariance and invariance properties to the capsule network idea. Our work can be divided into two contributions. First, we present a generic routing by agreement algorithm defined on elements of a group and prove that equivariance of output pose vectors, as well as invariance of output activations, hold under certain conditions. Second, we connect the resulting equivariant capsule networks with work from the field of group convolutional networks. Through this connection, we provide intuitions of how both methods relate and are able to combine the strengths of both approaches in one deep neural network architecture. The resulting framework allows sparse evaluation of the group convolution operator, provides control over specific equivariance and invariance properties, and can use routing by agreement instead of pooling operations. In addition, it is able to provide interpretable and equivariant representation vectors as output capsules, which disentangle evidence of object existence from its pose.

1 Introduction

Equivariance and invariance can help disentangle pose from other factors of variation, but general capsule networks do not guarantee these properties. The paper proposes group-based capsule layers and routing to provide guarantees under specified conditions.

1.1 Capsule networks

  • Capsule networks encode object pose separately from activation, which represents evidence of object existence.Source: §1.1 · p. 1

1.2 Motivation and contribution

  • Current capsule architectures lack guaranteed equivariance and invariance because vector-space averaging and pose-agnostic spatial kernels can produce non-equivariant poses and votes.Source: §1.2 · p. 2
  • The paper introduces group capsule layers and a general routing-by-agreement scheme that guarantee pose equivariance and activation invariance under stated conditions.Source: §1.2 · p. 2
  • The framework also combines group capsules with group convolutions to build networks with guaranteed equivariance and invariance and disentangled representations.Source: §1 · p. 1, §1.2 · p. 2

2 Group equivariant capsules

Group capsule layers compute pose vectors and activations through iterative routing over group-valued poses. With suitable averaging and distance operations, their outputs have provable equivariance and invariance properties that persist when layers are stacked.

2 Group equivariant capsules

  • Group capsule layers map input poses and activations to output poses and activations, with separate functions computing each output.Source: §2 · p. 2
  • The layer aims for output pose equivariance and activation invariance under the group operation.Source: §2 · p. 3

2.1 Group capsule layer

  • Routing casts votes by transforming input poses, repeatedly averages them, updates weights using pose distances, and derives activations from agreement.Source: §2.1 · p. 3
  • Equivariance and invariance are guaranteed when the weighted mean is closed and left-equivariant on the group and the distance measure preserves distances under group actions.Source: §2.1 · pp. 3–4
  • Stacking layers satisfying the properties preserves invariant output activations and equivariant pose vectors throughout the network.Source: §2 · p. 3, §2.1 · p. 4

2.2 Examples of useful groups

  • For SO(2), the paper represents group elements as unit vectors and uses a renormalized Euclidean weighted mean and negative scalar product as the distance measure.Source: §2.2 · p. 4
  • Direct products of groups let pose vectors carry independent, disentangled information for each group component.Source: §2.2 · p. 4

3 Spatial aggregation with group capsules

Spatial aggregation can break equivariance because receptive-field positions and trainable transformations are matched independently of input pose. The proposed pose alignment adjusts those positions before generating transformations, making the vote construction independent of the input transformation.

3 Spatial aggregation with group capsules

  • Spatial aggregation can break equivariance because pose-agnostic transformations at fixed receptive-field positions make votes depend on the input transformation.Source: §3 · p. 4
  • For a rotated 2×2 SO(2) capsule block, kernel alignment restores matching between poses and transformations and full agreement with an equivariant mean pose.Source: Figure 2 · p. 5
  • The method aligns receptive-field positions using their mean pose before passing them to a trainable transformation generator.Source: §3 · p. 5
  • Using the induced representation equally for poses and transformations makes their combination independent of the input transformation.Source: §3 · p. 5

4 Group capsules and group convolutions

Group capsule networks combine capsule pose and activation outputs with group convolutions to create sparse group-convolution networks. The combination preserves activation invariance and supports equivariance in other group dimensions under the stated conditions.

4 Group capsules and group convolutions

  • Group convolutions are equivariant under the group law but do not directly compute disentangled representations; combining them with capsules aims to retain that capability.The combined approach evaluates group convolutions at a fixed set of arbitrary group elements rather than using a regular group discretization.Source: §4 · p. 5

4.1 Sparse group convolution

  • Capsule poses guide sparse evaluation of group-convolution feature maps, while agreement values dampen or amplify their contents.The capsule output acts like a sparse index/value representation of a group-convolution map.Source: §4.1 · p. 6
  • For a group convolution evaluated at capsule output poses, the authors prove invariance under joint left-applications to the capsule input poses and signal.The theorem concerns a group capsule layer with pose-vector outputs, an input signal, and a filter.Source: §4.1 · p. 6
  • For product groups, capsule poses can index one dimension while the convolution is evaluated densely over another, yielding invariance in the capsule-indexed dimension and equivariance in the dense dimension.The authors describe this as the basis of their proof-of-concept application with two-dimensional rotation and translation.Source: §4.1 · p. 6
  • Local input neighborhoods are transformed by the inverse capsule pose before convolution, and routing weights can aggregate feature maps instead of max or average pooling.The transformed neighborhoods can be aggregated with a continuous kernel function.Source: §4.1 · p. 6, Figure 3 · p. 6

5 Related work

Prior work introduced several ways to model transformations, including group convolutions and specialized rotationally equivariant networks. This work builds on capsule routing and connects it to group-equivariant networks while aiming for a broader framework and disentangled representations.

Related work

  • Group convolutional networks generalize convolution to group elements and have been applied in spherical and steerable CNNs.These approaches use special convolution realizations for SO(3) and SO(2), respectively.Source: §5 · p. 7
  • Capsule networks and dynamic routing by agreement provide the starting point for connecting capsule methods with group-equivariant networks.The cited capsule work includes Hinton et al. [2011], Sabour et al. [2017], and Hinton et al. [2018].Source: §5 · p. 7
  • Harmonic and vector field networks address two-dimensional rotational equivariance, while this work aims at different groups and disentangled representations.The experiment is described as similar to those approaches.Source: §5 · p. 7

6 Experiments

The experiments test rotational equivariance, invariance, classification, and pose representations in MNIST-based settings. The combined capsule-and-convolution model outperforms its component parts, while the reported results also show equivariant poses and controllable reconstructions with some errors.

6.1 Implementation and training details

  • Initial image poses are extracted from local Sobel gradients, using gradient length as activation and an undefined pose for zero gradients.The authors identify first pose extraction as a subject not yet tackled generally for group capsule networks.Source: §6.1 · p. 7
  • The implementation uses spline-based convolution to transform local neighborhoods without resampling grids after group transformations.The continuous definition range of B-spline kernels enables transforming relative positions on image-grid graph edges.Source: §6.1 · p. 7
  • Routing uses sigmoid weights rather than a softmax over output capsules, allowing information to reach multiple or no output capsules; pose proposals are computed in two iterations.Source: §6.1 · p. 7
  • The experiments use a five-layer capsule architecture with 235k trainable parameters, pairing each capsule with a pose-indexed convolution.The models train for 45 epochs; the capsule and convolution outputs use spread loss and cross-entropy loss, respectively.Source: §6.1 · p. 7

6.2 Results

  • Training on non-rotated MNIST and testing on rotations by multiples of π/2 yields the same 99.02% accuracy as the non-rotated test set, with output activations unchanged and pose-vector errors below 0.0001.For arbitrary test rotations, the non-rotated-trained network achieves 89.12% accuracy.Source: §6.2 · p. 8
  • On affNIST, the combined model surpasses Sabour et al.’s 79% result using 235k rather than 6.8M parameters.The authors also report that the combined architecture significantly outperforms the capsule and convolution parts alone, though it does not reach Weiler et al.’s 99.29% rotated-MNIST result.Source: §6.2 · p. 8
  • Capsule poses are more accurate than hierarchical averaging of local poses, and reconstruction loss improves pose extraction despite the absence of direct pose supervision.The study measures average pose error under random rotations; Figure 4 also shows mostly equivariant output poses, with an ambiguous-pattern error case and internal-pose differences from discretization and resampling.Source: §6.2 · p. 8, Figure 4 · p. 9
  • Rotating the output pose vector gives fine-grained control over reconstructed image orientation, but not all representations reconstruct correctly.The displayed examples were selected from visually correct reconstructions.Source: §6.2 · p. 9

7 Limitations

The method restricts capsule poses to groups with suitable averaging operation M and distance function δ, limiting the pose vectors it can extract. Product groups may broaden the representation, but finding applicable operators and group realizations remains future work.

7 Limitations

  • Capsule poses must belong to a group for which suitable M and δ are available, so arbitrary pose vectors from original capsule networks cannot be extracted.The restriction is the stated source of the method's limitations.Source: §7 · p. 9
  • Product groups can combine groups to produce more general pose vectors with internally disentangled information, provided suitable M and δ can be found.Source: §7 · p. 9
  • For Lie groups, an equivariant Karcher mean would suffice for M, but identifying suitable realizations and evaluating them is future work.The Karcher mean minimizes the weighted sum of geodesic distances on the manifold.Source: §7 · p. 9

8 Conclusion

The paper concludes that group equivariant capsule networks provide provable equivariance and invariance, with routing, spatial aggregation, and group-convolution integration. It reports proof-of-concept validation and identifies further group applications and supporting appendix material.

8 Conclusion

  • The proposed network combines provable equivariance and invariance with routing by agreement, spatial aggregation, and integration of group convolutions.The authors report proofs, proof-of-concept experiments, disentangled pose vectors, and an example of sparse group-equivariant CNNs using guiding poses.Source: §8 · p. 9
  • Future work will apply the framework to higher-dimensional groups to approach original capsule networks' expressiveness while preserving guarantees.Source: §8 · p. 9

Appendix

  • The appendix provides detailed proofs, information on applicable groups, formal capsule convolution with product groups, and quantitative pose-vector analysis.Source: Appendix · p. 11

A Proofs for theorems

The proofs establish conditions under which group capsule layers produce equivariant pose outputs and invariant activations, and extend the invariance result to group convolution. The routing proof uses invariant weights across iterations to preserve pose equivariance.

Algorithm 1

  • Algorithm 1 transforms input poses into votes, averages votes into output poses, iteratively refines routing weights, and returns poses with activations.Weights depend on distances between current output poses and votes, scaled by input activations.Source: A Proofs for theorems · p. 11

Theorem 1

  • Pose outputs from Algorithm 1 are equivariant under left group applications when M is equivariant and closed on G, and δ preserves distance under group actions.The proof proceeds by induction: invariant routing weights preserve equivariance at each iteration.Source: A Proofs for theorems · pp. 11–12

Theorem 2

  • Algorithm 1's output activations are invariant under joint left applications to the input pose vectors, under Theorem 1's conditions.The proof invokes Theorem 1 and δ-distance preservation.Source: A Proofs for theorems · p. 12

Theorem 3

  • Group convolution is invariant when group actions are jointly applied to capsule poses and the input signal, given the group capsule pose outputs.The proof uses the feature-map group action, substitution, and group inverse properties.Source: A Proofs for theorems · p. 12

B Examples for useful groups

The paper describes suitable group-specific choices of averaging M and distance δ, including rotation and translation examples. Translation capsules suggest sparse evaluation, but the authors note optimization difficulties in their current architectures.

The two-dimensional rotation group SO(2)

  • For SO(2), unit-vector poses, a renormalized Euclidean weighted mean, and negative scalar-product distance meet the framework's requirements when poses do not sum to zero.The zero-sum condition can be prevented through practical measures.Source: B Examples for useful groups · p. 12

Translation group (Rn, +)

  • Translation capsules could sparsely evaluate feature maps at predicted points of interest, reducing evaluations from output capsules times pixels to output capsules.The authors do not expect this construction to work in current architectures because gradients may not guide transformations; local filters and sparse evaluations may also yield zero outputs.Source: B Examples for useful groups · p. 13

Higher dimensional groups

  • For translation groups, the Euclidean weighted average and l2 distance satisfy the requirements, while SO(3) can use a projected weighted matrix mean and Frobenius distance.The SO(3) averaging operator is not trivial to compute as a differentiable neural-network module.Source: B Examples for useful groups · p. 13

C Product group convolutions

Direct-product groups let the framework route sparsely over one transformation group and convolve densely over another, while preserving the corresponding equivariance and invariance properties. For SO(2) × (R2, +), this means sparse rotation handling alongside dense spatial translation processing.

C Product group convolutions

  • For SO(2) × (R2, +), the framework evaluates spatial translations densely, evaluates rotations sparsely, and routes between rotations across layers.The product-group setup preserves equivariance and invariance for the routed group and equivariance for the unrouted group.Source: C Product group convolutions · p. 13
  • Theorem 4 states that group convolution is invariant under joint left-applications of an element of R to capsule input poses and the signal, for all t in T.The theorem applies to a direct product group R × T under its stated conditions.Source: C Product group convolutions · p. 13
  • For each spatial position and sparse rotation, the convolution can be implemented by rotating a local input window before applying convolution.This yields rotationally invariant activations and equivariant pose vectors.Source: C Product group convolutions · p. 14
  • The method uses a direct product rather than the entangled roto-translation group SE(2), keeping the group components separate for simpler proofs and disentangled information.The authors expect SE(2) modeling to be possible, but do not use it here.Source: C Product group convolutions · p. 14

D Quantitative analysis of pose vectors

With arbitrary rotations requiring image resampling, pose-vector errors vary across MNIST classes; large errors are less frequent than errors near the correct pose.

D Quantitative analysis of pose vectors

Related papers

Loading 1806.05086v2…