Exploiting Cyclic Symmetry in Convolutional Neural Networks

Sander Dieleman; Jeffrey De Fauw; Koray Kavukcuoglu

Source-linked AI summary

Exploiting Cyclic Symmetry in Convolutional Neural Networks

Sander Dieleman, Jeffrey De Fauw, Koray Kavukcuoglu

arXiv:1602.02660v2Feb 8, 2016cs.LGcs.CVcs.NE

TL;DR

CNNs often need to learn rotational behavior from data even when images have rotational symmetry, potentially duplicating orientation-specific filters. This paper introduces four insertable layers that share parameters across orientations and constructs partially or fully rotation-equivariant networks. Across rotationally symmetric datasets, the resulting networks improved performance while using fewer parameters.

Problem
CNNs can learn redundant filters for different orientations, while rotational symmetry is not encoded architecturally as translation symmetry is.
Method
The paper introduces four layers that can be combined with existing CNN architectures to encode cyclic rotational equivariance and share parameters across orientations.
Results
The resulting equivariant networks improved performance on datasets with full rotational symmetry while reducing the number of parameters.
Takeaways & Limitations
Architectural rotation equivariance can improve performance with smaller models on datasets exhibiting rotational symmetry.
Takeaways & Limitations
The current framework focuses on cyclic rotations, while extensions to non-90° rotations and volumetric data remain future work.

Abstract

from arXiv · show

Many classes of images exhibit rotational symmetry. Convolutional neural networks are sometimes trained using data augmentation to exploit this, but they are still required to learn the rotation equivariance properties from the data. Encoding these properties into the network architecture, as we are already used to doing for translation equivariance by using convolutional layers, could result in a more efficient use of the parameter budget by relieving the model from learning them. We introduce four operations which can be inserted into neural network models as layers, and which can be combined to make these models partially equivariant to rotations. They also enable parameter sharing across different orientations. We evaluate the effect of these architectural modifications on three datasets which exhibit rotational symmetry and demonstrate improved performance with smaller models.

1. Introduction

Neural networks encode useful prior knowledge through architecture, with CNN convolutions providing translation equivariance and parameter sharing. This paper extends that architectural approach to rotational symmetry using four layers for scalable, partially or fully rotation-equivariant CNNs.

1. Introduction: CNNs use convolutional layers to encode translation equivariance and pooling layers to provide local translation invariance.Together, these layers support hierarchical feature detectors that become progressively more robust to local variations.
1. Introduction: Convolutional parameter sharing improves capacity use, reduces overfitting, and supports very large images without dramatically increasing parameter counts.
1. Introduction: The proposed framework applies architectural parameter sharing to rotational invariance and equivariance.It introduces four layers that can be combined to build partially or fully rotation-equivariant CNNs.
1. Introduction: The framework is designed to be scalable and easy to implement in practice.

2. Cyclic symmetry

Rotationally repeated patterns can cause CNNs to learn redundant filters. The paper focuses on four grid-aligned rotations, using parameter sharing across transformed inputs to encode cyclic symmetry without interpolation.

2. Cyclic symmetry: CNNs may learn multiple copies of filters for patterns appearing in different orientations, creating redundancy when inputs exhibit rotational symmetry.Encoding rotational symmetry could reduce this redundancy, free model capacity, or reduce parameters and overfitting risk.
2. Cyclic symmetry: Cyclic symmetry restricts rotations to k · 90° for k ∈ {0, 1, 2, 3}, whose sampling grids align without interpolation.These rotations form the cyclic group C4 and can be implemented through cheap transposition and row or column flipping operations.
2. Cyclic symmetry: Dihedral symmetry adds a horizontal flip to the four rotations, yielding eight orientations obtainable without interpolation.
2. Cyclic symmetry: The framework encodes cyclic symmetry by applying one filter to four transformed input copies, producing four feature maps with differing relative input-filter orientations.Equivalently, four transformed filter copies can operate on the unchanged input.

3. Equivariance and invariance

Rotationally symmetric data can require representations that transform predictably, or remain unchanged, when inputs rotate. The paper motivates these distinctions through cyclic transformations and convolutional rotation behavior.

3. Equivariance and invariance: Rotationally symmetric data occurs in biology, astronomy, medicine, and aerial photography, while dihedral symmetry also appears in board configurations such as Go.
3. Equivariance and invariance: Cyclic symmetry is also known as discrete rotational symmetry of order 4.
3. Equivariance and invariance: Convolving with a rotated filter and inversely rotating the result matches convolving an inversely rotated image with the unrotated filter.The caption attributes this equivalence to rotation being distributive with respect to convolution.
3. Equivariance and invariance: A function is equivariant when transformed inputs produce correspondingly transformed outputs, expressed as f(Tx) = T′f(x).
3. Equivariance and invariance: Invariant representations do not change under input rotations and are therefore equivariant, while same-equivariance uses the same transformation on input and output.

4. Encoding equivariance in neural nets

The framework encodes cyclic rotation equivariance through four composable layers that transform, realign, stack, or pool feature maps. These operations support invariant or same-equivariant outputs while sharing parameters across orientations.

Motivation: Data augmentation can encourage rotation invariance, but architectural encoding targets rotation properties directly and avoids relying solely on learned invariance.The paper frames architectural encoding as an alternative to generating predictions over transformed inputs, which increases computation.
Framework: The framework introduces four neural-network layers that alter minibatch size, feature-map count, or both while remaining compatible with surrounding architectures.The operations are designed to work with architectures such as Inception and residual networks.
Cyclic slicing and pooling: Cyclic slicing stacks four rotated copies of each input in one minibatch, while cyclic pooling combines the four pathways with a permutation-invariant function.Slicing increases the minibatch by 4×; pooling reduces it by 4×. Pooling can use averaging or maximum operations.
Cyclic slicing and pooling: Pooling after dense layers yields cyclic rotation invariance, whereas preserving spatial realignment through the output yields same-equivariance.For spatial tasks such as segmentation, realignment preserves the output’s spatial structure and rotation behavior.
Practical constraints: Pooling cannot be followed by convolutional or spatial-pooling layers without relinquishing equivariance, and four-way slicing increases the effective intermediate batch size.The larger effective batch may require reducing the input batch size to avoid significantly slower training.
Cyclic rolling: The roll operation realigns feature maps from all cyclic pathways and stacks them along the feature dimension, increasing feature-map richness through copying.Its construction applies stacking to all cyclic permutations and preserves equivariance across layers.
Cyclic rolling: R is same-equivariant under cyclic permutation, while the resulting parameter sharing can reduce parameters or use equal capacity more effectively.Reducing filters after rolling can keep parameters approximately constant while increasing produced feature maps by a factor of two.

5. Related work

Prior work encodes rotation equivariance through modified probabilistic models, tied or shared filters, learned transformation structure, or explicit transformation estimation. The paper situates its framework alongside these alternatives and concurrent group-equivariant CNNs.

Architectural and probabilistic encoding: Earlier approaches modified Markov random fields and restricted Boltzmann machines, including tied weights, to obtain rotation-invariant or rotation-equivariant representations.These methods encode rotational structure in probabilistic or energy-based models rather than the paper’s CNN-layer framework.
Shared filters and pooling: Fasel and Gatica-Perez rotated inputs, shared filters across orientations, and progressively pooled representations to obtain output invariance.The paper identifies its input-rotation strategy as identical in this respect, while distinguishing its use of a single pooling layer.
Shared filters and pooling: Wu et al. rotated and flipped filters and max-pooled activations, whereas this paper concatenates orientations and prefers pooling only at the output.The stated goal is to preserve global equivariance, which multiple pooling stages would prevent.
Learning transformations: Other methods learn invariance or equivariance from data using grouped filters, transformation-group models, or reduced weight sharing.The paper characterizes learned equivariance as more flexible but requiring more training data.
Explicit transformation estimation: A separate line of work estimates the transformation applied to each input, as in transforming auto-encoders and spatial transformer networks.This strategy has also been investigated for face detection.
Concurrent work: Cohen and Welling’s concurrent group-equivariant CNNs provide a theoretically grounded formalism describing the same type of models constructible with this framework.The paper presents the two approaches as formal descriptions of equivalent model types.

6. Experiments

Experiments on three rotationally symmetric datasets tested cyclic slicing, pooling, and rolling modifications against competitive CNN baselines. The modifications generally preserved or improved performance while reducing parameters or using comparable parameter budgets more effectively.

6.2. Experimental setup: Baseline CNNs followed common architectures, including VGG-inspired 3×3 same convolutions with pooling for Plankton and Galaxies.The baselines achieved competitive estimated Kaggle rankings, while all experiments used Adam, scheduled learning rates, and rotation augmentation.
6.3. Pooling functions: Mean cyclic pooling without a nonlinearity achieved the best Plankton cross-entropy among six tested configurations.The six configurations varied mean, maximum, or RMS pooling and whether ReLU preceded pooling.
6.3. Pooling functions: Cyclic pooling modestly improved performance over the baseline on the Galaxies and Massachusetts buildings datasets.The reported metrics were Galaxies RMSE and Massachusetts buildings AUC, averaged with standard deviations across ten runs.
6.4. Networks with rolling layers: Roll-all 1/4 models significantly reduced parameters across all three datasets while having a limited effect on performance.Comparable baseline 1/2 models were included to test parameter-budget efficiency, with roughly similar training computation because rolling adds minimal cost relative to convolutions.
6.4. Networks with rolling layers: Roll-dense 1/2 slightly improved performance over baseline networks for Plankton and Galaxies while reducing the dominant first dense-layer parameters.Halving the first dense layer and adding a rolling layer doubles the next dense layer’s parameters, but produces a net reduction for the Plankton network.
6.4. Networks with rolling layers: On Massachusetts buildings, roll-all 1/2 achieved better performance than the fully convolutional baseline with roughly the same number of parameters.The authors note that similarly sized models would heavily overfit on the smaller Plankton and Galaxies datasets.

7. Conclusion and future work

The paper presents an easily inserted four-layer framework for rotation-equivariant neural networks. Across rotationally symmetric datasets, the resulting networks improved performance while reducing parameters, while future work targets broader transformation groups and volumetric data.

7. Conclusion and future work: The framework builds rotation-equivariant networks from four layers that can be inserted into existing architectures with no modifications beyond adapting minibatch size.The authors report improved performance on datasets with full rotational symmetry while reducing parameter counts.
7. Conclusion and future work: Future work will test the approach on data-scarce rotational-symmetry domains such as medical imaging, where additional parameter sharing may reduce overfitting.The authors also plan to study non-90° rotations and volumetric data, including the interpolation and realignment complexity those extensions require.