Source-linked AI summary
Harmonic Networks: Deep Translation and Rotation Equivariance
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, Gabriel J. Brostow
TL;DR
CNNs lack reliable patch-wise rotational equivariance, while augmentation is inexact and does not guarantee equivariance throughout the network. H-Nets hard-bake continuous rotation equivariance by constraining filters to circular harmonics, achieving state-of-the-art results on rotated-MNIST and boundary detection while incurring higher per-filter computational cost.
Problem
CNNs do not reliably transform feature vectors predictably under input rotations, and augmentation does not ensure exact patch-wise equivariance at every layer.
Method
H-Nets constrain CNN filters to steerable circular harmonics, hard-baking patch-wise 360° rotational equivariance into deep feature representations.
Results
H-Nets achieved state-of-the-art results on rotated-MNIST classification and boundary detection, including state-of-the-art performance for non-pretrained networks.
Takeaways & Limitations
The results support 360° rotational equivariance as both possible and useful, with good generalization using less or less augmented training data.
Takeaways & Limitations
Discrete sampling can shift or remove local centers of equivariance under striding or pooling, so max-pooling was not used in the experiments.
Abstract
from arXiv · showhide
Translating or rotating an input image should not affect the results of many computer vision tasks. Convolutional neural networks (CNNs) are already translation equivariant: input image translations produce proportionate feature map translations. This is not the case for rotations. Global rotation equivariance is typically sought through data augmentation, but patch-wise equivariance is more difficult. We present Harmonic Networks or H-Nets, a CNN exhibiting equivariance to patch-wise translation and 360-rotation. We achieve this by replacing regular CNN filters with circular harmonics, returning a maximal response and orientation for every receptive field patch. H-Nets use a rich, parameter-efficient and low computational complexity representation, and we show that deep feature maps within the network encode complicated rotational invariants. We demonstrate that our layers are general enough to be used in conjunction with the latest architectures and techniques, such as deep supervision and batch normalization. We also achieve state-of-the-art classification on rotated-MNIST, and competitive results on other benchmark challenges.
1. Introduction
CNNs provide patch-wise translation equivariance through shared weights, but rotations lack a predictable feature-space transformation. H-Nets hard-bake rotational equivariance into local representations using circular-harmonic filters.
- Equivariance links input transformations to corresponding feature transformations, but CNN rotations do not necessarily produce meaningful or predictable feature changes.
- Equivariant intermediate features preserve local pose information that invariant representations discard, supporting richer object representations.
- Shared weights give CNNs translational equivariance and reduce learnable parameters, motivating the analogous hard-baking of rotation.
- Aggressive rotation augmentation improves generalization but is inexact, misses local equivariances, and leaves feature maps less interpretable while learning redundant rotated filters.
- H-Nets constrain filters to circular harmonics, whose steerability represents rotated filters with finite linear combinations and yields predictable rotationally equivariant feature maps.
2. Related Work
Prior approaches encode rotation through discrete filter or feature-map copies, rotated inputs, or learned transformations. H-Nets are positioned within this broader effort to obtain rotational equivariance without relying exclusively on such strategies.
- Existing rotational-equivariance methods commonly introduce filter or feature-map copies at different rotations, but none has become standard practice.
- Steerable filters: Steerable filters represent arbitrary rotated filters as finite linear combinations of basis filters, avoiding multiple learned copies and keeping memory requirements constant.
- Hard-baked transformations in CNNs: Related work distinguishes global from patch-wise equivariance and approaches that rotate filters from those that rotate feature maps.
- Hard-baked transformations in CNNs: Some methods copy filters across 90° rotations and flips, while others use larger sets of rotated copies or handcrafted filters for more rotations.
- Hard-baked transformations in CNNs: Input-rotation fusion methods provide global equivariance but do not produce interpretable feature maps, while feature-map copying extends equivariance within CNNs.
- Learning generalized transformations: Learned-transformation approaches are appealing, but some prior demonstrations remain limited to shallow architectures, motivating hard-baked transformations for interpretability and reliability.
3. Problem analysis
H-Nets use circular harmonics to make feature responses transform predictably under continuous rotations. Their rotation orders determine feature transformations, and network paths must preserve a consistent equivariance condition.
- H-Nets constrain CNN filters to circular harmonics so their feature representations are equivariant to 360° rotations.
- Continuous domain feature maps: The analysis uses continuous feature-map domains before converting the results back to discrete implementation.
- Equivariance: Equivariance means that transforming the input and then mapping features agrees with mapping features and then applying an associated feature transformation.
- The Complex Circular Harmonics: Circular-harmonic filters have learned radial profiles and phase offsets, producing complex-valued responses whose phases are tied explicitly to rotation angles.
- Rotational Equivariance: A response to a rotated input equals the unrotated response multiplied by e^(imθ), so rotation order m determines the feature-space transformation.
- Arithmetic and the Equivariance Condition: Rotation orders add under chained cross-correlations, while magnitude-only nonlinearities and same-order summation preserve equivariance.
- Arithmetic and the Equivariance Condition: The equivariance condition requires every path’s rotation-order sum to equal the target output order, preventing entanglement of differently rotating responses.
- Deep architecture: H-Nets organize feature maps into constant-rotation-order streams, using cross-correlations whose orders match differences between streams.
4. Method
H-Nets construct deep feature maps with predictable rotation behavior by organizing circular-harmonic responses into separate rotation-order streams. Their discrete implementation uses sampling and resampling operations that preserve the relevant equivariance structure, while practical choices constrain pooling and filter representation.
- Harmonic Network architecture: Circular-harmonic cross-correlations add rotation orders, so H-Nets maintain separate streams to prevent responses with different orders from entangling.Streams use order-zero cross-correlations and nonlinearities internally, while cross-correlations between streams use the difference between their rotation orders.
- Harmonic Network architecture: Responses of the same rotation order can be summed, reducing memory while preserving the stream organization.
- Discrete implementation: Sampling and cross-correlation commute, allowing continuous-domain H-Nets to be implemented on pixel-domain data; Gaussian blur and resampling adapt the representation to discrete grids.Sampling and point-wise nonlinearities also commute with the deep feature mapping, while Gaussian resampling supports filters defined in the polar domain.
- Harmonic Network architecture: H-Nets replace regular CNN filters with radially reweighted, phase-shifted circular harmonics, producing responses equivariant to input rotations with order m.Complex nonlinearities act only on feature-map magnitudes, preserving rotational equivariance; analogous batch-normalization operations are also available.
- Harmonic Network architecture: Each H-Net layer contains feature maps of different rotation orders that transform predictably under input rotation, achieving 360° equivariance with finite computation.
- Discrete implementation: Discrete sampling creates local centers of equivariance at feature-map patch locations, while max-pooling is avoided because it makes those centers input-dependent and nonlinear.Strided cross-correlation or average pooling can ablate or shift the centers of equivariance.
- Discrete implementation: Complex cross-correlations can be implemented using four real cross-correlations, and circular-harmonic filters can be grid-resampled for current deep-learning frameworks.
- Computational trade-offs: The representation increases cross-correlation cost in exchange for continuous rotational equivariance.The paper analyzes this cost in terms of the number of multiplications for standard and harmonic networks.
5. Experiments
Experiments evaluate H-Nets on rotated-MNIST classification and BSD500 boundary detection, while probing rotational stability, filter usage, data efficiency, and feature-map structure.
- Benchmarks: H-Nets achieve a new state-of-the-art on rotated MNIST, reducing test error by 26%.
- Benchmarks: H-Nets outperform non-pretrained neural-network baselines on BSD500 boundary detection.
- Benchmarks: H-DSN uses only 5% of the parameters of the compared architecture while delivering superior BSD500 performance.
- Model Insight: Magnitude responses remain nearly flat as input rotation changes for harmonic orders m∈{0,1,2}.Periodic small fluctuations are attributed to inexact anti-aliasing.
- Model Insight: H-Net filters show few reoriented copies, no blank filters, and broad phase usage, indicating low redundancy and full use of phase information.The visualizations show filters detecting boundaries and feature maps encoding edges, corners, objects, negative space, and outlines.
- Model Insight: Hard-baked rotation equivariance makes H-Nets more data-efficient than regular CNNs on rotated MNIST.Regular CNNs trained with data augmentation still perform worse because they learn global invariance rather than local equivariances at each layer.
6. Conclusions
The paper concludes that circular harmonics enable local equivariance to continuous 360° rotations with few parameters, supporting strong performance and interpretable feature representations.
- Conclusions: H-Nets provide local equivariance to patch-wise translation and continuous 360° rotation by hard-baking rotation into the architecture.The approach can also be implanted onto other architectures.
- Conclusions: Circular harmonics provide full rotational equivariance using few parameters, supporting good generalization with less or less-augmented training data.
- Conclusions: The reported disadvantage is higher per-filter computational cost, balanced against a more expressive representation.
- Conclusions: H-Nets set state-of-the-art results on rotated-MNIST classification and on non-pretrained boundary detection.
- Conclusions: Feature-map interpretability improves because the network specifies how representations transform under input-image rotations.Deep feature visualizations show consistent orientation coding across multiple layers.
- Future Work: Future work may hard-bake additional transformations, potentially extending the approach to 3D.
Abstract
The supplementary material includes proofs and derivations of circular harmonics’ rotational equivariance properties, together with parameter-count calculations for network architectures.
- Abstract: The paper provides proofs and derivations for rotational equivariance properties and demonstrates parameter-count calculations across network architectures.
A. Equivariance properties
The appendix proves that cross-correlation with circular harmonics is a 360°-rotation-equivariant feature transform.
- Equivariance properties: The appendix proves the 360°-rotation equivariance of cross-correlation with circular harmonics and develops related arithmetic properties.
A.1. Equivariance of the Circular Harmonics
Circular-harmonic cross-correlation is rotationally equivariant, and this property composes across chained filters and magnitude-based nonlinearities. Feature-map summation also preserves rotational order when the summed maps share that order.
- Equivariance of rotational cross-correlation: Rotational cross-correlation is equivariant about the origin, relating image rotations to predictable feature-map transformations.The construction introduces rotation and translation operators and defines rotational cross-correlation about the image origin.
- Equivariance of rotational cross-correlation: A rotated signal cross-correlated with circular harmonic filter Wm produces the zero-rotation response multiplied by the phase shift e^(imθ).The filter is written as Wm=R(r)e^(i(mφ+β)), and the phase factor is denoted ψθ.
- Chained cross-correlations: Chained cross-correlations add the rotation orders of their individual filters.The result is established for two filters and extended to longer chains by induction.
- Nonlinearities: Point-wise nonlinearities acting only on feature-map magnitudes preserve rotational equivariance.Because the nonlinearity changes magnitude while retaining the phase component, the transformed output remains equivariant.
- Summation of feature maps: Point-wise summation of feature maps with the same rotation order yields a new feature map with that same order.The result follows because the summed expression depends on the order and rotation angle while phase offsets remain fixed.
B. Number of parameters
The appendix breaks down how parameter counts are computed for the network architectures used in the experiments. It distinguishes cross-correlations, pooling, and 1×1 cross-correlations in the architecture diagrams.
- Network architecture: Network diagrams identify cross-correlations, pooling, and 1×1 cross-correlations using red, blue, and green boxes, respectively.Pooling is average pooling for H-Nets and max pooling for regular CNNs.
B.1. Standard CNN
Standard CNN parameter counts depend on input channels, output channels, filter size, biases, and batch-normalization factors. H-Net parameters instead use radial profiles and per-filter phase offsets.
- Standard CNN: A standard CNN layer with i input channels, o output channels, and k×k weights has iok^2+o learnable parameters including bias.With batch normalization, the count becomes iok^2+2o.
- H-Net: H-Net filters learn radial-profile elements determined by concentric equal-distance rings and one phase offset per filter.A 5×5 example has six radial-profile terms and one phase offset.