Source-linked AI summary
Scale-Equivariant Steerable Networks
Ivan Sosnovik, Michał Szmaja, Arnold Smeulders
TL;DR
CNNs are translation-equivariant but lack built-in handling for scale changes, motivating a dedicated scale-equivariant architecture. The paper develops steerable-filter scale convolution and scale-equivariant network blocks, then reports state-of-the-art supervised results on MNIST-scale and STL-10.
Problem
CNNs lack an embedded mechanism for scale changes, while some approaches preserve scale equivariance by breaking translation equivariance or rely on interpolation.
Method
The paper develops scale-equivariant convolutional networks using steerable filters, scale convolution, and scale-equivariant nonlinearities and pooling.
Results
The proposed approach outperforms other scale-equivariant and locally scale-invariant CNNs and achieves state-of-the-art supervised results on MNIST-scale and STL-10.
Takeaways & Limitations
Scale-equivariant steerable networks provide equivariant and invariant model variants while avoiding image resampling and interpolation artifacts during training.
Abstract
from arXiv · showhide
The effectiveness of Convolutional Neural Networks (CNNs) has been substantially attributed to their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transformations. In this work, we pay attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-convolution and generalize other common blocks to be scale-equivariant. We demonstrate the computational efficiency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-art results on MNIST-scale dataset and on STL-10 dataset in the supervised learning setting.
1 INTRODUCTION
The paper addresses CNNs’ lack of built-in scale handling by developing scale-equivariant networks with steerable filters. The approach supports arbitrary discrete scales without interpolation and achieves comparable convolutional computation time.
- Motivation: CNNs provide translation equivariance but lack a dedicated mechanism for handling scale changes.Existing coordinate-based approaches can preserve scale equivariance only by breaking translation equivariance.
- Approach: The paper develops a theory of scale-equivariant networks with steerable filter parametrization and derives scale-equivariant convolution.The method also generalizes common network blocks and compares against scale-equivariant and locally scale-invariant methods.
- Properties: The model supports scale transformations with arbitrary discrete scale factors rather than only integer or image-grid-tailored scales.
- Properties: It avoids image resampling during training, producing deep scale-equivariant representations without interpolation artifacts.
- Efficiency: The algorithm combines tensor expansion with 2-dimensional convolution and has the same computation time as a general CNN with a comparable filter bank.
2 PRELIMINARIES
The preliminaries define scale transformations, equivariance, and the scale-translation group, then show why standard convolution is not scale-equivariant. Steerable filters reparameterize filter scaling to simplify the analysis and implementation.
- Motivation: Scale transformations are incorporated directly into CNNs to make predictions equivariant to scaling transformations.
- Scaling and Convolution: For a downscaled input, convolution can be rewritten as convolution with a correspondingly upscaled filter followed by downscaling.
- Scaling and Convolution: Standard convolution is therefore not scale-equivariant.
- Steerable Filters: Steerable filters reparameterize scaled filters so that scaling acts on their parameters, while linear combinations remain steerable.
- Scale-Translation Group: The scale-translation group combines discrete scales and translations, whose noncommuting operations make their order consequential.
- Group Convolution: Group-equivariant convolution evaluates the inner product between a function and a transformed filter using the group’s invariant measure.
3 SCALE-EQUIVARIANT MAPPINGS
The paper constructs scale-equivariant layers from group convolution on the scale-translation group and steerable filters. Pooling operations then provide controlled transitions between scale-equivariant and scale-invariant representations.
- Scale Convolution: Scale convolution is derived by splitting the scale-translation group integral, selecting Haar measures, and applying steerable-filter properties.
- Scale Convolution: The resulting convolution supports multiple input and output channels through sums of spatial convolutions across scale-indexed filters.
- Scale Convolution: The scale convolution is equivariant to transformations from the scale-translation group.
- Nonlinearities: Point-wise nonlinearities such as ReLU preserve scale equivariance because they commute with scaling transformations.
- Pooling: Maximum projection over scales preserves scale equivariance while discarding explicit scale information.
- Pooling: Spatial pooling retains scale information while producing spatially scale-invariant representations, and combining both pooling operations yields scale invariance.
4 IMPLEMENTATION
The implementation discretizes the scale group and represents filters with a fixed steerable basis, reducing scale-equivariant convolution to tensor operations and standard 2D convolution. Scale-axis interactions can be omitted or modeled across neighboring scales.
- Filter Basis: Direct implementation is infeasible because the scale group is infinite and filters must be optimized in functional space.
- Filter Basis: The method truncates the scale group to N_S scales, discretizes translations, and represents filters as trainable combinations of basis functions.
- Filter Basis: The experiments use fixed 2D Hermite-polynomial basis functions with 2D Gaussian envelopes, precomputed for all scales.
- Convolution Modes: For T-to-H convolution, expanded filters are applied with standard 2D convolution, producing feature maps whose spatial positions contain S-component vectors.
- Convolution Modes: H-to-H convolution may use one scale or an extent K_S across neighboring scales, with the latter providing interscale interaction.
- Implementation Illustration: Figure 1 illustrates steerable-filter construction and scale convolution, where a 3-channel input with four H scales produces two output channels with four scales without scale interaction.
5 RELATED WORK
Prior work extends CNNs toward scale handling through rescaling, scale-space constructions, and steerable filters, but differs in equivariance, interscale interaction, and computational cost. SESN uses steerable filters to support fast scale convolution while retaining flexibility for equivariant and invariant models.
- Group-equivariant networks: Group-equivariant CNN research has largely focused on rotational symmetries, while scale symmetry has received comparatively less attention.The paper treats scale symmetry as fundamentally different from the rotations emphasized in much prior work.
- Scale-space methods: Scale-space methods analyze images alongside downscaled versions and connect scale symmetry to image structure and formation.This tradition predates deep scale-equivariant CNNs and includes foundational work on image scale-space.
- CNN scale methods: SiCNN and SI-ConvNet use image or filter rescaling, with SI-ConvNet selecting maximum responses across scales to obtain local scale invariance.These methods differ in whether intermediate layers interact across scales and rely on interpolation or rescaling operations.
- CNN scale methods: SEVF relies on image rescaling, DSS uses filter dilation but is restricted to integer downscale factors, and SS-CNN uses scale-steerable filters without discussing equivariance.The cited approaches therefore differ in speed, supported scale factors, and whether equivariance is explicitly addressed.
- SESN comparison: SESN uses steerable filters for fast scale convolution without limiting flexibility and supports both equivariant and invariant model constructions.Table 1 compares SESN with SiCNN, SI-ConvNet, SEVF, DSS, and SS-CNN using criteria including interscale interactions and grid-aligned scales.
6 EXPERIMENTS
Experiments evaluate SESN’s numerical equivariance, efficiency, and classification performance against scale-equivariant and scale-invariant alternatives. SESN remains accurate for deep networks, trains faster than rescaling-based methods, and achieves the strongest reported results on MNIST-scale and STL-10.
- Efficiency: SESN trains several times faster than methods that rely on image rescaling.The speed advantage is attributed to the algorithm proposed for SESN implementation.
- Equivariance error: Discretization keeps equivariance error below 6% for networks with 50 layers, although error increases when inputs are downscaled more than 16 times.Interscale interaction adds extra error, so the experiments use either no scale interaction or interaction across 2 scales.
- MNIST-scale: SESN significantly outperforms other methods across all four MNIST-scale regimes, with Vector versions improving further over Scalar versions.The global architectures are held constant across rows, highlighting the role of how scale convolution is performed.
- STL-10: On STL-10, SI-ConvNet and SS-CNN perform worse than the baseline, while SESN-B improves both WideResNet and DSS.SESN-C performs significantly worse than SESN-A and SESN-B because interscale interaction causes high equivariance error.
- STL-10: The proposed method achieves a new state-of-the-art result on STL-10 in the supervised learning setting.The comparison includes Harm WRN, previously reported as having the lowest error.
7 DISCUSSION
The paper presents SESN as a scale-equivariant framework with potential relevance to computer vision under changing object-camera distances. It especially identifies siamese visual tracking as a promising application direction.
- Discussion: The paper concludes that SESN achieves state-of-the-art results on MNIST-scale and STL-10 in supervised learning.Its theory provides exact scale-equivariant mappings and a discretized implementation.
- Discussion: SESN is proposed as a potential fit for autonomous-vehicle vision because changing object distances create substantial scale variation.The paper specifically highlights siamese visual tracking, where equivariance to principal transformations is important.
A PROOF OF EQUIVARIANCE
The proof establishes that scale-convolution is equivariant to translations and scale transformations, using the semidirect-product structure of the scale-translation group.
- Translation equivariance: Scale-convolution is first shown to be equivariant to translations.This is stated as the initial equivariance property proved for Equation 6.
- Scale equivariance: The proof then establishes scale equivariance for scale-convolution.The argument directly addresses transformations of the scale variable.
- Scale-translation group: The semidirect-product property combines translation and scale equivariance into equivariance under the scale-translation group.Equation 13 expresses the composed transformation through successive group actions.
B TIME PERFORMANCE
The paper compares training time for scale-equivariant and locally scale-invariant methods, using average epoch time at two input resolutions. It attributes observed efficiency differences to the computational structure of the filters and algorithms.
- Average epoch time is measured for training inputs at 28 × 28 and 56 × 56 resolutions.The experiments use the setups from Section 6.2 and one Nvidia GeForce GTX 1080Ti GPU.
- Methods using image rescaling during training have significantly worse time performance than methods using steerable filters or filter dilation.
- The proposed method outperforms SS-CNN by a wide margin despite similar filter sizes and comparable parameter counts.The passage attributes this difference to the algorithm proposed in Section 4.
- DSS can be slightly faster because each convolution uses fewer FLOPs, while dilated filters are sparse and steerable filters are dense.
D.1 MNIST-SCALE
Table 5 records model capacity and scale coverage for the MNIST-scale experiments. It reports convolutional channels, fully connected units, and the number of scales used by each model.
- Table 5 compares models by convolutional-layer channels, fully connected units, and number of scales.
D.2 STL-10
Table 6 records the channel counts and scale counts used by models in the STL-10 experiments. Channel counts are reported up to the widening factor.
- Table 6 compares models by convolutional-block channels and number of scales, with channel counts reported up to the widening factor.