Source-linked AI summary
Locally Scale-Invariant Convolutional Neural Networks
Angjoo Kanazawa, Abhishek Sharma, David Jacobs
TL;DR
ConvNets lack an explicit mechanism for scale, so recognizing patterns at multiple scales can require redundant filters and more training data. SI-ConvNets share filters across scales and pool aligned responses; on modified MNIST, they outperform ConvNets in reported comparisons, including for unfamiliar scales.
Problem
ConvNets lack an explicit scale mechanism, so covering multiple scales requires separate filters that consume resources, increase overfitting risk, and respond only to scales seen during training.
Source: §1 · pp. 1–2Method
SI-ConvNets apply shared filters to transformed inputs at multiple scales, align the responses, and pool over scales to form locally scale-invariant representations.
Source: §1 · p. 2, §3.1 · p. 4Results
SI-ConvNets outperform ConvNets across reported aspects, including classification with less training data and robustness to unfamiliar scales.
Source: §1 · p. 2, §5 · p. 8, §4.1.3 · p. 8Takeaways & Limitations
The authors conclude that sharing weights across scales can yield more efficient feature learning with the same parameter count and reduced chances of overfitting.
Source: §5 · p. 8The stated scale-invariance qualification assumes patterns share the same center; when centers shift, the output is similar at a different location.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Convolutional Neural Networks (ConvNets) have shown excellent results on many visual classification tasks. With the exception of ImageNet, these datasets are carefully crafted such that objects are well-aligned at similar scales. Naturally, the feature learning problem gets more challenging as the amount of variation in the data increases, as the models have to learn to be invariant to certain changes in appearance. Recent results on the ImageNet dataset show that given enough data, ConvNets can learn such invariances producing very discriminative features [1]. But could we do more: use less parameters, less data, learn more discriminative features, if certain invariances were built into the learning process? In this paper we present a simple model that allows ConvNets to learn features in a locally scale-invariant manner without increasing the number of model parameters. We show on a modified MNIST dataset that when faced with scale variation, building in scale-invariance allows ConvNets to learn more discriminative features with reduced chances of over-fitting.
1 Introduction
ConvNets handle patterns at different locations but lack an explicit mechanism for scale, so recognizing a pattern across scales can require redundant filters and more training data. The paper proposes SI-ConvNets, which share filters across scales to learn locally scale-invariant features without increasing the number of parameters.
1 Introduction
- Recognizing patterns across scales with separate filters uses resources, makes networks harder and slower to train, and increases overfitting risk.Even with more training data, the network responds only to scales seen during training, and separate filters do not share training samples.Source: §1 · pp. 1–2
- SI-ConvNets apply shared filters at multiple scales and pool their responses to produce locally scale-invariant features with traditional layer-output dimensionality.The paper reports better classification performance with less training data on modified MNIST with digits at multiple scales.Source: §1 · p. 2
2 Background
The background reviews ConvNets and prior ways to address scale variation, distinguishing the paper’s layer-level scale-invariant feature learning from multi-scale approaches that combine outputs or learn filters independently.
2 Background
- Prior approaches include transformation-invariant RBMs and tiled ConvNets, while other multi-scale ConvNets either learn independent models or apply each layer only at the original scale.The paper contrasts these approaches with explicitly shared feature detectors across scales within each layer.Source: §2 · p. 2
- Farabet et al. apply tied-weight ConvNets across a Laplacian pyramid for scene parsing, then align and concatenate scale responses before fully connected layers.The paper instead pools responses at each spatial location in each layer for locally scale-invariant features used in image classification.Source: §2 · pp. 2–3
2.1 Convolutional Neural Networks
- A ConvNet is a supervised feed-forward multilayer architecture whose layers learn increasingly complex feature detectors and whose final layer classifies or regresses.The network is jointly optimized by stochastic gradient descent with back-propagation.Source: §2.1 · p. 3
- ConvNets convolve trainable local filters across image positions, producing feature maps that pass through a nonlinear activation and may undergo spatial pooling.Weight sharing across locations reduces trainable parameters compared with using distinct weights at each location.Source: §2.1 · p. 3
3 Scale-Invariant Convolutional Neural Network
The proposed SI-ConvNet builds scale handling into convolution layers: shared filters are applied across transformed inputs, responses are aligned and pooled, and the resulting representation retains the standard layer output size. The paper also describes the transformation cost, a shared-center qualification, and how gradients are propagated.
3.1 Forward Propagation
- The scale-invariant layer applies the same filters to multiple transformed versions of the input, aligns the resulting feature maps, and max-pools responses over scales at each spatial location.Pooling produces a locally scale-invariant representation while retaining the output size of a standard convolution layer.Source: §3.1 · p. 3, Figure 1 · p. 4
- For patterns with the same center, a single detector can respond to different scales; shifted centers yield similar responses at different locations.The latter case is described as a shift-equivariant representation.Source: §3.1 · pp. 3–4
- Pooling aligned responses lets one detector recognize the same pattern at multiple scales, as illustrated by inputs with differently sized “V” patterns.The final response at each location is the maximum activation, with the winning scale shown in the illustration.Source: §3.1 · p. 4
- Using n scales is analogous to increasing feature maps n-fold without increasing the number of learned parameters, allowing a more expressive model without increasing overfitting chances.The analogy is between convolving a pyramid with one filter and convolving one image with filters of different sizes.Source: §3.1 · p. 4
- The scale transformation uses bilinear interpolation, and its coefficients can be precomputed, but applying it increases the number of convolution operations.The increase is dominated by the largest scale factor and the step sizes between scales.Source: §3.1 · p. 4
3.2 Backward Propagation
- Because the layer uses linear and max operations, its gradients can be computed by modifying back-propagation.Scale-pooling gradients use argmax indices, while transformation gradients pass through the bilinear coefficients.Source: §3.2 · p. 5
4 Experiments
On MNIST-Scale, SI-ConvNet outperforms ConvNet and hierarchical ConvNet while keeping the number of parameters fixed. Across measurements of invariance, training-data and feature-map variation, and scale variation, it performs better or is more robust than ConvNet.
4 Experiments
- The experiments compare SI-ConvNet with ConvNets using matched hyperparameters and architecture except for the convolution layers, on modified MNIST with randomly scaled digits.The modified MNIST-scale dataset uses 28×28 grayscale images with scale factors sampled from U(0.3, 1).Source: §4 · pp. 5–6
4.1 MNIST-Scale
- SI-ConvNet outperforms ConvNet and hierarchical ConvNet by more than 10% on MNIST-Scale while keeping the number of parameters fixed.The reported relative error difference is 10% for SI-ConvNet, comparable to 9.8% for SI-RBM.Source: Table 1 · p. 6, §4.1 · p. 6
4.1.1 Measuring Invariance
- SI-ConvNets produce more scale-invariant features than ConvNets when invariance is measured under image scalings from 0.3 to 1.2 in 0.1 steps.The measure is the average score of the top 20% of neurons; higher scores indicate greater invariance relative to selectivity.Source: §4.1.1 · pp. 6–7, Figure 3 · p. 6
4.1.2 Effect of training data and number of parameters
- SI-ConvNets consistently achieve lower test error than ConvNets as feature-map counts and training-data amounts vary, with the performance gap narrowing as either increases.The experiments report test error on 10,000 images.Source: Figure 4 · p. 7, §4.1.2 · p. 7
4.1.3 Robustness to Unfamiliar Scales and Scale Variation
- SI-ConvNet reduces relative error by 25% on average and by 20% and 47% at test scales 0.4 and 1.6, respectively, versus ConvNet.Training scales are sampled from a Gaussian centered at 1; the test scales 0.4 and 1.6 are about two standard deviations from the mean.Source: Figure 5 · p. 8, §4.1.3 · p. 8
- With parameter count and training samples fixed, SI-ConvNets consistently outperform ConvNets as the scale range in training and test data increases, and their error grows more slowly.The scale factors are sampled from a uniform distribution whose range is increased.Source: Figure 5 · p. 8, §4.1.3 · p. 8
5 Conclusion
The proposed architecture learns locally scale-invariant features while preserving the parameter count of traditional ConvNets. Experiments show that SI-ConvNets outperform ConvNets in several respects.
5 Conclusion
- SI-ConvNets learn locally scale-invariant feature representations by sharing weights across scales and locations and pooling detector responses across scales, while keeping the parameter count equal to traditional ConvNets.Scale-invariance is built into each convolution layer independently.Source: §5 · p. 8
A Back-propagation
The section rewrites the scale-invariant convolution layer’s forward operations as matrix-vector multiplications to align its back-propagation with an ordinary multilayer network. It describes how the error signal is distributed across scales and propagated through the layer’s transformations and weights.
A Back-propagation
- Bilinear interpolation represents the spatial transformation as a matrix whose rows each contain four nonzero interpolation coefficients.The matrix maps between the original and transformed input dimensions.Source: A Back-propagation · p. 10
- The forward-pass notation encodes convolution with a Toeplitz matrix and uses transformation matrices and their inverses to scale inputs and align convolution responses.The inverse transformation matrix aligns responses for each scale.Source: A Back-propagation · p. 10
- Back-propagation applies the activation derivative, uses argmax indices to undo scale-wise max-pooling, and propagates errors through the weights and initial spatial transformation.Errors from the initial spatial transformation are accumulated to complete propagation through the layer.Source: A Back-propagation · p. 10
B Time Analysis
The section analyzes the additional convolution work required by a scale-invariant layer by summing the work over its scaled inputs. It treats the quadratic image-size terms as a geometric series and gives the sum for the experiment’s scale settings.
B Time Analysis
- A scale-invariant layer computes linear combinations over inputs at multiple sizes, from skn down to the smaller scaled images.The analysis compares this work with a traditional layer, which computes local kernel combinations O(n^2) times.Source: B Time Analysis · p. 10
- The sum of the quadratic terms across the scaled inputs forms a geometric series.The terms include s^2kn^2, n^2, and s^2(k−t)n^2.Source: B Time Analysis · p. 10
- For s = 1.26, t = 5, and k = 2, the geometric series sums to 10.These are the values used in the experiments.Source: B Time Analysis · pp. 10–11
C Experimental Details
The experiments use a three-layer network with convolutional and pooling stages followed by a fully connected layer and logistic regression. Inputs are normalized, and first-layer kernel size and weight decay are tuned using a validation set.
C Experimental Details
- Inputs are preprocessed by subtracting the training mean and scaling pixel values to the [0, 1] range.All experiments use this preprocessing.Source: C Experimental Details · p. 11
- The base network has a convolutional first layer, a second convolutional layer, a 150-unit fully connected layer, and a 10-unit logistic regression output.The convolutional layers have 36 and 64 feature maps, respectively, and are followed by max-pooling.Source: C Experimental Details · p. 11
- The first-layer kernel size and weight decay are tuned on a 10k validation set before the final network is trained on all training data.For experiments on 40×40 images, the selected first-layer kernel size is 9×9.Source: C Experimental Details · p. 11