PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Transformer Networks

Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu

arXiv:1506.02025v3cs.CV

TL;DR

CNNs have limited efficient invariance to large spatial transformations, so the paper introduces learnable Spatial Transformer modules that actively warp feature maps; these models achieve state-of-the-art performance across multiple tasks.

  • Problem

    CNN feature maps are not invariant to large input transformations because conventional pooling provides only limited, predefined spatial invariance.

  • Method

    The Spatial Transformer is a differentiable, input-conditioned module that predicts a sampling grid and warps feature maps within existing CNNs.

  • Results

    Spatial transformer models achieve state-of-the-art performance across multiple tasks, reaching 3.6% error on 64×64 Street View House Numbers images versus 3.9% previously.

  • Takeaways & Limitations

    Spatial transformers provide a self-contained end-to-end way for neural networks to model spatial variation and learn pose-normalised representations.

  • Takeaways & Limitations

    In a purely feed-forward network, the number of parallel spatial transformers limits the number of objects the network can model.

Abstract

from arXiv · show

Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.

1 Introduction

CNNs achieve state-of-the-art results across multiple vision tasks, but their spatial invariance is limited. The paper introduces Spatial Transformer modules that learn sample-conditioned spatial transformations within standard neural networks without extra supervision.

  • CNNs have achieved state-of-the-art results in classification, localisation, semantic segmentation, and action recognition.
  • Max-pooling provides some spatial invariance, but small receptive fields limit this property to a deep hierarchy of pooling and convolution.
  • Spatial Transformer modules provide sample-conditioned spatial transformation capabilities within standard neural network architectures, learned for the task without extra supervision.
  • In CNNs, a spatial transformer can crop and scale-normalize relevant image regions, simplifying classification when digit position and size vary across samples.

2 Related Work

Prior work models transformations, studies invariant or equivariant representations, and uses attention or detection for feature selection. The paper frames spatial transformers as a generalization of differentiable attention to arbitrary spatial transformations.

  • Research directions: Related work spans neural-network transformation models, transformation-invariant representations, and attention or detection mechanisms for feature selection.These strands are identified as the central areas of prior research discussed in the section.
  • Transformation models: Earlier models assigned canonical reference frames to object parts or modeled 2D affine transformations in generative models.The generative training scheme used transformed input images as targets and supplied transformations as additional network inputs.
  • Invariant representations: Prior studies analyzed CNN invariance and equivariance through representation relationships and symmetry groups, while other architectures explicitly increased invariance to those groups.These approaches focused on characterizing or designing transformation-related properties of learned representations.
  • Attention and detection: Selective-attention methods used crops, reinforcement learning, differentiable Gaussian kernels, region proposals, or CNN-based salient-region regression.These mechanisms support spatial selection and, in some cases, translation invariance without requiring a differentiable attention mechanism.
  • Attention and detection: The proposed framework can be viewed as a generalization of differentiable attention to any spatial transformation.This positions the framework alongside prior attention mechanisms while extending their spatial scope.

3 Spatial Transformers

Spatial Transformers are differentiable, input-conditioned modules that spatially warp feature maps through a localisation network, grid generator, and sampler. Their parameterised transformations and differentiable sampling enable flexible spatial manipulation and integration into CNNs with low computational overhead.

  • 3 Spatial Transformers: A spatial transformer applies one input-conditioned spatial transformation to a feature map during a single forward pass, using identical warping across channels.The module produces a single output feature map, while preserving spatial consistency between channels.
  • 3 Spatial Transformers: The mechanism comprises a localisation network, a sampling-grid generator, and a sampler that produces the transformed output feature map.The localisation network predicts transformation parameters from the input feature map before the grid and sampling operations are applied.
  • Transformation: An affine transformation requires only 6 parameters and supports cropping, translation, rotation, scale, and skew of the input feature map.More general differentiable forms include plane projective transformations with 8 parameters, piecewise affine transformations, and thin plate splines.
  • Sampling: Differentiable sampling allows loss gradients to flow through sampled values and grid coordinates back to the transformation parameters and localisation network.Bilinear sampling provides the required partial derivatives, while sub-gradients handle discontinuities and GPU implementation can restrict computation to each kernel’s support region.
  • Spatial Transformer Networks: The resulting module can be inserted at any point and in any number within a CNN, with very little time overhead and possible speedups in attentive models after downsampling.Multiple transformers can operate at increasing depths or in parallel to transform increasingly abstract representations or multiple objects and parts.

4 Experiments

Experiments show that spatial transformers improve robustness to diverse distortions, achieve state-of-the-art SVHN recognition, and improve fine-grained bird classification. Across tasks, learned transformations provide spatial invariance and focus processing on informative image structure.

  • 4.1 Distorted MNIST: On distorted MNIST, spatial-transformer networks outperform their corresponding fully connected or convolutional baselines across rotation, RTS, projective, and elastic-warping distortions.The experiments compare FCN and CNN baselines with spatial-transformer variants; elastic warping can be destructive and non-invertible.
  • 4.1 Distorted MNIST: 0.5% and 0.6% error are achieved by ST-CNN models on RTS distortions, compared with 0.8% for a CNN and 0.8% for ST-FCN.ST-CNN models consistently outperform ST-FCN models because their max-pooling layers provide additional invariance.
  • 4.1 Distorted MNIST: Thin plate spline transformations are the most powerful for elastically deformed digits, reshaping inputs toward prototype instances without overfitting simpler data.All spatial-transformer models transform inputs toward a standard upright digit pose.
  • 4.2 Street View House Numbers: 3.6% error on 64×64 SVHN images surpasses the previous state of the art of 3.9%, while ST-CNN reaches 3.9% on 128×128 images versus 4.5% previously.The models recognize sequences of one to five digits in real-world images with variable scale and spatial arrangement.
  • 4.3 Fine-grained bird classification: 84.1% accuracy on CUB-200-2011 exceeds the strong baseline’s 82.3% by 1.8% using two or four parallel spatial transformers.The transformers act on the input image to capture discriminative image parts for fine-grained classification.

5 Conclusion

The paper introduces spatial transformers as self-contained, differentiable modules that explicitly transform features within neural networks. They can be inserted into networks and learned end-to-end without changing the loss function, yielding accuracy gains across multiple tasks.

  • 5 Conclusion: Spatial transformers are self-contained modules that perform explicit spatial transformations of neural-network features.They can be dropped into an existing network to expand how it models data.
  • 5 Conclusion: The module is learned end-to-end without requiring changes to the network’s loss function.This enables spatial transformation behavior to be learned as part of standard network training.
  • 5 Conclusion: Spatial transformers improve accuracy relative to CNNs across multiple tasks.The passage describes CNNs as a strong baseline and reports gains using spatial transformers.

A Appendix

The appendix reports additional experiments on spatial transformers, including multi-object MNIST addition, semi-supervised co-localisation, and a 3D extension. It also provides expanded experimental details for Sections 4.1–4.3.

  • Appendix: The appendix covers MNIST addition with multiple objects, semi-supervised co-localisation, a 3D extension, and expanded details for the main experiments.These topics are presented in Sections A.1–A.6, corresponding to the experiments in Sections 4.1–4.3.

A.1 MNIST Addition

The MNIST addition experiment tests whether parallel spatial transformers can independently handle two digits undergoing random rotation, scale, and translation. Two transformers enable an FCN to focus on separate digits and achieve the lowest error among similarly sized models.

  • A.1 MNIST Addition: The task requires summing two digits presented in separate 42 × 42 channels, with each digit independently transformed by random rotation, scale, and translation.The network receives only the sum label as supervision.
  • A.1 MNIST Addition: Two parallel spatial transformers process both input channels, concatenate their two 2-channel outputs into a 4-channel feature map, and feed the result to an FCN.This architecture is denoted 2×ST-FCN, with approximately matched parameter counts across models.
  • A.1 MNIST Addition: 47.7% is the FCN’s minimum error, while max-pooling CNNs reduce error to 14.7% on this complex task.These results establish the baseline difficulty before adding spatial transformers.
  • A.1 MNIST Addition: 18.5% error results from adding one spatial transformer, which focuses the FCN on a single region containing both digits.A single transformer improves the FCN but does not independently separate the digits.
  • A.1 MNIST Addition: Using two transformers lets each focus on one digit despite receiving both channels, with co-adaptation producing stable digit representations in two of four output channels.The table reports that 2×ST-FCN achieves the lowest error among the approximately parameter-matched models.

A.2 Co-localisation

The co-localisation experiment uses spatial transformers to localise a shared but unknown object class from image sets without labels or location ground truth. On MNIST, the method achieves 100% accuracy without clutter and 75–93% with clutter.

  • Task: Co-localisation finds bounding boxes for a common unknown object class using only sets of images, without object labels or location ground truth.The task is framed as semi-supervised spatial localisation.
  • Method: The method trains a spatial transformer as the localiser with a triplet loss that brings correctly localised object crops closer than random crops in embedding space.The transformation parameters are interpreted as the object bounding box and optimized with stochastic gradient descent.
  • Results: By SGD step 180, the model has correctly localised three digits in the optimization-dynamics example.The example tracks predictions for three of 100 dataset images.
  • Experimental setup: Experiments use translated MNIST digits and translated, cluttered digits containing 16 random 6 × 6 distractor crops.Each 28 × 28 digit is placed randomly in an 84 × 84 black background, and datasets contain 100 distorted samples of one digit class.
  • Results: 100% accuracy is achieved for correctly localising MNIST digits without clutter, while 75–93% of digits are correctly localised with clutter.Correct localisation requires predicted and ground-truth bounding boxes to have intersection-over-union greater than 0.5.

A.3 Higher Dimensional Transformers

Spatial Transformers extend naturally beyond 2D to learnable 3D affine warping, including intelligent projection of 3D data into 2D for downstream classification. The paper demonstrates this approach on randomly translated and rotated 3D MNIST voxel digits, trained end-to-end using only classification labels.

  • 3D extension: Spatial Transformers extend to higher-dimensional inputs, including a module capable of performing 3D affine transformations.The 3D transformer can be inserted into neural networks to warp data in 3D space, where the third dimension may represent space or time.
  • 3D MNIST classification: The 3D MNIST classifier is trained end-to-end with only classification supervision.The demonstrated input is an extruded random MNIST digit placed inside a 60 × 60 × 60 voxel volume.
  • 3D-to-2D projection: A 3D transformer can flatten its output across one dimension, creating a 2D projection that reduces subsequent processing dimensionality and complexity.This projection enables later network layers to operate on a 2D representation of the original 3D data.
  • 3D MNIST classification: On 3D object classification, the network learns to project randomly translated and rotated extruded MNIST digits into a representation classifiable by subsequent layers.The inputs are 3D voxel volumes containing randomly placed digits, and the task is to output the digit class.

A.4 Distorted MNIST Details

The distorted MNIST experiments define several transformation-based datasets and evaluate fully connected, convolutional, and spatial-transformer networks under standardized architectures and training settings.

  • Data: The rotated dataset samples MNIST digit rotations uniformly between −90◦ and +90◦.This section expands the distorted MNIST experiments in Sect. 4.1.
  • Data: The RTS dataset combines random ±45◦ rotations, scaling factors between 0.7 and 1.2, and random placement in a 42×42 image.Each transformation is sampled using uniform distributions.
  • Data: The projected dataset scales digits between 0.75 and 1.0 and perturbs each corner using a zero-mean normal distribution with 5 pixel standard deviation.
  • Networks: All networks use rectified linear nonlinearities and softmax classifiers, with specified fully connected or convolutional architectures.CNNs use 9 × 9 and 7 × 7 convolutional layers, each followed by 2 × 2 max-pooling with stride 2.
  • Training: Networks are trained with SGD for 150k iterations using batch size 256, base learning rate 0.01, no weight decay, and no dropout.The learning rate decreases tenfold every 50k iterations; localization regression layers initialize to the identity transform.

A.5 Street View House Numbers Details

The SVHN experiments use a specified CNN architecture and validation-selected training setup, with localisation-network weights initialized for identity transformations. Results are averaged across two full training runs with different random seeds.

  • Training setup: 400k iterations use SGD with batch size 128, base learning rate 0.01 reduced tenfold every 80k iterations, weight decay 0.0005, and dropout 0.5 except in the first convolutional and localisation layers.Hyperparameters are selected using a 5k-image validation set drawn from the training data.
  • CNN architecture: The CNN comprises convolutional, max-pooling, and fully connected layers, with rectified linear units following each weight layer.The listed network includes fully connected layers with 3072 units each near its output.
  • Initialization and reporting: Two full training runs use different random seeds, and the reported accuracy is the average obtained by a single model.Localisation-network final regression layers are initialized to regress the identity transform using zero weights and identity-transform bias.

A.6 Fine Grained Classification Details

The fine-grained classifier uses spatial transformers as differentiable attention to discover discriminative object parts from image labels. Its Inception-based architecture supports multiple sampling scales and achieves 27.1% top-1 error on the ILSVRC validation set with a single crop.

  • Attention mechanism: Each transformer predicts an attention window’s (x,y) location, fixes scale at 50% of the image, and samples a 224 × 224 crop.The sampled crops are processed by the classification architecture.
  • Classification performance: 27.1% top-1 error on the ILSVRC validation set is achieved with a single image crop using batch-normalised Inception pre-trained on ILSVRC.Training used single-scale images resized so that the smallest side was 256.
  • Localisation architecture: The shared localisation network truncates Inception before its final pooling layer, producing 7 × 7 spatial features with 1024 channels.Three added weight layers predict the transformations, beginning with a 1 × 1 convolution reducing channels from 1024 to 128.
  • Findings and limitations: The architecture was not optimised for parameter count or computation time, but it quantitatively and qualitatively discovered meaningful object parts using image labels alone.More complex location-and-scale and affine transformations produced similar results, with severe over-fitting attributed to the 6k-image, 200-class training set.
  • Input resolutions: The model evaluates 224 × 224 and 448 × 448 transformer inputs, while both settings output 224 × 224 crops for compatibility with crop description networks.A fixed 2× downscaling layer keeps the localisation network input at 224 × 224 in the larger-input setting, which better suits small-scale crops.
Loading 1506.02025v3…