PaperlayerView exact revision on arXiv

Source-linked AI summary

Polar Transformer Networks

Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, Kostas Daniilidis

arXiv:1709.01889v3cs.CV

TL;DR

CNNs provide translation equivariance, but broader learned equivariance has been limited. PTN combines spatial transformation with canonical log-polar coordinates to model rotation and scale, achieving strong results on rotated MNIST and SIM2MNIST and extending to 3D through cylindrical coordinates.

  • Problem

    CNNs are inherently translation-equivariant, while learned equivariance to transformations beyond translation has concentrated mainly on rotation.

  • Method

    PTN predicts a polar origin, applies a differentiable log-polar transform, and classifies the resulting representation in a three-stage end-to-end network.

  • Results

    PTN achieves state-of-the-art performance on rotated MNIST and SIM2MNIST and improves rotated-MNIST performance by a large margin while outperforming tested methods on SIM2MNIST.

  • Takeaways & Limitations

    PTN learns representations invariant to translation and equivariant to rotation and dilation, with the approach extended to 3D object classification through cylindrical coordinates.

  • Takeaways & Limitations

    The 3D coordinate extensions can achieve scale equivariance, but they do not provide equivariance to arbitrary 3D rotations.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are inherently equivariant to translation. Efforts to embed other forms of equivariance have concentrated solely on rotation. We expand the notion of equivariance in CNNs through the Polar Transformer Network (PTN). PTN combines ideas from the Spatial Transformer Network (STN) and canonical coordinate representations. The result is a network invariant to translation and equivariant to both rotation and scale. PTN is trained end-to-end and composed of three distinct stages: a polar origin predictor, the newly introduced polar transformer module and a classifier. PTN achieves state-of-the-art on rotated MNIST and the newly introduced SIM2MNIST dataset, an MNIST variation obtained by adding clutter and perturbing digits with translation, rotation and scaling. The ideas of PTN are extensible to 3D which we demonstrate through the Cylindrical Transformer Network.

1 INTRODUCTION

PTN extends learned CNN equivariance beyond translation by combining spatial transformation with canonical log-polar coordinates. It predicts an object center, transforms the input, and learns representations equivariant to rotation and dilation.

  • PTN combines Spatial Transformer Network ideas with canonical coordinates to achieve translation invariance and rotation- and dilation-equivariance.
  • The three-stage architecture predicts the object center, transforms the input into log-polar coordinates, and classifies the resulting representation.The polar origin is learned as the centroid of a heatmap predicted by a fully convolutional network.
  • In log-polar coordinates, rotations around the origin become vertical shifts, while dilations become horizontal shifts.The distances between corresponding lines encode the rotation angle or scale factor.
  • PTN captures rotations and dilations at arbitrary precision without the challenging parameter regression used by STN.Its scope is limited to global deformations.
  • PTN reports state-of-the-art performance on rotated MNIST and the newly introduced SIM2MNIST dataset.

2 RELATED WORK

Related work spans hand-crafted invariant descriptors, integral transforms, steerable representations, scattering, orbit pooling, and CNN architectures that constrain filters or use filter orbits. PTN belongs to the filter-orbit vein while using canonical log-polar coordinates to include scaling.

  • The Fourier-Mellin transform is equivariant to rotation and scale, while its modulus is invariant.
  • Steerability represents responses to group actions through interpolation from a finite filter basis, with frameworks covering rotation, scale, and non-Abelian groups.
  • The scattering transform composes rotated and dilated wavelets, producing translation-invariant and locally rotation- and scale-equivariant coefficients.
  • Orbit pooling achieves transformation invariance by pooling feature maps over transformed inputs, but requires forward and backward passes for each orbit element.
  • CNN equivariance methods constrain filter structure or use equivariant filter orbits; PTN follows the latter vein and adds scaling through log-polar coordinates.
  • For 3D objects, prior approaches commonly use transformed-input augmentation or pooling, whereas PTN's extension explicitly represents transformations through cylindrical coordinates.

3 THEORETICAL BACKGROUND

The paper develops equivariance for similarity transformations by reparameterizing translation, rotation, and scale into canonical coordinates. Its PTN pipeline learns an object center, centers the image, applies a log-polar transform, and uses planar convolutions to obtain predictable dilated-rotation responses.

  • 3.1 GROUP EQUIVARIANCE: Group-convolutions generalize translational convolution and are provably equivariant when defined with an appropriate group measure.
  • 3.2 EQUIVARIANCE IN SIM(2): SIM(2) combines translations with dilated rotations, decomposing each transformation into t ∈ R2 and r ∈ SO(2) × R+.
  • 3.2 EQUIVARIANCE IN SIM(2): SIM(2) equivariance is achieved by learning the transformation center, shifting the image, and converting it to canonical coordinates.
  • 3.2 EQUIVARIANCE IN SIM(2): The centered image is transformed into log-polar coordinates, where planar convolutions implement SO(2) × R+ group-convolutions.
  • 3.2 EQUIVARIANCE IN SIM(2): The resulting representation shifts predictably under input rotation and scaling, as illustrated by the corresponding heatmaps in canonical coordinates.
  • 3.2 EQUIVARIANCE IN SIM(2): The complete network builds translational convolutions, estimates a centroid, recenters the image, applies a log-polar transform, and processes it with a second convolutional network.

4 ARCHITECTURE

PTN uses a learned heatmap centroid to set a polar-transform origin, then converts the image into a representation processed by a conventional classifier CNN. Differentiable centroid computation and sampling enable end-to-end learning, while angular wrap-around padding and origin augmentation address training and representation-specific issues.

  • Architecture: PTN connects a polar origin predictor and a conventional fully convolutional classifier through a polar transformer module.The origin predictor estimates the transform center from the input image before classification.
  • Polar origin predictor: The origin predictor outputs a single-channel heatmap whose centroid provides the polar-transform origin.This avoids directly regressing coordinates and supplies two coordinates to the transformer.
  • Polar origin predictor: Using the heatmap centroid instead of argmax gives nonzero gradients with respect to all heatmap points, making the latent origin learnable.The centroid is used because argmax gradients are zero almost everywhere during backpropagation.
  • Polar transformer module: The polar transformer performs differentiable image sampling to produce a log-polar representation from the predicted origin and input image.Its source sampling coordinates are defined from the transform origin and target regular grid.
  • Polar transformer module: Wrap-around padding is applied along the angular dimension because input rotations become vertically shifting, periodic patterns in polar coordinates.The top and bottom rows are padded from one another rather than with zeros.
  • Training: Randomly shifting the regressed polar origin during training improves robustness at little computational cost relative to rotating input images.The performance gains from this augmentation are quantified in Table 5.

5 EXPERIMENTS

Experiments evaluate PTN on rotated and transformed MNIST variants, visualize its learned invariance and equivariance, and extend the approach to 3D voxel classification. PTN generally performs strongly, with results highlighting the value of predicting the transformation origin and representing rotations and dilations as shifts.

  • Rotated MNIST: PTN variants are compared under restricted and unrestricted settings, including differences in network size and rotation augmentation.The restricted setting uses smaller networks without rotation augmentation; the unrestricted setting imposes no such restrictions.
  • Rotated MNIST: Harmonic Networks slightly outperform PTN among restricted approaches, but require almost 4x more training time because complex-valued convolutions are costlier.
  • Rotated MNIST: Most PTN-B variants outperform the current state of the art in unrestricted settings, especially with CCNN combination and/or test-time augmentation.Test-time augmentation makes performance 8x slower.
  • Other MNIST variants: On MNIST variants with mild deformations, PTN mostly matches STN, whereas on SIM2MNIST its stronger transformations and 5x smaller training set expose the weakness of PCNN.The authors report that PTN achieves the best performance by a large margin on SIM2MNIST and is more efficient in parameters and training time.
  • Visualization: Figure 4 shows predicted origins rejecting clutter while preserving translation invariance and rotation- and dilation-equivariance through the polar transform and deeper convolutional layers.A 180° rotation becomes a half-height vertical feature-map shift, while 2.4× dilation becomes a rightward shift.
  • Extension to 3D object classification: For ModelNet40, the 3D extension predicts a rotation axis and applies channel-wise polar transforms, producing cylindrical representations where vertical-axis rotations become horizontal translations.The reported method outperforms published voxel-based methods without test-time augmentation, although multi-view methods generally perform better.

6 CONCLUSION

PTN produces translation-invariant representations that remain equivariant to rotations and dilations, while extending the approach to 3D through cylindrical coordinates.

  • PTN outputs representations invariant to translation and equivariant to rotations and dilations.
  • The network learns translation and uses canonical coordinates so group convolutions capture scaling and rotation without fully connected pose regression.
  • PTN improves state-of-the-art performance on rotated MNIST and outperforms tested methods on SIM2MNIST.
  • The authors expect PTN to apply to problems where varying orientations and scales hinder conventional CNNs.

A ARCHITECTURES DETAILS

The experiments compare conventional, polar, spatial-transformer, and related architectures, with small and big variants plus a cylindrical 3D transformer.

  • The implementation section introduces the architectures used for comparison before specifying their variants and cylindrical-network design.
  • The comparison includes conventional CNNs, Polar CNNs with fixed image-center origins, and Spatial Transformer Networks.
  • Architecture suffixes distinguish network scale and augmentation configurations.
  • Small networks use seven 20-filter blocks with one subsampling round, while big networks use eight blocks with increasing filter counts.
  • The cylindrical transformer uses four 2D axis-prediction blocks and eight 3D convolutional blocks, totaling approximately 1M parameters.

B DATASET DETAILS

The appendix describes rotated MNIST variants, challenging ROTSVHN samples with cluttering side digits, and SVHN results showing PTN's advantage under rotation.

  • Rotated MNIST contains 28×28 handwritten digits rotated through 360°, with 10k training, 2k validation, and 50k test images.
  • ROTSVHN is created from larger SVHN images so digits can be rotated without introducing cropping artifacts.
  • Visible neighboring digits in ROTSVHN challenge both classification and PTN origin prediction.
  • Table 4 reports SVHN classification performance, including variants that remove the indistinguishable 6 and 9 classes.

C SVHN EXPERIMENTS

The SVHN experiments apply PTN with a ResNet32 classifier to rotated real-world digit images, finding greater robustness to rotation despite slightly worse unperturbed performance.

  • Experiments use SVHN and the introduced ROTSVHN dataset, which rotates digits extracted from larger images without artifacts.
  • PTN-ResNet32 uses eight residual convolutional layers as an origin predictor followed by a ResNet32 classifier.
  • Removing 6s and 9s addresses their usual indistinguishability in house numbers and is denoted by a minus dataset suffix.
  • The conventional ResNet error increases from 2.09% to 5.39% under rotation, whereas PTN error increases from 2.85% to 3.96%.
  • PTN is slightly worse on unperturbed data but clearly superior when rotations are present.

D ABLATION STUDY

The ablation study evaluates wrap around padding, polar origin augmentation, and training-time rotation augmentation in PTN-B on Rotated MNIST. Removing any operation reduces performance, indicating that all three contribute to lower error.

  • Removing wrap around padding, polar origin augmentation, or training-time rotation augmentation consistently reduces PTN-B performance on Rotated MNIST.The ablation removes one operation at a time.
  • Wrap around padding contributes to reducing error in the PTN-B model.
  • Polar origin augmentation during training contributes to reducing error in the PTN-B model.
  • Rotation augmentation during training contributes to reducing error in the PTN-B model.
Loading 1709.01889v3…