PaperlayerView exact revision on arXiv

Source-linked AI summary

Warped Convolutions: Efficient Invariance to Spatial Transformations

João F. Henriques, Andrea Vedaldi

arXiv:1609.04382v5cs.CV

TL;DR

CNNs efficiently encode translation equivariance, but useful spatial transformations also include scaling and rotation, and prior generalized convolutions can be costly. This paper uses a fixed image warp followed by standard convolution to obtain equivariance to classes of transformations, reporting encouraging pose-estimation results across vehicle and face settings.

  • Problem

    CNNs exploit translation, but other useful image transformations include scaling and rotation, while prior generalized convolutions can carry high computational cost.

    Source: §1 · p. 1

  • Method

    The method pre-warps the image with a fixed transformation-dependent warp, then applies standard convolution to achieve equivariance for suitable transformation groups.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Warped convolutions show encouraging results in vehicle pose estimation and face pose estimation, with the face experiments reporting better exploitation of the data’s natural 3D rotation equivariance than the tested non-equivariant models.

    Source: §6.2 · p. 7, §7 · p. 8, §6.3 · p. 8

  • Takeaways & Limitations

    The construction reuses highly optimized convolution blocks for image tasks involving a range of transformations, including continuous ones.

    Source: §1 · p. 2, §7 · p. 8

    The efficient construction is restricted to transformation groups that are Abelian and have only two parameters.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) are extremely efficient, since they exploit the inherent translation-invariance of natural images. However, translation is just one of a myriad of useful spatial transformations. Can the same efficiency be attained when considering other spatial invariances? Such generalized convolutions have been considered in the past, but at a high computational cost. We present a construction that is simple and exact, yet has the same computational complexity that standard convolutions enjoy. It consists of a constant image warp followed by a simple convolution, which are standard blocks in deep learning toolboxes. With a carefully crafted warp, the resulting architecture can be made equivariant to a wide range of two-parameter spatial transformations. We show encouraging results in realistic scenarios, including the estimation of vehicle poses in the Google Earth dataset (rotation and scale), and face poses in Annotated Facial Landmarks in the Wild (3D rotations under perspective).

1. Introduction

Encoding data-relevant invariances can regularize learning, and natural images exhibit useful transformations beyond translation. The paper proposes fixed image warps that let standard convolutions handle continuous transformations while retaining efficient convolutional implementations.

1. Introduction

  • Encoding data-present invariances in an architecture provides regularization that reduces the training data required for learning.Source: §1 · p. 1
  • Natural image statistics are also largely invariant to isotropic scaling and rotation, while some applications call for invariances beyond translation.Source: §1 · p. 1
  • Earlier generalized convolutions incur high computational burden from repeated warping, while filter-basis methods often use coarsely discretized transformations.The passage notes examples such as flips and 90° rotations.Source: §1 · p. 1
  • Fixed image warps reduce convolution for certain two-parameter transformation groups to standard convolution, preserving its computational advantages.The warp is constant for images and filters and can be implemented with bilinear resampling.Source: §1 · pp. 1–2

2. Generalizing convolution

The section extends ordinary convolution from discrete image grids to continuous inputs and then to transformations in a group. Under an additive exponential-map parameterization, group convolution can be represented as standard convolution in a warped space, with an Abelian-group restriction simplifying the construction.

2.1. Discrete and continuous convolution

  • Continuous convolution expresses the image-filter operation as an integral, allowing intermediate input values to be obtained by interpolation and supporting continuous deformations.The continuous functions are defined on real-valued two-dimensional inputs; bilinear interpolation is one way to obtain intermediate values.Source: §2.1 · p. 2

2.2. Convolution on groups

  • Group convolution replaces translated filters with transformations from a group, and its equivariance means transforming the input corresponds to transforming the output.The group-convolution definition uses a Haar integral and measure.Source: §2.2 · p. 2

2.3. From groups to images

  • When the transformation group acts on the image domain, group-domain functions can be related to images, with group translation corresponding to warping the image.The image domain is typically chosen as the orbit of a pivot point under the group.Source: §2.3 · p. 3

2.4. Standard convolutions with exponential maps

  • For a smooth, bijective, additive exponential-map parameterization, group convolution becomes standard convolution in warped space; additivity requires an Abelian group.The authors focus mainly on the two-dimensional parameter case and identify commutativity as the restriction that simplifies the calculation.Source: §2.4 · p. 3

3. Warped convolutions

Warped convolution implements certain group convolutions by pre-warping the input and filter, then applying standard convolution. Its layer is equivariant to the supported transformations, while using a single image warp instead of many.

3. Warped convolutions

  • Certain group convolutions can be implemented by pre-warping the input image and filter, then applying standard convolution.The warp is fixed by the transformations rather than the image content.Source: §3 · p. 3, §3.2 · p. 4

3.1. Warped convolution layer

  • Warping an input by h = exp(u) translates the convolution result by u, preserving the equivariant relationship.The layer takes an image and learned filter coefficients, and outputs a new image on vector space V.Source: §3.1 · p. 3

3.2. Implementation and intuition

  • The implementation replaces the many warps used by group convolution with a single warp, followed by standard convolution.The warp grid is sampled from transformation parameters and can be computed once offline for reuse across images.Source: §3.2 · p. 4

4. Discussion

Warped convolution can be viewed as filtering in warped image space, with filter reflection and Jacobian weighting absorbed into learned filters in practice. It retains efficient standard-convolution implementations under restrictions on the transformation group.

Warped convolution

  • In practice, filter reflection and Jacobian weighting can be absorbed into a learned filter, avoiding those calculations.The Jacobian determinant accounts for spatial stretching and shrinking under the nonlinear map.Source: Warped convolution · p. 4

4.2. Efficiency vs. generality

  • Warped convolution can use efficient standard-convolution implementations, but requires an Abelian transformation group with only two parameters.The supported class includes transformations considered in prior works.Source: §4.2 · p. 4
  • Figure 1 illustrates warps for horizontal scaling, planar rotation, and 3D rotation around the vertical axis.A small translation in the warped space corresponds to each transformation after mapping back.Source: Figure 1 · p. 5
  • Direct computation over a discrete transformation group has O(|G|2) complexity, potentially O(N 4) when |G| is O(N 2).With an M×M filter whose support is smaller than the image, the stated practical complexity may reduce to O(N 2M 2).Source: §4.2 · p. 5
  • Direct group-convolution computation can lose regular local sampling and access to faster routines such as Winograd or FFT.Spatially varying interpolation makes high-performance implementation difficult, particularly on GPUs.Source: §4.2 · p. 5

5. Examples of spatial transformations

The section gives warp constructions for scale and aspect ratio, planar scale and rotation, and perspective yaw and pitch. These examples show how different transformations can be represented in warped image spaces.

5.1. Scale and aspect ratio

  • Bounding-box size prediction motivates equivariance to horizontal and vertical scale, represented as scale and aspect ratio.The warp uses exponential output coordinates to preserve additive scale parameters, with the image domain restricted to the positive quadrant.Source: §5.1 · p. 6

5.2. Scale and rotation (log-polar warp)

  • Planar rotation and scaling around the origin produce a warp grid that corresponds exactly to the log-polar domain.The domain excludes the origin, which cannot reach other points under rotation or scaling; log-polar correlation inspired this generalization.Source: §5.2 · p. 6

5.3. 3D sphere rotation under perspective

  • For perspective yaw and pitch, the warp assumes rotated 3D points remain on a sphere under a simplified camera and world model.The resulting grid wraps the 2D image around a discrete sphere mesh, so translation in warped space corresponds to moving between mesh vertices.Source: §5.3 · p. 6

6. Experiments

The experiments pair warped CNNs with soft-argmax pose prediction and evaluate them on vehicle and face pose tasks. Warped CNN performs best on most reported comparisons while matching CNN-baseline runtime on the vehicle task.

6.1. Architecture

  • The architecture warps the input, applies a standard CNN, and uses soft argmax to select the strongest pose-space response before calibrating predictions with learnable scale and bias.The soft argmax returns two spatial coordinates, and training minimizes L1 loss against ground-truth pose.Source: §6.1 · pp. 6–7

6.2. Google Earth

  • Warped CNN achieves the best vehicle results except for scale, where Dieleman et al. performs better; it matches CNN-baseline runtime while Dieleman et al. is 16× slower.The comparison covers angular and scale error on the Google Earth validation set.Source: §6.2 · p. 7, Table 1 · p. 7

6.3. Faces

  • On AFLW face pose estimation, Warped CNN better exploits natural 3D rotation equivariance than the same-CNN FC and STN baselines.The task evaluates yaw and pitch angular error; the dataset contains about 25K faces, with 24,384 retained after excluding faces with yaw above 90 degrees.Source: §6.3 · pp. 7–8, Table 2 · p. 8

7. Conclusions

Warped convolutions reuse optimized convolution blocks to achieve equivariance to transformations beyond image translation, including 3D transformations. The conclusion emphasizes their simplicity and flexibility while identifying support for more than two parameters as future work.

7. Conclusions

  • A simple input warp lets optimized translation-equivariant convolution blocks exhibit equivariance to other operators, including 3D transformations.The warp uses off-the-shelf deep-network components.Source: §7 · p. 8
  • Compared with other works, warped convolutions rely on optimized convolution routines and flexibly handle many continuous transformations.Source: §7 · p. 8
  • Generalizing warped convolutions to support more than two parameters is identified as a fruitful direction for future work.Source: §7 · p. 8

A. Spatial transformation for 3D sphere rotation under perspective

The appendix constructs the perspective yaw-and-pitch warp by mapping image points onto a sphere, rotating their spherical coordinates, and projecting them back into image space.

A. Spatial transformation for 3D sphere rotation under perspective

Related papers

This paper cites

Loading 1609.04382v5…