PaperlayerView exact revision on arXiv

Source-linked AI summary

Spherical CNNs

Taco S. Cohen, Mario Geiger, Jonas Koehler, Max Welling

arXiv:1801.10130v3cs.LGstat.ML

TL;DR

Spherical images require models that account for rotations rather than planar translations. The paper develops rotation-equivariant spherical correlations and evaluates FFT-based spherical CNNs, finding stable, accurate computations and near state-of-the-art results across 3D recognition and molecular energy regression.

  • Problem

    Planar convolutions are unsuitable for spherical signals because spherical patterns move under 3D rotations rather than translations, motivating rotation-aware models.

  • Method

    The paper defines S2 and SO(3) cross-correlations and computes them efficiently with a generalized FFT-based correlation algorithm.

  • Results

    Spherical CNNs effectively generalize across rotations and achieve near state-of-the-art results in competitive 3D model recognition and molecular energy regression challenges.

  • Takeaways & Limitations

    The results provide empirical support for spherical CNNs as a task-agnostic approach to rotation-invariant learning problems without excessive feature engineering and task-tuning.

  • Takeaways & Limitations

    The discretized implementation is not exactly equivariant, so discretization artifacts may affect equivariance and the effectiveness of weight sharing across many layers.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have become the method of choice for learning problems involving 2D planar images. However, a number of problems of recent interest have created a demand for models that can analyze spherical images. Examples include omnidirectional vision for drones, robots, and autonomous cars, molecular regression problems, and global weather and climate modelling. A naive application of convolutional networks to a planar projection of the spherical signal is destined to fail, because the space-varying distortions introduced by such a projection will make translational weight sharing ineffective. In this paper we introduce the building blocks for constructing spherical CNNs. We propose a definition for the spherical cross-correlation that is both expressive and rotation-equivariant. The spherical correlation satisfies a generalized Fourier theorem, which allows us to compute it efficiently using a generalized (non-commutative) Fast Fourier Transform (FFT) algorithm. We demonstrate the computational efficiency, numerical accuracy, and effectiveness of spherical CNNs applied to 3D model recognition and atomization energy regression.

1 INTRODUCTION

Spherical CNNs adapt convolutional architectures to spherical signals by replacing planar translations with rotations and using generalized Fourier methods for efficient computation. The paper addresses interpolation and computational challenges while evaluating the approach on rotation-invariant learning problems.

  • Spherical CNNs seek to detect patterns on a sphere regardless of their 3D rotation, analogous to position-independent pattern detection in planar CNNs.
  • Replacing filter translations with rotations yields spherical correlation on SO(3), requiring higher-layer feature maps to be signals on the rotation group rather than the sphere.
  • Planar projections distort spherical signals, so translation-based convolution cannot faithfully emulate spherical rotations.
  • Spherical CNNs face interpolation challenges because no perfectly symmetrical spherical grid provides a simple one-pixel rotation, while naive SO(3) correlation costs O(n6).
  • Generalized Fourier theory supports efficient S2 and SO(3) correlations through generalized FFT algorithms.
  • The paper evaluates mathematical fidelity and practical utility through discretization analysis, rotation-invariant classification, 3D shape recognition, and molecular energy regression.

2 RELATED WORK

The paper extends group-equivariant CNNs to spherical data by targeting the continuous, non-commutative rotation group SO(3). It also uses generalized Fourier transforms for fast group correlation, distinguishing the work from prior discrete-group or non-equivariant approaches.

  • The work is presented as the first CNN architecture equivariant to a continuous, non-commutative group, SO(3).
  • Earlier symmetry-based networks were generally limited to discrete groups, while other spherical-image methods did not define an equivariant architecture.
  • Generalized FFT algorithms are used to perform cross-correlations efficiently on the sphere and rotation group.

3 CORRELATION ON THE SPHERE AND ROTATION GROUP

The paper develops spherical and rotation-group correlations by replacing planar translations with rotations and formalizing the required operators and inner products. The resulting spherical correlation produces an SO(3)-indexed output and supports equivariance across network layers.

  • Spherical Correlation: Spherical correlation replaces planar filter translations with rotations represented by the three-dimensional manifold SO(3).The paper introduces the construction by analogy with planar correlation while accounting for the different space of spherical motions.
  • Mathematical Setup: A spherical signal is modeled as a continuous function on S2 with K channels, while three-dimensional rotations act through SO(3).S2 is parameterized by spherical coordinates, and SO(3) by ZYZ-Euler angles.
  • Mathematical Setup: The rotation operator applies a function to R^-1x, enabling filters and spherical signals to transform consistently under rotations.The inverse rotation gives the operator its composition property, L_R L_R′ = L_RR′.
  • Spherical Correlation: The spherical correlation evaluates an inner product between the input signal and a filter rotated by R, yielding a function on SO(3).Unlike the conventional spherical convolution, this definition does not restrict the filter to circular symmetry about the Z axis.
  • Rotation Group Correlation: The rotation operator is generalized to SO(3)-valued signals by composing rotations through matrix multiplication.The analogous term R^-1Q represents composition of rotations rather than rotation of a point on S2.
  • Equivariance: Equivariance ensures that correlating a rotated input produces the correspondingly rotated output for both spherical and rotation-group correlation.The derivation uses the rotation operator's unitarity and applies to both correlation types.

4 FAST SPHERICAL CORRELATION WITH G-FFT

The paper accelerates spherical and rotation-group correlations with generalized Fourier analysis and GFFT algorithms. In implementation, transforms combine ordinary FFTs over angular coordinates with linear contractions using precomputed basis-function samples.

  • Generalized Fourier Analysis: The ordinary Fourier theorem reduces planar correlation from naive O(n^2) spatial computation to FFT-based O(n log n) computation.This motivates the analogous generalized Fourier treatment for functions on S2 and SO(3).
  • Generalized Fourier Analysis: The generalized Fourier transform projects functions on S2 or SO(3) onto orthogonal basis functions, including spherical harmonics and Wigner D-functions.The transform is paired with a corresponding fast generalized FFT algorithm.
  • Generalized Fourier Analysis: The maximum frequency b is the bandwidth and is related to the resolution of the spatial grid.The inverse transform is defined for SO(3), with an analogous construction for S2.
  • Fourier Theorems: The SO(3) correlation satisfies a Fourier theorem in which transformed signals are combined by block-matrix multiplication with the filter transform's adjoint.For S2 correlation, the transformed signals combine through an outer product to produce SO(3)-Fourier coefficients.
  • Implementation: Figure 2 computes spherical correlation by transforming the signal and locally supported filter, block-wise tensoring them, summing input channels, and inverse transforming.Because the filter is locally supported, matrix multiplication via a DFT is faster than using an FFT for that filter.
  • Implementation: The SO(3) FFT first applies a 2D translational FFT over α and γ, then contracts the β axis with precomputed Wigner-d samples.The contraction is implemented as a custom GPU kernel because the Wigner-d shape depends on l.
  • Implementation: S2-FFTs similarly apply an FFT over α and a linear contraction over β using precomputed Legendre functions.The paper provides an implementation for these generalized transforms and makes its code available online.

5 EXPERIMENTS

The experiments assess numerical equivariance and apply spherical CNNs to rotated-image classification, 3D shape recognition, and molecular energy regression. Results show manageable discretization error, rotation robustness, strong SHREC17 performance, and improved QM7 regression over several baselines.

  • 5.1 EQUIVARIANCE ERROR: The discretized SO(3) correlation has manageable equivariance error across the resolutions of interest, although error grows with resolution and layer count.With ReLU activations, the error is higher but remains flat, indicating that feature-map rotation rather than network depth causes the observed error.
  • 5.2 ROTATED MNIST ON THE SPHERE: Spherical MNIST tests compare planar and spherical CNNs under non-rotated and randomly rotated training and test conditions.The spherical dataset includes digits projected onto the sphere, with one version additionally applying random rotations.
  • 5.2 ROTATED MNIST ON THE SPHERE: The planar CNN performs well only in the non-rotated regime, whereas the spherical CNN remains largely unaffected by rotations.When trained on non-rotated data and tested on rotated data, the planar CNN performs no better than random chance; the spherical CNN still performs very well.
  • 5.3 RECOGNITION OF 3D SHAPES: For SHREC17, S2CNN classifies randomly rotated 3D models across 55 categories using six-channel ray-cast spherical representations.The dataset contains 51300 models, and the representation is discretized on a Driscoll-Healy grid with bandwidth b = 128.
  • 5.3 RECOGNITION OF 3D SHAPES: On SHREC17, S2CNN ranks third for precision and F1@N and is runner-up on each other reported metric.The authors interpret this as strong empirical support despite using a task-agnostic architecture and lossy input representation; the network contains ≈1.4M parameters.
  • 5.4 PREDICTION OF ATOMIZATION ENERGIES FROM MOLECULAR GEOMETRY: For QM7 atomization-energy regression, the spherical learned representation outperforms all kernel-based approaches and an MLP trained on sorted Coulomb matrices.Only an MLP trained on randomly permuted Coulomb matrices performs better, but sufficient random-permutation sampling grows exponentially with molecule size N.

6 DISCUSSION & CONCLUSION

The paper presents Spherical CNNs as rotation-aware models for spherical signals and reports stable, accurate correlations with near state-of-the-art results on 3D recognition and molecular energy regression.

  • The paper defines S2 and SO(3) cross-correlations and implements a Generalized FFT-based correlation algorithm.The contribution includes both theoretical analysis and an efficient implementation.
  • Numerical results confirm the stability and accuracy of the Generalized FFT-based correlation algorithm, including in deep networks.
  • Further improvements for intrinsically volumetric tasks like 3D model recognition may come from extending the framework beyond SO(3) to the roto-translation group SE(3).
  • Omnidirectional vision is identified as a compelling future application as omnidirectional sensors become more prevalent in drones, robots, and autonomous cars.The paper notes that little omnidirectional image data is currently available in public repositories.

APPENDIX A: PARAMETERIZATION OF AND INTEGRATION ON S2 AND SO(3)

The appendix parameterizes points on S2 and rotations in SO(3), emphasizing that the sphere is a quotient of the rotation group and enabling Fourier analysis on S2 through SO(3).

  • SO(3) rotations are parameterized with ZYZ Euler angles, with α, β, and γ ranging over their stated angular intervals.Z and Y denote rotations around the Z and Y axes.
  • The sphere’s point parameterization uses the north pole as the reference point for rotations and coset representatives.
  • The sphere is parameterized as the quotient S2 = SO(3)/SO(2), where SO(2) consists of rotations around the Z axis that leave the north pole invariant.A spherical point corresponds to a coset of SO(2) in SO(3).
  • The appendix introduces normalized Haar measures for the sphere, SO(2), and SO(3), using the quotient structure to relate the measures.The relation dR = dx dh reflects the quotient structure.
  • Functions on S2 can be represented as γ-invariant functions on SO(3), allowing the S2 Fourier transform to be obtained from the SO(3) Fourier transform.

APPENDIX B: CORRELATION & EQUIVARIANCE

The appendix defines spherical correlation through rotated filters, establishes equivariance, and contrasts this construction with a spherical convolution whose output and integration differ.

  • The defined correlation operation is equivariant under rotations.
  • The compared spherical convolution produces a function on the sphere, whereas the paper’s cross-correlation produces a function on SO(3).
  • The compared convolution integrates over SO(3), unlike the paper’s correlation definition.
  • Because the compared convolution averages a filter over γ, it becomes circularly symmetric and is considered too limited for spherical-CNN pattern matching.

APPENDIX C: GENERALIZED FOURIER TRANSFORM

The appendix develops the generalized Fourier transform on SO(3) using Wigner D-functions as an orthogonal and complete basis for representing functions.

  • Wigner D-functions are matrix-valued basis functions associated with irreducible unitary representations of SO(3).They are parameterized by degree l and order parameters m and n.
  • The Wigner D-functions are orthogonal, supporting coefficient extraction through inner products.
  • The Wigner D-functions are complete, so any well-behaved function on SO(3) can be expressed as a linear combination of them.This provides the basis for the generalized Fourier transform on SO(3).
  • The generalized Fourier transform maps functions on SO(3) to their coefficients in the Wigner D-function basis.

APPENDIX D: FOURIER THEOREMS

The appendix derives Fourier convolution theorems on SO(3) and relates spherical convolution and Fourier analysis on S2 to SO(3) representations and transforms.

  • SO(3) Fourier foundations: Wigner D-matrices provide the representation-theoretic identities used to derive the SO(3) Fourier theorems.They are irreducible and unitary representations of SO(3), with unitarity expressed by D^l(R)D^l†(R) = I.
  • SO(3) Fourier foundations: The SO(3) Fourier transform converts convolution into matrix multiplication of the corresponding Fourier transforms.The derivation uses invariance of the integration measure on SO(3).
  • S2 Fourier analysis: Spherical harmonics form a complete orthogonal family on S2 and are related to the Wigner D functions.This relationship provides the harmonic basis for the analogous Fourier transform on the sphere.
  • S2 convolution: The S2 convolution of two functions is equivalent to the SO(3) convolution of their associated right-invariant functions.This equivalence allows spherical convolution results to be derived from the SO(3) convolution theorem.
  • S2 Fourier analysis: The S2 Fourier transform equals the n = 0 column of the SO(3) Fourier transform for the associated right-invariant function.The appendix presents this as a correspondence between spherical and rotational Fourier representations.
Loading 1801.10130v3…