Source-linked AI summary
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
Risi Kondor, Zhen Lin, Shubhendu Trivedi
TL;DR
Spherical CNNs provide rotation-equivariant learning for spherical images, but real-space nonlinearities require costly Fourier conversions. This paper uses Clebsch–Gordan decomposition as the sole nonlinearity in a fully Fourier-space SO(3)-equivariant architecture, reporting improved benchmark performance and broader compact-group applicability.
Problem
Existing equivariant networks commonly compute nonlinearities in real space, incurring repeated Fourier transforms, numerical errors, and quadrature complications.
Method
The paper represents activations as spherical-harmonic fragments and applies tensor products followed by Clebsch–Gordan decomposition, with subsequent operations implemented as matrix operations.
Results
7.97 RMSE for Ours (FFS2CNN), compared with 8.47 for Spherical CNN on the reported benchmark.
Takeaways & Limitations
The architecture avoids repeated forward and backward Fourier transforms and suggests a formalism applicable to equivariant networks for any compact continuous group.
Takeaways & Limitations
Allowing the number of channels to grow quadratically after each Clebsch–Gordan transform would be untenable, requiring channel limiting; the spherical treatment also assumes continuous functions.
Abstract
from arXiv · showhide
Recent work by Cohen \emph{et al.} has achieved state-of-the-art results for learning spherical images in a rotation invariant way by using ideas from group representation theory and noncommutative harmonic analysis. In this paper we propose a generalization of this work that generally exhibits improved performace, but from an implementation point of view is actually simpler. An unusual feature of the proposed architecture is that it uses the Clebsch--Gordan transform as its only source of nonlinearity, thus avoiding repeated forward and backward Fourier transforms. The underlying ideas of the paper generalize to constructing neural networks that are invariant to the action of other compact groups.
1 Introduction
The paper generalizes rotation-equivariant spherical CNNs by representing activations as SO(3)-covariant fragments and using Clebsch–Gordan decomposition for fully Fourier-space nonlinearities. This avoids repeated Fourier transforms while broadening the architecture’s applicability to compact groups.
- Equivariance applies the same filters across transformed input locations and is a key architectural source of success in CNNs.
- Spherical CNNs extend convolution to spherical images using SO(3) Fourier analysis for rotation-equivariant learning.Applications include 360-degree video, drone navigation, and molecular chemistry.
- Earlier spherical CNNs compute nonlinearities in real space, requiring repeated forward and backward SO(3) Fourier transforms that are costly and numerically problematic.The sphere and rotation group lack a regular discretization analogous to the Euclidean square grid.
- The proposed architecture uses SO(3)-covariant vectors called fragments instead of requiring each layer activation to be a vector-valued function on SO(3).This makes the architecture strictly more general than the earlier spherical CNN formulation.
- Clebsch–Gordan decomposition replaces real-space pointwise nonlinearities by tensor-product decomposition into irreducible fragments, avoiding repeated Fourier transforms.The authors report that the resulting network is more flexible and easier to implement, with better performance on some standard datasets.
- Because every compact group has a Clebsch–Gordan decomposition, the approach may extend beyond SO(3) to networks equivariant under other compact-group transformations.
2 Convolutions on the sphere
The paper develops spherical cross-correlation through continuous functions and harmonic analysis, then expresses the resulting operations in Fourier space. This framework motivates matrix-based filters and activations for spherical CNNs.
- Convolutional analogy: Classical CNN layers perform cross-correlation followed by a pointwise nonlinearity, with the correlation measuring local pattern matching.
- Convolutions on the sphere: Unlike the plane, S2 lacks a regular rotation-invariant discretization, so the paper treats spherical activations and filters as continuous complex-valued functions.The functions are parameterized by polar and azimuthal angles.
- Convolutions on the sphere: A naive spherical generalization of planar correlation is incorrect because it omits rotation of the filter around a third axis.The correct cross-correlation is therefore defined as a function on SO(3).
- Fourier representations: Spherical harmonics form an orthonormal Fourier-like basis for L2(S2), with practical expansions truncated at a resolution-dependent frequency L.
- Fourier representations: Noncommutative harmonic analysis represents functions on SO(3) with matrices indexed by irreducible representations, using the Haar measure.The representation matrices are inherently complex-valued, motivating complex activations and filters.
- Fourier-space operations: In Fourier space, spherical cross-correlation components reduce to outer products, and spherical CNN activations can be identified with spherical-harmonic coefficient vectors.For higher layers on SO(3), the corresponding Fourier components and filters are matrices.
3 Generalized spherical CNNs
The generalized spherical CNN represents activations as collections of SO(3)-covariant irreducible fragments, applies covariant linear maps and Clebsch–Gordan nonlinearities entirely in Fourier space, and produces exact rotation-invariant outputs.
- Covariant activations: Activations are collections of SO(3)-covariant vectors, or fragments, grouped by irreducible representation rather than required to be functions on SO(3).This generalizes the spherical CNN architecture while retaining equivariance to rotations.
- Covariant linear transformations: SO(3)-covariant linear maps mix only fragments belonging to the same irreducible representation, using learnable matrices W_s_ℓ that need not be square.The resulting matrix formulation is more general than Fourier-space cross-correlation, though less directly interpretable geometrically.
- Covariant nonlinearities: The Clebsch–Gordan transform supplies a fully Fourier-space nonlinearity by decomposing tensor products of irreducible representations into direct sums of irreducibles.The network computes this operation between all pairs of fragments.
- Covariant nonlinearities: The Clebsch–Gordan nonlinearity is essentially quadratic and is experimentally effective for learning spherical images, while higher Clebsch–Gordan powers would increase computational cost.Its quadratic structure is unusual for deep neural networks.
- Limiting the number of channels: Because each fragment acts as a separate channel, the Clebsch–Gordan transform squares channel count, requiring a subsequent learnable linear transformation to cap fragments per representation.The architecture naturally orders the Clebsch–Gordan transform before the convolution-like reduction step.
- Summary of algorithm: After the initial spherical harmonic transform, every operation is a matrix operation, avoiding repeated forward and backward Fourier transforms during training and testing.The final ℓ=0 Fourier components are guaranteed rotation-invariant up to finite-precision limitations.
4 Experiments
Experiments compare FFS2CNN with Spherical CNNs and other baselines on rotated spherical digits, molecular energy prediction, and 3D shape recognition. The proposed method maintains rotation-robust performance and improves on the direct Spherical CNN comparison.
- Rotated MNIST on the Sphere: On rotated spherical MNIST, the baseline CNN deteriorates toward random chance in the R/R setting, whereas the proposed method remains roughly stable across rotation regimes.The comparison includes NR/NR, NR/R, and R/R training-test rotation conditions.
- Rotated MNIST on the Sphere: The proposed method reports better results than Cohen et al.’s Spherical CNN on rotated spherical MNIST.The authors associate the competing model’s slight worsening with its loss of equivariance.
- Atomization Energy Prediction: The QM7 representation constructs spherical signals around atoms, discretized on a Driscoll–Healy grid with bandwidth b = 10.Each molecule is represented by a sparse tensor with dimensions N × T × 2b × 2b.
- Atomization Energy Prediction: 7.97 RMSE for FFS2CNN versus 8.47 RMSE for Spherical CNN on QM7 atomization-energy prediction.The MLP trained on randomly permuted Coulomb matrices performs better at 5.96 RMSE but is described as unlikely to scale to large molecules.
- 3D Shape Recognition: For SHREC17, the task uses roughly 51,300 randomly rotated 3D models across 55 categories and compares task-agnostic FFS2CNN with specialized models and Cohen et al.’s model.The dataset is split 70/10/20 for training, validation, and testing.
- 3D Shape Recognition: The SHREC17 model uses a ResNet-style architecture that progressively increases the maximum frequency band from L = 8 to Lmax = 14.Later blocks concatenate fragments from earlier blocks before further training.
5 Conclusion
The paper presents an SO(3)-equivariant architecture for spherical data that remains entirely in Fourier space by using Clebsch–Gordan decomposition as its only nonlinearity. Its framework is intended to extend to fully Fourier-equivariant networks for any compact continuous group.
- 5 Conclusion: The architecture operates completely in Fourier space, avoiding the repeated switches between Fourier and real space required by earlier models.Clebsch–Gordan decomposition supplies the network’s only source of nonlinearity.
- 5 Conclusion: The approach suggests a general formalism for designing fully Fourier neural networks equivariant to any compact continuous group.The paper’s specific focus is spherical data and SO(3)-equivariance.