PaperlayerView exact revision on arXiv

Source-linked AI summary

Spin-Weighted Spherical CNNs

Carlos Esteves, Ameesh Makadia, Kostas Daniilidis

arXiv:2006.10731v2cs.CVcs.LG

TL;DR

Spherical CNNs face a trade-off between the efficiency of spherical convolutions and the expressive filters of computationally costly SO(3) convolutions. This paper builds an equivariant CNN from spin-weighted spherical functions, reporting superior performance on its attempted tasks at reasonable computational cost.

  • Problem

    Spherical CNNs either use efficient spherical convolutions with isotropic filters or more expressive SO(3) convolutions that are computationally expensive.

    Source: §1 · p. 1

  • Method

    The paper defines spin-weighted convolutions and builds a CNN using sets of spin-weighted spherical functions as features and filters.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    The model achieves superior performance on the tasks attempted, at reasonable computational cost.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Spin-weighted features can represent equivariant spherical vector fields, supporting tasks with vector fields as inputs or outputs.

    Source: §1 · p. 2

    For bandwidth B, the model requires |W| transforms per layer and is asymptotically |W|B/log2 B times slower than SFT-based spherical convolution, but B/|W| times faster than SOFT-based convolution.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Learning equivariant representations is a promising way to reduce sample and model complexity and improve the generalization performance of deep neural networks. The spherical CNNs are successful examples, producing SO(3)-equivariant representations of spherical inputs. There are two main types of spherical CNNs. The first type lifts the inputs to functions on the rotation group SO(3) and applies convolutions on the group, which are computationally expensive since SO(3) has one extra dimension. The second type applies convolutions directly on the sphere, which are limited to zonal (isotropic) filters, and thus have limited expressivity. In this paper, we present a new type of spherical CNN that allows anisotropic filters in an efficient way, without ever leaving the spherical domain. The key idea is to consider spin-weighted spherical functions, which were introduced in physics in the study of gravitational waves. These are complex-valued functions on the sphere whose phases change upon rotation. We define a convolution between spin-weighted functions and build a CNN based on it. The spin-weighted functions can also be interpreted as spherical vector fields, allowing applications to tasks where the inputs or outputs are vector fields. Experiments show that our method outperforms previous methods on tasks like classification of spherical images, classification of 3D shapes and semantic segmentation of spherical panoramas.

1 Introduction

The introduction asks how spherical CNNs can combine expressive filters with the efficiency of spherical-domain convolutions. It proposes spin-weighted spherical functions as the basis for an equivariant CNN that handles anisotropic filters and vector fields, and outlines evaluations across several spherical tasks.

1 Introduction

  • Existing spherical CNNs trade efficient spherical convolutions with isotropic filters against more expressive but costly convolutions on SO(3).The introduction frames the goal as achieving SO(3)-convolution expressivity with spherical-convolution efficiency and scalability.Source: §1 · p. 1
  • Spin-weighted spherical functions enable more expressive representations and anisotropic filters without lifting features to SO(3).They are complex-valued spherical functions whose phases change upon rotation.Source: §1 · p. 2
  • The proposed CNN defines SO(3)-equivariant convolutions and cross-correlations of spin-weighted functions, with adapted nonlinearities, normalization, and pooling.For bandlimited inputs, the operations can be computed exactly in the spectral domain.Source: §1 · p. 2
  • Spin-weighted features can be interpreted as equivariant spherical vector fields, supporting vector-field inputs or outputs that current spherical CNNs cannot handle equivariantly.The introduction illustrates the distinction between scalar and vector-field rotations in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper evaluates spherical image and vector-field tasks, 3D shape classification, and spherical-panorama segmentation.The vector-field dataset projects MNIST images and their gradients onto the sphere and includes classification and scalar-to-vector or vector-to-scalar prediction.Source: §1 · p. 2

2 Related work

Prior spherical CNNs either lift spherical inputs to SO(3), increasing computational cost, or convolve on the sphere with isotropic filters. The proposed approach uses spin-weighted spherical functions to avoid both constraints.

2 Related work

  • One branch lifts spherical inputs to SO(3), while purely spherical convolutions use zonal, isotropic filters that limit representational power.The SO(3) branch is computationally more expensive because it processes functions on a higher-dimensional group.Source: §2 · pp. 2–3
  • The approach uses sets of spin-weighted spherical functions as filters and features, allowing non-isotropic filters without lifting features to SO(3).It is positioned between the two established spherical-CNN branches.Source: §2 · p. 3
  • Related work includes non-equivariant spherical CNNs that assume aligned inputs and manifold methods for vector-field features, while this work demonstrates vector-field tasks with a fully spectral implementation.The paper also distinguishes its spherical spin-weighted features from related planar complex-valued features.Source: §2 · p. 3

3 Background

The background establishes spherical harmonics and their Fourier representation, then extends the discussion to spin-weighted functions and their rotation behavior. It also reviews spherical convolution and cross-correlation as the basis for the later spin-weighted generalization.

3 Background

  • The background reviews spherical convolution and cross-correlation in the spectral domain as the operations later generalized to spin-weighted functions.The two operations have Fourier coefficients associated with spherical functions and functions on SO(3), respectively.Source: §3 · pp. 3–4
  • Spherical harmonics form an orthonormal basis for square-integrable functions on the sphere, enabling Fourier decomposition and exact reconstruction.A function has bandwidth B when its expansion contains only components of order ℓ≤B.Source: §3 · p. 3
  • Spherical harmonics are related to SO(3) representations, whose rotation formula determines how a spherical function’s Fourier coefficients transform.The background identifies the Euler angles and Wigner-D matrices used in this representation.Source: §3 · pp. 3–4
  • Spin-weighted spherical functions are complex-valued and acquire a phase change under rotation; spin weight zero recovers conventional spherical functions.Their expansion coefficients transform among themselves, which the paper identifies as crucial for defining equivariant convolutions.Source: §3 · p. 4

4 Method

The method builds a fully convolutional network from spin-weighted convolutions and cross-correlations, with filters and features represented as combinations of spin-weighted spherical functions. It adapts filtering and standard CNN components to preserve the spin structure.

4.1 Spin-Weighted Convolutions and Cross-Correlations

  • Spin-weighted convolution uses input components selected by input spins and returns SWSFs at the filter spins, letting layers specify their output spin sets.The operation is defined in the spectral domain.Source: §4.1 · p. 5
  • The convolution is SO(3)-equivariant, and ordinary spherical convolution is a spin-0 special case.Source: §4.1 · p. 5
  • Cross-correlation uses spins shared by inputs and filters but can produce all spins; sparsifying filter spectra restricts output spins, and the spin-0 case recovers spherical cross-correlation.Source: §4.1 · p. 5

4.2 Spin-weighted spherical CNNs

  • Feature maps combine channels across spins, with equal channel counts per spin; spectrally smooth filters are interpolated from anchors to encourage localization.This interpolation uses |WF*K||WF| more parameters per layer.Source: §4.2 · pp. 5–6
  • Spin-0 features are made real for ReLU and standard batch normalization; positive-spin maps use magnitude-based nonlinearities and normalization without mean subtraction or offset addition.Source: §4.2 · p. 6
  • SWSFT costs O(B^3), asymptotically slower than SFT yet faster than SOFT; typical experimental values are B=32 and |W|=2.Source: §4.2 · p. 6

5 Experiments

The experiments evaluate SWSCNNs on spherical image and vector-field tasks, 3D shape classification, and spherical-panorama segmentation. Across these settings, the results show strong performance, including advantages on rotation generalization and tasks requiring vector-field equivariance.

5.1 Spherical Image Classification

  • SWSCNNs outperform previous spherical CNNs in every Spherical MNIST rotation mode with parameter counts matched to the comparison architecture.The model is described as more expressive than isotropic methods and more efficient than previous anisotropic spherical CNNs.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 Spherical Vector Field Classification

  • When vector-field equivariance is required, SWSCNNs have a larger performance gap over planar and spherical baselines, which fail to generalize to unseen vector-field rotations.The dataset uses spherical projections of MNIST images and their gradients; the classification model maps spin 1 inputs to spin 0 and 1 features.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Spherical Vector Field Prediction

  • On SVFMNIST prediction tasks, SWSCNNs outperform the planar baseline on unseen rotations and are clearly superior on the harder semantic tasks.The easy gradient-derived tasks are solvable by discrete integration or differentiation, but isotropic filters cannot approximate a simple image-gradient operator; the harder colored-digit tasks require implicit classification to predict colors and vector directions.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Classification of 3D shapes

  • SWSCNNs outperform previous spherical CNNs on ModelNet40 while using small input size and low parameter count.The model represents normal directions uniquely by splitting normals into radial spin-0 and tangent spin-1 components.Source: Table 4 · p. 9, §5.4 · p. 9
  • For upright ModelNet40 inputs, replacing final equivariant layers with regular 2D convolutions improves performance, while it worsens performance on rotated inputs.The modified model is called “Ours + BE”; the upright mode allows only azimuthal rotations, whereas rotated inputs are unrestricted.Source: §5.4 · p. 9

5.5 Semantic segmentation of spherical panoramas

  • On Stanford 2D3DS segmentation, SWSCNNs match state-of-the-art performance despite upright inputs not requiring global SO(3) equivariance.The method can represent surface normals uniquely, including relative to local tangent frames; competing methods do not usually use normals.Source: Table 5 · p. 9, §5.5 · p. 9

6 Conclusion

The conclusion presents SWSCNNs as spherical-image and vector-field models with superior performance on the attempted tasks at reasonable computational cost, and points to further application areas.

6 Conclusion

  • SWSCNNs use sets of spin-weighted spherical functions as features and filters, processed by newly introduced spin-weighted spherical convolution layers.The model handles spherical images and spherical vector fields.Source: §6 · p. 9
  • The model achieves superior performance on the tasks attempted at reasonable computational cost.Source: §6 · p. 9
  • The authors foresee applications in 3D shape analysis, climate and atmospheric data analysis, and other tasks involving spherical images or vector fields.Source: §6 · p. 9

Broader Impact

The broader-impact discussion connects equivariant CNNs to potential efficiency and accessibility benefits while noting that the technology could also be used harmfully.

Broader Impact

  • Equivariant CNNs aim to reduce model and sample complexity and improve generalization.Source: Broader Impact · p. 10
  • These aims could translate to more energy-efficient models and greater access for people without large computational resources.Source: Broader Impact · p. 10
  • Making the technology more accessible also risks enabling bad actors to use it for harmful purposes.Source: Broader Impact · p. 10

Appendices

The supplementary material points readers to dataset, experiment, and transform-implementation details in its appendices.

Appendices

B Datasets

The dataset and experiment details describe SVFMNIST samples, challenge-modified prediction targets, training settings, and classification and dense-prediction architectures.

B Datasets

C MNIST Experiments Details

  • MNIST experiments train for 12 epochs with Adam, batch size 32, and 64 × 64 inputs; classification uses cross-entropy and dense prediction uses mean squared error.Source: C MNIST Experiments Details · p. 13

C.1 Classification

  • Classification models share the same architecture topology; SWSCNN uses fewer channels while matching parameter counts, with spins 0 and 1 throughout.The spherical baseline uses 16, 16, 32, 32, 58, 58 channels; SWSCNN uses 16, 16, 20, 24, 28, 32, and the planar baseline uses 16, 16, 32, 32, 54, 54.Source: §C.1 · p. 14

C.2 Spherical vector field/image prediction

  • Dense prediction uses a fully convolutional U-Net with spin-weighted convolutions, pooling in the first two layers, and nearest-neighbor upsampling in the last two.Its channel counts are 16, 32, 32, 32, 32, 16; the spherical and planar baselines use different channel configurations.Source: §C.2 · p. 15

C.3 Input-output samples

The examples compare spherical vector-field tasks under rotated inputs: standard and spherical CNNs do not preserve vector-field equivariance, while SWSCNNs do.

C.3 Input-output samples

  • Even with rotated training samples, standard and spherical CNNs are not equivariant in the vector-field sense and cannot match SWSCNN accuracy.The examples include vector-field-to-image and image-to-vector-field tasks.Source: §C.3 · p. 15, Figure 7 · p. 15
  • The examples also show input/output samples for the spherical image-to-vector-field task.Source: §C.3 · p. 15
  • For vector-field-to-image inputs, rotated instances of the same input illustrate that SWSCNNs respect spherical vector-field equivariance, unlike standard and spherical CNNs.Source: Figure 7 · p. 15

D Classification of 3D shapes

The appendix specifies evaluation and model setups for 3D shape classification and spherical panorama segmentation, then outlines an FFT-based implementation of the spin-weighted spherical harmonics transforms.

D Classification of 3D shapes

  • ModelNet40 evaluation follows a protocol with multiple rotated copies in training and testing, distinguishing azimuth-only upright rotations from arbitrary 3D rotations.The dataset has 9,843 training and 2,468 test CAD models.Source: D Classification of 3D shapes · p. 15
  • The 3D-shape model uses a residual architecture with 1.2 M parameters, while training uses 48 epochs and label smoothing with ϵ = 0.2.The passage also describes the channel schedule and spin-order-specific filter parameters.Source: D Classification of 3D shapes · p. 15

E Semantic segmentation of spherical panoramas

  • Panorama segmentation is evaluated on Stanford 2D3DS using pixelwise accuracy and mean intersection-over-union averaged over three official folds, with class weights to mitigate imbalance.The dataset contains 1,413 RGB-D panoramas with semantic labels and normals.Source: E Semantic segmentation of spherical… · p. 16
  • Segmentation uses a fully convolutional U-Net with 2.5 M parameters and a 48-epoch training setup with label smoothing.The architecture uses average pooling and nearest-neighbor upsampling as channel counts change.Source: E Semantic segmentation of spherical… · p. 16

F Spin-Weighted Spherical Harmonics Transforms

  • The spin-weighted spherical harmonics transform implementation follows Huffenberger and Wandelt, leveraging a relation between the harmonics and Wigner-D matrices.The passage identifies dℓ as a Wigner-d matrix.Source: F Spin-Weighted Spherical Harmonics… · pp. 16–17
  • A periodic torus extension handles the spherical function's lack of periodicity, enabling FFT computation; the discretized transform applies precomputed weights before a 2D FFT.A derived 1D discrete convolution can be computed as an FFT of a spatial-domain multiplication.Source: F Spin-Weighted Spherical Harmonics… · pp. 17–18

Related papers

Loading 2006.10731v2…