PaperlayerView exact revision on arXiv

Source-linked AI summary

CubeNet: Equivariance to 3D Rotation and Translation

Daniel Worrall, Gabriel Brostow

arXiv:1804.04458v1cs.CVcs.AIcs.LGstat.ML

TL;DR

3D CNNs can produce pose-sensitive representations of voxelized objects, making rotated instances difficult to relate. CubeNet addresses this by building linear equivariance to translations and right-angle 3D rotations into group convolutions, achieving state-of-the-art ModelNet10 performance and comparable ISBI 2012 segmentation performance.

  • Problem

    3D CNNs are sensitive to transformations, so voxelized objects and their rotated versions can produce unrelated representations.

  • Method

    CubeNet is a group-convolutional CNN with linear equivariance to translations and right-angle rotations in three dimensions.

  • Results

    CubeNet achieves state-of-the-art performance on ModelNet10 and comparable performance on the ISBI 2012 connectome segmentation benchmark.

  • Takeaways & Limitations

    The architecture preserves a 3D shape’s identity while respecting pose through discrete right-angle rotation processing.

  • Takeaways & Limitations

    The approach leaves continuous rotations, scale transformations, and rotation groups not aligned to the Cartesian voxel grid for future work.

Abstract

from arXiv · show

3D Convolutional Neural Networks are sensitive to transformations applied to their input. This is a problem because a voxelized version of a 3D object, and its rotated clone, will look unrelated to each other after passing through to the last layer of a network. Instead, an idealized model would preserve a meaningful representation of the voxelized object, while explaining the pose-difference between the two inputs. An equivariant representation vector has two components: the invariant identity part, and a discernable encoding of the transformation. Models that can't explain pose-differences risk "diluting" the representation, in pursuit of optimizing a classification or regression loss function. We introduce a Group Convolutional Neural Network with linear equivariance to translations and right angle rotations in three dimensions. We call this network CubeNet, reflecting its cube-like symmetry. By construction, this network helps preserve a 3D shape's global and local signature, as it is transformed through successive layers. We apply this network to a variety of 3D inference problems, achieving state-of-the-art on the ModelNet10 classification challenge, and comparable performance on the ISBI 2012 Connectome Segmentation Benchmark. To the best of our knowledge, this is the first 3D rotation equivariant CNN for voxel representations.

1 Introduction

Standard CNN activations are sensitive to local and global pose, while data augmentation offers a simple but costly route to invariance. CubeNet instead builds linear equivariance to 3D rotations and translations into the architecture, reducing reliance on augmentation and achieving strong benchmark performance.

  • Motivation: CNN filters are selective to orientation, scale, reflection, and parity, making intermediate activations and global outputs sensitive to local pose.
  • Motivation: Data augmentation can obtain invariance by training on transformed copies, but it requires an efficient and realistic augmentation pipeline.
  • Motivation: Architectural equivariance can reduce the training data needed because some transformation properties are enforced a priori rather than learned.
  • Contribution: CubeNet introduces a CNN linearly equivariant to 3D rotations and translations of voxelized data.The architecture uses weight tying to exploit classification-task symmetries.
  • Results: CubeNet achieves state-of-the-art performance on ModelNet10 and comparable performance on the ISBI 2012 connectome segmentation benchmark.

2 Background

The paper formalizes equivariance, invariance, groups, and convolution before extending convolution over transformation groups. Group convolution replaces spatial-only filtering with filtering over transformations, producing outputs indexed by transformation parameters and enabling linear equivariance.

  • Definitions: Equivariance explicitly relates transformations of inputs to transformations in feature space, whereas invariance is the special case whose feature-space transformation is the identity.
  • Convolution: The standard 3D convolution computes inner products between filters and patches extracted from activation tensors using a translational sliding window.
  • Convolution: The output is indexed by transformation parameter g, so each activation records the response of a correspondingly transformed filter.
  • Group Convolution: Group convolution replaces the standard convolution’s summation domain Z3 with a transformation group G, enabling sliding-and-rotating, flipping, or scaling inner products.
  • Group Convolution: For 3D roto-translations, filters are rotated and shifted using a 3D rotation matrix Rg and translational offset zg.

3 Related Work

Prior work includes handcrafted, learned, and 2D rotation-equivariant approaches, while 3D voxel methods largely rely on augmentation or auxiliary orientation prediction. CubeNet addresses this gap with a discrete 3D rotation-group construction for voxelized inputs.

  • 3D Methods: Earlier predefined-equivariance CNNs mainly focused on 2D, with notable 3D alternatives operating on spherical projections or point clouds rather than voxelized data.
  • Handcrafted Equivariance: Handcrafted equivariant models include scale-space and steerable-filter approaches, with exact steering for some non-compact transformations requiring infinitely many coefficients.
  • 2D Rotation Invariant Neural Networks: 2D rotation-invariant methods average predictions or responses across rotated inputs, with some scattering approaches potentially losing discriminative information through phase removal.
  • 2D Rotation Equivariant Neural Networks: Some 2D equivariant methods convert rotations and scalings about one point into translations, but this provides only global equivariance and does not generalize to 3D.
  • 3D Methods: Existing 3D voxel classifiers use standard 3D convolutions, heavy data augmentation and rotation averaging, or an auxiliary orientation-estimation task.

4 Method

The method extends convolution to discrete 3D rotations and translations using group structure, preserving shape identity while encoding pose through equivariant feature transformations. CubeNet uses selected rotation subgroups, Cayley-table composition, and permutation-based group convolutions to implement this behavior.

  • CubeNet filters voxelized 3D shapes at discrete right-angle orientations while preserving shape identity and respecting pose.The architecture selects Klein’s four-group for a 3D roto-translation equivariant CNN and extends standard convolution to transformations.
  • The cube group S4 contains 24 right-angle rotations, while T4 and Klein’s four-group provide 12- and 4-state subsamplings.T4 is a subgroup of S4, and Klein’s four-group is a subgroup of both with four rotations about independent axes.
  • 4.1 Cayley tables: Cayley tables determine transformation composition and therefore identify how group-convolution indices change across layers.Rows and columns encode composed rotations; their permutation structure supports determining deeper-layer group convolutions.
  • 4.2 Discrete Group Equivariance and Permutations: Rotating an input permutes the group-convolution output vector, with the permutation determined by the input transformation and transformation group.The output dimensions correspond to group elements, and the permutation can be represented by a permutation matrix associated with the Cayley table.
  • 4.3 Implementation: Roto-translational group-convolution: Roto-translations combine a rotation and translation, and the implementation rotates the filter before applying an efficient translational convolution.The filter has values indexed by translation and input rotation; the rotated kernel is then shifted and convolved using standard GPU operations.

5 Experiments and results

CubeNet is evaluated on ModelNet10 classification and ISBI 2012 volumetric segmentation using group-convolutional architectures designed for 3D transformations. It achieves state-of-the-art single-model ModelNet10 accuracy and competitive ISBI performance without extensive test-time rotation averaging.

  • 5.1 ModelNet10: The ModelNet10 experiment compares four-group and rotational tetrahedral group models using single-view and rotation-averaged accuracy across rotated test views.Single-view accuracy averages predictions over 12 rotated test views, whereas rotation-averaged accuracy evaluates the average of all 12 predictions.
  • 5.1 ModelNet10: The four-group, rotation-averaged CubeNet achieves state-of-the-art single-model accuracy on ModelNet10.The benchmark table states that only Brock et al.'s six-model ensemble performs better overall; restricting comparison to single models, CubeNet holds state-of-the-art accuracy.
  • 5.1 ModelNet10: The four-group model improves slightly with rotation averaging, while the T4 model performs worse under a fixed total filter budget.The authors attribute the T4 result to reduced filter diversity from increasing the number of filter copies.
  • 5.2 ISBI 2012 Challenge: Connectome Segmentation: For ISBI 2012 segmentation, CubeNet's Vrand is slightly better than UNet and Quan et al., but below Weiler et al., while Vinfo trails the other methods.The comparison uses a 3D group-equivariant FusionNet-like architecture and no post-processing.
  • 5.2 ISBI 2012 Challenge: Connectome Segmentation: The ISBI experiment demonstrates relatively good 3D segmentation performance without extensive test-time rotation averaging or post-processing.The method uses Klein's four-group to address anisotropic imaging and weight tying to reduce trainable parameters.

6 Conclusion

The paper concludes that CubeNet provides 3D CNN equivariance to right-angle rotations through an extension of convolution. Its design can achieve strong classification performance while leaving continuous rotations, scale, and non-Cartesian rotation groups for future work.

  • 6 Conclusion: CubeNet is equivariant to right-angle rotations in three dimensions through an extension of standard 3D convolution.The architecture is designed so the CNN need not learn to overcome these rotations in the same way as a standard CNN.
  • 6 Conclusion: On ModelNet10, CubeNet achieves state-of-the-art performance for a single model while avoiding the need for time-costly test-time rotation averaging.The conclusion contrasts this with larger models that rely on heavy data augmentation.
  • 6 Conclusion: CubeNet commits to a predefined transformation group and architecture instead of learning to disentangle pose across feature spaces.The trade-off is reduced effort in learning pose structure, at the cost of restricting the model to the selected group.
  • 6 Conclusion: Future work must examine generalization to continuous rotations, scale, and 3D rotation groups not aligned with the Cartesian voxel grid.These transformations and group choices remain outside the demonstrated scope.
Loading 1804.04458v1…