Source-linked AI summary
Vector Neurons: A General Framework for SO(3)-Equivariant Networks
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas
TL;DR
Pointcloud networks need rotation-aware processing, but prior solutions can be mathematically complex or tied to specific architectures. This paper introduces Vector Neurons, which lift scalar neurons to 3D vectors and provide equivariant versions of common network operations. The resulting networks support arbitrary poses across multiple tasks, with consistent advantages over augmentation-based approaches and a small reconstruction-quality gap on aligned inputs.
Problem
Existing pointcloud methods either use complex equivariant formulations or are restricted to specific data types and architectures, motivating more accessible and versatile SO(3)-equivariant layers.
Method
Vector Neurons lift scalar latent features to ordered lists of 3D vectors and provide equivariant linear layers, nonlinearities, pooling, and normalization.
Results
The framework yields rotation-equivariant PointNet and DGCNN variants evaluated on classification, segmentation, and reconstruction, with consistent advantages under arbitrary rotations and state-of-the-art classification and segmentation performance.
Takeaways & Limitations
Vector Neurons provide a lightweight, versatile route to incorporating SO(3) equivariance across pointcloud backbones and learning tasks.
Takeaways & Limitations
On aligned inputs, particularly reconstruction, VN-OccNet falls slightly short of vanilla OccNet's reconstruction quality.
Abstract
from arXiv · showhide
Invariance and equivariance to the rotation group have been widely discussed in the 3D deep learning community for pointclouds. Yet most proposed methods either use complex mathematical tools that may limit their accessibility, or are tied to specific input data types and network architectures. In this paper, we introduce a general framework built on top of what we call Vector Neuron representations for creating SO(3)-equivariant neural networks for pointcloud processing. Extending neurons from 1D scalars to 3D vectors, our vector neurons enable a simple mapping of SO(3) actions to latent spaces thereby providing a framework for building equivariance in common neural operations -- including linear layers, non-linearities, pooling, and normalizations. Due to their simplicity, vector neurons are versatile and, as we demonstrate, can be incorporated into diverse network architecture backbones, allowing them to process geometry inputs in arbitrary poses. Despite its simplicity, our method performs comparably well in accuracy and generalization with other more complex and specialized state-of-the-art methods on classification and segmentation tasks. We also show for the first time a rotation equivariant reconstruction network.
1. Introduction
Pointcloud networks must handle unordered inputs and arbitrary 3D rotations, but existing solutions are either approximate, augmentation-heavy, mathematically intricate, or narrowly integrated. The paper introduces Vector Neurons and lightweight equivariant building blocks that support versatile SO(3)-equivariant architectures and tasks.
- Motivation: Pointcloud networks respect set symmetry, but rotation handling remains important because global object pose should not affect tasks such as similarity search.PointNet's spatial transformers provide only approximate pose invariance and require extensive training-time augmentation.
- Motivation: Tensor Field Networks and SE(3)-Transformers guarantee equivariance by construction but use intricate formulations and are restricted to convolutional, relative-position-based pipelines.These constraints make them harder to incorporate into existing architectures.
- Vector Neurons: Vector Neurons lift latent representations from scalar lists to ordered sequences of 3D vectors, enabling a direct mapping of input rotations to intermediate layers.The resulting latent matrix representation supports SO(3)-equivariant processing.
- Equivariant toolbox: The framework provides equivariant linear layers, learned-direction nonlinearities, invariant pooling, and normalization layers for constructing fully equivariant networks.The learned activation direction preserves equivariance where standard ReLU would not commute with rotations.
- Scope and results: Vector Neuron modules are lightweight and can be incorporated into existing architectures, with demonstrations using popular backbones across classification, segmentation, and reconstruction.The paper reports state-of-the-art performance for its VN versions on classification and segmentation and presents a 3D equivariant reconstruction network.
2. Related Work
Related work addresses rotation robustness through invariant operations, pose estimation, and equivariant designs. These approaches variously use reference frames, local and global information, additional surface normals, or pose supervision.
- Rotation invariant methods: Rotation-invariant architectures target tasks such as shape classification and segmentation using specially designed invariant operations.Examples include methods based on multi-scale PCA reference frames and combinations of local invariant information with global context.
- Rotation invariant methods: Some invariant methods augment point coordinates with surface normals, while SFCNN follows a multi-view-like strategy.The related-work discussion presents these as alternative ways to address rotation handling.
- Pose estimation: Pose-estimation methods study instance-level or category-level canonical poses, with extensions to articulated objects and methods using explicit 2D-to-3D supervision.The discussion also notes approaches that relax supervision through ideas from Transforming Auto-Encoders.
3. Method
The method lifts scalar neurons to 3D Vector Neurons, making rotations act directly on latent vectors and enabling equivariant linear, nonlinear, pooling, and normalization layers.
- Vector Neuron representation: Vector Neurons lift scalar neurons z ∈ R to vectors v ∈ R3, forming matrix-valued latent features for pointcloud processing.A pointcloud with N points is represented as N vector-list features with C vector channels.
- Equivariance: Rotation acts through standard rotation matrices, and the learnable mapping is required to commute with every R ∈ SO(3).The framework removes the layer index when operations are understood per layer.
- Linear layers: Linear layers preserve equivariance because applying a channel-mixing weight matrix before or after rotation gives the same rotated output.Biases are omitted because adding a constant vector would interfere with equivariance; centering enables SE(3) equivariance.
- Non-linear layers: VN non-linearity predicts a direction k from the input and clips the component of q in the negative half-space while leaving the orthogonal component unchanged.The construction generalizes ReLU and remains equivariant because q and k are linear maps and inner products are rotation invariant.
- Pooling layers: VN max pooling learns data-dependent directions and selects the vector channel best aligned with each direction, supporting global and local aggregation.Local pooling applies the same operation separately within groups of k nearest neighbours.
- Normalization layers: Batch normalization operates on vector 2-norms because averaging features from arbitrarily rotated inputs may be meaningless, whereas other normalizations generalize directly to vector distributions.The vector norms provide the invariant component used for batch statistics.
4. Network Architectures
Vector Neuron layers are incorporated into PointNet and DGCNN, demonstrating that the framework adapts to architectures with different pointcloud processing structures.
- Backbone integration: The framework plugs Vector Neurons into PointNet and DGCNN, covering a convolution-free backbone and a dynamic-graph edge-convolution backbone.The paper presents these backbones as representative of diverse pointcloud architectures.
- VN-DGCNN: VN-DGCNN modifies DGCNN by using vector-list representations while retaining its adjacent edge-feature computation and local max pooling.The modification is described as a straightforward replacement within the existing architecture.
- VN-PointNet: VN-PointNet uses a shared VN-MLP for per-point processing and adds an input edge convolution to map single-vector coordinates into C > 1 vector channels.The added layer avoids linearly dependent vector components before subsequent VN-MLP operations.
5. Experiments
Experiments evaluate VN networks across classification, segmentation, and reconstruction under multiple rotation train/test settings. The results show consistent rotation robustness, while the reconstruction model combines an equivariant encoder with an invariant decoder for pose-coherent outputs.
- Evaluation scope: Experiments cover classification, part segmentation, and reconstruction, spanning rotation-invariant and rotation-equivariant output tasks.Classification and segmentation are evaluated as rotation-invariant tasks, whereas reconstruction is rotation-equivariant.
- Classification: ModelNet40 classification uses z/z, z/SO(3), and SO(3)/SO(3) train/test settings, contrasting axis-limited and arbitrary rotation augmentation.The z setting uses rotations around the vertical axis, while SO(3) uses arbitrary rotations.
- Classification: VN networks achieve consistently good classification results across rotation settings, and VN-DGCNN outperforms point-coordinate equivariant or invariant methods in z/SO(3) and SO(3)/SO(3).Methods using surface normals still achieve slightly better results.
- Segmentation: ShapeNet part segmentation reports overall average category mean IoU over 16 categories in two rotation settings, with VN-DGCNN achieving the best performance among compared works.The comparison includes a method using surface normals in addition to point coordinates.
- Reconstruction: Reconstruction uses sparse, noisy ShapeNet pointclouds and predicts occupancy probability functions from latent codes derived by an encoder.The input consists of 300 subsampled surface points perturbed with zero-mean noise of standard deviation 0.005.
- Reconstruction: The equivariant-encoder/invariant-decoder reconstruction model remains coherent across poses, with only a minor accuracy loss in the aligned train/test setting.Replacing the equivariant encoder with a standard PointNet while retaining the invariant decoder slightly improves performance in all three settings.
6. Conclusions
The paper introduces Vector Neurons as a framework for rotation-equivariant networks and applies it across pointcloud classification, segmentation, and reconstruction. The approach is versatile and advantageous for arbitrarily posed inputs, although VN-OccNet slightly trails vanilla OccNet on aligned reconstruction.
- Core contribution: Vector Neurons lift standard neural-network representations into 3D to facilitate rotation-equivariant neural networks.The framework extends standard modules and supports pointcloud architectures and multiple tasks.
- Empirical scope: The work extends rotation-equivariant pointcloud processing to reconstruction, with meshes extracted using MISE from neural implicits.Figure 6 compares OccNet and VN-OccNet reconstruction results on ShapeNet.
- Core contribution: The framework supplies vector-neuron counterparts of linear layers, non-linearities, pooling, and normalization.These modules support constructing equivariant versions of PointNet and DGCNN.
- Empirical scope: VN architectures demonstrate a consistent advantage over augmentation-based approaches when input shapes have arbitrary poses.The paper evaluates classification, segmentation, and reconstruction.
- Limitation: On aligned input shapes, VN-OccNet falls slightly short of vanilla OccNet in reconstruction quality.The authors identify this as a limitation and future-work direction.
- Future directions: The framework is presented as potentially generalizable to higher-dimensional pointclouds and other modalities or transformation groups.Suggested extensions include meshes, voxel grids, images, and the full affine group.
7. Discussions
The discussion develops alternative equivariant layer designs and analyzes their practical trade-offs. It also describes normalization behavior and reports ablation settings for aligned classification and segmentation experiments.
- Linear and non-linear layers: A detached non-linear layer separates the linear transformation from the non-linearity, unlike the built-in design.Figure 7 depicts the detached architecture.
- Linear and non-linear layers: Detached linear and non-linear layers can improve results in some cases but double network depth and lengthen training.The comparison concerns VN-PointNet and VN-DGCNN backbones.
- Other non-linearities: Equivariant non-linearities can be extended beyond VN-ReLU, including VN-LeakyReLU and arbitrary scalar functions applied along an input-dependent direction.The construction applies the scalar function to the component along the learned direction.
- Pooling: Local pooling can aggregate features from nearest neighbours in either primal space or feature space.Feature-space locality follows the DGCNN approach and uses the same local-pooling formulation.
- Batch normalization: Logarithmic VN batch normalization avoids negative normalized 2-norm entries but can cause gradient explosion and fails for zero 2-norms.The instability arises from applying log and exp element-wise.
- Experimental settings: The additional experiments use aligned-data training for ModelNet40 classification and ShapeNet part segmentation.The table captions identify the reported measures as accuracy and mIoU, respectively.
8. Additional Experiments
Additional experiments test VN robustness without rotation augmentation and examine pooling and invariant-layer choices. The results emphasize robustness to test-time rotations, while pooling and invariant-layer differences are generally modest.
- Pooling ablation: Mean and VN-MAX pooling give comparable results, with mean pooling slightly better in more cases.VN-MAX also introduces additional learnable weights.
- Invariant-layer ablation: Improvements from concatenating the global mean and using a three-layer VN-MLP in the invariant layer are minor.The ablation compares these choices with alternatives using a local feature and a single VN linear layer.
- Rotation robustness: Training without augmentation and testing on rotated shapes further highlights VN robustness relative to rotation-sensitive counterparts.The additional classification and part-segmentation results cover I/I, I/z, and I/SO(3) settings.
- Ablations: The ablations compare non-linearity, pooling, and invariant-layer choices in VN networks on ModelNet40 classification.Tables 6–8 cover these three components.