Source-linked AI summary
VV-Net: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation
Hsien-Yu Meng, Lin Gao, YuKun Lai, Dinesh Manocha
TL;DR
Point-cloud segmentation must preserve sparse local geometry while handling irregular data and symmetry. VV-Net voxelizes the cloud, encodes subvoxel distributions with an RBF-VAE, and applies 3D group convolutions. It reports improved robustness and outperforms state-of-the-art methods on ShapeNet and S3DIS.
Problem
Boolean voxel occupancy captures limited detail, while point-cloud processing must handle sparse distributions and internal symmetry.
Method
VV-Net uses RBF interpolation within subdivided voxels, a variational autoencoder for compact latent features, and group convolutions defined on 3D symmetry sets.
Results
VV-Net outperforms state-of-the-art segmentation methods by 2.7% mean IoU on ShapeNet and 16.12% on S3DIS.
Takeaways & Limitations
The representation combines regular voxel structure with detailed point distributions, while group convolutions increase expressive capacity without increasing parameters.
Abstract
from arXiv · showhide
We present a novel algorithm for point cloud segmentation. Our approach transforms unstructured point clouds into regular voxel grids, and further uses a kernel-based interpolated variational autoencoder (VAE) architecture to encode the local geometry within each voxel. Traditionally, the voxel representation only comprises Boolean occupancy information which fails to capture the sparsely distributed points within voxels in a compact manner. In order to handle sparse distributions of points, we further employ radial basis functions (RBF) to compute a local, continuous representation within each voxel. Our approach results in a good volumetric representation that effectively tackles noisy point cloud datasets and is more robust for learning. Moreover, we further introduce group equivariant CNN to 3D, by defining the convolution operator on a symmetry group acting on $\mathbb{Z}^3$ and its isomorphic sets. This improves the expressive capacity without increasing parameters, leading to more robust segmentation results. We highlight the performance on standard benchmarks and show that our approach outperforms state-of-the-art segmentation algorithms on the ShapeNet and S3DIS datasets.
1. Introduction
VV-Net combines an information-rich voxel representation with 3D group convolutions to address sparse point distributions and internal point-cloud symmetry. The method reports stronger segmentation performance on ShapeNet and S3DIS while retaining robustness to noisy labels.
- Voxel representation: The RBF-VAE converts sparse, unstructured point clouds into a compact latent voxel representation that captures spatial distributions.Each voxel’s distribution is encoded into an l-dimensional latent representation.
- Segmentation framework: The method combines voxel features with per-point MLP features to support point-cloud segmentation.The architecture targets both object-part segmentation and semantic segmentation of 3D scenes.
- Results: 2.7% and 16.12% mean IoU improvements over state-of-the-art methods are reported on ShapeNet and S3DIS, respectively.The paper also reports meaningful segmentation when some ground-truth point labels are incorrect.
- Voxel representation: VV-Net encodes point distributions within voxels using subvoxel-level RBF interpolation and a variational autoencoder instead of Boolean occupancy alone.This combines regular voxel structure with detailed local point-distribution information.
- Group convolutions: 3D group convolutions model internal point-cloud symmetry on Z^3 and its isomorphic sets without increasing network parameters.The convolutions are intended to improve feature co-occurrence detection and expressive capacity.
2. Related Work
Prior point-cloud methods address irregular data through pointwise, neighborhood, graph, mesh, voxel, or symmetry-aware representations. VV-Net’s related-work context motivates combining detailed voxel distributions with symmetry-aware processing.
- Deep learning on 3D data: PointNet applies shared pointwise MLPs and symmetric operations to handle permutation, while a transform network improves rotation robustness.It is described as a state-of-the-art approach for 3D object classification and semantic segmentation.
- Deep learning on 3D data: Other point-cloud methods use hierarchical segmentation, pairwise similarity matrices, recursive sampling, recurrent networks, or sparse convolutions to capture structure and scale.Mesh-based approaches instead exploit graph structures and graph-convolution operators for arbitrary connectivity.
- Neighborhood mining: Neighborhood-mining methods use K-nearest neighbors, self-organizing maps, graph structures, or shape priors to encode local information.The text notes challenges from missing topology and neighborhoods spanning nearby points from different objects.
- Voxel processing: Voxel methods regularize point data for neural networks, but resolution is limited by sparsity and computational cost and they do not directly process raw point clouds.Farthest-point sampling is used in prior work to reduce computation and point-distribution irregularity.
- Group convolutions: Group-convolution research extends CNN equivariance beyond translations to transformations such as 90° rotations and dihedral flips.These developments motivate symmetry-aware operators for 3D point-cloud processing.
3. Voxel VAE Net with Group Convolutions
VV-Net combines an RBF-based voxel VAE with 3D group-equivariant convolutions to represent local point distributions and exploit symmetry in point-cloud segmentation.
- Voxel Representation: The voxel grid divides the point cloud into D×H×W cells, whose dimensions are determined by the scene ranges and grid resolution.Voxel sizes are vD = ˜D/D, vH = ˜H/H and vW = ˜W/W.
- RBF-VAE Scheme: RBF kernels replace Boolean voxel occupancy with a continuous representation of point distributions across subvoxels.The representation uses a variational autoencoder to produce a compact l-dimensional latent vector for each voxel.
- Variational Auto-Encoder: The VAE maps each voxel’s RBF subvoxel data to a latent vector and decodes a plausible corresponding datapoint.After pretraining, the fixed VAE produces a D × H × W × l point-cloud representation while reducing memory footprint.
- Experiments: The full group-conv + RBF-VAE pipeline outperforms the state-of-the-art method by 2.5% mean IoU on ShapeNet part segmentation.Removing group convolution or RBF-VAE lowers mIoU by 1.3% or 1.4%, respectively; replacing RBF-VAE with standard occupancy voxels prevents convergence.
- Symmetry Group and Equivariant Representations: The p4 group combines translations with 90° rotations about axes in a 3D grid and acts on voxel coordinates through matrix transformations.Its parameters encode rotations rx, ry, rz and translations tx, ty, tz.
4. Implementation and Performance
VV-Net combines RBF-VAE voxel representations with group convolutions for point-cloud segmentation, achieving strong ShapeNet and S3DIS results while remaining robust to missing points.
- 4.1. Part segmentation: ShapeNet contains 16,881 shapes across 16 categories with 50 annotated parts, evaluated using point-level mean IoU.Category IoU is computed by averaging instance IoU, with empty ground-truth and prediction unions assigned IoU 1.
- 4.1. Part segmentation: 2.5% mIoU: group-conv + RBF-VAE outperforms RSN on ShapeNet and performs better in 12 of 16 categories.The configuration captures 4 × 4 × 4 subvoxels using 8 latent variables.
- 4.2. Semantic segmentation of scenes: 16.12% Mean IoU: group-conv + RBF-VAE outperforms SPG on S3DIS and exceeds either component used alone.S3DIS contains six benchmark types, with points assigned to 13 semantic categories; results also report category IoU, overall accuracy, and AP at IoU threshold 0.5.
- 4.3. Robustness test: 0.77% accuracy loss: VV-Net retains performance when 87.5% of ShapeNet point-cloud data is missing.The robustness test uses farthest point sampling and evaluates multiple missing-data ratios.
- 4.4. Comparison of Different RBF Kernels: Gaussian RBF kernels outperform inverse quadratic kernels, while RBF-VAE reduces input size from 16MB to 2MB on the reported S3DIS example.The example contains 4096 points over 64 × 64 × 128 subvoxels, averaging 0.008 points per subvoxel; replacing G-CNN with traditional CNN decreases mean IoU by 7.79%.
- 4.5. Ablation study: Replacing VAE with AE yields about 2.2× higher test reconstruction loss, although training reconstruction losses are close.The authors attribute VAE’s better generalizability to its prior-distribution assumption and reduced potential overfitting.
5. Conclusions, Limitations and Future Work
The authors conclude that VV-Net improves robust point segmentation through RBF-based VAE representations and group convolutions, while identifying failure cases and future extensions.
- Conclusions: VV-Net combines an RBF-based variational autoencoder with group convolutions for robust point segmentation.The architecture is evaluated for object-part segmentation and semantic scene segmentation.
- Limitations: The method improves accuracy and robustness on ShapeNet and S3DIS, although it occasionally performs poorly on some input shapes.One reported failure classifies a Cap as a Table, possibly because group convolution encodes 90° symmetry.
- Future work: Future work includes improving accuracy, evaluating more complex point-cloud datasets, and applying VV-Net to tasks such as normal estimation.