PaperlayerView exact revision on arXiv

Source-linked AI summary

SO-Net: Self-Organizing Network for Point Cloud Analysis

Jiaxin Li, Ben M. Chen, Gim Hee Lee

arXiv:1803.04249v4cs.CV

TL;DR

Learning from orderless 3D point clouds requires representations that avoid sparse voxel inefficiency while capturing spatial structure and hierarchical local features. SO-Net builds a permutation-invariant SOM-based architecture with adjustable receptive-field overlap and reports competitive or superior performance across tasks, with significantly faster training. The paper also identifies suboptimal SOM configurations and reduced accuracy from an explored convolutional variant as limitations.

  • Problem

    Deep learning on 3D data is challenging because voxelization wastes computation on sparse inputs, while existing point-cloud networks inadequately capture local features or spatial distribution.

  • Method

    SO-Net builds a permutation-invariant SOM to model point-cloud spatial distribution, extracts hierarchical features from points and SOM nodes, and controls receptive-field overlap with point-to-node kNN search.

  • Results

    Competitive or superior performance is reported across reconstruction, classification, part segmentation, and shape retrieval, with significantly faster training than existing point-cloud recognition networks.

  • Takeaways & Limitations

    SOM-based spatial modeling provides a single global feature representation while supporting permutation-invariant point-cloud analysis across multiple recognition tasks.

  • Takeaways & Limitations

    SOM training can converge to local minima with isolated nodes outside the input coverage, and replacing later fully connected layers with 2D convolutions slightly decreased classification accuracy.

Abstract

from arXiv · show

This paper presents SO-Net, a permutation invariant architecture for deep learning with orderless point clouds. The SO-Net models the spatial distribution of point cloud by building a Self-Organizing Map (SOM). Based on the SOM, SO-Net performs hierarchical feature extraction on individual points and SOM nodes, and ultimately represents the input point cloud by a single feature vector. The receptive field of the network can be systematically adjusted by conducting point-to-node k nearest neighbor search. In recognition tasks such as point cloud reconstruction, classification, object part segmentation and shape retrieval, our proposed network demonstrates performance that is similar with or better than state-of-the-art approaches. In addition, the training speed is significantly faster than existing point cloud recognition networks because of the parallelizability and simplicity of the proposed architecture. Our code is available at the project website. https://github.com/lijx10/SO-Net

1. Introduction

SO-Net addresses challenges in learning from orderless 3D point clouds by explicitly modeling spatial distribution with a SOM and aggregating features hierarchically. The architecture provides adjustable receptive-field overlap, permutation invariance, and competitive performance with faster training across several tasks.

  • 3D deep learning remains challenging because voxel representations waste computation on sparse data and suffer resolution loss with rising computational cost.
  • SO-Net builds a SOM to model point-cloud spatial distribution and performs hierarchical feature extraction on individual points and SOM nodes.The network ultimately compresses each input point cloud into a single feature vector.
  • SO-Net is permutation invariant and explicitly utilizes point-cloud spatial distribution.Permutation invariance follows from the network design and permutation-invariant SOM training.
  • Point-to-node kNN search on the SOM systematically adjusts receptive-field overlap during hierarchical feature aggregation.The overlap is controlled through the kNN assignment parameter.
  • The proposed network supports classification, autoencoder reconstruction, part segmentation, and shape retrieval, with similar or better performance and significantly faster training than state-of-the-art approaches.A point-cloud autoencoder is also proposed as pre-training to improve performance across tasks.

2. Related Work

Prior point-cloud methods use voxel, multi-view, tree, or direct point representations, but each has limitations in efficiency, scalability, hierarchy, or explicit spatial modeling. SO-Net is positioned as a point-based approach that models spatial distribution while enabling hierarchical local feature aggregation.

  • Voxel methods are compatible with 3D ConvNets but face redundancy, resolution, and computational-cost problems on sparse 3D data.
  • Multi-view methods convert 3D shapes into 2D images so standard 2D ConvNets can process them, with pooling used to aggregate view features.
  • Kd-tree and octree representations provide scalable indexing structures whose regular organization supports deep-learning operations.
  • PointNet provides permutation invariance through shared per-point processing and max pooling but lacks ConvNet-like hierarchical feature aggregation.PointNet++ adds multi-level grouping to extract features at multiple scales.
  • SO-Net explicitly models spatial distribution during hierarchical extraction and uses adjustable receptive-field overlap for local feature aggregation, unlike heuristic grouping in PointNet++.

3. Self-Organizing Network

SO-Net uses a Self-Organizing Map to model point-cloud spatial distribution and guide hierarchical, permutation-invariant feature extraction. Point-to-node kNN search controls receptive-field overlap, while the architecture supports classification, segmentation, reconstruction, and efficient feature aggregation.

  • Overview: SO-Net converts an input point set into SOM nodes, then aggregates point features into node features and ultimately a global feature vector.The network processes points and SOM nodes hierarchically and supports classification, per-point segmentation, and reconstruction.
  • Permutation Invariant SOM: Fixed SOM initialization and deterministic batch updates make SOM training permutation invariant and GPU-efficient.Initial nodes are fixed for each SOM configuration, while batch updates accumulate all point effects before updating nodes.
  • Encoder Architecture: Point-to-node kNN search normalizes each point relative to k associated SOM nodes, and k controls overlap among the M receptive fields.The resulting kN normalized points are processed by shared fully connected layers before node-level max pooling.
  • Encoder Architecture: The encoder separates points into M mini point clouds, encodes them with shared PointNet-like processing, and assembles node features into a global representation.For M = 64 and k = 3, a typical mini point cloud for 2048 input points contains around 90 points within a small local coordinate region.
  • Reconstruction and Extensions: The architecture also explores convolutional node-feature processing and uses a two-branch decoder with up-convolution components for point-cloud reconstruction.The up-convolution branch recovers the main body, while a fully connected branch provides flexible detail recovery.
  • Segmentation: For segmentation, global, node, and normalized point features are combined into kN features and pooled into N per-point outputs.Average or max pooling removes redundancy created by receptive-field overlap.

4. Experiments

Experiments evaluate SO-Net on reconstruction, classification, segmentation, and robustness, using a shared encoder and autoencoder pre-training across tasks. The network achieves strong classification and segmentation results, trains efficiently, and remains reasonably robust to point and SOM corruption.

  • Experimental setup: Experiments evaluate SO-Net on point cloud autoencoding, object classification, and object part segmentation, with the autoencoder encoder also used for pre-training.The encoder structure and SOM configuration remain identical across experiments except for 2D MNIST classification.
  • Point cloud autoencoder: The autoencoder obtains a testing Chamfer distance of 0.033, recovering main object bodies and some details but losing finer structures such as earphone details.The convolution branch recovers the main body, while the fully connected branch focuses on details such as table legs.
  • Classification: SO-Net outperforms state-of-the-art methods by 1.7% on ModelNet10 and 1.5% on ModelNet40 in instance accuracy.It also achieves a relative 13.7% error rate reduction on MNIST, while some multi-network methods remain better on ModelNet classification.
  • Pre-training: Autoencoder pre-training improves ModelNet10 classification by 0.5% and ModelNet40 classification by 0.2%.The pre-trained encoder is also reported to improve performance in other tasks.
  • Robustness: With 50% of points missing, accuracy drops by 1.7%; with 75% missing, it drops by 14.2%, compared with a 3.8% drop for PointNet under its stated 50% corruption.Testing uses point clouds reduced from 2048 to 1024 or 512 points.
  • Robustness: With SOM Gaussian noise of σ = 0.2, SO-Net reaches 91.1% accuracy on ModelNet40 and 94.6% on ModelNet10.The noise is considerable because input points are normalized into a unit cube.
  • Hierarchical layers: A three-layer SO-Net improves accuracy to 1.5% higher than PointNet++ on ModelNet40 and 1.7% higher than Kd-Net on ModelNet10, although excessive depth may overfit.The added layer groups each SOM node with its k′ = 9 nearest nodes and processes them with PointNet.
  • Training speed: Training the best ModelNet40 network takes about 3 hours on a GTX1080Ti, enabled by deterministic SOM preprocessing and GPU-parallel feature aggregation.The batch training of SOM and hierarchical aggregation can be implemented efficiently on GPUs.

5. Conclusion

The paper concludes that SO-Net models point-cloud spatial distribution with a SOM and adjustable receptive-field overlap for hierarchical feature extraction. Across reconstruction, classification, segmentation, and retrieval, it achieves competitive performance, with advantages in classification and training speed.

  • Conclusion: SO-Net hierarchically extracts point-cloud features by explicitly modeling spatial distribution and systematically adjusting receptive-field overlap.The architecture converts point clouds into feature matrices through SOM-based processing.
  • Conclusion: Across reconstruction, object classification, and part segmentation, SO-Net achieves competitive performance and outperforms state-of-the-art deep-learning approaches in classification and shape retrieval.The conclusion also reports significantly faster training speed.
  • Future direction: Because the SOM preserves input-space topology, the authors identify classical or graph-based ConvNets on SO-Net feature matrices as a future direction for deeper aggregation.This is presented as a promising future direction rather than an achieved result.

Supplementary

The supplementary material adds shape-retrieval experiments and technical analyses of SO-Net’s complexity and permutation-invariant SOM training. It also provides further experiments and results.

  • Supplementary experiments: Shape retrieval experiments use the ShapeNet Core55 dataset.They are presented in Section B of the supplementary document.
  • Technical analyses: The supplementary document analyzes time and space complexity and details the permutation-invariant SOM training algorithms.These analyses appear in Sections C and D, respectively.
  • Shape retrieval: SO-Net’s classification network can perform shape retrieval by using classification scores as feature vectors and comparing query-library distances.Similarity is computed from distances between the query and candidate feature vectors.

B.1. Dataset

The ShapeNet Core55 retrieval experiment uses SHREC16’s category-based evaluation, while the network-complexity comparison evaluates point-cloud classifiers on ModelNet40. SO-Net’s retrieval metrics include macro and micro contexts, and its reported results outperform state-of-the-art approaches on most metrics.

  • Dataset: ShapeNet Core55 contains 51,190 shapes across 55 categories and 204 subcategories for the 3D shape retrieval experiment.The SHREC16 split assigns 70% of models to training, 10% to validation, and 20% to testing; CAD models are sampled into 5,000 points with normals.
  • Evaluation: The retrieval experiment uses a classification score vector of length 55 as the feature vector and computes L2 distances within each predicted category.The network uses an 8×8 SOM with k = 3 under the same configurations as the ModelNet40 classification experiment.
  • Evaluation: Macro metrics average equally across categories, whereas micro metrics weight categories by their number of shapes.SHREC16 reports precision-recall, F-score, mAP, and NDCG under both contexts.
  • Results: SO-Net outperforms state-of-the-art approaches on most retrieval metrics, and its precision-recall curves have the largest AUC.The comparison is reported in Table 3 and Figure 9.
  • Complexity comparison: Model size, forward time, and training time are evaluated for point-cloud networks on ModelNet40 classification.Forward timings use batch size 8 and input point clouds of 1024 points, comparing selected best-accuracy PointNet and PointNet++ configurations.

D.1. Initialization for SOM Training

SO-Net initializes its SOM with a fixed, permutation-invariant configuration generated by a potential-field procedure. Repulsion spreads nodes while attraction keeps them within the unit ball.

  • Initialization: Fixed initialization is used for every point cloud to preserve permutation invariance and reduce susceptibility to SOM local minima.The paper also mentions principal component initialization as another permutation-invariant possibility.
  • Potential-field method: The potential-field method initializes SOM nodes by applying pairwise repulsion and attraction toward the origin.The weighting parameter λ controls the balance between repulsion and attraction.
  • Potential-field method: The resulting SOM nodes are constrained to lie within the unit ball.This follows from balancing repulsive and attractive forces during initialization.
  • Algorithm: Algorithm 1 specifies the potential-field procedure used to initialize the SOM.The procedure iteratively updates the node set until convergence.

D.2. Batch Update Training

SO-Net trains the SOM with a batch update rule that aggregates all point contributions before updating nodes, making each iteration independent of input-point order. The procedure uses Gaussian neighborhoods and GPU-efficient matrix operations.

  • Batch update: Batch SOM updates accumulate the effect of all points before applying one update, so each iteration is independent of point order.Each training sample contributes to its winner node and neighboring nodes.
  • Neighborhood function: The neighborhood function is defined as a Gaussian distribution during SOM training.The Gaussian controls how training samples affect the winner node and its neighbors.
  • Training schedule: Algorithm 2 represents input points and SOM nodes explicitly and decreases the learning rate and neighborhood parameters slowly during training.The input points are P = {p_i} and SOM nodes are S = {s_j}.
  • Implementation: The batch training algorithm can be implemented as matrix operations that are highly efficient on GPUs.This implementation supports the method’s computational efficiency.

E.1. MNIST Classification

The MNIST experiment converts nonzero image pixels into noisy 2D point clouds and adapts SO-Net’s SOM and feature layers to the smaller 2D setting. SO-Net outperforms existing point-cloud networks but remains below the best-engineered 2D ConvNets.

  • Dataset: MNIST provides 60,000 training images and 10,000 test images for the 2D point-cloud experiment.The experiment extracts 2D coordinates from nonzero pixels.
  • Dataset: Gaussian noise N(0, 0.01) augments pixel coordinates when upsampling them into point clouds of 512 points.Other data augmentation follows the ModelNet and ShapeNetPart experiments.
  • Configuration: The 2D configuration uses a 4×4 SOM and k = 4, with reduced shared fully connected layers for point and node feature extraction.These reductions reflect the smaller 2D point clouds and their lower dimensionality.
  • Results: SO-Net outperforms existing point-cloud deep networks on MNIST, but the best performance remains with well-engineered 2D ConvNets.Using point clouds, SO-Net also performs better than Network in Network and LeNet5.
  • SOM contribution: The SOM contributes both directly and indirectly through node-feature concatenation, point normalization, kNN search, and max pooling.The experiment also evaluates classification using SOM nodes without point coordinates.
  • Results: Table 6 reports the MNIST classification results, while Table 5 reports classification using SOM nodes and kd-tree split directions.These tables organize the two evaluation comparisons described in the experiment.
  • SOM contribution: SOM-only classification achieves reasonable performance, but remains substantially behind the full SO-Net, indicating that integrating SOM and point clouds is important.The SOM-based MLP also performs better than the split-based MLP using kd-tree directions.

E.3. Result Visualization

The paper visualizes SO-Net outputs for shape retrieval, point-cloud reconstruction, and object part segmentation across ShapeNetPart and ModelNet40 examples.

  • Point-cloud reconstruction: The ShapeNetPart autoencoder consumes 1024 points and reconstructs 1280 points, with examples shown for one instance per category.Odd rows show inputs and even rows show reconstructions; red and green points indicate the convolutional and fully connected branches, respectively.
  • Point-cloud reconstruction: The ModelNet40 autoencoder consumes 5000 points and reconstructs 4608 points, with examples shown for one instance per category.Odd rows show inputs and even rows show reconstructions; red and green points indicate the convolutional and fully connected branches, respectively.
  • Object part segmentation: ShapeNetPart object part segmentation examples use 1024-point clouds with corresponding surface normals and compare ground-truth with predicted segmentations.Odd rows show ground-truth segmentation and even rows show predicted segmentation, with one instance per category.
  • Shape retrieval: Top-5 shape retrieval results are shown with query shapes followed by retrieved shapes ordered by feature similarity.The first column contains queries, while columns 2–6 contain retrieved shapes.
Loading 1803.04249v4…