Source-linked AI summary
Pointwise Convolutional Neural Networks
Binh-Son Hua, Minh-Khoi Tran, Sai-Kit Yeung
TL;DR
Point-cloud use with convolutional neural networks remains underexplored despite the flexibility of point-cloud representations. The paper introduces pointwise convolution and builds fully convolutional networks for semantic segmentation and object recognition, achieving competitive accuracy while remaining simple to implement. The approach also exposes scope boundaries around point ordering and large-scale scenes.
Problem
Point clouds are flexible 3D representations, but their capability with neural networks has not been fully explored.
Method
The paper applies pointwise convolution at each point and stacks these operators into networks for semantic segmentation and object recognition.
Results
The network produces 81.5% per-point accuracy for semantic segmentation, comparable to PointNet, while the paper reports competitive accuracy across its evaluated tasks.
Takeaways & Limitations
Pointwise convolution offers a simple way to build effective neural networks that learn local features from point clouds.
Takeaways & Limitations
Large-scale semantic segmentation is handled by dividing scenes into blocks and resampling each block to a fixed number of points, while input ordering remains a design consideration.
Abstract
from arXiv · showhide
Deep learning with 3D data such as reconstructed point clouds and CAD models has received great research interests recently. However, the capability of using point clouds with convolutional neural network has been so far not fully explored. In this paper, we present a convolutional neural network for semantic segmentation and object recognition with 3D point clouds. At the core of our network is pointwise convolution, a new convolution operator that can be applied at each point of a point cloud. Our fully convolutional network design, while being surprisingly simple to implement, can yield competitive accuracy in both semantic segmentation and object recognition task.
1. Introduction
The paper addresses the underexplored use of point clouds with neural networks by introducing pointwise convolution for fully convolutional 3D scene understanding.
- Stacking pointwise convolution operators enables fully convolutional networks for scene segmentation and object recognition.
- Point clouds are compact and broadly exportable from CAD and 3D reconstruction software, but their use with neural networks remains underexplored.
- The paper presents a convolutional neural network for semantic segmentation and object recognition on 3D point clouds.
- Pointwise convolution applies a new convolution operator at each point to learn pointwise features.
- The network design is described as simple to implement while achieving competitive accuracy against previous techniques.
2. Related Works
Prior 3D scene-understanding methods use volumes, projections, point sets, or complex point-cloud architectures. This paper instead proposes a simple fully convolutional approach operating directly on ordered point clouds.
- 3D deep-learning methods represent data as volumes, multi-view images, or point sets, each offering different implementation and representation trade-offs.
- PointNet introduced an architecture that handles point clouds by learning an order-invariant function to canonicalize them.
- PointCNN explored equivariance instead of invariance and reported competitive performance on point-cloud data.
- The paper argues that PointNet-based network structures are rather complex and proposes pointwise convolution for simpler 3D point-cloud networks.
- Semantic segmentation: For semantic segmentation, prior methods include RGB-D approaches, volume-based voxel classification, interactive propagation, and graph neural networks.
- Semantic segmentation: Unlike one cited graph-based method, this work trains from scratch and accepts CAD models or meshes reconstructed from RGB-D sensors.
3. Pointwise Convolution
Pointwise convolution applies a kernel at every point, using neighboring points in local 3D sub-domains to produce pointwise features. The design supports trainable, fully convolutional point-cloud networks without pooling, while accommodating ordered inputs, efficient neighbor queries, and expanded receptive fields.
- Convolution: Pointwise convolution centers a kernel at each point, allowing neighboring points within its support to contribute to the center-point output.Kernel size or radius controls the number of neighboring points considered at each layer.
- Convolution: The operator partitions each kernel support into sub-domains and combines values from points in those sub-domains using learned kernel weights.The formulation indexes sub-domains around point i, counts their points, and uses weights w_k to compute outputs across layers.
- Gradient backpropagation: Gradient backpropagation computes derivatives with respect to input data and kernel weights, enabling pointwise convolution to be trained with a loss function.Input gradients iterate over neighboring points, while kernel-weight gradients iterate over all points.
- Implementation: Uniform 3 × 3 × 3 kernels and neighbor-query acceleration structures allow the operator to run efficiently on CPUs and GPUs.All points within each kernel cell share the same weights.
- Network design: Unlike volumetric convolution, the network omits pooling, preserving the point cloud and requiring neighbor-query acceleration structures to be built only once.This avoids point-cloud downsampling and upsampling and speeds computation.
- Network design: The method accepts points sorted in a specific order and can use à-trous convolution to enlarge the receptive field without processing too many points.Point coordinates remain available for neighbor queries, while point attributes such as colors, normals, or learned features are stored separately.
4. Evaluations
The evaluation covers semantic segmentation and object recognition across point-cloud datasets, then examines design choices affecting accuracy, ordering, neighborhood queries, depth, and training. The method achieves competitive results while remaining simpler than several alternatives, though clutter, ambiguity, and overlap noise remain challenges.
- Semantic segmentation: On S3DIS, the network produces 81.5% per-point segmentation accuracy, comparable to PointNet.Predictions and ground truths are visualized in Figure 3.
- Semantic segmentation: On SceneNN, structures and chairs are relatively accurate, while tables and desks are more ambiguous; overlap regions introduce prediction noise.The method works competitively with SemanticFusion and outperforms VoxNet in the reported comparison.
- Object recognition: For object recognition, the method performs comparably to PointNet on ObjectNN, although both methods are less effective because CAD and RGB-D objects produce ambiguous features.ModelNet40 performance is compared across network architectures using different 3D object representations.
- Ablation studies: Feature concatenation, atrous convolution, SELU activation, and dropout each contribute small accuracy improvements in the ablation study.The corresponding experiment is summarized in Table 6.
- Point order: Morton-curve point ordering yields comparable accuracy to other tested orders while improving memory coherence by storing nearby points close together.The result indicates that following an order is sufficient without requiring a particular ordering.
- Neighborhoods and depth: Adaptive-radius k-nearest-neighbor convolution shows no significant accuracy difference from fixed-radius convolution for object recognition.Deeper 8- and 16-layer networks take longer to train and achieve slightly slower accuracy than the basic model.
5. Conclusion
The paper concludes that pointwise convolution supports simple neural networks for point-cloud scene understanding while maintaining competitive accuracy. It also identifies large-scale point-cloud handling as an unresolved challenge addressed only by block partitioning and resampling.
- Pointwise convolution enabled neural networks for scene segmentation and object recognition with point-cloud data.The operator produces local point features and supports fully convolutional designs.
- Large-scale point clouds remain an open challenge because semantic segmentation currently divides scenes into blocks and resamples each block to a fixed number of points.
- 81.4 average per-class accuracy was obtained on ModelNet40, compared with PointNet's 86.3.
- 57.1 average per-class accuracy was obtained on ObjectNN, compared with PointNet's 56.0.
A. Layer Visualization
The layer visualization displays learned 3 × 3 × 3 filters for the object-recognition network, while the paper relates local features to strengths and difficulties in classification.
- Local features work most effectively for structures and common furniture, but tables and desks remain challenging to differentiate.
- Each row visualizes one 3 × 3 × 3 filter, with 27 rows in the first layer and 81 rows in subsequent layers.
- Blue and red encode positive and negative filter values, while white encodes zero.