Source-linked AI summary
Point Attention Network for Semantic Segmentation of 3D Point Clouds
Mingtao Feng, Liang Zhang, Xuefei Lin, Syed Zulqarnain Gilani, Ajmal Mian
TL;DR
Semantic segmentation of 3D point clouds is difficult because the data are sparse, irregular, unordered, and incompletely observed. The paper introduces a point attention network that combines multi-directional local edge attention with point-wise spatial attention for long-range context, and reports performance on par with or better than existing state-of-the-art methods on challenging benchmarks.
Problem
3D point cloud semantic segmentation must handle sparse, irregular, unordered data with missing regions, unlike image segmentation on regular grids.
Method
The network combines multi-directional LAE-Conv local graph aggregation with point-wise spatial attention in an encoder-decoder architecture.
Results
The method performs on par with or better than existing state-of-the-art methods on challenging point-cloud benchmark datasets.
Takeaways & Limitations
The proposed architecture adaptively integrates local point features and long-range contextual information for 3D point cloud semantic segmentation.
Takeaways & Limitations
Multi-view frameworks may not fully retain 3D geometric information in their 2D projections.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) have performed extremely well on data represented by regularly arranged grids such as images. However, directly leveraging the classic convolution kernels or parameter sharing mechanisms on sparse 3D point clouds is inefficient due to their irregular and unordered nature. We propose a point attention network that learns rich local shape features and their contextual correlations for 3D point cloud semantic segmentation. Since the geometric distribution of the neighboring points is invariant to the point ordering, we propose a Local Attention-Edge Convolution (LAE Conv) to construct a local graph based on the neighborhood points searched in multi-directions. We assign attention coefficients to each edge and then aggregate the point features as a weighted sum of its neighbors. The learned LAE-Conv layer features are then given to a point-wise spatial attention module to generate an interdependency matrix of all points regardless of their distances, which captures long-range spatial contextual features contributing to more precise semantic information. The proposed point attention network consists of an encoder and decoder which, together with the LAE-Conv layers and the point-wise spatial attention modules, make it an end-to-end trainable network for predicting dense labels for 3D point cloud segmentation. Experiments on challenging benchmarks of 3D point clouds show that our algorithm can perform at par or better than the existing state of the art methods.
1. Introduction
The paper addresses semantic segmentation of sparse, irregular, unordered 3D point clouds by combining local geometric modeling with long-range contextual reasoning. It proposes LAE-Conv, point-wise spatial attention, and a U-shaped network for dense labeling.
- Motivation: 3D point cloud semantic segmentation is challenging because point clouds are sparse, irregular, unordered, and contain missing regions.These properties make direct transfer of image-oriented deep-learning methods incomplete.
- Method: LAE-Conv searches neighborhood points along multiple directions, assigns edge attention coefficients, and aggregates neighbor features as a weighted sum.The multi-directional search uses 16 directions within a ball query to model local geometry more generally across space.
- Motivation: Local LAE-Conv receptive fields learn hierarchical features, but existing approaches do not sufficiently model long-range contextual relationships among points of the same categories.The paper identifies long-range context as important for semantic segmentation.
- Architecture: The network stacks point-wise spatial attention after LAE-Conv layers at multiple stages within an encoder-decoder architecture.This design targets both more accurate local geometric features and long-range relationships.
- Results: Experiments report performance on par with or better than existing state-of-the-art methods on challenging benchmark datasets.The paper also reports that its point attention block can generalize to other networks and improve their performance.
2. Related Work
Prior 3D point-cloud methods use indirect representations, graph operations, or direct point processing, but commonly struggle to preserve geometry or jointly capture local detail and long-range context. The paper positions its attention-based point-cloud network as a response to this gap.
- Indirect methods: Indirect methods transform point clouds into canonical forms such as voxels, lattices, projections, or other ordered representations before applying conventional operations.Voxel methods can require substantial memory and computation, while multi-view projections may not fully retain 3D geometry.
- Indirect methods: Multi-view frameworks process predefined 2D projections, but their projections do not always retain the full 3D geometric information.This is identified as the main drawback of the multi-view approach.
- Graph methods: Graph convolutional methods operate on irregular data in spectral or spatial domains, but some local methods do not account for irregular point distributions or fully exploit local correlations.The paper contrasts these limitations with its attention-based local feature learning.
- Point-cloud methods: PointNet processes unordered points with MLPs and global max pooling, whereas PointNet++ adds local geometric features but treats points within local regions individually.The related-work discussion identifies missing neighborhood relationships as a limitation of PointNet++.
- LAE-Conv operation: The described LAE-Conv operation searches neighbors, shifts them into a local coordinate system, computes normalized edge attention, aggregates features, and applies an MLP.The procedure is presented as the implementation sequence for filtering a central point.
- Research gap: The paper identifies a common limitation: prior methods cannot simultaneously exploit fine local details and long-range contextual information.Its proposed network combines edge-attention-based local geometry with interactions over longer distances.
3. Proposed Approach
The proposed point attention network combines LAE-Conv for direction-aware local geometric features with point-wise spatial attention for long-range contextual relationships. An encoder–decoder architecture uses these modules to predict dense point-cloud segmentation labels.
- Local Attention-Edge Convolution: LAE-Conv divides each local ball into directional bins, selects nearby points, and constructs a multi-directional neighborhood graph.The search uses 16 directions and selects m nearest points per bin, with radius and m set as hyperparameters.
- Local Attention-Edge Convolution: Attention coefficients weight neighboring edges, and commutative aggregation forms each central-point feature as a weighted sum.The edge coefficients are normalized across the reference point’s neighbors before aggregation.
- Point-wise Spatial Attention: The point-wise spatial attention module computes correlations among all points and selectively aggregates long-range contextual features.MLP-transformed features produce spatial relationships, which are normalized and applied to transformed point features before residual summation.
- Point Attention Network: The network combines LAE-Conv layers and spatial attention modules in an encoder–decoder with downsampling, upsampling, and skip connections.The architecture is designed for dense point-label prediction across multiple object scales.
4. Comparison with Existing Methods
The point attention network generalizes established point-cloud operators by using attention-weighted local aggregation and flexible neighborhood contributions. Its formulation also relates PointNet++ and DGCNN to special cases or contrasting design choices within this framework.
- PointNet++ becomes a special case of LAE-Conv when one neighbor receives coefficient 1 and all others receive 0.
- LAE-Conv assigns different attention coefficients to neighbors, allowing their contributions to the central point to vary.
- DGCNN gives all neighbors equal contribution, corresponding to the proposed operator when every edge coefficient equals 1.
- DGCNN keeps a constant receptive field of K across layers, whereas PointCNN changes neighborhood size through its dilation ratio.
- The point-wise operation models long-range dependency, increasing correlation between points with more similar feature representations.
5. Experiments and Discussion
Experiments evaluate the point attention network on ScanNet, S3DIS, and ShapeNet, including quantitative comparisons, qualitative scenes, efficiency, and ablations. The method achieves strong segmentation results, with global context helping rare categories, while attention placement and neighborhood selection affect performance.
- Experimental setup: The network is evaluated on ShapeNet, ScanNet, and S3DIS, with ablations and comparisons against existing state-of-the-art methods.ScanNet contains 1,513 scans across 21 categories and is evaluated using xyz coordinates as input features.
- ScanNet results: On ScanNet, the point attention network achieves the highest reported mIoU and OA among the compared PointNet++ and PointCNN baselines.The comparison uses mean per-class IoU and per-voxel overall accuracy, with baseline results taken from the original papers.
- Efficiency: The proposed architecture improves segmentation results with only marginal extra computation cost compared with representative methods.Model size and inference time are compared using TensorFlow model files and average runtime over five scene evaluations.
- ScanNet results: Qualitative ScanNet comparisons show more regular predictions for embedded or incomplete structures, although all methods struggle with rare categories and ambiguous shapes.Examples include windows, doors, skeletal tables, bookshelves, chairs near floors, and irregular desks.
- Ablation studies: Ablations indicate that excessive attention blocks do not improve performance, lower-dimensional placement can deteriorate results, and neighborhood counts that increase parameters can reduce accuracy.The selected configuration places three attention blocks after LAE-Conv layers 3, 4, and 5; multi-directional search is compared with KNN and ball query.
- S3DIS results: On S3DIS, the method outperforms PointNet, SPGraph, RSNet, 3DRCNN, and PointCNN, including higher accuracy on several rare object classes.The reported rare categories include beam, column, window, board, and clutter.
6. Conclusion
The proposed point attention network combines local geometric modeling with long-range contextual information for 3D point cloud semantic segmentation, achieving performance at or above existing state-of-the-art methods.
- The network adaptively integrates local point features and long-range contextual information for 3D point cloud semantic segmentation.
- LAE-Conv captures geometric details through attention on a local graph formed by each point and its neighborhood.
- The point-wise spatial attention module refines LAE-Conv features and can improve the accuracy of other networks.
- A U-shaped network combines LAE-Conv and point-wise spatial attention modules for dense semantic labels.
- Experiments on challenging benchmark datasets show performance at or above existing state-of-the-art methods, both quantitatively and qualitatively.