Source-linked AI summary
Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia
TL;DR
3D semantic scene labeling needs stronger modeling of relations between points and their local context without the information loss and resource demands of voxelization. The paper introduces interacting point and hierarchical edge branches, with multiscale edge refinement and semantic-consistency supervision. It reports strong performance on S3DIS and ScanNet, including 6.2% higher absolute mIoU on the ScanNet v2 test set than other point-based methods.
Problem
Voxelization can lose information and require substantial memory and computation, while point-based models provide weak connections to local context despite its importance for segmentation.
Method
The method combines an encoder-decoder point branch with a hierarchical edge branch that integrates multiscale point features, feeds edge context back to points, and supervises semantic consistency.
Results
6.2% higher absolute mIoU and 11.2% better relatively than other point-based methods on the ScanNet v2 test set, with strong results also reported on S3DIS.
Takeaways & Limitations
The interacting point-edge framework uses local semantic relationships and multiscale contextual information for 3D semantic segmentation across S3DIS and ScanNet.
Abstract
from arXiv · showhide
We achieve 3D semantic scene labeling by exploring semantic relation between each point and its contextual neighbors through edges. Besides an encoder-decoder branch for predicting point labels, we construct an edge branch to hierarchically integrate point features and generate edge features. To incorporate point features in the edge branch, we establish a hierarchical graph framework, where the graph is initialized from a coarse layer and gradually enriched along the point decoding process. For each edge in the final graph, we predict a label to indicate the semantic consistency of the two connected points to enhance point prediction. At different layers, edge features are also fed into the corresponding point module to integrate contextual information for message passing enhancement in local regions. The two branches interact with each other and cooperate in segmentation. Decent experimental results on several 3D semantic labeling datasets demonstrate the effectiveness of our work.
1. Introduction
The paper addresses inefficient voxel-based processing and weak local-context modeling in point-based 3D segmentation by explicitly modeling semantic relations through hierarchical edges. Its interacting point and edge branches achieve strong results on S3DIS and ScanNet.
- Voxelizing sparse 3D scenes loses information and requires high-resolution models, increasing memory and computation costs.
- PointNet++ captures local regions hierarchically, but its MLP-based point processing provides only weak connections to local context.
- The method explicitly connects points with contextual neighbors and adds a hierarchical edge branch with auxiliary edge supervision.
- The edge branch progressively integrates point features across layers, while edge features return to the point branch to provide local geometric and semantic context.
- Semantic-consistency supervision regularizes final edge features, increasing discrimination between inter-category and intra-category point pairs.
2. Related Work
Prior work processes 3D data with sparse voxel, multiview, surface, point, and graph-based methods, but point and graph approaches have limitations in local aggregation and multiscale construction. This paper uses hierarchically propagated edge features to fuse multiscale context and support message passing without large memory overhead.
- PointNet++ introduces hierarchical local aggregation, but simple max-pooling does not fully utilize regional information.
- Prior graph methods learn local relationships through graph convolutions, edge features, point-set kernels, or neighborhood connections.
- Unlike fixed-resolution or independently constructed multiscale graphs, this method hierarchically constructs one graph and propagates edge features across scales.
- Multiscale edge propagation enables longer-distance message passing without large memory overhead, while semantic edge loss increases discrimination among point features from different categories.
- The point branch concatenates each point feature with max-pooled corresponding edge features, differing from methods that learn adaptive weights or kernels along edges.
3. Our Approach
The approach couples a PointNet++-style encoder-decoder point branch with a hierarchical edge branch that progressively integrates multi-layer point features. Edge features guide point prediction through local contextual message passing, while semantic-consistency supervision regularizes final edge representations.
- 3.1. Edge Branch: The model combines a hierarchical point encoder-decoder with an edge branch that progressively fuses point features across layers.The edge branch receives features from corresponding point modules and previous edge modules during decoding.
- 3.2.2 Hierarchical Architecture: Edge upsampling maps previous-layer edge features onto the current graph, enabling hierarchical feature enrichment and longer-range edge message passing.The edge module combines current-layer point features with upsampled previous-layer edge features, and the final edge features preserve information from multiple layers.
- 3.1.1 Edge Module: Each edge feature encodes both endpoint features and their relative 3D position, preserving point information while representing local relationships.The edge function concatenates the two point features with pj − pi before producing the edge feature.
- 3.1.2 Incorporation of Edges in Point Prediction: Edge features are max-pooled for each point and concatenated with its point feature, supplying regional geometry and semantic context for message passing.This region guidance enlarges the effective message-passing range and lets point feature extraction see farther within each layer.
- 3.2.1 Graph Initialization: The graph starts at the coarsest layer and is enlarged during decoding using current-layer coordinates together with connections inherited from the previous layer.The initial graph connects each point to its k0-nearest neighbors, including itself; later layers construct candidate local connections and check previous-layer edge support.
- 3.3. Loss Function: Semantic-consistency supervision makes edge features distinguish intra-class from inter-class feature pairs and provides auxiliary supervision for point features.The final edge features are regularized according to whether the two connected points are semantically consistent.
4. Experiments
Experiments on S3DIS and ScanNet v2 show strong semantic segmentation performance, with ablations examining edge construction, message passing, and hierarchical graph connectivity.
- Experimental Setup: The framework is evaluated on S3DIS and ScanNet v2 using mIoU, mAcc, and OA.Ablations are reported on ScanNet v2 validation and S3DIS Area 5.
- Main Results: 61.85% mIoU on S3DIS Area 5 exceeds the former best by 3.58%, while the model leads all three reported metrics.Table 1 evaluates semantic segmentation on S3DIS Area 5.
- Main Results: The method achieves first place across all three metrics in S3DIS six-fold cross validation.The comparison covers different architectures on the S3DIS dataset.
- Main Results: On the ScanNet v2 test set, the approach improves absolute mIoU by 6.2% and relative mIoU by 11.2% over other point-based methods.All compared methods use point clouds with RGB input without voxelization.
- Ablation Study: Concatenation gives the best edge-function result because it preserves most point information, while ConcatSub performs similarly by allowing two-point features to be restored.Summation, subtraction, and Hadamard product are reported to lose point-feature information.
- Ablation Study: Max-pooling followed by concatenation yields larger performance gains than graph-convolution-style adaptive aggregation for message passing.The authors attribute this to preserving each point’s distinctiveness while representing its relation to the neighborhood.
- Ablation Study: Removing hierarchical graph construction reduces ScanNet results to 57.01/66.52/83.57 mIoU/mAcc/OA, versus 63.36/72.61/86.13 for the full framework.The connected edge branch incorporates point features from different layers for edge-feature learning.
5. Conclusion
The hierarchical point-edge interaction network uses an edge branch alongside the point branch to integrate multi-layer point features and contextual information for semantic segmentation. Its authors report high-quality predictions and applicability across different datasets, while proposing multi-range edge construction as future work.
- The network combines an edge branch with the encoder-decoder point branch to use local semantic relationships through edges.The edge branch progressively integrates point features, and edge features are incorporated into the point branch.
- The hierarchical graph framework progressively integrates point features from different layers into the edge branch.Final edge features are supervised by semantic consistency to regularize point features.
- Edge features provide contextual information that enhances point-cloud semantic segmentation.The interaction between branches uses local context through edges for point prediction.
- The authors report high-quality point predictions and generality across different datasets.
- Future work will explore multi-range edge construction to gather closerange and long-distance contextual information.