Source-linked AI summary
Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
Shiyi Lan, Ruichi Yu, Gang Yu, Larry S. Davis
TL;DR
The paper addresses limited explicit modeling of geometric relationships within local 3D point-cloud regions. It introduces GeoConv and Geo-CNN to decompose and angle-weight edge features, achieving state-of-the-art performance on ModelNet40 and KITTI, with rotation analysis scoped to the z-axis.
Problem
Existing point-cloud methods model local regions but give limited attention to explicitly preserving geometric structure among neighboring points.
Method
GeoConv decomposes edge-feature extraction onto three orthogonal bases and aggregates the resulting features according to angles between edge vectors and those bases.
Results
Geo-CNN achieves state-of-the-art performance on the ModelNet40 and KITTI datasets.
Takeaways & Limitations
GeoConv is presented as a generic module that can be integrated into pipelines for 3D shape classification, segmentation, and object detection.
Takeaways & Limitations
The paper’s rotation analysis refers to rotation around the z-axis, though the authors state it can be expanded to other cases.
Abstract
from arXiv · showhide
Recent advances in deep convolutional neural networks (CNNs) have motivated researchers to adapt CNNs to directly model points in 3D point clouds. Modeling local structure has been proven to be important for the success of convolutional architectures, and researchers exploited the modeling of local point sets in the feature extraction hierarchy. However, limited attention has been paid to explicitly model the geometric structure amongst points in a local region. To address this problem, we propose Geo-CNN, which applies a generic convolution-like operation dubbed as GeoConv to each point and its local neighborhood. Local geometric relationships among points are captured when extracting edge features between the center and its neighboring points. We first decompose the edge feature extraction process onto three orthogonal bases, and then aggregate the extracted features based on the angles between the edge vector and the bases. This encourages the network to preserve the geometric structure in Euclidean space throughout the feature extraction hierarchy. GeoConv is a generic and efficient operation that can be easily integrated into 3D point cloud analysis pipelines for multiple applications. We evaluate Geo-CNN on ModelNet40 and KITTI and achieve state-of-the-art performance.
1. Introduction
3D point-cloud methods increasingly process local neighborhoods, but many do not explicitly preserve the geometric relationships among neighboring points. Geo-CNN addresses this with GeoConv, which decomposes edge features along orthogonal directions and aggregates them by vector orientation.
- Voxel methods face insufficient resolution and rapidly increasing memory costs as 3D grids become finer.
- PointNet++ improves local feature extraction hierarchically, but existing point-set methods still ignore local geometric structure.
- EdgeConv models center-neighbor edge features but captures distance while ignoring vector direction, losing local geometric information.
- Current methods may struggle because coordinate variance overwhelms geometric relationships and deep feature hierarchies may not preserve Euclidean structure.
- GeoConv decomposes edge features into three orthogonal bases, applies direction-associated weights, and aggregates features according to angles between the edge vector and bases.
- Geo-CNN stacks GeoConv layers with expanding neighborhoods, uses channel-wise max pooling for permutation invariance, and targets classification, segmentation, and detection.
2. Related Work
Related work develops point-cloud representations through global point aggregation, local neighborhood graphs, and multi-view image representations. Geo-CNN differs by explicitly modeling geometric structure in native 3D point neighborhoods.
- Earlier work used hand-crafted descriptors for local geometric structure, whereas more recent deep networks learn point-cloud representations directly from data.
- PointNet and PointNet++ aggregate point features with symmetric functions, while related methods construct local neighborhood graphs and operate on connecting edges.
- View-based techniques represent 3D objects as collections of 2D views and pool features across orientations, often outperforming a single view.
3. Our Approach
Geo-CNN hierarchically extracts point-cloud features with GeoConv, explicitly preserving local Euclidean geometry through spherical neighborhoods, basis-based edge modeling, and multi-view feature aggregation.
- 3.1. Hierarchical Feature Extraction with Geo-CNN: GeoConv applies to each point and its radius-defined spherical neighborhood, while increasing radii across layers enlarge receptive fields for hierarchical feature extraction.Features from the center point and neighboring edge relationships are aggregated at each layer.
- 3.1. Hierarchical Feature Extraction with Geo-CNN: Neighbor contributions are weighted by a distance function that decreases with point separation and becomes less discriminative among similarly distant points as the radius grows.The distance weighting is defined over the center point, neighboring point, and neighborhood radius.
- 3.2. GeoConv: Local Geometric Modeling with Basis-based Decomposition and Aggregation: GeoConv decomposes each edge vector into orthogonal directional components, applies direction-associated weight matrices, and aggregates them using angle-based coefficients.The neighboring point’s quadrant selects three bases; coefficients cos2(θ_pq,b) represent squared projected-component ratios and sum to 1.
- 3.2. GeoConv: Local Geometric Modeling with Basis-based Decomposition and Aggregation: Basis-based decomposition learns directional representations independently, reducing the complexity of learning from high-variance coordinates while preserving edge geometry through the hierarchy.The method explicitly models geometric relationships between each point and its neighbors.
- 3.3. Approximating 3D Multi-view Augmentation at the Feature Level using Geo-CNN: At the feature level, Geo-CNN approximates multi-view processing by sharing directional edge computations and changing only the aggregation model using recomputed rotated-edge angles.Learned weights fuse the resulting multi-view features.
- 3.4. Implementation: A bottleneck-like GeoConv implementation first extracts lower-dimensional edge features, then enlarges their dimensionality to match center-point features before aggregation.Figure 3 depicts point-feature dimensionalities and the operations used in this structure.
4. Implementation Details
Geo-CNN uses radius-defined spherical neighborhoods and GeoConv layers to extract point features, with application-specific integrations for shape classification and object detection.
- GeoConv Module: Each GeoConv layer maps n × C_in point features to n × C_out features using a radius-defined spherical neighborhood.A center-point weight matrix extracts center features, while a bottleneck reduction module extracts and expands edge features.
- Shape Classification: ModelNet40 classification uses 1,000 sampled points with coordinates and surface normals as six input channels.The architecture includes two branches, including local groups of 16 nearest neighbors and GeoConv processing.
- Shape Classification: The three GeoConv neighborhood radii for ModelNet40 are 0.15, 0.3, and 0.6, followed by channel-wise max pooling for classification.Batch normalization and ReLU follow every fully connected layer and GeoConv reduction layer.
- 3D Object Detection: For detection, GeoConv replaces fully connected layers in Frustum PointNet’s 3D instance-segmentation feature extractor while preserving output dimensionalities.The replacement covers all nine fully connected layers across three MLP modules.
- 3D Object Detection: The detection pipeline removes background points through frustum segmentation, then uses the original box-estimation network for orientation, size, and centroid prediction.GeoConv is implemented the same way as in ModelNet40.
- Implementation Comparison: Compared with baselines, GeoConv weights edge features by distance and uses separate directional weights rather than averaging features and using one weight matrix.These differences define the edge-feature extraction comparison used in the implementation.
5. Experiments
Experiments evaluate Geo-CNN on ModelNet40 shape classification and KITTI 3D detection, including comparisons, ablations, and multi-view augmentation. The reported results show strong benchmark performance and attribute gains to geometric modeling rather than parameter growth alone.
- Experimental Setup: Geo-CNN is evaluated on ModelNet40 classification and integrated into Frustum PointNet for KITTI autonomous-driving detection.The experiments target both CAD-generated point clouds and sensor-acquired real-world point clouds.
- ModelNet40: ModelNet40 contains 12,311 CAD models across 40 categories, with 9,843 training and 2,468 testing samples represented by 10,000 points.The prepared dataset supports sampling smaller point sets as well.
- ModelNet40 Results: Geo-CNN achieves state-of-the-art ModelNet40 classification performance on both reported metrics, with a 1.6% gain from replacing fully connected edge modeling with GeoConv.Feature-level multi-view approximation adds a further 0.5% gain.
- ModelNet40 Results: The ModelNet40 performance reference uses Accuracy Overall because the two reported metrics are very similar.This clarifies which metric the paper means when it refers to performance on this dataset.
- Multi-view Augmentation: The multi-view approximation gain remains robust from 10 to 40 views, changing by less than ±0.1.The reported stability concerns the number of approximated views.
- KITTI Results: The KITTI comparison reports average precision in percent on the validation set and identifies Frustum Geo-CNN as better than both Frustum PointNet baselines.The implementation is based on Frustum PointNet v1.
- KITTI Results: On KITTI, Frustum Geo-CNN outperforms both Frustum PointNet v1 and v2 after replacing the v1 segmentation network’s PointNet feature extractor with GeoConv.KITTI includes Car, Pedestrian, and Cyclist categories evaluated at easy, moderate, and hard difficulty levels.
- Geometric Modeling Ablation: Directly learning geometry from coordinates or learning aggregation coefficients does not improve performance, whereas GeoConv captures local geometric structure effectively.These alternatives are evaluated as Baseline + 3D Coords and GeoConv - Learned-Agg.
6. Conclusion
GeoConv explicitly models local geometric structure, and Geo-CNN hierarchically preserves geometric relationships among points through feature extraction.
- GeoConv explicitly models local geometric structure by decomposing edge feature extraction onto three orthogonal directions and aggregating features using edge-to-basis angles.
- Geo-CNN stacks GeoConv layers to hierarchically extract features while preserving geometric relationships among points in Euclidean space.
- GeoConv achieves state-of-the-art performance on the challenging ModelNet40 and KITTI datasets.