Source-linked AI summary
FPConv: Learning Local Flattening for Point Convolution
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han
TL;DR
Point-cloud analysis needs efficient operations that capture surface geometry without the costs and limitations of volumetric or graph intermediates. FPConv learns projection weights to flatten local neighborhoods onto a 2D grid for regular 2D convolution, improves surface-style methods, matches volumetric-style performance, and complements volumetric convolutions when jointly trained.
Problem
Point-cloud analysis is challenging because sparse, unordered data make efficient local processing difficult, while volumetric methods have high memory and computational costs and surface methods can miss fine details.
Method
FPConv learns projection weights that flatten local point-cloud regions onto a 2D grid, then applies regular 2D convolutions as a surface-style operator.
Results
FPConv significantly improves surface-style convolution methods, achieves comparable performance with state-of-the-art volumetric-style methods, and reaches state-of-the-art results when jointly trained with volumetric convolutions.
Takeaways & Limitations
Surface-style convolution can complement volumetric-style convolution, with joint training boosting overall performance.
Takeaways & Limitations
The reported fusion experiments use rigid KPConv because released deformable-model weights and hyperparameter settings were unavailable.
Abstract
from arXiv · showhide
We introduce FPConv, a novel surface-style convolution operator designed for 3D point cloud analysis. Unlike previous methods, FPConv doesn't require transforming to intermediate representation like 3D grid or graph and directly works on surface geometry of point cloud. To be more specific, for each point, FPConv performs a local flattening by automatically learning a weight map to softly project surrounding points onto a 2D grid. Regular 2D convolution can thus be applied for efficient feature learning. FPConv can be easily integrated into various network architectures for tasks like 3D object classification and 3D scene segmentation, and achieve comparable performance with existing volumetric-type convolutions. More importantly, our experiments also show that FPConv can be a complementary of volumetric convolutions and jointly training them can further boost overall performance into state-of-the-art results.
1. Introduction
FPConv addresses the difficulty of efficient, surface-aware point-cloud analysis by learning local flattening directly and applying 2D convolutions. It improves surface-style methods, matches volumetric-style performance, and benefits from joint training with volumetric convolutions.
- Point-cloud analysis remains challenging because point clouds are sparse and unordered, while existing approaches face limitations in local processing, memory, computation, or surface-detail capture.
- FPConv learns projection weights to flatten local surface neighborhoods onto a 2D plane, then applies regular 2D convolutions.
- FPConv is a novel surface-style convolution that directly learns local flattening for 3D point-cloud feature extraction.
- FPConv improves previous surface-style methods and achieves comparable performance with state-of-the-art volumetric-style methods in classification and segmentation.
- Surface-style and volumetric-style convolutions are complementary, and joint training boosts overall performance into state-of-the-art.
2. Related Work
Prior point-cloud methods use volumetric grids, graphs, or projected planes, each introducing efficiency, topology, viewpoint, or geometric limitations. FPConv instead learns a nonlinear local flattening onto a 2D grid and applies 2D convolutions while retaining comparable volumetric-style performance.
- Volumetric-style methods use grids or point-based approximations, but grids incur high computational load and low representation efficiency.
- Graph-style methods establish point relationships for efficient convolution but can be impractical to construct and sensitive to local topology.
- Surface-style methods operate on surface geometry, but intermediate projections can introduce redundant multi-view representations, viewpoint ambiguity, tangent-estimation dependence, or suboptimal linear projections.
- Although surface learning loses one-dimensional information, FPConv achieves comparable performance with volumetric-style convolutions and can be integrated with them.
3. FPConv
FPConv models surface convolution by learning to flatten each local neighborhood onto a discrete 2D grid, then applying conventional 2D convolution. Its learned projection weights are normalized to support either dense or sparsity-preserving grid features.
- Local surface convolution: FPConv begins with a local neighborhood around point p and represents its surface signal using a virtual 2D plane.The neighborhood contains N points, and the plane is discretized to an Mw × Mh grid.
- Local flattening: The surface convolution is reformulated as learning projection weights that combine projection and interpolation into one weight matrix.The resulting matrix depends on each point’s location relative to the neighborhood center.
- Local flattening: The learned module uses permutation-invariant local distribution features together with local coordinates to adapt projection weights to neighborhood geometry.This design addresses unordered point clouds while conditioning the projection on local geometric information.
- Feature extraction: After flattening, 2D convolution extracts features from the grid plane, with global convolution or pooling producing a local feature vector.Normalization is introduced because uneven pixel intensities can destabilize training and hinder convergence.
- Grid normalization: Dense grid normalization uses softmax so the intensities received at each pixel sum to one, similarly to bilinear interpolation.Softmax also avoids division by zero during normalization.
- Grid normalization: Sparse grid normalization first balances intensity emitted by each neighbor, then balances intensity received at each pixel while preserving continuous sparsity.The second step uses a column-based normalization scheme rather than simple division by column sums.
- Network integration: The broader network uses residual FPConv blocks and multi-scale segmentation architectures with downsampling, upsampling, and skip connections.The residual block stacks shared MLPs around FPConv, while downsampling uses farthest point sampling and pooling.
4. Architecture
FPConv networks combine local feature extraction, multi-scale point-cloud processing, and complementary convolution operators. Fusion is implemented either in parallel convolutional blocks or at a later feature level, depending on operator compatibility.
- Architecture: The bottleneck residual FPConv block stacks shared MLPs around FPConv, reducing and restoring feature dimensions like 1 × 1 convolutions.This block forms the core unit for classification and segmentation networks.
- Multi-Scale Analysis: Multi-scale analysis uses farthest point sampling, neighborhood max-pooling, FPConv on downsampled points, and K-nearest-neighbor interpolation for upsampling.Encoder-decoder segmentation models use four downsampling and upsampling layers, while classification uses global pooling followed by fully connected layers.
- Fusing Two Convolutions: FPConv can be fused with other convolutions because complementary operators may improve performance beyond either operator alone.The paper specifically motivates combining surface-style and volumetric-style convolutions.
- Fusing Two Convolutions: The parallel residual strategy applies multiple compatible convolutions to one input in parallel and concatenates their outputs as a fused feature.This strategy suits methods sharing point-cloud inputs and downsampling, such as PointNet++ modules and PointConv.
- Fusing Two Convolutions: For incompatible operators, the network concatenates output features near the end and applies a small fusion network.This strategy is used for operators with differing inputs or downsampling schemes, including TextureNet and KPConv.
5. Experiments
FPConv is evaluated for ModelNet40 classification and S3DIS and ScanNet semantic segmentation. It outperforms prior surface-style methods and reaches strong large-scene segmentation results, including a fused state-of-the-art S3DIS model.
- Datasets: Experiments evaluate FPConv on ModelNet40 shape classification and S3DIS and ScanNet large-scale point-cloud segmentation.The study uses standard datasets spanning object classification and indoor scene segmentation.
- Classification: ModelNet40 contains 12,311 meshed models across 40 categories, with 9,843 training and 2,468 testing models; FPConv achieves state-of-the-art performance among surface-style methods.Normals and z-axis rotation and jittering augmentations are used.
- Segmentation: S3DIS evaluation reports mean class-wise IoU, overall point-wise accuracy, and mean class-wise accuracy using Area 5 and six-fold cross-validation settings.ScanNet evaluation reports mIoU on its benchmark.
- Segmentation: 63.9% mIoU on the ScanNet benchmark surpasses the previous best surface-style method by 7.3%.FPConv also achieves state-of-the-art performance on S3DIS when fused with KPConv.
- Segmentation: FPConv exceeds KPConv on selected S3DIS classes, including ceiling, floor, and board, which are identified as relatively flat objects.This observation motivates the subsequent curvature analysis.
6. Ablation Study
Ablations examine convolution fusion, curvature-dependent performance, and FPConv configuration choices. They show complementary behavior between surface-style and volumetric-style operators, while higher plane resolution improves performance at greater memory cost.
- Ablation Study: The ablation study covers fusion of surface-style and volumetric-style convolutions plus normalization and plane-size configurations.These experiments test both architectural combinations and detailed FPConv settings.
- Performance vs. Curvature: FPConv outperforms PointConv and KPConv at small curvatures but performs worse on structures with large curvatures.Incorrect predictions concentrate in high-curvature regions, while FPConv performs well in flatter areas.
- Fusion: FPConv and volumetric-style KPConv show opposite strengths: FPConv is better on flat areas, whereas KPConv is better on rough areas.The reported fusion experiments support treating the two convolution types as complementary.
- Fusion: Fusing FPConv with PointConv or KPConv produces a great improvement, while fusing PointConv with KPConv produces little improvement.The visualizations likewise report better flat-structure capture by FPConv, better complex-structure capture by KPConv, and stronger combined results.
- Configuration Ablations: Sparse-norm performs better than dense-norm, and higher-resolution grid planes may improve performance while increasing memory cost.Table 4 compares 6x6 and 5x5 plane sizes on S3DIS Area 5.
7. Conclusion
FPConv learns projection weights to flatten local point-cloud regions onto a 2D grid before applying regular 2D convolutions. Experiments show improved surface-style convolution performance, complementary behavior with volumetric convolutions, and state-of-the-art results from joint training.
- Conclusion: Experiments show that FPConv significantly improves the performance of surface-style convolution methods.The conclusion also reports comparable performance with volumetric-style convolutions.
- Conclusion: Surface-style convolution can complement volumetric-style convolution, and jointly training them can boost performance to state-of-the-art.The paper presents this complementarity as a promising direction for 3D feature learning.
Supplementary
The supplementary material reports fusion results, additional large-scale scene-segmentation evaluations, and qualitative and quantitative findings.
- The supplementary material includes fusion results for FPConv with PointConv on ScanNet.
- It includes fusion results for FPConv with KPConv-deform on S3DIS.
- Additional qualitative and quantitative results are provided for large-scale scene-segmentation tasks.
A. More results of the proposed fusion strategy
Experiments evaluate FPConv fusion with PointConv on ScanNet and with KPConv-deform on S3DIS under reported benchmark settings. The S3DIS fusion model reaches state-of-the-art among existing methods.
- Experiments fuse FPConv with PointConv on ScanNet using matched architecture and hyperparameter settings.The ScanNet experiments reduce sampled points to 8k within a 1.5m × 1.5m block for all methods.
- Table 5 reports quantitative segmentation results on the ScanNet evaluation dataset, including a PointConv reimplementation.
- The FPConv and KPConv-deform fusion model wins all existing methods on S3DIS and reaches the state-of-the-art.
B. Parameter Comparison
The parameter comparison contrasts individual convolution operators with fusion forms, emphasizing that different-type fusion improves performance even after reducing fusion-block channel size.
- Fusion of same-type convolution operators does not bring significant improvement and can perform worse.
- FPConv ⊗PointConv performs much better after fusion, even when the fusion block's middle channel size is halved.The comparison covers trainable parameters for PointConv, FPConv, and their fusion forms on ScanNet.
C. More Results on Segmentation Tasks
Detailed S3DIS evaluations show FPConv performs better on flat-shaped objects, while KPConv performs better on complex structures; additional visualizations cover ScanNet and S3DIS segmentation.
- On S3DIS 6-fold cross validation, FPConv achieves higher scores on flat-shaped objects such as ceilings, floors, tables, and boards.
- KPConv performs better than FPConv on complex structures in the detailed S3DIS comparison.
- Table 8 reports detailed semantic-segmentation scores on S3DIS 6-fold cross validation.
- Figures 12 and 13 provide visual segmentation results for ScanNet and qualitative comparisons on S3DIS area 5.Figure 13 marks fusion at the final feature level with ⊕.