PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers

Xumin Yu, Yongming Rao, Ziyi Wang, Jiwen Lu, Jie Zhou

arXiv:2301.04545v1cs.CVcs.AI

TL;DR

Point cloud completion must recover geometry missing from incomplete, unordered sensor observations, while conventional approaches can lose fine-grained information. PoinTr reformulates completion as Transformer-based set-to-set translation, and AdaPoinTr adds adaptive queries and denoising alongside geometry-aware modeling. The paper reports state-of-the-art results across diverse object benchmarks, extends the approach to scene-level completion, and introduces more realistic evaluation datasets.

  • Problem

    Incomplete point clouds from occlusion and sensor limitations require reconstruction, while existing encoder-decoder methods can lose fine-grained information in a single encoded feature.

  • Method

    PoinTr uses geometry-aware Transformer encoder-decoder set-to-set translation with point proxies; AdaPoinTr adds adaptive query generation and auxiliary denoising.

  • Results

    The method establishes state-of-the-art performance across object-completion benchmarks, including newly proposed diverse benchmarks, and extends to scene-level completion.

  • Takeaways & Limitations

    The architecture supports point-cloud completion across diverse object settings and scene-level tasks while preserving geometric information through Transformer-based modeling.

  • Takeaways & Limitations

    Transformer processing assumes order-invariant point-proxy features and is motivated by computational constraints on directly processing long point sequences.

Abstract

from arXiv · show

In this paper, we present a new method that reformulates point cloud completion as a set-to-set translation problem and design a new model, called PoinTr, which adopts a Transformer encoder-decoder architecture for point cloud completion. By representing the point cloud as a set of unordered groups of points with position embeddings, we convert the input data to a sequence of point proxies and employ the Transformers for generation. To facilitate Transformers to better leverage the inductive bias about 3D geometric structures of point clouds, we further devise a geometry-aware block that models the local geometric relationships explicitly. The migration of Transformers enables our model to better learn structural knowledge and preserve detailed information for point cloud completion. Taking a step towards more complicated and diverse situations, we further propose AdaPoinTr by developing an adaptive query generation mechanism and designing a novel denoising task during completing a point cloud. Coupling these two techniques enables us to train the model efficiently and effectively: we reduce training time (by 15x or more) and improve completion performance (over 20%). We also show our method can be extended to the scene-level point cloud completion scenario by designing a new geometry-enhanced semantic scene completion framework. Extensive experiments on the existing and newly-proposed datasets demonstrate the effectiveness of our method, which attains 6.53 CD on PCN, 0.81 CD on ShapeNet-55 and 0.392 MMD on real-world KITTI, surpassing other work by a large margin and establishing new state-of-the-arts on various benchmarks. Most notably, AdaPoinTr can achieve such promising performance with higher throughputs and fewer FLOPs compared with the previous best methods in practice. The code and datasets are available at https://github.com/yuxumin/PoinTr

1 INTRODUCTION

Point cloud completion addresses incomplete sensor data by reconstructing missing geometry. PoinTr and AdaPoinTr use Transformer-based set-to-set translation, geometry-aware modeling, adaptive queries, denoising, and broader benchmarks to improve completion across object and scene settings.

  • Sensor artifacts such as occlusion and limited resolution make recovering complete point clouds from sparse observations important.
  • PoinTr reformulates completion as set-to-set translation, using a Transformer encoder-decoder to model interactions between known and missing point groups.
  • The model represents local point regions as point proxies and adds geometry-aware blocks to explicitly model 3D geometric relationships.
  • AdaPoinTr adds adaptive query generation and auxiliary denoising to support diverse incompleteness and stabilize efficient training.The denoising task addresses low-quality query initialization, while adaptive queries handle cases with substantially different missing-point ratios.
  • The paper introduces ShapeNet-55/34 and Projected-ShapeNet-55/34 to provide more diverse and realistic incomplete-point-cloud benchmarks.Online cropping is flexible and efficient, whereas noised back-projecting better approximates real scans.
  • The approach extends to scene-level completion through a geometry-enhanced semantic framework that supplements voxel models with fine-grained geometric information.
  • Experiments report state-of-the-art performance across object-completion benchmarks and first place in the MVP Completion challenge.

2 RELATED WORK

Related work spans structured voxel or distance-field methods, unstructured point-cloud completion, semantic scene completion, and Transformer architectures. The field trades computational efficiency, fine-grained detail, geometric structure, and long-range dependencies across these approaches.

  • 3D Shape Completion: Voxel grids and distance fields enable 3D convolutions for shape completion but incur heavy memory consumption and computational burden.
  • 3D Shape Completion: PointNet and its variants process unordered 3D coordinates directly, motivating point-cloud completion methods such as PCN's encoder-decoder with FoldingNet.
  • Semantic Scene Completion: Semantic Scene Completion jointly predicts volumetric occupancy and semantic labels from a single-view depth map or RGBD image.
  • PoinTr Pipeline: PoinTr's pipeline downsamples partial clouds, extracts local features, adds position embeddings, predicts missing-part proxies with a Transformer, and reconstructs points coarse-to-fine.
  • Transformers: Transformers combine encoder-decoder, self-attention, and cross-attention mechanisms to exploit encoder information and model long sequences.

3 POINT CLOUD COMPLETION

PoinTr reformulates point cloud completion as set-to-set translation with Transformer encoder-decoder generation over point proxies. Its design combines position-aware local representations, geometry-aware attention, adaptive queries, and unified reconstruction of known and missing parts.

  • Set-to-Set Translation: PoinTr converts point cloud completion into set-to-set translation, mapping partial point proxies to proxies for the missing point cloud.The Transformer encoder produces features V, while dynamic decoder queries Q generate missing proxies H.
  • Point Proxy: Point proxies summarize local regions using DGCNN features around farthest-sampled centers, with an MLP encoding each proxy’s global location.This representation limits the quadratic attention cost that would result from directly processing all input points.
  • Geometry-aware Transformer Block: The geometry-aware Transformer block uses kNN neighborhoods to model local geometric relations alongside feature-based self-attention.Neighbor features are aggregated with a linear layer and max-pooling, then combined with the attention features.
  • Query Generation: Dynamic queries are generated from encoder outputs so decoder states reflect the sketch of the complete point cloud rather than fixed initial queries.The query generator summarizes encoder features, projects coordinates, and produces query embeddings for missing proxies.
  • Multi-Scale Point Cloud Generation: The multi-scale generation framework reuses query-produced coordinates as local centers and reconstructs neighboring points around them.Predicted proxies and recovered point clouds are both supervised during training before concatenating the recovered missing part with the input.
  • Adaptive Denoising Queries: AdaPoinTr concatenates known and predicted parts at the proxy level, then uses a shared reconstruction model to produce the complete point cloud.Adaptive selection constrains the total number of retained queries while allowing flexible contributions from the input and output query sets.

4 POINTR FOR SEMANTIC SCENE COMPLETION

The paper extends point cloud completion to semantic scene completion, where models must infer occluded geometry and semantic labels from incomplete observations. Its geometry-enhanced framework supplements conventional scene completion models with Transformer-based point-wise interactions.

  • Task definition: Semantic Scene Completion jointly completes a voxelized 3D scene and predicts object semantic labels from a single-view depth map or RGBD image.Scenes are voxelized at 60×36×60 resolution, with classification performed for each voxel.
  • Challenges: Occluded-space geometry remains an open challenge because incomplete local observations create ambiguity in understanding the whole scene.Existing approaches encode visible geometry with depth-derived TSDFs and use RGB images for semantic information.
  • Framework: The proposed framework combines a conventional 3D-convolutional semantic scene model with a geometry-enhanced block built using Transformers.The block is designed to capture more precise geometric information and point-wise interactions from point clouds converted from depth maps.
  • Training and evaluation: The scene-completion objective uses voxel-wise cross-entropy between predicted class probabilities and ground-truth voxel labels.Empty voxels are treated as an additional category, and evaluation includes observed and occluded voxels in the view frustum.

5 EXPERIMENTS

Experiments cover diverse object-completion benchmarks, established datasets, real-world LiDAR data, and semantic scene completion. AdaPoinTr consistently achieves strong completion, transfer, efficiency, and scene-level results across these evaluations.

  • ShapeNet-34: AdaPoinTr achieves the best performance on both seen and unseen ShapeNet-34 categories, demonstrating strong transferability to novel object categories.Models trained on 34 seen categories are evaluated on 21 categories absent from training.
  • Projected-ShapeNet-55: 1.1 CD-ℓ1 improvement and 0.086 F-Score@1% improvement over PoinTr are reported on the evaluated categories and overall result.AdaPoinTr obtains the best performance on the ten reported categories and overall CD.
  • Existing benchmarks: AdaPoinTr establishes new state-of-the-art performance on PCN and achieves better qualitative and quantitative performance on real-world KITTI LiDAR scans.The method also ranks first and wins the first prize in the MVP completion challenge.
  • Ablation study: The query generator improves the ablation baseline by 0.34 Chamfer distance, while adaptive query generation adds about 1.28 improvement to PoinTr.The study also finds that a single geometric block can be sufficient, whereas adding more blocks may cause over-fitting.
  • Efficiency and scene completion: The method reports favorable performance with practical efficiency, while its geometry-enhanced block improves existing semantic scene completion models on NYUV2 and NYUCAD.The block also improves geometric recognition of scene instances such as tables and chairs.

6 CONCLUSION

The paper introduces PoinTr as a Transformer-based set-to-set formulation for point cloud completion and extends it toward diverse benchmarks and scene-level tasks. Its results support applying the architecture to broader 3D completion problems.

  • Conclusion: PoinTr reformulates point cloud completion as set-to-set translation and applies a Transformer architecture to achieve state-of-the-art performance.The paper also introduces more challenging diverse-completion benchmarks and verifies usefulness for scene-level tasks.
  • Future direction: The authors identify extending the Transformer architecture to additional 3D tasks as an interesting future direction.

APPENDIX A TECHNICAL DETAILS ON TRANSFORMERS

The appendix describes the Transformer encoder-decoder used to transform point-proxy inputs into predicted proxies for missing points. Its attention layers combine projected queries, keys, and values, followed by feed-forward processing.

  • Encoder-decoder architecture: The encoder processes point proxies through stacked self-attention and feed-forward layers, while the decoder uses query embeddings and encoder memory to predict missing-point proxies.The predicted proxies represent the missing part of the point cloud.
  • Multi-head attention: Multi-head attention jointly attends to information from different representation subspaces at different positions.The mechanism takes values V, keys K, and queries Q as inputs.
  • Attention projections: Attention heads use linear projections into separate subspaces, with d_k denoting the input-feature dimension.The output is combined using the output linear-layer weights W^O.
  • Feed-forward network: Each feed-forward network uses two linear layers with ReLU activations and dropout.

APPENDIX B IMPLEMENTATION DETAILS

The implementation uses point proxies extracted with lightweight DGCNN layers, while Transformer and geometry-aware-block settings vary between object completion and semantic scene completion. Qualitative visualizations illustrate predicted centers and completed point clouds against inputs and ground truth.

  • Point Proxies: Point proxies are extracted by hierarchically downsampling the input to center points and applying lightweight DGCNN layers to capture local geometric relationships.The feature-extraction pipeline begins with a linear layer and successive DGCNN layers.
  • Object Point Cloud Completion: Object completion uses a 6-layer encoder, 8-layer decoder, 6 attention heads, and hidden dimension 384, with kNN settings of 16 and 8.These settings apply to the Transformer and the DGCNN feature extractor or geometry-aware block as specified.
  • Semantic Scene Completion: Semantic scene completion uses 3-layer encoder and decoder Transformers, with the same kNN settings of 16 and 8 for feature extraction and geometry-aware processing.The scene-completion training configuration uses SGD with cosine learning-rate scheduling.
  • Qualitative Visualization: Predicted point proxies provide centers that support accurate point-center prediction and subsequent point-cloud completion, with ground truth shown for comparison.The visualization contrasts partial inputs, predicted centers, completed outputs, and ground-truth clouds.

APPENDIX C QUALITATIVE RESULTS

Qualitative results show the method recovering point-cloud structure and details across object completion, diverse incomplete patterns, and semantic scene completion. The geometry-enhanced block is presented as preserving geometric information and guiding scene-level predictions.

  • Predicted Centers: A coarse-to-fine strategy first predicts local centers and then adds nearby points to produce the final point cloud.The predicted proxies represent the overall structure, while added points provide finer details.
  • Point Cloud Completion: On ShapeNet-55, PoinTr is reported to recover details more strongly and remain more robust across varied incomplete patterns.These observations come from additional qualitative completion results.
  • Semantic Scene Completion: On NYUCAD semantic scene completion, the Geometry-Enhanced block is reported to retain more geometric information and guide the final prediction.The qualitative results are shown for additional scene-completion examples.

Ablation on the number of point proxies and the number

Ablations show that increasing input point proxies or generated queries improves performance, but proxy counts face attention-complexity limits and saturation beyond 256. Proxy formulation also affects results.

  • Number of Queries and Proxies: Increasing either the number of generated queries q or input point proxies N improves performance on the PCN benchmark.The ablation evaluates both quantities directly.
  • Number of Queries and Proxies: Attention’s quadratic complexity makes unlimited increases in N or q computationally impractical, while performance saturates when N > 256.The saturation behavior differs for q because the input point cloud contains only 2048 points.
  • Formulation of Point Proxies: Removing positional embeddings from point proxies causes a performance drop of 0.48.The positional embedding encodes the point proxy’s location alongside local structure features.
  • Formulation of Point Proxies: Replacing the mini-DGCNN feature extractor with PointNet causes a performance drop of 0.23.This ablation compares alternative feature extractors for point-proxy construction.

APPENDIX E ADDITIONAL DESCRIPTION TO KITTI METRIC

The KITTI MMD metric measures resemblance to a typical car by comparing each output with the closest PCN car point cloud under Chamfer Distance, while fidelity measures input preservation.

  • KITTI Metrics: MMD is the Chamfer Distance between the output and the closest PCN car point cloud, measuring resemblance to a typical car.The closest reference is selected according to Chamfer Distance.
  • KITTI Metrics: Fidelity is the average distance from each input point to its nearest neighbor in the output, measuring how well the input is preserved.The metric focuses on preservation of observed input points.

APPENDIX F DETAILED EXPERIMENTAL RESULTS

The appendix reports detailed point cloud completion results across ShapeNet-34, Projected-ShapeNet-34, ShapeNet-55, and Projected-ShapeNet-55. Results compare multiple methods under varying difficulty settings and evaluation metrics, alongside qualitative examples on ShapeNet-55 and NYUCAD.

  • ShapeNet-34: ShapeNet-34 results compare FoldingNet, PCN, TopNet, PFNet, GRNet, SnowflakeNet, and the proposed method across simple, moderate, and hard settings using CD-ℓ2.Table 12 reports category-level results for novel objects from 21 ShapeNet-34 categories.
  • Projected-ShapeNet-34: Projected-ShapeNet-34 results report category-level CD-ℓ1 and F-score@1% for each method.The evaluation covers novel objects from 21 categories.
  • ShapeNet-55: ShapeNet-55 results compare methods across simple, moderate, and hard settings using category-level CD-ℓ2.Table 14 presents detailed results for each category.
  • Projected-ShapeNet-55: Projected-ShapeNet-55 results report category-level CD-ℓ1 and F-score@1% for each method.Table 15 provides the detailed comparisons for the dataset.
  • Qualitative results: Additional qualitative results are presented for ShapeNet-55 and NYUCAD, with visual comparisons including inputs, competing methods, the proposed method, and ground truth.The qualitative comparison labels include Input, FoldingNet, PCN, TopNet, GRNet, Ours, SnowflakeNet, and G.T.
Loading 2301.04545v1…