PaperlayerView exact revision on arXiv

Source-linked AI summary

D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features

Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, Chiew-Lan Tai

arXiv:2003.03164v1cs.CV

TL;DR

D3Feat addresses the limited joint learning of 3D feature detection and description, which is important for establishing sparse, discriminative registration matches. It uses a KPConv-based fully convolutional network with density-invariant keypoint selection and self-supervised detector guidance. The method achieves strong indoor and outdoor benchmark results and performs especially well with few keypoints.

  • Problem

    Learning-based 3D feature research has largely emphasized descriptors while overlooking jointly learned keypoint detection, despite the importance of reliable sparse matches for registration.

  • Method

    D3Feat densely predicts per-point descriptors and detection scores with a KPConv-based fully convolutional network, using density-invariant selection and matching-guided self-supervision.

  • Results

    D3Feat outperforms state-of-the-art methods across indoor 3DMatch and outdoor KITTI evaluations, with strong generalization reported on ETH.

  • Takeaways & Limitations

    Using a reliable learned detector allows accurate and fast point-cloud alignment with a smaller number of sampled features.

  • Takeaways & Limitations

    The original D2-Net-inspired loss did not guarantee convergence in the 3D context, motivating D3Feat’s explicit detector-loss guidance.

Abstract

from arXiv · show

A successful point cloud registration often lies on robust establishment of sparse matches through discriminative 3D local features. Despite the fast evolution of learning-based 3D feature descriptors, little attention has been drawn to the learning of 3D feature detectors, even less for a joint learning of the two tasks. In this paper, we leverage a 3D fully convolutional network for 3D point clouds, and propose a novel and practical learning mechanism that densely predicts both a detection score and a description feature for each 3D point. In particular, we propose a keypoint selection strategy that overcomes the inherent density variations of 3D point clouds, and further propose a self-supervised detector loss guided by the on-the-fly feature matching results during training. Finally, our method achieves state-of-the-art results in both indoor and outdoor scenarios, evaluated on 3DMatch and KITTI datasets, and shows its strong generalization ability on the ETH dataset. Towards practical use, we show that by adopting a reliable feature detector, sampling a smaller number of features is sufficient to achieve accurate and fast point cloud alignment.[code release](https://github.com/XuyangBai/D3Feat)

1. Introduction

D3Feat addresses the underdeveloped joint learning of 3D keypoint detection and description by using dense prediction and self-supervised detector guidance. It targets accurate, efficient registration from a small set of reliable keypoints.

  • Motivation: Point cloud registration depends on robust local keypoint detection and description to establish matches between partially overlapping fragments.These components support applications including SLAM and 3D LiDAR-based mapping.
  • Motivation: Random point sampling can produce poorly localized, nondiscriminative, and inefficient matches, motivating learned keypoint selection.Full scene coverage may require oversampling, while smooth-surface points can introduce matching noise.
  • Approach: D3Feat jointly learns detection and description with a KPConv-based fully convolutional network for dense 3D feature prediction.The framework avoids dual structures for fast inference and directly processes unstructured point clouds.
  • Approach: A density-invariant keypoint selection strategy addresses varying point-cloud densities, while a self-supervised detector loss uses on-the-fly matching results during training.The loss guides detection scores according to correspondence reliability.
  • Evaluation: Experiments on 3DMatch, KITTI, and ETH demonstrate strong performance across indoor and outdoor settings.The paper reports superiority over state-of-the-art methods and generalization to ETH.

2. Related Work

Prior 3D local-feature methods include handcrafted, patch-based learned, and fully convolutional approaches, while joint detector–descriptor learning remains relatively underexplored. D3Feat is positioned as an efficient single-network alternative to existing methods.

  • Handcrafted Methods: Handcrafted 3D descriptors and detectors can degrade under noise, occlusion, and real-world scan conditions.Handcrafted descriptors generally lack robustness against noise and occlusion, while detectors rely on local geometric properties.
  • Patch-Based Networks: Most learned 3D descriptors use point-cloud patches, but reusing intermediate activations across adjacent patches is difficult, limiting high-resolution output efficiency.Patch-based approaches include image, voxel, and PointNet-style representations.
  • Joint Learning: D3Feat fuses detection and description into one network that predicts keypoint locations and per-point features in a single forward pass.This design is intended to reduce memory and computation relative to separate or patch-based processing.

3. Joint Detection and Description Pipeline

D3Feat uses a KPConv-based fully convolutional network to produce dense point features, then derives keypoint scores from channel and neighborhood responses. Density-invariant scoring makes selection more robust to non-uniform sampling.

  • Network Design: D3Feat adapts D2-Net’s dual-role detector–descriptor design to irregular, variably sparse 3D point clouds.The method uses a single network rather than separate detection and description networks.
  • Network Design: KPConv serves as the backbone operation for dense feature extraction directly on raw point clouds.KPConv uses kernel points carrying weights to emulate convolution on unstructured 3D data.
  • Network Design: A UNet-like fully convolutional architecture with skip connections and residual blocks outputs a dense feature map F ∈ R^N×c.Each point receives a descriptor represented by its corresponding row of the feature map.
  • Dense Keypoint Detection: The detector first selects the strongest feature channel and then checks whether that response is a spatial maximum within the point’s radius neighborhood.Training softens this selection using saliency and channel-max scores.
  • Dense Keypoint Detection: The density-invariant saliency score compares a point’s feature with the mean feature of its local neighborhood rather than a neighborhood sum.Using the average response prevents the score from depending on the number of neighboring points.
  • Dense Keypoint Detection: The final keypoint score combines saliency and channel-max scores, after which points with the highest scores are selected.This produces the keypoints from the dense score map.

4. Joint Optimizating Detection & Description

The method jointly learns descriptors and detectors using metric learning plus a self-supervised detector loss guided by feature matchability. Detector scores are explicitly encouraged to prioritize correspondences that the current descriptor can match reliably.

  • Descriptor loss: Contrastive loss optimizes descriptors, using hardest-in-batch sampling to focus training on difficult pairs.The paper reports better convergence for contrastive loss than alternative metric-learning strategies.
  • Descriptor loss: Negative pairs use the hardest sample outside a safe radius of the true correspondence, with separate positive and negative margins.The safe radius excludes geometrically nearby points from negative selection.
  • Detector loss: The prior joint loss lacked explicit score guidance and did not guarantee convergence in this setting.The authors motivate their revised detector loss as a response to this observed limitation.
  • Detector loss: The detector objective seeks higher scores for easily matchable correspondences and lower scores for correspondences that remain difficult to match.Matchability is assessed from feature-matching results produced during training.
  • Detector loss: The proposed detector loss explicitly guides score gradients using on-the-fly matching results under the current descriptor network.This self-supervised signal evaluates each correspondence’s discriminativeness without requiring detector labels.
  • Detector loss: When positive descriptors are nearer than the hardest negative, the corresponding detection scores are increased; otherwise, they are decreased.The rule couples detector optimization to whether nearest-neighbor matching succeeds for each correspondence.

5. Implementation Details

Training uses overlapping fragment pairs to construct correspondences, while implementation choices specify grid-based neighborhoods and augmentation. Testing applies hard keypoint selection to obtain separated, higher-quality keypoints.

  • Training: Training uses point-cloud fragment pairs with more than 30% overlap to construct correspondence sets.Correspondences are formed by transforming randomly sampled anchors and accepting nearest neighbors within a distance threshold.
  • Training: Correspondence construction transforms randomly sampled anchors and retains nearest neighbors only when their Euclidean distance is below a threshold.This procedure uses the ground-truth transformation between fragments P and Q.
  • Network settings: The first network layer uses a 0.03m grid, with neighborhood radii set to 2.5 times the current layer’s grid size.The keypoint-detection neighborhood matches the first-layer radius neighborhood.
  • Training: Augmentation adds Gaussian noise with standard deviation 0.005, random scaling in [0.9, 1.1], and random rotations.These augmentations are applied to each point-cloud fragment during training.
  • Testing: Hard keypoint selection at test time emulates non-maximum suppression and prevents selected keypoints from lying too close together.The paper uses hard rather than soft selection to obtain better-quality keypoints.

6. Experiments

Experiments evaluate D3Feat for indoor and outdoor registration, varying thresholds, keypoint counts, datasets, and detector components. The learned detector improves matching quality, robustness with fewer keypoints, repeatability, and cross-dataset performance.

  • 6.1. Indoor Settings: 3DMatch dataset: On 3DMatch, D3Feat achieves the best feature matching recall when a learned detector is used, for both original and randomly rotated fragments.Results are reported for D3Feat(rand) and D3Feat(pred) using 5000 points or predicted keypoints.
  • 6.1. Indoor Settings: 3DMatch dataset: At τ2 = 20%, D3Feat notably outperforms other methods, while its varying-τ1 performance is slightly below PerfectMatch at smaller tolerated distance errors.D3Feat uses 3cm voxel downsampling, whereas PerfectMatch uses a 1.875cm voxel size.
  • 6.1. Indoor Settings: 3DMatch dataset: With fewer than 5000 points, D3Feat(pred) maintains high matching quality across evaluation metrics and outperforms comparative methods by a large margin.Without a detector, PerfectMatch, FCGF, and D3Feat(rand) degrade similarly as the sampled point count decreases.
  • 6.1. Indoor Settings: 3DMatch dataset: D3Feat(pred) is the only method whose inlier ratio improves with fewer points, indicating that its highest-scored keypoints are more likely to match.This behavior suggests that the detector ranks keypoints by matching reliability.
  • 6.1. Indoor Settings: 3DMatch dataset: D3Feat empirically achieves strong rotation invariance through low-cost data augmentation with a KPConv-based fully convolutional network.This behavior is reported from the rotated-fragment results in Table 1.
  • 6.1. Indoor Settings: 3DMatch dataset: The proposed detector loss improves convergence over D2 Triplet, boosts performance over D2 Contrastive, and strengthens the descriptor through joint detector learning.The ablation compares D3Feat(rand), D2 Triplet, D2 Contrastive, and a model without detector learning.
  • 6.2. Outdoor Settings: KITTI dataset: On KITTI and ETH, experiments assess outdoor registration and generalization using separate voxel-size settings and prescribed dataset protocols.The 3DMatch-trained model is evaluated on ETH to test generalization.
  • 6.3. Outdoor settings: ETH Dataset: On ETH, D3Feat generalizes better than FCGF under the 5cm setting, and enabling its detector remarkably improves results, although PerfectMatch remains best.PerfectMatch’s advantage is attributed to its smoothed density value representation.

7. Conclusion

D3Feat jointly detects and describes dense 3D point features using a fully convolutional network, density-invariant saliency, and self-supervised detector guidance. Experiments on 3DMatch and KITTI support effectiveness, especially with few keypoints.

  • D3Feat uses a dual-role fully convolutional network for dense 3D feature detection and description.
  • A density-invariant saliency score selects keypoints under varying point-cloud densities.
  • A self-supervised detector loss guides prediction toward highly repeatable keypoints and jointly improves detector and descriptor performance.
  • Experiments on indoor 3DMatch and outdoor KITTI show the effectiveness of the detector and distinctiveness of the descriptor.
  • D3Feat outperforms state-of-the-art methods especially when using a small number of keypoints.

8. Supplementary Material

The supplementary material examines density normalization, network architecture, rotation augmentation, and related experimental settings. It reports that normalization improves matching across sparsity levels, while removing rotation augmentation causes failure on 3DRotatedMatch.

  • Supplementary Material: The supplement details KPConv modifications, rotation augmentation, detector-loss ablations, FCGF integration, and experiments on 3DMatch, KITTI, and ETH.
  • Normalization term: A density normalization term is added to KPConv to make convolution invariant to point-cloud sparsity.
  • Normalization term: With uniform downsampling, normalization maintains high matching quality and outperforms the model without normalization by a large margin.
  • Network architecture: The KPConv-based network uses encoder channels of 64, 128, 256, 512, and 1024, skip connections, and 32-dimensional output features.
  • Rotation invariance: Without rotation augmentation, D3Feat fails on 3DRotatedMatch because the network cannot learn rotation invariance from the data.

8.3. Ablation on Detector Loss

The detector-loss ablation finds that the proposed loss benefits both descriptor quality and detector performance, particularly by improving the ranking of distinctive keypoints.

  • Ablation on Detector Loss: The detector-loss ablation compares D3Feat with models without the detector loss on random and predicted keypoints.
  • Ablation on Detector Loss: The detector loss strengthens the descriptor when evaluated on random keypoints and boosts the detector when evaluated on predicted keypoints.
  • Ablation on Detector Loss: Only D3Feat(pred) improves Inlier Ratio when reducing the number of points, indicating better ranking of keypoints by distinctiveness.
  • Ablation on Detector Loss: Because detection scores are computed from dense descriptors, the detector can be incorporated with dense description models such as FCGF.
  • Ablation on Detector Loss: FCGF trained with the proposed detector loss maintains high performance with fewer points.

8.5. Runtime

The runtime section compares D3Feat and FCGF under the same voxel size while defining the feature-matching and registration-recall evaluation procedures. The stated runtime difference mainly comes from FCGF’s time-consuming sparse-convolution hashing.

  • Runtime: Runtime is compared on 3DMatch using the same 2.5cm voxel size, roughly 20k points, and includes both detection and description for D3Feat.
  • Runtime: The runtime difference mainly reflects FCGF’s sparse convolution, which is time-consuming during hashing.
  • Evaluation metrics: Feature Matching Recall evaluates descriptor quality through mutually nearest-neighbor correspondences between partially overlapping fragments.
  • Evaluation metrics: A correspondence is an inlier below τ1 = 10cm, and a fragment pair counts as matched above τ2 = 5% inlier correspondences.
  • Evaluation metrics: Registration recall evaluates features within a reconstruction system by estimating a rigid transformation with RANSAC and measuring transformed ground-truth correspondence error.

8.7. Dataset Preprocessing

The datasets use distinct preprocessing and evaluation protocols, including overlap-based fragment pairing, rotation perturbations, ICP refinement, and author-provided ground truth.

  • 3DRotatedMatch: 3DRotatedMatch tests rotation robustness by randomly rotating every 3DMatch test fragment about all three axes with angles sampled from [0, 2π).The model is translation invariant because it uses relative coordinates.
  • KITTI: KITTI uses sequences 0–5 for training, 6–7 for validation, and the final three for testing after ICP refinement of noisy GPS alignments.Pairs are sampled at least 10m apart, yielding 1358 training, 180 validation, and 555 test pairs.
  • ETH: ETH evaluation directly uses the authors’ raw point clouds, ground-truth transformations, and overlap ratios to extract features and assess registration.The dataset-processing section covers 3DMatch, 3DRotatedMatch, KITTI, and ETH.

8.8. Qualitative Visualization

The paper presents challenging registration examples and visualizes detected keypoints across 3DMatch, ETH, and KITTI datasets.

  • Keypoint Visualizations: Figures 7–9 visualize detected keypoints on 3DMatch, ETH, and KITTI, respectively.The accompanying text identifies these figures as additional keypoint visualizations.
  • Registration Results: Figure 6 shows qualitative registration results for 3DMatch, with input fragments in the first two columns and registration outputs in the third.The figure recommends viewing results with color and zoom-in.
Loading 2003.03164v1…