PaperlayerView exact revision on arXiv

Source-linked AI summary

3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration

Zi Jian Yew, Gim Hee Lee

arXiv:1807.09413v1cs.CV

TL;DR

Point cloud registration needs reliable 3D correspondences, but existing deep-learning pipelines generally require impractical manual feature labels. 3DFeat-Net jointly learns feature detection and description from GPS/INS-tagged point clouds using weak supervision, and compares favorably with prior methods on several outdoor gravity-aligned datasets.

  • Problem

    Existing deep-learning approaches generally learn descriptors without learned detection because manually labeling salient 3D features and matching clusters is impractical.

  • Method

    3DFeat-Net uses a weakly supervised Siamese architecture with attention, triplet loss, and learned orientations to jointly detect and describe keypoints from GPS/INS-tagged point clouds.

  • Results

    3DFeat-Net compares favorably with handcrafted and learned methods across several outdoor gravity-aligned datasets, including strong generalization performance on KITTI without fine-tuning.

  • Takeaways & Limitations

    Weak supervision can support a holistic 3D feature detector-and-descriptor pipeline without explicitly specifying point-level correspondences.

  • Takeaways & Limitations

    Training is constrained when point-cloud poses are too inaccurate to infer point-to-point correspondences, limiting the weak-supervision signal.

Abstract

from arXiv · show

In this paper, we propose the 3DFeat-Net which learns both 3D feature detector and descriptor for point cloud matching using weak supervision. Unlike many existing works, we do not require manual annotation of matching point clusters. Instead, we leverage on alignment and attention mechanisms to learn feature correspondences from GPS/INS tagged 3D point clouds without explicitly specifying them. We create training and benchmark outdoor Lidar datasets, and experiments show that 3DFeat-Net obtains state-of-the-art performance on these gravity-aligned datasets.

1 Introduction

Point-cloud registration is valuable because 3D scenes remain matchable across strong lighting and seasonal changes, but establishing reliable point correspondences remains difficult. 3DFeat-Net addresses the lack of labeled salient features with weakly supervised joint detector–descriptor learning from GPS/INS-tagged clouds.

  • 3D point-cloud registration supports Lidar mapping, robot localization, archaeological studies, geo-surveying, and architectural inspection.
  • Compared with images, point clouds vary less and remain matchable across day–night and summer–winter conditions.
  • Existing learned 3D methods generally learn descriptors but not detectors because supervised training requires impractical manual labels for salient 3D features.
  • 3DFeat-Net jointly learns a 3D feature detector and descriptor from GPS/INS-tagged point clouds using weak supervision.
  • The Siamese network uses PointNet inputs, triplet loss, descriptor similarities, and attention weights to learn descriptors and point saliency.
  • The paper contributes a weakly supervised network, an attention-based saliency mechanism, and Oxford RobotCar training and benchmark datasets.

2 Related Work

Prior local 3D features comprise handcrafted and learning-based approaches, while learned methods largely focus on descriptors rather than keypoint detection. The paper motivates 3DFeat-Net against this limitation and the instability of image matching under changing conditions.

  • Local 3D feature methods are broadly categorized as handcrafted or learning-based approaches influenced by 2D local-feature research.
  • 2D images often produce wrong matches across times or seasons even after RANSAC, whereas the corresponding 3D scene remains largely stable and easier to match.
  • Handcrafted 3D Features: Handcrafted methods detect keypoints using geometric variation or curvature and describe neighborhoods with normals, histograms, or related constructs.
  • Learned 2D Features: Learned 2D methods can learn detectors and descriptors from image matches, but handcrafted 3D features provide less robust matches for this strategy.
  • Learned 3D Features: Learned 3D methods such as 3DMatch, PPFNet, CGF, and LORAX primarily learn or reduce descriptor representations.
  • Learned 3D Features: Existing learned 3D approaches generally compute descriptors on all or randomly sampled points, with limited joint detector learning.

3 Problem Formulation

The problem is to learn point correspondences between local regions of two point clouds using only coarse pose information. Training uses geographically defined positive and negative cloud pairs, while point-level feature selection remains unspecified.

  • A point cloud P is modeled as N 3D points, and each cloud is cropped to a fixed-radius ball around its centroid.
  • GPS/INS provides absolute pose during training, but its accuracy is insufficient to infer point-to-point correspondences.
  • Cloud distance is defined as the Euclidean distance between centroids, d(m, n) = ∥c_m − c_n∥_2.
  • Training triplets contain an anchor, positive, and negative cloud, with positives below τ_p distance and negatives above τ_n distance from the anchor.
  • The network seeks correspondences between subsets of points in two clouds rather than requiring every point to participate.
  • Weak supervision supplies only model-level relative poses and does not explicitly specify which point subset should form the 3D features.

4 Our 3DFeat-Net

3DFeat-Net jointly learns point-cloud detectors and descriptors in a three-branch Siamese architecture. It samples local clusters, predicts attention and orientation, computes canonical descriptors, aligns descriptor matches, and trains with a triplet loss.

  • Network Architecture: A three-branch Siamese network processes anchor, positive, and negative point clouds with a triplet loss.Each branch accepts an entire point cloud and produces cluster-level representations.
  • Feature Alignment Triplet Loss: The triplet objective reduces anchor-positive distance while increasing anchor-negative distance by a margin.The hinge loss uses γ as the enforced margin between positive and negative pairs.
  • Clustering: PointNet++-style sampling forms local clusters around sampled points to support local descriptor computation.Iterative farthest point sampling is used, while other coverage-oriented sampling schemes are also suitable.
  • Detector: The detector predicts each cluster’s orientation and attention score, using attention as a saliency measure and orientation for canonicalization.Inference retains high-attention points after non-maximal suppression and thresholding.
  • Descriptor: The descriptor network rotates clusters to canonical configurations and generates a descriptor for each cluster, incorporating contextual information.The detector’s predicted orientation is passed into the descriptor network.
  • Feature Alignment Triplet Loss: Descriptor alignment matches each descriptor to its closest counterpart, while normalized attention weights determine cluster contributions to the loss.For non-matching pairs, closest-descriptor alignment supplies hardest negatives without explicit hard-negative mining.

5 Evaluations and Results

The evaluation uses Oxford RobotCar traversals with GPS/INS-derived 3D point clouds, disjoint training and testing splits, and ICP-refined test poses. The supplied passages specify the dataset construction and split but not performance values.

  • Oxford RobotCar: Oxford RobotCar provides repeated traversals of the same central Oxford route at different times over one year.Push-broom 2D scans are accumulated into 3D point clouds using GPS/INS poses.
  • Oxford RobotCar: 3D point clouds are created within a 30m radius at 10m intervals when reliable GPS/INS poses are available.Each cloud is downsampled with a 0.2m VoxelGrid filter.
  • Oxford RobotCar: The first 35 traversals form the training split and the last 5 traversals form the testing split, yielding 21,875 training and initially 828 testing point-cloud sets.The training and testing sets are disjoint.
  • Evaluation Protocol: GPS/INS relative poses provide test ground truth, then ICP refinement improves test-set fidelity before evaluation.The refinement registers test point clouds within 10m to corresponding clouds in a reference traversal.

KITTI Dataset

The supplied passages describe evaluations on KITTI, ETH, and training settings, including preprocessing, optimization, staged training, and inference parameters.

  • ETH Dataset: The ETH evaluation uses ground-Lidar scenes with largely unstructured vegetation and cross-season registration between global and local point clouds.Gazebo and Wood scenes are accumulated across seasons for registration.
  • ETH Dataset: ETH point clouds use a finer 0.1m VoxelGrid resolution because vegetation contains finer features.This preprocessing choice differs from the coarser Oxford and KITTI settings described elsewhere.
  • Training Settings: Training uses six triplets per batch, ADAM with learning rate 1e-5, 512 clusters of radius 2.0m, and positive and negative thresholds of 5m and 50m.Point clouds are randomly downsampled to 4096 points during training.
  • Training Settings: Training proceeds in two phases because the end-to-end network can be difficult to optimize in practice.The first phase pretrains the descriptor without the detector for two epochs before second-phase initialization.
  • Inference Settings: Inference uses dataset-specific non-maximal-suppression radii and attention thresholds, with up to 1024 keypoints and approximately 0.8s processing for 16,384 points.The ETH global model allows 2048 keypoints.

5.3 Baseline Algorithms

The evaluation compares handcrafted and learned descriptors, detector combinations, and descriptor dimensionalities. 3DFeat-Net’s descriptor performs best overall, with diminishing returns beyond 32 dimensions and further gains from its dedicated detector.

  • Descriptor matching: The study compares handcrafted descriptors, a modified weakly supervised PointNet++ baseline, and 3DFeat-Net under a fixed 2.0m cluster radius.The handcrafted baselines include FPFH, Spin-Image, and USC; descriptor matching uses 30,000 balanced matching and non-matching pairs.
  • Descriptor dimensionality: Above 32 dimensions, descriptor improvements diminish, so the remaining experiments use d = 32.The dimensionality choice is based on the matching-error trend shown in Fig. 3.
  • Descriptor matching: 3DFeat-Net achieves lower matching error than every other descriptor despite having a similar or smaller dimension.It significantly outperforms FPFH, the strongest handcrafted descriptor, while modified PointNet++ performs substantially worse despite sharing the descriptor network structure.
  • Descriptor matching: The learned descriptor struggles with sparse clusters for some histogram-based descriptors, whereas 3DMatch distinguishes random clusters using a larger feature dimension.The comparison links performance differences to descriptor dimensionality and the ability to represent sparse point-cloud regions.

5.5 Keypoint Detection and Feature Description

The joint evaluation varies keypoint detectors and descriptors using precision plots on Oxford RobotCar. The learned detector–descriptor combination gives the strongest precision, while replacing its detector reduces performance.

  • Evaluation procedure: The evaluation matches each source keypoint descriptor to its nearest target descriptor and plots precision while varying the correctness distance threshold.The ground-truth distance to the nearest neighbor determines whether each match is correct.
  • Results: The proposed keypoint detector and descriptor achieve the best precision across all distance thresholds, reaching 15.6% at 1m.The comparison includes baseline descriptors paired with ISS, random sampling, and Random Sphere Cover Set keypoints.
  • Detector–descriptor coupling: Using the learned descriptor with random sampling or ISS lowers performance, indicating the importance of a dedicated feature detector.The result supports jointly designing detector and descriptor rather than relying on a generic or random detector.

5.6 Geometric Registration

Geometric registration uses nearest-neighbor keypoint matching followed by RANSAC to estimate a rigid transformation. On Oxford, the jointly learned detector and descriptor achieve the highest success rates and lowest errors.

  • Registration pipeline: Registration estimates a rigid transformation from nearest-neighbor keypoint matches using RANSAC, without subsequent ICP refinement.The RANSAC iteration count is adjusted for 99% confidence and capped at 10,000 iterations.
  • Oxford results: On Oxford RobotCar, the learned detector–descriptor combination produces the highest success rates and lowest registration errors.The comparison is reported in Table 3 for the Oxford dataset.
  • Detector and descriptor effects: Replacing random sampling with a detector improves geometric registration, including for 3DMatch, while the learned descriptor remains effective with alternative detectors.These findings support simultaneous detector and descriptor design and indicate generalization to generic point clusters.

Performance on Oxford RobotCar

The reported evaluations show strong registration performance and generalization, including favorable comparisons against handcrafted and learned baselines under differing evaluation protocols.

  • KITTI odometry: On KITTI odometry, 3DFeat-Net outperforms most other algorithms across registration measures without network fine-tuning.It uses the same parameters as on Oxford and is evaluated for geometric registration performance against ISS + FPFH.
  • KITTI odometry: 3DFeat-Net slightly trails CGF in RTE but achieves a significantly higher success rate with far fewer RANSAC iterations.
  • ETH and Gazebo/Wood comparisons: Against LORAX, 3DFeat-Net outperforms LORAX and most baselines despite using only the best match, one pose hypothesis, and no pose refinement.LORAX instead uses three descriptor matches per keypoint, multiple pose hypotheses, and ICP refinement.
  • ETH and Gazebo/Wood comparisons: The ETH evaluation reports results for FPFH and LORAX, with the final column giving success rate over the entire dataset.

6 Conclusion

The paper presents 3DFeat-Net as a weakly supervised model for jointly detecting and describing Lidar keypoints, with favorable results on outdoor gravity-aligned datasets. It also identifies noise sensitivity, input-size limits, and incomplete rotational invariance as future-work challenges.

  • Conclusion: 3DFeat-Net jointly learns keypoint detection and description in Lidar point clouds using weak supervision and a triplet loss.The loss incorporates individual descriptor similarities and the saliency of input 3D points.
  • Conclusion: Its learned detector and descriptor compare favorably with previous handcrafted and learned methods on several outdoor gravity-aligned datasets.
  • Limitations: The network does not train well on overly noisy point clouds, while PointNet limits the maximum input point-cloud size.
  • Limitations: The descriptor is not fully rotationally invariant, which the authors leave for future work.
Loading 1807.09413v1…