PaperlayerView exact revision on arXiv

Source-linked AI summary

Lepard: Learning partial point cloud matching in rigid and deformable scenes

Yang Li, Tatsuya Harada

arXiv:2111.12591v2cs.CV

TL;DR

Partial point-cloud matching must handle both rigid and deformable scenes while coping with ambiguous geometry and changing positions. Lepard separates feature and position representations, encodes relative 3D distances, and repositions cross-cloud coordinates; it reports strong rigid recall and higher non-rigid matching recall than prior art.

  • Problem

    Partial point-cloud matching must work across rigid and deformable scenes, where transformation-invariant geometry features can be ambiguous for repetitive structures.

  • Method

    Lepard combines learned geometry features with disentangled position representations, relative 3D positional encoding, and cross-point-cloud repositioning.

  • Results

    Lepard achieves 93.9% / 71.3% registration recall on 3DMatch / 3DLoMatch and +27.1% / +34.8% higher non-rigid matching recall than prior art on 4DMatch / 4DLoMatch.

  • Takeaways & Limitations

    Positional knowledge supports Lepard’s feature matching results across both rigid and deformable point clouds.

  • Takeaways & Limitations

    Lepard is coarse, does not explicitly handle topological changes, and faces particular difficulty with low-overlap matching and registration.

Abstract

from arXiv · show

We present Lepard, a Learning based approach for partial point cloud matching in rigid and deformable scenes. The key characteristics are the following techniques that exploit 3D positional knowledge for point cloud matching: 1) An architecture that disentangles point cloud representation into feature space and 3D position space. 2) A position encoding method that explicitly reveals 3D relative distance information through the dot product of vectors. 3) A repositioning technique that modifies the crosspoint-cloud relative positions. Ablation studies demonstrate the effectiveness of the above techniques. In rigid cases, Lepard combined with RANSAC and ICP demonstrates state-of-the-art registration recall of 93.9% / 71.3% on the 3DMatch / 3DLoMatch. In deformable cases, Lepard achieves +27.1% / +34.8% higher non-rigid feature matching recall than the prior art on our newly constructed 4DMatch / 4DLoMatch benchmark.

1. Introduction

Lepard targets robust partial point-cloud matching across rigid and deformable scenes by adding 3D positional knowledge to learned geometry features. It introduces disentangled position and feature representations, relative positional encoding, repositioning, and new deformable benchmarks, achieving strong rigid and non-rigid results.

  • Partial point-cloud matching supports applications including SLAM, dynamic tracking, and reconstruction across rigid and deformable scenes.
  • Translation and rotation invariance can make repetitive geometries ambiguous because points with similar local appearance may occupy different locations.
  • Lepard disentangles point-cloud representations into feature and position spaces, encodes 3D relative distances through vector dot products, and repositions cross-cloud coordinates.
  • 4DMatch and 4DLoMatch provide benchmarks containing point clouds that deform across time, making matching and registration more challenging than rigid cases.
  • 93.9% / 71.3% registration recall is achieved on 3DMatch / 3DLoMatch, while non-rigid matching recall is +27.1% / +34.8% higher on 4DMatch / 4DLoMatch than prior art.

2. Related work

Related work spans learned local descriptors and direct registration for rigid matching, alongside learned and geometry-processing approaches for non-rigid correspondence. Lepard focuses on general non-rigid feature matching for partial point-cloud scans while enhancing descriptors with 3D positional knowledge.

  • Rigid Point Cloud Matching and Registration: Rigid matching commonly combines local descriptor prediction with robust RANSAC-based optimization, progressing from hand-engineered to learned descriptors.
  • Rigid Point Cloud Matching and Registration: Direct registration methods optimize pose using ICP, FGR, global SE(3) search, or learned correspondence weighting and pose optimization.
  • Non-Rigid Correspondence: Non-rigid correspondence research addresses real-world sensor data for online reconstruction through descriptors, learned correspondences, alignment optimization, and outlier rejection.
  • Non-Rigid Correspondence: Geometry-processing methods also study non-rigid correspondence for manifold surfaces using deformation assumptions, latent codes, and functional maps.
  • Lepard develops a general non-rigid feature-matching method for partial point-cloud scans and enhances point-cloud descriptors with 3D positional knowledge.

3. Problem Definition

The problem is to find correspondences between partial source and target point clouds that recover an aligning warp. Lepard treats the warp as rigid in rigid scenes or as a dense per-point field in deformable scenes, using overlap-aware inputs and a staged matching pipeline.

  • Problem Definition: Given source S and target T point clouds, the goal is to find matches K that recover a warp W aligning S to T.
  • Problem Definition: Rigid scenes parameterize W with a SE(3) transformation, whereas deformable scenes use a dense per-point warp field.
  • Problem Definition: An inlier match satisfies ||Wgt(Si)−Tj||2 < σ, using the ground-truth warp and a tolerance radius σ.
  • Partial Overlap: Partial overlap arises because object motion or viewpoint changes mean that points in S do not necessarily have corresponding points in T.
  • Proposed Pipeline: The pipeline downsamples input clouds, extracts geometry features, encodes relative positions, applies transformer attention and differentiable matching, estimates rigid fitting, then repositions source position codes.

4. Method

Lepard combines geometry features with explicit 3D positional information through Transformer-based matching, differentiable confidence estimation, and iterative rigid fitting. Its repositioning module updates cross-cloud positions before a second matching stage, while focal and warping losses supervise correspondence and alignment.

  • Feature extraction: Lepard extracts down-sampled multi-level geometry features with a KPFCN backbone for efficient Transformer and matching computation.The backbone uses translation-equivariant, local geometry extraction, and its output feature dimension is d = 528.
  • Relative 3D positional encoding: 3D positional encoding complements translation-invariant features by exposing relative distance information through encoded-feature dot products.The encoding is orthogonal, preserving feature length while changing feature direction, and is used in both Transformer and matching layers.
  • Position-aware feature matching: Dual-softmax converts the scoring matrix into a confidence matrix, from which thresholded mutual-nearest-neighbor matches are selected; Sinkhorn is an alternative.The matching threshold θc is examined in the supplementary ablation study.
  • Transformer: Position codes and geometry features remain in separate streams, combining only when similarity matrices are computed so position affects attention weights without becoming part of values.Self-attention aggregates context within each cloud, while cross-attention exchanges information between clouds using query, key, and value projections.
  • Rigid fitting and repositioning: Repositioning applies the estimated rigid fitting to source position codes, bringing corresponding points closer in position space for cross-attention and position-aware matching.The fitting uses top-confidence matches and a soft Procrustes procedure based on SVD; the updated positions feed the subsequent TMP stage.
  • Supervision: Training combines focal matching loss and warping loss across two TMP blocks, with warping regularizing false positives in rigid scenes and approximating a root pose in deformable scenes.The total loss weighting factor λw and the number of TMP blocks are evaluated in supplementary ablations.

5. 4DMatch

4DMatch and 4DLoMatch benchmark partial point-cloud matching with time-varying geometry, using synthesized partial scans split by a 45% overlap threshold. Evaluation reports inlier ratio and non-rigid feature matching recall, with qualitative examples spanning overlap and deformable matching cases.

  • Benchmark construction: 4DMatch is built from DeformingThings4D sequences with ground-truth dense correspondence and synthesized depth-image scans.The benchmark uses 1,761 selected animations, split into 1,232/176/353 train/validation/test sequences.
  • Benchmark construction: The 353 testing sequences are divided into 4DMatch or 4DLoMatch using a 45% overlap-ratio threshold.The overlap ratio is computed relative to the source point cloud, and partial overlap reflects scene deformation and camera viewpoint change.
  • Qualitative evaluation: Qualitative benchmark examples include varying overlap ratios and deformable matching cases, with inliers and outliers visualized for the latter.The deformable example uses a bilaterally symmetric dragon to illustrate feature discrimination between its two wings.
  • Evaluation metrics: Inlier ratio measures the fraction of predicted correspondences that are correct within σ = 0.04m after applying the ground-truth warp.Correctness is evaluated using the Euclidean distance between the warped prediction and its matched point.
  • Evaluation metrics: Non-rigid feature matching recall measures the fraction of ground-truth correspondences recovered by predicted correspondences after inverse-distance flow interpolation.The sparse scene-flow field is constructed from predicted matches and propagated from anchor points.

6. Experimental Results

Lepard improves rigid and deformable point-cloud matching through positional design choices, with strong benchmark results and gains over prior methods. Ablations support the contributions of disentanglement, relative positional encoding, repositioning, and dual softmax matching.

  • Ablation study: +3.8% higher NFMR on 4DLoMatch and +1.5% / +2.3% higher RR on 3DMatch / 3DLoMatch result from relative rather than absolute positional encoding.The comparison uses absolute sinusoidal encoding as the alternative.
  • Ablation study: +2.9% / +3.3% NFMR gains on 4DMatch / 4DLoMatch and +0.9% / +1.0% RR gains on 3DMatch / 3DLoMatch come from repositioning.Random rotation-based positioning causes significant performance drops, while oracle fitting configurations achieve near-perfect results.
  • Ablation study: The dual softmax operator achieves higher scores on all benchmarks than the Sinkhorn approach.This ablation compares the two matching operators used for correspondence prediction.
  • Deformable matching: +27.1% / +34.8% higher NFMR is achieved on 4DMatch / 4DLoMatch than Predator.The same comparison reports +22.3% / +28.2% higher IR on 4DMatch / 4DLoMatch.
  • Rigid matching: +1.7% / +6.6% higher RR is achieved on 3DMatch / 3DLoMatch than Predator.Point-to-point ICP further improves the registrations, and Lepard also produces better RRE and RTE than Predator.

7. Conclusion

Lepard uses positional knowledge to obtain state-of-the-art feature matching results for rigid and deformable point clouds. The conclusion identifies unresolved limitations in refinement, topology changes, and low-overlap matching.

  • Conclusion: Lepard demonstrates state-of-the-art feature matching results for both rigid and deformable point clouds by leveraging positional knowledge.The paper also identifies end-to-end registration as a promising direction.
  • Limitations: Lepard is a coarse matching approach, leaving fine-grained correspondence to potential learning-based refinement.The limitation is presented as an issue yet to be addressed.
  • Limitations: Lepard does not explicitly handle topological changes in deformable cases.The paper suggests jointly learning matching with motion segmentation as a potential solution.
  • Limitations: Low-overlap matching and registration remain particularly challenging because of data incompleteness.Figure 9 shows failure cases involving similar patterns in non-overlapping regions.

Supplementary Material: Lepard: Learning partial point cloud matching in rigid and deformable scenes

The supplementary material provides ablation studies, implementation details, formal definitions, and additional results for 3DMatch and 4DMatch.

  • Supplementary contents: The supplement includes an ablation study, implementation details, a formal definition of non-rigid registration, and more 3DMatch and 4DMatch results.These materials are located in the supplementary sections listed by the paper.

I. Ablation study

The ablations show that training losses, confidence thresholds, positional design, and network depth affect matching and registration differently across rigid and deformable benchmarks. They also support relative positional encoding, repositioning, and a confidence threshold of θc=0.1 as useful design choices.

  • Influence of warping loss: Warping loss generally increases NFMR and IR on 4DMatch and 4DLoMatch, while increasing IR but decreasing RR on 3DMatch and 3DLoMatch.In low-overlap deformable cases, performance grows steadily as motion loss weight increases.
  • Influence of confidence threshold: Increasing the confidence threshold decreases rigid registration recall but increases deformable IR while lowering NFMR.The paper attributes this trade-off to suppressing borderline correspondences.
  • Influence of confidence threshold: θc=0.1 provides a reported trade-off between precision and recall.This threshold is selected after observing the rigid and deformable ablation trends.
  • Number of TMP layers: Additional TMP layers do not improve 3DMatch results, while 3 TMP layers achieve the best 4DMatch results and increase training time.The corresponding repositioning-layer counts are 2 and 3 for 3 and 4 TMP layers, respectively.
  • Metrics and efficiency: The method’s metrics include RR, FMR, NFMR, and MNN, while inference is reported as about twice as efficient as Predator in time and memory.The efficiency comparison is reported for 4DLoMatch inference on an Nvidia A100 (80G) GPU.

V. KPFCN backbone architecture

The paper models non-rigid deformation with an embedded deformation graph and optimizes correspondence and regularization energies using Gauss–Newton iterations.

  • Deformation Model: Non-rigid motion is represented by an embedded deformation graph whose nodes are sampled over the source surface and connected by edges.Each node has a 3D location and a parameterized rotation.
  • Non-rigid Warping Function: The warping function transforms each point by blending node motions according to skinning weights.The weights measure each node’s influence on the point and are normalized to sum to one.
  • Energy Function: The N-ICP energy combines a correspondence term weighted by match confidence with an ARAP regularization term.The total objective is formed from these correspondence and regularization components.
  • Residual and Partial Derivatives: The optimization uses sparse residual and Jacobian structures built from correspondence and graph-edge terms.The residual vector contains 3D blocks for correspondences and edges, while most Jacobian blocks are zero.
  • Non-rigid Optimization: Gauss–Newton repeatedly recomputes the Jacobian and residual vector, then solves for a deformation increment.The resulting linear system is solved using LU decomposition.

VII. Qualitative Results

The qualitative evaluations show matching and registration examples for deformable 4DMatch and rigid 3DLoMatch point clouds, alongside quantitative non-rigid registration metrics.

  • Quantitative Results: Tables 9 and 10 report quantitative non-rigid registration results using 3D endpoint error and motion estimation accuracy.Accuracy thresholds are reported at <0.05m or 5% and <0.1m or 10%.
  • 4DMatch: 4DMatch qualitative results compare predicted matches and registration outcomes using a 4cm inlier threshold.The examples include the effects of N-ICP-based refinement on matching outliers.
  • 4DMatch: N-ICP refinement can remedy moderately displaced outlier matches but cannot handle outliers connecting distant parts.In the Mutant example, both methods register the left and right legs together.
  • 3DLoMatch: The 3DLoMatch qualitative results present point cloud matching and registration examples.The passage identifies these as qualitative results without reporting a numerical comparison.
Loading 2111.12591v2…