PaperlayerView exact revision on arXiv

Source-linked AI summary

CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence

Siyu Ren, Yiming Zeng, Junhui Hou, Xiaodong Chen

arXiv:2207.05483v3cs.CVcs.MM

TL;DR

Visual pose estimation determines an image’s 6DOF camera position in a 3D scene, with finding image–scene correspondence as a key step. CorrI2P detects overlapping image and point-cloud regions, builds dense correspondence from their features, and estimates pose with EPnP within RANSAC. CorrI2P outperforms state-of-the-art image-to-point-cloud registration methods on KITTI and NuScenes.

  • Problem

    Visual pose estimation determines an image’s 6DOF camera position in a 3D scene, with finding image–scene correspondence as a key step.

    Source: §I · p. 1

  • Method

    CorrI2P detects overlapping image and point-cloud regions, builds dense correspondence from their features, and estimates pose with EPnP within RANSAC.

    Source: §I · p. 1, B. Symmetric Overlapping Region… · p. 3

  • Results

    CorrI2P outperforms state-of-the-art image-to-point-cloud registration methods on KITTI and NuScenes.

    Source: D. Results · p. 7, §V · p. 9

  • Takeaways & Limitations

    The authors suggest the approach may benefit cross-modal tasks such as distillation and semantic segmentation.

    Source: §V · p. 10

Abstract

from arXiv · show

Motivated by the intuition that the critical step of localizing a 2D image in the corresponding 3D point cloud is establishing 2D-3D correspondence between them, we propose the first feature-based dense correspondence framework for addressing the image-to-point cloud registration problem, dubbed CorrI2P, which consists of three modules, i.e., feature embedding, symmetric overlapping region detection, and pose estimation through the established correspondence. Specifically, given a pair of a 2D image and a 3D point cloud, we first transform them into high-dimensional feature space and feed the resulting features into a symmetric overlapping region detector to determine the region where the image and point cloud overlap each other. Then we use the features of the overlapping regions to establish the 2D-3D correspondence before running EPnP within RANSAC to estimate the camera's pose. Experimental results on KITTI and NuScenes datasets show that our CorrI2P outperforms state-of-the-art image-to-point cloud registration methods significantly. We will make the code publicly available.

I. INTRODUCTION

Visual pose estimation depends on 2D–3D correspondence, which is challenging across image and point-cloud modalities. CorrI2P proposes a feature-based dense-correspondence framework and reports state-of-the-art performance on KITTI and NuScenes.

A. Visual Pose Estimation

  • Visual pose estimation determines an image’s 6DOF camera position in a 3D scene, with correspondence followed by methods such as EPnP used to estimate pose.The paper identifies this task as a basis for SLAM and Structure-from-Motion.Source: §I · p. 1

I. INTRODUCTION

  • Image-to-point-cloud registration requires cross-modal 2D–3D correspondence, which is harder to estimate than same-modal 2D–2D or 3D–3D correspondence.Prior same-modal approaches cannot simply be extended to establish cross-modal correspondence.Source: §I · p. 1
  • CorrI2P is the first feature-based dense-correspondence framework for image-to-point-cloud registration and achieves state-of-the-art performance on KITTI and NuScenes.Its contributions include a symmetric overlapping-region detector and a joint loss for overlap detection and dense correspondence.Source: §I · p. 1
  • CorrI2P embeds image and point-cloud inputs, detects their overlapping regions, establishes correspondence from overlap features, and estimates pose with EPnP within RANSAC.The pipeline is illustrated in Fig. 1.Source: §I · p. 1, Figure 1 · p. 2

B. 2D-3D Correspondence for Registration

Prior image-to-point-cloud registration approaches use selected keypoints, frustum classification, or coarse image-region classification, each with stated constraints. CorrI2P instead builds dense cross-modal correspondence from overlapping image and point-cloud regions and uses it to estimate pose.

B. 2D-3D Correspondence for Registration

  • Independent keypoint detection in 2D3D-MatchNet can reduce inlier ratio and registration accuracy, while DeepI2P’s frustum-border classifications can be erroneous and its region labels provide only coarse correspondence.DeepI2P uses point classification and inverse camera projection to optimize pose.Source: B. 2D-3D Correspondence for Registration · p. 2
  • Other correspondence approaches include methods requiring accurate initialization, methods using projected point clouds, and image-reconstruction methods whose point-cloud features come from images.These approaches establish correspondences differently from CorrI2P’s use of a camera image and point-cloud features.Source: B. 2D-3D Correspondence for Registration · p. 2

III. PROPOSED METHOD

  • CorrI2P’s symmetric overlapping-region detector uses cross-attention to map image and point-cloud features into each other’s feature spaces, then decoders fuse original and mapped features to predict overlap.Fig. 2 illustrates this detector architecture.Source: Figure 2 · p. 3
  • The registration objective is to estimate camera pose in the point-cloud coordinate space from a query image.The pose is denoted T = [R|t], with R in SO(3) and t in R^3.Source: §III · pp. 2–3
  • CorrI2P embeds an image and point cloud, predicts their overlap, builds dense 2D–3D correspondence, and runs EPnP with RANSAC to estimate camera pose.The method’s three modules are feature embedding, symmetric overlap detection, and pose estimation.Source: §III · p. 3

A. Feature Embedding

CorrI2P embeds images and point clouds with separate hierarchical networks, producing local and global features. Its symmetric detector fuses cross-modal features, predicts overlapping regions, and collects features from selected pixels and points.

A. Feature Embedding

  • ResNet and SO-Net separately embed images and point clouds hierarchically, producing pixel-wise and point-wise features at two layers and global features by max-pooling the second-layer features.The feature extractors differ because images and point clouds have different structures and characteristics.Source: A. Feature Embedding · p. 3

B. Symmetric Overlapping Region Detection

  • The symmetric detector applies image-to-point-cloud and point-cloud-to-image cross-attention to map features into each other’s spaces, then uses image and point-cloud decoders to predict overlap.The weighted features are combined with the original modality features before decoding.Source: B. Symmetric Overlapping Region… · p. 3
  • For image-to-point-cloud attention, the image global and point-cloud local features feed an MLP and Softmax to produce weights applied to image local features; the reverse module is symmetric.The resulting weighted image and point-cloud features are used in overlap detection.Source: B. Symmetric Overlapping Region… · pp. 3–4

WP, F2

  • The image decoder produces pixel-wise scores and features, while the point-cloud decoder uses PointNet and point upsampling to produce point-wise scores and features.A shared threshold filters the scores to select overlapping pixels and points and collect their corresponding features.Source: B. Symmetric Overlapping Region… · p. 4, WP, F2 · p. 4

C. Dense Correspondence-based Pose Estimation

CorrI2P matches each 3D point to the nearest image-pixel feature within the detected overlap, assuming matched pairs have similar features and non-matches do not. EPnP within RANSAC estimates pose while rejecting outliers.

C. Dense Correspondence-based Pose Estimation

  • Under the assumption that matched pixel-point pairs have similar features and non-matched pairs have distinct features, nearest-neighbor search in feature space establishes correspondence.Source: C. Dense Correspondence-based Pose… · p. 4
  • For each 3D point in the overlap, CorrI2P selects the overlapping image pixel with the nearest feature as its correspondence.This choice addresses cases where occlusion and limited image resolution project multiple points onto one pixel.Source: C. Dense Correspondence-based Pose… · p. 4
  • EPnP within RANSAC estimates the camera pose from correspondences while rejecting outliers that could otherwise reduce registration accuracy or produce a wrong transformation.The method notes that nearest-feature matching can generate incorrect correspondences.Source: C. Dense Correspondence-based Pose… · p. 4

D. Loss Function

CorrI2P trains overlap detection and feature matching with a joint descriptor and detector loss. The descriptor loss separates matched from non-matched features, while the detector loss scores overlapping pixels and points above non-overlapping ones.

D. Loss Function

  • The joint loss trains both feature matching and overlapping-region detection through descriptor and detector losses.Source: D. Loss Function · p. 4
  • The descriptor loss encourages similar features for matched pixel-point pairs and different features for non-matched pairs.Source: D. Loss Function · p. 4
  • The detector loss encourages high scores for pixels and points inside overlap and low scores outside it.The decoder detectors produce coordinates and features for overlapping regions.Source: D. Loss Function · pp. 4–5, Figure 4 · p. 5
  • Positive descriptor pairs are sampled from ground-truth 2D-3D correspondences in the overlapping region, and their feature distance uses cosine distance.The feature vectors represent image and point-cloud features.Source: D. Loss Function · p. 4
  • For a negative pair, the method selects the smallest feature distance among candidate pixels farther than safe radius R from the projected point.Source: D. Loss Function · p. 4
  • Positive and negative margins are introduced to accelerate training of the descriptor loss.Source: D. Loss Function · p. 4

IV. EXPERIMENTS

The experiments evaluate CorrI2P on KITTI Odometry and NuScenes using dataset-specific image-point cloud pair construction and train-test splits. Both datasets use transformation ranges described as similar.

A. Dataset

  • CorrI2P is evaluated on KITTI Odometry and NuScenes.Source: A. Dataset · p. 5
  • On KITTI, image-point cloud pairs come from the same frame, and the first eight sequences train while the last two test.The camera and LiDAR have fixed relative positions; transformations include up-axis rotation and ground-plane translation within 10 m.Source: A. Dataset · p. 5
  • On NuScenes, point clouds aggregate nearby frames while images come from the current frame; 850 scenes train and 150 test.The point cloud uses a 360-degree-field-of-view LiDAR, and the transformation range is similar to KITTI.Source: A. Dataset · p. 5

B. Implementation Details

Training uses ground-truth camera pose to supervise image-point correspondences, with specified network settings and loss parameters. Testing thresholds the predicted overlap and uses RANSAC with a fixed iteration count and reprojection-error threshold.

B. Implementation Details

  • Ground-truth camera pose and camera intrinsics define the projection used to supervise 3D-point-to-image correspondences.Source: B. Implementation Details · p. 5
  • The image input is scaled to 1/16, while the implementation specifies feature scales, point counts, neighborhood size, and channel dimensions.It uses 25 training epochs per dataset with batch size 24.Source: B. Implementation Details · p. 5
  • Training uses Adam with initial learning rate 0.001, reduced by 0.25 every five epochs to 0.00001; safe radius R is 1 pixel and margins are 0.2 and 1.8.The loss-balance parameter λ is 0.5.Source: B. Implementation Details · p. 6
  • At test time, overlap threshold τ is 0.9, and RANSAC uses 500 iterations with a 1-pixel inlier reprojection-error threshold.Source: B. Implementation Details · p. 6

C. Compared Methods

CorrI2P is compared with DeepI2P-based grid-classification and frustum-classification settings. The evaluation uses RTE and RRE for registration and pair and fragment recall for feature matching.

C. Compared Methods

  • CorrI2P is compared with DeepI2P’s grid-classification-plus-PnP and frustum-classification-plus-inverse-projection settings.The grid setting establishes correspondence from image-grid classifications; the frustum setting uses inverse camera projection.Source: C. Compared Methods · p. 6
  • Grid Cls. + PnP classifies which 32×32 image grid contains each point’s projection, then estimates pose with EPnP and RANSAC.Source: C. Compared Methods · p. 6
  • Frus. Cls. + Inv.Proj. uses frustum classification and inverse camera projection, with 60-fold random initialization and 2D or 3D projection variants.Source: C. Compared Methods · p. 6
  • Registration is evaluated with RTE and RRE, while feature matching is assessed with pair recall and fragment recall.Source: C. Compared Methods · p. 6

D. Results

CorrI2P outperforms the compared methods on KITTI and NuScenes registration, with overlap detection and dense correspondence providing key comparisons. Registration errors are lower on KITTI, where point clouds are denser than NuScenes frames.

D. Results

  • Across KITTI and NuScenes, CorrI2P outperforms all compared methods by a noticeable margin in registration; Fig. 5 compares recall across RTE and RRE thresholds.Source: Figure 5 · p. 7, D. Results · p. 7
  • Frustum-boundary errors can yield wrong poses, while 32×32 grids produce sparse, coarse matches despite classification accuracy above 0.50; CorrI2P instead uses dense correspondence.Source: D. Results · p. 7
  • On KITTI, RTE/RRE modes are ~0.5m/2° versus ~1.5m/2° on NuScenes, with smaller RTE and RRE variances on KITTI.Source: D. Results · p. 7
  • CorrI2P’s overlap detection outperforms Random, SIFT, and ISS, exceeds DeepI2P for point-cloud detection, and has precision above 0.9 for both modalities.The comparison evaluates recall, precision, and F2-score on KITTI.Source: D. Results · p. 8
  • KITTI point clouds are dense enough for direct use, whereas NuScenes frames require adjacent-frame splicing; dynamic objects can leave clouds incompletely aligned and hinder feature extraction.Source: D. Results · p. 8

E. Ablation Study

Ablations examine cross-attention, overlap detection, and point-cloud density. Results indicate that removing overlap detection or reducing point density lowers registration accuracy, while cross-attention benefits feature matching.

E. Ablation Study

  • The ablation without cross-attention fusion and detection loss assesses feature matching; the authors report that cross-attention benefits matching and 2D-3D correspondence.Fig. 8a shows feature-matching recall across inlier-ratio thresholds.Source: E. Ablation Study · p. 8, Figure 8 · p. 9
  • Removing overlap detection lowers registration accuracy; removing point-cloud selection hurts more than removing image selection, as few cloud points project into the image while image overlap is large.The comparison includes random, keypoint, and confidence-score selection strategies.Source: E. Ablation Study · pp. 8–9, Figure 8 · p. 9
  • Reducing point-cloud density lowers registration accuracy because omitted structure weakens feature descriptions and leads to incorrect correspondences.Source: E. Ablation Study · p. 9

F. Efficiency Analysis

CorrI2P showed significant advantages over state-of-the-art methods in experiments on KITTI and NuScenes. Its efficiency trade-offs differ between neural-network inference and pose estimation, and the authors identify potential uses in other cross-modal tasks.

F. Efficiency Analysis

  • Classification-based methods use fewer GPU resources and run faster during inference than CorrI2P; Grid Cls. + EPnP is fastest for pose estimation, while DeepI2P is much slower.CorrI2P extracts features and detects overlap for both modalities; DeepI2P uses 60-fold pose initialization before optimization.Source: F. Efficiency Analysis · p. 9

V. CONCLUSION

  • CorrI2P showed significant advantages over state-of-the-art methods in extensive experiments on KITTI and NuScenes, alongside comprehensive ablation studies.The authors suggest the method may benefit cross-modal distillation and semantic segmentation.Source: §V · pp. 9–10
  • The authors suggest CorrI2P may benefit cross-modal distillation and semantic segmentation, tasks that transform different data types into the same feature space.Source: §V · p. 10

Related papers

Loading 2207.05483v3…