PaperlayerView exact revision on arXiv

Source-linked AI summary

DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

Wenxin Zhang, Hang Li, Zhiwei Xu, Qiankun Dong, Gang Wang, Tao Li

arXiv:2608.26589v1cs.CV

TL;DR

Image-to-Point Cloud Registration must estimate camera pose across heterogeneous image and sparse point-cloud modalities, but reliable cross-modal correspondence remains challenging. DPA-I2P combines metric-depth and projection-consistent feature construction with early query pruning, reducing KITTI RTE and RRE by 45.0% and 55.6% over the strongest implicit baseline, respectively, while improving accuracy on nuScenes.

  • Problem

    Image-to-Point Cloud Registration remains challenging because image and sparse LiDAR point-cloud representations have an inherent modality gap that complicates reliable cross-modal correspondence learning.

  • Method

    DPA-I2P uses RMDE and PVL to construct geometry-aware cross-modal features, while CQP suppresses unreliable queries during early correspondence refinement.

  • Results

    45.0% and 55.6% reductions in RTE and RRE, respectively, are reported on KITTI over the strongest implicit baseline, with improved registration accuracy on nuScenes.

  • Takeaways & Limitations

    DPA-I2P demonstrates superior registration performance, robustness, and cross-dataset generalization on KITTI and nuScenes.

Abstract

from arXiv · show

Image-to-Point Cloud Registration aims to estimate the camera pose of a given image within a 3D scene point cloud, which is a fundamental task in autonomous driving and large-scale outdoor localization. Recent implicit correspondence learning methods have improved registration performance by learning cross-modal alignment in an end-to-end framework, leading to more accurate camera pose estimation. However, due to the inherent modality discrepancy between images and sparse LiDAR point clouds, reliable cross-modal correspondence learning remains challenging. To address this issue, we propose Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration (DPA-I2P). Unlike naive depth or feature concatenation, Ray-Conditioned Metric Depth Encoding (RMDE) and Projection-Consistent Vision Lifting (PVL) exploit depth and visual cues in a structured, geometry-aware manner. In addition, Cross-Modal Query Pruning (CQP) suppresses unreliable queries during early refinement to improve matching stability. Experiments on KITTI and nuScenes demonstrate the effectiveness of the proposed method. On KITTI, DPA-I2P reduces RTE and RRE by 45.0% and 55.6% over the strongest implicit baseline, respectively. On nuScenes, DPA-I2P also improves registration accuracy over the evaluated baselines, suggesting better transferability to different driving scenes.

I. INTRODUCTION

Image-to-Point Cloud Registration estimates camera pose in 3D point clouds but remains difficult because image and sparse LiDAR modalities have inconsistent representations. DPA-I2P addresses this through depth- and projection-aware feature construction plus early query pruning, achieving strong KITTI results.

  • Motivation: Image-to-Point Cloud Registration estimates an image camera pose within a 3D scene point cloud for autonomous driving applications.The task supports visual localization, camera relocalization, SLAM, motion planning, and 3D reconstruction.
  • Motivation: Cross-modal feature inconsistency makes reliable correspondence learning difficult, especially with sparse LiDAR observations in large-scale outdoor scenes.Image features may lack metric-scale geometric awareness, while point-cloud features lack reliable visual grounding, creating correspondence ambiguity and optimization instability.
  • Method: DPA-I2P constructs a more discriminative cross-modal feature space by integrating metric depth and projection-aware visual cues.The framework uses depth and projection consistency to improve geometry-aware image and point representations.
  • Method: RMDE injects metric depth into image features, while PVL transfers projection-consistent visual information to point features.Together, the modules enhance cross-modal geometric consistency for implicit correspondence learning.
  • Method: CQP suppresses unreliable candidates during early correspondence refinement to improve correspondence stability and pose-estimation robustness.This targets instability during the early stages of implicit correspondence refinement.
  • Results: 45.0% and 55.6% reductions in RTE and RRE, respectively, are reported on KITTI relative to the strongest implicit baseline.The paper also reports qualitative robustness and cross-scene generalization results on nuScenes.

B. Monocular Metric Depth Estimation

DPA-I2P uses frozen monocular metric depth and confidence to construct ray-aware image features rather than simply concatenating depth. RMDE samples and aggregates features around predicted metric surfaces before implicit correspondence learning.

  • Monocular Metric Depth Estimation: UniDepthV2 supplies frozen metric depth and pixel-wise confidence maps for geometry-aware image and point representations.Confidence indicates prediction reliability and guides RMDE and PVL.
  • Monocular Metric Depth Estimation: RMDE samples N candidate points along each viewing ray using predicted metric depth as the reference surface.Higher confidence narrows the sampling range, while lower confidence permits a broader ray interval.
  • Monocular Metric Depth Estimation: Each sampled point encodes 3D location, ray direction, and depth offset relative to the predicted surface.A shared encoder produces sampled ray features that are aggregated along each ray.
  • Monocular Metric Depth Estimation: RMDE injects the aggregated ray-aware representation into image features through a gated residual update.This gives image tokens uncertainty-aware local ray distributions instead of appended monocular depth.

C. Projection-Consistent Vision Lifting

PVL addresses the remaining view-independence of point features by projecting 3D points into the image feature plane under a coarse pose prior. It lifts the gathered visual cues through a bounded residual update, while avoiding noisy late-stage priors.

  • Projection-Consistent Vision Lifting: PVL projects 3D points onto the image feature plane using the coarse pose prior and lifts projection-consistent visual cues into point features.This supplies vision consistency to otherwise view-independent point representations.
  • Projection-Consistent Vision Lifting: Projected image features are gathered at valid point locations and mapped into the point-feature space.Each projection has a validity indicator, and center-cell feature gathering retrieves the corresponding image feature.
  • Projection-Consistent Vision Lifting: PVL incorporates the lifted vision cue through a bounded residual update to reduce sensitivity to coarse pose errors.Invalid projections preserve the original point features.
  • Projection-Consistent Vision Lifting: PVL is enabled only in early stages and disabled during late refinement to avoid noisy vision priors.

D. Cross-Modal Query Pruning

CQP stabilizes early correspondence exploration by constructing a projective support prior from projected points and pruning queries with very low support. Pruning is relaxed in later refinement stages.

  • Cross-Modal Query Pruning: CQP introduces a projective support prior because early correspondence queries can drift toward geometrically unsupported regions.
  • Cross-Modal Query Pruning: At each refinement stage, points are projected onto the feature plane using the pose estimate from the preceding stage.The projections include validity indicators and are used to construct dense support.
  • Cross-Modal Query Pruning: The normalized support prior is flattened and combined with image-token similarity to produce support-aware query-token matching.
  • Cross-Modal Query Pruning: A conservative pruning mask removes regions with extremely low support during early refinement stages.Query pruning is gradually relaxed later.

E. Loss Function

DPA-I2P jointly optimizes pose regression and support-guided query regularization. Training first optimizes the coarse pose branch independently, then trains the full framework end-to-end.

  • Loss Function: The overall objective jointly optimizes pose regression and support-guided query regularization.
  • Loss Function: Pose regression uses translation error plus geodesic rotation distance, with λR balancing the two terms.
  • Loss Function: Support-guided regularization aligns early correspondence exploration with projective support.The support distribution is defined for queries over early refinement stages.
  • Loss Function: Training first optimizes the coarse pose branch independently, after which the entire framework is trained end-to-end.

IV. EXPERIMENTS

The experiments evaluate DPA-I2P quantitatively on KITTI and qualitatively on KITTI and nuScenes, with ablations assessing individual components.

  • Experiments report quantitative KITTI results, qualitative visualizations on KITTI and nuScenes, and ablations of each component.

A. Implementation Details

DPA-I2P uses ResNet-FPN and KPFCNN backbones with attention-based correspondence refinement, while frozen UniDepthV2 priors provide metric depth and confidence cues.

  • The image and point branches use a 4-stage ResNet-FPN and 4-stage KPFCNN, respectively, producing 128-dimensional features.
  • The framework uses 4-head attention, 128 correspondence queries, and three refinement stages.
  • UniDepthV2 metric depth and confidence maps are pre-computed offline and frozen as geometric priors without fine-tuning.

1) KITTI [29].:

The evaluation uses KITTI for quantitative registration assessment and nuScenes for qualitative cross-scene comparison, with RTE, RRE, and registration accuracy as metrics.

  • KITTI: KITTI contains 22 synchronized image–LiDAR sequences, with sequences 0–8 for training and 9–10 for evaluation.
  • Evaluation metrics: The evaluation uses average RTE, average RRE, and registration accuracy, where Acc requires RTE < 2 m and RRE < 5°.
  • Quantitative comparison: DPA-I2P reduces RTE from 0.20 m to 0.11 m and RRE from 1.24° to 0.55° versus ICLI2P, while reaching 99.70% registration accuracy.
  • Qualitative comparison: Qualitative KITTI and nuScenes comparisons project point clouds into image space using predicted poses and camera intrinsics, with colors encoding depth.
  • Ablation study: The ablation study evaluates variants without RMDE, PVL, or CQP to assess each component's contribution.

1) Effects of the RMDE.:

Ablations show that RMDE improves all KITTI metrics, PVL chiefly benefits rotation estimation, and CQP is important for stable early correspondence refinement.

  • Effects of RMDE: RMDE improves KITTI RTE from 0.12 m to 0.11 m, RRE from 0.58° to 0.55°, and Acc from 99.61% to 99.70%.
  • Effects of PVL: Without PVL, RRE increases from 0.55° to 0.74°, the largest degradation among variants, while other metrics remain relatively stable.
  • Effects of CQP: Without CQP, RTE increases from 0.11 m to 0.18 m and Acc drops from 99.70% to 98.82%, the most significant performance degradation.
  • RMDE design choices: Naive metric-depth concatenation only marginally improves RRE from 0.58° to 0.57° and Acc from 99.61% to 99.63%.
  • RMDE design choices: The full RMDE reaches 0.11 ± 0.12 m RTE, 0.55 ± 0.67° RRE, and 99.70% Acc through ray geometry, local sampling, and confidence-aware aggregation.

5) Effects of the Pruning Schedule.:

The pruning schedule is most effective when CQP is applied during the first two refinement stages, while DPA-I2P maintains moderate computational overhead and improved registration accuracy.

  • Pruning schedule: RTE drops from 0.18 m to 0.11 m and RRE from 0.65° to 0.55° when query pruning is applied only in the first two refinement stages.Applying pruning throughout all stages performs worse, indicating that later refinement benefits from more flexible local correspondence exploration.
  • Pruning schedule: Applying pruning throughout all refinement stages yields inferior results compared with pruning only in the first two stages.The authors attribute this to the need for more flexible local correspondence exploration during later refinement.
  • Efficiency: DPA-I2P uses 179.93 MB of network storage, 11.15 GB of GPU memory, and 36.81 ms for end-to-end inference.Measurements use 160 × 512 images, 40,960-point clouds, batch size 4, and an NVIDIA GeForce RTX 4090 GPU.
  • Efficiency: Compared with ICLI2P, DPA-I2P increases network size by approximately 2.3%, GPU memory by 3.8%, and inference time by 4.8%.The reported increases are paired with substantially improved registration accuracy.
  • Efficiency: The overall inference efficiency remains within the range of recent implicit correspondence learning methods despite the added RMDE, PVL, and CQP modules.Reported inference includes feature extraction, correspondence refinement, and differentiable pose solving.
Loading 2608.26589v1…