Source-linked AI summary
DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models
Yungsoo Han, Youngseok Jang, Seungwon Roh, Jeongyeon Seo, H. Jin Kim
TL;DR
CMPR must align visual appearance with LiDAR geometry, while existing unified-encoder approaches can require modality-specific backbones, complex preprocessing, or added semantics. DXPR converts camera and LiDAR data into unified depth images and uses geometry-aware overlap mining with a VFM-based feature extractor and aggregator. On KITTI, our method achieves the highest overall performance; DXPR achieves R@1 = 81.75%, R@5 = 87.90%, and R@10 = 90.09%, outperforming RangeBEV [10] and SALAD.
Problem
CMPR must align visual appearance with LiDAR geometry, while existing unified-encoder approaches can require modality-specific backbones, complex preprocessing, or added semantics.
Source: §I · p. 1Method
DXPR converts camera and LiDAR data into unified depth images and uses geometry-aware overlap mining with a VFM-based feature extractor and aggregator.
Source: §I · p. 2, 2) Geometry-Aware Overlap Score · p. 4Results
On KITTI, our method achieves the highest overall performance; DXPR achieves R@1 = 81.75%, R@5 = 87.90%, and R@10 = 90.09%, outperforming RangeBEV [10] and SALAD.
Source: C. KITTI · p. 5, D. Boreas · p. 7Takeaways & Limitations
CMPR enables camera-only platforms to localize against pre-built LiDAR maps, combining camera practicality with LiDAR geometric accuracy and robustness.
Source: §I · p. 1
Abstract
from arXiv · showhide
We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale alignment of camera and LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.
I. INTRODUCTION
Prior place-recognition systems face a modality gap, and existing unified approaches often require modality-specific backbones or complex preprocessing. DXPR maps camera and LiDAR data to depth images for shared VFM processing and introduces geometry-aware overlap mining.
I. INTRODUCTION
- Camera place recognition is sensitive to appearance changes, while LiDAR is more robust but carries greater hardware, power, and processing costs.CMPR seeks to combine camera practicality with LiDAR geometric accuracy and robustness, but faces a modality gap between visual appearance and LiDAR geometry.Source: §I · p. 1
- Existing unified-encoder approaches can require complex preprocessing or additional semantics, while VFM integration with unified camera–LiDAR representations remains relatively unexplored.The cited VFM attempt used DINO features only from RGB images.Source: §I · p. 2
- DXPR converts camera and LiDAR data into a unified depth representation for VFM-based cross-modal place recognition.Its geometry-aware miner aligns depth scales, computes overlap, reclassifies ambiguous pairs, and adjusts the positive loss margin.Source: §I · p. 2
- Earlier pair-mining strategies rely on pose thresholds or predefined FoV areas, which do not fully exploit available sensor geometry.DXPR uses scale alignment and depth projection between views to compute overlap, reclassify ambiguous pairs, and reduce false positives from weakly overlapping views.Source: §I · p. 2
- The pipeline is modular, with interchangeable components that can be extended to other sensing modalities representable as depth images.The contribution summary also reports an ablation study of depth-image representations.Source: §I · p. 2
II. RELATED WORK
Prior CMPR work commonly maps inputs to comparable representations, but often retains modality-specific encoders or requires complex preprocessing. DXPR uses a shared VFM for unified depth inputs and geometry-aware depth overlap for pair mining.
II. RELATED WORK
- CMPR methods commonly map LiDAR to 2D range or BEV images; other approaches combine camera disparity, depth-derived BEV, FoV cues, or semantic features.Examples include LC2, I2P-Rec, ModaLink and RangeBEV, [10],,, VXP, C2L-PR, and InsCMPR.Source: §II · p. 2
- Most systems use modality-specific encoders, while tighter-alignment approaches can require complex preprocessing or added semantic information.DXPR instead uses unified depth inputs with a shared VFM backbone and aggregator.Source: §II · p. 2
- Most CMPR pair mining uses pose thresholds or FoV overlap, while depth-driven geometric covisibility has rarely been used.DXPR aligns cross-modal scales and forward-warps depth to compute overlap scores for reclassifying ambiguous pairs.Source: §II · pp. 2–3
III. METHOD
DXPR combines shared depth-image inputs, geometry-aware overlap mining, and VFM feature aggregation. It evaluates alternative depth encodings because representation choices affect training stability, modality alignment, and retrieval.
A. Overview
- The overview combines unified depth inputs, geometry-aware overlap mining, and a VFM feature extractor and aggregator trained with an overlap-adaptive multi-similarity loss.At inference, camera descriptors are matched against a LiDAR descriptor database.Source: A. Overview · p. 3
B. Depth Representation
- Camera and LiDAR depth maps use metric-depth outputs, and the method crops their overlapping fields of view.Depth Anything and DOC-Depth generate the camera and LiDAR maps, respectively; the camera output is explicitly metric to make their geometric scales comparable.Source: B. Depth Representation · p. 3
- Random image-training augmentations are used to mitigate density and geometric-pattern differences between the modalities.The study compares raw, logarithmic, inverse, three-channel coordinate, and surface-normal depth encodings; log depth achieves the best overall performance among the evaluated representations.Source: B. Depth Representation · p. 3
C. Geometry-Aware Overlap Mining
The module uses LiDAR geometry to correct camera-depth scale, then evaluates cross-view visibility overlap to select more reliable training pairs.
C. Geometry-Aware Overlap Mining
- The overlap miner uses relative pose and depth to correct scale discrepancies and compute visibility-overlap scores, reclassifying ambiguous pairs for more reliable supervision and reduced overfitting.Source: C. Geometry-Aware Overlap Mining · p. 3
1) Scale Alignment:
- LiDAR depth provides a geometric reference for correcting residual scale errors in camera-derived depth.LiDAR points are projected into the camera frame and compared with camera depth.Source: 1) Scale Alignment · p. 3
- Candidate depth scales are scored by inlier pixels under an adaptive depth tolerance, and golden-section search selects the scale used to correct the camera depth map.The tolerance accounts for increasing uncertainty with distance; search stops when |a−b| < 10^-3.Source: 1) Scale Alignment · p. 3
2) Geometry-Aware Overlap Score:
The geometry-aware overlap score measures whether two scale-aligned depth views capture the same scene with consistent pixel coverage. It combines overlap measured in both projection directions, capturing scale and pixel-level consistency that conventional FoV overlap metrics miss.
2) Geometry-Aware Overlap Score:
- The score measures both shared-scene visibility and consistency of pixel coverage between two scale-aligned depth views.This makes overlap sensitive to how consistently the views represent the scene, not just whether they observe it.Source: 2) Geometry-Aware Overlap Score · p. 4
- Measurements are projected in both directions, and pixel-level inlier sets are formed using sampled pixels and an adaptive depth-error tolerance.The method samples on a uniform grid and dilates by the sampling stride during matching.Source: 2) Geometry-Aware Overlap Score · p. 4
- The final score is the harmonic mean of the two directional overlap scores.Each directional score is computed using intersection over union (IoU).Source: 2) Geometry-Aware Overlap Score · p. 4
- Unlike conventional FoV-overlap metrics [10],,, the score incorporates pixel-level consistency and local measurements without requiring a global 3D map.Views observing the same space at different scales receive a substantially lower score, even when their FoV overlap remains similar.Source: 2) Geometry-Aware Overlap Score · p. 4
3) Mining and Training Loss:
The method assigns strong positives and negatives from pose thresholds and uses geometry-aware overlap to resolve ambiguous pairs. It trains a VFM-based feature-extraction and aggregation pipeline with a positive margin adapted to overlap.
3) Mining and Training Loss:
- Ambiguous pairs are classified using geometry-aware overlap, while pose thresholds identify strong positives and negatives without overlap computation.Pairs with very similar poses are strong positives; pairs beyond the negative-distance threshold are strong negatives, and ambiguous pairs are positive only when their overlap exceeds Op.Source: 3) Mining and Training Loss · p. 4
- Fig. 3 contrasts similar-scale views, which yield a higher geometry-aware overlap score, with different-scale views, which yield a much lower score.Conventional predefined FoV overlap scores are similar in the two cases because they do not account for pixel-level resolution differences.Source: Figure 3 · p. 5
- Mined depth pairs and overlap scores train a DINOv2-SALAD backbone–aggregator pipeline with modified multi-similarity loss.The positive base margin is λ(O) = a + b · O, adapting it to overlap to reduce overfitting to weakly overlapping views.Source: 3) Mining and Training Loss · pp. 4–5
A. Datasets
The evaluation uses KITTI odometry and Boreas, with Boreas providing repeated-route data across diverse weather and lighting. The implementation uses dataset-specific depth handling, evaluation protocols, and pair-mining thresholds.
A. Datasets
- Boreas provides routes recorded over more than a year in diverse weather and lighting, supporting intra- and inter-sequence robustness evaluation.KITTI is used primarily for comparison with prior work, while Boreas tests performance under challenging conditions.Source: A. Datasets · p. 5
- KITTI uses two evaluation splits: sequences 11–21 train against sequences 00–10, or the first 3,000 frames of sequence 00 train with later frames and sequences 02, 05, 06, and 08 evaluated.The two protocols follow [10] and InsCMPR, respectively.Source: A. Datasets · p. 5
- Boreas follows the training, validation, and evaluation split in Table I; comparisons with RangeBEV [10] and DINOv2-SALAD reproduce their papers and publicly released code.For the Boreas comparison, DINOv2-SALAD uses relative rotation and translation thresholds of 20° and 10 m to define positive pairs.Source: Table I · p. 5, A. Datasets · p. 5
B. Implementation Details
- KITTI depth uses the pretrained Depth Anything metric-depth model without further scale alignment, whereas Boreas depth is aligned because its estimated scale differs from LiDAR ground truth by about 0.6 on average.Depth images are resized to 150 × 744 for KITTI and 112 × 341 for Boreas; Boreas Recall@1 excludes the query sequence from the database.Source: B. Implementation Details · p. 5
- Pair classification uses θp = 10°, dp = 5 m, dn = 20 m, and minimum positive inlier ratio Op = 0.35.The adaptive-margin parameters are a = 0.2 and b = −0.1; alignment uses τabs = 0.5 m, τrel = 0.1, and scale range [0.3, 2.0].Source: B. Implementation Details · p. 5
C. KITTI
DXPR achieves the highest overall performance among the compared methods on KITTI. Its results are especially strong when sufficient training data is available and the environment is not highly ambiguous, while repetitive scenery remains difficult.
C. KITTI
- Inter-sequence success rate approaches 100.0% when training data is sufficient and the environment is not highly ambiguous.KITTI’s synchronized LiDAR and camera observations for database and query construction also yield relatively high success rates.Source: C. KITTI · p. 5
- DXPR achieves the highest overall performance among the compared algorithms on KITTI.The comparison is reported in Tables II and III.Source: C. KITTI · p. 5, Table II · p. 6, Table III · p. 6
- On the entire validation set, our method provides more stable and higher Recall@1 performance than RangeBEV; Sequences 01 and 04 show a drop because repetitive, indistinguishable scenery is difficult to resolve with geometric features alone.The notation @N m denotes a true-positive threshold of N m.Source: Figure 4 · p. 6, C. KITTI · p. 6
D. Boreas
On Boreas, DXPR performs strongly in both same-sequence and cross-sequence retrieval across repeated routes and changing conditions. Its inter-sequence results exceed RangeBEV and SALAD, including under severe appearance changes such as nighttime.
D. Boreas
- Boreas repeats routes across seasons and environmental conditions, enabling separate evaluation of same-sequence retrieval and cross-sequence retrieval.Fig. 5 places intra-sequence results on the diagonal and inter-sequence results off the diagonal.Source: D. Boreas · p. 6, Figure 5 · p. 7
- On the Boreas dataset at a 5 m true-positive threshold, DXPR achieves R@1 = 81.75%, R@5 = 87.90%, and R@10 = 90.09%, outperforming RangeBEV [10] and SALAD.DXPR also remains robust under severe appearance changes, including nighttime, where competing methods degrade substantially.Source: Table IV · p. 6, D. Boreas · p. 7
- In intra-sequence evaluation, DXPR achieves R@1 = 95.34%, R@5 = 97.14%, and R@10 = 97.71%, below single-modal DINOv2-SALAD.SALAD's same-sensor, identical-condition retrieval setting can retain highly similar database samples even after self-matching is excluded, unlike cross-modal retrieval.Source: Table IV · p. 6, D. Boreas · p. 6
- The authors attribute robustness to fine-tuned DINOv2 attention to static structure, reliable monocular depth estimation, and LiDAR's resilience to appearance changes.Fig. 6 illustrates the attention change after fine-tuning.Source: D. Boreas · p. 7, Figure 6 · p. 7
E. Ablation Study
The ablation identifies log depth as the best-performing tested representation, and the conclusion presents DXPR as a unified depth-based CMPR framework with future extensions to modalities and model components.
E. Ablation Study
- Raw depth is replicated across three channels and normalized before network input, so scale calibration is unnecessary and scale correction has no measurable effect.Log and inverse depth are also replicated across three channels; Log+Normal combines log depth with camera-frame y and z surface-normal components.Source: E. Ablation Study · p. 7
- Log depth achieves the best overall performance among the evaluated depth representations.The ablation compares replicated raw depth, log depth, inverse depth, log depth with surface normals, and camera-frame XYZ coordinates.Source: E. Ablation Study · p. 7, Table V · p. 7
V. CONCLUSION
- The conclusion describes a unified depth-based CMPR framework using VFMs and geometry-aware overlap mining, with future work extending to more modalities and backbone–aggregator combinations.The framework's stated extensibility supports integrating future VPR backbones, aggregators, and objectives into cross-modal recognition.Source: §V · p. 7