PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Aligned Semantic Matching for Cross-Modal Planar Image Registration

Zhiwei Wang, Defeng He, Yuxing Li, Meilu Zhu, Edmund Y. Lam

arXiv:2610.03167v1cs.CV

TL;DR

Cross-modal matching must balance stable correspondences with precise localization, yet semantic similarity alone may miss geometric matches and fine-scale CNN cues can be modality-sensitive. CDPM adapts DINOv3 with geometric patch supervision and preserves its dominant role during CNN-assisted refinement. On VIS-IR, it achieves the best performance across all reported metrics and uses 45.6% fewer FLOPs than RoMa v2.

  • Problem

    Cross-modal appearance differences make feature similarity unreliable for geometric matching, while semantic features lack spatial precision and fine-scale features can cause refinement drift.

    Source: §I · pp. 1–2

  • Method

    CDPM adapts DINOv3 using geometrically consistent cross-modal patch pairs and uses a DINO-Centric Feature Pyramid with auxiliary CNN details for localization.

    Source: §I · p. 2

  • Results

    On VIS-IR, CDPM achieves the best performance across all evaluation metrics and reduces FLOPs by 45.6% compared with RoMa v2.

    Source: C. Comparison with State-of-the-Art… · p. 8, E. Complexity Discussion · p. 10

  • Takeaways & Limitations

    The reported results show CDPM combining stable cross-modal correspondences with accurate fine-grained localization for planar registration.

    Source: §V · p. 11

Abstract

from arXiv · show

Cross-modal image matching establishes stable and accurate geometric correspondences across modalities for planar registration. Existing semantic representations provide cross-modal consistency, but semantic similarity does not necessarily imply geometric correspondence. Meanwhile, fine-grained CNN features provide accurate local details but lack global cross-modal semantic guidance for stable refinement. To address these issues, we propose CDPM, which first establishes geometrically consistent semantic representations and then preserves their dominant role in correspondence estimation during fine-grained localization. Specifically, we progressively adapt DINOv3 using geometrically consistent cross-modal patch pairs, enabling feature similarity to better reflect true cross-modal spatial correspondences. We then construct a DINO-Centric Feature Pyramid, where multi-scale DINO representations maintain stable cross-modal correspondences, while a lightweight CNN branch provides auxiliary structural details for precise local refinement. Extensive experiments on three cross-modal datasets demonstrate the superior performance of CDPM. On VIS-IR, compared with the dense matcher RoMa, CDPM improves AUC@3/5/10/20 by 7.36, 13.40, 13.75, and 10.42 percentage points, respectively, and reduces mACE from 5.83 to 2.78 pixels. It also outperforms RoMa v2 across all metrics while requiring 45.6% fewer FLOPs. The online demo and dataset are available, and the code will be released on our project page at https://warren-wzw.github.io/CDPM/.

I. INTRODUCTION

Cross-modal planar registration depends on reliable correspondences, but semantic stability and fine localization are difficult to achieve together across modality gaps. CDPM adapts DINOv3 with geometric supervision and keeps DINO features dominant during CNN-assisted refinement.

I. INTRODUCTION

  • Cross-modal appearance differences make pixel correspondences difficult, requiring a balance between correspondence stability and localization accuracy.Robust cross-modal features can be too coarse, while fine-grained features tend to be modality-sensitive.Source: §I · p. 1
  • Semantic similarity may not identify the same physical location, especially with repetitive structures, low light, or substantial modality gaps.CNN refinement can sharpen responses yet shift their peaks away from ground truth.Source: §I · pp. 1–2
  • CDPM adapts DINOv3 with geometrically consistent positive and negative patch pairs, then uses a DINO-Centric Feature Pyramid to keep DINO correspondences primary and CNN details auxiliary.The design aims to improve geometric correspondence while maintaining stable matching during fine-scale refinement.Source: §I · p. 2
  • CDPM is evaluated on two public datasets and PCB-Layout, an industrial dataset of PCB images and design layouts.The authors report superior registration performance across these cross-modal settings.Source: §I · p. 2

A. Cross-modal Homography Estimation

  • CDPM focuses on reliable dense cross-modal correspondences for homography recovery, unlike methods that directly estimate homographies or match dense grids.The homography is recovered from correspondences using robust estimation.Source: A. Cross-modal Homography Estimation · p. 2

B. Detector-Free and Dense Matching

Detector-free and dense matchers use coarse-to-fine features, but cross-modal appearance variation can undermine correspondences during refinement. CDPM adapts DINOv3 with geometric patch supervision and propagates DINO correspondence cues across scales, with CNN features supplementing localization.

B. Detector-Free and Dense Matching

  • Under substantial modality gaps, modality-sensitive fine-scale features can disturb reliable coarse matches and cause refinement drift.Prior detector-free and dense approaches progressively refine correspondences, but cross-modal feature consistency remains a challenge.Source: B. Detector-Free and Dense Matching · p. 2

C. Generalizable Cross-modal Image Matching

  • Foundation-model features support robust coarse matching, but generic semantic similarity is not explicitly aligned with cross-modal geometric correspondence.Prior methods combine DINO features with CNN localization, add multi-view interaction, or scale cross-modal training data.Source: C. Generalizable Cross-modal Image… · pp. 2–3

A. Problem Formulation and Overview

  • CDPM predicts a dense correspondence field and confidence map before selecting reliable matches to estimate the homography robustly.Its stated challenges are unreliable cross-modal feature similarity, limited spatial precision of semantic features, and refinement drift.Source: A. Problem Formulation and Overview · p. 3
  • CDPM adapts DINOv3 from geometrically corresponding cross-modal patches, then propagates multi-level DINO features toward higher resolutions with CNN structural cues for localization.The framework uses coarse-scale correspondences followed by residual refinement, limiting updates at modality-sensitive fine scales.Source: A. Problem Formulation and Overview · pp. 3–4

B. Patch-level DINOv3 Adaptation

  • Random homographies place the same content patch at different token positions, discouraging reliance on fixed token indices as a correspondence shortcut.Mapped corresponding tokens become positive pairs, other locations are negatives, and a patch-level contrastive loss trains their cross-modal identifiability.Source: B. Patch-level DINOv3 Adaptation · p. 4
  • DINOv3 is adapted in stages, first optimizing Layers 8–15 for structural correspondences and then Layers 16–23 for high-level semantic consistency.Only attention Q/K/V projection matrices are updated in each stage; the adapted backbone is frozen during downstream matching training.Source: B. Patch-level DINOv3 Adaptation · p. 4

C. DINO-Centric Feature Pyramid

The DINO-Centric Feature Pyramid (DCFP) keeps multi-level DINO representations as the primary cross-modal correspondence features and selectively adds CNN details. It progressively builds a coarse-to-fine DINO hierarchy to recover spatial structure while maintaining alignment.

C. DINO-Centric Feature Pyramid

  • DCFP uses multi-level DINO representations as primary correspondence features across scales, with lightweight CNN features selectively adding structural cues for precise localization.DINO correspondences propagate from coarse to fine scales, while CNN details supplement local spatial information.Source: C. DINO-Centric Feature Pyramid · p. 4

1) Multi-level DINO Pyramid Construction:

  • L23 features establish cross-view correspondences at P16, and bidirectional cross-attention produces cross-view-enhanced features for robust matching.The projected features are augmented with positional encoding before cross-attention.Source: 1) Multi-level DINO Pyramid Construction · pp. 4–5
  • At P8 and P4, the Spatial Feature Block upsamples the preceding coarse feature, concatenates it with a scale-aligned DINO feature, and processes the result with depthwise-separable convolution.Intermediate DINO layers are projected into a shared channel space before progressive spatial recovery.Source: 1) Multi-level DINO Pyramid Construction · p. 5
  • The hierarchy restores spatial structure at P8 and P4 and propagates the aggregated representation to P2, while injecting cross-attended P16 features into P8 and P4 to maintain coarse-to-fine alignment.The pyramid construction is illustrated in Fig. 3.Source: 1) Multi-level DINO Pyramid Construction · p. 5, Figure 3 · p. 5

2) DINO-guided CNN Detail Enhancement:

The detail-enhancement design preserves DINO as the main correspondence pathway and adds CNN features as auxiliary structural cues. Its scale-specific fusion limits the influence of modality-sensitive details while supporting fine localization.

2) DINO-guided CNN Detail Enhancement:

  • The DINO pyramid remains the semantic anchor because patch-level features alone struggle to capture fine local structures, while CNN features can be more susceptible to modality gaps and correspondence drift.CNN details are therefore incorporated selectively rather than treated as equal correspondence features.Source: 2) DINO-guided CNN Detail Enhancement · pp. 5–6
  • At P8 and P4, DINO features serve as the primary correspondence representations and corresponding CNN features enter through residual fusion.The CNN branch extracts detail features at P1, P2, P4, and P8; P2 and P1 preserve finer edge and texture cues.Source: 2) DINO-guided CNN Detail Enhancement · p. 5
  • At P2, a conditioned adaptive gate uses DINO and CNN features to modulate CNN details at each spatial location and feature channel before residual injection.At P1, a shallow full-resolution CNN branch is fused with upsampled P2 context, and P1 is retained for limited local refinement.Source: 2) DINO-guided CNN Detail Enhancement · pp. 5–6
  • The enhanced P4 feature retains stable DINO correspondences while improving localization with fine structural details.P4 is the principal fine-scale correspondence representation; P1 and P2 mainly supply auxiliary structural cues for limited refinement and detail fusion.Source: 2) DINO-guided CNN Detail Enhancement · p. 6
  • DINO remains the primary correspondence representation, while CNN features selectively supplement local structures to balance cross-modal stability and spatial precision.The design is intended to mitigate refinement drift as reliable coarse correspondences are refined into localizations.Source: 2) DINO-guided CNN Detail Enhancement · p. 6

D. Coarse-to-Fine Matching

CDPM first estimates global correspondences at P16, then progressively refines the correspondence field at finer pyramid scales. Local correlation and residual updates refine positions, while certainty estimates support selecting reliable correspondences for homography recovery.

D. Coarse-to-Fine Matching

  • The cross-attended P16 features form a global correlation matrix used to predict the coarse correspondence field and certainty map that initialize refinement.The final dense flow and certainty map are upsampled to image resolution, and reliable correspondences are selected to estimate the homography using RANSAC.Source: D. Coarse-to-Fine Matching · p. 6
  • Starting from P16, the correspondence field is progressively refined at P8, P4, P2, and P1 using upsampled estimates and local correlation volumes around predicted positions.At each scale, a lightweight refinement module predicts a residual displacement.Source: D. Coarse-to-Fine Matching · p. 6
  • As feature resolution increases, the local search range shrinks; CDPM performs two refinement iterations at P8 and P4 and one at P2 and P1.Each refinement scale also predicts a certainty map for the updated correspondences.Source: D. Coarse-to-Fine Matching · p. 6

E. Loss Function

CDPM uses separate training phases for DINO adaptation and matching, with a patch-level contrastive objective and a multi-scale correspondence objective.

E. Loss Function

1) Patch-level contrastive loss:

The patch-level contrastive loss trains cross-modal features to distinguish geometrically corresponding patches from other reference patches.

1) Patch-level contrastive loss:

2) Multi-scale correspondence loss:

The multi-scale objective supervises predicted correspondence fields and certainty maps against homography-derived targets, with fine-scale supervision restricted by the preceding prediction's refinement range.

2) Multi-scale correspondence loss:

  • At each pyramid scale and refinement iteration, the network predicts a correspondence field and certainty map, while the ground-truth field comes from applying the homography to the query grid.Source: 2) Multi-scale correspondence loss · p. 6
  • Correspondence regression is evaluated over valid locations at each scale and iteration using a robust regression function.Source: 2) Multi-scale correspondence loss · p. 6
  • The certainty map uses binary cross-entropy supervision, and fine-scale supervision is limited to correspondences within the preceding coarser prediction's refinement range.The overall matching objective combines these losses with α = 0.5, δ = 10^-4, and λ_cert = 0.01.Source: 2) Multi-scale correspondence loss · p. 7

IV. EXPERIMENTS

CDPM is evaluated on visible–infrared, satellite–map, and PCB image–layout registration, with the strongest reported results across the three datasets. The experiments also assess qualitative matching, implementation settings, and computational efficiency.

A. Datasets and Evaluation Protocol

  • The evaluation uses VIS-IR and GoogleMap training sets with 1,000 test pairs each, plus PCB-Layout with 3,000 training and 500 test pairs.PCB-Layout contains 100 PCB surfaces and 700 images captured under dark, normal, and bright illumination.Source: A. Datasets and Evaluation Protocol · p. 7
  • For correspondence-based methods, the protocol retains the top 5,000 confidence-ranked matches and estimates homographies with RANSAC.Higher AUC and lower mACE indicate better accuracy.Source: A. Datasets and Evaluation Protocol · p. 7

B. Implementation Details

  • DINO adaptation optimizes Q/K/V projections in Layers 8–15 and 16–23 sequentially, while downstream matching training keeps the adapted backbone frozen.Images are resized to 448 × 448, and adaptation is performed separately using each dataset's training set.Source: B. Implementation Details · p. 7, Figure 5 · p. 8

C. Comparison with State-of-the-Art Methods

  • CDPM achieves the best VIS-IR results across all metrics and outperforms the second-best methods on GoogleMap and PCB-Layout.Source: C. Comparison with State-of-the-Art… · pp. 8–9
  • On VIS-IR, CDPM improves AUC@3, AUC@5, AUC@10, and AUC@20 over the corresponding second-best methods by 7.36, 13.40, 12.50, and 6.83 percentage points, respectively.It reduces mACE from 3.62 to 2.78 pixels.Source: C. Comparison with State-of-the-Art… · pp. 8–9
  • On GoogleMap, CDPM exceeds GFNet by 1.09, 1.58, 1.35, and 0.75 on AUC@3, AUC@5, AUC@10, and AUC@20, respectively.On PCB-Layout, it surpasses the corresponding second-best results by 5.17, 8.24, 7.47, and 3.94 percentage points on those metrics, respectively.Source: C. Comparison with State-of-the-Art… · p. 9
  • Qualitative comparisons show CDPM maintaining more accurate alignment and more continuous, dense, geometrically consistent correspondences in challenging regions.Other methods show boundary deviations or erroneous matches in low-light, weak-texture, appearance-discrepant, and repetitive regions.Source: C. Comparison with State-of-the-Art… · p. 9

D. Cross-modal Transferability

A model trained on GoogleMap is applied without fine-tuning to unseen modality combinations, where qualitative examples show geometrically coherent correspondences. CDPM also reports lower computational cost than the compared dense matchers.

D. Cross-modal Transferability

  • Without fine-tuning, the GoogleMap-trained model establishes geometrically coherent correspondences on unseen RGB-T, RGB-Event, RGB-Depth, and Retina data.The transferability examples are qualitative and include challenging illumination and modalities with substantially different imaging characteristics.Source: D. Cross-modal Transferability · p. 9, Figure 7 · p. 10

E. Complexity Discussion

  • CDPM uses 45.6% fewer FLOPs and 64.1% less inference time than RoMa v2, and 52.2% fewer FLOPs and 69.7% less inference time than RoMa.The comparison uses identical settings and 448 × 448 input resolution.Source: E. Complexity Discussion · p. 10

F. Ablation Study

The ablations show that geometric patch-level DINO adaptation and a DINO-centric feature pyramid support stable cross-modal correspondences and accurate fine-scale localization.

F. Ablation Study

  • The ablations vary DINO adaptation, feature construction, and refinement on VIS-IR, with other components held fixed within each group.Cross-modal patch retrieval also assesses representation-level correspondence under different DINO adaptation strategies.Source: F. Ablation Study · p. 10, Table IV · p. 10

1) Effect of DINO Adaptation:

2) Effect of DCFP and Refinement:

  • 41.43 AUC@3 and 2.78-pixel mACE are achieved with two refinement iterations at P8–P4 and one at P2/P1.This refinement schedule gives the best overall performance among the tested strategies.Source: Table IV · p. 10, 2) Effect of DCFP and Refinement · p. 11
  • Multi-scale DINO reduces mACE from 3.40 to 3.12 pixels versus DINO-P16 with CNN features, and complete DCFP reduces it further to 2.78 pixels.In challenging examples, CDPM reduces localization error from 9.98 to 1.11 pixels in low light and from 8.59 to 1.24 pixels with repetitive textures.Source: Table IV · p. 10, 2) Effect of DCFP and Refinement · p. 11

V. CONCLUSION

  • CDPM preserves multi-scale DINO correspondences while adding CNN structural details, yielding accurate localization and superior registration performance across multiple cross-modal datasets.The conclusion also reports fewer FLOPs and lower inference time than representative dense matchers.Source: §V · p. 11, 2) Effect of DCFP and Refinement · p. 11
Loading 2610.03167v1…