PaperlayerView exact revision on arXiv

Source-linked AI summary

MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training

Xingyi He, Hao Yu, Sida Peng, Dongli Tan, Zehong Shen, Hujun Bao, Xiaowei Zhou

arXiv:2501.07556v1cs.CV

TL;DR

Cross-modality image matching lacks broadly generalizable models because annotated training data are scarce. The paper uses large-scale pre-training with synthetic cross-modal signals and diverse data, achieving universal generalization across more than eight unseen tasks with one network weight while outperforming existing methods.

  • Problem

    Existing detector-free matchers trained mainly on single-modality data generalize poorly to cross-modality tasks, while dense annotated correspondences for each task are scarce.

  • Method

    The framework pre-trains detector-free matchers using diverse data and synthetic image-style translations to learn fundamental structures across modalities.

  • Results

    Using a single network weight, the trained models generalize across more than eight unseen real-world cross-modal tasks and outperform existing matching and alignment methods.

  • Takeaways & Limitations

    The framework extends cross-modality matching capabilities across scientific applications including medical imaging, histopathology, remote sensing, and autonomous systems.

  • Takeaways & Limitations

    The framework performs poorly for aerial-to-ground matching because it does not effectively mimic the extreme perspective and appearance differences between these views.

Abstract

from arXiv · show

Image matching, which aims to identify corresponding pixel locations between images, is crucial in a wide range of scientific disciplines, aiding in image registration, fusion, and analysis. In recent years, deep learning-based image matching algorithms have dramatically outperformed humans in rapidly and accurately finding large amounts of correspondences. However, when dealing with images captured under different imaging modalities that result in significant appearance changes, the performance of these algorithms often deteriorates due to the scarcity of annotated cross-modal training data. This limitation hinders applications in various fields that rely on multiple image modalities to obtain complementary information. To address this challenge, we propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to train models to recognize and match fundamental structures across images. This capability is transferable to real-world, unseen cross-modality image matching tasks. Our key finding is that the matching model trained with our framework achieves remarkable generalizability across more than eight unseen cross-modality registration tasks using the same network weight, substantially outperforming existing methods, whether designed for generalization or tailored for specific tasks. This advancement significantly enhances the applicability of image matching technologies across various scientific disciplines and paves the way for new applications in multi-modality human and artificial intelligence analysis and beyond.

1. Introduction

Cross-modality image matching is essential for registration and analysis across scientific disciplines but is difficult because imaging differences cause substantial appearance changes and annotated data are scarce. The proposed large-scale pre-training framework uses cross-modal stimulus signals to improve detector-free matchers’ generalization to unseen real-world tasks, achieving universal performance across more than eight tasks with one network weight.

  • Motivation: Cross-modality image matching finds corresponding pixel locations across imaging principles, supporting registration and applications in medical analysis, histopathology, remote sensing, and autonomous systems.Estimated matches help recover image transformations and combine complementary information from modalities such as MRI, CT, PET, and SPECT.
  • Challenge: Approximately 250 hours were required for 9 experts to annotate matches for 481 histopathology image pairs, making human labeling impractical for large datasets.The paper motivates computer-vision methods as an alternative to labor-intensive manual correspondence annotation.
  • Challenge: Substantial appearance changes from differing imaging principles make cross-modality matching difficult, while detector-based methods struggle to identify reliable cross-image keypoints.Detector-free Transformer methods are more robust in low-textured scenes and under large perspective changes, but existing models trained mainly on single-modality data have limited generalization.
  • Approach: The proposed large-scale pre-training framework uses cross-modal stimulus signals to teach appearance-insensitive, fundamental image structures and enhance generalization to unseen cross-modal tasks.The framework addresses the lack of annotated cross-modal training data and can be applied to detector-free matching methods without modifications.
  • Results: More than eight unseen real-world cross-modal tasks were handled with a single network weight and without further training by models pre-trained with the framework.Experiments on nine datasets assessed models based on ROMA and ELoFTR across disciplines including medical image analysis, histopathology, remote sensing, and autonomous systems.

2. Results

Across nine datasets spanning more than eight cross-modality registration tasks, the proposed pre-training framework consistently improves image matching across medical, histological, retinal, thermal-visible, and satellite-view settings. The trained models generalize to unseen modalities and datasets, often outperforming specialized or existing baseline methods.

  • Experimental scope: Experiments cover nine datasets and more than eight cross-modality registration tasks across medical imaging, histopathology, remote sensing, UAV positioning, and autonomous driving.The comparisons include semi-dense ELoFTR and dense ROMA against state-of-the-art matching and registration methods.
  • Multi-Modality Tomography Image Registration: The proposed models substantially outperform all baselines on cross-modality tomography registration despite never being exposed to medical images.Severe appearance and structural differences, particularly between SPECT and MR, cause existing matchers to produce inaccurate matches and erroneous registration or fusion.
  • Registration of Histology Images with Different Stains: The framework generalizes to previously unseen histology tissue slices and stains, outperforming optimization-based methods and the histology-specific DeepHistReg model.The improvement is especially pronounced on the average metric.
  • Retina Image Registration: ROMA achieves the best accuracy on the AUC-Easy metric for visible-light retina matching, surpassing SuperRetina despite the framework’s cross-modality focus.The trained models retain comparable performance on visible-light matching tasks.
  • Thermal and Visible Light Image Registration: 16.8% and 11.6% relative improvements are achieved by ROMA and ELoFTR, respectively, on the SR@10° metric in outdoor street scenes.Across thermal-visible datasets, the framework consistently outperforms all baseline methods, while SuperFusion generalizes poorly to out-of-distribution data.
  • Thermal and Visible Light Image Registration: 93.3% and 72.5% success rates are achieved by ROMA and ELoFTR, respectively, on the unseen Visible-SAR satellite-view dataset.The models show substantial improvements despite SAR modality and satellite viewpoints being completely unseen during training.

3. Discussion

The paper presents a large-scale pre-training framework for universal cross-modality matching, addressing the limited generalizability of existing models. Its main limitation is poor performance on aerial-to-ground matching because training data do not capture the extreme perspective and appearance differences.

  • Contribution: The framework enables state-of-the-art detector-free matchers to achieve universal cross-modality matching for multimodality image registration.Cross-modality matching supports applications in medical imaging, histopathology, remote sensing, and autonomous systems.
  • Limitation: The models perform poorly on aerial-view-to-ground-view matching because the framework cannot effectively mimic their drastic perspective changes and appearance differences.The authors attribute this limitation to a lack of relevant training data and suggest future fine-tuning on specific cross-modality tasks.

4. Methods · 4.1. Preliminaries about Detector-Free Matchers · 4.2. Multi-Resources Data Mixture Training

The framework combines detector-free matcher architectures with multi-resource, cross-modality pre-training to learn generalizable correspondences. It mixes geometrically grounded, video-derived, and synthetically warped data, using complementary strengths to support training at scale.

  • 4. Methods: The method targets accurate correspondences between images from different unseen modalities through large-scale, multi-resource, cross-modality pre-training.The framework integrates multi-data-resource and multi-modal stimulus signals.
  • 4.1. Preliminaries about Detector-Free Matchers: The framework supports detector-free matchers without architectural modifications and uses ROMA as a dense base model and ELoFTR as a semi-dense base model.ROMA emphasizes robustness, while ELoFTR balances efficiency and effectiveness.
  • 4.1.1 Dense Matcher: ROMA: ROMA predicts dense warping fields using transformer-based decoding over coarse and fine features, making it suited to challenging appearance and perspective changes.Its expansive encoder and decoder parameter spaces support large-scale pre-training and generalization to unseen cross-modality tasks.
  • 4.1.2 Semi-Dense Matcher: ELoFTR: ELoFTR uses coarse-to-fine transformer matching with global self- and cross-attention, offering substantially better efficiency despite underperforming ROMA.Its lighter architecture suits applications where speed and computational-resource efficiency are critical.
  • 4.2. Multi-Resources Data Mixture Training: The data-mixture strategy combines multi-view geometry, video sequences, and warped single-image datasets to address costly reconstruction requirements and increase training diversity.These resources provide complementary combinations of realistic viewpoint changes, ground-truth availability, and diversity.
  • 4.2.1 Multi-View Images with Geometry: 1079 scenes from MegaDepth, ScanNet++, and BlendedMVS provide geometrically grounded training pairs spanning indoor and outdoor environments.Depth warping lifts left-image points into 3D and projects them into the right view; matches are retained when ed < 0.05 and ec < 3 pixels.
  • 4.2.2 Video Sequences: The video pipeline builds long-range pseudo ground-truth matches by merging adjacent-frame correspondences into trajectories and refining them with a transformer-based multi-view method.Training pairs use frames more than 10 apart with at least 300 co-visible correspondences; DL3DV contributes 10K high-quality video sequences.
  • 4.2.3 Image Warping: Single-image homography warping supplies diverse synthetic pairs, but mixture training with geometry and video data is needed because planar transformations poorly mimic real-world perspective changes.The transformation samples rotation, translation, scaling, and shearing, with matches obtained by warping dense input-image pixel locations.

4.3. Cross-Modality Stimulus Data Generation · 4.4. Training Details · 4.5. Related Works

The framework uses synthetic cross-modality stimuli and multi-resource training to learn structural correspondences that transfer to unseen tasks. It combines broad pretraining with universal matching models to address limited cross-modality data and generalization challenges.

  • 4.3. Cross-Modality Stimulus Data Generation: Synthetic cross-modality pairs preserve input structure while introducing substantial appearance changes, training networks to match fundamental structures across modalities.Pixel-aligned image generation models transform training images into other modalities, and the resulting pairs support transfer to real-world cross-modality tasks.
  • 4.3. Cross-Modality Stimulus Data Generation: Depth maps add larger structural changes than style translation, addressing modality differences that cannot be modeled by appearance variation alone.The method introduces visible-light and depth-map pairs because textured regions may disappear in thermal imagery despite corresponding scene content.
  • 4.4. Training Details: The training mixture generates ∼800M image pairs spanning visible-visible, synthetic thermal, synthetic nighttime, and depth-map combinations.ROMA and ELoFTR are trained with their official implementations, identical hyperparameters, and identical loss functions for fair comparison.
  • 4.4. Training Details: A single pre-trained model weight is used for all experiments, demonstrating the framework’s strong generalizability across tasks.Training uses 16 NVIDIA A100-80G GPUs with batch size 64.
  • 4.5. Related Works: The proposed universal cross-modality matchers provide accurate correspondences for transformation estimation across varied registration tasks.The approach targets planar, 3D, and nonrigid registration pipelines that support image fusion, calibration, and localization.
  • 4.5. Related Works: Existing cross-modality registration methods often have limited task coverage, accuracy, perspective robustness, or modality generalization because of restricted training diversity and displacement assumptions.Examples include task-specific deformation regression, optical-flow-based methods, and XoFTR trained on synthetic thermal data.
  • 4.5. Related Works: Detector-free and dense matching methods improve robustness to large perspective changes, but dense architectures are generally slower than sparse and semi-dense alternatives.These methods directly produce semi-dense or dense correspondences rather than relying on detected keypoints.
  • 4.5. Related Works: The framework addresses the cross-modality data bottleneck by combining multi-view images, single-image warping, video sequences, and coarse-to-fine correspondence generation.This design harnesses complementary training sources while obtaining long-range, accurate matches from unlabelled videos.

4.6. Experimental Details

The evaluation spans multiple cross-modality datasets, using transformation-based registration metrics tailored to each task. Experiments average results over five random seeds and compare against handcrafted, optimization-based, and learning-based baselines.

  • Evaluation protocol: Experiments were conducted five times with different random seeds, reporting average results and standard-deviation error bars.
  • Medical datasets: The Liver CT-MR evaluation contains 555 warped image pairs, with affine registration assessed by success rate below an n-pixel mean warping-error threshold.Four source-image corners serve as control points for the warping-error calculation.
  • Medical datasets: The Harvard Brain dataset contains 810 warped CT-MR, PET-MR, and SPECT-MR pairs evaluated using the same success-rate metric as Liver CT-MR.
  • Histology and retina datasets: The ANHIR histology dataset contains 251 expert-checked test pairs across stains, evaluated with average and median relative Target Registration Error metrics.The dataset covers tissue sections from multiple organs and conditions, including lesions, lungs, mammary glands, COAD, kidneys, and breasts.
  • Baselines: Comparisons include SIFT, SRIF, MCNet, ROMA, DKM, GIM, SuperFusion, ELoFTR, MatchFormer, AspanFormer, and SP+LG, with SuperRetina used for retina registration.Learning-based matching methods use their outdoor models, while SuperFusion and MCNet use task-specific training weights.

4.7. Ablation Studies

Ablation studies show that cross-modality signals, joint multi-resource training, and coarse-to-fine trajectory construction are important design choices in the pre-training framework. Removing or replacing these components reduces performance across cross-modality evaluation tasks.

  • Cross-Modality Activation Signals: Removing cross-modality training data significantly decreases performance across all cross-modality evaluation datasets, reducing generalization to unseen modalities.The ablation uses only visible image pairs for training.
  • Cross-Modality Activation Signals: 9.7% drop of SR@10◦ metric occurs on real-world thermal-visible registration when synthetic thermal data are removed.Other cross-modality tasks also consistently decline, indicating synthetic thermal data improve real-world thermal generalizability.
  • Multi-Resources Dataset Mixture Training: Multi-stage training underperforms the proposed multi-resources dataset mixture strategy, even with carefully tuned learning rates.The multi-stage variant sequentially trains on single-image, multiview-geometry, and video datasets.
  • Coarse-to-Fine Video Dataset Ground Truth Generation: 7.1% improvement on SR@5 pixels is achieved on the visible-SAR dataset by the proposed coarse-to-fine trajectory construction strategy.The comparison replaces coarse-to-fine construction with merging matches separated by less than 1 pixel.

Supplementary Material · A. Details about training data generation · A.1. Multi-view images with geometry datasets

The training data generation uses multi-view datasets spanning outdoor, indoor, and mixed indoor–outdoor scenes. MegaDepth training excludes reported low-quality scenes and scenes overlapping with the IMC test set, while ScanNet++ and BlendedMVS pairs are constructed using geometry-based overlap checks.

  • A.1. Multi-view images with geometry datasets: MegaDepth provides outdoor imagery from 196 different scenes.Training pairs are sampled according to LoFTR’s procedure.
  • A.1. Multi-view images with geometry datasets: MegaDepth training pairs are sampled by LoFTR.
  • A.1. Multi-view images with geometry datasets: 16 reported low-quality MegaDepth scenes are removed from training.The passage lists the excluded scene identifiers explicitly.
  • A.1. Multi-view images with geometry datasets: 8 MegaDepth scenes overlapping with the IMC test set are removed from training.The excluded identifiers are ‘0024’, ‘0021’, ‘0025’, ‘1589’, ‘0019’, ‘0008’, ‘0032’, and ‘0063’.
  • A.1. Multi-view images with geometry datasets: ScanNet++ contains 380 indoor scenes, while BlendedMVS contains 502 indoor and outdoor scenes.
  • A.1. Multi-view images with geometry datasets: Training pairs for ScanNet++ and BlendedMVS are constructed using an image overlap ratio r.Pixels are warped using depth maps and camera parameters, then checked for depth consistency.
  • A.1. Multi-view images with geometry datasets: The overlap ratio r is defined as consistent warped pixels divided by the number of pixels in the left image.

A.2. Video sequences dataset

The dataset constructs training image pairs with ground-truth matches from unlabeled DL3DV video sequences. It combines trajectory-based matching and refinement with strict pair-selection criteria for reliable cross-frame supervision.

  • Source data and matching: 10K high-quality DL3DV video sequences from indoor and outdoor scenes provide the unlabeled source data.Sequences are downsampled every 4 frames, and each image is matched with its consecutive 10 images.
  • Trajectory construction and refinement: Coarse point trajectories merge matches using non-maximum suppression with a 7 × 7 sliding window, then undergo off-the-shelf multi-view refinement.The refinement step targets high-accuracy trajectories before training-pair construction.
  • Training-pair selection: 20 frames, 300 co-visible matches, and 30 pixels define the thresholds for selecting training pairs with sufficient separation, overlap, and relative motion.Processing all video sequences uses 16 A100-80GB GPUs and takes about 72 hours.

A.3. Single image datasets

The framework trains on GoogleLandmark and SA-1B single-image datasets by synthesizing perspective-varied views through random homography transformations. To stabilize cross-modality training, supervision is restricted to regions with positive estimated depth, excluding unreliable sky areas.

  • Synthetic view generation: GoogleLandmark and SA-1B provide large-scale single-image training data, with random homographies synthesizing views under perspective changes.The homography sampling combines random rotation, translation, scale, and nonisotropic skew factors.
  • Depth-based supervision: Supervision is restricted to regions whose DepthAnything-estimated depth exceeds zero, preventing unreliable sky matches from destabilizing cross-modality training.Outdoor sky regions can receive incorrect ground-truth matches because depth estimation is unreliable there.

B. Experimental details

Experiments used resized image pairs with a longest edge of 840 pixels to evaluate the trained models against established matching and registration baselines. Synthetic transformations varied by dataset, with specified rotation, translation, and scale ranges.

  • Evaluation setup: Image pairs were resized so their longest edge equaled 840 pixels for all trained models and listed baselines.Baselines included ROMA, DKM, GIM, ELoFTR, MatchFormer, AspanFormer, SuperPoint+LightGlue, and SIFT.
  • Evaluation setup: GIM used its best-performing large-scale model trained with DKM.
  • Transformation settings: For the visible-vectorized map and Harvard Brain datasets, transformations sampled rotation from [−10°, 10°], translation factor from [−0.1, 0.1], and scale factor from [0.8, 1.25].
Loading 2501.07556v1…