XoFTR: Cross-modal Feature Matching Transformer

1. Introduction

• Visible–TIR matching remains sub-optimal across viewpoints, scales, and poor textures, while modality differences include nonlinear intensity and texture variation and typically lower TIR resolution and field of view.TIR imagery is robust to adverse lighting and weather, but thermal radiation differs from the reflected light captured in visible images.Source: §1 · p. 1
• XoFTR adapts LoFTR [69] through masked image modeling pre-training and fine-tuning with pseudo-TIR augmentation to address modality differences.The augmentation extends the cosine transform.Source: §1 · pp. 1–2
• At 1/8 resolution, coarse matching allows one-to-one and one-to-many assignments; a customized decoder upsamples matches to 1/2 scale before rematching and sub-pixel regression.Rematching filters low-confidence matches, while regression prevents one point from matching multiple points in the other image.Source: §1 · p. 2
• XoFTR outperforms strong baselines and achieves state-of-the-art visible-thermal matching results, with Figure 1 showing improvements over LoFTR [69].The new dataset covers viewpoint differences and sunny and cloudy weather conditions.Source: Figure 1 · p. 1, §1 · p. 2
• The introduced dataset supports evaluation across viewpoint differences and sunny and cloudy conditions, addressing the lack of a suitable urban visible-thermal benchmark.The paper also evaluates homography estimation on public datasets.Source: §1 · p. 2

Local Feature Matching

• Detector-free Transformer matchers such as LoFTR [69], AdaMatcher [38], and PMatch [89] address low-texture, scale, or viewpoint challenges in visible imagery, but their application to visible-thermal pairs is limited.Multimodal alternatives include handcrafted and learning-based methods, as well as ReDFeat [18], Shape-Former, and MIVI.Source: §2 · p. 2

Unsupervised Pre-training in Vision

• XoFTR adapts PMatch’s paired pre-training [89] to LoFTR for visible-to-thermal applications, whereas prior multimodal masked-image pre-training has been sparse and recent.Earlier approaches include MultiMAE, Complementary Random Masking, and a multimodal transformer [16].Source: §2 · p. 3

RGB - Thermal Image Conversion

• Handcrafted cosine transform is computationally light for RGB-to-thermal conversion but produces images lacking true thermal characteristics; the paper prefers it for reliability over generative networks that can produce artifacts.The cited generative methods include GAN-based translation [21] [85].Source: §2 · p. 3

3. Methods

• XoFTR extracts multiscale features, matches them at coarse and fine levels, then refines fine matches to sub-pixel locations.Its CNN backbone uses 1/8, 1/4, and 1/2 scales.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Coarse-Level Matching Module

• At 1/8 scale, coarse matching uses one-to-one, one-to-many, and many-to-one assignments to address feature inconsistency from scale or viewpoint variation.The approach is intended to avoid manual methods such as image cropping.Source: §3.1 · p. 3

3.2. Fine-Level Matching Module

• At 1/2 scale, fine matching re-matches coarse predictions and filters low-confidence candidates to favor texturally reliable matches.A customized decoder supports both pre-training and fine-tuning and carries LoFTR-processed information to upper layers.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Sub-pixel Refinement Module

• Sub-pixel refinement jointly regresses local coordinates in both images and adds them to fine-level match locations, preventing either point from matching multiple points.The regression uses concatenated feature vectors and an MLP with a Tanh function.Source: §3.3 · p. 5

3.4. Masked Image Modeling

• Masked image modeling pre-trains on real visible-thermal pairs by reconstructing randomly masked inputs, then transfers the encoder and decoder layers to fine-tuning.The method masks 50% of each image in 64×64 patches and uses mean squared error between target and reconstructed images.Source: §3.4 · p. 5

3.5. Data Augmentation

• During fine-tuning, randomized cosine-transform augmentation converts visible images into pseudo-thermal examples to address the lack of urban visible-thermal matching data.The method also applies random 5×5 Gaussian blur; the authors say this augmentation gives the network endurance against nonlinear intensity variation.Source: §3.5 · p. 5

3.6. Supervision

• Training combines coarse-level, fine-level, and sub-pixel losses, with coarse ground-truth matches projected between images using depth maps and camera poses.Fine-level ground truth adds a mutual-nearest-neighbor constraint for one-to-one matches, while the sub-pixel loss uses symmetric epipolar distance and jointly supervises coordinates without explicit ground-truth pairs.Source: §3.6 · pp. 5–6

• METU-VisTIR covers six diverse scenes with ground-truth camera poses, including extreme viewpoint and weather-induced lighting and temperature changes.Source: §4 · p. 6
• The dataset uses closely positioned thermal and RGB cameras; pose ground truth comes from RGB-based COLMAP and HLOC, with same-modality calibration for both cameras.The thermal camera is 640 × 512 pixels with a 61° field of view; the RGB camera is 3840 × 2160 pixels with an 84° field of view.Source: §4 · p. 6
• Two benchmark sets contain 1,382 cloudy-cloudy pairs and 1,208 cloudy-sunny pairs, each with ground-truth camera poses.Source: §4 · p. 6

5.1. Implementation Details

• Implementation pre-training uses 95,000 KAIST visible-thermal pairs, followed by fine-tuning on MegaDepth at 640 × 640 resolution.Pre-training takes 9 epochs; fine-tuning converges after 24 hours.Source: §5.1 · p. 7

5.2. Experiment 1: Relative Pose Estimation

• Relative pose is evaluated with AUC at 5°, 10°, and 20° thresholds, using RANSAC to estimate the essential matrix from predicted matches.Cloudy-cloudy and cloudy-sunny sets are evaluated separately.Source: §5.2 · p. 7
• XoFTR outperforms other methods by a large margin in relative pose estimation, though performance decreases with varying weather conditions.Source: Table 3 · p. 7, §5.2 · pp. 7–8

5.3. Experiment 2: Homography Estimation

• XoFTR surpasses other baselines at 10- and 20-pixel homography error thresholds, while ASpanFormer [14] performs best at 5 pixels.The comparison uses LGHD LWIR/RGB [2] and FusionDN [82] datasets.Source: Table 4 · p. 8, §5.3 · p. 8

5.4. Experiment 3: Ablation Study

• Ablations on METU-VisTIR cloudy-sunny show AUC drops without pre-training, augmentation, one-to-many assignment, fine-level modules, sub-pixel refinement, confidence thresholding, or absolute positional bias.Source: Table 5 · p. 8, §5.4 · p. 8
• XoFTR runs in 116 ms at 640 × 512 resolution on an A5000 GPU, versus 102 ms for LoFTR [69].The passage attributes the small runtime increase to one-to-many assignment increasing the number of coarse matches.Source: §5.4 · p. 8

6. Conclusion

• XoFTR is a novel pipeline for cross-view visible-thermal image matching.Source: §6 · p. 8
• XoFTR’s two-stage approach significantly outperforms compared methods on several benchmarks.Source: §6 · p. 8
• The paper introduces a novel challenging dataset for evaluating methods.Source: §6 · p. 8