Source-linked AI summary
CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
Paul Schneider, Nazim Haouchine
TL;DR
Multimodal matching remains difficult because modality-specific sensing changes appearance, while existing solutions require modality-pair retraining or large models. CrossFeat learns a descriptor-space crossing function that alters appearance while preserving geometry, and experiments show improved performance across multimodal tasks with greater efficiency than dense matchers.
Problem
Existing cross-modal approaches require retraining descriptors for each modality pair or using large models with increased runtime, leaving test-time adaptation of pretrained descriptors insufficiently addressed.
Method
CrossFeat learns a non-linear descriptor-space operator with geometry-appearance disentanglement, adapting appearance while preserving the original descriptor’s geometric structure.
Results
CrossFeat consistently improves multimodal matching across descriptors, modality pairs, and tasks, while achieving approximately 100× fewer parameters and 5–10× faster inference than dense matchers.
Takeaways & Limitations
Descriptor-level modality bridging offers an effective, simple, and efficient alternative to modality-specific descriptors or entirely new matching architectures.
Takeaways & Limitations
Cross-modal crossing must preserve geometric content even when structural cues are attenuated or absent in some modalities.
Abstract
from arXiv · showhide
Most advances in keypoint descriptions address monomodal settings, where image variations arise from viewpoint, illumination, or contrast changes. Multimodal scenarios involve images produced by fundamentally different sensing processes, such as multispectral imaging, RGB-depth, satellite imagery, or medical imaging, causing the same structures to appear differently. A common solution to cross-modal description is to train descriptors for each modality pair, which requires retraining whenever the modalities change, or to train large models, which incur a significant increase in runtime. Instead, we propose CrossFeat, a framework that enables an existing monomodal descriptor to operate across modalities. Our method learns a crossing function in descriptor space that maps features from one modality to a representation compatible with another. To preserve the structural information captured by the original descriptor, CrossFeat introduces a geometry-appearance disentanglement such that only appearance is altered while the geometric properties are preserved. Experiments across multiple domains and datasets demonstrate improved performance in multimodal matching.
1 Introduction
CrossFeat adapts existing monomodal descriptors for multimodal matching by translating descriptor representations while preserving their geometric structure. Evaluations across medical, urban, and aerial domains demonstrate improved multimodal matching performance.
- CrossFeat learns a non-linear operator that translates a chosen descriptor across at least two modalities without retraining the original descriptor.
- The framework applies to any chosen descriptor and different modality pairs by training the crossing function for the specified descriptor and modalities.
- Its disentanglement mechanism separates appearance from geometry, changing only modality-specific appearance while retaining geometric properties.
- Evaluations span medical, urban, and aerial domains across multiple datasets.
- CrossFeat improves performance over state-of-the-art multimodal matching methods.
2 Related Work
Related work improves matching through descriptor adaptation, local-frame normalization, inference-time augmentation, and adaptive matching. However, adapting a previously trained descriptor to modality-specific appearance differences remains an open gap.
- Cross-spectral matching addresses modality gaps from different sensing physics, including thermal–visible, depth–RGB, and MRI–ultrasound imaging.
- Learned detector, descriptor, and matcher pipelines include context-aware assignment, detector-free matching, and modality-conditioned coarse-to-fine reasoning.
- Large-scale synthetic or cross-modal pretraining seeks scalability and generalization across modality pairs, including unseen combinations.
- Descriptor-space transforms, patch-frame normalization, inference-time augmentation, and adaptive matchers improve robustness without always redesigning the full pipeline.
- A remaining gap is adapting a previously trained descriptor at test time to appearance differences caused by the target modality.
3 Method
CrossFeat learns a descriptor-space crossing function that maps source descriptors into a target modality space while preserving geometric information. It explicitly separates geometry from appearance, transforms only appearance, and trains the components with reconstruction, crossing, alignment, adversarial, and contrastive objectives.
- Problem formulation: CrossFeat maps a source descriptor into a target feature space so standard nearest-neighbor matching can produce cross-modal correspondences without retraining the original descriptor.The crossing function is conditioned on source and target modalities.
- Geometry–appearance disentanglement: The architecture decomposes each descriptor into geometry and appearance codes, preserves geometry, and applies the crosser exclusively to appearance.The decoder recombines the source geometry with crossed appearance to reconstruct a target-space descriptor.
- Encoder: A shared encoder uses separate projection heads for geometry and appearance, with a variational appearance code and deterministic geometry code.At inference, the appearance code uses its posterior mean, while geometry bypasses the crosser unchanged.
- Crosser and decoder: The modality-conditioned crosser combines FiLM modulation with a residual MLP, initialized near identity and connected through a decoder conditioned on the target modality.The decoder concatenates the two codes, processes them with an MLP, and ℓ2-normalizes the output onto the descriptor sphere.
- Training objective: Training combines descriptor reconstruction, target-descriptor crossing, geometry alignment, modality-adversarial learning, and InfoNCE hard-negative discrimination.Geometry alignment moves modality-specific information into appearance, while contrastive learning helps the correct target rank above alternatives.
- Disentanglement analysis: Sensitivity analysis shows geometry changes more under rotation, whereas appearance changes more across modality transitions, supporting the intended disentanglement.Cross-modality correspondences are obtained after crossing appearance while preserving geometry.
4 Experiments
CrossFeat is evaluated across medical, driving, and satellite multimodal tasks using sparse and dense matching metrics, descriptor comparisons, robustness tests, discriminability analyses, and ablations. Results show consistent matching improvements, preserved geometric robustness, and strong dependence on key design components and sampling strategy.
- Datasets, Metrics and Training: Evaluation spans medical imaging, autonomous driving, and satellite imagery, with Success Rate and AUC for dense matching plus Precision, Recall, and match count for sparse methods.The medical task includes ceT1, T2, FLAIR, and ultrasound modalities; driving and satellite tasks cover additional modality pairs.
- Comparison with state-of-the-art methods: CrossFeat improves sparse matching across domains, achieving the best SR and AUC on medical and driving tasks and non-zero success rates on the satellite task.On satellite imagery, most baselines fail to produce reliable matches, while CrossFeat performs best without TTA.
- Comparison with state-of-the-art methods: CrossFeat primarily improves match reliability rather than density, often producing fewer but more geometrically consistent correspondences and consequently strong SR@1 and AUC@1.TTA can increase recall but may reduce precision when the initial inlier set is small or noisy, particularly on satellite data.
- Comparison with state-of-the-art methods: CrossFeat substantially outperforms dense baselines on driving and performs strongly on medical matching, while dense methods struggle on satellite correspondences.On medical data, CrossFeat with TTA achieves the highest AUC across thresholds and best SR@1, although another method has the highest SR at larger thresholds.
- Comparison with state-of-the-art methods: CrossFeat requires approximately 0.5–1 s per image, compared with roughly 5 s for dense matching methods.The reported runtime includes detection, description, crossing, and matching.
- CrossFeat improves descriptor performances: CrossFeat consistently improves SIFT, SuperPoint, ALIKED, and DISK across medical modality pairs, with particularly pronounced gains for SIFT.The results indicate that both classical and learned descriptors benefit from adapting to multimodal appearance changes.
- Robustness and discriminability: CrossFeat preserves geometric properties under increasing shear, with ∆TRE increasing more slowly than without crossing and matching descriptors showing greater similarity separation.Crossing shifts true-correspondence similarities higher while leaving non-matching pairs largely unchanged, increasing d′.
- Ablation study: Removing geometry-appearance disentanglement causes the largest ablation drop at −13.3 pp SR@1, while removing contrastive loss collapses SR@1 from 52.9% to 15.1%.The adversarial discriminator, FiLM-conditioned decoder, reconstruction, and latent regularization also contribute, while the variational bottleneck has a moderate effect.
5 Discussion and Conclusion
CrossFeat improves multimodal correspondence by adapting descriptor appearance while preserving geometric structure. It consistently outperforms sparse pipelines, competes with dense matchers, and does so with substantially lower computational cost.
- Discussion and Conclusion: CrossFeat adapts descriptor appearance while preserving the geometric structure encoded by the original representation.This behavior is observed across different descriptors and modality pairs, with robustness maintained under distortions.
- Discussion and Conclusion: CrossFeat consistently outperforms sparse matching pipelines across tasks, especially for challenging modality gaps where baselines often fail to produce reliable correspondences.
- Discussion and Conclusion: CrossFeat is competitive with, and sometimes superior to, recent dense matching approaches despite using the classical detector–descriptor–matching pipeline.
- Discussion and Conclusion: Approximately 100× fewer parameters and 5–10× faster inference make CrossFeat substantially more efficient than dense matchers.Its efficiency comes from sparse descriptors and a lightweight crossing network rather than large correlation-based architectures.
- Discussion and Conclusion: Descriptor-level modality bridging offers a simple and efficient alternative to modality-specific descriptors or entirely new matching architectures.