PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Spectral Dense Correspondence for Multimodal Spectral Medical Imaging

Eric L. Wisotzky, Jost Triller, Simon W. Härtl, Oliver T. Bruns, Peter Eisert, Anna Hilsmann

arXiv:2608.28341v1cs.CVcs.AI

TL;DR

Cross-spectral dense correspondence lacks realistic supervision because non-overlapping spectral sensitivities create radiometric changes and real medical ground truth is difficult to obtain. The paper addresses this with sensor-agnostic intensity projection, spectral-response modulation, and a synthetic benchmark. The resulting training strategy improves robustness across architectures under spectral mismatch while preserving competitive RGB performance and supports spatially coherent spectral fusion in heterogeneous systems.

  • Problem

    Cross-spectral correspondence is difficult because non-overlapping sensitivities produce contrast changes and intensity inversions, while dense ground truth for real surgical HSI is difficult to obtain.

  • Method

    The paper generates cross-spectral benchmark variants and a synthetic benchmark, using sensor-agnostic single-channel inputs and view-specific spectral-response modulation during training.

  • Results

    The training strategy substantially improves robustness across modern correspondence architectures under cross-spectral inputs while preserving competitive performance on RGB benchmarks.

  • Takeaways & Limitations

    Realistic cross-spectral data generation provides a practical foundation for training and evaluating robust correspondence before deployment on clinical spectral data.

  • Takeaways & Limitations

    Single-channel input cannot jointly exploit complementary structures distributed across multiple wavelengths, trading spectral completeness for sensor-independent plug-and-play operation.

Abstract

from arXiv · show

Precise dense correspondence is a fundamental prerequisite for multimodal spectral imaging systems that fuse disparate wavelength ranges for subsequent analysis in medical and scientific imaging. Corresponding image points are often observed with non-overlapping spectral sensitivities, leading to wavelength-dependent contrast changes, intensity inversions, and appearance shifts for which dense ground truth is difficult to obtain and conventional RGB-based training data provides only limited supervision. We address this data gap by introducing a sensor-agnostic cross-spectral modulation protocol on established correspondence benchmarks with intensity input projection, and by proposing a synthetic cross-spectral correspondence benchmark simulating physically plausible radiometric differences. Evaluation on several modern dense correspondence backbones trained with our unified cross-spectral protocol showed substantial improvements under severe spectral mismatch while maintaining performance on standard RGB benchmarks. Ablation experiments show that view-dependent channel selection and nonlinear radiometric transformations provide complementary robustness, indicating that the primary limitation of existing models is not their structural matching capacity but the mismatch between training distribution and spectral characteristics of the target image pair. Qualitative evaluations on heterogeneous medical spectral acquisition systems demonstrate the practical relevance of the proposed training data augmentation protocol as an enabler for spatially coherent spectral fusion in HSI workflows.

1 Introduction

Cross-spectral correspondence is difficult because wavelength-dependent radiometric changes violate photometric assumptions and real surgical ground truth is scarce. The paper addresses this gap with sensor-agnostic spectral modulation, synthetic evaluation data, and architecture-spanning validation.

  • Motivation: Accurate dense alignment is required before spectral measurements from heterogeneous views and channels can be fused into coherent representations.The motivation includes spatially resolved tissue assessment, spectral fusion, and possible depth recovery.
  • Motivation: Non-overlapping spectral sensitivities cause contrast changes and intensity inversions that undermine conventional dense correspondence assumptions.These effects make corresponding points photometrically inconsistent across views.
  • Data gap: Real surgical HSI lacks reliable dense ground truth because tissue motion, specularities, moist surfaces, limited texture, and clinical constraints complicate annotation and calibration.This data scarcity limits realistic training and evaluation.
  • Contributions: The framework generates cross-spectral variants of established RGB benchmarks while preserving dense ground truth and adds a synthetic benchmark with controlled wavelength-pair variation.The synthetic benchmark supports systematic evaluation under non-overlapping spectral sensitivities.
  • Contributions: A sensor-agnostic input representation and spectral-response modulation expose existing models to wavelength-dependent radiometric variation and contrast inversions.The approach targets generalization across heterogeneous multi- and hyperspectral systems rather than a task-specific architecture.
  • Results: Across several modern correspondence architectures, the training strategy improves cross-spectral robustness while maintaining performance on standard RGB benchmarks and supports heterogeneous medical acquisition settings.The work demonstrates applicability for spatially coherent spectral fusion in stereo-HSI and HSI light-field workflows.

2 Related Work

Related work establishes the value of spectral medical imaging and modern correspondence models, while emphasizing limited clinical annotation and the need for data-centric robustness to spectral mismatch.

  • Spectral medical imaging: Medical MSI and HSI provide spatially resolved spectral signatures relevant to tissue characterization, perfusion assessment, surgical understanding, and spectral reconstruction.These modalities extend conventional RGB imaging with material and physiological information.
  • Data limitations: Annotated clinical data and geometric ground truth are difficult to obtain in vivo, motivating simulations, phantoms, structured-light acquisition, and controlled setups.Prior surgical HSI augmentation addressed geometric domain shifts without changing network architecture.
  • Data-centric methods: Cross-spectral correspondence requires augmentations that preserve geometry while changing the radiometric relationship between paired views.The paper adopts this data-centric perspective through spectral-response augmentation and synthetic evaluation data.
  • Surgical correspondence: Dense correspondence supports surgical 3D perception, augmented reality, and image-guided intervention, but surgery introduces weak texture, reflections, deformation, motion, and workflow constraints.These factors complicate acquisition and quantitative evaluation.
  • Dense correspondence: Modern correspondence architectures perform strongly on RGB benchmarks, but cross-spectral matching violates brightness constancy and can suppress local contrast.This motivates methods explicitly designed for spectral appearance variation.

3 Modality-Robust Dense Correspondence

The proposed modality-robust framework estimates geometry-consistent 2D displacement while separating spatial structure from wavelength-dependent appearance. It combines sensor-agnostic intensity inputs with structured radiometric modulation during training.

  • Problem formulation: Cross-spectral correspondence is formulated as pixel-wise 2D displacement estimation between views with different viewpoints and non-overlapping spectral sensitivities.The formulation generalizes beyond specific modality pairs and rectified geometries.
  • Spectral heterogeneity: Heterogeneous systems vary in band count, spectral coverage, and radiometric response, often without shared channels, making naive similarity measures ambiguous.Local contrast may be suppressed, distorted, or inverted across views.
  • Framework design: The framework uses a geometry-agnostic displacement formulation and sensor-agnostic inputs to account explicitly for modality-induced appearance variation.Its design separates structural alignment from fixed spectral channel semantics.
  • Backbone: Paired images are processed with feature extraction, correlation reasoning, and iterative refinement to predict one displacement vector per reference-view pixel.The strategy is realized within dense correspondence architectures.
  • Input representation: Inputs are normalized into a single intensity channel through intensity projection or individual-band selection, removing fixed wavelength semantics while preserving spatial layout.This representation is designed for heterogeneous MSI and HSI systems.
  • Radiometric modulation: One view receives a deterministic, sample-level nonlinear intensity mapping that simulates sensor-dependent spectral responses rather than pixel-wise photometric noise.The protocol explicitly models modality-induced variation during training.
  • Data generation: RGB benchmark pairs are converted using either non-overlapping channel pairing or grayscale conversion with spectral-response modulation.The two strategies create disjoint-band or transformed grayscale cross-spectral inputs.
  • Data generation: The transformation family includes identity or inversion, square-root variants, power laws with n ∈{2, 4}, and logarithmic mappings to approximate cross-spectral contrast effects.These mappings can represent contrast suppression and inversion associated with spectral tissue behavior.

4 Experiments

Experiments evaluate the framework across multiple correspondence architectures, benchmark-derived and synthetic data, and heterogeneous medical acquisition settings. Synth isolates geometric displacement from wavelength-dependent appearance variation, while clinical tests cover varied spectral and geometric configurations.

  • Architectures: Several dense correspondence architectures, including RAFT-style, SKFlow, DIP, SEA-RAFT, and GMA, are used to assess generalizability.The models represent recurrent, correlation-based, efficiency-oriented, and attention-augmented designs.
  • Evaluation sources: Training and evaluation use established benchmarks, a controlled synthetic cross-spectral dataset, and real medical data from heterogeneous imaging systems.This combines quantitative benchmark testing with qualitative clinical assessment.
  • Benchmark evaluation: Benchmark-derived spectral-mismatch variants include FlyingChairs, FlyingThings3D, MPI-Sintel, HD1K, KITTI, Middlebury Stereo, ETH3D, and InStereo2K.The listed datasets support training, validation, or evaluation under modified cross-spectral conditions.
  • Synthetic benchmark: Synth provides 768 × 512 triplets with dense 2D displacement ground truth and 100 samples per motion regime and spectral range.It is a controlled benchmark for material-dependent radiometric shifts, not a tissue simulator.
  • Synthetic benchmark: Synth separates difficulty into non-rigid, perspective-parallax, and zero-motion regimes to test geometric displacement and appearance-induced spurious motion independently.Synth-Zero evaluates whether spectral changes alone produce predicted displacement.
  • Clinical evaluation: Clinical evaluation spans rectified MSI stereo, unrectified RGB-SWIR and SWIR-SWIR stereo, and hyperspectral light-field sub-aperture pairs.These settings vary in viewpoint, wavelength separation, rectification, and narrow-band filtering.
  • Clinical evaluation: The VIS-NIR setup uses disjoint ranges of [450, 650] nm and [675, 1000] nm, while RGB-SWIR evaluates full 2D correspondence without rectification under a strong modality gap.The RGB-SWIR images use RGB and SWIR resolutions of 1080×1440 and 1032×1296 pixels.
  • Clinical evaluation: HSI light-field pairs combine viewpoint changes with narrow-band filters of ±10 nm bandwidth across [350, 1000] nm and small or large baseline offsets.Each sub-view has resolution 400×400 pixels.

5 Results

Cross-spectral training preserves performance on conventional RGB benchmarks while substantially improving correspondence under spectral mismatch. Results across benchmarks, Synth regimes, architectures, feature analyses, and real medical setups support data-distribution alignment as the main robustness factor.

  • Benchmark evaluation: Cross-spectral models maintain comparable performance to their baseline architectures on original RGB benchmarks.The reported total mean deviations are 0.032 EPE for MPI-Sintel, 0.022 EPE for Middlebury Stereo, and 0.019 EPE for KITTI.
  • Benchmark evaluation: 11.48% mean deviation separates cross-spectral models from their original counterparts across the evaluated RGB datasets.Architecture-specific changes are mixed: DIP improves by 10.12%, while the smallest deterioration is 4.63% for the RAFT-based model.
  • Cross-spectral evaluation: Cross-spectral models retain RGB-level accuracy on modified cross-spectral datasets, whereas original models perform about one order of magnitude worse.This pattern holds across the evaluated cross-spectral architectures and modified benchmark inputs.
  • Synthetic benchmark: Cross-spectral training consistently lowers error across Synth displacement regimes and suppresses spurious motion in the zero-motion regime.In Synth-Zero, the views differ only spectrally, so predicted displacement directly measures radiometrically induced false motion.
  • Synthetic benchmark: 9.78 average EPE is achieved by cross-spectral variants across architectures and wavelength pairings, while original-model error increases with spectral separation.The cross-spectral variants remain stable across camera pairings, indicating robustness to wavelength-dependent radiometric changes.
  • Ablation and representation analysis: View-dependent channel selection and nonlinear radiometric modulation provide complementary robustness, with the combined configuration achieving the lowest EPE.The former introduces spatially heterogeneous, content-dependent contrast changes; the latter adds nonlinear mappings and contrast inversions.
  • Ablation and representation analysis: Cross-spectral training increases mean Synth cosine feature similarity from 0.386 to 0.604 and maintains r > 0.9 across radiometric variants.For Mixed Monotonic, similarity increases by 0.75 on average, while original models collapse to 0.17 under mixed radiometric changes.
  • Medical applications: Qualitative evaluations across heterogeneous medical setups show coherent cross-spectral displacement fields with fewer artifacts than original models.The evaluated settings include VIS-NIR MSI stereo, unrectified RGB-SWIR stereo, and HSI light-field views; real-world data lack dense ground truth.

6 Conclusion

The paper presents a data-centric, architecture-agnostic framework for generating and using realistic cross-spectral correspondence data. Across modern architectures, this strategy improves robustness under spectral mismatch while preserving competitive RGB performance.

  • The framework generates cross-spectral variants of established benchmarks and the synthetic Synth benchmark with dense ground-truth correspondence.These datasets support controlled training and evaluation under spectral appearance variation.
  • Across several modern correspondence architectures, training substantially improves robustness under cross-spectral inputs while preserving competitive RGB benchmark performance.
  • Synth results indicate that conventional models primarily fail because their learned representations depend on RGB training appearance statistics rather than lacking geometric matching capacity.
  • Ablations identify compatibility between the training distribution and target spectral domain as a dominant factor for correspondence under spectral mismatch.
  • Realistic synthetic data and benchmark construction support developing, comparing, and validating correspondence models before deployment on clinical spectral data.The framework is positioned as a practical foundation for cross-spectral vision and multimodal medical imaging.

A.1 Need for Dense Cross-Spectral Correspondence

RGB-SWIR correspondence methods recover reliable matches mainly in locally distinctive regions, leaving substantial portions of scenes unsupported. This exposes the need for dense correspondence methods robust to cross-spectral appearance differences.

  • Reliable RGB-SWIR matches from DUSt3R and RoMa concentrate in locally distinctive regions, while large scene areas remain without correspondence support.
  • Both methods recover plausible correspondences where intensity distributions behave similarly across views, but their matches remain spatially uneven.

A.2 Synth: Controlled Benchmark for Spectral Mismatch

Synth is a synthetic cross-spectral dataset designed to provide dense ground truth and spectral pairing metadata for controlled evaluation under spectral mismatch. It uses measured reflectance spectra spanning broad wavelength ranges and presents image pairs with displacement fields.

  • Synth contains dense ground-truth displacement and spectral pairing metadata for synthetic cross-spectral correspondence evaluation.
  • The dataset uses measured USGS Spectral Library reflectance spectra from natural, biological, and manmade materials.
  • Its spectra span approximately 0.2−200µm, covering UV, VIS, NIR, SWIR, MWIR, and LWIR ranges.
  • Generated triplets contain a left image IL, a middle image IR, and a right-column displacement field u′.

A.3 Data Augmentation by Spectral-Response Modulation

The augmentation protocol projects image pairs into intensity representations and applies view-dependent channel selection with nonlinear radiometric transformations. These transformations preserve geometry while modeling controlled, realistic violations of brightness constancy.

  • Image pairs are converted to single-channel representations through normalized grayscale conversion or independent random RGB-channel selection for each view.
  • One resulting intensity image receives a structured nonlinear radiometric transformation to emulate sensor- and wavelength-dependent appearance changes.
  • The transformation family includes identity, inversion, square-root, power-law, and logarithmic mappings.Examples include f(x) = x, f(x) = 1 −x, square-root variants, power-law mappings, and logarithmic variants.
  • These transformations preserve geometric structure while varying the radiometric relation between views, keeping the original displacement annotations valid.
  • The functions are compact approximations of dominant radiometric effects rather than camera-specific physical sensor models.Real RGB-SWIR examples include a 1064 nm response resembling a square root of red and a 1370 nm response resembling a fourth-order blue-channel response.

A.4 Rationale and Limitations of the Single-Channel Input Representation

The framework uses a sensor-agnostic single-channel input to avoid sensor-specific channel semantics while preserving original multi-band measurements for later spectral fusion. This representation improves cross-spectral feature consistency and displacement-map coherence, but trades off simultaneous use of complementary spectral information.

  • The single-channel projection avoids sensor-specific channel semantics across systems with different band counts, wavelengths, bandwidths, and spectral ranges.
  • Normalized intensity inputs can aggregate multiple bands or select one band, preserving structures that are pronounced only in a specific wavelength range.
  • The representation cannot jointly exploit complementary structures distributed across multiple wavelengths in a single forward pass.Band aggregation may attenuate narrow-band structures, while individual-band selection limits simultaneous use of complementary spectral information.
  • Adaptive spectral projections or attention could use complementary spectral information more effectively, but would reduce the framework’s sensor-independent, plug-and-play character.These alternatives introduce assumptions about the number, ordering, or spectral characteristics of available bands.
  • The single-channel projection estimates geometry, while original multi-band measurements remain available for alignment and spatially coherent spectral fusion.
  • Cross-spectral models produce modality-invariant features and coherent displacement maps, whereas original models show modality-dependent clusters and artifacts or displacement flips.The feature comparison is illustrated with RGB-SWIR t-SNE embeddings and qualitative medical displacement maps.
Loading 2608.28341v1…