PaperlayerView exact revision on arXiv

Source-linked AI summary

SeeRF: Learning to See Beyond Fields of View via Generative Image Registration and Fusion

Xunpeng Yi, Yibing Zhang, Qinglong Yan, Yifan Lu, Han Xu, Jiayi Ma

arXiv:2610.35637v1cs.CV

TL;DR

Existing multimodal fusion methods struggle to integrate scene information beyond sensors’ shared fields of view, where viewpoint and coverage mismatches leave regions incompletely represented. SeeRF couples generative image registration with diffusion-based completion and fusion; experiments report superior performance and robustness across multiple datasets. The authors also identify increased computational complexity and inference overhead as a remaining limitation.

  • Problem

    Existing fusion methods struggle to integrate complete scene information beyond shared fields of view when multimodal sensors have viewpoint and coverage mismatches.

    Source: §I · p. 1

  • Method

    SeeRF couples stationary-velocity-field-based multimodal registration with generative diffusion for completion and fusion beyond the shared field of view.

    Source: §I · p. 2

  • Results

    SeeRF achieves superior performance and robustness across multiple datasets for fusion beyond the field of view.

    Source: §I · p. 2

  • Takeaways & Limitations

    SeeRF addresses missing information and residual artifacts in non-shared field-of-view regions through registration and generative completion and fusion.

    Source: §I · p. 2, §VI · p. 15

    Visual foundation and diffusion models increase computational complexity and inference overhead, leaving room for further lightweighting and optimization.

    Source: §V · p. 14

Abstract

from arXiv · show

Due to the inherent discrepancies in imaging viewpoints and field-of-view mismatches among multi-modal sensors, existing fusion methods have long struggled to achieve complete scene-level information integration beyond the limitation in sensor fields of view. Consequently, the generated fusion results often contain modality-specific regions and fusion artifacts, limiting the comprehensive representation of scene structures and semantic information. To address this challenge, we propose a generative registration-fusion framework, termed SeeRF. Specifically, it first introduces a deformation generation strategy based on the diffeomorphic stationary velocity field, combined with visual foundation model-guided feature extraction and a coarse-to-fine progressive multi-modal image registration module, to achieve accurate cross-modal flow estimation and generate valid field-of-view masks. Subsequently, the registration module is tightly coupled with the generative diffusion module for multi-modal completion and fusion, where the cross-modal control encoding module and the control residual feature encoder-decoder mechanism provide effective guidance. By leveraging the powerful prior modeling capability of generative diffusion models, the proposed framework overcomes the limitations imposed by shared fields of view and enables effective completion and natural fusion of non-overlapping field-of-view regions. Extensive experiments on multiple public datasets and real-world scenarios demonstrate that the proposed method achieves superior performance, validating its effectiveness, superiority, and robustness. The code will be available at https://github.com/XunpengYi/SeeRF.

I. INTRODUCTION

Existing joint registration–fusion methods often assume shared field-of-view coverage, leaving non-overlapping regions and modality-specific information inadequately integrated. SeeRF couples registration with generative completion and fusion to address both viewpoint misalignment and field-of-view inconsistency.

I. INTRODUCTION

  • Existing joint registration–fusion methods often assume consistent field-of-view coverage, leaving non-overlapping regions without reliable cross-modal integration and risking discontinuities or reduced spatial coverage.These issues can compromise semantic integrity and downstream-task performance.Source: §I · pp. 1–2
  • SeeRF achieves superior performance and robustness across multiple datasets for fusion beyond the field of view.The reported experiments also cover cross-modal registration and fusion, with improved downstream semantic-task performance.Source: §I · p. 2
  • SeeRF jointly addresses spatial misalignment and field-of-view inconsistency through generative image registration and fusion.The authors describe it as the first image-fusion framework to explicitly explore and address field-of-view discrepancies.Source: §I · p. 2
  • Its registration module uses diffeomorphic stationary-velocity-field deformation generation, foundation-model-guided features, and progressive flow estimation for cross-modal alignment.The module also generates valid field-of-view masks for the coupled fusion process.Source: §I · p. 2
  • A coupled diffusion module uses registration masks to complete and fuse information in regions outside the shared field of view.The framework aims to exploit complementary information beyond the sensors’ overlapping coverage.Source: §I · p. 2

II. RELATED WORK

The section reviews image-fusion and registration-fusion approaches before formulating SeeRF’s joint registration and generative-completion approach to field-of-view mismatch.

A. Classical Image Fusion Methods

  • Image fusion methods progressed from autoencoders with hand-crafted rules to end-to-end CNNs and Transformer-, diffusion-, and lookup-table-based approaches.Fusion has also been coupled with upstream registration and downstream vision applications.Source: A. Classical Image Fusion Methods · pp. 2–3

B. Image Registration and Fusion Methods

  • Existing registration-fusion methods address misalignment but remain unable to resolve field-of-view inconsistencies, limiting visual appearance and semantic information.Examples include RFNet, UMF-CMGR, C2RF, MulFS-CAP, and FusionRegister.Source: B. Image Registration and Fusion Methods · p. 3

III. METHODOLOGY

  • The methodology comprises problem formulation, a registration module, a generative diffusion module for completion and fusion, diffeomorphic deformation generation, and loss functions.Source: §III · p. 3

A. Problem Formulation

  • Classical fusion aligns misregistered images before fusion, but registration alone cannot recover information beyond a single-modality field of view.The proposed formulation instead integrates registration with generative completion.Source: A. Problem Formulation · p. 3
  • SeeRF couples registration with generative completion to use information beyond the field of view and improve fused appearance and downstream-task performance.The framework includes a registration module and a generative diffusion module for completion and fusion.Source: A. Problem Formulation · pp. 3–4, Figure 3 · p. 3

B. Deformation Generation Strategy

SeeRF synthesizes global and local deformations using homography and a diffeomorphic stationary velocity field, producing invertible forward–inverse training pairs for registration.

B. Deformation Generation Strategy

  • A global homography and local diffeomorphic deformation form a deformation model that supports spatially varying non-rigid motion while preserving continuous invertibility.The local deformation is parameterized by a stationary velocity field, avoiding non-invertible random elastic fields in training-sample construction.Source: B. Deformation Generation Strategy · p. 4
  • The stationary velocity field induces a unit-time flow through an exponential map, while reversing the field yields the inverse flow from the same generator.The flow exponential converts infinitesimal motion into a finite spatial transformation.Source: B. Deformation Generation Strategy · p. 4
  • Forward and inverse deformation fields generate elastically deformed infrared–visible image pairs that supervise registration transformations.Source: B. Deformation Generation Strategy · p. 4

C. Multi-Modal Image Registration Module

SeeRF’s registration module combines foundation-model and pyramid features with coarse-to-fine flow estimation, then derives a validity mask for the registered overlap.

C. Multi-Modal Image Registration Module

  • The registration module addresses viewpoint discrepancies and supplies registration-confidence masks that support its coupling with completion and fusion.Source: C. Multi-Modal Image Registration Module · p. 4
  • Registration features combine pyramid features with deep DINOv3 foundation features through gated injection to form robust representations.The foundation features provide semantic-level information, while the gate modulates their injection into deep pyramid levels.Source: C. Multi-Modal Image Registration Module · pp. 4–5
  • Progressive coarse-to-fine flow estimation refines a coarse field with local correlation until the full-resolution flow can warp the source image.The module uses global matching followed by progressively upsampled local refinement.Source: C. Multi-Modal Image Registration Module · p. 5
  • The full-resolution flow maps pixel coordinates, and the validity mask marks locations that fall within the corresponding image boundary.The resulting mask characterizes the geometrically valid region defined by the estimated flow.Source: C. Multi-Modal Image Registration Module · p. 5

D. Generative Diffusion Module for Multi-Modal Completion and Fusion

The diffusion module couples registered inputs and a validity mask to complete and fuse regions beyond the shared field of view. Full-view fusion targets, visible-image control, and a trainable residual feature path guide generation.

D. Generative Diffusion Module for Multi-Modal Completion and Fusion

  • The module is tightly coupled with registration through the registered source image and validity mask to unify fields of view and fuse the modalities.Source: D. Generative Diffusion Module for… · p. 5
  • Pre-aligned full-view source images are passed through a pretrained fusion network to produce fused targets that transfer prior knowledge into the diffusion model.The target is encoded with a VAE before the forward diffusion process.Source: D. Generative Diffusion Module for… · p. 5
  • The registered infrared image and validity mask condition the completion branch, while visible-image latent features provide spatial guidance through a cross-modal control encoder.The infrared image is treated as the main completion branch because its field of view is typically smaller.Source: D. Generative Diffusion Module for… · pp. 5–6
  • The reverse diffusion process iteratively refines latent features, and training first pretrains the diffusion model on full-view infrared targets before freezing its UNet and training the control encoder.The optimization objective is defined for the reverse diffusion process.Source: D. Generative Diffusion Module for… · p. 6
  • A trainable control residual feature encoder injects features through residual connections and zero convolutions into a frozen pretrained decoder to enhance fine-grained texture.Decoder and injected features are coupled by zero convolution and addition.Source: D. Generative Diffusion Module for… · p. 6

V AE

The passage states that the fused image is obtained by estimating an approximate output and introduces an optimization objective.

V AE

  • The fused image is produced by estimating an approximate output from the latent representation, followed by a defined optimization objective.Source: V AE · p. 6

E. Loss Functions

The loss design separates registration supervision from diffusion-based fusion supervision. It constrains multi-scale flow estimates and the generated fused image.

E. Loss Functions

  • Registration is supervised by a flow loss that constrains multi-scale predicted flow fields against ground-truth flow.At level l, the target is the ground-truth flow and the prediction is the network-estimated flow.Source: E. Loss Functions · p. 6
  • Fusion uses a latent-space diffusion loss to supervise the denoising network and support generative completion and fusion.Source: E. Loss Functions · p. 6
  • A fused-image consistency loss constrains the generated completion against the full-view fused image.The full-view image is the reference, and the estimated fused image is the completion result.Source: E. Loss Functions · p. 6

IV. EXPERIMENTS

The experiments evaluate registration, fusion, and joint registration-fusion using public infrared-visible datasets, full-field-of-view source images, and established comparison methods.

IV. EXPERIMENTS

  • Experiments use MFNet, FMB, and LLVIP, with separate training and testing image-pair splits for each dataset.The splits are 784/393 pairs for MFNet, 1220/280 for FMB, and 1973/432 for LLVIP; real-world datasets are also tested.Source: §IV · p. 7
  • Registration training uses 520 × 520 patches, and fusion training uses original-resolution images; synthetic transformations include homographies and bounded elastic deformations.Homography rotations range from −4° to 4°, translations are capped at 8% of image dimensions, and elastic displacements at 5 pixels.Source: §IV · p. 7
  • Registration is assessed with EPE, MAE, MI, CC, and SSIM, while fusion is assessed with SD, VIF, CC, QAB/F, and SSIM using full-field-of-view source images.Lower EPE and MAE indicate better registration; higher values indicate better alignment or fusion quality for the other listed metrics.Source: §IV · p. 7
  • Comparisons include RIFT, GLUNet, and CrossRAFT for registration, and U2Fusion and DDFM for fusion.Source: §IV · p. 7
  • Joint registration-fusion comparisons include UMF-CMGR, MURF, SuperFusion, AUNet, MulFS-CAP, and CPMFusion.Source: §IV · p. 7

A. Results and Analyses

SeeRF achieves strong registration and fusion results across MFNet, FMB, and LLVIP, including more complete, spatially consistent fused scenes despite source-image misalignment.

Qualitative Results on Image Registration

  • SeeRF achieves the strongest fine-grained object alignment and global spatial consistency in the registration visualizations.The assessment compares registered infrared structure with ground truth and spatial correspondence between infrared edges and visible-image gradients.Source: A. Results and Analyses · p. 7, Figure 6 · p. 8

Quantitative Results on Image Registration

  • SeeRF achieves the best overall registration performance across MFNet, FMB, and LLVIP on EPE and MAE, MI, CC, and SSIM metrics.The results indicate more accurate flow estimates and stronger structural consistency with reference images.Source: A. Results and Analyses · p. 8, Table I · p. 8

Qualitative Results on Image Fusion

  • SeeRF produces more complete, natural fused scenes by recovering structures in non-overlapping field-of-view regions while maintaining spatial consistency.The comparisons include missing pedestrian information and structures such as exhaust pipes, tires, road surfaces, handlebars, and feet.Source: A. Results and Analyses · pp. 8–9
  • The fusion comparisons use source images that are themselves spatially misaligned.The visualizations cover MFNet, FMB, and LLVIP.Source: Figure 7 · p. 9

Quantitative Results on Image Fusion

  • Across MFNet, FMB, and LLVIP, SeeRF shows superior fusion performance; it leads four MFNet metrics and all LLVIP metrics, with competitive results on other measures.On MFNet, SeeRF improves over Reg.+U2Fusion by 1.8385 SD, 0.1962 VIF, 0.0398 CC, and 0.1154 QAB/F.Source: A. Results and Analyses · p. 9, Table II · p. 10

B. Downstream Task Performance Validation

SeeRF improves downstream task performance, achieving the best reported object-detection metrics on LLVIP and the best semantic-segmentation mIoU on MFNet and FMB.

Semantic Segmentation

Object Detection

C. Ablation Study

Ablations show that the registration and fusion components contribute to the framework’s task performance, while disabling generative completion reduces the credibility of generated out-of-view content.

C. Ablation Study

  • Removing any individual registration component worsens registration results, increasing EPE and MAE while lowering MI, CC, and SSIM.The variants remove foundation-model guidance, diffeomorphic deformation generation, or coarse-to-fine flow regression.Source: C. Ablation Study · p. 12
  • The complete model performs best on all fusion metrics except SD; removing CRFE, CCE, or VFM degrades fused-scene quality.CRFE removal yields structural distortion and detail loss, CCE removal weakens cross-modal guidance, and disabling VFM generation produces incomplete, discontinuous out-of-view scenes.Source: C. Ablation Study · p. 12
  • Disabling generation beyond the shared field of view lowers segmentation mIoU, while removing CRFE or CCE also degrades downstream segmentation.The qualitative comparison links missing out-of-view information and fusion discontinuities with truncated target contours.Source: C. Ablation Study · pp. 12–13, Table VII · p. 13, Figure 12 · p. 13

D. Generative Credibility Analysis Experiment

E. Real-World Validation Experiment

Real-world comparisons show SeeRF aligning targets and boundaries while completing missing regions; its diffusion-based fusion component carries substantial computational overhead.

E. Real-World Validation Experiment

  • In real-world scenes, SeeRF achieves more accurate target and boundary alignment and reconstructs missing out-of-view content into more complete, coherent fused images.Competing methods show residual misalignment, boundary distortion, or discontinuities between observed and missing regions.Source: E. Real-World Validation Experiment · pp. 13–14, Figure 13 · p. 14

F. Computational Overhead

  • SeeRF-Fus. uses 1701.765M parameters, 1968.819G FLOPs, and 5.293s inference time, compared with 113.367M parameters, 437.667G FLOPs, and 0.142s for SeeRF-Reg.The fusion component is diffusion-based and carries higher computational costs.Source: F. Computational Overhead · p. 14, Table IX · p. 14

V. LIMITATION

  • The foundation and diffusion models improve task performance but increase computational complexity and inference overhead, leaving room for lightweighting and optimization.The authors identify single-step generation and flow matching as possible future directions.Source: §V · p. 14

VI. CONCLUSION

  • SeeRF combines diffeomorphic stationary-velocity-field deformation, foundation-model-guided registration, and coupled diffusion completion and fusion to address shared-field-of-view limits.Experiments on public datasets and real-world data demonstrate the method’s effectiveness and superiority.Source: §VI · p. 15

Related papers

Loading 2610.35637v1…