PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation

Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, Xin Fan

arXiv:2308.02097v1cs.CV

TL;DR

SegMiF addresses the difficulty of jointly achieving strong multi-modality image fusion and segmentation by enabling feature interaction between the two tasks. It combines hierarchical interactive attention and dynamic weighting, and reports visually appealing fused images, improved segmentation, and a full-time benchmark with 15 pixel-level categories.

  • Problem

    Prior efforts often optimize fusion or segmentation separately, while visible sensors can fail in complex environments and infrared sensors have limited spatial resolution.

  • Method

    SegMiF jointly uses fusion and segmentation networks with hierarchical interactive attention and dynamic weighting to exchange features and automatically balance the two tasks.

  • Results

    7.66% higher mIOU on FMB than the second-ranked method, alongside visually appealing fused images and strong segmentation performance.

  • Takeaways & Limitations

    The method and FMB benchmark support joint evaluation and development of multi-modality fusion and segmentation for real-world scene understanding.

Abstract

from arXiv · show

Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{https://github.com/JinyuanLiu-CV/SegMiF}.

1. Introduction

Multi-modality fusion and segmentation must jointly deliver visually appealing fused images and accurate pixel-level scene parsing, especially where visible or infrared sensing alone is limited. SegMiF addresses this challenge with mutually interactive features, hierarchical attention, dynamic weighting, and a full-time benchmark.

  • Motivation: Visible sensors may fail in complex environments, while infrared sensors avoid those issues but suffer from low spatial resolution.Fusing infrared and visible images is presented as a mainstream solution for better scene understanding.
  • Problem: Joint multi-modality fusion and segmentation must produce high-quality images while assigning accurate category labels to every pixel.The paper frames these as simultaneous requirements for robust scene parsing.
  • Problem: Existing fusion methods often prioritize visual appearance, while multitask approaches still struggle to learn unified features suitable for both tasks.This limits downstream scene parsing and makes feature sharing between fusion and segmentation difficult.
  • Problem: Existing multi-modality collections either focus on fusion or lack whole-image pixel-level segmentation labels, hindering study of fusion–segmentation correlation.The paper identifies well-aligned image pairs and broad pixel annotations as requirements for this joint problem.
  • Proposed Approach: SegMiF jointly processes fusion and segmentation, using hierarchical interactive attention to fine-grainedly integrate semantic- and modality-oriented features.Its fusion and segmentation networks exchange intrinsic features through the proposed attention mechanism.
  • Proposed Approach: A dynamic weighting factor automatically learns task parameters, while FMB provides 1,500 registered infrared–visible pairs with 15 pixel-level categories across severe environments.The benchmark includes dense fog, heavy rain, and low-light conditions.

2. Related Works

Multi-modality fusion research has advanced through network and loss-function design, but relatively few methods address downstream-task adaptation. Recent work instead cascades fusion with downstream tasks to improve task performance.

  • Multi-modality Image Fusion: Deep learning has driven significant progress in multi-modality image fusion, with early methods adjusting network structures or loss functions.These approaches primarily target improved fusion effects.
  • Multi-modality Image Fusion: Only a minority of fusion methods consider whether downstream tasks can adapt well to the fused output.This distinguishes downstream-task compatibility from optimizing fusion quality alone.
  • Multi-modality Image Fusion: Recent methods cascade fusion and downstream tasks, focusing on improving task performance.The passage presents cascading as a newer direction in multi-modality fusion research.

3. The Proposed Method

SegMiF jointly optimizes image fusion and segmentation through interactive feature learning in a cascade architecture. Hierarchical attention exchanges modality and semantic features, while dynamic weighting balances the two tasks during alternating training.

  • 3.1. Problem formulation: SegMiF jointly formulates image fusion and segmentation so semantic and pixel-based features can interact across tasks.The fused image and segmentation map are produced by separate networks, with a feature-learning constraint coupling their optimization.
  • 3.1. Problem formulation: The cascade architecture uses parallel dilated residual dense blocks for visible and infrared features and SegFormer for semantic parsing.Two scales of interpolated semantic features are embedded into the fusion network.
  • 3.2. Feature interaction architecture: Hierarchical interactive attention maps fusion modality features and segmentation features through two complementary attention mechanisms.SoAM emphasizes semantic information for modality features, while MoAM extracts significant modality features from semantic contexts before residual aggregation.
  • 3.3. Loss function: The fusion loss combines structural similarity, saliency-based pixel, and multi-scale gradient losses, while the segmentation loss uses the segmentation objective.The fusion term is summarized as Lf = LSSIM + LMSE + ηLgrad, and the total objective combines fusion and segmentation losses.
  • 3.4. Dynamic factor for interactive learning: Dynamic weighting factors measure task-loss importance from convergence behavior and automatically balance interactive learning.The strategy uses a temperature T and avoids the complicated computation of various task gradients associated with GDN.
  • 3.4. Dynamic factor for interactive learning: Fusion and segmentation are trained alternately end-to-end until convergence, and the task-agnostic strategy can incorporate other high-level vision tasks.The segmentation network is optimized using the generated fusion image and semantic features.

4. Full-time Multi-modality Benchmark

The FMB benchmark addresses limitations in existing multi-modality segmentation datasets by providing aligned visible/infrared data across varied driving conditions with extensive pixel-level labels.

  • Existing datasets have few label categories, sparse annotation, and monotonous scenes.
  • The benchmark uses a roof-mounted binocular system combining a visible camera with an infrared sensor operating at 8–14 µm.
  • The sensors produce 1,500 aligned image pairs at 800×600 resolution through calibration, homography estimation, projection, and cropping.
  • FMB covers real driving scenes under varied illumination, including rain, fog, strong light, and Tyndall Effect scenarios.
  • The dataset labels 98.16% of pixels across 14 categories relevant to real-world autonomous driving and semantic understanding.

5. Experiments

Experiments evaluate SegMiF for image fusion, semantic segmentation, and component effectiveness on MFNet and FMB. The method shows strong fusion quality, improved segmentation, and benefits from HIA and dynamic weighting.

  • Experimental setup: SegMiF is trained and evaluated on MFNet and FMB using augmentation and joint optimization of fusion and segmentation networks.Training uses resizing, cropping, brightness distortion, normalization, and Adam optimization with poly learning-rate adjustment.
  • Image fusion results: SegMiF preserves salient infrared characteristics and improves visual details in qualitative fusion comparisons on MFNet and FMB.The reported advantages include highlighting infrared information and enhancing texture details for observation.
  • Image fusion results: SegMiF achieves consistent superiority across entropy, standard deviation, spatial frequency, and SCD in quantitative fusion comparisons.The paper associates higher EN and SCD with preserved source information, higher SD with pixel contrast, and higher SF with texture details.
  • Semantic segmentation results: SegMiF produces more accurate segmentation in challenging scenes, while competing fusion and dual-stream methods show errors on dim, distant, occluded, or complex objects.The authors attribute the improvement to transferring complementary characteristics between segmentation and fusion through interactive feature learning.
  • Semantic segmentation results: 7.66% and 1.45% are the reported mIoU improvements over the second-ranked method on FMB and MFNet, respectively.The method also ranks second in mACC and performs strongly for categories such as Car, Person, traffic sign, building, and bump.
  • Ablation studies: HIA improves cross-task feature bridging, preserving salient infrared features while retaining semantic information and supporting both visual observation and segmentation.Removing SoAM or MoAM causes confusion, detail loss, or color distortion in the reported variants.
  • Ablation studies: The dynamic factor avoids laborious manual adjustment and better introduces task preferences into optimization for joint visualization and segmentation performance.The paper reports that other training strategies poorly coordinate the two tasks, whereas the dynamic factor achieves strong results on both.

6. Conclusion

The paper presents SegMiF as a multi-interactive architecture for jointly addressing fusion and segmentation, using HIA and dynamic factors to bridge cross-task features. It also introduces a full-time benchmark with registered targets, abundant scenes, and diverse labels.

  • Conclusion: SegMiF formulates image fusion and segmentation jointly through a multi-interactive architecture with hierarchical interactive attention and dynamic factors.These components bridge cross-task feature gaps from architectural and learning perspectives.
  • Conclusion: The proposed benchmark provides well-registered targets, abundant scenes, and affluent labels for full-time multi-modality evaluation.
Loading 2308.02097v1…