Source-linked AI summary
Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection
Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, Zhongxuan Luo
TL;DR
Infrared-visible fusion methods often prioritize visual quality while neglecting modality differences important for object detection. The paper jointly optimizes fusion and detection with TarDAL and a detection network, and builds M3FD for diverse evaluation. TarDAL produces visually appealing fusion and higher detection mAP than competing methods, while M3FD broadens multi-scenario benchmarking.
Problem
Existing infrared-visible fusion approaches neglect complementary modality differences and often lack comprehensive data for learning and evaluating detection across varied conditions.
Method
The paper unrolls a bilevel fusion-detection formulation into TarDAL, using one generator and target-aware dual discriminators, with cooperative training and a commonly used detection network.
Results
TarDAL achieves higher detection mAP than other methods on two datasets, improving by 1.4% and 1.1% over DenseFuse and GANMcC, respectively.
Takeaways & Limitations
The method preserves discriminative infrared targets and visible texture while supporting visually friendly fusion and object detection, and M3FD enables evaluation across diverse scenarios.
Takeaways & Limitations
The benchmark uses image pairs from one infrared-visible sensor set, while binocular-camera depth data is reserved for future publication.
Abstract
from arXiv · showhide
This study addresses the issue of fusing infrared and visible images that appear differently for object detection. Aiming at generating an image of high visual quality, previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks. These approaches neglect that modality differences implying the complementary information are extremely important for both fusion and subsequent detection task. This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network. The fusion network with one generator and dual discriminators seeks commons while learning from differences, which preserves structural information of targets from the infrared and textural details from the visible. Furthermore, we build a synchronized imaging system with calibrated infrared and optical sensors, and collect currently the most comprehensive benchmark covering a wide range of scenarios. Extensive experiments on several public datasets and our benchmark demonstrate that our method outputs not only visually appealing fusion but also higher detection mAP than the state-of-the-art approaches.
1. Introduction
Infrared and visible images provide complementary target and texture information, but their appearance discrepancy makes fusion for object detection difficult. The paper jointly optimizes fusion and detection with TarDAL and introduces M3FD to support learning and evaluation across diverse scenarios.
- Visible images provide high-resolution details under suitable lighting, whereas infrared images highlight thermally salient targets but often have blurred, lower-resolution details.
- Existing fusion methods emphasize common information and typically neglect modality differences that preserve infrared target structure and visible background texture for detection.
- The paper formulates fusion and detection jointly through bilevel optimization and unrolls it into a target-aware dual-adversarial fusion network with a detection network.
- TarDAL uses one generator and two discriminators to preserve infrared target information and visible textural details while enabling fast fusion and detection inference.
- M3FD contains 4, 177 aligned infrared-visible image pairs and 23, 635 annotated objects across varied environments, illumination, seasons, and weather.
2. Related Works
Prior work includes transform, optimization, representation, subspace, deep-network, and adversarial approaches for infrared-visible fusion, alongside benchmarks with limited scale or scenario diversity. The paper presents its methodology framework and positions M3FD as a broader multi-modality benchmark for detection-oriented fusion.
- Deep IVIF methods learn feature representations or weighting strategies, but training often depends on redundant, well-prepared image pairs.
- GAN-based fusion methods improve visual appearance through adversarial distribution matching, but identical adversarial treatment of modalities can lose vital infrared information.
- Existing benchmarks include datasets such as TNO, INO, OSU Color-Thermal, RoadScene, and Multispectral, covering different sensor and application settings.
- Figure 2 organizes the paper's approach into bilevel fusion-detection optimization, target-aware dual-adversarial fusion, and cooperative training.
- Limited resolution, object counts, scenario types, and labels restrict existing datasets' use for higher-level multi-modality detection.
3. The Proposed Method
The method jointly optimizes fusion and detection through a bilevel formulation unrolled into cooperative fusion and detection networks. TarDAL combines common and modality-specific information using target- and detail-aware adversarial supervision.
- Problem formulation: The bilevel formulation seeks fused images and detector parameters that mutually favor fusion quality and object detection.Fusion is represented by Φ, detection by Ψ, and the formulation is converted to a single-level optimization using learned network parameters.
- Target-aware dual adversarial network: TarDAL uses one generator and two discriminators to preserve infrared target structures and visible background textures.The target discriminator evaluates infrared target intensity consistency, while the detail discriminator compares visible and fused gradient distributions.
- Target and detail discriminators: Infrared targets and visible background details are separated using a saliency-derived target mask before adversarial discrimination.The mask defines target and background regions, while gradient operations such as Sobel support detail discrimination.
- Generator: The generator combines structural similarity, saliency-weighted pixel fidelity, and adversarial objectives to produce the fused image.The fused output has the same size as the source images; SSIM encourages structural preservation, while SDW-based pixel loss balances source intensity distributions.
- Cooperative training strategy: Cooperative training backpropagates detection gradients through both networks while adding fusion-loss gradients to the fusion parameters.A fusion regularizer and trade-off parameter λ balance fusion and detection objectives rather than relying on a manually designed weighting rule.
- Cooperative training strategy: The trained system is designed to generate visually appealing fused images and accurate detections, with more efficient convergence than independent training schemes.This is the stated outcome of jointly optimizing the fusion and detection networks.
4. Multi-scenario Multi-modality Benchmark
M3FD is a high-resolution infrared–visible benchmark designed for multi-modality detection across diverse object types and scenarios. It combines synchronized imaging, calibration, and alignment to produce corresponding image pairs.
- M3FD contains high-resolution infrared and visible images covering diverse object types under various scenarios.
- A synchronized system uses binocular optical and infrared sensors to capture corresponding two-modality images of natural scenes.
- Visible and infrared cameras differ in resolution, imaging range, and wavelength coverage.Visible images are 1024×768, while infrared images are 640×512 with an 8−14µm wavelength range.
- Camera calibration and a homography matrix produce well-aligned infrared–visible pairs warped to a common 1024 × 768 coordinate system.
- The dataset spans Daytime, Overcast, Night, and Challenge conditions, with ten sub-scenarios and six annotated object classes.The classes are People, Car, Bus, Motorcycle, Truck, and Lamp.
5. Experiments
Experiments evaluate TarDAL for infrared-visible fusion, object detection, computational efficiency, and component effectiveness across public datasets and M3FD. Results show strong visual and detection performance, while ablations connect target and detail discriminators and cooperative training to distinct outcomes.
- Experimental setup: Experiments use four datasets for fusion and detection, with cropped and augmented multimodality patches for training.The setup includes TNO, RoadScene, M3FD, and Multispectral data, with 320×320-pixel crops.
- Fusion results: TarDAL preserves discriminative infrared targets and abundant visible-image texture, producing clearer contours and sharper, more visual-friendly fused results.Qualitative comparisons cover typical image pairs from TNO, RoadScene, and M3FD.
- Fusion results: TarDAL achieves the largest or second-largest mean value across MI, EN, and SD on three datasets, with lower variance indicating more stable visual-scene handling.The largest average MI indicates stronger information transfer, while EN and SD indicate abundant information and high target-background contrast.
- Object detection results: TarDAL improves detection mAP by 1.4% over DenseFuse and 1.1% over GANMcC on the two evaluated datasets.The comparison uses YOLOv5 as the detector on fused outputs from competing methods.
- Computational analysis: TarDAL simultaneously achieves the highest running speed and lower FLOPs and training-parameter complexity among the compared methods.The reported efficiency combines fusion with the follow-up high-level vision application.
- Ablation studies: Removing the target discriminator blurs infrared targets, removing the detail discriminator loses visible texture, and omitting both can inflate EN and SD through heavy noise.The ablation identifies distinct visual roles for the two discriminators and warns that higher EN or SD can reflect noise.
- Ablation studies: Cooperative training improves both detection performance and visual effects compared with direct and task-oriented training strategies.Task-oriented training uses only detection loss and produces worse visual observation quality.
6. Conclusion
The paper jointly formulates fusion and detection through bilevel optimization, unrolls it into a fusion and detection network, and introduces a synchronized multimodality benchmark. It concludes that this framework produces visual-friendly fused results for both fusion and object detection.
- The paper proposes bilevel optimization for jointly realizing infrared-visible image fusion and object detection.
- The unrolled model combines a designed fusion network with a commonly used detection network to generate visual-friendly fused results.
- The paper contributes a synchronized visible-infrared imaging system and a multi-scenario multi-modality benchmark for future research.