Source-linked AI summary
Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks
Heng Zhang, Elisa Fromont, Sébastien Lefevre, Bruno Avignon
TL;DR
Multispectral detection benefits from visible and thermal information, but their differing views can make feature fusion inconsistent. The paper introduces a cyclic halfway fusion module that repeatedly fuses and refines spectral features, and reports improved detection across KAIST and FLIR, including new state-of-the-art KAIST results and a strong FLIR baseline.
Problem
Visible and thermal features are complementary but may be inconsistent, creating a difficult information-fusion problem for multispectral object detection.
Method
The Cyclic Fuse-and-Refine module repeatedly uses fused features to refine visible and thermal features within a halfway-fusion network.
Results
The method improves detection over competing multispectral methods on KAIST and establishes a first strong baseline on FLIR ADAS.
Takeaways & Limitations
Cyclic fusion provides a multispectral feature-fusion approach that accounts for the balance between feature consistency and complementarity.
Abstract
from arXiv · showhide
Multispectral images (e.g. visible and infrared) may be particularly useful when detecting objects with the same model in different environments (e.g. day/night outdoor scenes). To effectively use the different spectra, the main technical problem resides in the information fusion process. In this paper, we propose a new halfway feature fusion method for neural networks that leverages the complementary/consistency balance existing in multispectral features by adding to the network architecture, a particular module that cyclically fuses and refines each spectral feature. We evaluate the effectiveness of our fusion method on two challenging multispectral datasets for object detection. Our results show that implementing our Cyclic Fuse-and-Refine module in any network improves the performance on both datasets compared to other state-of-the-art multispectral object detection methods.
1. INTRODUCTION
Visible and thermal features offer complementary information but can be inconsistent, making multispectral fusion difficult. The paper proposes cyclic fusion and refinement to improve this balance and reports strong results on KAIST and FLIR.
- Visible images provide color and texture, while thermal images capture object temperature and can help detect pedestrians at night.
- Although the visible-thermal pairs are aligned, their predicted segmentation masks can differ substantially, indicating inconsistent multispectral features.
- The Cyclic Fuse-and-Refine module repeatedly refines mono-spectral features using fused features to increase feature quality and consistency.
- Controlling the number of fuse-and-refine loops adjusts the balance between feature consistency and complementarity.
- Experiments report new state-of-the-art results on KAIST and a first strong baseline on FLIR ADAS.
2. RELATED WORK
Prior CNN-based multispectral detection work primarily varies when and how features are fused. Evidence from earlier studies established halfway fusion as the prevailing strategy.
- Existing multispectral CNN approaches mainly differ in the timing and mechanism used to fuse spectral features.
- Earlier studies compared early and late fusion, then found that halfway fusion performed better than either alternative.
- Halfway fusion subsequently became the default strategy in deep-learning multispectral and multimodal methods.
3. PROPOSED APPROACH
The proposed module cycles through fusion and refinement operations, using fused features to refine visible and thermal representations. Auxiliary semantic supervision and averaged intermediate features support training and final detection.
- Overview: Fusion and refinement operations repeat through multiple cycles to increase multispectral feature consistency and decrease complementarity.
- Fuse-and-Refine: In each loop, fused features are computed from visible and thermal features, then assigned as residuals to refine both spectral representations.
- Fuse-and-Refine: The fusion operation uses feature concatenation followed by a 3 × 3 convolution and batch normalization, with shared weights across loops.
- Semantic supervision: An auxiliary semantic segmentation task predicts separate pedestrian masks for refined thermal and visible features to provide supervision and help tune loop count.
- Final fusion: The final fused representation averages all refined spectral features because the optimal cycling number may vary across image pairs.
4. EXPERIMENTS
The experiments evaluate Cyclic Fuse-and-Refine on KAIST and FLIR using CNN-based pedestrian detectors, including comparisons, implementation details, and loop ablations. The method improves detection while adding only small inference overhead, and its benefit varies with the number of refinement loops.
- Experimental setup: The proposed module is evaluated on the KAIST and FLIR multispectral pedestrian detection datasets against state-of-the-art methods.KAIST uses sanitized and original annotations for comparison; FLIR uses a sanitized dataset because of image-pair misalignment.
- Network architecture: The module is implemented on the FSSD single-stage detector, with VGG16 extracting visible and thermal features fused halfway through the network.The baseline uses element-wise averaging, while experiments integrate different numbers of Cyclic Fuse-and-Refine loops.
- On KAIST: On KAIST, the proposed method obtains better detection results than all competitors for both sanitized and original training annotations.The comparison uses the same input resolution and VGG16 backbone across deep learning-based methods.
- On KAIST: Each fuse-and-refine cycle adds approximately 0.4 ms of inference time.The reported overhead applies during inference to each cycle.
- Ablation study: Figure 4 compares visible- and thermal-derived pedestrian segmentation masks after one, two, and three fuse-and-refine loops on day and night image pairs.The visualization examines mask quality and similarity across successive refinements.
- Ablation study: On KAIST ablations, detection accuracy improves from zero to three loops but decreases after four loops, while DICE scores continue increasing.DICE measures similarity between pedestrian masks predicted from refined visible and thermal features.
5. CONCLUSION
The paper proposes Cyclic Fuse-and-Refine to improve multispectral feature fusion by balancing complementary and consistent information. Experiments report substantial accuracy improvements on KAIST and FLIR, while FLIR misalignment remains an application-relevant challenge for future work.
- Conclusion: Cyclic Fuse-and-Refine improves multispectral feature fusion by accounting for the complementary and consistency balance between features.The conclusion identifies the module as the paper’s central contribution.
- Conclusion: Integrating the module into a vanilla multispectral pedestrian detector leads to substantial accuracy improvements on the KAIST and FLIR datasets.The conclusion reports this outcome across both evaluated datasets.
- Conclusion: FLIR contains misaligned visible/thermal image pairs, a problem that may become more serious in real-world applications because of calibration errors or temporal shifts.The authors identify general cyclic alignment as future work.