Source-linked AI summary
Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond
Guanyao Wu, Haoyu Liu, Hongming Fu, Yichuan Peng, Jinyuan Liu, Xin Fan, Risheng Liu
TL;DR
IVIF must preserve complementary infrared and visible information while supporting downstream tasks, but existing approaches struggle to optimize both goals together. SAGE uses SAM semantic priors through SPA and distills them into a smaller network for practical inference. Experiments report superior fusion and task performance across multiple datasets, including a 3.0 mIoU improvement on FMB over the second-best approach.
Problem
IVIF methods struggle to preserve visual quality while adapting fused images to downstream tasks because their optimization objectives can conflict.
Method
SAGE combines Semantic Persistent Attention with bi-level triplet-loss distillation to retain source information, integrate SAM priors, and decouple inference from SAM.
Results
3.0 mIoU improvement over the second-best approach on FMB was reported for traditional segmentation comparisons, alongside superior fusion and task performance across multiple datasets.
Takeaways & Limitations
The distilled model delivers high-quality visual results and downstream task performance without direct SAM involvement during inference.
Abstract
from arXiv · showhide
Multi-modality image fusion, particularly infrared and visible, plays a crucial role in integrating diverse modalities to enhance scene understanding. Although early research prioritized visual quality, preserving fine details and adapting to downstream tasks remains challenging. Recent approaches attempt task-specific design but rarely achieve "The Best of Both Worlds" due to inconsistent optimization goals. To address these issues, we propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability, namely SAGE. Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM. More importantly, to eliminate the impractical dependence on SAM during inference, we introduce a bi-level optimization-driven distillation mechanism with triplet losses, which allow the student network to effectively extract knowledge. Extensive experiments show that our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency. The code is available at https://github.com/RollingPlain/SAGE_IVIF.
1. Introduction
Infrared and visible images offer complementary strengths, but IVIF methods struggle to balance visual quality with downstream task adaptability. SAGE addresses this tension by using SAM semantic priors and distillation to retain adaptability while enabling practical inference.
- Infrared images resist smoke, obstruction, and low light, while visible images provide resolution, contrast, and texture detail.
- Existing IVIF methods either prioritize fused-image visualization or couple fusion with task objectives, creating conflicting optimization goals.
- SAM provides robust semantic information suited to IVIF and downstream tasks, but directly using the full model during inference is computationally impractical.
- SAGE combines a Semantic Persistent Attention module with a bi-level distillation scheme to integrate SAM priors, preserve source information, and remove SAM from inference.
- The proposed framework targets a balance between visual quality and downstream task adaptability through a smaller distilled sub-network.
2. Related Works
Related work has progressed from fusion-quality optimization toward joint fusion and downstream-task learning, while SAM offers broad semantic generalization for low-level vision and IVIF. The proposed workflow uses SAM-guided semantic processing and distillation to support this integration.
- Deep learning IVIF methods improve details, structural consistency, and noise reduction but generally do not address downstream detection or segmentation.
- Task-oriented methods optimize fusion for detection or segmentation, and multi-task approaches jointly improve performance across these downstream tasks.
- SAM is a large-scale pretrained model with zero-shot generalization across segmentation and detection tasks and applications in several low-level vision problems.
- The proposed workflow processes SAM-generated semantic-prior patches with SPA, preserves source information through a persistent repository, and transfers knowledge to a dense-block sub-network.
3. The Proposed Method
SAGE combines SAM-derived semantic priors with source-image information through SPA, then transfers this knowledge to a SAM-independent sub-network using bi-level distillation. Its design uses persistent contextual memory and complementary losses to preserve semantic, structural, and intensity information during fusion.
- Bi-level distillation: The framework transfers SAM-driven knowledge from a main network to a lightweight sub-network through bi-level optimization, reducing dependence on SAM during inference.The two networks are jointly optimized with outputs from the counterpart network, creating a bidirectional distillation dependency.
- Semantic Persistent Attention: SPA combines SAM semantic patches with source features by using a Persistent Repository to provide stable modality-specific contextual information.The repository stores source-feature representations and key-value pairs used during cross-attention.
- Semantic Persistent Attention: Cross-attention enriches limited-coverage semantic patches with full-scene context from the visible and infrared source images.The encoded semantic features are queried against repository key-value pairs to address incomplete scene coverage.
- Triplet loss distillation: The distillation scheme uses feature alignment, context consistency, and contrastive semantic losses to guide the sub-network toward the main network’s fused representation.The context component includes gradient and MSE terms, while the contrastive term separates fused features from individual modality features.
- Triplet loss distillation: The context loss preserves structural and intensity consistency while comparing fused outputs with both the reference fusion and original source images.Applying these terms to source images is intended to maintain reconstruction fidelity during distillation.
- Triplet loss distillation: The main network additionally uses segmentation cross-entropy, whereas the sub-network is trained through the distillation objectives.The segmentation loss is applied to the main network to avoid potential optimization conflicts in the sub-network.
4. Experiments
Experiments evaluate SAGE across fusion quality, semantic segmentation, ablations, and computational efficiency. The results report strong visual and segmentation performance, while distillation reduces inference costs and preserves semantic information.
- Experimental Setup: Experiments use five datasets and compare SAGE with nine state-of-the-art fusion methods.The datasets are TNO, RoadScene, MFNet, FMB, and M3FD.
- Fusion Quality: SAGE preserves visible and infrared details while reconstructing structures robustly under smoke, fog, nighttime, and other interference.Examples include vegetation, chimney smoke, pedestrian crossing lines, and distant building outlines.
- Fusion Quality: SAGE demonstrates consistent superiority across EN, SD, SCD, MS-SSIM, BRISQUE, NIQE, MUSIQ, and PaQ-2-PiQ evaluations.The reported metrics assess source information preservation, detail and texture richness, pixel distribution, and perceptual alignment.
- Segmentation: SAGE achieves strong segmentation results on FMB and MFNet, including a 3.0 mIoU improvement over the second-best approach on FMB.The method also segments difficult objects and structures, including trucks, buses, distant pedestrians, sidewalks, and nighttime lane curves.
- Ablation Studies: Ablation results indicate that the Persistent Repository preserves critical information, while removing SPA components causes significant information loss.The study evaluates variants without latent representations, key-value pairs, or the Persistent Repository.
- Computational Efficiency: The distilled model processes M3FD inputs in 10.47 ms with 52.06 G FLOPs and 0.136M parameters, reducing SAM-related inference overhead while preserving semantic information.The authors attribute the efficiency to the distillation scheme and its lightweight sub-network.
5. Conclusion
SAGE uses SAM-derived semantic priors for infrared-visible image fusion, combining source-detail retention with downstream task performance while reducing reliance on SAM during inference.
- SAGE integrates SAM-derived semantic priors into infrared-visible image fusion while retaining source details.
- The Semantic Persistent Attention module incorporates semantic information while preserving source-image details.
- A bi-level distillation scheme with triplet loss decouples fusion from SAM during inference and reduces computational complexity.
- Extensive experiments across multiple datasets show superior fusion and task performance compared with other methods.