S2A:Semantic-to-Spatial Alignment for Alignment-Free RGB-T Salient Object Detection

1. INTRODUCTION

• GGHF uses high-level semantic guidance to suppress background interference and selectively integrate multi-level intra-modal features.Source: §1 · p. 1
• AFCA exchanges complementary semantics through channel correlations, bypassing strict pixel-wise alignment during cross-modal interaction.Source: §1 · p. 1
• SDCA predicts adaptive sampling offsets to recover fine-grained local correspondence after semantic interaction.Source: §1 · pp. 1–2
• S2A establishes cross-modal semantic correspondence before recovering spatial correspondence, reducing feature contamination from misalignment.This avoids premature spatial interaction between features that may come from different physical regions.Source: §1 · pp. 1–2
• Extensive experiments across multiple public alignment-free RGB-T benchmarks verify S2A’s effectiveness, robustness, and generalizability.Source: §1 · p. 2

Global-Guided Hierarchical Fusion Module (GGHF)

• GGHF uses global semantics to suppress background interference and refine hierarchical intra-modal features.It combines global guidance with local cues for bidirectional refinement, aggregates the refined features into shared hierarchical attention, and applies that attention to enhance the representations.Source: §2 · p. 2

Alignment-Free Cross-modal Attention Module (AFCA)

• AFCA exchanges cross-modal semantics through channel-wise attention, reducing reliance on strict pixel-wise correspondence.It uses fmain for Q and V and fguide for K; after spatial flattening, global aggregation produces channel correlations.Source: §2 · pp. 2–3

Spatial Deformable Cross-Attention Module (SDCA)

• SDCA predicts offsets from an RGB query to resample thermal features and recover local cross-modal correspondence.It uses bilinear grid sampling, then projects the calibrated feature into keys and values for cross-attention; the attended feature is concatenated with RGB features and refined by a 3 × 3 convolution.Source: §2 · p. 3

Decoder and Loss Function

• A lightweight top-down decoder aggregates fused multi-scale features into a prediction map, trained end-to-end with binary cross-entropy, smoothness, and soft Dice losses.The decoder uses bilinear upsampling and convolutional refinement.Source: §2 · p. 3

3.1. Evaluation

• Evaluation uses the same training set as previous methods, tests two unaligned and three weakly aligned datasets, and reports Sm, Em, and Fm.Source: §3.1 · p. 3
• S2A achieves the best performance on almost all metrics and outperforms TPS-SCL on all three metrics across all five evaluated datasets.The five datasets comprise two unaligned and three weakly aligned datasets.Source: §3.1 · p. 3, Table 1 · p. 4
• Visual comparisons report more accurate saliency maps, more complete object structures, and sharper boundaries across challenging conditions.The reported conditions include cluttered backgrounds, weak alignment, low illumination, degraded or diffused thermal signals, and fine structures.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2. Ablation Studies

• Replacing GGHF with standard convolutions increases false positives, replacing AFCA with concatenation drops performance, and replacing SDCA with rigid fusion degrades all metrics.Source: §3.2 · p. 4, Table 2 · p. 4
• Reversing the semantic-to-spatial alignment order into spatial-to-semantic degrades performance.Source: §3.2 · p. 4

• S2A sequentially refines intra-modal representations, interacts across modalities semantically, and recovers local spatial correspondence.Source: §4 · p. 4
• Experiments on multiple public benchmarks demonstrate S2A's effectiveness under varying degrees of spatial misalignment.Source: §4 · p. 4
• The paper concludes that semantic-to-spatial alignment is an effective strategy for robust multimodal perception under imperfect spatial correspondence.Source: §4 · p. 4