Source-linked AI summary
STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification
Xingguo Xu, Zhanyu Liu, Weixiang Zhou, Yuansheng Gao, Junjie Cao, Yuhao Wang, Jixiang Luo, Dell Zhang
TL;DR
Multi-modal ReID must exploit complementary modalities despite distribution discrepancies, token loss from hard filtering, background interference, and limited high-order fusion. STMI combines segmentation-guided feature modulation, semantic token reallocation, and cross-modal hypergraph interaction, with experiments reporting state-of-the-art performance across three public datasets. Its CHI ablation reaches 81.2% mAP, while four learnable tokens perform best in the reported token-count analysis.
Problem
Multi-modal ReID faces modality distribution discrepancies, loss of discriminative details from hard token filtering, background interference, and limited modeling of high-order semantic relationships.
Method
STMI combines SAM-guided feature modulation, learnable-query semantic token reallocation, and unified-hypergraph cross-modal interaction.
Results
STMI achieves state-of-the-art performance on three public multi-modal ReID datasets; its CHI configuration reaches 81.2% mAP.
Takeaways & Limitations
STMI provides a multi-modal ReID framework that preserves visual details, suppresses background noise, and models high-order semantic relationships across modalities.
Abstract
from arXiv · showhide
Multi-modal object Re-Identification (ReID) aims to exploit complementary information from different modalities to retrieve specific objects. However, existing methods often rely on hard token filtering or simple fusion strategies, which can lead to the loss of discriminative cues and increased background interference. To address these challenges, we propose STMI, a novel multi-modal learning framework consisting of three key components: (1) Segmentation-Guided Feature Modulation (SFM) module leverages SAM-generated masks to enhance foreground representations and suppress background noise through learnable attention modulation; (2) Semantic Token Reallocation (STR) module employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens; (3) Cross-Modal Hypergraph Interaction (CHI) module constructs a unified hypergraph across modalities to capture high-order semantic relationships. Extensive experiments on public benchmarks (i.e., RGBNT201, RGBNT100, and MSVR310) demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.
Introduction
Multi-modal ReID must learn across modality distribution discrepancies while retaining discriminative details and suppressing background interference. STMI addresses these challenges with segmentation-guided modulation, token reallocation, and cross-modal hypergraph interaction.
- Multi-modal ReID remains challenging because substantial distribution discrepancies exist across modalities.
- Existing methods may discard critical details through hard token cropping and inadequately model high-order semantic relationships during feature fusion.
- STMI introduces SFM, which uses SAM-generated foreground masks and learnable modulation parameters to emphasize foreground regions and suppress background noise.
- STR uses learnable query tokens and cross-attention to extract compact semantic representations while preserving fine-grained visual details instead of hard-filtering tokens.
- CHI models high-order cross-modal relationships by treating RGB, NIR, and TIR semantic tokens as nodes in a unified hypergraph.
- Experiments on three public multi-modal ReID datasets show that STMI achieves state-of-the-art performance and robustness.
Related Work
Prior multi-modal ReID methods model cross-modal interactions but remain vulnerable to background noise. Related segmentation work supplies rich masks, while captioning methods face modality inconsistency and semantic ambiguity.
- Multi-Modal Object Re-Identification: TOP-ReID, MambaPro, and DeMo use cross-attention, Mamba-based dependency modeling, or mixture-of-experts feature aggregation for multi-modal interaction.
- Multi-Modal Object Re-Identification: Methods that model all image tokens remain vulnerable to background noise, which deteriorates feature quality and limits overall performance.
- Semantic Segmentation for Feature Enhancement: Pre-trained segmentation models such as OpenPifPaf, SAM, and SAM2 can generate masks containing rich semantic information for visual tasks.
- Multi-Modal Caption Generation: Figure 2 compares IDEA with the proposed approach on caption clarity, accuracy, and unknown-attribute frequency across training and test sets.
Proposed Method
STMI combines segmentation-guided attention modulation, semantic token reallocation, and cross-modal hypergraph interaction to enhance foreground modeling, preserve informative tokens, and capture higher-order multimodal relationships.
- Segmentation-Guided Feature Modulation: SFM uses segmentation masks to construct token-level foreground interactions, enhancing foreground attention and suppressing background noise with learnable modulation strengths.The class token is treated as foreground, while patch-token labels derive from spatial overlap with the segmentation mask.
- Segmentation-Guided Feature Modulation: During training, mask perturbation flips selected background labels to foreground, addressing boundary ambiguity and incorrect segmentation to improve robustness.Only background tokens are perturbed, with probability p; foreground tokens remain unchanged.
- Semantic Token Reallocation: STR applies cross-attention between learnable modality-specific query tokens and backbone patch tokens to reconstruct compact semantic representations without discarding tokens.A shared global CLIP text feature is appended to the query sequence as a cross-modal semantic prior.
- Cross-Modal Hypergraph Interaction: CHI concatenates semantic tokens from RGB, NIR, and TIR, then dynamically forms hyperedges from intra- and inter-modal similarities exceeding threshold τ.Each hyperedge connects multiple related nodes, enabling high-order associations across modalities.
- Cross-Modal Hypergraph Interaction: Hypergraph convolution aggregates information from nodes to hyperedges and back, while a residual connection preserves independent modality-specific semantics.The resulting multimodal semantic tokens are used for the final fused representation.
Experiments
Experiments evaluate STMI on three public multi-modal ReID datasets and use ablations and feature visualizations to assess its modules and design choices.
- Datasets and Evaluation Protocols: STMI is evaluated on RGBNT201, MSVR310, and RGBNT100 using mAP and CMC at ranks 1, 5, and 10.RGBNT201 contains 4,787 triplets, MSVR310 contains 2,087, and RGBNT100 contains 17,250.
- Ablation Study: 81.2% mAP and 83.4% Rank-1 are achieved when SFM, STR, and CHI are all enabled.The corresponding baseline without these modules achieves 70.3% mAP and 72.1% Rank-1.
- Fusion Strategies: CHI achieves 81.2% mAP, outperforming direct concatenation at 78.1%, MLP fusion at 78.0%, and multi-head self-attention at 78.4%.The comparison is reported as an ablation of different cross-modal fusion strategies.
- Token Configuration: Using four learnable tokens gives the best results, while adding more tokens leads to diminishing returns and potential overfitting.The reported trend links token-count selection to performance and model generalization.
- Feature Visualization: The full STMI model produces the clearest and most structured t-SNE distribution, with more compact identity clusters and better inter-class separation after adding SFM and STR.Figure 4 colors represent different identities.
Conclusion
STMI addresses token loss and weak semantic alignment in multi-modal object ReID through foreground-aware modulation, semantic token extraction, and unified cross-modal interaction.
- SFM enhances foreground regions and suppresses background noise using SAM-generated masks.
- STR extracts compact, informative semantic tokens with learnable queries and cross-attention without hard filtering.
- CHI captures high-order semantic relationships across modalities through a unified hypergraph structure.
- A caption generation strategy fuses multi-modal inputs to produce reliable textual descriptions.
- Extensive experiments on three public multi-modal ReID benchmarks demonstrate state-of-the-art performance, effectiveness, and generalizability.