PaperlayerView exact revision on arXiv

Source-linked AI summary

STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification

Xingguo Xu, Zhanyu Liu, Weixiang Zhou, Yuansheng Gao, Junjie Cao, Yuhao Wang, Jixiang Luo, Dell Zhang

arXiv:2603.00695v1cs.CV

TL;DR

Effective representation learning across modalities remains a challenge in multi-modal object ReID. STMI combines segmentation-guided feature modulation, semantic token reallocation, and cross-modal hypergraph interaction; in the reported CHI ablation, its full model achieves the best result at 81.2% mAP.

  • Problem

    Effective multi-modal representation learning remains a fundamental challenge in multi-modal object ReID.

    Source: Introduction · p. 1

  • Method

    STMI combines segmentation-guided feature modulation, learnable-query semantic token reallocation, and unified-hypergraph interaction to model cross-modal semantic relationships.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Results

    81.2% mAP: the full STMI model outperforms the reported CHI alternatives, whose results range from 78.0% to 78.4% mAP.

    Source: Ablation Studies · p. 7

  • Takeaways & Limitations

    STMI is presented as addressing token loss and weak semantic alignment in multi-modal object ReID.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Multi-modal object Re-Identification (ReID) aims to exploit complementary information from different modalities to retrieve specific objects. However, existing methods often rely on hard token filtering or simple fusion strategies, which can lead to the loss of discriminative cues and increased background interference. To address these challenges, we propose STMI, a novel multi-modal learning framework consisting of three key components: (1) Segmentation-Guided Feature Modulation (SFM) module leverages SAM-generated masks to enhance foreground representations and suppress background noise through learnable attention modulation; (2) Semantic Token Reallocation (STR) module employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens; (3) Cross-Modal Hypergraph Interaction (CHI) module constructs a unified hypergraph across modalities to capture high-order semantic relationships. Extensive experiments on public benchmarks (i.e., RGBNT201, RGBNT100, and MSVR310) demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.

Introduction

STMI addresses information loss from hard token filtering and weak modeling of cross-modal relationships through segmentation-guided modulation, semantic token reallocation, and hypergraph interaction. The introduction reports state-of-the-art performance on three public multi-modal ReID datasets.

Introduction

  • Existing methods risk discarding discriminative details through hard token filtering and underuse cross-modal complementary information by omitting high-order semantic relationships.These limitations are especially consequential in scenes with background clutter or occlusions.Source: Introduction · p. 1, Figure 1 · p. 1
  • STMI combines segmentation priors, semantic token reconstruction, and high-order relational modeling for multi-modal object ReID.The framework comprises SFM, STR, and CHI modules.Source: Introduction · p. 2
  • SFM uses SAM-generated foreground masks and learnable modulation parameters to emphasize foreground tokens while suppressing background noise without discarding tokens.The parameters adaptively reweight token features under segmentation guidance.Source: Introduction · p. 2
  • STR uses learnable query tokens and cross-attention to produce compact semantic representations while preserving fine-grained visual details instead of hard-filtering patches.Source: Introduction · p. 2
  • CHI treats RGB, NIR, and TIR semantic tokens as nodes in a unified hypergraph, using similarity-based cross-modal hyperedges to model high-order relationships.Source: Introduction · p. 2
  • The introduction reports that STMI achieves state-of-the-art performance and effectiveness and robustness on three public multi-modal ReID datasets.Source: Introduction · p. 2

Related Work

Prior multi-modal ReID methods model cross-modal interactions or select salient tokens, but background noise and hard pruning can impair features; segmentation masks have been used in ReID without fine-grained token-level modulation.

Multi-Modal Object Re-Identification

Semantic Segmentation for Feature Enhancement

  • Prior ReID methods use segmentation masks as auxiliary inputs, whereas STMI embeds segmentation into attention for fine-grained, token-level feature modulation across modalities.Mask-Guided uses binary masks for body and background attention, and MaskReID combines segmentation results with RGB images.Source: Semantic Segmentation for Feature… · p. 2

Proposed Method

STMI combines multi-modal caption generation with three feature-modeling modules—SFM, STR, and CHI—and supervises global and fused semantic representations with classification and triplet losses.

Multi-Modal Caption Generation

  • Caption generation combines same-identity RGB, NIR, and TIR images for holistic MLLM input with confidence-aware attribute extraction and LLM selection to produce descriptions.Attribute–value–confidence triplets from individual modalities and the concatenated image guide selection of reliable values.Source: Multi-Modal Caption Generation · p. 3

Segmentation-Guided Feature Modulation

  • SFM derives foreground labels for patch tokens from segmentation-mask overlap and modulates self-attention to enhance foreground interactions while suppressing background interactions.The class token is treated as foreground, and learnable parameters control the enhancement and suppression strengths.Source: Segmentation-Guided Feature Modulation · pp. 3–4
  • SFM randomly flips some background-token labels to foreground during training, addressing mask ambiguity and incorrect segmentation while reducing overfitting to mask guidance.Foreground labels remain unchanged during perturbation.Source: Segmentation-Guided Feature Modulation · p. 4

Semantic Token Reallocation

  • STR uses modality-specific learnable semantic queries and a shared CLIP text feature to query visual patch tokens through cross-attention, producing representations for subsequent semantic alignment and interaction.The visual patch tokens serve as both keys and values, and an FFN follows cross-attention.Source: Semantic Token Reallocation · pp. 4–5

Cross-Modal Hypergraph Interaction

  • CHI forms a cross-modal hypergraph from semantic tokens, propagates information through node-to-hyperedge and hyperedge-to-node convolution with residuals, then uses global image features to query the fused tokens.Hyperedges connect nodes whose semantic similarity meets a threshold; the cross-attention output is fed back into the global representation.Source: Cross-Modal Hypergraph Interaction · p. 5

Objective Function

  • The objective applies label-smoothed cross-entropy and triplet losses to the concatenated global feature, fused semantic feature, and global text feature.The passage identifies these as jointly supervised representations.Source: Objective Function · p. 5

Experiments

STMI achieves the strongest reported mAP results across the person and vehicle benchmarks. Ablations and feature visualizations support the contributions of its modules and configuration choices.

Datasets and Evaluation Protocols

Implementation Details

  • The experiments use pre-trained CLIP encoders and fine-tune learnable parameters with Adam, decaying the learning rate from 3.5×10−6 to 3.5×10−7.Training uses an NVIDIA A800, random flipping, cropping, and erasing; batch sizes are 72 for RGBNT201, 64 for MSVR310, and 128 for RGBNT100.Source: Implementation Details · p. 6

Comparison with State-of-the-Art Methods

  • 81.2% mAP makes STMI best on RGBNT201, beating IDEA by +1.0%; it reaches 89.1% on RGBNT100 and 64.8% on MSVR310, +17.8% over IDEA.On RGBNT100, STMI also outperforms IDEA (87.2%) and DeMo (86.2%); on RGBNT201, it exceeds TOP-ReID (72.3%) and EDITOR (66.5%) by +8.9% and +14.7%, respectively.Source: Comparison with State-of-the-Art Methods · p. 6

Ablation Studies

  • On RGBNT201, SFM raises mAP from 70.3% to 76.1%, STR raises it to 78.1%, and full STMI with CHI reaches 81.2%.The baseline's Rank-1 accuracy is 72.1%.Source: Ablation Studies · p. 6, Table 3 · p. 6
  • CHI achieves 81.2% mAP, versus 78.1% with concatenation, 78.0% with an MLP, and 78.4% with multi-head self-attention.The comparison tests alternative cross-modal fusion strategies in the STMI framework.Source: Table 4 · p. 6, Ablation Studies · p. 7
  • Layer-wise SFM modulation with shared head parameters achieves 81.2% mAP, compared with 77.2% for sharing parameters across all layers and 77.1% for head-wise parameters.Modulating only early or late layers is also reported as suboptimal.Source: Ablation Studies · p. 7, Table 5 · p. 7
  • A small amount of randomness improves generalization, higher levels degrade performance through excessive noise, and four learnable tokens perform best.Adding more than four tokens yields diminishing returns and may overfit.Source: Ablation Studies · p. 7, Figure 5 · p. 7

Visualization

  • t-SNE visualizations show more compact identity clusters and better separation after SFM, further improvement with STR, and the clearest structure with full STMI.Different colors represent different identities.Source: Visualization · p. 7, Figure 4 · p. 7

Conclusion

STMI addresses token loss and weak semantic alignment in multi-modal object ReID with segmentation-guided modulation, semantic token reallocation, and cross-modal hypergraph interaction. Experiments on three public benchmarks demonstrate state-of-the-art performance.

  • STMI achieves state-of-the-art performance on three public multi-modal ReID benchmarks, supporting its effectiveness and generalizability.Source: Conclusion · p. 7
  • SFM uses SAM-generated masks to enhance foreground regions and suppress background noise, while STR uses learnable queries and cross-attention to extract compact semantic tokens without hard-filtering information loss.Source: Conclusion · p. 7
  • CHI captures high-order semantic relationships across modalities through a unified hypergraph structure.Source: Conclusion · p. 7
  • A caption-generation strategy fuses multi-modal inputs to produce reliable textual descriptions.Source: Conclusion · p. 7

Related papers

Loading 2603.00695v1…