Source-linked AI summary
Learning Progressive Modality-shared Transformers for Effective Visible-Infrared Person Re-identification
Hu Lu, Xuezhang Zou, Pingping Zhang
TL;DR
VI-ReID requires reliable identity matching across visible and infrared images, but modality gaps and modality-dependent feature unreliability complicate this task. PMT progressively learns modality-invariant features using gray-scale images, MSEL, and DCL, and experiments on SYSU-MM01 and RegDB report performance better than most state-of-the-art methods. The framework also improves performance when applied to CNN-based backbones.
Problem
VI-ReID is challenged by modality gaps, unreliable modality-specific visual features, and large intra-class differences that hinder reliable cross-modality matching.
Method
PMT progressively learns modality-invariant features with gray-scale images as an auxiliary modality, then applies MSEL and DCL to enhance reliability and discrimination.
Results
PMT performs better than most state-of-the-art methods on SYSU-MM01 and RegDB, with additional improvements when its methods are applied to CNN-based backbones.
Takeaways & Limitations
The reported results support PMT as an effective framework for extracting reliable and discriminative modality-invariant features in VI-ReID.
Abstract
from arXiv · showhide
Visible-Infrared Person Re-Identification (VI-ReID) is a challenging retrieval task under complex modality changes. Existing methods usually focus on extracting discriminative visual features while ignoring the reliability and commonality of visual features between different modalities. In this paper, we propose a novel deep learning framework named Progressive Modality-shared Transformer (PMT) for effective VI-ReID. To reduce the negative effect of modality gaps, we first take the gray-scale images as an auxiliary modality and propose a progressive learning strategy. Then, we propose a Modality-Shared Enhancement Loss (MSEL) to guide the model to explore more reliable identity information from modality-shared features. Finally, to cope with the problem of large intra-class differences and small inter-class differences, we propose a Discriminative Center Loss (DCL) combined with the MSEL to further improve the discrimination of reliable features. Extensive experiments on SYSU-MM01 and RegDB datasets show that our proposed framework performs better than most state-of-the-art methods. For model reproduction, we release the source code at https://github.com/hulu88/PMT.
Introduction
VI-ReID must match identities across visible and infrared modalities despite modality gaps, missing information, and clothing changes. PMT addresses these challenges by progressively learning reliable modality-invariant features with auxiliary gray-scale images, MSEL, and DCL.
- Challenges: VI-ReID faces modality gaps, reduced infrared discriminability under lighting changes, and clothing changes over long time spans.These factors make identity-related feature alignment and robust feature extraction difficult.
- Existing methods: Existing dual-stream methods extract modality-specific features before learning shared features but struggle to obtain effective modality-shared representations.Their separate non-shared components help handle inter-modality differences, yet shared-feature extraction remains difficult.
- Existing methods: Overreliance on fine-grained discriminative features can produce wrong matches when posture, viewpoint, modality, lighting, or clothing changes obscure information.The paper illustrates these failure cases in Fig. 1 and notes that recent work therefore emphasizes reducing modality differences.
- Proposed framework: PMT progressively reduces visible-infrared differences by using Transformers, gray-scale images as an auxiliary modality, and an improved hard triplet loss.The strategy is designed to learn modality-independent patterns before further feature enhancement.
- Proposed framework: MSEL enhances modality-shared features, while DCL addresses large intra-class differences and improves the discrimination of modality-invariant features.Together, the losses target feature reliability and discrimination within the PMT framework.
- Results: Experiments on SYSU-MM01 and RegDB show that PMT achieves state-of-the-art performance and performs better than most existing methods.The reported evaluation supports the framework’s effectiveness for VI-ReID.
Related Work
Related VI-ReID work primarily uses modality-aware architectures, auxiliary modalities, and alignment mechanisms to improve cross-modality representations. Transformer research in person ReID has also explored side information, part tokens, and patch-based features.
- Visible-infrared person ReID: VI-ReID methods commonly use dual-stream networks and modality-shared feature mining to represent identities across image modalities.Prior work includes parameter sharing and heterogeneous-center objectives for improving feature representation.
- Visible-infrared person ReID: Auxiliary gray-scale images, cross-modality architecture search, modality confusion, and center aggregation have been used to reduce modality differences.These approaches target either augmented multimodal learning or more effective cross-modality structures and alignment.
- Transformers for person ReID: Transformer-based visible-visible ReID methods learn discriminative representations using side-information embeddings, jigsaw patch modules, and part tokens.These developments provide relevant Transformer design directions for person ReID.
The Proposed Method
PMT progressively learns modality-independent and modality-shared features using grayscale, infrared, and visible images. MSEL suppresses modality-specific unreliability, while DCL improves discrimination by increasing intra-class compactness and inter-class separability.
- Progressive learning strategy: PMT first uses grayscale and infrared images with a weight-shared Transformer to learn modality-independent discriminative patterns.The first stage applies identity and progressive hard triplet supervision while sampling positives and negatives within each modality.
- Progressive learning strategy: The second stage replaces grayscale inputs with visible images and uses cross-modality learning to retain modality-specific information for finer-grained feature learning.The framework uses visible and infrared images with LGlobal, which no longer distinguishes modalities when selecting positive and negative samples.
- Modality-Shared Enhancement Loss: MSEL suppresses features appearing in only one modality and enhances reliable modality-invariant features by penalizing differences between intra-modality and cross-modality distances.The loss captures anomalies when modality-specific discriminative features increase the distance difference between Dintra and Dcross.
- Discriminative Center Loss: DCL computes identity feature centers and a dynamic negative margin to improve the discrimination of reliable modality-shared features.Its dynamic sampling focuses on relatively difficult examples and uses modality-specific features to capture additional relationships.
- Discriminative Center Loss: Minimizing DCL improves intra-class compactness and inter-class separability in the learned features.The paper describes these geometric effects as the principal benefits of LDCL.
- Training objectives: The hybrid training objective uses LID and LIntra in stage one, then combines LGlobal, MSEL, and DCL in stage two.λ1 and λ2 balance the MSEL and DCL terms in the second-stage objective.
Experimental Setting
The experiments use two public VI-ReID datasets, standard retrieval metrics, and a specified Transformer-based implementation protocol.
- SYSU-MM01 contains 491 identities, with 395 for training and 96 for testing under the described query-gallery protocol.
- RegDB contains 412 identities, split into 206 for training and 206 for testing across ten random evaluation splits.
- Performance is evaluated using CMC, mAP, and mINP.
- The method uses an ImageNet-pretrained ViT-B/16 backbone, 256×128 inputs, augmentation, and batches containing eight identities.
Ablation Studies
Ablations show that progressive learning, MSEL, and DCL each improve VI-ReID performance, with gains extending to CNN backbones and visual evidence of more reliable, discriminative features.
- Image modality and progressive learning: 5.16% Rank-1, 5.04% mAP, and 5.74% mINP improvements over “Baseline (RGB)” demonstrate the progressive learning strategy’s benefit.
- MSEL and DCL: MSEL with Euclidean distance improves “Base (PL)” by 3.44% Rank-1, 2.21% mAP, and 1.91% mINP.
- MSEL and DCL: Dynamic DCL selection improves “Base (PL)” by 2.38% Rank-1, 3.50% mAP, and 4.89% mINP, while combining MSEL adds further gains.The combined improvement is 2.42% Rank-1, 3.51% mAP, and 4.46% mINP.
- CNN-based backbones: With CNN-based backbones, the full “MSEL+PL+DCL” modules yield gains of 8.90% Rank-1, 7.75% mAP, and 8.34% mINP.Adding MSEL alone produces gains of 3.97% Rank-1, 3.16% mAP, and 2.86% mINP.
- Parameter sensitivity: LMSEL is not sensitive to trade-off parameters, while LDCL remains stable within a certain parameter range.
- Visualization and similarity analysis: MSEL focuses attention on more discriminative regions and increases positive cross-modality similarity, while DCL expands separation between positive and negative pairs.
Comparison with State-of-the-Arts
PMT outperforms competing methods across SYSU-MM01 and RegDB, with particularly strong gains in Rank-1 and mAP. Its results indicate robust performance across datasets and query directions.
- SYSU-MM01: On SYSU-MM01, PMT exceeds weight-shared methods by at least 12.24% in Rank-1 and 11.09% in mAP under all-search.
- SYSU-MM01: Under SYSU-MM01 indoor-search, PMT performs better in Rank-1, mAP, and mINP, while remaining comparable to CM-NAS in Rank-10 and Rank-20.
- RegDB: On RegDB, PMT achieves 84.83% Rank-1 and 76.55% mAP in the Visible to Thermal mode, a 4% gain over other best methods.
- RegDB: PMT also shows performance advantages in the more challenging Thermal to Visible mode, indicating robustness across query patterns and datasets.
Conclusion
The paper concludes that PMT improves VI-ReID by exploring reliable modality-invariant features through progressive learning, MSEL, and DCL. The methods also generalize to CNN-based backbones, while future work will examine more effective Transformer structures.
- Conclusion: PMT improves VI-ReID by exploring reliable modality-invariant features with gray-scale-assisted progressive learning.
- Conclusion: MSEL and DCL extract more reliable and discriminative features, producing stronger performance and robustness.
- Conclusion: Applying the proposed methods to CNN-based backbones also brings significant performance improvements, supporting their generalization.
- Conclusion: Future work will explore more effective Transformer structures to further improve feature representation ability.