Source-linked AI summary
Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification
Menglin Wang, Xiaojin Gong
TL;DR
Unsupervised visible-infrared person re-ID needs reliable cross-modality association despite noisy clusters and modality-dependent similarity. The paper unifies modality-unified prototypical contrast with prototype-guided self-distillation, and reports that the resulting framework is effective on standard VI-ReID benchmarks.
Problem
Unsupervised visible-infrared re-ID must estimate reliable intra- and cross-modality associations despite significant modality discrepancy, noisy clusters, and suboptimal hard matching.
Method
The framework jointly optimizes intra- and cross-modality prototype similarities and uses each instance’s self-prototype as an online teacher for soft similarity distillation.
Results
Extensive comparisons and ablations on standard VI-ReID benchmarks validate the effectiveness of the proposed method and its components.
Takeaways & Limitations
The unified prototype framework provides a simple approach to improving association estimation and learning modality-invariant, identity-discriminative representations.
Takeaways & Limitations
The method is evaluated in the unsupervised setting with separate unlabeled visible and infrared datasets and a shared-backbone baseline with modality-specific initial blocks.
Abstract
from arXiv · showhide
Estimating reliable cross-modality association is crucial to unsupervised visible-infrared person re-ID. While optimal transport is shown to be a practical solution for cross-modality association, it suffers from the rigidness of hard label assignment without considering the impact of cluster noise. Moreover, enforcing only cross-modality contrast is also suboptimal, as it fails to jointly optimize the similarity relation within and across modality. In this paper, we propose a novel framework for cross-modality learning by well exploitation of prototypes: First, instead of contrasting with cross-modality prototypes, we show that modality-unified prototypical contrast facilitates better modality invariance by jointly and simultaneously optimizing similarity relation within and across-modality. Taking self-prototype as a steady teacher, we further refine the instance-prototype online relation through prototype-guided self-distillation. The two components are optimized in a unified framework, leading to a simple yet effective model. On standard VI-ReID benchmarks, we perform extensive comparison and analysis, validating the effectiveness of our proposed method. Code is available at: https://github.com/Terminator8758/PoSeD.
1 Introduction
Unsupervised visible-infrared re-ID must estimate reliable intra- and cross-modality associations despite modality discrepancy and noisy clusters. The paper addresses these issues with modality-unified prototypical contrast and prototype-guided self-distillation in a unified framework.
- Motivation: Unsupervised visible-infrared re-ID estimates associations between daytime visible and nighttime or low-light infrared person images without costly annotation.
- Challenges: Noisy intra-modality clusters make hard optimal-transport matching suboptimal, while evolving prototypes can drift from offline associations.The paper motivates soft association based on semantic consistency and online prototype similarity dynamics.
- Challenges: Cross-modality contrast alone ignores the lower cross-modality similarity relative to intra-modality similarity, potentially weakening generalization under severe modality shift.
- Method: Modality-unified contrastive optimization jointly compares instances with intra- and cross-modality prototypes to reduce the similarity gap while preserving identity discrimination.This optimization uses intra-modality similarity as a reference for cross-modality semantic relations.
- Method: Prototype-guided self-distillation uses each instance’s centroid prototype as a steady teacher, refining online instance-to-prototype similarity and complementing offline association.Centroid prototypes incorporate intra-cluster variation through online updates, providing an instance-adaptive soft distillation target.
- Contributions: The unified framework combines offline matching with online distillation to improve cross-modality association estimation and learn modality-invariant, identity-discriminative representations.
2 Related Work
Related work in unsupervised visible-infrared re-ID addresses intra-modality clustering, cross-modality association, and representation learning. Self-distillation and prototypes provide the conceptual foundations for the paper’s approach.
- Unsupervised visible-infrared re-ID: Unsupervised visible-infrared re-ID methods commonly combine iterative intra-modality clustering with cross-modality association or representation-learning strategies.
- Cross-modality learning: Cross-modality association methods include optimal transport, label smoothing and propagation, matched-pair ranking, and related cluster-association strategies.
- Self-distillation: Self-distillation typically uses predictions from one augmented view or network as supervision for another view of the same unlabeled instance.
- Prototype learning: Prototype-based unsupervised learning uses cluster representations to guide discriminative representation learning through prototypical contrast.Prior designs capture camera-view variation, diversity, or improved prototype updating mechanisms.
3 Methodology
The framework learns modality-invariant representations through intra-modality prototypes, optimal-transport cross-modality association, modality-unified contrast, and prototype-guided self-distillation.
- 3.2 The UnVI-reID Baseline: The framework first clusters visible and infrared images separately, initializes dual prototype memories, and trains with intra-modality prototypical contrast.Centroid prototypes capture cluster means, while hard prototypes use the farthest positive instance and are updated online.
- 3.2.2 Cross-modality learning: Optimal transport estimates visible–infrared cluster correspondence from cosine similarities between modality-specific cluster prototypes.The resulting correspondence supports cross-modality contrastive learning against matched prototypes.
- 3.3 Modality-unified Prototypical Contrast: Modality-unified prototypical contrast assigns shared pseudo-labels and jointly optimizes similarity to intra- and cross-modality prototypes.Unlike cross-modality-only contrast, the unified loss explicitly addresses the modality-induced similarity gap.
- 3.4 Prototype-guided Self-distillation: Prototype-guided self-distillation uses online prototype similarity as a soft teacher to refine noisy and drifting instance–prototype relations.The teacher combines instance-to-prototype and prototype-to-prototype similarity, while separate modality normalization avoids intra-modality dominance.
- 3.4 Prototype-guided Self-distillation: The method repurposes online-updated centroid prototypes as steady, informative teachers for distilling instance-to-prototype similarity distributions.This prototype-based distillation perspective extends the use of prototypes beyond ordinary assistance in unsupervised learning.
4 Experiment
Experiments on SYSU-MM01, RegDB, and LLCM show strong unsupervised VI-ReID performance, while ablations and analyses support the complementary roles of modality-unified contrast and prototype-guided self-distillation.
- Comparison with State-of-the-Art Methods: 0.9% Rank-1 improvement on SYSU-MM01 and 4.4% Rank-1 plus 4.9% mAP improvements on RegDB Visible-to-Thermal search over prior unsupervised methods.The method consistently outperforms state-of-the-art unsupervised methods on SYSU-MM01 and RegDB.
- Comparison with State-of-the-Art Methods: 7.3% and 5.9% Rank-1 gains over PGM on LLCM for Visible-to-Infrared and Infrared-to-Visible evaluation, respectively.The method also surpasses other unsupervised VI-ReID methods on LLCM.
- Ablation Study: Prototype-guided online self-distillation improves Rank-1 (All Search) accuracy by 6.4% over M2, while adding offline modality-unified contrast further increases Rank-1 by 0.6% on both SYSU-MM01 and RegDB.The ablation identifies complementary contributions from online distillation and offline association under the same modality-unified framework.
- Further Analysis on Prototype-guided Self-distillation: Teacher temperature achieves its best performance at 𝛾=0.05, while association recall rises during training and union association consistently exceeds offline-association recall.After around 30 epochs, association recall becomes stable; offline and online signals become more consistent as training converges.
- Further Analysis on Prototype-guided Self-distillation: Separate intra- and cross-modality distillation decreases All Search performance, whereas replacing instance-adaptive fusion with prototype-to-prototype similarity causes a noticeable drop.These comparisons support unified optimization and instance-adaptive similarity fusion for handling modality discrepancy and intra-cluster noise.
- Visualization: T-SNE visualizations show tighter intra-class feature distributions for the proposed method than for the baseline across visible and infrared images.The comparison describes baseline identity features as dispersed into sub-clusters, while the proposed method associates intra-class features more compactly.
5 Conclusion
The proposed framework jointly optimizes modality-unified contrastive similarity and prototype-guided self-distillation for unsupervised visible-infrared person re-ID. Extensive experiments on benchmark VI-ReID datasets validate the method and its components.
- The unified framework jointly optimizes contrastive similarity across modalities and uses prototype-guided self-distillation to address modality discrepancy.Centroid prototypes serve as steady online teachers, complementing offline association with stronger supervision for modality-unified contrastive learning.