Source-linked AI summary
Cross-modality Person re-identification with Shared-Specific Feature Transfer
Yan Lu, Yue Wu, Bin Liu, Tianzhu Zhang, Baopu Li, Qi Chu, Nenghai Yu
TL;DR
Conventional cross-modality person re-identification focuses on shared representations, but modality discrepancy and discarded modality-specific information remain challenges. The paper proposes cm-SSFT to transfer shared and specific features among and across modalities, with experiments validating superior performance and component effectiveness.
Problem
Cross-modality person re-identification faces modality discrepancy, while learning only shared representations discards modality-specific information.
Method
cm-SSFT models intra- and inter-modality affinities and uses near-neighbor propagation to transfer shared and specific features among and across modalities.
Results
Extensive experiments validate the superior performance of cm-SSFT and the effectiveness of each algorithm component.
Takeaways & Limitations
Using modality-shared and modality-specific features enables cm-SSFT to compensate for lacking specific information and enhance overall discriminative representation.
Takeaways & Limitations
The method's testing procedure is described for settings with one query image and gallery images when auxiliary query samples are unavailable or few.
Abstract
from arXiv · showhide
Cross-modality person re-identification (cm-ReID) is a challenging but key technology for intelligent video analysis. Existing works mainly focus on learning common representation by embedding different modalities into a same feature space. However, only learning the common characteristics means great information loss, lowering the upper bound of feature distinctiveness. In this paper, we tackle the above limitation by proposing a novel cross-modality shared-specific feature transfer algorithm (termed cm-SSFT) to explore the potential of both the modality-shared information and the modality-specific characteristics to boost the re-identification performance. We model the affinities of different modality samples according to the shared features and then transfer both shared and specific features among and across modalities. We also propose a complementary feature learning strategy including modality adaption, project adversarial learning and reconstruction enhancement to learn discriminative and complementary shared and specific features of each modality, respectively. The entire cm-SSFT algorithm can be trained in an end-to-end manner. We conducted comprehensive experiments to validate the superiority of the overall algorithm and the effectiveness of each component. The proposed algorithm significantly outperforms state-of-the-arts by 22.5% and 19.3% mAP on the two mainstream benchmark datasets SYSU-MM01 and RegDB, respectively.
1. Introduction
Cross-modality person re-identification addresses modality discrepancies between RGB and infrared images, where shared-only representations discard useful modality-specific cues. cm-SSFT transfers shared and specific information using near-neighbor affinities to improve representation quality.
- RGB-based ReID performance degrades in dark environments, motivating infrared cameras as complementary sensors.
- cm-ReID is difficult because RGB and infrared cameras use intrinsically distinct imaging processes, causing modality discrepancies and missing cues such as color.
- Shared feature learning embeds modalities into one space but eliminates modality-specific information, limiting feature discrimination.
- cm-SSFT models intra- and inter-modality near-neighbor affinities, propagating shared and specific information among samples.
- 22.5% and 19.3% mAP gains over state-of-the-art methods are reported on SYSU-MM01 and RegDB, respectively.
- The method combines end-to-end shared-specific transfer with complementary learning for discriminative shared and specific features.
2. Related Work
Related work covers person ReID, cross-modality matching, and cross-modality person ReID. Existing RGB-IR approaches primarily learn shared features or use generative compensation, while cm-SSFT transfers shared and specific features through affinity modeling.
- Person ReID: Person ReID searches a large gallery for images matching a query person, using deep feature learning, body-part modeling, or metric learning.
- Person ReID: Earlier ReID methods process samples independently, whereas self-attention and graph-based methods model relationships between sample pairs.
- Cross-modality matching: Cross-modality matching includes heterogeneous tasks such as face recognition, text-to-image retrieval, and tracking across distinct modalities.
- Cross-modality person ReID: Cross-modality person ReID matches queries from one modality against a gallery from another, including RGB-IR, RGB-Depth, and text-image settings.
- Cross-modality person ReID: RGB-IR methods include shared embedding, adversarial training, and GAN-based modality translation, but shared-only methods ignore specific features.
- Cross-modality person ReID: GAN-based compensation faces ambiguity because multiple generated targets can be reasonable for person ReID.
3. Cross-Modality Shared-Specific Feature Transfer
cm-SSFT transfers shared and modality-specific information using affinity-guided propagation among intra- and inter-modality neighbors. Complementary learning purifies shared features, separates specific features, and constrains information loss through adversarial and reconstruction objectives.
- Two-stream feature extractor: The two-stream extractor produces shared and modality-specific features for each RGB or infrared input.The streams separate at shallow convolutional layers, and classification plus modality-appropriate triplet losses promote discriminability.
- Shared-Specific Transfer Network: Features are padded into [RGB-specific; shared; Infrared-specific] vectors so both modalities share a unified representation.Zero padding marks the absent specific features of the opposite modality before transfer.
- Shared and specific information propagation: Affinity-guided propagation transfers shared and specific information among near neighbors, compensating for missing modality-specific features and enhancing shared features.The propagated representation is fused by a learnable nonlinear transformation and optimized with classification and both cross- and single-modality triplet losses.
- Affinity modeling: SSTN models intra-modality affinity from specific features and inter-modality affinity from shared features, retaining top-k neighbors for propagation.The resulting affinity matrix represents relationships within and across modalities and guides feature transfer.
- Complementary feature learning: Complementary learning purifies shared features with modality adversarial learning, separates specific features through project adversarial learning, and constrains information loss with reconstruction.The combined strategy guides shared and specific features toward self-discriminative and mutually complementary representations.
- Optimization: The complete algorithm is trained end-to-end with adversarial min-max games and combined classification and triplet-based feature-learning losses.The loss design treats classification and triplet objectives as equally important.
4. Experiments
Experiments evaluate cm-SSFT on RGB-IR person ReID benchmarks, compare it with state-of-the-art methods, and analyze its components, augmentation choices, reconstruction behavior, and query requirements.
- Experimental setup: The experiments use SYSU-MM01 and RegDB under standard cross-modality protocols with CMC and mAP evaluation.SYSU-MM01 includes indoor-search and all-search modes; queries and galleries are from different modalities.
- Comparison with state-of-the-art methods: 22.5% higher mAP and 19.2% higher Rank-1 than AlignGAN are reported on SYSU-MM01 all-search single-shot evaluation.The method also achieves the best results in indoor-search mode and its mAP drops only 1.2% from single-shot to multi-shot evaluation.
- Comparison with state-of-the-art methods: 19.3% higher mAP and 14.4% higher Rank-1 are reported for Visible-to-Thermal RegDB evaluation, while Thermal-to-Visible gains are 18.3% mAP and 14.7% Rank-1.The reported advantages occur in both cross-modality directions.
- Ablation study: Specific feature streams add 5.7% Rank-1, while separating streams at shallow layers adds 4.2% gains for more discriminative features.The specific streams back-propagate modality-specific gradients to low-level feature maps.
- Ablation study: The complete complementary learning scheme improves feature-extractor and SSTN performance by 8% and 12%, respectively.The evaluated modules are modality adaptation, project adversarial learning, and reconstruction enhancement.
- Ablation study: Feature transfer contributes 12.0% Rank-1 and 13.5% mAP improvements overall, with specific-feature transfer adding further gains beyond shared-feature transfer alone.Shared-only transfer yields 5.5% Rank-1 and 6.7% mAP improvements; transferring both feature types adds 6.5% and 6.8% increments over that setting.
5. Conclusion
The proposed cross-modality shared-specific feature transfer algorithm utilizes modality-specific features alongside shared features for person re-identification, with complementary learning and extensive experiments supporting its effectiveness.
- cm-SSFT utilizes specific features ignored by conventional shared feature learning.
- The algorithm propagates information among and across modalities to compensate for lacking specific information and enhance discriminative features.
- A complementary learning strategy learns self-discriminate and complementary features, while experiments validate the algorithm and its components.