Source-linked AI summary
Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification
Seokeon Choi, Sumin Lee, Youngeun Kim, Taekyung Kim, Changick Kim
TL;DR
VI-ReID must handle intra-modality variation and additional cross-modality discrepancy when matching visible and infrared images. Hi-CMD disentangles identity-discriminative from pose and illumination factors using ID-PIG and HFL, and it outperforms state-of-the-art methods on two VI-ReID datasets.
Problem
VI-ReID must match visible and infrared pedestrians despite cross-modality discrepancy in addition to intra-modality variation, especially when visible cameras capture poor appearance information under low illumination.
Method
Hi-CMD uses ID-PIG to generate cross-modality images with changed pose and illumination while preserving identity, and HFL to extract common ID-discriminative factors.
Results
Hi-CMD outperforms state-of-the-art VI-ReID methods on the RegDB and SYSU-MM01 datasets.
Takeaways & Limitations
The method uses ID-discriminative features for cross-modality matching while providing generated images for analyzing matching results and alleviating insufficient data.
Takeaways & Limitations
Most existing style-transfer methods do not address the lack of cross-view paired training data, limiting exploitation of view-invariant representations.
Abstract
from arXiv · showhide
Visible-infrared person re-identification (VI-ReID) is an important task in night-time surveillance applications, since visible cameras are difficult to capture valid appearance information under poor illumination conditions. Compared to traditional person re-identification that handles only the intra-modality discrepancy, VI-ReID suffers from additional cross-modality discrepancy caused by different types of imaging systems. To reduce both intra- and cross-modality discrepancies, we propose a Hierarchical Cross-Modality Disentanglement (Hi-CMD) method, which automatically disentangles ID-discriminative factors and ID-excluded factors from visible-thermal images. We only use ID-discriminative factors for robust cross-modality matching without ID-excluded factors such as pose or illumination. To implement our approach, we introduce an ID-preserving person image generation network and a hierarchical feature learning module. Our generation network learns the disentangled representation by generating a new cross-modality image with different poses and illuminations while preserving a person's identity. At the same time, the feature learning module enables our model to explicitly extract the common ID-discriminative characteristic between visible-infrared images. Extensive experimental results demonstrate that our method outperforms the state-of-the-art methods on two VI-ReID datasets. The source code is available at: https://github.com/bismex/HiCMD.
1. Introduction
VI-ReID must address both intra-modality and cross-modality discrepancies when matching visible and infrared pedestrians. Hi-CMD disentangles identity-discriminative information from pose and illumination, then uses the former for cross-modality matching.
- VI-ReID matches pedestrians across visible and infrared cameras with different spectra, where poor illumination limits visible-camera appearance capture.
- Cross-modality discrepancy adds to the traditional intra-modality problem, making intra-class distances potentially larger than inter-class distances.
- Hi-CMD hierarchically separates ID-discriminative factors, such as body shape and clothes pattern, from ID-excluded pose and illumination factors.
- The HFL module extracts common ID-discriminative factors between RGB-IR images, and the resulting feature supports cross-modality image retrieval.
- The ID-PIG network changes pose and illumination while preserving a person's identity, using image-level constraints without pose-related supervision.
- Hi-CMD outperforms state-of-the-art methods on two VI-ReID datasets, according to extensive experimental results.
2. Related Work
Prior VI-ReID work uses feature learning, metric learning, adversarial learning, and image generation, but existing generation methods address only selected structural or style factors. Hi-CMD instead targets intra- and cross-modality discrepancies jointly through hierarchical disentanglement.
- VI-ReID methods include feature-embedding networks, adversarial learning, metric learning, and approaches that reduce distribution divergence.
- ReID image-generation methods commonly perform pose transfer or style transfer, addressing pose coverage or camera-domain differences respectively.
- Most style-transfer methods do not address the lack of cross-view paired training data, so view-invariant representations are hardly exploited.
- Existing image-generation methods handle only structural information or image style, whereas VI-ReID requires simultaneous consideration of intra- and cross-modality characteristics.
- Disentangling common identity information from remaining attributes is particularly challenging in VI-ReID because cross-modality and intra-modality discrepancies coexist.
- Hi-CMD addresses both discrepancies through hierarchical disentanglement of ID-discriminative and ID-excluding factors without pose supervision.
3. Proposed Method
Hi-CMD addresses VI-ReID’s simultaneous intra- and cross-modality discrepancies by hierarchically separating identity-discriminative information from pose, illumination, and other identity-excluded factors. It combines ID-PIG image generation with HFL feature learning and reconstruction, distribution, and adversarial objectives.
- Framework overview: Hi-CMD hierarchically disentangles ID-discriminative factors from ID-excluded pose and illumination factors in cross-modality images.The method uses image-level constraints to address both intra- and cross-modality discrepancies.
- Hierarchical representation: The prototype encoder extracts identity-related appearance such as body shape and clothes pattern, while attribute codes represent style, illumination, and pose.The attribute representation is further divided into style, illumination, and pose codes.
- Disentangling ID-excluded factors: ID-PIG synthesizes cross-modality images by swapping identity-excluded factors between same-identity images while preserving shared identity characteristics.Cross-modality reconstruction uses an L1 loss, encouraging the generator to create sharp images and learn the identity-excluded representations.
- Reconstruction loss: The reconstruction objective combines cross-modality, same-modality, cycle, and code reconstruction losses to preserve identity-discriminative and identity-excluded information.Same-modality reconstruction regularizes the generation network, while the overall loss weights the four reconstruction terms.
- KL divergence loss: KL divergence regularizes identity-excluded codes toward a Gaussian prior, enabling continuous changes in the latent space.The distribution constraint is applied to cross-modality and intra-modality characteristics.
- Hierarchical feature learning: HFL shares the prototype and attribute encoders with ID-PIG to extract common RGB-IR identity factors and improve image generation quality.The resulting identity-discriminative feature is used for cross-modality retrieval, and the network is trained end to end.
4. Experiments
Experiments evaluate Hi-CMD on RegDB and SYSU-MM01 using standard retrieval metrics, comparisons, ablations, and qualitative analyses. Results show strong state-of-the-art performance and support the proposed disentanglement and image-generation components.
- Datasets and Settings: Experiments use RegDB and SYSU-MM01, with CMC rank-k identification rates and mAP as evaluation metrics.RegDB evaluation repeats 10 random splits for statistically stable results.
- Comparison with State-of-the-art Methods: 70.93% rank-1 and 66.04% mAP on RegDB, plus 34.94% rank-1 and 35.94% mAP on SYSU-MM01, significantly outperform state-of-the-art VI-ReID methods.The comparison covers feature learning, ranking-loss, metric-learning, distribution-divergence, and image-generation approaches.
- Further Evaluations and Analysis: Removing any ID-PIG loss produces blurriness or color shifts, whereas the complete model generates sharper and more realistic images across modalities.The ablation removes reconstruction, disentanglement, or adversarial losses while keeping the network structure and training strategy fixed.
- Further Evaluations and Analysis: HFL with alternate sampling minimizes intra-class distance, maximizes inter-class distance, and significantly improves performance over the original image set.Diverse code combinations also outperform learning style-attribute or prototype codes alone.
- Person Attribute Exploration: Swapping ID-excluded factors changes pose and illumination while preserving clothing and patterns; swapping ID-discriminative factors changes clothing while preserving pose and illumination.These visualizations support separation of identity-related and identity-excluded factors.
- Person Attribute Exploration: ID-PIG independently transforms pose without pose-estimation supervision and smoothly interpolates pose and illumination in latent space.The interpolation results indicate synthesis of unseen poses or illuminations.
5. Conclusion
The conclusion presents Hi-CMD as a hierarchical disentanglement method for reducing cross- and intra-modality discrepancies in VI-ReID. Quantitative and qualitative evaluations report superiority over state-of-the-art methods.
- Conclusion: Hi-CMD disentangles ID-discriminative and ID-excluded factors to reduce cross- and intra-modality discrepancies simultaneously.The method uses an ID-preserving person image generation network to analyze matching and address insufficient data.