Source-linked AI summary
Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible-Infrared Person Re-identification
Yukang Zhang, Hanzi Wang
TL;DR
VIReID must overcome large VIS–IR modality gaps with limited training data and insufficient low-light evaluation. The paper introduces DEEN for diverse embedding generation and multistage feature aggregation, alongside the LLCM dataset. DEEN outperforms other state-of-the-art methods on SYSU-MM01, RegDB, and LLCM.
Problem
VIReID faces large VIS–IR modality gaps, limited training samples, and insufficient existing evaluation data for drastic illumination changes or low illumination.
Method
DEEN augments embeddings through DEE and CPM, aggregates multistage features with MFA, and is trained in an end-to-end framework.
Results
DEEN outperforms several other state-of-the-art methods on the SYSU-MM01, RegDB, and LLCM datasets.
Takeaways & Limitations
The paper provides a diverse embedding approach and the LLCM low-light cross-modality dataset for advancing VIReID toward practical applications.
Takeaways & Limitations
GAN-based image-level methods can generate noisy cross-modality images because VIS–IR image pairs are unavailable.
Abstract
from arXiv · showhide
For the visible-infrared person re-identification (VIReID) task, one of the major challenges is the modality gaps between visible (VIS) and infrared (IR) images. However, the training samples are usually limited, while the modality gaps are too large, which leads that the existing methods cannot effectively mine diverse cross-modality clues. To handle this limitation, we propose a novel augmentation network in the embedding space, called diverse embedding expansion network (DEEN). The proposed DEEN can effectively generate diverse embeddings to learn the informative feature representations and reduce the modality discrepancy between the VIS and IR images. Moreover, the VIReID model may be seriously affected by drastic illumination changes, while all the existing VIReID datasets are captured under sufficient illumination without significant light changes. Thus, we provide a low-light cross-modality (LLCM) dataset, which contains 46,767 bounding boxes of 1,064 identities captured by 9 RGB/IR cameras. Extensive experiments on the SYSU-MM01, RegDB and LLCM datasets show the superiority of the proposed DEEN over several other state-of-the-art methods. The code and dataset are released at: https://github.com/ZYK100/LLCM
1. Introduction
VIReID must bridge large VIS–IR modality gaps under limited data and challenging illumination, while existing datasets largely omit low-light conditions. The paper proposes DEEN for diverse embedding learning and introduces LLCM, with experiments showing DEEN outperforms state-of-the-art methods.
- VIReID is harder than conventional ReID because VIS and IR images have an additional cross-modality discrepancy.
- Existing feature-level methods struggle to project cross-modality images into a common space, while image-level translation can introduce noise without VIS–IR pairs.
- DEEN augments embeddings with a DEE module and CPM loss, while MFA aggregates multistage features to mine diverse channel-wise and spatial representations.
- LLCM contains 46,767 images of 1,064 identities captured under illumination changes and low illuminations, targeting more practical VIReID evaluation.
- Extensive experiments on SYSU-MM01, RegDB, and LLCM show that DEEN outperforms other state-of-the-art VIReID methods.
2. Related Work
VIReID methods address modality discrepancy either by translating images between modalities or by learning shared and modality-specific feature spaces. The proposed DEEN pipeline instead generates and aggregates diverse embeddings for cross-modality representation learning.
- VIReID methods generally fall into image-level and feature-level categories.
- Image-level methods use GAN-based transformations to align cross-modality images and alleviate limited data, but unpaired translation can add noise.
- Feature-level methods seek modality-shared and modality-specific spaces to minimize modality gaps through metric, center, pixel, or discriminator-based learning.
- DEEN generates additional embeddings with DEE and CPM, then aggregates multistage features through MFA to mine diverse representations.
3. Method
DEEN expands VIS and IR features in the embedding space, constrains generated embeddings to be diverse and discriminative, and aggregates multistage channel-wise and spatial representations.
- Overall architecture: DEEN uses a two-stream ResNet-50 backbone, a DEE module for generating embeddings, CPM loss for diversity, and MFA for multistage feature aggregation.The generated embeddings are concatenated and used as input to the next backbone stage.
- Diverse Embedding Expansion: The DEE module uses multi-branch dilated convolutions with dilation ratios 1, 2, and 3, followed by feature recombination, ReLU activation, and a 1 × 1 convolution.This structure generates additional embeddings from the input feature map.
- Center-Guided Pair Mining Loss: CPM pulls generated embeddings toward opposite-modality embeddings, pushes them away from same-modality originals, and enforces smaller intra-class than inter-class distances.The loss uses Euclidean distances between embeddings and class centers, with a margin term α balancing its terms.
- Center-Guided Pair Mining Loss: Orthogonal loss encourages embeddings from different DEE branches to capture different informative representations by minimizing overlapping elements.The loss is defined through pairwise inner products between generated embeddings from different branches.
- Multistage Feature Aggregation Block: MFA aggregates low- and high-level features to mine diverse channel-wise and spatial representations using learned similarity-based operations.The block considers feature maps before and after each backbone stage and combines them through channel-spatial aggregation.
- Optimization: The network is optimized end-to-end with cross-entropy, triplet, CPM, and orthogonal losses, whose relative contributions are controlled by λ1 and λ2.The total objective is L_total = L_ce + L_tri + λ1L_cpm + λ2L_ort.
4. LLCM Dataset
LLCM is a low-light VIS-IR dataset collected with a nine-camera network and designed to expose illumination, appearance, and real-world variability challenges in VIReID.
- Dataset construction: LLCM uses a 9-camera network to capture VIS images in daytime and IR images at night, with face regions blurred for privacy.Each annotated identity is captured by both VIS and IR cameras.
- Low-light conditions: 46,767 bounding boxes from 1,064 identities are captured under severe illumination changes and low-light conditions in both modalities.These conditions can alter clothing colors and remove clothing texture information.
- Temporal and appearance variation: LLCM was collected over 100 days from January to April across different climate conditions and clothing styles.The long-term collection supports studying VIReID across climate and clothing variation.
- Real-world challenges: The dataset includes motion blur, pose variation, camera-view changes, occlusion, and low resolution alongside low-light conditions.These challenges are intended to reflect conditions encountered in real-world VIReID applications.
- Dataset split: The training split contains 30,921 bounding boxes from 713 identities, while the testing split contains 13,909 bounding boxes from 351 identities.The split ratio is approximately 2:1, with VIS and IR samples represented in both splits.
5. Experiments
Experiments on SYSU-MM01, RegDB, and LLCM show that DEEN achieves strong VIReID performance, while ablations explain how its components, placement, and hyperparameters affect results.
- Comparison with State-of-the-art Methods: DEEN achieves the best reported performance against state-of-the-art methods on SYSU-MM01 and RegDB.It reports 74.7% Rank-1 and 71.8% mAP in SYSU-MM01 All-Search, 80.3% and 83.3% in Indoor-Search, and 91.1%/85.1% and 89.5%/83.4% across RegDB directions.
- Comparison with State-of-the-art Methods: The best competing method reaches only 54.9% Rank-1 accuracy and 62.9% mAP on LLCM under IR to VIS retrieval.DEEN achieves the best performance in both VIS to IR and IR to VIS modes, while the results indicate that LLCM is highly challenging.
- Ablation Studies: CPM makes DEE-generated embeddings more diverse, substantially improving the baseline and reducing the VIS–IR modality discrepancy.DEE alone slightly improves the baseline, whereas adding CPM greatly improves performance; MFA contributes by aggregating multistage features.
- Ablation Studies: DEE performs best when plugged after ResNet-50 stage-3, whereas stage-4 causes a rapid performance drop.The paper attributes the stage-4 degradation to CPM enlarging distances between generated and original embeddings, complicating optimization.
- Ablation Studies: Three DEE branches achieve the best performance on both LLCM and SYSU-MM01, while additional branches introduce redundant features.Performance rises from two to three branches but declines beyond three.
- Ablation Studies: MFA outperforms the Non-Local block by 1.1% Rank-1 accuracy and 2.2% mAP.The results also characterize MFA and DEE as complementary for generating diverse embeddings and reducing modality gaps.
- Ablation Studies: The best hyperparameter setting is λ1 = 0.8, λ2 = 0.1, and α = 0.2.These values produce the highest performance in the reported LLCM comparison.
6. CONCLUSION
The paper introduces DEEN to generate diverse embeddings and mine diverse feature representations for reducing VIS–IR modality discrepancy. It also contributes the challenging LLCM dataset, and experiments across three datasets show DEEN's superiority over several state-of-the-art methods.
- Conclusion: DEEN generates diverse embeddings and mines diverse channel-wise and spatial representations to reduce VIS–IR modality discrepancy.The method is designed for informative feature learning in VIReID.
- Conclusion: The LLCM dataset introduces challenging low-light cross-modality data intended to facilitate VIReID research toward practical applications.The dataset adds new low-light conditions beyond existing sufficiently illuminated VIReID datasets.
- Conclusion: Experiments on SYSU-MM01, RegDB, and LLCM show DEEN's superiority over several other state-of-the-art methods.This conclusion is reported across all three evaluated datasets.