Source-linked AI summary
Learning by Aligning: Visible-Infrared Person Re-identification using Cross-Modal Correspondences
Hyunjong Park, Sanghoon Lee, Junghyup Lee, Bumsub Ham
TL;DR
VI-reID must match identities across visible and infrared images despite intra-class variation, misalignment, occlusion, and modality discrepancies. The paper learns dense cross-modal correspondences with pixel-level losses and reports state-of-the-art performance on standard benchmarks, while failure cases remain for similar silhouettes and ambiguous objects.
Problem
VI-reID must handle intra-class variations, imperfect alignment, and cross-modal discrepancies that limit coarse image- or rigid part-level representations.
Method
The framework establishes probabilistic dense RGB–IR correspondences, uses person masks, and trains with ID consistency and dense triplet losses based on pixel-level associations.
Results
The method achieves a new state of the art on standard VI-reID benchmarks and significantly outperforms other VI-reID methods overall.
Takeaways & Limitations
Dense cross-modal correspondences produce person representations that are robust to intra-class variations and modality discrepancies while remaining discriminative.
Takeaways & Limitations
The method still fails for persons with similar silhouettes and for images containing objects with ambiguous correspondences, such as backpacks.
Abstract
from arXiv · showhide
We address the problem of visible-infrared person re-identification (VI-reID), that is, retrieving a set of person images, captured by visible or infrared cameras, in a cross-modal setting. Two main challenges in VI-reID are intra-class variations across person images, and cross-modal discrepancies between visible and infrared images. Assuming that the person images are roughly aligned, previous approaches attempt to learn coarse image- or rigid part-level person representations that are discriminative and generalizable across different modalities. However, the person images, typically cropped by off-the-shelf object detectors, are not necessarily well-aligned, which distract discriminative person representation learning. In this paper, we introduce a novel feature learning framework that addresses these problems in a unified way. To this end, we propose to exploit dense correspondences between cross-modal person images. This allows to address the cross-modal discrepancies in a pixel-level, suppressing modality-related features from person representations more effectively. This also encourages pixel-wise associations between cross-modal local features, further facilitating discriminative feature learning for VI-reID. Extensive experiments and analyses on standard VI-reID benchmarks demonstrate the effectiveness of our approach, which significantly outperforms the state of the art.
1. Introduction
VI-reID retrieves matching identities across visible and infrared images but is challenged by intra-class variation, noisy misalignment, and modality differences. The paper addresses these issues with dense cross-modal correspondences and pixel-level losses to learn robust, discriminative person representations.
- VI-reID retrieves infrared images matching an RGB query and vice versa, extending person re-identification across visible and infrared modalities.Visible cameras perform poorly in poor illumination, whereas infrared cameras capture overall scene layout without texture and color details.
- The task is difficult because of intra-class variation, noisy samples from misalignment or occlusion, and substantial RGB–IR discrepancies.Relevant variations include viewpoint, pose, illumination, and background clutter.
- Previous methods learn coarse image-level or rigid part-level representations while assuming person images are roughly aligned.Misaligned features can distract representation learning and impair handling of cross-modal discrepancies.
- The proposed framework uses dense cross-modal correspondences to align RGB and IR person representations at the pixel level during training.It encourages same-identity representations to reconstruct one another across modalities and uses parameter-free person masks to focus on person regions.
- ID consistency and dense triplet losses exploit pixel-level associations to reduce modality discrepancies and enhance discriminative person representations.The framework is presented as a unified feature-learning approach for VI-reID.
2. Related work
Related work spans single-modality person re-identification, VI-reID, cross-modal image retrieval, and dense correspondence learning. Existing approaches address modality gaps through metric learning, feature disentanglement, generation, attention, or correspondence-based methods, but some impose test-time computational costs.
- ReID: Person re-identification methods commonly formulate single-modality matching as multi-class classification and use triplet loss to structure feature space.These methods place same-identity images nearby and different-identity images farther apart.
- VI-reID: VI-reID methods learn discriminative features that generalize across RGB and IR modalities while addressing cross-modal discrepancies.Representative approaches include graph attention for cross-modal relations and GANs that synthesize identity-preserving or cycle-consistent infrared images.
- Cross-modal image retrieval: Cross-modal image retrieval typically uses siamese networks to learn pairwise metrics or disentangles modality-shared and modality-specific embeddings.These approaches target matching across image modalities such as sketch and natural images or RGB and IR images.
- Correspondence: Dense correspondences support tasks including depth prediction, optical flow, 3D reconstruction, colorization, and person re-identification.A prior person-reID method using dense correspondences required its learned metric function at test time, increasing computational and memory demands.
3. Approach
The framework aligns RGB and IR feature maps through dense probabilistic correspondences and person-focused soft warping. It trains with identity, cross-modal consistency, and dense local constraints, while using the alignment module only during training for efficient inference.
- Training objective: The framework uses original and reconstructed features during training with ID, ID consistency, and dense triplet losses.All components are differentiable, enabling end-to-end training with ground-truth identity labels.
- Inference: At test time, person representations are obtained by pooling RGB and IR features, and retrieval uses cosine distances without CMAlign.Using CMAlign only during training supports efficient inference.
- Feature extractor: A two-stream CNN extracts RGB and IR feature maps, using modality-specific shallow parameters and shared parameters in later layers.The feature maps have dimensions h×w×d.
- CMAlign: CMAlign computes bidirectional dense cross-modal correspondences from local cosine similarities and matching probabilities, then performs soft feature warping.Matching probabilities avoid the brittleness of hard argmax correspondences, which can be distracted by background clutter, occlusion, and modality-specific appearance.
- CMAlign: Parameter-free person masks restrict reconstruction and alignment to foreground regions, reducing ambiguous matches involving background.The reconstructed feature retains original features outside person regions while aggregating cross-modal features probabilistically within them.
- Training objective: ID consistency trains reconstructed cross-modal representations to retain the identity labels of their original positive counterparts, suppressing modality-related features.The loss applies cross-entropy using the same classifier as the ID loss.
- Training objective: The dense triplet loss compares local original and reconstructed features within mutually visible foreground regions defined by co-attention maps.This avoids enforcing local alignment over occluded or disassociated body regions.
4. Experiments
Experiments evaluate the approach on RegDB and SYSU-MM01, comparing benchmark performance, efficiency, and model variants. Ablations and correspondence visualizations examine the contributions of losses, co-attention, and module placement.
- Datasets and protocols: The evaluation uses RegDB and SYSU-MM01, with visible-to-IR and IR-to-visible testing on RegDB and all-search and indoor-search modes on SYSU-MM01.RegDB results are averaged over 10 training/test splits, while SYSU-MM01 results are averaged over 4 runs.
- Benchmark comparison: The model sets a new VI-reID state of the art except in SYSU-MM01 indoor-search mode, where DDAG performs better.The comparison reports mAP (%) and rank-1 accuracy (%) on RegDB and SYSU-MM01.
- Efficiency: The method is fastest among the compared methods and uses the fewest test-time parameters by requiring only the backbone network.Other methods use additional layers or networks at test time.
- Ablation study: Using all losses with the co-attention map gives the best ablation performance, indicating that the components are complementary.The ID consistency term addresses cross-modal discrepancies at image level, while the dense triplet term handles them at pixel level and strengthens discriminative representations.
- Ablation study: Adding CMAlign modules to both conv4-6 and conv5-3 gives the best results by considering cross-modal discrepancies at multiple feature levels.The co-attention map is especially important for the dense triplet term because distractive regions can hinder discriminative representation learning.
- Correspondence analysis: Visualized correspondences match same-identity RGB and IR persons and remain robust to scale, occlusion, viewpoint, and background variations.Viewpoint robustness includes matching a sweatshirt or trousers across front and side views.
5. Conclusion
The framework uses dense cross-modal correspondences and pixel-level associations to improve VI-reID representations, achieving state-of-the-art benchmark performance.
- Dense correspondences make VI-reID representations robust to intra-class variations and cross-modal discrepancies across RGB and IR images.
- ID consistency and dense triplet losses exploit pixel-level associations to learn more discriminative person representations.
- The approach sets a new state of the art on standard VI-reID benchmarks and significantly outperforms other VI-reID methods.
Supplementary Material
The supplementary material reports additional evaluations, qualitative comparisons, and visualizations covering metrics, tasks, and learned correspondences.
- Additional performances are reported under different quantitative metrics, alongside results from applying the method to single-modality person reID.
- The supplement presents qualitative comparisons with state-of-the-art VI-reID methods and visualizations of co-attention maps and cross-modal correspondences.
S1. Quantitative comparisons
The supplementary quantitative comparison reports rank-10 accuracy on the RegDB and SYSU-MM01 datasets, averaged over four training and test runs.
- Rank-10 accuracy (%) is reported on the RegDB and SYSU-MM01 datasets as an additional robustness-oriented evaluation metric.
- The reported results average values across 4 training and test runs.
S2. RGB reID
The framework generalizes to single-modality person reID, improving mAP and rank-1 accuracy while adding no test-time parameters or computational overhead.
- The CMAlign module and the proposed losses improve mAP and rank-1 accuracy for both tested single-modality reID configurations.The configurations use a ResNet50 backbone and PCB with or without CMAlign and the proposed losses.
- Applying the approach to single-modality reID demonstrates effectiveness beyond the VI-reID setting and boosts performance for other reID methods.
- The approach requires no additional parameters or computational overhead at test time.
S3. Qualitative comparisons
The qualitative comparisons evaluate retrieval behavior on SYSU-MM01 and RegDB, while also documenting failure cases for similar silhouettes and ambiguous objects.
- Retrieval results on SYSU-MM01 and RegDB are sorted by cosine distance for each query image.
- The model retrieves same-identity person images despite severe intra-class variations across person images.
- Current VI-reID methods, including this model, fail to reliably discriminate persons with similar silhouettes.
- The method tends to fail when correspondences are ambiguous, such as for a backpack, making CMAlign ineffective for enhancing local-feature discriminability.
S4. Co-attention map
Co-attention maps focus dense-triplet reconstruction on image regions visible in both RGB and infrared images, excluding disassociated human parts.
- Co-attention maps focus reconstructions on regions visible in both RGB and infrared person images for dense-triplet feature learning.
- The map ignores disassociated human parts, such as feet that are not visible in the infrared image.
- This masking concentrates the dense triplet loss on cross-modal regions that can be associated.
S5. Cross-modal correspondences
The correspondence analysis shows that the full model produces semantically related dense matches and local features robust to modality differences and severe intra-class variation.
- The model produces local features robust to RGB-infrared discrepancies under severe scale, occlusion, and viewpoint variations.
- The full model establishes dense correspondences between semantically related regions.
- Top-20 matches are selected by matching probabilities for each RGB-infrared image pair.
- Comparisons across training-loss configurations show that the full model yields the strongest qualitative correspondence structure among the displayed configurations.