Source-linked AI summary
RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment
Guan'an Wang, Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, Zengguang Hou
TL;DR
RGB-IR person Re-ID is challenged by large cross-modality variation, and feature alignment alone makes direct matching difficult. AlignGAN jointly aligns pixels and features using a pixel generator, feature generator, and joint discriminator. On SYSU-MM01, it reports absolute gains of 15.4% Rank-1 and 12.9% mAP, with favorable results on two standard benchmarks.
Problem
RGB-IR Re-ID must bridge large cross-modality variation, while existing methods mainly rely on feature alignment and direct matching in a shared feature space is difficult.
Method
AlignGAN jointly uses a pixel generator, feature generator, and joint discriminator to align RGB-IR data in pixel and feature spaces while learning identity-consistent features.
Results
15.4% Rank-1 and 12.9% mAP are the reported absolute gains on SYSU-MM01, with favorable performance against state-of-the-art methods on two standard benchmarks.
Takeaways & Limitations
The paper supports jointly modeling pixel and feature alignment for RGB-IR Re-ID while reducing cross-modality and intra-modality variation and learning identity-consistent features.
Takeaways & Limitations
Separate alignment strategies may not preserve identity consistency for unseen test identities, and generated fake IR images retain discrepancies from viewpoint, pose, and occlusion changes.
Abstract
from arXiv · showhide
RGB-Infrared (IR) person re-identification is an important and challenging task due to large cross-modality variations between RGB and IR images. Most conventional approaches aim to bridge the cross-modality gap with feature alignment by feature representation learning. Different from existing methods, in this paper, we propose a novel and end-to-end Alignment Generative Adversarial Network (AlignGAN) for the RGB-IR RE-ID task. The proposed model enjoys several merits. First, it can exploit pixel alignment and feature alignment jointly. To the best of our knowledge, this is the first work to model the two alignment strategies jointly for the RGB-IR RE-ID problem. Second, the proposed model consists of a pixel generator, a feature generator, and a joint discriminator. By playing a min-max game among the three components, our model is able to not only alleviate the cross-modality and intra-modality variations but also learn identity-consistent features. Extensive experimental results on two standard benchmarks demonstrate that the proposed model performs favorably against state-of-the-art methods. Especially, on SYSU-MM01 dataset, our model can achieve an absolute gain of 15.4% and 12.9% in terms of Rank-1 and mAP.
1. Introduction
RGB-IR Re-ID must address heterogeneous modalities and difficult intra-class variation, while existing feature-alignment methods struggle to match RGB and IR images directly. AlignGAN jointly performs pixel and feature alignment with a joint discriminator to learn identity-consistent features.
- Motivation: RGB-IR Re-ID is needed because visible cameras can fail in poor illumination, while RGB and IR images differ intrinsically in channels and wavelength ranges.RGB images contain three visible-light color channels; IR images contain one channel.
- Existing approaches: Existing methods mainly bridge RGB-IR differences through feature representation learning, but direct matching in a shared feature space is difficult under large cross-modality variation.The cited comparison reports weaker cmGAN performance under cross-modality than single-modality settings.
- Proposed strategy: Pixel alignment generates identity-maintained fake IR images from RGB images, after which feature alignment matches them with real IR images.This strategy uses a pixel generator and a feature generator to align the modalities in pixel and feature spaces.
- Methodological gap: Separate pixel and feature alignment may fail to complement each other because fitting training identities cannot guarantee identity-consistent aligned features for unseen test identities.Viewpoint changes, pose variations, and occlusions also create intra-class discrepancies among generated and real IR images.
- Proposed method: AlignGAN combines a pixel generator, feature generator, and joint discriminator in a min-max game to reduce cross-modality and intra-modality variation.The feature generator uses identity-based classification and triplet losses, while the joint discriminator supports identity-consistent features.
- Results: 15.4% Rank-1 and 12.9% mAP are the reported absolute gains of AlignGAN on SYSU-MM01.The paper also reports favorable performance against state-of-the-art methods on two standard benchmarks.
2. Related Works
Prior work on person Re-ID includes handcrafted, metric-learning, and deep-learning approaches, while RGB-IR Re-ID additionally requires cross-modality matching. GAN research has focused on image translation or domain adaptation, motivating task-specific cross-modality methods.
- RGB-RGB Person Re-Identification: RGB-RGB Re-ID matches pedestrian RGB images across disjoint visible cameras and addresses variation from views, poses, illuminations, and occlusions.Existing approaches include handcrafted descriptors, metric learning, and deep learning algorithms.
- RGB-IR Person Re-Identification: RGB-IR Re-ID matches RGB and IR images across disjoint cameras and adds cross-modality variation to the difficulties of RGB-RGB Re-ID.The related work describes SYSU RGB-IR Re-ID and methods using modality-specific and modality-shared representations.
- RGB-IR Person Re-Identification: Prior RGB-IR methods include deep zero-padding, modality-specific and shared metric losses, bi-directional ranking losses, and cmGAN for reducing RGB-IR feature divergence.These methods target cross-modality matching through network design, metric learning, ranking, or adversarial feature alignment.
- Generative Adversarial Networks: GANs learn data distributions through adversarial training and have been applied to image translation and domain adaptation, but those works do not target RGB-IR cross-modality matching.Pix2Pix, CycleGAN, and StarGAN are cited as image-translation examples.
3. Alignment Generative Adversarial Network
AlignGAN jointly aligns RGB-IR images in pixel and feature spaces using pixel and feature generators with a joint discriminator. Its training combines realism, identity, modality, and feature-alignment objectives to produce identity-consistent representations.
- AlignGAN contains pixel alignment, feature alignment, and joint discriminator modules for RGB-IR re-identification.The pixel module translates RGB images into identity-maintained fake IR images, while the feature module processes fake and real IR images in a shared space.
- 3.3. Joint Discriminator Module: Joint adversarial training encourages realistic fake IR images, aligned feature distributions, and identity consistency between generated images and their corresponding RGB inputs.The paper describes the discriminator as making pixel and feature modules learn from each other through a min-max game.
- 3.1. Pixel Alignment Module: The pixel generator maps RGB images to realistic IR-style images while preserving their original identities.Adversarial training makes generated images resemble real IR data, while identity classification, triplet, and cycle-consistency objectives constrain content and identity preservation.
- 3.2. Feature Alignment Module: The feature generator encodes fake and real IR images into a shared feature space using classification, triplet, and feature-space GAN losses.The feature-space discriminator reduces distribution divergence between real and fake IR feature maps.
- 3.3. Joint Discriminator Module: The joint discriminator evaluates image-feature pairs and treats only same-identity real IR pairs as real.Other combinations, including mismatched identities or pairs containing fake elements, are classified as fake.
- 3.4. Train and Test: At test time, AlignGAN uses the pixel and feature modules, translating RGB inputs to fake IR images before feature extraction.IR images are passed directly through the feature module, whereas RGB images first pass through pixel alignment.
4. Experiments
Experiments evaluate AlignGAN on SYSU-MM01 and RegDB using repeated-split CMC and mAP protocols, comparing benchmarks, baselines, variants, parameter settings, and visualizations. AlignGAN achieves strong cross-modality performance, with gains attributed to joint pixel-feature alignment and the joint discriminator.
- Datasets and Evaluation: Experiments use SYSU-MM01 and RegDB, with CMC and mAP evaluated over repeated random splits.SYSU-MM01 results average 10 repeated gallery/probe splits, while RegDB results average 10 repeated training/testing splits.
- SYSU-MM01 Results: AlignGAN significantly outperforms the state-of-the-art method on SYSU-MM01 by 15.4% Rank-1 and 12.9% mAP.The comparison follows the single-shot&all-search analysis unless otherwise specified.
- RegDB Results: AlignGAN outperforms state-of-the-art methods on RegDB by 23.5% and 24.4% Rank-1 in thermal2visible and visible2thermal modes, respectively.It also exceeds HSME by 6.1%/7.0% Rank-1 on RegDB and gains 21.72%/17.58% Rank-1 on SYSU-MM01.
- Model Analysis: PixAlign improves Basel. by 11.0% Rank-1, FeatAlign by 4.5%, and PixAlign exceeds FeatAlign by 6.5%, showing both alignment modules contribute unequally.The pixel alignment module has the larger isolated improvement in this ablation.
- Model Analysis: The joint-discriminator AlignGAN performs much better than PixAlign and FeatAlign, indicating that jointly complementing the alignment modules improves performance.A simple stack of separate pixel and feature alignment modules does not improve over PixAlign in the reported comparison.
- Parameter Analysis: Across different λpix_gan settings, AlignGAN stably outperforms Basel., demonstrating robustness to the tested parameter weights.Figure 4 reports this parameter analysis on SYSU-MM01 under the single-shot&all-search mode.
5. Conclusion
AlignGAN jointly exploits pixel and feature alignment for RGB-IR RE-ID, generating identity-maintained fake IR images and learning identity-consistent features while performing favorably against state-of-the-art methods.
- Joint pixel and feature alignment is proposed for the RGB-IR RE-ID task.
- AlignGAN generates fake IR images that maintain the identities and contents of original RGB images while adopting IR style.
- AlignGAN learns features that better maintain identity-consistency across modalities.
- The model performs favorably against state-of-the-art methods on two standard benchmarks.