PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification

Guan-An Wang, Tianzhu Zhang. Yang Yang, Jian Cheng, Jianlong Chang, Xu Liang, Zengguang Hou

arXiv:2002.04114v2cs.CV

TL;DR

RGB-IR person re-identification must bridge large modality differences despite unpaired RGB and IR images, making set-level alignment prone to instance misalignment. JSIA-ReID generates cross-modality paired images through feature disentanglement and exchange, then performs joint set-level and instance-level alignment. Experiments on two datasets demonstrate effectiveness, including gains of 9.2% Rank-1 and 7.7% mAP on SYSU-MM01.

  • Problem

    Unpaired RGB and IR images make cross-modality alignment difficult, while global set-level alignment can misalign individual instances.

  • Method

    JSIA-ReID disentangles modality-invariant and modality-specific features, exchanges features to generate paired images, and jointly learns set-level and instance-level aligned features.

  • Results

    Experiments on two standard datasets demonstrate that JSIA-ReID performs favourably against state-of-the-art methods, with gains of 9.2% Rank-1 and 7.7% mAP on SYSU-MM01.

  • Takeaways & Limitations

    Generating cross-modality paired images enables direct instance-level alignment alongside set-level alignment for RGB-IR Re-ID.

Abstract

from arXiv · show

RGB-Infrared (IR) person re-identification is very challenging due to the large cross-modality variations between RGB and IR images. The key solution is to learn aligned features to the bridge RGB and IR modalities. However, due to the lack of correspondence labels between every pair of RGB and IR images, most methods try to alleviate the variations with set-level alignment by reducing the distance between the entire RGB and IR sets. However, this set-level alignment may lead to misalignment of some instances, which limits the performance for RGB-IR Re-ID. Different from existing methods, in this paper, we propose to generate cross-modality paired-images and perform both global set-level and fine-grained instance-level alignments. Our proposed method enjoys several merits. First, our method can perform set-level alignment by disentangling modality-specific and modality-invariant features. Compared with conventional methods, ours can explicitly remove the modality-specific features and the modality variation can be better reduced. Second, given cross-modality unpaired-images of a person, our method can generate cross-modality paired images from exchanged images. With them, we can directly perform instance-level alignment by minimizing distances of every pair of images. Extensive experimental results on two standard benchmarks demonstrate that the proposed model favourably against state-of-the-art methods. Especially, on SYSU-MM01 dataset, our model can achieve a gain of 9.2% and 7.7% in terms of Rank-1 and mAP. Code is available at https://github.com/wangguanan/JSIA-ReID.

Introduction

RGB-IR person re-identification is difficult because RGB and IR images differ substantially, while unpaired training data limits fine-grained correspondence. JSIA-ReID generates cross-modality paired images and jointly performs set-level and instance-level alignment.

  • Introduction: RGB-IR Re-ID supports nighttime surveillance but is harder than RGB-RGB matching because the modalities are intrinsically heterogeneous.RGB images contain three visible-light color channels, whereas IR images contain one invisible-light channel.
  • Introduction: Existing methods mainly align entire RGB and IR sets, which can misalign individual instances.They reduce marginal distribution divergence but do not address joint distributions or every image pair.
  • Introduction: Unpaired RGB and IR images prevent direct instance-level alignment, and conventional translation models cannot represent IR-to-RGB one-to-many mappings.RGB and IR images are captured at different times, while CycleGAN and StarGAN learn one-to-one mappings.
  • Introduction: JSIA-ReID disentangles modality-specific and modality-invariant features to perform set-level alignment while explicitly removing modality-specific information.The method uses a generation module and a feature alignment module to learn aligned features.
  • Introduction: JSIA-ReID generates cross-modality paired images from unpaired images and minimizes distances between each pair for instance-level alignment.The paired images are produced by exchanging disentangled features before decoding.

Related Works

Prior work largely addresses RGB-RGB re-identification, while RGB-IR Re-ID introduces cross-modality variation as an additional challenge. Related approaches use specialized networks, generated samples, or unpaired image translation for re-identification and modality adaptation.

  • RGB-RGB Person Re-Identification: RGB-RGB Re-ID matches pedestrian images across disjoint visible cameras, and many deep methods learn identity-discriminative features.These methods include CNN fine-tuning and triplet-loss variants.
  • RGB-IR Person Re-Identification: Most existing methods focus on RGB-RGB Re-ID and cannot perform well on RGB-IR Re-ID in practical surveillance scenarios.RGB-IR Re-ID adds cross-modality variation between RGB and IR images.
  • RGB-IR Person Re-Identification: RGB-IR Re-ID methods include zero-padding networks designed to evolve domain-specific nodes across modalities.Wu et al. introduced the SYSU RGB-IR Re-ID dataset and explored three network structures.
  • Person Re-Identification with GAN: GAN-based Re-ID methods generate unlabeled samples, translate camera styles, or synthesize pedestrian images to reduce variation or augment training data.Examples use GAN, CycleGAN, and conditional GAN models.
  • Image Translation: Pix2Pix uses paired supervision, whereas CycleGAN and StarGAN learn image translations from unpaired data using cycle-consistency.GANs learn image distributions through adversarial training.

The Proposed Method

JSIA-ReID generates cross-modality paired images by disentangling modality-invariant and modality-specific features, then jointly learns set-level and instance-level aligned features. Reconstruction, cycle-consistency, adversarial, alignment, classification, and triplet objectives support realistic generation and identity discrimination.

  • Cross-Modality Paired-Images Generation Module: The generation module disentangles RGB and IR images into modality-invariant content features and modality-specific style features, then reconstructs original images.The modality-invariant encoder learns content information, while separate encoders learn RGB and IR style information.
  • Cross-Modality Paired-Images Generation Module: Cross-modality paired images are generated by exchanging style features between RGB and IR images while keeping exchanges intra-person to preserve identity.Generated RGB images combine IR content with RGB style, and generated IR images combine RGB content with IR style.
  • Cross-Modality Paired-Images Generation Module: Reconstruction, cycle-consistency, and adversarial losses supervise reconstruction, preserve content and style through translations, and improve generated-image realism.The cycle-consistency loss addresses missing supervision for cross-modality generation, while discriminators distinguish real from generated RGB and IR images.
  • Feature Alignment Module: The feature alignment module first aligns modality-invariant features at the set level, then minimizes Kullback-Leibler divergence between every cross-modality paired-image at the instance level.An instance-level encoder maps set-level aligned features into a new space before pairwise probability distributions are aligned.
  • Feature Alignment Module: Identity-discriminative learning averages instance-level feature maps into vectors and optimizes them with classification and triplet losses.The triplet loss uses positive same-person pairs, negative different-person pairs, and a margin parameter.
  • Feature Alignment Module: The overall objective weights reconstruction, cycle-consistency, adversarial, re-identification, and alignment terms; testing uses only feature extraction and cosine-similarity matching.At test time, the set-level and instance-level encoders extract feature vectors for probe-gallery comparison.

Experiment

Experiments on SYSU-MM01 and RegDB show strong benchmark performance, complementary set-level and instance-level alignment, robustness to alignment weighting, and more reliable paired-image generation than CycleGAN and StarGAN.

  • Dataset and evaluation: The model is evaluated on SYSU-MM01 and RegDB using Rank-1 and mAP-based protocols across standard query settings.SYSU-MM01 contains 491 identities from four RGB and two IR cameras; RegDB contains 412 persons with visible and thermal images.
  • Comparison with state-of-the-art: 9.2% and 7.7% gains over the state-of-the-art are reported on SYSU-MM01 in Rank-1 and mAP, respectively.On RegDB, the model exceeds state-of-the-art Rank-1 scores by 4.7% in thermal2visible and 5.1% in visible2thermal modes.
  • Ablation study: 38.1% Rank-1 is achieved with both set-level and instance-level alignment, versus 32.1% without either and 35.1% or 36.0% with only one.Removing disentanglement reduces Rank-1 by 1.3%, indicating its contribution to set-level alignment.
  • Similarity analysis: Set-level alignment concentrates intra-person similarity but also increases inter-person similarity, whereas instance-level alignment boosts intra-person similarity while keeping inter-person similarity unchanged.The similarity analysis uses cosine distance and attributes the latter behavior to improved instance-level alignment.
  • Parameter analysis: Performance remains significantly improved across different λalign values on SYSU-MM01 single-shot and all-search settings, demonstrating robustness to the alignment weight.The analysis reports stable improvement rather than a single sensitive optimum.
  • Generated-image visualization: The proposed method stably generates paired images with correct cross-modality appearance, while CycleGAN and StarGAN fail to preserve colors or poses reliably.The authors state that the competing generated images are unsuitable for learning instance-level aligned features.

Conclusion

JSIA-ReID jointly reduces set-level and instance-level modality variation by disentangling features and generating paired images from exchanged features. Results on two datasets show the proposed method's effectiveness.

  • Conclusion: JSIA-ReID performs set-level alignment by disentangling modality-specific and modality-invariant features, explicitly removing modality-specific information.This design is intended to reduce the modality gap.
  • Conclusion: Given cross-modality unpaired images, JSIA-ReID generates paired images by exchanging features and reduces instance-level variation by minimizing pairwise distances.The generated pairs support fine-grained instance-level alignment.
  • Conclusion: Experimental results on two datasets show the effectiveness of the proposed method.
Loading 2002.04114v2…