Source-linked AI summary
Hetero-Center Loss for Cross-Modality Person Re-Identification
Yuanxin Zhu, Zhao Yang, Li Wang, Sai Zhao, Xiao Hu, Dapeng Tao
TL;DR
Cross-modality person re-identification requires matching RGB and infrared pedestrian images despite modality differences, while prior work has focused more on inter-class separation than intra-class cross-modality similarity. The paper introduces Hetero-Center loss with CE loss and a Two-Stream Local Feature Network to learn modality-shared local representations. On the most difficult all-search single-shot mode, it reports gains of 29.99% in Rank1 and 27.15% in mAP over state-of-the-art methods.
Problem
Cross-modality person re-identification needs features that enlarge inter-class discrepancy and improve intra-class cross-modality similarity, but Cross-Entropy loss does not effectively enforce modality-shared information.
Method
The paper jointly trains Cross-Entropy and Hetero-Center losses while using TSLFN, a two-stream architecture that extracts and aligns local features across modalities.
Results
29.99% Rank1 and 27.15% mAP gains are reported over state-of-the-art methods in the all-search single-shot mode.
Takeaways & Limitations
HC loss and TSLFN provide a simple framework for learning discriminative, modality-shared local representations in cross-modality person re-identification.
Abstract
from arXiv · showhide
Cross-modality person re-identification is a challenging problem which retrieves a given pedestrian image in RGB modality among all the gallery images in infrared modality. The task can address the limitation of RGB-based person Re-ID in dark environments. Existing researches mainly focus on enlarging inter-class differences of feature to solve the problem. However, few studies investigate improving intra-class cross-modality similarity, which is important for this issue. In this paper, we propose a novel loss function, called Hetero-Center loss (HC loss) to reduce the intra-class cross-modality variations. Specifically, HC loss can supervise the network learning the cross-modality invariant information by constraining the intra-class center distance between two heterogenous modalities. With the joint supervision of Cross-Entropy (CE) loss and HC loss, the network is trained to achieve two vital objectives, inter-class discrepancy and intra-class cross-modality similarity as much as possible. Besides, we propose a simple and high-performance network architecture to learn local feature representations for cross-modality person re-identification, which can be a baseline for future research. Extensive experiments indicate the effectiveness of the proposed methods, which outperform state-of-the-art methods by a wide margin.
I. INTRODUCTION
Cross-modality person re-identification must bridge substantial RGB–infrared differences while improving both inter-class separation and intra-class cross-modality similarity. The paper addresses this gap with Hetero-Center loss and a local-feature two-stream architecture.
- Motivation: RGB-IR person re-identification is difficult because infrared images lose color information and use a different spectrum from RGB images.These modality differences limit the effectiveness of methods developed for RGB-only person re-identification.
- Related work: The proposed methods target modality-invariant representations while extending prior cross-modality approaches based on zero-padding, metric learning, and adversarial training.Prior methods include zero-padding, TONE, HCML, BDTR, and cmGAN.
- Research gap: Existing methods mainly enlarge inter-class feature discrepancy while giving less attention to intra-class cross-modality similarity.The paper identifies both objectives as important for cross-modality matching.
- Proposed loss: Hetero-Center loss constrains the distance between intra-class centers from heterogeneous modalities to encourage modality-shared feature information.It is jointly optimized with Cross-Entropy loss, combining cross-modality similarity with discriminative identity learning.
- Proposed architecture: TSLFN uses separate modality branches, stripe-based local feature extraction, and shared fully connected layers for corresponding stripes.The shared layers project corresponding local features from the two modalities into the same subspace.
III. METHODS
Cross-modality re-identification requires both inter-class separation and intra-class similarity across RGB and infrared images. Conventional CE supervision emphasizes identity-specific classification, which can preserve modality-specific cues and leave a large cross-modality gap.
- Modality-shared information such as contours and textures is invariant across modalities, whereas modality-specific information such as colors can reduce same-identity feature similarity.
- Cross-modality re-identification matches pedestrian images across RGB and infrared modalities using feature similarity.
- CE loss extracts identity-specific information for classification but does not effectively constrain the network to learn modality-shared information.
- CE-only features separate different identities but exhibit a large gap between modality distributions within each identity.The gap is reflected by the considerable distance between the two modality centers.
- 770 RGB and 300 infrared images from 8 SYSU-MM01 identities are visualized after t-SNE reduces feature dimensions to 2.Figure 2 uses colors for identities, shapes for modalities, and red points for modality-specific class centers.
B. Hetero-Center Loss
HC loss reduces intra-class cross-modality variation by penalizing the distance between RGB and infrared centers within each class. Combined with CE loss, it targets both cross-modality compactness and inter-class discrepancy.
- HC loss penalizes the distance between RGB and infrared feature-distribution centers within each class instead of directly comparing full distributions.The center distance is used because computing the distance between feature distributions is difficult.
- The modality centers are computed within each mini-batch, and balanced sampling selects T RGB and T infrared images for each of L identities.The resulting batch size is 2×L×T=K, with equal samples per class to reduce class-imbalance perturbations.
- HC loss alone improves intra-class cross-modality similarity but does not supervise discriminative inter-class representation, so it is jointly trained with CE loss.
- Increasing λ pulls different-modality feature distributions closer and reduces the distance between their class centers.The paper interprets this as increased intra-class cross-modality similarity and more consistent representations.
- At convergence, the learned representations have both inter-class discrepancy and intra-class cross-modality compactness.
C. Two-Stream Local Feature Network
TSLFN uses two modality-specific branches and local feature processing to learn representations for cross-modality person re-identification. Its local descriptors are embedded under joint HC and CE supervision and concatenated for retrieval.
- Horizontal body-structure representations are treated as modality-shared because body structure is an intrinsic pedestrian property.
- TSLFN contains separate RGB and infrared branches, each comprising feature extraction and feature embedding components.
- The feature map is divided into horizontal stripes, whose pooled local vectors are dimension-reduced by weight-sharing fully connected layers.Weight sharing projects corresponding local features from both modalities into the same subspace.
- During testing, all dimension-reduced local features are concatenated into the final descriptor, and gallery images are ranked by Euclidean distance to the probe descriptor.
- HC and CE losses supervise the embedded local features to enlarge inter-class discrepancy and improve intra-class cross-modality similarity.Feature vectors undergo L2 normalization before HC-loss computation.
A. Dataset description
SYSU-MM01 evaluates RGB-to-infrared person re-identification across six cameras, using all-search and indoor-search protocols with single-shot and multi-shot gallery settings. Experiments use repeated evaluation, specified preprocessing and optimization settings, and compare against handcrafted and deep-learning methods.
- Dataset: SYSU-MM01 contains 287,628 RGB images and 15,792 infrared images captured by six cameras, including four RGB and two infrared cameras.RGB cameras operate in light scenarios, while infrared cameras operate in dark environments.
- Dataset: The training set contains 22,258 RGB images and 11,909 infrared images from 395 persons, while testing uses RGB and infrared images from 96 identities.
- Evaluation protocol: Evaluation uses all-search and indoor-search modes, with all-search being more difficult because of greater scene diversity.Indoor-search uses cameras placed in indoor environments and is closer to an ideal condition without drastic environmental disturbance.
- Evaluation protocol: Single-shot galleries contain one randomly selected image per identity, whereas multi-shot galleries contain ten images per identity.Cam 3 probe images exclude Cam 2 gallery images because both cameras share scenes.
- Implementation: Images are resized to 288×144, and training uses random cropping, horizontal flipping, batch size 64, and four identities per batch.Each identity contributes eight RGB and eight infrared images in a batch.
- Implementation: The final descriptor has dimension 3072, formed by six stripes with 512 dimensions each, and HC loss weight λ is set to 0.5.
E. Ablation experiments
Ablation experiments separate the effects of sampling, local features, and HC loss. The results support the effectiveness of TSLFN, HC loss, and the proposed sampling strategy for cross-modality person re-identification.
- Ablation design: The ablations compare global and local representations with and without the proposed sampling strategy and HC loss.Baseline uses p=1, while TSLFN uses p=6 for local feature extraction.
- Network and loss components: TSLFN outperforms the baseline, indicating that local feature learning is effective for cross-modality person re-identification.
- Network and loss components: Adding HC loss improves both the baseline and TSLFN under joint CE and HC supervision.The comparisons indicate that HC loss improves intra-class cross-modality similarity.
- Sampling strategy: The sampling strategy improves models supervised by both CE loss and joint CE and HC loss.
A. Impact of
The study examines how the HC-loss weight λ affects TSLFN and a global-feature baseline, and evaluates whether HC-loss supervision introduces overfitting. TSLFN performs best near λ=0.5, while large λ values sharply reduce its performance.
- Impact of λ: λ is varied from 0.1 to 0.6 for TSLFN, and about 0.5 is the optimal value in all-search single-shot evaluation.
- Impact of λ: When λ exceeds 0.6, TSLFN performance drops sharply.The authors speculate that insufficient modality-shared information in local features may misdirect HC-loss optimization and cause overfitting.
- Baseline control: For the global-feature baseline, performance improves as λ increases from 0.1 to 1, unlike TSLFN.This control result supports the authors’ inference that local features contribute to the decline at large λ.
- Overfitting analysis: The modified HC loss adds a relaxation margin α, making the loss zero when the distance between modality centers is below α.The original HC loss corresponds to α=0.
- Overfitting analysis: As α increases from 0.1 to 1, model performance decreases, indicating that HC-loss supervision does not lead to model overfitting in this experiment.
D. Comparison among the distance metric
The paper compares weak HC constraints with stronger constraints that additionally align modality variances. Strong constraints perform slightly worse and require greater computational cost than weak constraints.
- Constraint design: Weak constraint denotes HC loss, which constrains the center distance between two modality feature distributions.
- Constraint design: Strong constraint additionally constrains the variances of the two modality distributions in a mini-batch.
- Comparison: Strong constraints achieve slightly lower performance than weak constraints in all-search single-shot evaluation.
- Comparison: Strong constraints have higher computational cost than weak constraints.
F. The number of parts
TSLFN performance initially improves as the number of local parts increases, but declines beyond six parts. The experiments also compare sampling strategies and HC loss with center loss for cross-modality feature alignment.
- The number of parts: When p = 1, TSLFN degenerates into the baseline network that extracts global features.
- The number of parts: Performance improves as p increases initially because narrower local-feature granularity directs the network toward finer details.
- The number of parts: Performance drops when p exceeds 6, partly because excessively fine granularity cannot provide efficient information for discriminative descriptors.
- Sampling strategy: The proposed sampling strategy is more effective than the original strategy, with its benefit increasing as the number of parts grows.
- HC loss comparison: HC loss produces smaller cross-modality center distances than center loss under the same λ, indicating closer feature distributions between modalities.
- HC loss comparison: HC loss outperforms center loss by a large margin on baseline and TSLFN, and the margin increases with λ.
VI. CONCLUSIONS
The paper proposes HC loss and TSLFN for cross-modality person re-identification. Joint CE and HC supervision targets both inter-class discrepancy and intra-class cross-modality similarity, while TSLFN learns local feature representations.
- HC loss constrains distances between heterogeneous-modality centers to improve intra-class cross-modality similarity.
- TSLFN learns discriminative local feature representations from heterogeneous images using a simple two-stream architecture.
- Joint CE and HC supervision enables feature representations targeting inter-class discrepancy and intra-class cross-modality similarity simultaneously.