PaperlayerView exact revision on arXiv

Source-linked AI summary

Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization

Xin Jin, Tianyu He, Kecheng Zheng, Zhiheng Yin, Xu Shen, Zhen Huang, Ruoyu Feng, Jianqiang Huang, Xian-Sheng Hua, Zhibo Chen

arXiv:2103.15537v4cs.CV

TL;DR

CC-ReID must match people across long-term clothing changes, but single-image settings lack the video sequences commonly used for gait recognition. GI-ReID uses predicted gait as an auxiliary regulator for image ReID, with semantic consistency regularization, and reports state-of-the-art performance across image-based CC-ReID benchmarks.

  • Problem

    Long-term clothing changes undermine image-based ReID, while using gait for CC-ReID from a single image remains under-studied and challenging under viewpoint variation and occlusion.

  • Method

    GI-ReID jointly trains an image ReID-Stream and auxiliary Gait-Stream, predicts gait sequences from one image, and applies high-level semantic consistency regularization before discarding Gait-Stream at inference.

  • Results

    GI-ReID achieves state-of-the-art performance on image-based cloth-changing ReID benchmarks.

  • Takeaways & Limitations

    Gait can serve as a training-time regulator for learning cloth-agnostic image ReID representations without retaining the Gait-Stream during inference.

  • Takeaways & Limitations

    GSP predictions can be inaccurate under occlusion, partial observation, and multi-person images because gait and ReID data differ in viewpoints and environments.

Abstract

from arXiv · show

Cloth-Changing person re-identification (CC-ReID) aims at matching the same person across different locations over a long-duration, e.g., over days, and therefore inevitably meets challenge of changing clothing. In this paper, we focus on handling well the CC-ReID problem under a more challenging setting, i.e., just from a single image, which enables high-efficiency and latency-free pedestrian identify for real-time surveillance applications. Specifically, we introduce Gait recognition as an auxiliary task to drive the Image ReID model to learn cloth-agnostic representations by leveraging personal unique and cloth-independent gait information, we name this framework as GI-ReID. GI-ReID adopts a two-stream architecture that consists of a image ReID-Stream and an auxiliary gait recognition stream (Gait-Stream). The Gait-Stream, that is discarded in the inference for high computational efficiency, acts as a regulator to encourage the ReID-Stream to capture cloth-invariant biometric motion features during the training. To get temporal continuous motion cues from a single image, we design a Gait Sequence Prediction (GSP) module for Gait-Stream to enrich gait information. Finally, a high-level semantics consistency over two streams is enforced for effective knowledge regularization. Experiments on multiple image-based Cloth-Changing ReID benchmarks, e.g., LTCC, PRCC, Real28, and VC-Clothes, demonstrate that GI-ReID performs favorably against the state-of-the-arts. Codes are available at https://github.com/jinx-USTC/GI-ReID.

1. Introduction

Cloth-changing disrupts image-based person re-identification, especially in long-term settings, while gait offers cloth-independent biometric cues. GI-ReID addresses single-image CC-ReID by using an auxiliary gait stream to regularize image representations during training.

  • Motivation: Long-term clothing changes cause significant performance degradation for ReID methods that assume consistent query and gallery clothing.The paper links this degradation to large clothing variations across long-duration datasets.
  • Motivation: Contour- and body-shape-based CC-ReID methods remain vulnerable to estimation errors from single-view inference.These approaches attempt to remove clothing dependence but can be unreliable when inferred from one view.
  • Motivation: Gait provides unique, relatively cloth-invariant motion cues that can support identity matching across different clothing and long-distance, low-quality imaging.The paper contrasts gait with appearance characteristics such as face, body shape, and contour.
  • Research gap: Existing gait studies mainly require video sequences, creating latency and storage costs while leaving single-image CC-ReID under-studied under viewpoint variation and occlusion.The paper identifies single-image gait-assisted CC-ReID as a more challenging setting.
  • Proposed approach: GI-ReID combines an image ReID-Stream with an auxiliary Gait-Stream that learns cloth-agnostic representations from one image and is removed during inference.A Gait Sequence Prediction module forecasts continuous gait frames to enrich gait information.
  • Reported outcome: GI-ReID achieves state-of-the-art performance on image-based cloth-changing ReID benchmarks while retaining an inference design that discards the auxiliary gait stream.The framework is evaluated with multiple ReID-specific network baselines.

2. Related Work

Related work addresses ReID through static appearance, shape, contour, or gait sequences, but these strategies do not directly resolve single-image CC-ReID under difficult imaging conditions. GI-ReID instead predicts gait sequences and uses gait-derived features to regularize image ReID features.

  • General ReID: General ReID methods rely substantially on static spatial texture information, which becomes unreliable when clothing changes.The related work also considers viewpoint, pose, semantic, and occlusion misalignment.
  • Gait recognition: Gait recognition is cloth-independent but typically uses gait sequences and cannot be directly applied to image-based CC-ReID.Large viewpoint variations, occlusions, and complex environments make direct gait recognition unsuitable for this setting.
  • GI-ReID: GI-ReID feeds a single image’s extracted silhouette into Gait-Stream, predicts a gait sequence, and extracts discriminative gait features with GaitSet.The predicted sequence augments gait information before recognition.
  • Gait Sequence Prediction: The Gait Sequence Prediction module uses an encoder-decoder and middle-frame position information to reduce ambiguity when forecasting continuous gait frames.The input silhouette is treated as the middle frame of a fixed-length predicted sequence.
  • Gait Sequence Prediction: GSP predicts a fixed number N of gait frames and uses an L2 prediction loss to match the predicted sequence to ground truth.Predicted frame height and width match the input silhouette dimensions.
  • Joint learning: Semantics Consistency aligns transformed gait and ReID features by minimizing their mean and variance discrepancies with an MMD-based constraint.This regularization encourages the ReID stream to emphasize cloth-independent gait biometrics.

4. Experiment

Experiments across cloth-changing ReID datasets evaluate GI-ReID’s gait assistance, sequence prediction, semantic consistency, inference strategies, and compatibility with existing ReID networks. Results show that gait-based regularization improves cloth-changing recognition while allowing the auxiliary Gait-Stream to be discarded at inference.

  • Datasets, Metrics and Setups: GI-ReID is evaluated on Real28, VC-Clothes, LTCC, PRCC, and MARS using CMC Rank-1/-10/-20 and mAP.Real28, VC-Clothes, LTCC, and PRCC support image-based cloth-changing evaluation, while MARS highlights the difficulty of image-based CC-ReID.
  • Ablation Study: All Gait-Stream schemes improve mAP by over 2.7% over Baseline, while GSP adds 3.3%/6.7%/2.3% mAP on Real28/VC-Clothes/LTCC over the non-GSP scheme.The ablation attributes the additional gains to gait sequence prediction augmenting gait information; semantic consistency produces the best performance in most evaluation cases.
  • Ablation Study: Silhouette-ReID is 1.06% lower in mAP than Baseline on LTCC, indicating that directly adding silhouettes does not explain GI-ReID’s cloth-changing gains.The authors associate direct silhouette use with increased attention to foreground appearance and clothing color.
  • Design Choices in Our GI-ReID Framework: GI-ReID performs best with predicted sequence length N = 8, middle-frame input exceeds arbitrary-position input by 2.3% mAP, and LMSE replacement lowers mAP by nearly 2.0%.These studies support the selected GSP length, middle-frame design, and distribution-level MMD regularization.
  • Design Choices in Our GI-ReID Framework: Using aligned feature fusion improves mAP by 0.4%/0.5% over GI-ReID but requires Gait-Stream inference, whereas GI-ReID retains lower computation and beats reconstructed-vector inference by 1.3% mAP.Direct predicted gait sequence use is unsatisfactory, and reconstructed ReID vectors lose information.
  • Comparison with State-of-the-Arts: GI-ReID (OSNet) surpasses PRCC-contour by 3.17% Rank-1, Gait-Stream improves LTCC-shape by 1.79%/1.59% mAP, and GI-ReID (ResNet-50) gains 2.28%/2.13% mAP over Baseline.The results also indicate compatibility with existing ReID-specific networks, including OSNet.

5. Conclusion

GI-ReID uses unique gait as an auxiliary regulator for cloth-agnostic representation learning in single-image cloth-changing ReID. GSP and SC support this gait-involved two-stream framework, whose effectiveness is demonstrated across multiple benchmarks.

  • GI-ReID uses an auxiliary Gait-Stream as a regulator to encourage cloth-agnostic learning in the image-based ReID-Stream.
  • The Gait-Stream is discarded during inference, while the ReID-Stream remains the deployed representation-learning component.
  • GSP enriches gait utilization by generating gait information, and SC enforces high-level semantic consistency between the two streams.
  • Extensive experiments on multiple cloth-changing ReID benchmarks demonstrate GI-ReID’s effectiveness and superiority.

Supplementary

GSP fine-tuning uses resized and zero-padded person masks because gait training data typically have a 1:1 height-width ratio.

  • Figure 5 applies resize+zero padding to person masks when fine-tuning GSP on ReID-specific datasets.
  • Raw gait training data typically use a 1:1 height-width ratio, which is important for satisfactory GSP gait prediction.

1. Detailed Network Structures of GI-ReID

GI-ReID combines a flexible ReID-Stream with a Gait-Stream containing GSP and GaitSet, using predicted gait sequences and position-aware processing to provide cloth-independent motion cues.

  • GI-ReID consists of an auxiliary Gait-Stream and a mainstream ReID-Stream, whose backbone can use common or ReID-specific architectures.
  • The GSP module contains an encoder, decoder, position embedder, and feature aggregator for gait information processing.
  • The encoder uses four convolutional layers followed by batch normalization and ReLU, producing a 100-dimensional feature through a fully connected layer.
  • GSP uses position embedding and feature aggregation to incorporate the input middle-frame index and reduce prediction ambiguity.
  • The decoder mirrors the encoder with a fully connected reshaping stage, four deconvolutional up-sampling layers, and a final sigmoid output.
  • GaitSet processes silhouette-frame sets through CNN features, set pooling, and horizontal pyramid mapping for multi-scale spatial discrimination.

2. Training Details of our GI-ReID

GI-ReID training proceeds through staged pre-training and joint optimization of GaitSet, GSP, Gait-Stream, SC layers, and the ReID-Stream using gait and ReID datasets.

  • Phase-1: Pre-training for GaitSet: GaitSet is first pre-trained for 80 epochs on aligned 64 × 44 silhouettes with set cardinality 30 using Adam optimization.
  • Phase-2: Joint Training for GSP module and GaitSet: GSP and GaitSet are jointly trained after GaitSet pre-training, using middle-frame inputs and position and prediction losses before adding separate triplet supervision.
  • Phase-3: Joint Training for Gait-Stream and ReID-Stream: The full GI-ReID framework jointly trains GSP, GaitSet, SC layers, and the ReID backbone on ReID data for 240 epochs.
  • Phase-3: Joint Training for Gait-Stream and ReID-Stream: The total objective jointly includes position, prediction, separate triplet, classification, hard-mined triplet, MMD, and reconstruction terms.
  • Phase-3: Joint Training for Gait-Stream and ReID-Stream: Phase-3 uses classification and hard-mined triplet losses for ReID-Stream training, while Gait-Stream losses receive weight 0.1.

3. Details of Datasets

The experiments use one video ReID dataset and four image-based cloth-changing ReID datasets, spanning real and synthetic scenarios with varied identities, cameras, clothing, and evaluation protocols.

  • Dataset Overview: The evaluation covers MARS plus the image-based cloth-changing datasets Real28, VC-Clothes, LTCC, and PRCC.MARS is used for video ReID, while the other four datasets target cloth-changing ReID.
  • MARS: MARS contains 20,715 track-lets from 1,261 pedestrians captured by at least two cameras.
  • Real28: Real28 contains 4,324 images of 28 identities collected across three days and four cameras, and is used only for evaluation.Its query and gallery sets contain 336 and 3,988 images, respectively.
  • VC-Clothes: VC-Clothes is a GTA5-rendered dataset with 512 identities, four scenes, and 19,060 images, split equally by identity into training and testing sets.
  • LTCC: LTCC contains 17,138 images from 152 identities, averaging five different clothes per person, with outfit changes ranging from 2 to 14.
  • PRCC: PRCC contains 221 identities across three camera views, with same-clothing images in Cameras A/B and different-clothing images in Camera C.Its test identities are split into gallery and probe sets for single-shot matching.

4. Experimental Results of Different Settings

The experiments evaluate gait assistance under real, general-video, and imitated-cloth-changing settings, comparing gait-stream variants and semantic consistency against baselines.

  • Experimental Settings: The study uses three settings: real cloth-changing image ReID, general video ReID, and imitated cloth-changing video ReID.
  • Inference Efficiency: GI-ReID’s Gait-Stream is unnecessary during inference, supporting computational efficiency while gait prediction and regularization contribute to performance.
  • Imitated Cloth-Changing Video ReID: On CC-MARS, the baseline reaches 68.52% mAP versus 79.12% on raw MARS after synthetic clothing changes.
  • Imitated Cloth-Changing Video ReID: On CC-MARS, Baseline+GS (concat) and Baseline+GS+SC improve the baseline by near 5.0% in mAP.
  • Imitated Cloth-Changing Video ReID: The gait ‘concat’ scheme is slightly superior on CC-MARS, but it retains Gait-Stream during inference and therefore incurs extra computational cost.

5. Comparison with State-of-the-Arts (Complete version)

The state-of-the-art comparisons evaluate GI-ReID on LTCC and PRCC, including same-clothing and cross-clothing conditions. GI-ReID maintains stronger cross-clothing performance than several conventional ReID approaches.

  • Comparison Scope: The complete comparison reports additional approaches and evaluation settings on LTCC and PRCC.
  • PRCC: 85.97% Rank-1 accuracy is achieved by GI-ReID on PRCC’s same-clothes setting, exceeding all handcrafted metric-learning methods and most deep learning methods.
  • PRCC: 37.55% Rank-1 accuracy is achieved by GI-ReID on PRCC under the cloth-changing scenario.
  • PRCC: AlexNet, VGG16, HA-CNN, and PCB show sharp performance drops when clothing changes, highlighting the difficulty of cross-clothing ReID.

6. Study on Failure Cases (Limitations)

GSP can produce inaccurate gait sequences when ReID images differ substantially from gait-training data or contain occlusion, partial views, or multiple people.

  • Failure Conditions: GSP predictions become inaccurate under viewpoint and environment differences between gait and ReID training data.
  • Failure Conditions: Occlusion, partial observations, and multiple people can cause large estimation errors in predicted gait frames.
  • Robustness Strategy: These gait-prediction failures can hurt ReID performance, motivating indirect use of predictions through two-stream knowledge regularization.

7. Social Impact

GI-ReID is presented as a way to bridge image-based ReID research and practical surveillance applications, while raising privacy concerns about surveillance data and deployments.

  • Social impact: GI-ReID uses gait to support cloth-agnostic image-based ReID and is intended to bridge ReID algorithms with practical applications.The framework uses a gait stream as a regulator and discards it during inference.
  • Social impact: The approach targets efficient pedestrian identification from a single image, which supports latency-sensitive surveillance applications.The Gait-Stream is discarded at inference for high efficiency.
  • Social impact: ReID surveillance can raise risks of leaking private information, and collected or downloaded data may violate human privacy.The paper calls for research that understands and mitigates risks arising from surveillance applications.
Loading 2103.15537v4…