PaperlayerView exact revision on arXiv

Source-linked AI summary

When Person Re-identification Meets Changing Clothes

Fangbin Wan, Yang Wu, Xuelin Qian, Yixiong Chen, Yanwei Fu

arXiv:2003.04070v3cs.CV

TL;DR

Person ReID must identify people despite clothing changes, but prior benchmarks largely overlook this setting and encourage reliance on clothing appearance. The paper introduces real and synthetic benchmarks, then shows that clothing changes hinder representation learning and generalization to unseen clothes while motivating preliminary robustness solutions.

  • Problem

    Clothing inconsistency is an overlooked ReID setting, while common benchmarks’ clothing consistency encourages reliance on appearance rather than identity-sensitive features.

  • Method

    The paper constructs a small real dataset and a larger GTA5-based synthetic dataset, then uses them to study clothing inconsistency and preliminary robustness approaches.

  • Results

    Clothing changes make ReID harder by hindering representation learning and challenging generalization to people wearing unseen clothes.

  • Takeaways & Limitations

    The two benchmark datasets and preliminary solutions provide baselines for further research on clothing-robust person ReID.

  • Takeaways & Limitations

    The real dataset is small because collecting large-scale real-scene data is difficult, so it is used for testing rather than deep-learning training.

Abstract

from arXiv · show

Person re-identification (ReID) is now an active research topic for AI-based video surveillance applications such as specific person search, but the practical issue that the target person(s) may change clothes (clothes inconsistency problem) has been overlooked for long. For the first time, this paper systematically studies this problem. We first overcome the difficulty of lack of suitable dataset, by collecting a small yet representative real dataset for testing whilst building a large realistic synthetic dataset for training and deeper studies. Facilitated by our new datasets, we are able to conduct various interesting new experiments for studying the influence of clothes inconsistency. We find that changing clothes makes ReID a much harder problem in the sense of bringing difficulties to learning effective representations and also challenges the generalization ability of previous ReID models to identify persons with unseen (new) clothes. Representative existing ReID models are adopted to show informative results on such a challenging setting, and we also provide some preliminary efforts on improving the robustness of existing models on handling the clothes inconsistency issue in the data. We believe that this study can be inspiring and helpful for encouraging more researches in this direction. The dataset is available on the project website: https://wanfb.github.io/dataset.html.

1. Introduction

The paper identifies clothes inconsistency as an overlooked challenge that exposes ReID models’ reliance on clothing appearance. It introduces real and synthetic benchmarks, studies the resulting difficulties, and offers preliminary robustness solutions.

  • Motivation: ReID searches for specific people across surveillance cameras despite substantial appearance changes among similar-looking candidates.The task is valuable for public safety but technically challenging because identities must be distinguished under changing appearances.
  • Problem: Existing benchmarks often preserve clothing consistency, encouraging models to learn clothing appearance instead of identity-sensitive biological features.Clothing occupies much of the body, so models can fail when people change clothes or wear similar clothing.
  • Problem: Clothes inconsistency has been overlooked because ReID was traditionally assumed to operate over short periods and large realistic datasets are expensive to collect.The short-period assumption overlooks coat, bag, or outfit changes, while privacy constraints make large-scale real data difficult to gather.
  • Datasets: The paper builds a small real benchmark and a larger synthetic VC-Clothes dataset to systematically study clothes inconsistency in person ReID.The real dataset contains 28 volunteers; VC-Clothes provides realistic data with 512 identities and 19,060 images across four scenes.
  • Findings: Experiments show that clothing changes hinder representation learning and generalization to people wearing unseen clothes.The paper also presents preliminary methods for learning more identity-sensitive and clothing-insensitive features.
  • Contributions: The study contributes two benchmark datasets and preliminary solutions intended to support further research on clothing-robust ReID.These contributions address the historically overlooked same-identity clothing inconsistency problem.

2. Related Work

Prior work provides limited resources for studying clothing inconsistency in ReID. The paper distinguishes its datasets by combining realistic clothing changes with broader support for this problem.

  • Existing datasets: Few ReID studies address clothing inconsistency, and existing datasets have important limitations in scale, modality, environment, or realism.RGB-D is indoor-only and small, iQIYI-VID uses Internet videos, and PRCC uses a laboratory setting with simple backgrounds.
  • Existing datasets: RGB-D addresses clothing inconsistency with depth information but contains only 79 identities and 1-2 clothes per identity.Its depth cue is applicable only in indoor environments, limiting broader surveillance use.
  • Existing datasets: iQIYI-VID offers 600K video clips from 5,000 celebrities, but its Internet-video source differs from surveillance-camera data.Its strengths include scale and variation in pose, face quality, clothing, and makeup.
  • Dataset examples: Real28 provides samples across different days, while VC-Clothes depicts the same people across four cameras and up to three suits.These visual samples illustrate the intended clothing and viewpoint variation in the two datasets.
  • Existing datasets: PRCC contains 33,698 real images from 221 identities but is collected indoors against simple laboratory backgrounds.The paper therefore proposes a large synthetic dataset intended to better represent real demands.
  • Proposed datasets: The proposed datasets contain 512 identities with 1~3 clothing suits per identity and are designed to support research on clothing inconsistency.Synthetic data enables automated generation when ideal real-world datasets require substantial human labor.

3. New Benchmark Datasets

The paper introduces Real28, a small real-world dataset, and VC-Clothes, a larger synthetic benchmark designed to study person re-identification under changing clothes. The datasets vary clothing alongside cameras, scenes, illumination, occlusion, and pose, supporting systematic evaluation of ReID models.

  • Real28 Dataset: Real28 contains 4,324 images from 28 identities across three days, four cameras, two indoor scenes, and two outdoor scenes.Because it is too small to train deep models, the authors recommend using Real28 for evaluation.
  • VC-Clothes Dataset: GTA5 enables controlled rendering of four indoor and outdoor scenes with configurable clothing, illumination, viewpoints, backgrounds, and cameras.The synthetic construction changes clothing while preserving identity, with each identity assigned one to three outfits.
  • Dataset Challenges: The benchmark captures clothes inconsistency together with similar clothing across people, occlusion, illumination changes, and pose variance.These factors create the principal visual challenges for identifying people independently of clothing.
  • VC-Clothes Dataset: VC-Clothes contains 512 identities, four scenes, approximately nine images per scene per identity, and 19,060 images.The dataset is split evenly by identity into 256 training and 256 testing identities.
  • Benchmark Evaluation: VC-Clothes is evaluated against CUHK03 using representative handcrafted and deep ReID methods, including LOMO+XQDA, GOG+XQDA, MDLA, PCB, Part-aligned, and ResNet50.The evaluation uses mAP and Rank-1, Rank-5, and Rank-10 accuracy.

4. Influence of Clothing Change

Clothing changes substantially reduce person ReID performance and expose weaknesses in learned representations and generalization to unseen clothes. Models trained with clothing inconsistency generalize across both consistent and changing-clothes settings more reliably.

  • 4.1. Making ReID a Much Harder Problem: Clothing inconsistency causes a 30% to 50% performance drop for DNN-based methods in mAP and Rank-1 accuracy, with larger drops for hand-crafted methods.The comparison uses Same Clothes and Change Clothes settings across camera pairs.
  • 4.1. Making ReID a Much Harder Problem: The primary cause of the dramatic performance decrease is changing clothes, although camera diversity may also influence results.
  • 4.1. Making ReID a Much Harder Problem: Part-aligned modeling has the smallest performance drop among compared methods, indicating that finer and more detailed appearance modeling helps handle clothing changes.
  • 4.2. Changing Learned Representations: Under clothing changes, part maps attend more clearly to faces and feet, suggesting identity-sensitive representations require details from smaller body regions rather than large body areas.Same-clothes appearance maps are brighter and more scattered than change-clothes maps.
  • 4.3. Generalization to Unseen Clothes: Unseen suits create significant extra difficulties, limiting model generalization even when querying with a new camera is not itself a major issue.Models are trained on Cam3 and Cam4, then tested on Cam2 with seen suits or Cam1 with unseen suits.
  • 4.4. Benefits of Training with Clothes Inconsistency: Training with all clothes reduces the cross-setting performance loss to about 1%, whereas a same-clothes model tested on all clothes loses 27.7% in Rank-1 and 39.8% in mAP.Models trained with clothing inconsistency perform well in both clothes-consistency and clothes-inconsistency settings.

5. Preliminary Solutions

The paper explores preliminary ways to improve robustness to changing clothes, including richer training data, face-feature fusion, and transfer to real data. These experiments suggest that unseen clothing variation and fine-grained identity cues can improve performance under clothes inconsistency.

  • Enriching Training Data: Adding one unseen suit/person significantly improves all methods, whereas adding one seen suit/person has the opposite effect.The comparison uses training on Cam3 and Cam4, adding either Cam1 or Cam2.
  • Face-Feature Fusion: The 3APF model combines holistic and face features to improve ReID under clothes inconsistency.Its two main components are a holistic feature extractor and a local face feature extractor.
  • Face-Feature Fusion: Holistic features are produced by combining appearance and body-part branches through bi-linear pooling, while face features are extracted separately.The final feature vector is a weighted mean of the two feature types.
  • Face-Feature Fusion: At weight 0.95, the 3APF model reaches 82.1% mAP, outperforming the single face and holistic extractors.The weight controls the relative contribution of the ReID and face features.
  • Transfer to Real Data: Directly applying VC-Clothes-pretrained models to real data performs poorly, but fine-tuning on a third real dataset substantially improves results.Synthetic pretraining also improves models initially trained on real data for clothes-inconsistent tasks, especially on Real28.

6. Conclusions and Discussions

The paper studies clothes inconsistency in person ReID through real and synthetic benchmarks, pilot experiments, preliminary robustness solutions, and transfer to real data. Its conclusion is intended to motivate further research on this setting.

  • Datasets: The study uses a small real dataset for testing and a large GTA5-rendered synthetic dataset for broader investigation.The real dataset is limited by the difficulty of collecting large-scale surveillance data in real scenes.
  • Findings: The benchmark datasets support pilot studies of the influence and importance of clothes inconsistency in person ReID.The paper also investigates straightforward performance improvements and transfer to a real test dataset.
  • Future Work: The authors present the work as a basis for further studies of clothes inconsistency in person ReID.The conclusion explicitly frames the issue as one that future research should continue exploring.
Loading 2003.04070v3…