PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Person Re-identification: A Survey and Outlook

Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, Steven C. H. Hoi

arXiv:2001.04193v2cs.CV

TL;DR

Person Re-ID has achieved strong results in research-oriented closed-world settings but still faces a substantial gap to practical open-world deployment. This survey analyzes both settings, proposes AGW and mINP, and identifies remaining limitations and open issues.

  • Problem

    Person Re-ID remains difficult because practical deployment involves visual variation, domain shift, unseen scenarios, changing cameras, and other conditions beyond controlled benchmarks.

  • Method

    The paper surveys closed-world Re-ID through feature representation learning, metric learning, and ranking optimization, then organizes open-world challenges across practical system components.

  • Results

    AGW achieves state-of-the-art performance on twelve datasets for four Re-ID tasks, while mINP supplements CMC/mAP by measuring the cost of finding all correct matches.

  • Takeaways & Limitations

    The survey provides a strong baseline and an additional evaluation criterion for developing and assessing Re-ID systems across research and practical settings.

  • Takeaways & Limitations

    Performance remains lower on newer or challenging datasets, drops dramatically under cross-dataset evaluation and adversarial attack, and unsupervised Re-ID still trails supervised results in the cited examples.

Abstract

from arXiv · show

Person re-identification (Re-ID) aims at retrieving a person of interest across multiple non-overlapping cameras. With the advancement of deep neural networks and increasing demand of intelligent video surveillance, it has gained significantly increased interest in the computer vision community. By dissecting the involved components in developing a person Re-ID system, we categorize it into the closed-world and open-world settings. The widely studied closed-world setting is usually applied under various research-oriented assumptions, and has achieved inspiring success using deep learning techniques on a number of datasets. We first conduct a comprehensive overview with in-depth analysis for closed-world person Re-ID from three different perspectives, including deep feature representation learning, deep metric learning and ranking optimization. With the performance saturation under closed-world setting, the research focus for person Re-ID has recently shifted to the open-world setting, facing more challenging issues. This setting is closer to practical applications under specific scenarios. We summarize the open-world Re-ID in terms of five different aspects. By analyzing the advantages of existing methods, we design a powerful AGW baseline, achieving state-of-the-art or at least comparable performance on twelve datasets for FOUR different Re-ID tasks. Meanwhile, we introduce a new evaluation metric (mINP) for person Re-ID, indicating the cost for finding all the correct matches, which provides an additional criteria to evaluate the Re-ID system for real applications. Finally, some important yet under-investigated open issues are discussed.

1 INTRODUCTION

Person Re-ID retrieves people across cameras but remains difficult because real scenes involve substantial visual, environmental, and deployment variability. This survey organizes the field around practical system components, closed- versus open-world settings, and new evaluation and baseline contributions.

  • Motivation: Person Re-ID determines whether a queried person appears at another place or time using images, videos, or text descriptions.Its practical importance is tied to intelligent surveillance and public safety.
  • Challenges: Viewpoint, resolution, illumination, pose, occlusion, modality, clutter, camera environments, and unreliable detections create substantial uncertainty.Domain shift, unseen scenarios, incremental updating, changing clothing, and large galleries further complicate deployment.
  • Contributions: The paper contributes an in-depth survey, the AGW baseline, the mINP metric, and discussion of under-investigated open issues.AGW is reported as achieving state-of-the-art performance on twelve datasets for four Re-ID tasks, while mINP measures the cost of finding all correct matches.
  • System Pipeline: A practical Re-ID system proceeds through raw-data collection, bounding-box generation, training annotation, model training, and pedestrian retrieval.Bounding boxes commonly come from detection or tracking, while cross-camera labels support discriminative model learning.
  • Closed- and Open-World Settings: The survey separates closed-world research assumptions from open-world applications and compares them across practical system dimensions.These dimensions include data modality, input format, annotation, and other deployment conditions.

2 CLOSED-WORLD PERSON RE-IDENTIFICATION

Closed-world Re-ID studies deep representations under controlled assumptions, including visible-camera inputs, bounding boxes, sufficient labeled data, correct annotations, and known identities. The surveyed representation strategies include global, local, auxiliary, and video features, each addressing different information or robustness needs.

  • Setting Assumptions: Closed-world Re-ID assumes visible-camera image or video inputs, mostly identity-pure bounding boxes, sufficient correct annotations, and query identities present in the gallery.These assumptions define the widely studied research-oriented setting.
  • Global Feature Representation Learning: Global feature learning extracts one feature vector for each person image without additional annotation cues.It was an early primary choice for applying deep learning to Re-ID.
  • Local Feature Representation Learning: Local feature learning aggregates part-level features to capture fine-grained information and improve robustness to misalignment.Human parsing or pose estimation provides semantic parts but can introduce noisy detections, while horizontal stripes are more flexible but vulnerable to occlusion and clutter.
  • Auxiliary Feature Representation Learning: Auxiliary feature learning uses attributes, viewpoints, domain information, generated images, or augmented samples to reinforce representation learning.Data augmentation methods enrich supervision with varied training samples and can improve testing-set generalizability.
  • Video Feature Representation Learning: Video feature learning represents each person with multiple frames, combining appearance and temporal information while requiring methods to handle noisy or outlier tracking frames.Attention, consensus, diversity regularization, affine-hull modeling, and occlusion completion are surveyed as responses to these challenges.

2.2 Deep Metric Learning

Deep metric learning for person Re-ID has shifted from hand-crafted distance functions toward deep loss designs that guide feature representation learning. The survey organizes these designs around identity, verification, and triplet losses, alongside sampling, memory, and adaptive multi-loss strategies.

  • Deep Re-ID loss functions have largely replaced hand-crafted Mahalanobis-distance and projection-matrix learning as the main mechanism guiding feature representation.
  • Identity Loss: Identity loss treats Re-ID training as multi-class classification, using each identity as a distinct class and extracting testing features from the pooling or embedding layer.It is generally easy to train and automatically mines hard samples.
  • Verification Loss: Verification loss optimizes pairwise relationships through contrastive or binary verification objectives that distinguish positive and negative image pairs.Contrastive loss uses Euclidean embedding distance, a binary pair label, and a margin parameter.
  • Triplet Loss: Triplet loss formulates training as retrieval ranking, requiring an anchor-positive distance to be smaller than an anchor-negative distance by a predefined margin.Because easy triplets can dominate optimization, informative mining methods select harder triplets; quadruplet networks further add two mined negatives.
  • Combined Losses: Identity and triplet losses are frequently combined because the survey describes them as mutually beneficial for discriminative feature representation learning.
  • Other Losses: Memory-bank OIM loss compares online instance features with stored labeled and unlabelled identity features, supporting settings with many non-target identities.A temperature parameter controls the similarity space, and the memory scheme is also used in unsupervised domain-adaptive Re-ID.
  • Training Strategies: Identity sampling selects several images from randomly chosen identities in each batch, enabling informative positive and negative mining despite imbalanced pairs.Adaptive sampling, sample re-weighting, and reference constraints provide alternative ways to adjust sample contributions.
  • Training Strategies: Adaptive multi-loss training dynamically reweights identity and triplet losses and reports consistent performance gains.

2.3 Ranking Optimization

Ranking optimization improves person Re-ID retrieval after an initial ranking list is produced. The survey covers gallery-based re-ranking, query-adaptive retrieval, human interaction, and fusion of multiple ranking lists.

  • Ranking optimization improves retrieval by reordering an initial list through gallery-to-gallery similarity mining, human interaction, or rank and metric fusion.
  • Re-ranking: Re-ranking exploits gallery-to-gallery similarity to pull top-ranked similar items, push dissimilar items, and model contextual or manifold information.Examples include k-reciprocal, geometric-manifold, cross-neighborhood, and local-blurring methods.
  • Query Adaptive: Query-adaptive retrieval replaces uniform searching with query-specific strategies, including locality-preserving projections and local metric adaptation using mined negatives.
  • Human Interaction: Human-interaction methods use feedback as supervision during re-ranking, with incremental learning improving ranking performance on the fly.
  • Rank Fusion: Rank fusion combines multiple ranking lists using similarity, dissimilarity, query-adaptive late fusion, or graph-based consensus decisions.

2.4 Datasets and Evaluation

Closed-world Re-ID research evaluates image- and video-based methods on growing benchmark datasets using CMC and mAP, with recent deep-learning systems achieving strong results while remaining less robust on challenging settings. Analyses highlight part-level features, attention, multi-loss training, and spatial-temporal modeling as recurring design patterns.

  • Datasets: The survey covers 11 image datasets and 7 video datasets, emphasizing general large-scale datasets for deep-learning methods.Dataset scale and camera counts have increased, while annotation and model-generalization challenges also grow.
  • Evaluation Metrics: CMC-k measures whether a correct match appears among the top k results, whereas mAP evaluates retrieval across multiple ground truths.CMC is limited when galleries contain multiple ground truths; mAP captures performance on additional hard matches.
  • Image-based Re-ID: 96.2% Rank-1 accuracy and 91.6% mAP are reported for VAL on Market-1501, exceeding the cited human Rank-1 accuracy of 93.5%.The reported setting does not use re-ranking or additional annotated information, while viewpoint information is identified as VAL’s major advantage.
  • Image-based Re-ID: Part-level features, attention, and multi-loss training recur among strong image-based methods, addressing misalignment, occlusion, and discriminative representation learning.Recent methods commonly aggregate part-level and full-body features; attention schemes appear in methods achieving the best performance on each dataset.
  • Video-based Re-ID: 88.5%/82.3% Rank-1 accuracy/mAP is reported for STC on MARS, while STA reaches 96.2% Rank-1 accuracy and 94.9% mAP on Duke-Video.Video methods commonly use spatial-temporal aggregation and attention across multiple frames.
  • Video-based Re-ID: Video benchmark gains have generally saturated below 1% accuracy, while LS-VID remains challenging at 63.1%/44.43% Rank-1 accuracy/mAP.LS-VID contains substantially more identities and video sequences than the other discussed benchmarks.

3 OPEN-WORLD PERSON RE-IDENTIFICATION

Open-world person Re-ID addresses practical settings that relax closed-world assumptions, including heterogeneous modalities, raw-image/video inputs, limited labels, noisy annotations, and absent gallery matches. The survey reviews these challenges and methods, highlighting substantial unsupervised progress alongside remaining gaps.

  • Scope: Open-world Re-ID covers heterogeneous modalities, end-to-end processing from raw images or videos, limited-label learning, noisy annotations, and open-set retrieval.The survey organizes open-world Re-ID into five aspects, including depth-RGB, text-image, visible-infrared, cross-resolution, raw-image/video, tracking, unsupervised, noise-robust, and open-set tasks.
  • Heterogeneous Re-ID: Depth, infrared, text, and resolution variants address settings where visual conditions or query modalities differ from standard visible-camera imagery.Depth supports illumination- and clothing-changing environments; text-to-image retrieval applies when no query image is available; visible-infrared matching targets low-light conditions; cross-resolution Re-ID handles resolution variation.
  • End-to-end Re-ID: End-to-end Re-ID jointly handles detection and identification from raw images or videos, while multi-camera tracking links person hypotheses across cameras.Joint detection and re-identification are challenging because the two components have different focuses; tracking methods combine holistic appearance and pose-related information.
  • Unsupervised and Domain Adaptation: Unsupervised Re-ID methods estimate cross-camera labels or adapt models across domains, using approaches such as graph matching, camera constraints, and target-image generation.Target-image generation transfers source-domain images into target-domain styles to enable supervised learning with unlabeled target data.
  • Unsupervised Re-ID Results: 90.3%/76.7% Rank-1 accuracy/mAP on Market-1501 and 82.9%/68.8% on DukeMTMC mark increases from 54.5%/26.3% and 30.0%/16.4%, respectively, for unsupervised Re-ID.The survey reports these gains over three years and says the gap with the supervised upper bound narrowed significantly.
  • Open Issues: Unsupervised Re-ID remains under-developed, with attention, target-domain generation, and annotated source data identified as opportunities for improvement, while a supervised–unsupervised gap persists.On Market-1501, supervised ConsAtt reaches 96.1% Rank-1 accuracy compared with about 90.3% for unsupervised SpCL.

4 AN OUTLOOK: RE-ID IN NEXT ERA

This outlook introduces mINP and the AGW baseline as guidance for future Re-ID research, then evaluates AGW across single- and cross-modality tasks while highlighting practical open challenges.

  • 4.1 mINP: A New Evaluation Metric for Re-ID: mINP measures the penalty of finding the hardest correct match and supplements CMC and mAP for practical Re-ID evaluation.Its calculation is efficient, but value differences become smaller with large gallery sizes.
  • 4.2 A New Baseline for Single-/Cross-Modality Re-ID: AGW combines non-local attention, generalized-mean pooling, and weighted regularization triplet loss on top of BagTricks.The framework uses ResNet50 as its backbone and adopts the batch-normalization output as the testing feature representation.
  • 4.2 A New Baseline for Single-/Cross-Modality Re-ID: AGW improves substantially over BagTricks under various metrics, while achieving 45.7% mINP versus 42.1% for ABD-Net on DukeMTMC.The authors interpret the higher mINP as requiring less effort to find all correct matches.
  • 4.2 A New Baseline for Single-/Cross-Modality Re-ID: AGW+ achieves better Rank-1, mAP, and mINP than Co-Seg in most single-modality video Re-ID cases.AGW+ adds frame-level average pooling for sequence representation and constraint random sampling during training.
  • 4.2 A New Baseline for Single-/Cross-Modality Re-ID: AGW achieves comparable performance on partial Re-ID and much higher accuracy than existing models on visible-infrared cross-modality Re-ID.These experiments cover open-world partial and cross-modality tasks.
  • 4.3 Open Issues: Practical open-world Re-ID must handle heterogeneous modalities, changing resolutions, domain gaps, and unseen datasets.The survey identifies domain-generalizable models that transfer to unseen datasets without additional training as a practical direction.

5 CONCLUDING REMARKS

The paper surveys person Re-ID across closed-world and open-world settings, then proposes AGW and mINP while identifying open-world challenges as an important future direction.

  • 5 CONCLUDING REMARKS: The survey analyzes closed-world Re-ID through feature representation learning, deep metric learning, and ranking optimization, alongside open-world practical challenges.It reports performance saturation in several closed-world datasets and increasing attention to open-world settings.
  • 5 CONCLUDING REMARKS: AGW achieves competitive performance across four Re-ID tasks under various metrics and provides a strong baseline for future improvements.The paper also introduces an evaluation metric measuring the cost of finding all correct matches.

Supplemental Materials:

The supplementary material provides implementation details and expanded experiments for closed-world image and video Re-ID and open-world cross-modality and partial Re-ID.

  • Supplementary Materials: Supplementary experiments cover image-based Re-ID on four datasets and video-based Re-ID on four datasets.These are described as single-modality closed-world Re-ID tasks.
  • Supplementary Materials: Supplementary comparisons cover visible-infrared cross-modality Re-ID and partial Re-ID on two datasets each.These are presented as open-world Re-ID tasks.

A. Experiments on Single-modality Image-based Re-ID

The supplementary image-based AGW experiments specify its ResNet50 architecture, attention blocks, training losses, label smoothing, and optimization schedule.

  • Architecture Design: AGW uses ImageNet-pretrained ResNet50, changes the final downsampling stride from 2 to 1, and adjusts the fully connected layer for the training identities.The modified stride increases the spatial size of the output feature map.
  • Non-local Attention: Five dot-product non-local attention blocks are inserted after selected residual blocks, each using a 512-channel bottleneck.A BatchNorm layer follows the last linear layer representing Wz in each block.
  • Training Loss: Training combines identity classification loss, center loss, and weighted regularization triplet loss.The total loss uses balanced weights β1 = 0.0005 and β2 = 1.0 for the center and triplet terms.
  • Training Loss: Label smoothing replaces one-hot targets to reduce confidence in the true identity label and help prevent classification overfitting.The smoothing constant ϵ is set to 0.1.
  • Optimizer Setting: The model is trained with Adam for 120 epochs using weight decay, scheduled learning-rate decreases, and a ten-epoch warm-up.The learning rate increases from 3.5 × 10^-5 to 3.5 × 10^-4 during warm-up, then decreases at epochs 40 and 70.

B. Experiments on Video-based Re-ID

The video-based AGW baseline extends the image-based model by aggregating frame-level features, while AGW+ further models temporal information and adjusts training. Across four datasets, AGW variants achieve competitive or superior results against several baselines and state-of-the-art methods.

  • Implementation Details: The video-based AGW baseline averages frame-level feature vectors into a video-level representation before the BNNeck layer.Dense sampling uses all frames in a tracklet, improving performance at higher computational cost.
  • Detailed Comparison: AGW achieves competitive results on MARS and DukeVideo using video sequences with average frame pooling.The baseline also significantly outperforms BagTricks under multiple evaluation metrics.
  • Detailed Comparison: AGW+ substantially improves performance and remains competitive on PRID2011 and iLIDS-VID after modeling more temporal information and adjusting training.AGW+ outperforms most state-of-the-art methods on MARS, DukeVideo, and PRID2011.
  • Detailed Comparison: The comparison reports Rank at r accuracy, mAP, and mINP on SYSU-MM01 under the single-shot query setting.The supplied table caption specifies the reported metrics and query protocol but provides no numerical values here.

C. Experiments on Cross-modality Re-ID

The cross-modality AGW model uses modality-specific early processing and shared later features, with joint identity and weighted triplet losses. It consistently outperforms existing methods on the evaluated datasets without time-consuming image generation.

  • Architecture Design: The cross-modality backbone uses two streams: the first block captures modality-specific information, while later shared blocks learn modality-sharable features.This differs from the one-stream architecture used for single-modality Re-ID.
  • Training Strategy: Training samples 8 identities with 4 visible and 4 infrared images each, enabling hard positive and negative mining within and across modalities.Each batch contains 32 visible and 32 infrared images.
  • Training Loss: The model combines identity classification loss with weighted regularized triplet loss and uses a shared identity classifier for both modalities.The two losses receive equal weight, and the pooling parameter p_k is set to 3.
  • Optimizer Setting: The experiments use SGD rather than Adam because Adam produced much lower performance on the cross-modality task.The SGD momentum is 0.9, with a 60-epoch schedule and warm-up learning rate.
  • Detailed Comparison: The proposed AGW consistently outperforms current state-of-the-art cross-modality methods on SYSU-MM01 and RegDB without image generation.Its performance generally remains consistent across the two RegDB query settings.

D. Experiments on Partial Re-ID

For partial Re-ID, AGW is trained on Market-1501 and evaluated on Partial-REID and Partial-iLIDS using the single-modality image-based design. It improves over BagTricks and remains competitive despite relying only on global features.

  • Implementation Details: The partial-ReID AGW model keeps the single-modality image-based backbone and training strategy, training on Market-1501 before evaluation on two partial-ReID datasets.Partial-REID and Partial-iLIDS provide query and gallery sets without their own training sets.
  • Detailed Comparison: AGW is compared with DSR, SFR, and VPM on Partial-REID and Partial-iLIDS using Rank-1, Rank-3, and mINP.The supplied table caption identifies the datasets and metrics but does not provide numerical values here.
  • Detailed Comparison: AGW achieves competitive state-of-the-art performance on both partial-Re-ID datasets when using only global features.VPM achieves very high performance by modeling visibility and extracting region-level features.
  • Detailed Comparison: AGW significantly improves over BagTricks under multiple evaluation metrics, demonstrating effectiveness for partial Re-ID.This comparison is reported for both Partial-REID and Partial-iLIDS.

E. Overview of This Survey

The survey organizes person Re-ID around five system-development steps and separates closed-world from open-world settings. It analyzes closed-world methods, summarizes open-world challenges, introduces mINP and AGW, and discusses future open issues.

  • Closed-World Setting: Closed-world Re-ID is analyzed through feature representation learning, deep metric learning, and ranking optimization.The survey also summarizes image- and video-based datasets and state-of-the-art methods.
  • Outlook: The survey introduces mINP to evaluate the difficulty of finding all correct matches and develops AGW as a baseline for future work.AGW achieves competitive performance on four Re-ID tasks, while the survey also discusses under-investigated open issues.
  • Outlook: The paper combines a comprehensive state-of-the-art summary with future-direction analysis intended to guide further Re-ID development.The stated scope includes different subtasks and an outlook for future person Re-ID.
  • Overview: The survey overview contains three components: closed-world Re-ID, open-world Re-ID, and an outlook for the next era.These components correspond to Sections 2, 3, and 4.
Loading 2001.04193v2…