PaperlayerView exact revision on arXiv

Source-linked AI summary

Person Re-identification: Past, Present and Future

Liang Zheng, Yi Yang, Alexander G. Hauptmann

arXiv:1610.02984v1cs.CV

TL;DR

Person re-ID seeks to match a person across cameras under substantial appearance variation, while practical systems must also contend with detection, tracking, and large galleries. This survey synthesizes historical, hand-crafted, and deep-learning approaches across image- and video-based re-ID, then emphasizes end-to-end integration and efficient large-scale retrieval. Reported accuracy improved substantially across datasets over time, but evaluation of detector/tracker effects and very large galleries remains limited.

  • Problem

    Person re-ID must match people across cameras despite lighting, pose, and viewpoint changes, while detection/tracking effects and very large-gallery evaluation remain under-explored.

  • Method

    The survey organizes hand-crafted and deeply learned methods across image- and video-based re-ID and reviews end-to-end and large-scale retrieval directions.

  • Results

    Rank-1 accuracy increased by +51.9% on VIPeR and +31.62% on Market-1501 over reported historical results.

  • Takeaways & Limitations

    Person re-ID lies between classification and instance retrieval, combining learned discriminative representations with efficient retrieval approaches for large galleries.

  • Takeaways & Limitations

    Very large-gallery performance evaluation remains lacking, with the identified hashing work tested on CUHK03 splits containing 100 identities per test split.

Abstract

from arXiv · show

Person re-identification (re-ID) has become increasingly popular in the community due to its application and research significance. It aims at spotting a person of interest in other cameras. In the early days, hand-crafted algorithms and small-scale evaluation were predominantly reported. Recent years have witnessed the emergence of large-scale datasets and deep learning systems which make use of large data volumes. Considering different tasks, we classify most current re-ID methods into two classes, i.e., image-based and video-based; in both tasks, hand-crafted and deep learning systems will be reviewed. Moreover, two new re-ID tasks which are much closer to real-world applications are described and discussed, i.e., end-to-end re-ID and fast re-ID in very large galleries. This paper: 1) introduces the history of person re-ID and its relationship with image classification and instance retrieval; 2) surveys a broad selection of the hand-crafted systems and the large-scale methods in both image- and video-based re-ID; 3) describes critical future directions in end-to-end re-ID and fast retrieval in large galleries; and 4) finally briefs some important yet under-developed issues.

1 INTRODUCTION

Person re-ID evolved from multi-camera tracking into a retrieval task focused on matching people across cameras despite appearance changes. This survey reviews its history, methods, relationship to classification and retrieval, and emerging end-to-end and large-gallery directions.

  • 1 INTRODUCTION: Person re-ID identifies whether a queried person has appeared elsewhere or at another time through another camera.The task emerged alongside growing public-safety demands and widespread camera networks.
  • 1 INTRODUCTION: A practical surveillance system comprises person detection, tracking, and retrieval, although most re-ID research focuses on retrieval.The survey uses person re-ID to mean the retrieval module unless otherwise specified.
  • 1.1 Organization of This Survey: The survey emphasizes deep learning, end-to-end re-ID, and very large-scale retrieval while organizing prior work by image/video tasks and hand-crafted/deeply learned systems.It also discusses re-ID’s connections to classification and instance retrieval and identifies detection, tracking, and retrieval integration as a future direction.
  • 1.2 A Brief History of Person Re-ID: Re-ID began within multi-camera tracking, acquired explicit terminology in 2005, and separated into image-based visual matching in 2006.Early approaches used appearance, color, spatial-temporal cues, and foreground or interest-point representations.
  • 1.2 A Brief History of Person Re-ID: Using multiple frames per person improved video re-ID over single-frame matching, while accuracy saturated as more frames were selected.The 2010 multi-shot methods randomly selected frames and compared bounding-box sets using color and related features.
  • 1.2 A Brief History of Person Re-ID: Deep learning entered re-ID in 2014 through siamese networks, and later CNN systems became popular despite unstable performance on small datasets.Early methods differed in cost functions and body partitioning, and their datasets did not overlap directly.

2 IMAGE-BASED PERSON RE-ID

In the closed-world image-based formulation, a query is matched against a gallery of images representing different identities. The predicted identity is the gallery item with the highest similarity to the query.

  • 2 IMAGE-BASED PERSON RE-ID: The gallery contains N images assigned to N different identities, and a probe image q is used to determine the matching identity.This is presented as a closed-world toy model for single-image re-ID.
  • 2 IMAGE-BASED PERSON RE-ID: The predicted identity i* is the gallery identity maximizing sim(q, g_i).Here, sim(·, ·) denotes a similarity function between the query and a gallery image.

2.1 Hand-crafted Systems

Hand-crafted re-ID systems combine pedestrian descriptors with learned similarity functions. The surveyed methods span low-level color and texture features, attribute-based representations, metric learning, subspace learning, and alternative discriminative rankers.

  • Pedestrian Description: Toy hand-crafted re-ID systems require both image description and a distance metric.The survey identifies these as the two necessary components of a basic system.
  • Pedestrian Description: Color and texture descriptors remain central, while attribute-based features provide mid-level representations that are considered more robust to image translations.Examples include color histograms, SIFT, Gaussian region models, and annotated attire or soft-biometrics attributes.
  • Distance Metric Learning: Metric learning is critical because high-dimensional visual features often fail to capture invariant factors under sample variance.The survey categorizes methods by supervision and locality, with most person re-ID work using supervised global learning.
  • Distance Metric Learning: Mahalanobis-based learning keeps same-class vectors closer and different-class vectors farther apart through learned linear scalings and rotations of feature space.Its squared distance uses a positive semidefinite matrix M, and KISSME derives such a metric from a likelihood-ratio test.
  • Distance Metric Learning: Alternative approaches learn rankers, discriminative subspaces, or adaptive similarities instead of relying only on Mahalanobis distance.Examples include RankSVM, structural SVM, AdaBoost, LDA-like projections, null-space learning, and cross-patch bilinear similarity.

2.2 Deeply-learned Systems

Deeply learned re-ID systems primarily use CNN classification or siamese architectures. The survey describes pairwise, triplet, identification, and hybrid designs, while noting data scarcity and computational inefficiency as important constraints.

  • CNN Architectures: CNN-based re-ID models commonly use either classification architectures or siamese models that take image pairs or triplets as input.The survey traces their adoption to the success of CNNs in image classification and distinguishes identification from verification-style learning.
  • Learning Objectives: Limited training data has led many re-ID systems to favor siamese learning, although pairwise labels provide weaker supervision than identity classification labels.Datasets such as VIPeR may provide only two images per identity, constraining deep model training.
  • Learning Objectives: Siamese and triplet systems learn image embeddings by comparing paired or grouped inputs, sometimes combining local body parts with global representations.Reported designs use overlapping horizontal or body-part regions, cosine similarity, patch matching, and triplet loss.
  • Learning Objectives: Identification models use re-ID annotations more fully than siamese models by training directly with identity labels in a classification network.One surveyed approach jointly trains identities from multiple datasets with a softmax loss.
  • Alternative Designs: Hybrid networks can aggregate low-level descriptors into Fisher Vectors and optimize embeddings with discriminative objectives such as LDA.This provides an alternative to learning deep features directly from raw images.

2.3 Datasets and Evaluation

Re-ID evaluation spans increasingly large and realistic datasets, while CMC and mAP capture different retrieval objectives. Reported accuracy has improved substantially, but detector imperfections, limited scale, and large galleries remain important constraints.

  • Datasets: VIPeR is the most tested benchmark, with 632 identities, two images per identity, and 10 random train/test splits.Each split uses 316 identities for training and testing.
  • Datasets: Recent datasets CUHK03 and Market-1501 each exceed 1,000 identities and 10,000 bounding boxes, supporting deep-learning training but remaining smaller than needed.The survey explicitly calls for larger datasets.
  • Datasets: Detector-generated boxes introduce misalignment and false detections, and experiments show that detector errors and added gallery distractors reduce re-ID accuracy.Practical benchmarks should therefore include imperfections such as false detection and misalignment.
  • Datasets: Market-1501 uses up to six cameras per identity, requiring metric learning that generalizes across camera pairs rather than relying on pair-specific tuning.A city-scale network creates a combinatorial number of camera pairs, making per-pair annotation and metric training prohibitive.
  • Evaluation: CMC measures the probability that the first match appears within candidate-list sizes, whereas mAP evaluates retrieval of all true matches when multiple ground truths exist.CMC is suitable when top-ranked retrieval matters, while mAP better distinguishes systems with similar first-match accuracy but different recall.
  • Re-ID Accuracy Over the Years: +51.9%, +56.7%, +35.0%, +42.6%, +57.2%, and +31.62% performance increases were observed over time on VIPeR, CUHK01, i-LIDS, PRID 450S, CUHK03, and Market-1501, respectively.On VIPeR, rank-1 accuracy rose from 12.0% in 2008 to 63.9% in 2015; Market-1501 rose from 44.42% to 76.04%.
  • Re-ID Accuracy Over the Years: Deep learning achieved new state-of-the-art results on five of six datasets, while VIPeR’s small size limited its advantage over hand-crafted metric learning.The five datasets were CUHK01, i-LIDS, PRID 450S, CUHK03, and Market-1501.
  • Re-ID Accuracy Over the Years: 65.88% rank-1 accuracy without multiple queries on Market-1501 coexisted with 39.55% mAP, indicating difficulty retrieving hard positives and achieving high recall.A separate report found that a 10-fold gallery-size increase produced a 10-fold decrease in rank-1 accuracy.

3 VIDEO-BASED PERSON RE-ID

Video-based person re-ID extends image matching to sets of frames, using multi-shot similarity, temporal information, and increasingly pooled deep representations. Evaluation spans datasets with differing scale and difficulty, revealing saturation on ETHZ, progress on PRID-2011 and iLIDS-VID, and strong but incomplete deep-learning performance.

  • Video-based Formulation: Video-based re-ID replaces query and gallery images with sets of bounding boxes, requiring multi-shot matching and temporal-information integration.Each sequence contains nq and ng bounding boxes for query and gallery videos, respectively.
  • Hand-crafted Systems: Early video-based systems used color descriptors, optional foreground segmentation, and set similarity based on minimum Euclidean distance.These 2010 systems represent multi-shot re-ID, where similarity between frame sets determines matching.
  • Representation and Matching: Current video-based methods typically pool frame-level features into global vectors because pooling scales better than computationally expensive multi-match strategies.Pooling can use Fisher vectors, VLAD, direct CNN transfer, or learned temporal encodings.
  • Deeply-learned Systems: Temporal cues are incorporated into final representations through spatial-temporal descriptors or recurrent networks applied to appearance features.Appearance remains critical because motion can be nondiscriminative within a camera and vary across cameras.
  • Datasets and Evaluation: Video re-ID datasets range from relatively easy ETHZ to challenging iLIDS-VID and large-scale MARS with 1,261 identities in over 20,000 video sequences.iLIDS-VID is considered more challenging than PRID-2011 because of heavy occlusion.
  • Datasets and Evaluation: ETHZ is saturated near 100% rank-1 accuracy, while iLIDS-VID and PRID-2011 continue improving and deep methods dominate their best results.On MARS, rank-1 reaches 68.3% but mAP is 49.3%, with lower performance for individual camera pairs.

4 FUTURE: DETECTION, TRACKING AND PERSON RE-ID

The survey argues that practical re-ID must move beyond fixed, hand-drawn galleries by integrating and evaluating detection, tracking, and retrieval together. It highlights unresolved evaluation issues, detector/tracker effects, and future strategies for correcting or exploiting those effects.

  • 4 FUTURE: DETECTION, TRACKING AND PERSON RE-ID: End-to-end re-ID should integrate pedestrian detection, tracking, and retrieval from raw videos rather than assume a fixed gallery of hand-drawn boxes.Existing studies commonly assume given galleries and perfect detection, whereas practical galleries vary with detector and tracker parameters.
  • 4 FUTURE: DETECTION, TRACKING AND PERSON RE-ID: Dynamic galleries and unclear detection/tracking evaluation make end-to-end re-ID protocols difficult to standardize.Gallery size changes with detection or tracking thresholds, while the appropriate evaluation of detection and tracking in re-ID remains largely unknown.
  • 4.2.1 System Performance Evaluation: Under IoU = 0.7, detector and re-ID accuracy show a clearer linear relationship than under IoU = 0.5, although the relationship remains unsatisfactory.The survey therefore treats larger IoU thresholds as a more consistent evaluation criterion for detector quality in re-ID.
  • 4 FUTURE: DETECTION, TRACKING AND PERSON RE-ID: Detection errors compromise re-ID through misalignment, scale changes, missing parts, false positives, and missed detections.The survey identifies the intrinsic mechanism and feasible solutions as open challenges.
  • 4.2.2 The Influence of Detector/Tracker on Re-ID: Future work should study confidence-aware matching, context models, tracklet purification, and enlarged boxes to mitigate detection and tracking errors.The survey also notes that detection and tracking can help re-ID through pedestrian-labeled data and jointly trained models.

5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-

Very large galleries are an important future setting for person re-ID because current datasets and methods do not adequately address practical scale. The survey emphasizes both scalable learning and efficient retrieval through structures such as inverted indexes and hashing.

  • 5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-: Current re-ID datasets remain below practical surveillance scale; one 12-hour, 100-camera scenario would produce 43.2 × 10^6 bounding boxes.The survey contrasts this scale with earlier datasets containing only hundreds of gallery images.
  • 5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-: Very large-gallery re-ID requires simultaneous improvements in accuracy and efficiency for real-world applications.The survey identifies this setting as a critical future direction from both research and application perspectives.
  • 5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-: Global re-ID models are needed because pairwise training across n cameras becomes prohibitive as the number of camera pairs grows as O(n^2).The survey prioritizes adaptation across illumination conditions and camera locations, given long recording periods and limited annotations.
  • 5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-: Inverted indexes accelerate local-descriptor retrieval by organizing quantized descriptors into codebook-sized inverted lists.The baseline structure contains k entries, where k is the codebook size.
  • 5 FUTURE: PERSON RE-ID IN VERY LARGE GAL-: Hashing maps feature vectors to compact binary codes for approximate nearest-neighbor search when galleries are large or distance computation is costly.Supervised deep hashing is a natural fit for re-ID retrieval, but evaluation on very large galleries remains lacking.

6 OTHER IMPORTANT YET UNDER-DEVELOPED OPEN ISSUES

The survey identifies open issues spanning data annotation, learning from auxiliary data, re-ranking, and open-world recognition. These issues reflect practical constraints in scaling re-ID and handling identities absent from or entering the gallery.

  • Data Volume: Large-scale re-ID annotation is costly because annotators must assign identities across re-entries and cameras, not merely draw pedestrian bounding boxes.This makes collaboration difficult when identities reappear long after their first observation.
  • Data Volume: Tracking and detection annotations remain under-exploited, while additional pedestrian and background data can benefit subsequent re-ID descriptor training.The survey also notes attribute annotations from independent datasets as another auxiliary source.
  • Data Volume: Hard-negative retrieval from unlabeled data is presented as a way to enlarge training sets while reducing over-fitting risk.The strategy treats selected unlabeled examples as “true positives” for metric-learning or CNN training.
  • Re-ID Re-ranking: Re-ranking reorders an initial retrieval list to improve accuracy by exploiting relationships among easy and hard matches.The survey describes human-in-the-loop methods using negative feedback and incremental instance-model updates, alongside fully automatic approaches.
  • Open-World Person Re-ID: Open-world re-ID adds verification so queries can be rejected as identities absent from the gallery, but recognition remains low at low false accept rates.Dynamic galleries must also add new identities when probes match none of the existing watch-list identities.

7 CONCLUDING REMARKS

The paper surveys person re-ID from its history through image- and video-based methods, while emphasizing practical future directions. It highlights end-to-end systems, very-large-gallery retrieval, and unresolved issues as necessary areas for progress toward practical re-ID.

  • Concluding Remarks: The survey reviews person re-ID history, its links to image classification and instance retrieval, and hand-crafted and deep systems for image- and video-based tasks.It organizes the literature across these task and modeling distinctions.
  • Concluding Remarks: End-to-end re-ID integrating pedestrian detection, tracking, and retrieval is emphasized as an under-developed direction toward practical systems.The conclusion connects this direction with the need to optimize detector and tracking components alongside discriminative feature learning.
  • Concluding Remarks: Fast re-ID in very large galleries is highlighted as another critical future possibility requiring efficient retrieval and data structures.The paper frames very-large-gallery retrieval as closer to real-world application needs.
  • Concluding Remarks: The survey also identifies data volume, re-ranking, and open re-ID systems as important issues that remain under-developed.Its overall conclusion calls for integrating feature learning, detector and tracking optimization, and efficient data structures.
Loading 1610.02984v1…