Source-linked AI summary
A Pose-Sensitive Embedding for Person Re-Identification with Expanded Cross Neighborhood Re-Ranking
M. Saquib Sarfraz, Arne Schumann, Andreas Eberle, Rainer Stiefelhagen
TL;DR
Person re-identification must match people across non-overlapping camera views despite changes in view and pose. This paper learns pose-sensitive CNN embeddings and introduces expanded cross neighborhood re-ranking, achieving state-of-the-art performance on challenging datasets.
Problem
Person re-identification requires matching a person across non-overlapping camera views despite differing views and poses.
Method
The paper incorporates coarse camera-view and fine joint-location cues into CNN embeddings and proposes unsupervised expanded cross neighborhood re-ranking using distances or rank-list comparisons.
Results
The pose-sensitive model and re-ranking method each achieve state-of-the-art performance on four challenging datasets, while pose cues improve embedding accuracy across datasets and architectures.
Takeaways & Limitations
Direct rank-list comparison or even direct Euclidean distances can provide strong re-ranking gains without recomputing rank lists for each image pair.
Takeaways & Limitations
The pose-sensitive model currently relies on an external pose predictor, and view classifiers may require pretraining on a separate annotated pedestrian dataset.
Abstract
from arXiv · showhide
Person re identification is a challenging retrieval task that requires matching a person's acquired image across non overlapping camera views. In this paper we propose an effective approach that incorporates both the fine and coarse pose information of the person to learn a discriminative embedding. In contrast to the recent direction of explicitly modeling body parts or correcting for misalignment based on these, we show that a rather straightforward inclusion of acquired camera view and/or the detected joint locations into a convolutional neural network helps to learn a very effective representation. To increase retrieval performance, re-ranking techniques based on computed distances have recently gained much attention. We propose a new unsupervised and automatic re-ranking framework that achieves state-of-the-art re-ranking performance. We show that in contrast to the current state-of-the-art re-ranking methods our approach does not require to compute new rank lists for each image pair (e.g., based on reciprocal neighbors) and performs well by using simple direct rank list based comparison or even by just using the already computed euclidean distances between the images. We show that both our learned representation and our re-ranking method achieve state-of-the-art performance on a number of challenging surveillance image and video datasets. The code is available online at: https://github.com/pse-ecn/pose-sensitive-embedding
1. Introduction
Person re-identification must match people across non-overlapping camera views despite view- and pose-induced appearance changes. The paper addresses this with pose-sensitive embeddings and expanded cross-neighborhood re-ranking.
- Problem: Non-overlapping camera views make person re-identification a difficult matching problem because local image regions lack implicit correspondence.Prior approaches learn global or regional CNN representations, while pose and body-part information can establish correspondence.
- Problem: Camera view changes visible clothing cues, while pose changes body-part locations and can complicate alignment.Pose information may also guide attention when detections are mis-aligned.
- Pose-sensitive embedding: Coarse camera-view cues or fine joint locations suffice to learn discriminative embeddings with a simple classification loss.The approach learns view-specific feature maps and adds joint locations as additional information to a CNN.
- Re-ranking: The paper introduces expanded cross-neighborhood distance, aggregating distances among close neighbors of probe and gallery images for re-ranking.The method can use direct Euclidean distances or rank-list distances.
- Re-ranking: Directly obtained rank-list comparison achieves state-of-the-art re-ranking without recomputing new rank lists for each image pair.This addresses the computational burden of approaches based on common or reciprocal neighbors.
- Results: The pose-sensitive model and re-ranking method achieve state-of-the-art performance on four challenging datasets.The paper also reports scalability to very large galleries and person search in full camera images.
2. Related Work
Related work uses pose to improve re-identification through invariant features, view-specific matching, pose normalization, or explicit body-part modeling. This paper instead adds pose confidence maps and expanded neighbors without recomputing pair-specific rank lists.
- Pose-based re-identification: Prior re-identification methods use pose invariance, view-specific matching weights, pictorial structures, or CNN-based pose information.These approaches range from coarse view angles to fine-grained body-part representations.
- Pose-based re-identification: Several methods normalize person images, crop localized body parts, or explicitly detect and match body parts to address pose variation.These designs learn local and global representations or align inputs before matching.
- Pose-sensitive embedding: Unlike most prior pose-based approaches, the proposed method incorporates pose confidence maps as additional CNN input channels rather than explicitly normalizing or matching parts.This leaves the network to learn which body parts are relevant and reliable.
- Re-ranking: Recent re-ranking methods exploit common or reciprocal neighbors but often recompute neighborhood lists for each image pair, increasing computational complexity.Other work aggregates similarity and dissimilarity cues from neighbors or fuses reciprocal-neighbor distances with original distances.
- Pose-sensitive embedding: The PSE architecture combines detailed body joint maps with a coarse view predictor over a ResNet-50 or Inception-v4 baseline.This architecture provides the paper’s unified treatment of fine and coarse pose information.
- Re-ranking: Expanded neighbors aggregate cross distances between image pairs without recomputing new rank lists for each pair.The paper presents this as a more effective re-ranking framework than common- or reciprocal-neighbor approaches.
3. Pose-Sensitive Embedding
The pose-sensitive embedding incorporates both coarse camera orientation and fine body-joint information into a CNN. View prediction modulates feature processing, while joint maps guide attention through additional input channels.
- Coarse view information: The method represents coarse pose by quantizing camera orientation into front, back, and side views.Because orientation depends on both the camera and the person, the paper calls this information view information.
- Coarse view information: A ternary view classifier forms a side branch whose three prediction scores weight replicated tail units of the re-identification CNN.The weighting modulates gradient flow so strongly predicted views contribute more to the final embedding.
- Coarse view information: Because target re-identification datasets may lack view annotations, the view classifier is pretrained on RAP and transferred to the re-identification model.Early layers may be shared between the view predictor and re-identification network to reduce model complexity.
- Fine body pose: Fine-grained pose uses 14 body-joint keypoints from DeeperCut, added as 14 input channels rather than used to normalize person images.These channels guide CNN attention toward body-pose information.
- Training details: Training uses ImageNet initialization, staged fine-tuning for view or pose inputs, and Adam optimization with augmentation and learning-rate decay.The common protocol normalizes channels, randomly crops resized images, and applies horizontal flips.
- Combined embedding: The final PSE combines the full-body-pose model with the view predictor in one network.Training begins from the full-body-pose model, then adds and fine-tunes the view predictor using pose maps.
4. Expanded Cross Neighborhood Distance based Re-Ranking
Expanded Cross Neighborhood (ECN) re-ranking expands each probe and gallery image’s neighborhood through immediate neighbors and their neighbors, then aggregates cross-neighborhood distances. It can use direct Euclidean distances or rank-list similarities computed from initial rankings, avoiding recomputation of new rank lists.
- Distance choices: The underlying ECN distances can be direct Euclidean distances or rank-list-based distances derived from the initial distance matrix or rank lists.The paper reports that direct Euclidean distances provide similar rank-accuracy improvements, while rank-list distances can further enhance mAP.
- Initial ranking: ECN builds initial rank lists by sorting pairwise probe–gallery distances in increasing order.The initial distances are computed between probe and gallery images before neighborhood expansion.
- Expanded neighbors: Expanded neighbors combine the top t immediate neighbors with the top q neighbors of each, yielding M = t + t × q neighbors.The same two-level expansion is constructed for probe and gallery images.
- Cross-neighborhood distance: ECN aggregates distances between each image and the expanded neighbors of its paired counterpart, using the corresponding probe and gallery neighbor sets.For pair (p, g_i), the method combines distances involving neighbors from both expanded sets.
- Computational design: The proposed framework avoids recomputing new rank lists for each image pair, unlike re-ranking methods based on common or reciprocal neighbors.Initial ordered rank lists are sufficient to obtain the expanded sets and compute the rank-list similarity efficiently.
5. Evaluation
The evaluation tests pose-sensitive embeddings and ECN re-ranking across four datasets, comparing pose configurations, architectures, state-of-the-art methods, robustness, and computational cost. Pose cues improve embeddings, their combination further helps, and ECN provides strong re-ranking gains without recomputing neighbor lists.
- Evaluation setup: The evaluation covers Market-1501, Duke, MARS, and PRW using standard cross-camera single-query re-identification metrics, including CMC rank scores and mAP.PRW additionally tests robustness to false-positive or misaligned detections.
- 5.1. Study of Pose Information: Including either view or fine-grained pose information significantly improves embeddings over a pose-free baseline across datasets and CNN architectures.On ResNet, view information improves mAP by about 6–7% on both Market and Duke, while pose improves it by about 2–3%.
- 5.1. Study of Pose Information: Combining view and body-pose information further increases mAP over either cue alone, improving ResNet-50 by 2.1% on Market and 5.3% on Duke.The corresponding Inception-v4 gains are 3% on Market and 2.2% on Duke.
- 5.2. Study of Re-Ranking: ECN re-ranking achieves strong gains with either rank-list or direct Euclidean distances, while avoiding new rank-list computation for each image pair.Using direct distances produces similar rank-1 gains, and rank-list comparison within ECN further improves mAP.
- 5.2. Study of Re-Ranking: ECN has O(N^2logN) complexity but is faster than k-reciprocal on Duke: 115.3s with rank distances and 73.2s with original distances versus 124.6s.The timings average five runs over 19,889 images.
- 5.3. State-of-the-art: The pose-sensitive embedding achieves top accuracy on MARS and Duke, while combining it with ECN establishes new state-of-the-art results on Market, Duke, and MARS.The combined approach increases mAP by 11.4% on Market, 19.2% on Duke, and 4.5% on MARS.
- 5.3. State-of-the-art: On PRW, PSE remains stable as detections per image increase and outperforms related approaches by at least 6.3% mAP when averaging three detections per image.The result evaluates robustness to increasing false detections.
6. Conclusion
The paper presents two related but independent contributions for person re-identification and retrieval: a pose-sensitive CNN embedding and an unsupervised re-ranking method. The conclusion reports state-of-the-art performance while identifying external pose prediction as a remaining limitation.
- The pose-sensitive CNN embedding incorporates both fine and coarse body-pose cues for person re-identification.
- The re-ranking method is unsupervised and applicable to general image and video retrieval applications.
- Both contributions achieve state-of-the-art results independently and together on challenging datasets, while the PSE model relies on an external pose predictor.