Source-linked AI summary
A Discriminatively Learned CNN Embedding for Person Re-identification
Zhedong Zheng, Liang Zheng, Yi Yang
TL;DR
Person re-ID verification and identification models use different supervision and have complementary limitations. The paper combines their losses in a siamese network that predicts identities and pairwise similarity, producing discriminative embeddings that improve benchmark performance and also support instance retrieval.
Problem
Verification and identification CNNs have complementary advantages and limitations, motivating a model that uses both pairwise similarity and identity annotations.
Method
A siamese network jointly computes identification and verification losses while predicting the identities of paired images and whether they share an identity.
Results
The learned embedding outperforms the state of the art on two person re-ID benchmarks and shows potential for generic instance retrieval.
Takeaways & Limitations
Joint identity and similarity supervision learns both a discriminative embedding and a similarity measurement for pedestrian retrieval.
Abstract
from arXiv · showhide
We revisit two popular convolutional neural networks (CNN) in person re-identification (re-ID), i.e, verification and classification models. The two models have their respective advantages and limitations due to different loss functions. In this paper, we shed light on how to combine the two models to learn more discriminative pedestrian descriptors. Specifically, we propose a new siamese network that simultaneously computes identification loss and verification loss. Given a pair of training images, the network predicts the identities of the two images and whether they belong to the same identity. Our network learns a discriminative embedding and a similarity measurement at the same time, thus making full usage of the annotations. Albeit simple, the learned embedding improves the state-of-the-art performance on two public person re-ID benchmarks. Further, we show our architecture can also be applied in image retrieval.
I. INTRODUCTION
Person re-ID uses CNN embeddings, but verification and identification models exploit complementary supervision. The paper combines both losses in a siamese network to learn more discriminative embeddings and improve retrieval accuracy.
- Person re-ID matches pedestrians across cameras using learned CNN embeddings and large-scale datasets for training and evaluation.
- Verification models compare image pairs but rely on weak labels, limiting their use of relationships with other dataset images.
- Identification models predict person identities from individual images and use annotated identity information, but have complementary limitations to verification models.
- The proposed siamese network jointly predicts identities and pairwise similarity, combining identification and verification supervision.
- The method reports competitive accuracy on Market1501 and CUHK03 and applies the architecture to Oxford5k instance retrieval.
II. RELATED WORK
Prior person re-ID work is organized around verification and identification CNNs, with progress enabled by larger datasets and increasingly discriminative learned embeddings.
- Related approaches mainly use either verification models or identification models for person re-ID.
- Verification models compare image pairs using similarity scores, contrastive loss, data augmentation, or spatial matching operations.
- Large-scale CUHK03 and Market1501 datasets enable deeper classification models to learn discriminative embeddings without over-fitting.
C. Verification-identification Models
The paper distinguishes its joint verification-identification model from DeepID face-recognition networks through its loss, regularization, initialization, and evaluation setting.
- DeepID networks jointly train face identification and verification losses, providing a related precedent for the proposed architecture.
- The proposed method uses cross-entropy rather than contrastive loss because the authors find contrastive loss over-fits when person re-ID images are limited.
- Unlike DeepID, the method can apply dropout regularization to the embedding and benefits from ImageNet-pretrained networks.
- The proposed structure uses paired identical CaffeNet branches, identity predictions for both images, a Square Layer, and three softmax objectives.
- The method is evaluated on person re-ID and instance retrieval, extending analysis beyond face recognition.
A. Preview
The model combines implicit relationships from identification loss with explicit pairwise relationships from verification loss in a siamese CNN architecture.
- A. Preview: Identification loss creates implicit relationships among embeddings, whereas verification loss explicitly compares paired inputs.
- A. Preview: The joint model simultaneously uses verification and identification losses to combine their complementary strengths.
- A. Preview: Given resized image pairs, two pretrained CNN branches predict both image identities and their similarity score.
- A. Preview: The shared CaffeNet branches replace the pretrained classifier with a dataset-specific convolutional layer and softmax identity predictor.
D. Verification Loss
The verification branch compares high-level embeddings directly through a Square Layer and supervises them with a binary same/different identity objective. Cross-entropy is chosen instead of contrastive loss to support dropout and reduce overfitting concerns.
- Verification Loss: The model directly compares the identification network’s high-level features f1 and f2 for similarity estimation.These features are compact and discriminative compared with intermediate activations.
- Verification Loss: The Square Layer subtracts and squares corresponding elements of two 4,096-dimensional embeddings to produce fs.It is a non-parametric comparison layer whose output feeds the verification classifier.
- Verification Loss: The identification descriptors are directly supervised by the verification loss, linking similarity learning to embedding learning.This provides verification supervision at the high-level feature representation.
- Verification Loss: A convolutional layer followed by softmax maps fs to two probabilities representing whether the input images share an identity.The two probabilities sum to one, and the verification task uses binary cross-entropy.
- Verification Loss: The model avoids contrastive loss because it may overfit limited per-identity training data and is incompatible with dropout on the embedding.The authors instead use cross-entropy, which can work with dropout.
E. Identification vs. Verification
Jointly training identification and verification objectives outperforms either objective alone across three network structures. Visualizations suggest complementary activation patterns and embeddings that emphasize clothing color while tolerating some viewpoint variation.
- Identification vs. Verification: The learned embedding clusters people wearing similarly colored clothes and separates them from other people in t-SNE visualization.The visualization uses a Market1501 test split containing 354 identities and 6868 images.
- Identification vs. Verification: Joint training with identification and verification losses outperforms models trained with either loss individually across three network structures.The comparison is reported quantitatively in Table II after training each objective to convergence.
- Identification vs. Verification: The learned descriptor pays more attention to clothing color and is reported to be robust to some illusion and viewpoint variations.The authors identify color as especially important for person re-identification in realistic settings.
- Identification vs. Verification: The two single-objective networks exhibit different pedestrian activation patterns, while the proposed map is nearly their union.The combined model also activates more neurons.
F. Training and Optimization
Training uses resized, mean-subtracted images with random crops, horizontal mirroring, shuffled sampling, and progressively more negative than positive pairs. The model is optimized with mini-batch SGD under network-specific epoch and batch-size settings, then features are extracted efficiently for retrieval.
- Training and Optimization: Training samples positive or negative image pairs after shuffling, starting at a 1:1 ratio and increasing negatives to a 1:4 ratio.The schedule addresses the limited number of positive pairs.
- Training and Optimization: Mini-batch SGD trains CaffeNet, VGG16, and ResNet-50 with batch sizes of 128, 48, and 48 image pairs, respectively.Maximum epochs are 155 for CaffeNet, 65 for VGG16, and 75 for ResNet-50.
- Training and Optimization: At test time, one fine-tuned CaffeNet extracts each 4,096-dimensional pedestrian descriptor because the two CaffeNet branches share weights.Gallery descriptors are extracted and stored offline.
- Training and Optimization: The architecture is evaluated on person re-identification datasets and also applied to Oxford Buildings image retrieval, where performance is reported as comparable to the state of the art.The person re-identification evaluation includes Market1501 and CUHK03, plus Market1501+500k.
A. Dataset
The evaluation covers Market1501, CUHK03, and Oxford5k, spanning realistic person re-identification settings and landmark image retrieval. Performance is measured with rank-1 accuracy and mAP for re-ID, and mAP for Oxford.
- Dataset: Market1501 contains 32,668 annotated bounding boxes from 1,501 identities across at most six cameras.Its training and testing sets contain 12,936 images from 751 identities and 19,732 images from 750 identities plus distractors, respectively.
- Dataset: Market1501 retrieval targets ground-truth images within a 19,732-image candidate gallery for each query.The images are detected by DPM rather than manually boxed.
- Dataset: Adding 500,000 distractor bounding boxes to Market1501 creates a more difficult large-gallery retrieval setting.The distractors include background false alarms and people outside the original identities.
- Dataset: CUHK03 contains 14,097 cropped images of 1,467 identities observed from two camera views, with DPM-detected boxes used for evaluation.The dataset partition uses 1,367 training identities and 100 testing identities.
- Dataset: Oxford5k contains 5,062 internet images of 11 Oxford landmarks and provides 55 queries for image retrieval.The benchmark includes complex scenes and high-resolution images with a mean size of 851×921.
- Dataset: Re-identification is evaluated with rank-1 accuracy and mAP, while Oxford retrieval is evaluated with mAP.These metrics are applied to Market1501, Market1501+100k, and CUHK03 for re-ID.
B. Person Re-id Evaluation
The proposed model improves person re-identification across Market1501 and CUHK03 settings, including multi-shot retrieval and larger galleries. Its learned embeddings also show robustness across camera viewpoints, though accuracy declines as the search pool grows.
- 79.51% rank-1 accuracy and 59.87% mAP are achieved on Market1501 with ResNet-50, outperforming other state-of-the-art algorithms.
- 83.4% rank-1 accuracy and 86.4% mAP are obtained on CUHK03 in the single-shot setting, outperforming state-of-the-art performance.
- 88.3% rank-1 accuracy and 85.0% mAP are obtained on CUHK03 in the multi-shot setting, also exceeding state-of-the-art performance.
- 48.42% cross-camera average mAP and 54.42% average rank-1 accuracy are reported on Market1501, with smaller variation between cameras than previous results.
- The learned pedestrian descriptor clusters similar-color clothing and remains robust to some illusion and viewpoint variations.
- 45.24% mAP and 68.26% rank-1 accuracy remain at a gallery size of 500,000 + 19,732 images, although accuracy drops as the search pool expands.
C. Instance Retrieval
The identification-verification model is adapted to generic building-image retrieval using high-resolution features and a MAC pooling layer. It improves or matches prior methods while handling scale variation in Oxford5k retrieval.
- The model is adapted to Oxford5k by training on non-overlapping building images and combining building identification with same-architecture verification.
- A MAC pooling layer preserves high-resolution image information without fixed-size resizing and produces fixed-dimension retrieval features.
- The proposed method improves over CaffeNet and VGG16 baselines by 6.0% and 6.6%, respectively, on Oxford5k.
- The CaffeNet-based model outperforms the state of the art, while the VGG16-based model is comparable to state-of-the-art methods.
- Retrieval examples show robustness to object-scale variation, including correct retrieval of the top five candidates for a building-roof query.
V. CONCLUSION
The paper proposes a siamese network that jointly uses identification and verification losses to learn a discriminative embedding and similarity measurement. It reports state-of-the-art performance on two person re-identification benchmarks and potential for generic instance retrieval, while identifying broader applications as future work.
- The siamese network simultaneously considers identification and verification losses to learn a discriminative embedding and similarity measurement.
- The method outperforms the state of the art on two person re-identification benchmarks and shows potential for generic instance retrieval.
- Future work includes applying the method to car recognition and fine-grained classification.