Source-linked AI summary
Joint Detection and Identification Feature Learning for Person Search
Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, Xiaogang Wang
TL;DR
Person search addresses the gap between cropped-pedestrian benchmarks and searching for people in whole scene images. The paper jointly trains detection and identification in one CNN, using OIM for scalable feature learning. The framework outperforms separate approaches, while OIM converges faster and improves test performance more consistently than Softmax.
Problem
Person search must find target people in whole scene images, whereas most benchmarks assume manually cropped pedestrian bounding boxes unavailable in practical applications.
Method
A single CNN jointly trains a pedestrian proposal net and identification net, while OIM compares features against stored labeled and unlabeled identities.
Results
The framework outperforms 15 separate detection-and-re-identification baseline combinations, and OIM converges faster while consistently improving test performance than Softmax.
Takeaways & Limitations
Joint detection and identification plus OIM provides a person-search framework validated on a large-scale benchmark with numerous identities.
Takeaways & Limitations
The non-parametric OIM loss could overfit more easily, although L2-normalized low-dimensional features help reduce overfitting.
Abstract
from arXiv · showhide
Existing person re-identification benchmarks and methods mainly focus on matching cropped pedestrian images between queries and candidates. However, it is different from real-world scenarios where the annotations of pedestrian bounding boxes are unavailable and the target person needs to be searched from a gallery of whole scene images. To close the gap, we propose a new deep learning framework for person search. Instead of breaking it down into two separate tasks---pedestrian detection and person re-identification, we jointly handle both aspects in a single convolutional neural network. An Online Instance Matching (OIM) loss function is proposed to train the network effectively, which is scalable to datasets with numerous identities. To validate our approach, we collect and annotate a large-scale benchmark dataset for person search. It contains 18,184 images, 8,432 identities, and 96,143 pedestrian bounding boxes. Experiments show that our framework outperforms other separate approaches, and the proposed OIM loss function converges much faster and better than the conventional Softmax loss.
1. Introduction
Person search targets people in whole scene images rather than manually cropped pedestrians, motivating a jointly trained detection-and-identification CNN and a scalable OIM loss.
- Problem setting: Person search finds a target in whole scene images, unlike benchmarks with manually cropped pedestrian galleries.Real applications lack the manually cropped bounding boxes assumed by most person re-identification methods.
- Problem setting: Off-the-shelf pedestrian detectors introduce false alarms, misdetections, and misalignments that can significantly harm search performance.
- Joint framework: The proposed CNN jointly optimizes pedestrian detection and person identification from whole gallery images.A proposal net generates candidate boxes, while an identification net extracts comparison features; both adapt during joint optimization.
- Online Instance Matching: Online Instance Matching compares mini-batch features with stored labeled-identity features and unlabeled-identity negatives to improve scalability.The method uses a lookup table for labeled identities and a circular queue for unlabeled identities.
- Contributions: The framework jointly optimizes detection and identification and uses OIM to learn identification features effectively on large datasets.
- Benchmark: The benchmark contains 18,184 images, 8,432 identities, and 96,143 pedestrian bounding boxes.The dataset covers hundreds of scenes from street and movie snapshots and is released with code.
2. Related Work
Related work spans hand-crafted and CNN-based person re-identification and pedestrian detection, while prior person-search approaches used separate components; this framework shares convolutional features.
- Person re-identification: Person re-identification methods have progressed from hand-crafted features and metric learning to CNN models that jointly learn these aspects.
- Person search: Prior person-search work combined separate detection and re-identification methods with score re-weighting.
- Pedestrian detection: Pedestrian detection includes hand-crafted-feature detectors such as DPM, ACF, and Checkerboards alongside later CNN-based detectors.
- Proposed framework: The proposed framework shares underlying convolutional feature maps between pedestrian proposal and identification networks.
3. Method
The method jointly performs pedestrian proposal generation and person identification in one CNN, using OIM to compare each sample with stored labeled and unlabeled identities.
- 3.1. Model Structure: The model processes whole scene images through a stem CNN, pedestrian proposal network, and identification network for candidate-person matching.The identification network uses RoI-Pooling and extracts L2-normalized 256-d features for proposals.
- 3.1. Model Structure: The proposal network predicts pedestrian anchors, regresses their locations, and retains the top 128 boxes after non-maximum suppression.
- 3.2. Online Instance Matching Loss: OIM separates proposals into labeled identities, unlabeled identities, and background clutter, applying its loss only to the first two groups.
- 3.2. Online Instance Matching Loss: The lookup table stores labeled-identity features and updates the target class feature during backpropagation using a momentum rule with unit L2 normalization.
- 3.2. Online Instance Matching Loss: A circular queue stores recent unlabeled-identity features, which are used as negative classes and refreshed by pushing new vectors while removing stale ones.
- 3.2. Online Instance Matching Loss: OIM compares each mini-batch sample with stored labeled and unlabeled identities, increasing similarity to the target and decreasing similarity to others.
- 3.2. Online Instance Matching Loss: The LUT and circular queue are external buffers rather than CNN parameters, making OIM non-parametric but potentially more prone to overfitting.L2-normalized low-dimensional projections are used to reduce overfitting.
- 3.2. Online Instance Matching Loss: Subsampling labeled and unlabeled identities can reduce partition-function cost, while Softmax is limited by many identities, sparse per-iteration classes, and unusable unlabeled identities.
4. Dataset
The dataset combines street and movie snapshots to provide varied person-search scenes, with annotated identities and pedestrian boxes organized into separate training and test subsets. Evaluation uses query-specific galleries whose sizes vary widely and is measured with CMC top-K and mAP.
- Data sources: The dataset combines hand-held urban street snaps and movie snapshots to diversify viewpoints, lighting, and background conditions.
- Dataset statistics: 18,184 images contain 96,143 annotated pedestrian bounding boxes associated across images into 8,432 labeled identities.
- Data splits: The dataset separates training and test subsets without overlapping images or labeled identities.
- Evaluation protocols: Gallery-size protocols range from 50 to 4000 images, with gallery size 100 representing roughly 600 candidate pedestrians plus background clutter.
- Evaluation metrics: CMC top-K counts a match when a predicted box overlaps a ground truth by IoU ≥ 0.5, while mAP averages query-level AP values.
5. Experiments
Experiments compare the joint person-search framework with separate detection and re-identification pipelines, evaluate OIM against Softmax, and study scalability and design factors. The framework outperforms separate baselines, while OIM converges faster and improves test performance.
- 5.1. Experiment Settings: The evaluation compares the joint framework with 15 separate detection and re-identification baseline combinations on the person-search dataset.Baselines include three pedestrian detectors, five re-identification methods, and ground-truth boxes as a perfect detector.
- 5.2. Comparison with Detection and Re-ID: The joint framework outperforms the separate baselines by a large margin, with gains attributed to joint optimization and unlabeled identities in OIM.The comparison includes versions of the proposed framework with and without unlabeled identities.
- 5.2. Comparison with Detection and Re-ID: Different detectors significantly affect person-search performance, and off-the-shelf detection can bottleneck stronger re-identification methods.Relative re-identification performance remains consistent across detectors, while manually cropped re-identification datasets omit unlabeled identities and background clutter.
- 5.3. Effectiveness of Online Instance Matching: OIM converges much faster than Softmax and consistently improves test performance, whereas pretrained Softmax remains near 60% test mAP.Without classifier pretraining, Softmax stays at low accuracy; pretraining improves optimization only slowly.
- 5.3. Effectiveness of Online Instance Matching: Sub-sampling identities yields slightly inferior performance but faster convergence, supporting scalability to larger datasets with an appropriate sampling rate.The study evaluates sub-sampling sizes of 10, 100, and 1000 identities.
- 5.3. Effectiveness of Online Instance Matching: Using a 2048-d normalized feature directly makes test performance 20% worse, while 256–1024 dimensions perform similarly; 256-d is chosen for faster distance computation.The results support projecting features into a low-rank subspace for regularization.
- 5.4. Factors for Person Search: Higher detector recall does not necessarily improve person-search performance because re-identification can be confused by false alarms.This motivates jointly considering detections rather than training only with manually cropped pedestrians.
6. Conclusion
The paper presents a unified person-search CNN and an Online Instance Matching loss. Experiments validate faster and better convergence for the proposed loss.
- The framework jointly handles pedestrian detection and identification in a single CNN.It is designed for person search rather than treating detection and re-identification as separate tasks.
- Online Instance Matching loss trains the network effectively for person search.Its non-parametric nature is reported to support the training objective.
- The proposed loss achieves faster yet better convergence, validated through a series of experiments.