Source-linked AI summary
Neural Person Search Machines
Hao Liu, Jiashi Feng, Zequn Jie, Karlekar Jayashree, Bo Zhao, Meibin Qi, Jianguo Jiang, Shuicheng Yan
TL;DR
Person search must locate a queried person amid distracting regions and challenging visual conditions. NPSM recursively shrinks the search area using query memory and contextual cues, and outperforms other methods on two benchmark datasets across reported evaluation settings.
Problem
Person search must jointly localize and re-identify a specific queried person despite appearance variation, low resolution, clutter, and unfavorable camera settings.
Method
NPSM performs detection-free recursive region shrinkage with a query-informed primitive memory and Conv-LSTM-based neural search to focus on effective regions.
Results
NPSM outperforms state-of-the-art methods on two public benchmarks in most cases and improves average mAP per gallery size over OIM by around 2%.
Takeaways & Limitations
Query memory and contextual guidance support more precise person localization while allowing irrelevant regions to be ignored during recursive search.
Takeaways & Limitations
The reported comparison includes failure or error sources associated with distracting similar people and inaccurate detection in prior two-stage pipelines.
Abstract
from arXiv · showhide
We investigate the problem of person search in the wild in this work. Instead of comparing the query against all candidate regions generated in a query-blind manner, we propose to recursively shrink the search area from the whole image till achieving precise localization of the target person, by fully exploiting information from the query and contextual cues in every recursive search step. We develop the Neural Person Search Machines (NPSM) to implement such recursive localization for person search. Benefiting from its neural search mechanism, NPSM is able to selectively shrink its focus from a loose region to a tighter one containing the target automatically. In this process, NPSM employs an internal primitive memory component to memorize the query representation which modulates the attention and augments its robustness to other distracting regions. Evaluations on two benchmark datasets, CUHK-SYSU Person Search dataset and PRW dataset, have demonstrated that our method can outperform current state-of-the-arts in both mAP and top-1 evaluation protocols.
1. Introduction
Person search must jointly localize and re-identify a queried person despite distracting real-world conditions. NPSM addresses this with query-guided recursive region shrinkage that uses contextual cues and memory to focus attention on likely target regions.
- Person search jointly addresses person detection and re-identification while coping with appearance variation, low resolution, clutter, and unfavorable camera settings.
- Earlier methods generally detect candidate persons first and then exhaustively compare the query with candidates, exposing errors from similar distractors and inaccurate detection.
- NPSM reframes person search as detection-free recursive focusing on regions likely to contain the target, using coarse-to-fine localization.
- The model recursively shrinks its search region with query appearance memory, incorporates contextual cues, and ignores irrelevant regions after each shrinkage.
- NPSM uses a Conv-LSTM-based Neural Search Network to preserve spatial information while selecting effective subregions and suppressing distracting ones.
- Its external primitive memory integrates query information into representation learning and recursively guides attention toward effective regions.
2. Related Work
Related person-search research combines detection and re-identification, while adjacent work studies attention for emphasizing informative regions. NPSM differs by using query-aware rather than blind attention within a recursive search process.
- Person search can be viewed as combining person re-identification with person detection, drawing on distinct feature-learning and pedestrian-detection research traditions.
- Earlier person-search methods fused detection and re-identification into an integrated pipeline to address the task jointly.
- LSTM-based attention methods have been applied to image description, action recognition, and person re-identification to emphasize important visual elements.
- Unlike the blind attention mechanisms in related attention-based methods, NPSM uses query-aware attention for person search.
3. Proposed Neural Person Search Machines
NPSM performs person search by recursively shrinking an image region toward the target while using query memory, contextual cues, and neural attention to suppress distractors. Its Neural Search Networks preserve spatial information and select increasingly focused subregions until producing one final person-search result.
- 3.1. Architecture Overview: NPSM combines Primitive Memory with Neural Search Networks to recursively shrink the search area from the whole image toward the target person.The process retains contextual information from larger regions and provides robustness to distracting subregions.
- 3.1. Architecture Overview: The model is trained end-to-end to choose subregion attention at each step and produce one final bounding-box result without detection or NMS.It simultaneously performs region shrinking and person identification, leaving one person-search result after searching.
- 3.2.1 Neural Search Networks: NSN uses Conv-LSTM to preserve spatial structure during recursive search, while its additional query memory distinguishes relevant information from irrelevant regions.Conv-LSTM avoids vectorizing two-dimensional feature maps, and the query is modeled as time-independent primitive memory.
- 3.2.1 Neural Search Networks: Query features are integrated into NSN gates and cell states so internal updates emphasize query-relevant information and forget irrelevant information.The query feature is extracted from the same pre-trained ResNet-50 Part 1 used for gallery-region features.
- 3.2.2 Region Shrinkage with Primitive Memory: At each recursive step, NSN scores candidate subregions and highlights the subregion with the maximum average score for the next search step.Candidate regions are formed by grouping proposal boxes, and the attention map assigns scores reflecting confidence that a region contains the target.
- 3.2.2 Region Shrinkage with Primitive Memory: The model can shift to a previously unhighlighted region when it receives higher attention, reducing accumulated shrinkage error during recursive search.The score map is computed over K × K locations, and subregion scores are averaged across their spatial dimensions.
4.1. Datasets and Evaluation Protocol
The evaluation uses CUHK-SYSU, a large and diverse person-search dataset, and measures performance with mAP and top-1 matching rate. These metrics assess detection-like accuracy and ranked top-result localization, respectively.
- 4.1. Datasets: CUHK-SYSU contains 18,184 images, 8,432 persons, and 96,143 annotated pedestrian bounding boxes across diverse scenes.Each selected query appears in at least two images from different viewpoints, with variation in viewpoint, lighting, resolution, occlusion, and background.
- 4.1. Evaluation Protocol: The evaluation reports mean Average Precision (mAP) and top-1 matching rate, following the metrics used by OIM and the recent comparison method.mAP reflects detecting the query person in gallery images, while top-1 counts a match when the highest-ranked predicted box overlaps the ground truth.
4.2. Implementation Details
Implementation uses OIM-pretrained ResNet-50 feature extractors and ROI-pooled region and query features as inputs to NPSM. The network uses fixed convolutional dimensions and three clustered proposal subregions per recursive step.
- 4.2. Implementation Details: The Res50 Part 1 and Part 2 fully convolutional networks are pretrained with the OIM architecture before NPSM processing.ROI pooling converts each input region and query representation to 14 × 14 × 1024 features.
- 4.2. Implementation Details: NSN uses 3 × 3 convolutional kernels with 1024 channels and sets the number of clustered proposal subregions to C = 3 at each recursive step.The implementation uses Theano and Caffe, and training converges in roughly 50 hours for CUHK-SYSU.
4.3. Ablation Study
The ablation study evaluates attention prediction, contextual cues, and query primitive memory, showing that their combination supports the strongest person-search performance.
- Ablation setup: The study compares three NPSM variants by selectively removing attention prediction, contextual cues, and primitive memory, using a gallery size of 100 on CUHK-SYSU.OIM serves as the baseline without these three factors.
- Component effects: Primitive memory alone is insufficient, while adding attention prediction yields only slightly higher performance than the baseline.The “NPSM w/o A&C” and “NPSM w/o C” variants both lack contextual cue memory.
4.4. Comparison with State-of-the-art Methods
NPSM is compared with end-to-end and detector–recognizer person-search methods on CUHK-SYSU and PRW. It consistently outperforms the compared methods while using query-aware recursive region shrinkage.
- Results on CUHK-SYSU: NPSM beats all compared methods consistently on CUHK-SYSU for both mAP and top-1 metrics.The comparison uses a gallery size of 100 and includes methods based on detection, re-identification, and joint optimization.
- Results on CUHK-SYSU: Across gallery sizes 50, 100, 500, 1000, 2000, and 4000, NPSM remains ahead of other methods, improving average mAP over OIM by around 2%.mAP gradually decreases as gallery size increases for the compared methods.
- Results on PRW: NPSM outperforms OIM on PRW by 2.9% mAP and 3.2% top-1 accuracy.The comparison reports these gains against an OIM baseline implementation.
- Results on PRW: On PRW, NPSM achieves better performance than other state-of-the-art methods while retaining only one bounding box per gallery image for testing.The compared methods consider five bounding boxes per gallery image.
- Qualitative analysis: Attention-map visualizations from CUHK-SYSU and PRW show top-ranked search samples in which NPSM shrinks the search region to the correct person region.Green boxes denote ground truth, and red boxes denote regions highlighted by NPSM.
5. Conclusions
The paper introduces NPSM, which recursively localizes effective regions using memory of the query person. Experiments on two public benchmarks demonstrate superiority over state-of-the-art methods in most cases and improved recognition accuracy.
- NPSM solves person search by recursively localizing effective regions under guidance from the query person’s memory.
- Table 3 compares NPSM’s performance on the PRW dataset with state-of-the-art methods.
- Experiments on two public benchmarks show NPSM’s superiority over state-of-the-art methods in most cases.
- The experiments also demonstrate a benefit to recognition accuracy in person search.