Source-linked AI summary
Learning Context Graph for Person Search
Yichao Yan, Qiang Zhang, Bingbing Ni, Wendong Zhang, Minghao Xu, Xiaokang Yang
TL;DR
Person search must handle appearance variation and difficult scenes, while many methods rely primarily on individual appearance features. This paper expands target features with selectively matched context and learns a context graph on top of joint detection and feature learning. The framework achieves state-of-the-art performance on two widely used person-search benchmarks.
Problem
Person-search models that focus on individual appearance features struggle with illumination, pose variance, occlusion, and scene-context variation.
Method
The framework uses relative attention to filter informative context pairs and graph learning to aggregate them for target-similarity estimation within a joint detection and feature-learning system.
Results
The framework achieves state-of-the-art performance on two widely adopted person-search benchmarks.
Takeaways & Limitations
Contextual instance expansion and graph learning improve the robustness of person-search results by aggregating useful scene information.
Abstract
from arXiv · showhide
Person re-identification has achieved great progress with deep convolutional neural networks. However, most previous methods focus on learning individual appearance feature embedding, and it is hard for the models to handle difficult situations with different illumination, large pose variance and occlusion. In this work, we take a step further and consider employing context information for person search. For a probe-gallery pair, we first propose a contextual instance expansion module, which employs a relative attention module to search and filter useful context information in the scene. We also build a graph learning framework to effectively employ context pairs to update target similarity. These two modules are built on top of a joint detection and instance feature learning framework, which improves the discriminativeness of the learned features. The proposed framework achieves state-of-the-art performance on two widely used person search datasets.
1. Introduction
Person re-identification searches gallery recordings for a probe identity but remains difficult under cross-source variation, illumination, pose, detection errors, occlusion, and clutter. The proposed framework expands individual features with filtered scene context and graph-based similarity learning, achieving improvements over prior state-of-the-art methods.
- Person re-identification searches a gallery of known identity recordings for matches to a target probe across cameras.
- Different data sources, illumination, pose, inaccurate detection or tracking, occlusion, and background clutter create substantial appearance variation.
- Context can complement individual appearance because co-travelers and neighboring pedestrians may recur across cameras with the target person.
- The framework collects scene pedestrians as context candidates, uses relative attention to select matched informative pairs, and builds a context graph to update target similarity.
- Experiments on CUHK-SYSU and PRW show significant improvements over previous state-of-the-art methods.
- The system combines end-to-end human detection and multi-part feature learning with adaptive context selection and graph-based global similarity learning.
2. Related Work
Prior person re-identification work centers on feature representation, distance metric learning, and joint detection-reidentification, while newer graph methods generally omit scene context. This paper instead designs a target-context graph with pairwise graph learning for person search.
- Earlier methods used hand-crafted color, texture, and gradient features, whose representation capacity was limited for large-scale searching.
- Recent person-search methods jointly perform detection and re-identification but verify targets mainly through individual appearance, ignoring relationships among scene individuals.
- Existing graph methods for person re-identification model probe-gallery relationships without incorporating context information.
- The proposed approach designs a target-context graph and employs pairwise graph convolutional learning for visual similarity.
3. Methodology
The framework combines joint detection and part-based feature learning with contextual instance expansion and graph-based similarity learning for robust person search.
- Instance Detection and Feature Learning: The framework uses joint pedestrian detection and feature learning as its backbone, rather than separating detection from feature extraction.It follows a Faster R-CNN-style structure with a proposal network and RoI pooling.
- Contextual Instance Expansion: Contextual instance expansion filters context candidates with relative attention based on visual similarity between context pairs.The attention network processes four feature-pair inputs and outputs four normalized weights.
- Instance Detection and Feature Learning: Part-based pooling complements global features by extracting 2048-dimensional representations for specific human regions and normalizing them into 256-dimensional features.The framework models upper, middle, and lower body regions in addition to whole-box features.
- Instance Detection and Feature Learning: OIM loss supervises part-feature learning because large identity numbers and small mini-batches make conventional Softmax training difficult.The detection framework’s memory consumption limits mini-batch size, producing sparse identity coverage.
- Contextual Instance Expansion: A cosine embedding verification loss separates positive and negative pairs, while top-K matched context pairs are selected for further feature learning.The loss uses a margin for negative pairs and is jointly optimized with OIM loss.
- Contextual Graph Representation Learning: The contextual graph represents target and context pairs as nodes, connects the target node to context nodes, and applies a GCN to learn target-pair similarity.The graph is designed to propagate contextual information between paired instances rather than using separate image graphs.
4. Experiments
Experiments analyze part-based features, attention-based context selection, contextual graph learning, and comparisons with state-of-the-art person-search methods on CUHK-SYSU and PRW.
- Experiment settings: The experiments evaluate the framework on CUHK-SYSU and PRW using mean Average Precision and top-1 matching rate under the person-search protocol.A top-1 match is accepted only when the predicted and ground-truth boxes overlap by more than 0.5.
- Effective of Part-based Learning Framework: The part-based analysis splits each detection into upper, middle, and lower regions and also evaluates a global whole-box feature.Part-based pooling layers produce the regional features used for matching.
- Instance Expansion and Contextual Learning: About 2% improvement over uniform weighting is obtained by the relative attention module when learning pairwise-part weights for overall similarity.The module adaptively downweights occluded or misleading regions instead of combining part features uniformly.
- Instance Expansion and Contextual Learning: With K = 3 context pairs, the contextual graph reaches 86.5% rank-1 accuracy and 84.1% mAP on CUHK-SYSU, and 73.6% rank-1 accuracy and 33.4% mAP on PRW.The graph aggregates selected context information for global representation learning.
- Comparison with State-of-the-Art Methods: On CUHK-SYSU with gallery size 100, the framework improves over the original OIM baseline by about 8% on both mAP and top-1 matching rate.It also exceeds IAN by more than 6% while using a ResNet-50 backbone.
- Comparison with State-of-the-Art Methods: The method remains superior across gallery sizes from 50 to 4000, while performance of all methods declines as gallery size increases.On PRW, it achieves 73.6% top-1 accuracy and 33.4 mAP, outperforming previous state-of-the-art methods.
- Discussion on Graph Structures: The proposed graph outperforms alternative baseline graph structures because pairwise relations are contained within each graph node, making context easier to capture.The results support graph structures as useful for learning contextual features.
5. Conclusion
The paper concludes that contextual information improves robust person search through instance expansion and graph-based aggregation. The framework achieves state-of-the-art performance on two widely used person-search benchmarks.
- The instance expansion method finds useful contextual information, while graph learning aggregates it to update target similarity.These components are presented as the framework’s mechanisms for employing contextual information.
- The proposed framework achieves state-of-the-art performance on two widely adopted person-search benchmarks.