PaperlayerView exact revision on arXiv

Source-linked AI summary

Person Search by Multi-Scale Matching

Xu Lan, Xiatian Zhu, Shaogang Gong

arXiv:1807.08582v1cs.CV

TL;DR

Person search must match identities across noisy detections whose scales vary widely, and the paper argues that this multi-scale challenge is more limiting than detection after reliable localization. It proposes end-to-end CLSA, which aligns semantics across an in-network feature pyramid, and reports superior Rank-1 accuracy on CUHK-SYSU and PRW.

  • Problem

    Person search remains challenging because auto-detected boxes vary in scale, while multi-scale matching is under-studied relative to detection and localization.

  • Method

    CLSA uses an end-to-end in-network feature pyramid with a cross-level semantic alignment loss to make representations across pyramid levels identity-discriminative.

  • Results

    CLSA improves the best competitor by 7.3% on CUHK-SYSU and 11.9% on PRW in Rank-1 accuracy.

  • Takeaways & Limitations

    Multi-scale matching is more critical than person detection for overall person-search performance in the reported experiments.

Abstract

from arXiv · show

We consider the problem of person search in unconstrained scene images. Existing methods usually focus on improving the person detection accuracy to mitigate negative effects imposed by misalignment, mis-detections, and false alarms resulted from noisy people auto-detection. In contrast to previous studies, we show that sufficiently reliable person instance cropping is achievable by slightly improved state-of-the-art deep learning object detectors (e.g. Faster-RCNN), and the under-studied multi-scale matching problem in person search is a more severe barrier. In this work, we address this multi-scale person search challenge by proposing a Cross-Level Semantic Alignment (CLSA) deep learning approach capable of learning more discriminative identity feature representations in a unified end-to-end model. This is realised by exploiting the in-network feature pyramid structure of a deep neural network enhanced by a novel cross pyramid-level semantic alignment loss function. This favourably eliminates the need for constructing a computationally expensive image pyramid and a complex multi-branch network architecture. Extensive experiments show the modelling advantages and performance superiority of CLSA over the state-of-the-art person search and multi-scale matching methods on two large person search benchmarking datasets: CUHK-SYSU and PRW.

1 Introduction

Person search is intrinsically a multi-scale matching problem because unconstrained scene detections vary substantially in scale, while prior work has emphasized detection and localization. The paper argues that multi-scale matching matters more for performance and introduces CLSA to address it.

  • Problem: Auto-detected person boxes vary significantly in scale, making person search intrinsically a multi-scale matching problem.CUHK-SYSU covers a wider person-scale range than manually refined CUHK-03.
  • Motivation: Ground-truth person boxes improve ResNet-50 Rank-1 by only 1.5% on CUHK-SYSU, whereas multi-scale matching learning improves it by 6.0%.These comparisons motivate prioritizing matching over further detection improvements.
  • Contribution: CLSA learns an end-to-end in-network feature-pyramid representation with cross-level semantic alignment for variable-scale detections.The approach is presented as a response to the under-studied multi-scale matching challenge.
  • Results: CLSA improves the best competitor by 7.3% on CUHK-SYSU and 11.9% on PRW in Rank-1 accuracy.The reported comparisons cover two person-search benchmarks.

2 Related Work

Prior person-search studies largely enhance detection, while this paper identifies multi-scale matching as the more severe challenge after reliable localization is achieved.

  • Prior work: Existing person-search work addresses detection, re-identification, detection effects, or recursive localization refinement.The related work includes joint detection and re-identification, detection analysis, and search refinement.
  • Research gap: The paper reports that improved object detection makes localization less limiting than multi-scale matching for person search.It frames scale variation as the more severe challenge and the more promising source of performance gain.

3 Cross-Level Semantic Alignment for Person Search

CLSA combines person detection with identity matching over arbitrary-scale detections, using a ResNet-50 in-network feature pyramid to align semantic information across levels. Its alignment loss transfers stronger top-level identity semantics to lower levels, while the final representation concatenates pyramid-level features.

  • Framework: The framework detects people in gallery scenes, then matches a probe against gallery boxes at varying scales.Person detection supplies crops; re-identification performs identity matching on arbitrary-scale boxes.
  • Person detection: Faster-RCNN is used for detection, with resized region feature maps and max-pooling replacing conventional RoI pooling.The modified detector is designed to improve detection performance and efficiency.
  • Feature pyramid: CLSA builds a computationally efficient three-level feature pyramid from the last convolutional layers of the top ResNet-50 blocks.The deepest block is intended to provide the strongest semantic features.
  • Feature representation: Each pyramid level produces a normalized feature vector through average pooling, a fully connected layer, batch normalization, and ReLU.The resulting representations are denoted {x1, · · · , xK}.
  • Semantic alignment: Cross-level semantic alignment projects pyramid features into a shared identity-semantic space and transfers top-level semantics to lower levels.The alignment formulation uses a KL-divergence-based loss inspired by knowledge distillation.
  • Training and matching: The alignment objective is added to cross-entropy training, with a weighting parameter balancing the two loss terms.At deployment, feature vectors from all pyramid levels are concatenated for re-identification matching, avoiding an external image pyramid.

4 Experiments

Experiments on CUHK-SYSU and PRW show that CLSA improves person-search accuracy, scalability, and robustness while maintaining computational efficiency. Further analyses indicate that multi-scale matching contributes more to performance than person detection alone.

  • Comparisons to State-Of-The-Art Person Search Methods: CLSA surpasses NPSM by 7.3% in Rank-1 and 9.3% in mAP on CUHK-SYSU with 100-image galleries.The reported values are 88.5 versus 81.2 in Rank-1 and 87.2 versus 77.9 in mAP.
  • Comparisons to State-Of-The-Art Person Search Methods: CLSA takes 1.2 seconds to search 100 CUHK-SYSU gallery images, compared with 0.8 seconds for OIM and 120 seconds for NPSM.The comparison is presented as evidence that CLSA’s performance advantages do not sacrifice model efficiency.
  • Comparisons to State-Of-The-Art Person Search Methods: When gallery size increases from 100 to 4,000 images, CLSA drops 9.7% in mAP and 9.1% in Rank-1, compared with NPSM’s 24.9% mAP degradation.The authors report greater performance robustness and larger gains for CLSA at larger search scales.
  • Comparisons to State-Of-The-Art Person Search Methods: CLSA surpasses NPSM by 11.9% in Rank-1 and 14.5% in mAP on PRW.The reported values are 65.0 versus 53.1 in Rank-1 and 38.7 versus 24.2 in mAP.
  • Further Analysis and Discussions: Replacing CLSA with the ResNet-50 baseline reduces Rank-1 by 6.0% and mAP by 5.6% on CUHK-SYSU.The reported comparisons are 88.5 versus 82.5 in Rank-1 and 87.2 versus 81.6 in mAP.
  • Further Analysis and Discussions: Perfect person detection adds only 1.5% in Rank-1 and 0.9% in mAP, whereas CLSA’s multi-scale approach improves the baseline by 6.0% and 5.6%.These results support the paper’s comparison between detection quality and multi-scale representation learning as performance factors.

5 Conclusion

The paper presents CLSA, an end-to-end framework that addresses multi-scale matching in person search through an in-network feature pyramid and semantic alignment loss. Evaluations on CUHK-SYSU and PRW show superiority over state-of-the-art methods.

  • CLSA targets the under-studied cross-scale matching challenge in person search, where person images exhibit large and unconstrained scale variations.The framework emphasizes multi-scale matching rather than treating person detection as the primary bottleneck.
  • The method constructs an in-network feature pyramid and applies semantic alignment learning to make all pyramid levels identity-discriminative.This produces a hierarchical representation for matching person images across scale variations.
  • CLSA integrates person detection and person re-identification in an end-to-end deep learning framework for unconstrained scene images.
  • Extensive evaluations on CUHK-SYSU and PRW report performance superiority over state-of-the-art person search, person re-identification, and multi-scale learning methods.
Loading 1807.08582v1…