Source-linked AI summary
IAN: The Individual Aggregation Network for Person Search
Jimin Xiao, Yanchun Xie, Tammam Tillo, Kaizhu Huang, Yunchao Wei, Jiashi Feng
TL;DR
Person search must find people in uncropped gallery images despite unavailable pedestrian boxes and large appearance variation. IAN combines online pedestrian proposals from Faster R-CNN with center loss for more compact same-identity features. It reports state-of-the-art performance on CUHK-SYSU and PRW, including 77.23% mAP and 80.45% top-1 accuracy on CUHK-SYSU.
Problem
Person search must jointly handle unavailable pedestrian bounding boxes and substantial same-person appearance variation in whole gallery images.
Method
IAN builds person search on Faster R-CNN for online pedestrian proposals and adds center loss to increase intra-class feature compactness.
Results
77.23% mAP and 80.45% top-1 accuracy are reported on CUHK-SYSU, while state-of-the-art performance is also reported on PRW.
Takeaways & Limitations
IAN is reported to accurately localize pedestrians and reduce intra-person feature variation across the CUHK-SYSU and PRW benchmarks.
Takeaways & Limitations
IAN requires substantial GPU memory because center loss tracks feature centers for all classes.
Abstract
from arXiv · showhide
Person search in real-world scenarios is a new challenging computer version task with many meaningful applications. The challenge of this task mainly comes from: (1) unavailable bounding boxes for pedestrians and the model needs to search for the person over the whole gallery images; (2) huge variance of visual appearance of a particular person owing to varying poses, lighting conditions, and occlusions. To address these two critical issues in modern person search applications, we propose a novel Individual Aggregation Network (IAN) that can accurately localize persons by learning to minimize intra-person feature variations. IAN is built upon the state-of-the-art object detection framework, i.e., faster R-CNN, so that high-quality region proposals for pedestrians can be produced in an online manner. In addition, to relieve the negative effect caused by varying visual appearances of the same individual, IAN introduces a novel center loss that can increase the intra-class compactness of feature representations. The engaged center loss encourages persons with the same identity to have similar feature characteristics. Extensive experimental results on two benchmarks, i.e., CUHK-SYSU and PRW, well demonstrate the superiority of the proposed model. In particular, IAN achieves 77.23% mAP and 80.45% top-1 accuracy on CUHK-SYSU, which outperform the state-of-the-art by 1.7% and 1.85%, respectively.
I. INTRODUCTION
Person search must jointly address pedestrian localization in uncropped gallery images and large appearance variation across views. IAN combines online proposals from Faster R-CNN with center loss to compact same-identity features, achieving strong benchmark results.
- Challenge: Person search requires detecting pedestrians in whole gallery images because perfect pedestrian bounding boxes are unavailable.The detected pedestrians are then compared with the probe person.
- Challenge: End-to-end person search jointly handles detection and re-identification, reducing the influence of detection misalignments.Earlier re-identification systems generally assumed that pedestrian boxes were already available.
- IAN: IAN uses Faster R-CNN to produce high-quality pedestrian region proposals online and center loss to increase intra-class feature compactness.Center loss pulls sample features toward the center of their class and encourages same-identity representations to share similar characteristics.
- IAN: Center loss addresses appearance variation by reducing intra-person feature differences without requiring positive and negative verification samples.The paper also reports that dropout is incompatible with center loss.
- Results: 77.23% mAP and 80.45% top-1 accuracy are reported on CUHK-SYSU, with state-of-the-art performance also reported on PRW.These results are described as outperforming prior state-of-the-art performance.
II. RELATED WORKS
Related work spans Siamese and classification-based re-identification models, while person search also requires integrating pedestrian detection with retrieval. These approaches differ in how they use labels and whether detection boxes are assumed to be available.
- CNN-based re-identification: Person re-identification CNNs commonly use either Siamese models with image pairs or triplets, or classification models.These represent the two broad CNN model categories described in the related work.
- CNN-based re-identification: Siamese models use pairwise similarity labels but cannot fully exploit re-identification annotations.Classification models make fuller use of labels and perform well on large datasets such as PRW and MARS.
- Pedestrian detection: Person search must account for pedestrian detection because conventional re-identification typically matches queries against cropped or pre-detected pedestrians.CNN-based pedestrian detectors are part of the broader detection research landscape.
III. INDIVIDUAL AGGREGATION NETWORK
IAN combines Faster R-CNN-based online pedestrian proposals with center-loss feature learning to address unavailable boxes and intra-person appearance variation. Its training uses candidate boxes for softmax classification while applying center loss to ground-truth pedestrian features.
- IAN uses Faster R-CNN to produce pedestrian region proposals online for person search when bounding boxes are unavailable.At test time, only RPN boxes are available.
- The Faster R-CNN person-search mini-batch makes balanced positive and negative pair construction difficult for contrastive or triplet loss.Pedestrians within images have different, random, sparse, and unbalanced identities.
- The training framework extracts image features, applies ROI pooling to RPN and ground-truth boxes, and produces separate feature and bounding-box outputs.All candidate features enter random-sampling softmax, whereas only ground-truth features enter center loss.
- Center loss pulls same-identity features toward class centers, complementing softmax loss, which separates features from different classes.The center loss avoids aggregating positive and negative verification pairs.
- The overall objective combines Faster R-CNN losses with weighted center loss, while class centers are updated from mini-batch features using a controlled learning rate.The scalar α limits perturbations from few mislabeled samples.
B. Why to Avoid Dropout?
The paper argues that dropout conflicts with center loss because random unit removal creates feature inconsistency for identical inputs. An illustrative derivation and experiments support avoiding dropout in this setting.
- A VGGNet IAN with three dropout layers achieves about 10% lower CUHK-SYSU person-search mAP than the version without dropout.
- Dropout randomly removes units and creates uncertainty in features extracted from the same image across forward passes.This conflicts with center loss, which penalizes intra-class feature variation.
- For identical same-class images, different dropout masks prevent equal outputs unless the preceding feature vector is zero.The paper notes that zero features cannot properly represent the image samples.
A. Training Phase
Training jointly teaches IAN to detect pedestrians and produce re-identification features using five losses, with center loss restricted to ground-truth pedestrian boxes. Training proceeds through staged fine-tuning.
- IAN training uses five losses for bounding-box regression, pedestrian proposal classification, random-sampling softmax, and center-loss feature learning.The overall loss is summed and optimized by back propagation.
- Only features from ground-truth pedestrian boxes enter center loss during training to reduce sample noise.
- Training first fine-tunes residual features on cropped pedestrian and background boxes, then fine-tunes whole images without center loss, and finally with all five losses.The first stage uses resized 224 × 224 boxes and residual-network models.
B. Test Phase
At test time, IAN extracts features for gallery candidates and the query box, then compares them using pairwise Euclidean distances.
- Gallery images use RPN candidate boxes, while the query uses its given pedestrian bounding box for feature extraction.The gallery features are obtained with one network forward computation per gallery image.
- Person similarity is evaluated from pairwise Euclidean distances between query and gallery-candidate features.
V. EXPERIMENTS
Experiments evaluate IAN on CUHK-SYSU and PRW using mAP and top-1 matching rate, with a standard positive-window overlap threshold. The CUHK-SYSU protocol also varies gallery size and uses cross-validation for ablation studies.
- Both CUHK-SYSU and PRW are evaluated with mAP and top-1 matching rate, treating windows with overlap above 0.5 as positive.
- CUHK-SYSU contains 18,184 images, 8,432 persons, and 99,809 annotated bounding boxes.
- PRW contains 11,816 annotated frames and 43,110 pedestrian bounding boxes, with separate training and test identities.
- Ablation studies are based on CUHK-SYSU to compare IAN with E2E-PS and JDI-PS.
- CUHK-SYSU testing varies gallery size from 50 to 4,000, with gallery size 100 used by default.
A. Results on CUHK-SYSU dataset
On CUHK-SYSU, IAN achieves strong person-search performance and remains competitive across gallery sizes and difficult conditions. Ablations show that center-loss inputs, network architecture, dropout compatibility, and loss weight materially affect performance.
- Loss-weight sensitivity: Setting the center-loss weight to λ = 0.032 improves accuracy, while performance remains largely stable across λ values from 0.016 to 0.128.
- Overall performance: More than 7% gain over E2E-PS is obtained for both mAP and top-1 accuracy with ResNet-101.
- Center-loss inputs: Using ground-truth pedestrian boxes in center loss performs better than using all positive pedestrian boxes.
- Dropout compatibility: With dropout and center loss together, VGGNet loses almost 9% mAP after 40,000 iterations compared with models without center loss.
- Dropout compatibility: About 2% performance gain over E2E-PS is obtained for both metrics when dropout layers are removed from VGGNet.
- Gallery size and robustness: At gallery size 4,000, IAN gains more than 10% mAP over E2E-PS and 3% over JDI-PS.
- Gallery size and robustness: IAN significantly outperforms E2E-PS on both occlusion and low-resolution subsets.
B. Results on PRW dataset
On PRW, IAN outperforms the strongest reported detection-and-identification combination and E2E-PS, demonstrating improved person-search performance on this benchmark.
- IAN exceeds DPM-Alex+IDEdet by more than 14% top-1 accuracy on PRW.
- IAN also performs better than E2E-PS and DPM-Alex+IDEdet on PRW.
VI. CONCLUSIONS
IAN combines online pedestrian localization with reduced intra-person feature variation, achieving state-of-the-art performance on CUHK-SYSU and PRW while requiring substantial GPU memory.
- IAN uses Faster R-CNN to produce high-quality pedestrian region proposals online for person search.
- IAN incorporates center loss to reduce intra-person feature variation caused by large visual-appearance changes.
- IAN achieves state-of-the-art performance on both the CUHK-SYSU and PRW benchmarks.
- Large GPU memory requirements remain a limitation because center loss tracks feature centers for all classes.Reducing memory cost and network computational complexity is identified as future work.