Source-linked AI summary
Occlusion-aware R-CNN: Detecting Pedestrians in a Crowd
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, Stan Z. Li
TL;DR
Crowded pedestrian detection is difficult because pedestrians gather together and occlude one another. OR-CNN combines aggregation loss with part occlusion-aware pooling and achieves state-of-the-art results on CityPersons, ETH, and INRIA, while performing on-pair with state-of-the-art methods on Caltech.
Problem
Pedestrian detection in crowded scenes must handle frequent mutual occlusion, which makes detection challenging.
Method
OR-CNN uses aggregation loss to compactly align proposals and PORoI pooling to combine human-body structure information with visibility prediction.
Results
OR-CNN achieves state-of-the-art results on CityPersons, ETH, and INRIA, and performs on-pair with state-of-the-art methods on Caltech.
Takeaways & Limitations
The end-to-end OR-CNN improves pedestrian detection accuracy in crowded scenes by addressing proposal compactness and partial occlusion.
Takeaways & Limitations
Because of computational and memory constraints, OR-CNN is evaluated only with ×1 and ×1.3 input scales.
Abstract
from arXiv · showhide
Pedestrian detection in crowded scenes is a challenging problem since the pedestrians often gather together and occlude each other. In this paper, we propose a new occlusion-aware R-CNN (OR-CNN) to improve the detection accuracy in the crowd. Specifically, we design a new aggregation loss to enforce proposals to be close and locate compactly to the corresponding objects. Meanwhile, we use a new part occlusion-aware region of interest (PORoI) pooling unit to replace the RoI pooling layer in order to integrate the prior structure information of human body with visibility prediction into the network to handle occlusion. Our detector is trained in an end-to-end fashion, which achieves state-of-the-art results on three pedestrian detection datasets, i.e., CityPersons, ETH, and INRIA, and performs on-pair with the state-of-the-arts on Caltech.
1 Introduction
Crowded-scene pedestrian detection is difficult because pedestrians occlude one another, while prior proposal handling overlooks relations among detections. OR-CNN addresses this with aggregation loss and part occlusion-aware pooling, achieving state-of-the-art results on three datasets.
- 48.8% of annotated pedestrians in CityPersons are occluded by other pedestrians, underscoring the challenge of crowded-scene detection.
- Prior methods make detectors sensitive to NMS thresholds because they constrain proposals individually without modeling relations among pedestrians.
- OR-CNN introduces AggLoss to bring proposals close to corresponding objects while minimizing internal distances among proposals for the same object.
- PORoI pooling replaces the Fast R-CNN RoI pooling layer and combines human-body structure information with visibility prediction for partial occlusion.
- 11.3% MR−2 on CityPersons, 24.5% MR−2 on ETH, and 6.4% MR−2 on INRIA are reported as state-of-the-art results.
2 Related Work
CNN-based and part-based methods dominate pedestrian detection, but occlusion remains a major localization challenge. Existing approaches use part detectors, shape models, or visibility reasoning to recover partially occluded pedestrians.
- CNN-based methods have become dominant in pedestrian detection, following earlier sliding-window approaches using hand-crafted features and classifiers.
- Occlusion remains a major challenge because it increases the difficulty of pedestrian localization.
- Part-based approaches learn part detectors and fuse their outputs to localize partially occluded pedestrians.
- Other occlusion methods generate pedestrian hypotheses, refine visible regions, or estimate visibility across template blocks.
3 Occlusion-aware R-CNN
OR-CNN extends Faster R-CNN with aggregation loss and a part occlusion-aware RoI pooling unit. The loss promotes compact proposals, while visibility-aware part features support classification and regression under occlusion.
- Overall architecture: OR-CNN follows Faster R-CNN, using an RPN to generate proposals and Fast R-CNN to classify and regress their locations.The proposed aggregation loss and PORoI pooling unit are incorporated into these two modules.
- Aggregation loss: In the RPN, AggLoss complements classification loss by enforcing proposals to locate closely and compactly around ground-truth pedestrians.The classification term uses log loss, while the aggregation term targets proposal compactness.
- Aggregation loss: AggLoss pushes proposals toward their corresponding ground-truth objects while minimizing internal distances among proposals associated with the same object.It is used in both the RPN and Fast R-CNN modules to reduce false detections from adjacent overlapping pedestrians.
- Aggregation loss: For ground-truth objects associated with multiple anchors, the compactness loss compares average anchor predictions with the corresponding ground-truth object using smooth L1 loss.This term operates over groups of anchors associated with the same object.
- Part occlusion-aware RoI pooling: PORoI divides each pedestrian proposal into five parts, predicts each part's visibility, and combines visibility-weighted part features with a holistic proposal feature.The resulting feature is used for subsequent classification and regression; visibility supervision treats a part as visible when the overlap ratio with the visible ground-truth region exceeds 0.5.
- Part occlusion-aware RoI pooling: The occlusion process unit uses three convolutional layers and a softmax layer to estimate part visibility, then element-wise multiplication and summation produce the final 512 × 7 × 7 feature.The five part features and the whole-proposal feature are combined for Fast R-CNN classification and regression.
4 Experiments
Experiments evaluate OR-CNN and its components on four pedestrian-detection datasets, using MR−2 and comparisons with baselines and state-of-the-art methods. The results show gains from AggLoss and PORoI pooling, strong performance across occlusion levels, cross-dataset generalization, and a remaining dependence on input scale and backbone capacity.
- Experimental setup: Experiments cover CityPersons, Caltech-USA, ETH, and INRIA using Faster R-CNN-based OR-CNN configurations and MR−2 evaluation.The implementation uses VGG-16, dense small-scale anchors, and SGD training; MR−2 averages log miss rate over FPPI [10^-2, 100], with lower values better.
- Ablation studies: AggLoss reduces CityPersons validation MR−2 by 1.1%, from 12.5% to 11.4% at ×1.3 scale, surpassing Repulsion Loss at 11.6%.Before NMS, AggLoss produces more compact predictions with fewer boxes between adjacent ground-truth objects, making detection less sensitive to the NMS threshold.
- Ablation studies: PORoI pooling reduces baseline MR−2 by 0.8% on CityPersons validation at the reported ×1.3 setting.Its visibility predictions approach 1 for unoccluded parts and decrease for occluded parts, helping extract effective features for detection.
- CityPersons results: OR-CNN improves CityPersons validation MR−2 by 1.1% on Bare, 1.1% on Partial, and 4.0% on Heavy subsets, with the largest gain under severe occlusion.The subsets correspond to occlusion ≤10%, 10%–35%, and greater than 35%, respectively.
- Cross-dataset results: OR-CNN achieves 24.5% MR−2 on ETH without fine-tuning, reducing the state-of-the-art result of 30.2% by 5.7%.The CityPersons-trained model is evaluated on all 1,804 ETH frames from three video clips, supporting the reported cross-scenario generalization result.
- Cross-dataset results: OR-CNN achieves 6.4% MR−2 on INRIA and reports competitive Caltech-USA performance, while larger inputs and stronger backbones remain avenues for improvement.On CityPersons testing, the comparison is limited to ×1 and ×1.3 scales because of computational and memory constraints; Repulsion Loss used ×1.5 and ResNet-50 versus OR-CNN’s ×1.3 and VGG-16.
5 Conclusions
OR-CNN improves pedestrian detection in crowded scenes through aggregation loss and part occlusion-aware RoI pooling. It achieves state-of-the-art accuracy on CityPersons, ETH, and INRIA, and performs on par with state-of-the-art methods on Caltech.
- OR-CNN uses aggregation loss to keep proposals close to associated objects and compactly located.
- Its PORoI pooling unit replaces RoI pooling and combines human-body structure information with visibility prediction to handle partial occlusion.
- The detector is trained end-to-end.
- OR-CNN achieves state-of-the-art accuracy on CityPersons, ETH, and INRIA, and performs on par with state-of-the-art methods on Caltech.
- Future work will redesign PORoI pooling to jointly estimate part location, size, and occlusion status, and extend the method to other tasks.