PaperlayerView exact revision on arXiv

Source-linked AI summary

Repulsion Loss: Detecting Pedestrians in a Crowd

Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, Chunhua Shen

arXiv:1711.07752v2cs.CV

TL;DR

Crowd occlusion makes individual pedestrian localization difficult because overlapping people can cause detections to shift between targets. The paper analyzes this failure mode and introduces RepLoss, whose target attraction and surrounding-object repulsion improve crowd-robust localization and reported benchmark performance.

  • Problem

    Crowd occlusion causes pedestrians with similar appearance to interfere with localization, producing shifted boxes, missed detections, and false positives.

  • Method

    RepLoss combines attraction toward each proposal’s designated target with repulsion from surrounding ground-truth objects and predictions assigned to other targets.

  • Results

    The detector achieves the best reported performance on Caltech and CityPersons, improves occlusion-case accuracy, and reduces missed detections by relatively 10% with RepGT at a detection-score threshold near 0.9.

  • Takeaways & Limitations

    Repulsion-by-surrounding complements attraction-by-target and makes pedestrian detection more robust in crowd scenes, with usefulness also shown on PASCAL VOC.

  • Takeaways & Limitations

    The formulation considers two-class detection and assumes all ground-truth objects belong to the same category.

Abstract

from arXiv · show

Detecting individual pedestrians in a crowd remains a challenging problem since the pedestrians often gather together and occlude each other in real-world scenarios. In this paper, we first explore how a state-of-the-art pedestrian detector is harmed by crowd occlusion via experimentation, providing insights into the crowd occlusion problem. Then, we propose a novel bounding box regression loss specifically designed for crowd scenes, termed repulsion loss. This loss is driven by two motivations: the attraction by target, and the repulsion by other surrounding objects. The repulsion term prevents the proposal from shifting to surrounding objects thus leading to more crowd-robust localization. Our detector trained by repulsion loss outperforms all the state-of-the-art methods with a significant improvement in occlusion cases.

1. Introduction

Crowd occlusion is a major pedestrian-detection challenge because overlapping people can cause localization shifts, missed detections, and false positives. The paper studies this failure mode and proposes RepLoss, which combines attraction to the target with repulsion from surrounding objects.

  • Motivation: Crowd occlusion occurs when pedestrians occlude others of the same category, and it constitutes the majority of occlusion cases in pedestrian detection.In CityPersons, 48.8% of validation annotations overlap another pedestrian at IoU above 0.1, while 26.4% overlap at IoU above 0.3.
  • Motivation: Crowd overlap can shift a predicted box from its target pedestrian to a similar-looking neighboring pedestrian, producing inaccurate localization and more missed detections after NMS.The same failure can also harm instance-segmentation frameworks that depend on accurate detection results.
  • Motivation: Standard bounding-box regression attracts proposals toward designated targets but does not penalize shifts toward surrounding objects.This omission is problematic when neighboring pedestrians overlap and have similar appearance features.
  • Method: RepLoss adds repulsion from surrounding non-target objects and proposals to prevent predicted boxes from shifting toward adjacent overlapped pedestrians.Its two terms are attraction to the designated target and repulsion from surrounding objects.
  • Method: The paper proposes RepGT Loss for shifts toward other ground-truth objects and RepBox Loss for separation between predictions with different designated targets.RepBox also makes detection results less sensitive to NMS.
  • Results: The RepLoss detector outperforms state-of-the-art methods on CityPersons and Caltech-USA, with significant gains for occlusion cases and benefits on PASCAL VOC.The supplied result passage reports the cross-benchmark comparison and broader object-detection usefulness without providing numerical scores.

2. Related Work

Prior work advances object localization and pedestrian detection through regression, CNN-based detectors, part-based models, and NMS improvements. The paper identifies crowd occlusion as comparatively underexplored and examines baseline failures on pedestrian subsets.

  • Object Localization: Object localization is generally formulated as regression that relocates an initial proposal to its designated target.Earlier approaches use coordinate-distance regression, while SmoothL1 Loss and region proposal networks improve bounding-box regression frameworks.
  • Pedestrian Detection: Pedestrian detection has progressed from handcrafted-feature sliding-window detectors to CNN-based, multitask, part-based, and NMS-focused methods.These approaches target detection accuracy, occluded pedestrians, or post-processing robustness through different mechanisms.
  • Pedestrian Detection: Few previous works focus on studying and overcoming the impact of crowd occlusion in pedestrian detection.This gap motivates the paper’s focused analysis of crowd-related failures.
  • Pedestrian Detection: Figure 2 reports missed-detection numbers and MR−2 scores for a baseline on the reasonable, reasonable-occ, and reasonable-crowd subsets.The reasonable-occ subset is used to quantify the role of crowd occlusion in missed detections.

3. What is the Impact of Crowd Occlusion?

Experiments on CityPersons show that crowd occlusion is a major source of pedestrian detector errors, worsening missed detections and producing high-confidence false positives. These failures arise from inaccurate localization around neighboring or overlapping pedestrians.

  • Experimental setup: CityPersons evaluation uses MR−2 over FPPI [10−2, 100], with lower values indicating better detection.The experiments use reasonable train/validation sets and a Faster R-CNN-based baseline with a ResNet-50 backbone.
  • Experimental setup: The reasonable-crowd subset contains 479 of 1,579 non-ignored validation pedestrians, or 30.3%.Crowd occlusion requires occ ≥0.1 and IoU ≥0.1 with another annotated pedestrian.
  • Missed detections: 19.1 MR−2 on reasonable-crowd versus 18.6 on reasonable-occ and 14.6 on reasonable shows crowd occlusion is especially difficult.Crowd occlusion accounts for approximately 60% of missed detections, rising from 60.7% to 69.2% as the false-positive count increases from 100 to 500.
  • False positives: About 20% of all false positives are crowd errors, often caused by boxes shifting toward neighboring pedestrians or covering several overlapping pedestrians.These errors commonly have relatively high confidence, making them top-ranked false positives.

4. Repulsion Loss

Repulsion Loss augments attraction-based bounding-box regression with penalties that separate predictions from neighboring ground-truth objects and predictions. Its design uses RepGT and RepBox terms to reduce harmful overlap and improve robustness in crowd scenes.

  • Repulsion Loss: Repulsion Loss combines attraction to a designated target with repulsion from surrounding ground-truth objects and predicted boxes.The attraction term approaches the designated target, while LRepGT and LRepBox discourage overlap with neighboring objects and predictions having different targets.
  • Attraction Term: Each positive proposal is assigned the ground-truth box with maximum IoU as its designated target.The attraction term uses SmoothL1 distance with smooth parameter 2.
  • Repulsion Term (RepGT): RepGT penalizes overlap between a predicted box and its largest-overlap non-target ground-truth object.Its penalty increases as the predicted box overlaps more with the non-target object, discouraging shifts toward neighboring pedestrians.
  • Repulsion Term (RepBox): RepBox minimizes overlap between predictions assigned to different ground-truth targets, reducing their likelihood of being merged by NMS.The loss samples proposals from different target-based subsets and penalizes their predicted-box IoU.
  • Distance Metric: IoG and IoU are used for repulsion because their values are bounded in [0, 1], whereas SmoothL1 is boundless.IoG is selected for RepGT because its ground-truth denominator is constant, avoiding enlargement of the predicted box as a way to reduce loss.
  • Smooth Parameter σ: The smooth parameter σ adjusts repulsion-loss sensitivity to outliers, with smaller σ making Smoothln less sensitive to outliers.The paper notes that denser predicted boxes make large-overlap outlier pairs more likely in RepBox.

5. Experiments

Experiments evaluate RepLoss variants, loss balancing, occlusion levels, NMS sensitivity, and comparisons on CityPersons and Caltech-USA. RepLoss improves crowd-scene detection, especially under heavier occlusion, while making results less sensitive to NMS thresholds.

  • Ablation Study: 10% fewer missed detections were observed with RepGT near the commonly used detection-score threshold of ∼0.9.RepGT also produces a smaller proportion of crowd-occlusion false positives than the baseline.
  • Ablation Study: 3.5 points separate RepBox from the baseline at an NMS threshold of 0.35 and FPPI = 10^-2.RepBox’s performance curve is smoother, indicating lower sensitivity to the NMS threshold; visualizations show fewer predictions between adjacent ground truths.
  • Ablation Study: Experiments compare RepGT and RepBox losses, their smooth parameters, and their balance on CityPersons validation data.The combined-loss weights α = 0.5 and β = 0.5 yield the best empirical performance.
  • Comparisons with State-of-the-art Methods: 3.7 points is the RepLoss improvement on the Caltech-USA Heavy subset, compared with 1.8 points on the Partial subset and non-obvious improvement on Bare.On the new Caltech test subset, RepLoss achieves 13.2 MR^-2, an absolute 1.4-point improvement over the baseline.
  • Comparisons with State-of-the-art Methods: RepLoss achieves MR^-2 of 5.0 at .5 IoU and 26.3 at .75 IoU on the new Caltech-USA evaluation.The experiments compare the method with state-of-the-art detectors on CityPersons and Caltech-USA.

6. Extensions: General Object Detection

The paper extends RepLoss beyond pedestrian detection to general object detection using PASCAL VOC and Faster R-CNN. Its overall gain is not significant, but performance improves on the crowd subset.

  • General Object Detection: RepLoss is presented as a generic loss function for object detection in crowd scenes beyond pedestrian detection.The extension is evaluated on PASCAL VOC using a vanilla Faster R-CNN framework.
  • General Object Detection: 2.1 mAP is the RepLoss improvement over baseline on the PASCAL VOC crowd subset.The crowd subset contains same-category ground-truth objects with overlaps above 0.1 IoU.
  • General Object Detection: The gain over the entire PASCAL VOC dataset is not significant.The authors nevertheless report that the crowd-subset result demonstrates extension to general object detection.

7. Conclusion

The conclusion frames RepLoss as a localization loss that complements attraction to the target with repulsion from surrounding objects. Two repulsion variants produce strong benchmark results and improve occlusion-case detection.

  • Conclusion: RepLoss improves pedestrian detection performance, particularly in crowd scenes.The conclusion attributes its motivation to the insufficiency of attraction-by-target loss alone and the benefit of repulsion-by-surrounding.
  • Conclusion: Two repulsion losses implement the repulsion energy, and the paper reports its best performance on Caltech and CityPersons.The CityPersons result without pixel annotation exceeds the previously best result using pixel annotation by about 2%.
  • Conclusion: RepLoss improves detection accuracy by a large margin in occlusion scenarios and is also useful for generic object detection.The conclusion points to detailed comparisons and PASCAL VOC results as evidence for these claims.

A. The examples of missed detections and false positives

Supplementary visual examples illustrate RepLoss behavior on missed detections and false positives in challenging CityPersons scenes. The figures identify prediction scores and distinguish baseline from RepLoss outputs.

  • Examples of missed detections and false positives: Figure 9 uses blue boxes for false positives and red boxes for missed detections when comparing baseline with RepLoss.Samples above and below the dashed line illustrate false-positive elimination and detection of more missed pedestrians, respectively.
  • Examples of missed detections and false positives: Figure 10 highlights predicted pedestrians with scores greater than 0.8 on challenging CityPersons examples.Green bounding boxes represent these high-scoring predictions.
Loading 1711.07752v2…