Source-linked AI summary
Detection in Crowded Scenes: One Proposal, Multiple Predictions
Xuangeng Chu, Anlin Zheng, Xiangyu Zhang, Jian Sun
TL;DR
Crowded detection is difficult because highly overlapped instances share similar features and valid detections can be mistakenly suppressed by NMS. The paper lets each proposal predict a correlated instance set, using EMD Loss and Set NMS. It reports gains on CrowdHuman, CityPersons, and COCO, suggesting robustness across crowdedness levels.
Problem
Highly overlapped instances are difficult to detect because they have similar features and may be mistakenly suppressed by NMS.
Method
Each proposal predicts a set of potentially overlapped instances, supervised by EMD Loss and deduplicated with Set NMS.
Results
4.9% AP gain on CrowdHuman and 1.0% MR−2 improvement on CityPersons are reported, with a 1.0% AP gain on COCO.
Takeaways & Limitations
The method is reported to handle crowded and less crowded scenes and to cooperate flexibly with proposal-based detection frameworks.
Takeaways & Limitations
Re-scoring requires nearby proposals with similar features to produce different predictions, described as infeasible for neural networks on CrowdHuman; multiple-instance prediction may also increase false positives.
Abstract
from arXiv · showhide
We propose a simple yet effective proposal-based object detector, aiming at detecting highly-overlapped instances in crowded scenes. The key of our approach is to let each proposal predict a set of correlated instances rather than a single one in previous proposal-based frameworks. Equipped with new techniques such as EMD Loss and Set NMS, our detector can effectively handle the difficulty of detecting highly overlapped objects. On a FPN-Res50 baseline, our detector can obtain 4.9\% AP gains on challenging CrowdHuman dataset and 1.0\% $\text{MR}^{-2}$ improvements on CityPersons dataset, without bells and whistles. Moreover, on less crowed datasets like COCO, our approach can still achieve moderate improvement, suggesting the proposed method is robust to crowdedness. Code and pre-trained models will be released at https://github.com/megvii-model/CrowdDetection.
1. Introduction
Proposal-based detectors can miss highly overlapped instances because similar features make predictions difficult and NMS may suppress valid detections. The paper addresses this by having each proposal predict an instance set, supported by EMD Loss and Set NMS.
- Proposal-based detectors generate proposals, then predict one instance, confidence score, and refined location for each proposal.
- Highly overlapped instances have similar features, making it difficult to produce distinguishing predictions for each proposal.
- NMS can mistakenly suppress valid detections when instances heavily overlap.
- The proposed scheme makes each proposal predict a set of potentially overlapped instances rather than a single instance.
- The method adds EMD Loss for instance-set supervision, Set NMS for duplicate-set suppression, and an optional refinement module for potential false positives.
- 4.9% AP improvement on CrowdHuman and 1.0% MR−2 improvement on CityPersons are reported, while COCO also receives a 1.0% AP gain.
2. Background
The background analyzes why single-instance prediction and NMS are inadequate for crowded scenes. It motivates multiple-instance prediction as an alternative to increasingly complex or ineffective workarounds.
- Proposal-based detectors first generate object proposals and then predict refined instance results corresponding to each proposal.
- Single-instance prediction binds each proposal to one ground-truth instance, requiring overcomplete proposals that create duplicate predictions.
- In crowded scenes, NMS can suppress nearby people and cause missed detections.
- Advanced NMS: Advanced NMS methods soften or learn duplicate removal, but can rely on heuristics or make the pipeline more complex.
- Loss functions for crowded detection: New loss functions improve crowded-scene detections by compacting proposals or penalizing proposals that intertwine with multiple ground truths.
- Re-scoring: Re-scoring requires nearby proposals with similar features to produce different predictions, which is described as infeasible for neural networks on CrowdHuman.
- The analysis identifies predicting only one instance per proposal as the key issue and motivates multiple-instance prediction.
3. Our Approach: Multiple Instance Prediction
The method changes proposal-based detection from predicting one instance per proposal to predicting a correlated set of instances, using EMD Loss and Set NMS for crowded scenes.
- Multiple Instance Prediction: Each proposal predicts a correlated set of ground-truth instances instead of a single object, allowing nearby proposals to infer the same overlapped instances.This targets cases where highly overlapped proposals share similar features and distinguishing individual predictions is difficult.
- Instance Set Prediction: The detector extends existing frameworks with K detection functions that generate multiple predictions, with unused functions assigned to background.K is the maximum cardinality of the associated ground-truth set, and extra prediction branches implement the extension.
- EMD Loss: EMD Loss matches predictions to ground truths by searching permutations for the lowest combined classification and regression loss.Dummy background boxes pad smaller ground-truth sets, and K = 1 recovers the traditional single-instance loss.
- Set NMS: Set NMS skips suppression when two boxes originate from the same proposal, preserving multiple predictions that are unique within that proposal.Experiments indicate that multiple-instance prediction and Set NMS must be used together for significant crowded-detection improvement.
- Refinement Module: An optional refinement module combines proposal features with predicted boxes and performs a second prediction round to correct possible false positives.Generating more predictions may increase false positives, motivating this refinement step.
- Network Architecture: The approach attaches to the instance-prediction branch of proposal-based detectors and is evaluated with FPN and RoIAlign using K = 2.Only slight architectural modifications are required: an additional instance head and EMD Loss replace the original loss.
4. Experiment
The experiments evaluate crowded-scene detection across datasets with different instance densities, measuring precision, false positives, counting, and recall. The proposed components consistently improve CrowdHuman results, transfer to CityPersons and COCO, and preserve performance across crowdedness levels.
- Datasets: Evaluations use CrowdHuman, CityPersons, and COCO to test heavily, moderately, and slightly overlapped instance distributions.The experiments emphasize CrowdHuman, while COCO tests robustness to isolated objects and multi-class detection.
- Evaluation metrics: AP measures precision and recall, MR−2 emphasizes false positives, and Jaccard Index evaluates overlap between prediction and ground-truth sets.Higher AP and JI are better, whereas smaller MR−2 is better.
- CrowdHuman: 4.5% AP and 2.2% JI improvements over the FPN baseline occur without refinement, while refinement further improves MR−2 by approximately 0.8%.The ablation indicates that the refinement module primarily reduces false positives.
- CrowdHuman: Compared with alternative NMS and crowded-detection methods, the method achieves the best three-metric scores without refinement and adds little complexity.Other approaches can improve recall but may worsen false positives or require nontrivial tuning; AdaptiveNMS comparisons are not directly aligned because baselines differ.
- CrowdHuman: Our method improves crowded-instance recall from 54.4% to 63.3%, an 8.9% gain, while also slightly improving uncrowded recall.Recall thresholds are selected using the best JI score for each entry.
- CityPersons and COCO: The method improves AP and MR−2 by 0.9% and 1.0% over the CityPersons NMS baseline, and improves COCO AP by 1.0% over naïve NMS and 0.5% over Soft-NMS.COCO results also indicate no performance drop for multi-class and isolated-object detection.
5. Conclusion
The paper proposes a simple, effective proposal-based detector for crowded instance detection, using multiple instance prediction with EMD loss, Set NMS, and a refinement module. The approach is flexible across state-of-the-art proposal-based frameworks and generalizes to less crowded scenarios.
- The detector predicts multiple correlated instances per proposal for crowded detection.It incorporates EMD loss, Set NMS, and a refinement module.
- The method is designed to cooperate with most state-of-the-art proposal-based detection frameworks.
- The approach also generalizes well to less crowded scenarios.
A. CrowdHuman Testing Benchmark
The CrowdHuman testing subset contains 5,000 images without released annotations and serves as a benchmark for comparing detection methods. Using a larger model and higher-resolution inputs, the authors report outperforming all results in the challenge leaderboard.
- CrowdHuman’s testing subset contains 5,000 images, but its annotations have not been released.
- The Detection In the Wild Challenge uses CrowdHuman’s testing subset as a benchmark for comparing methods.
- The authors report that their method outperforms all challenge results after replacing ResNet-50 with SEResNeXt101 and resizing the short image edge to 1200 pixels.The baseline model for the reported leaderboard results is a reimplemented FPN.
B. Ablation on Number of Heads
The authors examine K, the method’s only hyper-parameter, using K = 2 as the default because it satisfies almost all CrowdHuman images and proposals. Larger K values are intended for more crowded scenes.
- K = 2 is used in the paper because it satisfies almost all images and proposals in CrowdHuman.
- Increasing K enables the network to detect instances in more crowded scenes.
- An experiment on CrowdHuman evaluates different K values while keeping the other settings unchanged.
C. More Results of Our Method
Additional evaluations include visualizations from a YouTube video and the CrowdHuman validation dataset, with a threshold used to remove redundant boxes. The supplementary experiments also cover K ablations and RetinaNet.
- The additional visualizations use a threshold of 0.7 to remove redundant result boxes.
- The supplementary results include a YouTube video and the CrowdHuman validation dataset, shown in Figure 5.
- When K = 1, the architecture is equivalent to the single-instance-prediction baseline.
- Experiments on RetinaNet indicate that one-stage detectors can also benefit from the proposed method.The RetinaNet baseline and proposed method use the same hyperparameters.
D. Our Method in One-Stage Detector
The method is evaluated in a one-stage RetinaNet detector using a ResNet-50 ImageNet-pretrained backbone and human-shaped anchor ratios. Figure 5 compares detection outputs across baseline methods and the proposed approach, including multiple predictions from one proposal.
- D. Our Method in One-Stage Detector: The one-stage evaluation uses RetinaNet with a standard ResNet-50 backbone pretrained on ImageNet.The experiment uses a batch size of 16 images across 8 GPUs, a learning rate of 0.005, and 50 training epochs.
- D. Our Method in One-Stage Detector: Anchor ratios are set to H : W = {1 : 1, 2 : 1, 3 : 1} to reflect human-instance shapes.Other hyperparameters are kept the same as the RetinaNet reference implementation.
- D. Our Method in One-Stage Detector: Figure 5 compares the baseline, GossipNet, RelationNet, and the proposed approach using colored boxes for detections, misses, redundancies, and multiple predictions.Blue boxes denote detections, white boxes missed instances, orange boxes redundant boxes, and green boxes multiple predictions from one proposal.