Source-linked AI summary
What makes for effective detection proposals?
Jan Hosang, Rodrigo Benenson, Piotr Dollár, Bernt Schiele
TL;DR
Object proposals reduce exhaustive detection search, but their trade-offs and evaluation criteria have been insufficiently characterized. The paper compares twelve proposal methods and four baselines across repeatability, recall, and detector performance, introducing average recall (AR) as a joint metric. It finds that localisation accuracy matters as much as recall and that AR correlates well with detector performance.
Problem
Proposal methods are widely used to avoid exhaustive sliding-window search, yet their trade-offs and comparative effectiveness have been insufficiently evaluated across metrics, datasets, and detectors.
Method
The paper systematically compares twelve proposal methods and four baselines using repeatability, recall across overlap thresholds, and effects on DPM, R-CNN, and Fast R-CNN.
Results
Improving proposal localisation accuracy is as important as improving recall, and average recall correlates well with detector performance across LM-LLDA, R-CNN, and Fast R-CNN.
Takeaways & Limitations
Average recall should be reported as a primary proposal metric, with its IoU range adapted for detectors that are more robust to localisation errors.
Takeaways & Limitations
The paper leaves unverified whether combining high proposal quality with greater repeatability would improve detection accuracy.
Abstract
from arXiv · showhide
Current top performing object detectors employ detection proposals to guide the search for objects, thereby avoiding exhaustive sliding window search across images. Despite the popularity and widespread use of detection proposals, it is unclear which trade-offs are made when using them during object detection. We provide an in-depth analysis of twelve proposal methods along with four baselines regarding proposal repeatability, ground truth annotation recall on PASCAL, ImageNet, and MS COCO, and their impact on DPM, R-CNN, and Fast R-CNN detection performance. Our analysis shows that for object detection improving proposal localisation accuracy is as important as improving recall. We introduce a novel metric, the average recall (AR), which rewards both high recall and good localisation and correlates surprisingly well with detection performance. Our findings show common strengths and weaknesses of existing methods, and provide insights and metrics for selecting and tuning proposal methods.
1 INTRODUCTION
Detection proposals replace exhaustive window search with candidate regions, but their trade-offs and evaluation criteria remain insufficiently understood. This paper systematically compares proposal methods and their effects on detection performance.
- Motivation: Single-scale sliding-window detection tests around 10^4–10^5 windows per image, rising to 10^6–10^7 with multiple scales and aspect ratios.This motivates proposal methods as a way to reduce the search space.
- Motivation: Detection proposals generate image regions likely to contain objects, reducing computation and enabling more sophisticated classifiers.They also alter the classifier’s data distribution, which may reduce spurious false positives.
- Research gap: Existing proposal studies typically compare limited subsets of metrics, datasets, and competing methods.The paper revisits proposals in a unified framework to better understand their benefits and limitations.
- Contributions: The paper systematically surveys proposal methods, defines reference baselines, and measures proposal repeatability.Repeatability is treated as a distinct property relevant to proposal evaluation.
- Contributions: Experiments evaluate object recall on PASCAL VOC 2007, ImageNet 2013, and MS COCO 2014, alongside effects on DPM, R-CNN, and Fast R-CNN.The broader datasets allow examination of possible biases toward PASCAL object categories.
- Evaluation scope: The study comprises over 500 experiments across multiple datasets, with evaluation scripts and method bounding boxes publicly released.The release is intended to facilitate reproduction of the evaluation.
2 DETECTION PROPOSAL METHODS
The paper organizes detection proposal methods into grouping, window-scoring, and alternative approaches, then compares diverse methods and baselines under controlled proposal counts. Its scope is class-agnostic, single-frame bounding-box detection.
- Overview: Detection proposals focus computation on candidate regions, enabling expensive classifiers while avoiding dense sliding-window evaluation.The paper contrasts this with the earlier use of interest points when dense feature computation was expensive.
- Method coverage: The survey covers 12 selected methods plus 4 baselines, chosen for source-code availability and diversity in quality, speed, and underlying approach.The evaluation uses only methods whose source code is available.
- Evaluation scope: The evaluation controls proposal counts per image because methods otherwise produce widely varying numbers of proposals and may return ranked or unranked outputs.The paper concentrates on class-agnostic proposals for single-frame bounding-box detection and excludes methods requiring temporal information.
- Grouping proposal methods: Grouping methods generate object-like segments by grouping superpixels, solving graph cuts, or using edge contours.The evaluation converts segmentation outputs to enclosing bounding boxes.
- Grouping proposal methods: SelectiveSearch greedily merges superpixels without learned parameters, whereas RandomizedPrim’s uses learned probabilities for randomized merging and substantially improves speed.Both methods are examples of superpixel-grouping approaches.
- Grouping proposal methods: Rantalankila combines superpixel merging with graph cuts, while Chang combines saliency and Objectness with a graphical model.These methods produce figure/background segmentations or additional proposals through different mechanisms.
- Grouping proposal methods: CPMC, Endres, Rigor, Geodesic, and MCG generate proposals through graph cuts, hierarchical segmentation, or edge-based figure/ground constructions.Their ranking and proposal-generation mechanisms use distinct combinations of seeds, cues, contours, and segment hierarchies.
- Window-scoring methods: Window-scoring methods rank candidate boxes and are usually faster, but sparse sampling typically produces lower localisation accuracy unless windows are refined.Objectness and Bing exemplify this family, while CrackingBing is image-independent.
3 PROPOSAL REPEATABILITY
Proposal repeatability measures whether methods place proposals on similar image content after small perturbations, because inconsistent proposal distributions can hinder detector scoring. Experiments show widespread sensitivity to subtle changes, with Bing notably more robust and superpixel-based methods especially unstable.
- Evaluation protocol: Repeatability measures whether proposals remain on similar image content when the same image undergoes slight modifications.The protocol applies synthetic transformations to PASCAL images and compares proposals from reference and perturbed versions.
- Evaluation protocol: 1000 proposals per image are projected back to the reference image, matched using IoU, and evaluated by recall across IoU thresholds.The repeatability score is the area under the recall-versus-IoU curve; proposal windows are also grouped by area to control for size effects.
- Results: Small image changes noticeably alter proposal sets for most methods, while Bing is the principal exception.The same broad trend appears for proposals sufficiently overlapping annotations, showing that instability also affects boxes that partially cover objects.
- Results: Superpixel-based methods are particularly sensitive to perturbations, whereas non-superpixel methods and content-agnostic baselines are more robust to small changes.The authors identify sensitivity in the underlying segmentation algorithm as a likely source of poor repeatability.
- Results: Scale changes, JPEG compression, illumination changes, blur, rotation, and sparse pixel noise all reduce repeatability, often even when perturbations are slight.Most methods lose repeatability after a single pixel changes, and substantial degradation occurs when merely ten pixels are modified.
- Implications: Because applications differ in sensitivity to repeatability, proposal methods should be selected with care for object detection.The section treats repeatability as one consideration alongside object recall.
4 PROPOSAL RECALL
The paper evaluates proposal recall across multiple IoU thresholds, proposal counts, datasets, and methods, showing that rankings depend on both localisation quality and dataset statistics. MCG performs best overall, while average recall (AR) provides a unified measure that reflects performance across overlap thresholds.
- Evaluation protocol: Different evaluation metrics produce different rankings of proposal methods, motivating a unified comparison protocol.The paper compares results across PASCAL VOC 2007, ImageNet 2013, and MS COCO 2014.
- Metrics: Proposal evaluation measures ground-truth coverage at fixed proposal counts across varying IoU thresholds, or varying proposal counts at a fixed IoU threshold.The paper also introduces AR between IoU 0.5 and 1 to summarise performance across thresholds.
- PASCAL results: MCG, EdgeBoxes, SelectiveSearch, Rigor, and Geodesic are the strongest PASCAL methods across proposal counts, while MCG performs well across the entire AR range.For fewer than 10^3 proposals, MCG, Endres, and CPMC provide strong results.
- PASCAL results: Proposal methods divide into well-localised methods with gradual recall loss at higher IoU and coarse methods whose recall drops rapidly.Bing has high repeatability but performs well mainly at IoU = 0.5 and declines sharply at higher overlap.
- Cross-dataset results: ImageNet produces recall behaviour almost identical to PASCAL, whereas MS COCO shows different absolute values and method rankings because of its distinct object-size distribution.On MS COCO, MCG and Endres improve relative to other methods, especially with more proposals.
- Cross-dataset results: MCG is the top-performing method across all evaluated datasets for both recall and AR at all settings, with no measurable overfitting to PASCAL categories.The results suggest proposal methods transfer adequately across object classes.
5 USING THE DETECTION PROPOSALS
The paper tests proposal methods with DPM, R-CNN, and Fast R-CNN to determine how proposal quality affects detection. Detection improves with proposal localisation, AR correlates strongly with detector performance, and proposal rankings are broadly consistent across detectors, subject to bounding-box regression and proposal-count limits.
- Detector responses: Detection scores for LM-LLDA and R-CNN increase almost linearly with proposal IoU, showing that better localisation matters alongside recall.Fast R-CNN also benefits from localisation, but its performance saturates earlier.
- Detection results: Using 1,000 proposals reduces LM-LLDA detection quality by about 1-2 mAP relative to the original sliding-window setup.The five top-performing methods achieve mAP between 32.0 and 33.0.
- Detection results: Methods with high average recall also achieve high mAP, and methods with lower AR achieve lower mAP.This ranking similarity holds when comparing proposal results with LM-LLDA detection performance.
- Cross-detector comparison: Across LM-LLDA, R-CNN, and Fast R-CNN, proposal methods show similar ordering rather than being specially adapted to particular detectors.The experiments identify some proposal methods as consistently better than others.
- Predicting detection performance: Recall at IoU values around 0.6 to 0.8 correlates strongly with detector performance, whereas recall at IoU = 0.5 is only weakly correlated.The reported correlation exceeds 0.9 for all detectors except Fast R-CNN with bounding-box prediction.
- Predicting detection performance: AR over IoU 0.5 to 1.0 is strongly and surprisingly linearly correlated with mAP, although the relationship does not perfectly predict detection performance.For Fast R-CNN with bounding-box regression, restricting AR to IoU 0.5 to 0.7 raises correlation to 0.949 from 0.877.
- Predicting detection performance: AR is a useful predictor for selecting and tuning proposal methods when the number of proposals is fixed.AR necessarily increases as proposal count increases, while detector performance can saturate or degrade.
6 DISCUSSION
The paper’s analysis identifies localisation accuracy and average recall as central to proposal effectiveness, while showing that proposal methods generalise across datasets but still leave detector errors to address.
- Future Directions: Proposal methods have limited repeatability, whereas future progress may involve tighter proposal–detector integration and greater use of segmentation masks.The paper also anticipates more top-down reasoning because purely bottom-up processes have difficulty generating perfect proposals.
- Localisation Accuracy: Improving proposal localisation accuracy is as important as improving recall for object detection.Recall at IoU 0.5 is not predictive of detection accuracy, and high-recall proposals with low overlap are ineffective.
- Average Recall: Average recall (AR) summarises recall across overlap thresholds and correlates well with LM-LLDA, R-CNN, and Fast R-CNN performance.AR supports both comparing proposal methods and tuning a specific method’s parameters.
- Top Methods: SelectiveSearch, Rigor, MCG, and EdgeBoxes consistently achieved top detection performance across diverse detectors.EdgeBoxes offers a speed–quality compromise, while the methods achieve similar performance despite using different proposal-generation mechanisms.
- Generalisation: Recall showed no significant drop from 20 PASCAL categories to 200 ImageNet categories, and method ordering mostly persisted on MS COCO.MS COCO contains more and smaller objects, yet the relative ordering remained mostly unchanged.
- Oracle Experiments: With EdgeBoxesAR proposals, Fast R-CNN achieved 67.8 mAP on PASCAL 2007 test, while oracle corrections raised performance to 71.2 and then 78.1 mAP.The remaining 21.9 mAP gap to perfect detection came from high-scoring background detections and object misclassifications.
- Discussion: Proposal methods may eventually become unnecessary if dense CNN evaluation and adequate training outperform the proposal–detector cascade.Increasing proposal counts or adding random proposals can harm detection performance.
- Broader Uses: Object proposals may support unknown-category detection and weakly supervised learning beyond the object-detection setting.The paper also notes emerging cross-pollination between proposal methods’ boundary and superpixel features and detectors’ CNN features.
APPENDIX A ANALYSIS OF METRICS
The appendix explains how average recall can be computed from overlap distributions and contrasts it with related metrics and matching procedures used in proposal evaluation.
- Average Recall: Average recall between 0.5 and 1 can be computed by averaging each annotation’s closest-proposal overlap.This integrates over the overlap axis and corresponds to ABO or BSS truncated at 0.5 IoU.
- Matching: The experiments assign proposals to annotations through bipartite matching using a greedy algorithm.This differs from ABO and BSS, where the closest proposal may match more than one annotation.
- Related Metrics: VUS computes recall across overlap and proposal count, but it is only an indirect predictor when detectors use a fixed proposal count.Its surface-volume formulation does not directly match the fixed-number setting used in practice.