Source-linked AI summary
How good are detection proposals, really?
Jan Hosang, Rodrigo Benenson, Bernt Schiele
TL;DR
Detection proposals promise faster object detection than exhaustive sliding-window search, but their trade-offs remain unclear. The paper evaluates ten proposal methods and four baselines across recall, repeatability, and detector impact, finding distinct practical compromises and common repeatability weaknesses. These results support choosing methods according to speed and proposal-count requirements, while leaving the direct effect of repeatability on detector performance for future work.
Problem
The trade-offs of using detection proposals during object detection remain unclear despite their popularity and potential to reduce exhaustive search.
Method
The paper compares ten proposal methods and four baselines using ground-truth overlap on Pascal VOC 2007 and ImageNet 2013, repeatability, and detector-performance evaluations.
Results
SelectiveSearch and EdgeBoxes perform consistently well across recall, repeatability, and evaluation speed, while MCG provides high recall below 10^3 proposals and top pure detection quality.
Takeaways & Limitations
The evaluation helps researchers choose proposal methods according to speed, quality, and proposal-count requirements, while most methods generalize beyond Pascal VOC categories.
Takeaways & Limitations
The paper leaves direct evaluation of how repeatability influences detector performance for future work because it requires additional parameters and a more complicated procedure.
Abstract
from arXiv · showhide
Current top performing Pascal VOC object detectors employ detection proposals to guide the search for objects thereby avoiding exhaustive sliding window search across images. Despite the popularity of detection proposals, it is unclear which trade-offs are made when using them during object detection. We provide an in depth analysis of ten object proposal methods along with four baselines regarding ground truth annotation recall (on Pascal VOC 2007 and ImageNet 2013), repeatability, and impact on DPM detector performance. Our findings show common weaknesses of existing methods, and provide insights to choose the most adequate method for different settings.
1 Introduction
Detection proposals replace exhaustive sliding-window search with a smaller set of likely object-containing windows, aiming to preserve recall while reducing computation. The paper compares existing methods in a common framework to clarify their benefits, limitations, and appropriate use.
- ∼10^6 classifier evaluations per image make exhaustive sliding-window detection computationally demanding.
- Detection proposals output likely object-containing windows, potentially achieving high recall with ∼10^4 or fewer windows and enabling faster, more sophisticated detectors.
- Using proposals changes the classifier’s data distribution and may improve detection quality by reducing false positives.
- The paper compares publicly available proposal methods in a unified framework to understand their benefits and limitations.
- Its evaluation covers proposal-method reviews, repeatability, ground-truth overlap on Pascal VOC 2007 and ImageNet 2013, detector impact, and released evaluation resources.
2 Detection proposal methods
Detection proposal methods use varied low-level cues and segmentation strategies to generate and rank candidate windows, while evaluations must account for proposal count and dataset bias. The paper excludes some methods because of limited applicability or unavailable code.
- Detection proposals can be viewed as fast discriminative filters that use low-level image features to discard unpromising windows.
- Objectness estimates whether a window contains an object using cues including saliency, colour contrast, edge density, location, size, and superpixel overlap.
- SelectiveSearch greedily merges low-level superpixels using engineered features and score functions without learning parameters.
- The data-driven objectness method of Kang et al. is excluded because its applicability beyond indoor scenarios appears limited.
- Most evaluated methods rely on low-level segmentation, whereas CPMC, Bing, and EdgeBoxes avoid computing low-level segments.
- Repeated reliance on Pascal VOC overlap may encourage overfitting, motivating evaluations beyond that dataset.
- Methods produce widely varying candidate counts, so experiments strive to equalize average proposals per image because more windows increase recall.
3 Proposal repeatability
Proposal repeatability is evaluated by matching projected proposals across synthetically perturbed Pascal images, with window-size normalization. Most methods lose repeatability under subtle changes, especially those relying on superpixels, while Bing and EdgeBoxes are comparatively robust.
- Evaluation protocol: Repeatability measures whether proposals remain on similar image content across slightly modified versions of the same image.The protocol projects proposals between reference and perturbed images and matches them using intersection over union (IoU).
- Evaluation protocol: The evaluation applies scale, blur, rotation, illumination, and JPEG-compression perturbations to Pascal VOC 2007 test images.Each reference image is paired with several synthetically transformed versions so proposal correspondences can be measured.
- Evaluation protocol: Proposal window sizes are binned into ten groups because larger windows match more easily and proposal methods produce different size distributions.Repeatability is summarized as the unweighted average area under the recall-versus-IoU curve across size bins.
- Repeatability results: Most methods show a large repeatability drop under subtle image changes, plausibly because unstable superpixels alter the proposal windows.The Superpixels baseline also drops strongly, while methods not based on superpixels are more robust to small changes.
- Repeatability results: All methods except Bing show a drastic drop with small scale changes, while JPEG artefacts, illumination, and rotation also affect repeatability.Bing is comparatively robust to JPEG and illumination changes, whereas rotated bounding-box matching affects all methods similarly.
- Repeatability results: Bing and EdgeBoxes appear more repeatable than the other evaluated methods, possibly because both use machine-based components.The authors identify superpixel sensitivity as a major cause of repeatability degradation in several proposal methods.
4 Proposal recall
The evaluation measures proposal quality by ground-truth coverage across IoU thresholds and proposal budgets, using Pascal VOC 2007 and ImageNet 2013. Results distinguish consistently strong methods, localization weaknesses, and the greater relative value of proposals when few windows are allowed.
- Evaluation protocol: Ground-truth recall is the fraction of annotated objects covered above a specified IoU threshold, making object coverage central because missed objects cannot be recovered.The unified evaluation also addresses inconsistent prior datasets and metrics, and includes the full Pascal VOC 2007 test set.
- Pascal VOC recall: MCG, EdgeBoxes, and SelectiveSearch perform best across different numbers of proposals on Pascal VOC 2007.With fewer than 10^3 proposals, MCG, Endres2010, and CPMC provide strong results.
- Pascal VOC recall: EdgeBoxes tuned for IoU 0.7 produces a clear recall bump, while at IoU = 0.5 it reaches about 93% recall.At 10^3 proposals and IoU = 0.9, tuned EdgeBoxes falls below MCG.
- Localization behavior: Proposal methods divide into well-localized approaches with gradual recall loss and approximate-localization methods whose recall drops dramatically as IoU increases.Bing has high repeatability but high recall only at IoU = 0.5, with a sharp decline at higher overlap.
- Proposal budgets: As proposal counts increase, Gaussian and SlidingWindow baselines become competitive at IoU > 0.7, and Uniform eventually becomes competitive with enough windows.Proposal methods offer their greatest relative gain when the number of windows is low.
- Cross-dataset evaluation: ImageNet 2013 recall curves are almost identical to Pascal VOC 2007 despite covering 200 categories and four times as many images.The authors interpret this as adequate transfer across object classes and no visible over-fitting to the 20 Pascal categories.
5 Using the detection proposals
The paper tests detection proposals by filtering detections from a pretrained LM-LLDA DPM detector, isolating their effect on detection quality rather than speed. Using 1000 proposals reduces overall quality versus sliding-window detection, but localization quality and class-specific effects matter.
- Experimental setup: The experiment filters pretrained LM-LLDA DPM detections with different proposal sets at test time, without measuring speed-up.The procedure evaluates detections inside proposal windows and around them, then applies non-maxima suppression and bounding-box regression.
- Detection quality: 1000 detection proposals decrease detection quality compared with sliding-window detection.The comparison reports left/right mAP before/after bounding-box regression.
- Detection quality: Methods with area under the recall curve above 69% also have high mAP, while higher recall in the high-precision region IoU ≥ 0.7 distinguishes some methods with similar overall area.The authors associate better proposal localization with increased detection quality.
- Per-class effects: Proposals hurt performance for bicycle, bottle, car, chair, motorbike, person, and potted plant, but improve it for aeroplane, cat, dining table, dog, and sofa.The paper reports both reduced recall and reduced precision in the classes where proposals clearly hurt.
6 Conclusion
The evaluation identifies proposal methods with different speed, recall, repeatability, and detection-quality trade-offs, while revealing low repeatability as a common weakness.
- SelectiveSearch and EdgeBoxes provide consistently good ground-truth recall, reasonable repeatability, and tolerable evaluation speed.
- For fast proposals, properly tuned EdgeBoxes offers the best compromise between speed and quality.
- With fewer than 10^3 proposals, MCG is preferred for high recall when speed is not a concern.
- MCG achieves top results for pure detection quality.
- Most methods suffer from low repeatability because of unstable superpixels, even under slight image perturbations.The authors identify more robust superpixel or boundary-estimation methods as a possible improvement.
- The study leaves missing recall and the relationship between detection proposals and object detection for future work.
A Method overview
The evaluation compares proposal methods and baselines under differing output-size, ranking, and implementation constraints, while applying image perturbations to study robustness.
- Table 3 covers eleven listed proposal methods, but only eight are evaluated because code is unavailable for three methods; four baselines are also included.
- Methods either return roughly the requested number of proposals or significantly fewer, and may rank them by likelihood or return them arbitrarily ordered.
- Subset selection generally uses returned ranking when available, but Rahtu2011 and MCG perform better when using the first n proposals rather than the best n.
- RandomizedPrim’s output is sampled using learned probabilities, so fixed proposal counts and first-returned candidates are used for different experiments.
- EdgeBoxes, Bing, Objectness, CPMC, Endres2010, and Selective Search use method-specific scores or ordering to select subsets.
- The perturbation protocol includes scale changes from 0.5 to 2, Gaussian blurring, rotations from -20 to 20 degrees, and illumination changes with over- and under-saturation.
C.5 JPEG artefacts
JPEG artefacts are generated across a wide range of compression qualities, including a lossless setting, and visible examples emphasize the low-quality end.
- JPEG artefacts are created by saving images with Matlab’s imwrite across very low through 100% quality settings, plus a lossless comparison.
- Pixel-wise differences from the original occur across the entire image even at 100% JPEG quality, although only low-quality examples are readily visible to humans.
D Repeatability results
Repeatability is evaluated under image perturbations and across datasets, with multiple recall views used because different curves can favor different methods.
- The paper includes repeatability results for all perturbations, with the blurring plot supplied separately as Figure 11e because of space constraints.
- Recall is analyzed against localization quality and proposal count, using fixed-IoU curves or area under the recall-versus-IoU curve.
- Different recall curves favor different methods, so selecting only one curve or part of a curve can be misleading.
- ImageNet 2013 validation experiments test whether methods trained and tuned on Pascal generalize to 200 classes beyond Pascal’s 20.
F Detection
The paper evaluates proposal quality and repeatability across Pascal VOC 2007 and ImageNet 2013, while reporting class-specific recall–precision behavior. Figures encode recall, IoU thresholds, proposal counts, perturbations, and test-time proposal-method comparisons.
- Proposal evaluation: Recall–precision behavior differs across object classes, so the paper supplements mean average precision with class-level curves.The reported mAP averages results over all classes; figures 14–17 provide the individual recall–precision curves.
- Proposal evaluation: Proposal-window size distributions and recall fluctuations under blur are included as additional diagnostics of proposal behavior.The size analysis compares methods on the Pascal VOC 2007 test set, while blur groups correspond to points in Figure 11a.
- Repeatability: Repeatability is analyzed under image-scale changes ranging from half to double the original size.The scale-change axis is logarithmic.
- Repeatability: Repeatability is also evaluated under rotations from −20 to 20 degrees.The rotation analysis uses the stated degree range on the x-axis.
- Proposal evaluation: Proposal quality is shown as recall versus IoU threshold, with area under each curve and average windows or proposals per image reported.Figure 12 covers Pascal VOC 2007; Figure 13 covers ImageNet 2013.