Source-linked AI summary
DeepBox: Learning Objectness with Convolutional Networks
Weicheng Kuo, Bharath Hariharan, Jitendra Malik
TL;DR
Existing proposal methods mainly rank regions using bottom-up cues, motivating a semantic, data-driven notion of objectness. DeepBox reranks bottom-up proposals with a lightweight CNN and improves proposal and detection performance, including on unseen categories.
Problem
Existing object proposal methods rely primarily on bottom-up grouping and saliency, while objectness may also involve higher-level structures shared across categories.
Method
DeepBox trains a lightweight CNN on annotated objects to rerank proposals generated by a bottom-up method, using shared convolutional features for efficient processing.
Results
4.5 points higher detection performance is achieved with 500 DeepBox proposals than with 500 Edge box proposals, while ranking improves by 26% relative AUC on VOC 2007.
Takeaways & Limitations
The learned semantic notion of objectness generalizes to categories unseen during training and improves proposal ranking beyond bottom-up saliency.
Takeaways & Limitations
On datasets with few small objects, increasing the number of reranked proposals beyond 2000 yields only marginal recall gains, unlike more challenging small-object datasets.
Abstract
from arXiv · showhide
Existing object proposal approaches use primarily bottom-up cues to rank proposals, while we believe that objectness is in fact a high level construct. We argue for a data-driven, semantic approach for ranking object proposals. Our framework, which we call DeepBox, uses convolutional neural networks (CNNs) to rerank proposals from a bottom-up method. We use a novel four-layer CNN architecture that is as good as much larger networks on the task of evaluating objectness while being much faster. We show that DeepBox significantly improves over the bottom-up ranking, achieving the same recall with 500 proposals as achieved by bottom-up methods with 2000. This improvement generalizes to categories the CNN has never seen before and leads to a 4.5-point gain in detection mAP. Our implementation achieves this performance while running at 260 ms per image.
1. Introduction
DeepBox treats objectness as a semantic, data-driven construct rather than relying only on bottom-up cues. A lightweight CNN reranks bottom-up proposals and improves proposal quality and downstream detection.
- Bottom-up proposal methods rely mainly on grouping and saliency cues for speed and category generality.
- DeepBox learns discriminative low-, mid-, and high-level object cues from annotated images using a CNN.
- 26% relative AUC improvement over Edge boxes on VOC 2007 demonstrates stronger proposal ranking.
- 4.5 points higher detection performance results when Fast R-CNN uses 500 DeepBox proposals instead of 500 Edge box proposals.
- The paper combines proposal-ranking experiments with a lightweight implementation and concludes with discussion of applications.
2. Related work
Prior proposal systems generate candidate regions through category-independent bottom-up processes and often rank them with low-level features. DeepBox instead uses a lightweight CNN to rerank such proposals.
- Category-independent proposal methods sample or segment image regions to generate putative objects.
- Figure 1 depicts bottom-up generation followed by CNN reranking, with higher-ranked boxes shown in green.
- Many proposal methods rank regions using low-level features such as saliency, with more involved ranking needed for smaller proposal sets.
- DeepBox differs from direct proposal-generation networks by reranking proposals from any bottom-up method with a lightweight architecture.
- The paper points readers to prior work for broader evaluations of bottom-up proposal methods.
3. Method
DeepBox first generates bottom-up proposals, then reranks them with a compact CNN trained through sliding-window and hard-negative stages. Shared convolutional features and pooling reduce reranking cost, while the method remains sensitive to proposal count and scale choices.
- The pipeline generates an initial proposal pool and reranks each proposal using DeepBox scores.
- For datasets with few small objects, reranking beyond 2000 proposals offers marginal recall gains, whereas challenging small-object datasets continue benefiting from more proposals.
- DeepBox uses a lightweight network because objectness prediction is simpler and should cost less than full object detection.
- The final architecture is conv(11, 96, 4)−pool(3, 2)−conv(5, 256, 1)−fc(1024)−fc(2), with ReLU layers and softmax outputs.
- Shared convolutional feature maps and spatial pyramid pooling avoid recomputing convolutional features for highly overlapping proposals.
- A multiscale variant selects among three feature-map scales because one scale may not suit every object.
- Fast DeepBox uses RoI Pooling and distinguishes the shared-feature implementation from the slower cropping-and-warping version.
- Training progresses from raster-scanned negatives to hard negatives supplied by bottom-up proposals, teaching the network to distinguish complete objects from proposer errors.
4. Experiments
Experiments compare DeepBox with Edge boxes on PASCAL VOC and COCO using proposal-ranking metrics, including recall and AUC. The evaluations vary proposal counts, datasets, overlap thresholds, and object sizes.
- Experiments use Edge boxes as the initial proposal pool, reranking the top 2048 proposals on PASCAL and all proposals on COCO.
- Training uses PASCAL trainval and COCO train sets, with evaluation on their respective test and validation sets.
- DeepBox outperforms Edge boxes across proposal counts and IoU regimes, with AUCs of 0.74(0.60) versus 0.60(0.47) on PASCAL at IoU=0.5(0.7).
- On COCO, DeepBox reaches AUCs of 0.40(0.28) versus 0.28(0.20) for Edge boxes at IoU=0.5(0.7).
- DeepBox’s gain is especially large for small objects, exceeding Edge boxes by more than 160% on COCO.
- The comparison with other proposal methods reports proposal counts needed for 25%, 50%, and 75% recall plus maximum recall using 5000 boxes.
4.3. Visualization of DeepBox proposals
DeepBox produces more successful and object-focused proposals than Edge boxes, particularly in cluttered scenes with small or partially obscured objects.
- Visualization of hits and misses: In cluttered scenes, DeepBox significantly outperforms Edge boxes by capturing small or difficult objects that Edge boxes misses.Examples include tiny boats, parked cars, donuts, and people in shade.
- Visualization of hits and misses: Figure 5 marks highly overlapping proposal hits in green and missed ground-truth boxes in red at an IoU threshold of 0.7.Blue boxes show the corresponding proposals; evaluations use 1000 proposals for COCO and 500 for PASCAL.
- Proposal distributions: DeepBox proposals are more densely concentrated on objects, whereas Edge boxes often spread proposals across image contours.The comparison uses the top 100 proposals from each ranking.
4.4. Generalization to unseen categories
DeepBox generalizes its learned objectness beyond the categories used for training. On unseen COCO categories, it improves proposal-ranking AUC over Edge boxes across IoU thresholds and proposal settings.
- Experimental setup: DeepBox was trained on 36 overlapping Imagenet–COCO categories and evaluated on the remaining 44 COCO categories as unseen categories.Training labels and positive sampling excluded the held-out categories, while evaluation considered only those categories.
- Unseen-category evaluation: 15.7% (15.3%) AUC improvement over Edge boxes was achieved at IoU=0.5 (0.7) when reranking the top 2048 proposals.The initial proposal pool was generated by Edge boxes.
- Unseen-category evaluation: 18.5% (16.2%) AUC improvement over Edge boxes was achieved at IoU=0.5 (0.7) when reranking all Edge boxes proposals.DeepBox outperformed Edge boxes in all regimes on unseen categories for both IoUs.
- Category-level generalization: The gains exceed 20% in more than half of the 44 testing categories, supporting a class-agnostic notion of objectness.Performance is especially strong for animals and weaker for sports categories with salient contours.
4.5. DeepBox using Large Networks
Larger CNNs capture objectness slightly better than DeepBox’s lightweight network, but the four-layer architecture remains substantially more efficient while retaining strong performance.
- PASCAL comparison: On PASCAL, VGG reaches AUCs of 0.78 (0.65) and AlexNet reaches 0.76 (0.62) at IoU=0.5 (0.7), versus 0.74 (0.60) for DeepBox.The comparison shows a modest performance advantage for the larger networks.
- COCO comparison: On COCO with the top 2048 proposals reranked, VGG reaches 0.43 and 0.31, AlexNet 0.42 and 0.30, versus DeepBox’s 0.38 and 0.27 at IoU=0.5 and 0.7.When reranking all proposals, the small network reaches 0.40 and 0.28.
- Efficiency and adequacy: The experiments suggest that the lightweight architecture captures objectness semantics while being more efficient to evaluate than VGG and AlexNet.This preserves the practical advantage of a smaller proposal-scoring network.
4.6. DeepBox with Other Proposers
DeepBox generalizes across bottom-up proposal methods, but effective reranking requires retraining for each proposer. Gains are strongest for Selective Search, while MCG sees little benefit.
- Retraining DeepBox separately for each proposal method removes the degradation observed when an Edge boxes model reranks Selective Search or MCG proposals.An Edge boxes-trained model provides little gain and can hurt on the other proposal methods.
- DeepBox produces large gains on Selective Search at both evaluated IoU thresholds.
- DeepBox does not hurt MCG proposals but provides little improvement.
- The category-wise evaluation indicates that DeepBox learns objectness that generalizes beyond its training categories.
4.7. Fast DeepBox
Fast DeepBox substantially improves proposal-ranking speed while retaining competitive proposal quality on COCO. Its speed–accuracy trade-offs depend on scale and the number of reranked proposals.
- 41%(44%) gains over Edge boxes are achieved by multiscale Fast DeepBox at IoU=0.5(0.7), with AUCs of 0.40(0.29) versus 0.28(0.20).
- 0.26s is required to rerank all proposals with multiscale Fast DeepBox, versus 2.5s for DeepBox.Reranking the top-2000 proposals takes 0.12s.
- 0.39 average recall with 1000 proposals exceeds GOP, Selective Search, and Edge boxes while nearly matching MCG.Fast DeepBox is reported to be almost 70 times faster than MCG in this comparison.
- 0.005 and 0.01 AUC decreases accompany single-scale Fast DeepBox when reranking the top-2000 and all proposals, respectively.The corresponding runtimes are 0.060s and 0.11s.
4.8. Impact on Object Detection
DeepBox proposals improve downstream object detection while reducing the number of proposals used. With 500 proposals, Fast R-CNN performs better than with substantially larger Edge Box sets.
- 37.8% mAP is achieved by Fast R-CNN with 500 DeepBox proposals on COCO Test at IoU=0.5.
- 2 points of performance improvement are obtained with 500 DeepBox proposals compared with 2000 Selective Search proposals.
5. Discussion and Conclusion
DeepBox learns an efficient semantic notion of objectness that generalizes to unseen categories. The authors argue that generic objectness can reduce detector burden and support object discovery for agents operating beyond fixed labeled datasets.
- DeepBox is an efficient CNN architecture that learns a semantic notion of objectness generalizing to unseen categories.
- A generic objectness system becomes more useful as the number of object categories increases because detector complexity also increases.Objectness can reduce the number of locations the detector examines and ease localization.
- Objectness can help an AI agent select candidate object locations and track them while collecting data for detectors built on the fly.This is motivated by agents encountering categories without labeled COCO-style training data.