Source-linked AI summary
DeNet: Scalable Real-time Object Detection with Directed Sparse Sampling
Lachlan Tychsen-Smith, Lars Petersson
TL;DR
Object detection entails estimating probabilities over an intractably large set of image boxes, while detectors also face demands for high evaluation rates and reduced manual engineering. DeNet uses Directed Sparse Sampling in an end-to-end CNN with corner-based RoI estimation and deconvolution. The authors report improved detection performance with real-time and near-real-time evaluation rates, and identify a timing cost in the wide variant.
Problem
Estimating class probabilities over all valid image boxes is intractable; for a 224 × 224 ImageNet image and 1000 classes, the distribution requires about 629 × 10^9 values.
Source: §1.2 · p. 2Method
DeNet applies Directed Sparse Sampling in an end-to-end CNN, estimating likely bounding-box corners for RoI selection and using deconvolution layers.
Source: §1 · p. 1, §1.2 · p. 2, §2.1 · p. 3Results
DeNet reports improved state-of-the-art detection performance with real-time and near-real-time evaluation rates.
Source: §5 · p. 8Takeaways & Limitations
The method offers 4.2 × 10^6 feasible sampling boxes, versus fewer than 2.5 × 10^4 for SSD512 and RPN; the authors suggest this supports finer localization and reduces reliance on reference boxes.
Source: §5 · p. 8The wide variant incurs considerable timing cost from increased classification and CPU-bound RoI identification.
Source: §2.3.2 · p. 5
Abstract
from arXiv · showhide
We define the object detection from imagery problem as estimating a very large but extremely sparse bounding box dependent probability distribution. Subsequently we identify a sparse distribution estimation scheme, Directed Sparse Sampling, and employ it in a single end-to-end CNN based detection model. This methodology extends and formalizes previous state-of-the-art detection models with an additional emphasis on high evaluation rates and reduced manual engineering. We introduce two novelties, a corner based region-of-interest estimator and a deconvolution based CNN model. The resulting model is scene adaptive, does not require manually defined reference bounding boxes and produces highly competitive results on MSCOCO, Pascal VOC 2007 and Pascal VOC 2012 with real-time evaluation rates. Further analysis suggests our model performs particularly well when finegrained object localization is desirable. We argue that this advantage stems from the significantly larger set of available regions-of-interest relative to other methods. Source-code is available from: https://github.com/lachlants/denet
1. Introduction
DeNet targets strong detection at predefined evaluation rates, combining sparse region-based detection with the speed and scene adaptability of dense methods. It frames detection as estimating a huge but sparse box-dependent distribution and proposes Directed Sparse Sampling with a corner-based RoI estimator and deconvolution layers.
1. Introduction
- The paper prioritizes detection performance at predefined 60 Hz and 30 Hz evaluation rates rather than accuracy under computationally unconstrained conditions.Source: §1 · p. 1
1.1. Related work
- DeNet combines sparse region-based methods’ training and classification strengths with dense methods’ fast training and evaluation, while reducing manual engineering.The introduction contrasts expensive region-based CNNs with dense detectors whose predefined regions require scene-dependent engineering.Source: §1.1 · p. 2
1.2. Probabilistic Object Detection
- Estimating the class probability for every valid image box is intractable: the ImageNet example requires about 629 × 10^9 values, or 2.5TB in 32-bit floats.Source: §1.2 · p. 2
1. Introduction
- Directed Sparse Sampling estimates a highly sparse box-dependent distribution with an end-to-end CNN, using a corner-based RoI estimator without manually defined reference boxes and deconvolution layers to improve evaluation rates.The introduction also identifies six implementations with competitive performance across benchmarks.Source: §1 · p. 1, §1.2 · p. 2
2. Directed Sparse Sampling (DSS)
Directed Sparse Sampling (DSS) uses a jointly optimized two-stage CNN: it estimates likely object locations, then sparsely classifies them. DeNet implements this with corner-based region proposals, a detection network, and optional skip and wide variants.
2. Directed Sparse Sampling (DSS)
- DSS jointly estimates likely non-null bounding boxes and classifies the selected boxes in a two-stage CNN.The stages are jointly optimized; the first estimates likely locations of interesting values, and the second sparsely classifies them.Source: §2 · p. 3
2.1. Corner-based RoI Detector
- The corner-based RoI detector estimates probabilities for four corner types, then ranks candidate boxes by non-null probability to select N × N sampling boxes.The number N balances the maximum number of detections against computational and memory requirements.Source: §2.1 · p. 3
- For each selected box, DeNet builds a feature vector from nearby sampled features and box dimensions, then predicts class probabilities and refined coordinates.The box center is omitted so the classifier is agnostic to image offsets.Source: §2.1 · p. 3
2.2. Training
- During training, DeNet augments sampled boxes with ground-truth and random boxes, then jointly optimizes corner, classification, and box-regression costs.Sampling boxes are held constant during gradient estimation, allowing the corner detector to be optimized alongside classification and estimation.Source: §2.2 · p. 3
2.3. Detection Model
- The detection model upsamples spatial features with deconvolution layers, then sparsely samples N × N boxes from 4.2M valid boxes for classification and box refinement.Each sampled feature uses a 7 × 7 grid plus box width and height; the model uses a shallow fully connected network for its final predictions.Source: §2.3 · p. 4
2.3.2 Wide Variant
- The skip variant adds projected features from the base model, while the wide variant raises map resolution to 128 × 128 and uses N = 48.The wide variant's 2304 RoIs bring considerable timing costs from classification and CPU-bound RoI identification; the authors suggest further engineering could reduce them.Source: §2.3.1 · p. 4, §2.3.2 · p. 5
3. Implementation Details
DeNet is implemented in a Theano-based CNN library. Training combines established SGD and image augmentation procedures with a corner-pruning search to form sampling boxes.
3. Implementation Details
- DeNet models are implemented in a Theano-based CNN library.The source code is available from the project repository.Source: §3 · p. 5
3.1. Training methodology
- Training uses Nesterov-style SGD and image augmentation, with randomly generated boxes added after overtraining was observed on Pascal VOC.The learning rate starts at 0.1 and is divided by 10 at epochs 30 and 60; training lasts 90 epochs, and crops are sampled with scale and aspect-ratio variation.Source: §3.1 · p. 5
3.2. Identifying Sampling Bounding Boxes (RoIs)
- The RoI search prunes low-probability corners, pairs the most likely remaining corners to form boxes, and keeps the highest-ranked boxes.Culling most corners in the initial thresholding step speeds the search over the brute-force approach of testing every possible box.Source: §3.2 · p. 5
4. Results and Analysis
DeNet's model overview compares input sizes, test batch sizes, activation layers, and approximate parameter counts. The authors report improved evaluation rates despite increased layers and parameters, while noting that comparisons across models are not always apples-to-apples.
4. Results and Analysis
- DeNet models obtain improved evaluation rates despite having more layers and parameters.The comparison uses a model overview table; the authors caution that base models, augmentation, dataset merging, and test settings differ across methods.Source: §4 · p. 5, Table 3 · p. 5
MAP (%)
The experiments examine DeNet’s sampling choices, timing, RoI coverage, and detection results across MSCOCO and Pascal VOC. They report competitive accuracy at real-time or near-real-time rates, with stronger RoI coverage at higher IoU thresholds.
4.1. Hyperparameter Optimization
- MAP improves with more training or testing bounding boxes, but gains diminish above 576 training samples and higher testing counts reduce evaluation rates; subsequent experiments use N = 24 for both.The experiments also report best results with λs = 1, λt = 100, and λb = 1.Source: §4.1 · p. 6, Figure 2 · p. 6, Table 4 · p. 6
4.2. Timing Breakdown and Evaluation Rates
- Deconvolution introduces spatial information later in the model, improving evaluation rates and allowing a more expressive base model at the same rate.The timing analysis identifies base-network corner generation as the dominant time cost; CPU-bound RoI generation can vary substantially and may need application-specific tuning.Source: §4.2 · pp. 6–7
4.3. RoI Coverage Comparison
- At higher IoU thresholds, DeNet models provide significantly better coverage than RPN (VGG) and R-FCN among the top 300 RoIs, although those methods cover better at low thresholds.RoI coverage does not necessarily translate into lower full-model MAP at low IoU, because the full model also uses NMS and bounding box regression.Source: §4.3 · p. 7, Table 7 · p. 7
4.4. MSCOCO
- MSCOCO’s evaluation metric integrates MAP from IoU=0.5 to IoU=0.95, placing greater emphasis on localization than Pascal VOC metrics.Source: §4.4 · p. 7
- DeNet-34 beats SSD300 by 6.2% MAP at the same evaluation rate and SSD512 by 2.6% at more than twice the evaluation rate on MSCOCO test-dev2015.DeNet-101 is beaten only by the much slower RPN+ model using multi-scale evaluation and bounding box refinement; analysis suggests DeNet’s advantage relates to large-object detection and fine localization.Source: §4.4 · p. 7, Table 8 · p. 8
4.5. Pascal VOC 2007
- DeNet-34 with skip layers improves upon SSD300’s peak MAP by 1.6% and 20Hz on Pascal VOC 2007.In the near-real-time domain, DeNet-101 matches SSD512 at a higher evaluation rate.Source: §4.5 · p. 7, Table 9 · p. 8
4.6. Pascal VOC 2012
- On Pascal VOC 2012, DeNet-34 matches SSD300, while DeNet-101 falls below SSD512 and is near-identical to RPN (ResNet) at an order-of-magnitude higher evaluation rate.Source: §4.6 · p. 8
5. Conclusion
The conclusion presents a computationally efficient CNN detection model that couples RoI prediction, classification, and bounding-box regression. It highlights fine localization and reduced manual engineering as advantages of sparse sampling.
5. Conclusion
- DeNet combines sparse CNN estimation with a novel RoI detector and classification model, reducing manual engineering while improving detection performance at real-time and near-real-time rates.Source: §5 · p. 8
- Deconvolution and skip layers support a computationally efficient model with tightly coupled RoI, class prediction, and bounding-box regression.Skip connections consistently improve detection rates for small and medium objects; the wide variant highlights corner-map resolution as relevant to those objects.Source: §5 · p. 8
- DeNet performs particularly well when finer localization is desirable, with 4.2 × 10^6 possible sampling boxes versus fewer than 2.5 × 10^4 for SSD512 and RPN.The authors propose that the larger candidate set can place a box closer to ground truth before regression, while avoiding manually defined reference boxes supports varied aspect ratios and scales.Source: §5 · p. 8