Source-linked AI summary
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, Yann LeCun
TL;DR
ConvNets had advanced image classification, but their integrated use for classification, localization, and detection remained to be demonstrated. OverFeat addresses this with a shared multiscale, sliding-window network, achieving first-place results in ILSVRC 2013 localization and detection.
Problem
The paper addresses how a single ConvNet can jointly perform image classification, localization, and detection rather than treating these tasks separately.
Method
OverFeat uses a shared ConvNet with multiscale sliding-window processing and accumulated bounding-box predictions for classification, localization, and detection.
Results
OverFeat ranked 4th in classification, 1st in localization, and 1st in detection on ILSVRC 2013.
Takeaways & Limitations
The results demonstrate that ConvNets can support challenging ImageNet localization and detection tasks alongside classification in an integrated framework.
Takeaways & Limitations
The localization approach does not backpropagate through the whole network and uses ℓ2 loss instead of directly optimizing the measured intersection-over-union criterion.
Abstract
from arXiv · showhide
We present an integrated framework for using Convolutional Networks for classification, localization and detection. We show how a multiscale and sliding window approach can be efficiently implemented within a ConvNet. We also introduce a novel deep learning approach to localization by learning to predict object boundaries. Bounding boxes are then accumulated rather than suppressed in order to increase detection confidence. We show that different tasks can be learned simultaneously using a single shared network. This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013) and obtained very competitive results for the detection and classifications tasks. In post-competition work, we establish a new state of the art for the detection task. Finally, we release a feature extractor from our best model called OverFeat.
1 Introduction
The paper proposes a single integrated ConvNet that simultaneously classifies, localizes, and detects objects, using accumulated bounding-box predictions to improve all three tasks. Experiments on ImageNet establish state-of-the-art results for ILSVRC 2013 localization and detection.
- Core contribution: A single ConvNet is trained to simultaneously classify, locate, and detect objects, with the paper claiming improved accuracy across all three tasks.The approach integrates object recognition, localization, and detection in one network.
- Core contribution: Accumulating predicted bounding boxes is introduced as a method for localization and detection.The method aggregates bounding-box predictions rather than treating them independently.
- Experimental results: The experiments use the ImageNet ILSVRC 2012 and 2013 datasets and establish state-of-the-art results on the ILSVRC 2013 localization and detection tasks.The reported results cover both localization and detection.
- Method motivation: The approach addresses variation in object size and position by applying a ConvNet across multiple image locations and scales in a sliding-window fashion.The motivation is that ImageNet objects may vary substantially in size and position.
- Task definitions: Localization and detection both involve predicting a bounding box for each object, differing only in the evaluation criterion used in the ImageNet 2013 competition.This terminology follows the competition’s definitions.
2 Vision Tasks
The paper examines classification, localization, and detection as progressively harder vision tasks within one framework and shared feature-learning base. Localization serves as an intermediate evaluation step between classification and detection, while detection additionally involves small-object data and detection-specific challenges.
- Task progression: The paper studies classification, localization, and detection in increasing order of difficulty, with each task a sub-task of the next.All three tasks are addressed using a single framework and shared feature-learning base, though they are described separately.
- Task definitions: Classification assigns each image one label for its main object, with five guesses allowed because images may contain multiple unlabeled objects.The classification and localization tasks use the same dataset.
- Task definitions: Localization extends classification by requiring a bounding box for the predicted object while retaining five guesses per image.It provides an intermediate step between classification and detection and enables independent evaluation of the localization method.
- Task definitions: Detection differs from classification and localization by involving detection-specific challenges, such as learning a background class, and additional data containing small objects.The paper reports results on ILSVRC2013 and shows localization/detection predictions alongside corresponding ground truth.
3 Classification
OverFeat combines a convolutional classification architecture with dense, multiscale sliding-window inference that explores image locations and scales efficiently. This approach achieves a 13.6% top-5 error rate, while the competition system obtains 14.2% accuracy and ranks fifth among 18 teams.
- Architecture: The classification architecture improves on Krizhevsky et al.’s design and inference step, while retaining rectified nonlinearities and max pooling with modified pooling and normalization choices.The model omits contrast normalization, uses non-overlapping pooling regions, and has larger first- and second-layer feature maps.
- Multiscale classification: Dense multiscale inference explores the entire image at multiple scales, producing many voting views while remaining efficient for ConvNets.The network is run at each location and across multiple input scales; convolution produces spatial maps of class vectors.
- Multiscale classification: Fine-stride resolution augmentation improves alignment by pooling layer-5 maps with offsets and applying the classifier in sliding-window fashion.Six input scales are used, with 3x3 pooling repeated for pixel offsets of {0, 1, 2}; outputs are assembled into spatial class maps.
- Results: 13.6% top-5 error is achieved with six scales, compared with 16.97% for the single-scale model.Using fewer scales hurts performance, while fine stride provides a relatively small improvement in the single-scale regime and matters for multiscale inference.
- Results: 14.2% accuracy is obtained by voting across 7 ConvNets, ranking OverFeat 5th out of 18 teams in the 2013 competition.Post-competition, larger models reduce the result to 13.6% error.
4 Localization
OverFeat localizes objects by replacing the classifier layers with a shared-feature regression network that predicts bounding boxes across spatial locations and scales, then greedily merges these predictions with classification scores. Multiscale and multi-view predictions yield 29.9% localization error on the ILSVRC2013 competition, while a shared regressor outperforms per-class regression.
- Regression architecture: The regression network predicts four bounding-box edge coordinates from pooled layer-5 features at every spatial shift and scale.It uses two fully connected hidden layers of 4096 and 1024 channels, with 3x3 copies induced by the spatial shifts.
- Regression training: OverFeat fixes classification feature layers 1–5 and trains class-specific regressors with an ℓ2 loss against shifted ground-truth boxes across the same scales used for classification.The final regressor layer has 1000 versions, one for each class.
- Prediction merging: Bounding-box predictions are greedily merged by averaging coordinates, and final scores cumulatively add classification outputs from the windows generating each box.The final prediction selects merged boxes with maximum class scores.
- Localization results: 29.9% error made the method the winner of the 2013 ImageNet localization competition.The evaluation used the competition’s localization criterion on the ImageNet 2012 validation set.
- Localization results: 40% error from a single centered crop fell to 31.5% with two-scale all-location predictions and 30.0% after adding third and fourth scales.These results demonstrate the importance of the multiscale and multi-view approach.
- Localization results: 44.1% versus 31.3% error shows that per-class regression did not outperform a single regressor shared across classes.The paper attributes this result possibly to relatively few class-level bounding-box examples and 1000 times more top-layer parameters.
5 Detection
OverFeat trains detection spatially with shared convolutional weights and an explicit background class, using hard-negative bootstrapping. It achieved 19.4% mAP during ILSVRC 2013 and later reached 24.3% mAP, establishing a new detection state of the art.
- Detection training: Detection training operates spatially, sharing convolutional weights across image locations and predicting a background class when no object is present.Multiple image locations can be trained simultaneously because the model is convolutional.
- Detection training: Hard-negative bootstrapping adds the most offending negative errors to the training set after initially sampling negative examples at random.This procedure follows the initial random selection of negative examples during detection training.
- Competition results: 19.4% mAP ranked OverFeat’s detection system third in the ILSVRC 2013 competition.The competition results are reported on the ILSVRC 2013 detection task.
- Competition results: 24.3% mAP established a new detection state of the art in post-competition work.The reported post-competition result improved over OverFeat’s competition result.
- Comparison with competing systems: Inference differs from competing systems that use segmentation to reduce candidate windows from approximately 200,000 to 2,000.Those systems reduce potential false positives and speed inference by discarding unlikely object locations.
- Result improvements: Longer training and context from feeding lower-resolution scales into each scale account for the improvement between OverFeat’s two reported results.OverFeat did not fine-tune on the detection validation set, unlike NEC and UvA.
6 Discussion
The paper presents a multiscale sliding-window ConvNet framework for classification, localization, and detection, achieving top ILSVRC 2013 results in localization and detection. It also identifies several opportunities to improve localization, including end-to-end backpropagation and directly optimizing IOU.
- Contributions: The multiscale sliding-window approach supports classification, localization, and detection within a unified ConvNet framework.The authors also explain how ConvNets can be effectively applied to detection and localization tasks.
- Results: 1st in localization and 1st in detection, while ranking 4th in classification on the ILSVRC 2013 datasets.These rankings are reported as the approach’s current competition performance.
- Limitations and future improvements: Localization may improve by back-propagating through the whole network rather than stopping short of end-to-end training.The current approach does not back-propagate through the entire network.
- Limitations and future improvements: Replacing ℓ2 loss with direct optimization of the intersection-over-union criterion may better align training with the measured localization performance.IOU remains differentiable provided there is some overlap.
Appendix: Additional Model Details
The appendix details the accurate model’s architecture, compares model parameterization, and specifies spatial dimensions for the multi-scale approach. It also explains how dense pooling produces output maps from unpooled feature maps.
- Architecture specifics: The accurate model differs from the fast model mainly in first-convolution stride, number of stages, and number of feature maps.These architecture specifics are summarized in Table 3.
- Model comparison: Table 4 reports the number of parameters and connections for different models.The table provides model-size and connectivity details.
- Multi-scale spatial dimensions: The multi-scale approach uses 6 input-image sizes, yielding layer 5 unpooled feature maps with differing spatial resolutions and 256 feature channels.Dense pooling uses (∆x, ∆y) = {0, 1, 2} to convert these maps into output maps.