Source-linked AI summary
DeepID-Net: Deformable Deep Convolutional Neural Networks for Object Detection
Wanli Ouyang, Xiaogang Wang, Xingyu Zeng, Shi Qiu, Ping Luo, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang, Chen-Change Loy, Xiaoou Tang
TL;DR
Object detection must handle appearance variation, deformation, and a mismatch between image-classification pretraining and detection fine-tuning. DeepID-Net combines task-suitable pretraining, deformation-constrained pooling, contextual refinement, and diverse model aggregation. It raises RCNN's mean AP from 31.0% to 50.3% on ILSVRC2014 and outperforms the challenge winner by 6.1% mAP.
Problem
Object detection faces appearance variation and deformation, while conventional image-classification pretraining mismatches detection's localization requirements.
Method
DeepID-Net jointly learns features and deformable parts using def-pooling, task-specific pretraining, contextual refinement, and diverse detector aggregation.
Results
50.3% mean AP on ILSVRC2014 improved RCNN's 31.0% mean AP and exceeded the ILSVRC2014 winner by 6.1% mAP.
Takeaways & Limitations
Component-wise analysis and a unified evaluation provide a global view of how detection-pipeline components improve deep-learning object detection.
Abstract
from arXiv · showhide
In this paper, we propose deformable deep convolutional neural networks for generic object detection. This new deep learning object detection framework has innovations in multiple aspects. In the proposed new deep architecture, a new deformation constrained pooling (def-pooling) layer models the deformation of object parts with geometric constraint and penalty. A new pre-training strategy is proposed to learn feature representations more suitable for the object detection task and with good generalization capability. By changing the net structures, training strategies, adding and removing some key components in the detection pipeline, a set of models with large diversity are obtained, which significantly improves the effectiveness of model averaging. The proposed approach improves the mean averaged precision obtained by RCNN \cite{girshick2014rich}, which was the state-of-the-art, from 31\% to 50.3\% on the ILSVRC2014 detection test set. It also outperforms the winner of ILSVRC2014, GoogLeNet, by 6.1\%. Detailed component-wise analysis is also provided through extensive experimental evaluation, which provide a global view for people to understand the deep learning object detection pipeline.
1. Introduction
DeepID-Net addresses object detection challenges from appearance variation, deformation, and a mismatch between image-classification pretraining and detection fine-tuning. It combines deformable modeling, task-suitable pretraining, diverse detectors, and component-wise evaluation.
- Object detection is challenged by intra-class appearance variation and object deformation.
- Deep CNN training faces difficult initialization and overfitting problems because it involves nonconvex optimization with millions of parameters.
- Image-level pretraining encourages scale and location robustness, whereas detection requires sensitivity to normalized candidate-region size and position for localization.
- Def-pooling learns shared deformation properties for visual patterns across object classes, semantic levels, and composition levels.
- DeepID-Net jointly learns feature representations and part deformation while investigating bounding-box rejection, training schemes, context modeling, and model averaging.
- 50.3% mean AP was achieved on ILSVRC2014, improving RCNN's 31.0% mean AP through component-wise additions.
2. Related work
Related work treats deformation, context, and other detection components separately, while DeepID-Net integrates them and evaluates them under a common large-scale detection setting.
- Deformable part-based approaches model translational movement, articulation, size change, rotation, or shared patterns, but use manually designed features.
- Max pooling and average pooling help handle deformation but cannot learn deformation penalties or geometric models of object parts.
- Whole-image classification scores are used as global contextual information to refine detection scores.
- The paper integrates feature learning, deformation modeling, context modeling, pretraining, network structures, bounding-box refinement, and model averaging under one experimental setting.
3. Method
The approach uses selective search and background rejection before DeepID-Net scores cropped regions, then refines detection with whole-image classification context. Its architecture combines a baseline model, deformable part branches, and an image-classification model.
- Overview of our approach: Selective search proposes candidate bounding boxes for the ImageNet object detection pipeline.
- Overview of our approach: RCNN rejects candidate boxes likely to be background before cropped regions enter DeepID-Net.
- Overview of our approach: DeepID-Net produces 200-class detection scores from cropped image regions, while 1000-class whole-image scores provide contextual refinement.
- Architecture of DeepID-Net: Def-pooling branches operate on conv5 features to learn deformation constraints for part filters.
- Architecture of DeepID-Net: The architecture includes a baseline deep model, branches with variable-size part filters and def-pooling layers, and a 1000-class image-classification model.
3.3. New pretraining strategy
The paper replaces image-classification pretraining with object-level pretraining on more diverse ImageNet data, then fine-tunes for detection. This strategy improves detection performance over the conventional RCNN training scheme.
- The conventional scheme pretrains on ImageNet image classification and localization data before fine-tuning on object detection annotations.
- The proposed scheme pretrains on image regions cropped from bounding boxes in the 1,000-class ImageNet Cls-Loc dataset.The model is then fine-tuned for the 200-class ImageNet Det task.
- 4.5% mean AP improvement is obtained on ImageNet Det val2 compared with RCNN training.
- 5.7% mean AP is lost when pretraining uses only the 200 target classes instead of all 1,000 Cls-Loc classes.
3.4. Def-pooling layer
Def-pooling extends convolutional pooling by combining part-detection responses with learned deformation penalties before block-wise max pooling. It supports learned, spatially structured deformation across visual patterns, abstraction levels, and object classes.
- Def-pooling treats convolutional outputs as part-detection maps and processes local spatial blocks around assumed part locations.The input is a convolutional layer producing C maps of size W × H.
- The layer sums each part-detection score with a learned deformation penalty and applies block-wise max pooling to produce the output.The deformation penalty is associated with displacement from an anchor position to a deformed position.
- Def-pooling generalizes max-pooling by learning a penalty map that implicitly determines the effective pooling kernel size.A sufficiently large penalty outside a permitted range reproduces max-pooling with a bounded kernel.
- 3.4.1 Analysis: The learned penalties can encode directional and spatial preferences, including diagonal, vertical, horizontal, and asymmetric displacement patterns.Figure 5 illustrates different penalty maps for different visual patterns.
- 3.4.1 Analysis: Def-pooling can model parts of different sizes and semantic levels because it can replace convolutional layers throughout the network.Examples include upper bodies, heads, and lower-level visual patterns.
- 3.4.1 Analysis: The layer shares learned visual patterns across object classes, including circular patterns appearing in traffic lights, cars, and ipods.Class-specific spatial distributions can still be modeled by subsequent convolutional connections.
3.5. Fine-tuning the deep model with hinge-loss
The approach merges deep-model fine-tuning and class-specific SVM learning by replacing softmax fine-tuning with binary hinge losses.
- RCNN separately fine-tunes a deep model with softmax loss and trains a linear binary SVM for each detection class.
- The proposed approach replaces softmax with 200 binary hinge losses during fine-tuning.This merges deep-model fine-tuning and SVM learning into one step.
3.6. Contextual modeling
Contextual modeling combines whole-image classification information with local object-detection scores. This supplements local evidence when visually similar objects occur in different scenes.
- Whole-image classification scores provide contextual features that are concatenated with local 200-class detection scores.The resulting 1,200-dimensional vector is used to train a linear SVM that refines detection scores.
- For volleyball detection, scene context helps distinguish volleyballs from bathing caps and golf balls when local regions are ambiguous.Beach, swimming-pool, grass-field, and stadium scenes provide different contextual cues.
3.7. Combining models with high diversity
The model-averaging scheme combines detectors trained with diverse architectures, pretraining schemes, losses, and components. Greedy selection of four models achieves higher mAP than the best single model.
- Diverse model settings: Models vary in net structures, pretraining schemes, loss functions, def-pooling usage, and bounding box rejection.These settings produce diverse models that are complementary for detection.
- Model averaging: 50.3% mAP is achieved by averaging four models automatically selected with greedy search on ImageNet Det val2.The result is measured on the ILSVRC2014 test data.
- Model averaging: 48.2% mAP is obtained by the best single model, below the 50.3% mAP achieved by model averaging.The comparison is reported on ILSVRC2014 test data.
4. Experimental results
Experiments evaluate DeepID-Net and its components on ImageNet Det and PASCAL VOC under controlled settings. Results show gains from pretraining choices, network structures, deformation modeling, context, and model averaging.
- ImageNet Det: 6.1% mAP is the margin by which the approach outperforms the ILSVRC2014 winner, GoogLeNet, on ImageNet Det.The paper reports the highest mAP for both single-model and model-averaging performance.
- Baseline and rejection: 1% mAP is gained by applying bounding box rejection to the RCNN baseline.The component also saves training and validation time for new models.
- Pretraining schemes: 5.7% mAP is the advantage of object-level pretraining on 1000 classes over object-level pretraining on 200 classes.Using 1000 classes improves the generalization capability of learned feature representations.
- Pretraining schemes: 4.4% mAP for A-net and 4.2% for Z-net are the gains from object-level over image-level pretraining with the same 1000 classes.The experiment supports object-level annotation as the better pretraining choice in these settings.
- Overall pipeline: 50.7% final mAP results from stepwise additions including network changes, object-level pretraining, candidate combination, def-pooling, context, regression, and model averaging.Table 6 reports the cumulative single-model pipeline result after the listed additions and model averaging.
5. Appedix A: Relationship between the deformation layer and the DPM
The appendix relates the quadratic deformation constraint to part detection maps. The formulation uses learned parameters to set part centers and deformation costs before producing a final output map.
- Quadratic deformation constraint: The quadratic deformation constraint modifies each element of the part detection map M around a predefined part anchor location.m(i,j) is the (i,j)th element of M, and (b1,b2) is the anchor location of the pth part.
- Parameter roles: a3/2a1 and a4/2a2 adjust the anchor location, while a1 and a2 determine the deformation cost.The center of the part is jointly determined by the anchor and these learned adjustments.
- Output: The final output is formed from the transformed deformation map, whose elements are denoted by ˜m(i,j).The appendix identifies ˜m(i,j) as the corresponding element of matrix ˜M in Eq. (2).
6. Conclusion
The paper presents an integrated deep-learning object-detection pipeline that combines deformation handling, task-specific pretraining, and diverse model configurations. It reports a substantial improvement over the prior RCNN result and analyzes the pipeline's components under a common setting.
- 50.3% mAP improves on RCNN's 31.0% result for the ImageNet object task.
- The pipeline integrates bounding-box rejection, pretraining, deformation handling, context modeling, bounding-box regression, and model averaging.
- The def-pooling layer accommodates diverse deformation-handling approaches and deep architectures.
- A proposed pretraining scheme targets object-detection feature representations with good generalization capability.
- Changing pipeline configurations produces diverse detectors that improve the effectiveness of model averaging.