Source-linked AI summary
DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection
Wanli Ouyang, Ping Luo, Xingyu Zeng, Shi Qiu, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang, Yuanjun Xiong, Chen Qian, Zhenyao Zhu, Ruohui Wang, Chen-Change Loy, Xiaogang Wang, Xiaoou Tang
TL;DR
Object detection must handle substantial appearance, deformation, and contextual variation while learning task-appropriate representations. DeepID-Net combines deformable pooling, multi-stage training, specialized pretraining, contextual modeling, and diverse model averaging; it ranks #2 in ILSVRC 2014 and raises RCNN mean AP from 31.0% to 45%.
Problem
Object detection must handle intra-class variation in appearance, lighting, backgrounds, and object-part deformation while learning suitable feature representations.
Method
DeepID-Net combines learned features, def-pooling for part deformation, object-level pretraining, multi-stage classifiers, contextual information, and diverse model averaging.
Results
45% mean AP versus RCNN’s 31.0% was obtained on ImageNet detection validation 2, and the approach ranked #2 in ILSVRC 2014.
Takeaways & Limitations
A globally designed detection pipeline with diverse components and models substantially improves the reported RCNN baseline.
Abstract
from arXiv · showhide
In this paper, we propose multi-stage and deformable deep convolutional neural networks for object detection. This new deep learning object detection diagram has innovations in multiple aspects. In the proposed new deep architecture, a new deformation constrained pooling (def-pooling) layer models the deformation of object parts with geometric constraint and penalty. With the proposed multi-stage training strategy, multiple classifiers are jointly optimized to process samples at different difficulty levels. A new pre-training strategy is proposed to learn feature representations more suitable for the object detection task and with good generalization capability. By changing the net structures, training strategies, adding and removing some key components in the detection pipeline, a set of models with large diversity are obtained, which significantly improves the effectiveness of modeling averaging. The proposed approach ranked \#2 in ILSVRC 2014. It improves the mean averaged precision obtained by RCNN, which is the state-of-the-art of object detection, from $31\%$ to $45\%$. Detailed component-wise analysis is also provided through extensive experimental evaluation.
1. Introduction
DeepID-Net addresses object detection challenges by jointly improving feature representation, deformation modeling, contextual reasoning, and classification. Its multi-stage architecture, specialized pretraining, def-pooling, and diverse model averaging raise RCNN-based detection performance.
- Object detection is challenged by intra-class variation in appearance, lighting, backgrounds, and deformation.
- The proposed architecture learns feature representations, object-part deformation models, and contextual information for generic object detection.
- 45% mean AP, versus RCNN’s 31.0%, was achieved step-by-step on the ImageNet detection validation 2 dataset, with a #2 ILSVRC 2014 ranking.
- DeepID-Net integrates feature representation learning, part deformation learning, sub-box feature extraction, context modeling, model averaging, and bounding box refinement.
- Object-level 1000-class pretraining is followed by fine-tuning on the 200-class object detection dataset.
- Def-pooling models part deformation with geometric constraints and penalties, while replacing max-pooling at different abstraction levels.
2. Related Work
Prior work commonly treats detection components separately or sequentially. DeepID-Net instead takes a global view, combining learned features, deformation handling, context, and classification toward joint learning.
- Existing pooling layers handle deformation but do not learn deformation penalties and geometric models for object parts.
- Earlier deformable-part approaches model part movement, articulation, size, rotation, or appearance mixtures, but use manually designed features that are not jointly learned with deformation.
- Traditional classifiers adapt to training data, but manually designed features can lose information that classification cannot recover.
- Contextual information includes surrounding regions, object-scene interactions, and relationships among objects’ presence, location, orientation, and size.
- DeepID-Net takes a global view of components previously treated individually or sequentially, moving toward joint learning for object detection.
3. Dataset overview
ILSVRC 2014 includes separate classification-localization and detection datasets. The detection dataset covers 200 categories and uses a validation split aligned with RCNN for model validation.
- ILSVRC 2014 contains classification-localization and detection datasets.
- The classification-localization dataset contains 1000 object categories, with train, validation, and test subsets.
- The detection dataset contains 200 object categories and 395,918 training, 20,121 validation, and 40,152 test images.
- Detection validation and test images have similar distributions, while detection training has a different distribution and includes extra negative images excluded here.
- The validation set is split into val1 for training models and val2 for validating performance, following RCNN’s split.
4. Method
The method builds on RCNN’s proposal, feature, classification, and localization pipeline, then adds rejection, DeepID-Net scoring, context, model averaging, and regression. Bounding-box rejection reduces computation while preserving most recall and improving mean AP.
- 4.1. The RCNN approach: RCNN uses selective search proposals, AlexNet features, 200 one-versus-all classifiers, and bounding-box regression for localization refinement.
- 4.3. Bounding box proposal by selective search: DeepID-Net applies an existing detector to reject likely background boxes before processing the remaining cropped and warped proposals.
- 4.3. Bounding box proposal by selective search: Selective search generates 2403 bounding boxes per validation image, creating a costly feature-extraction workload.
- 4.4. Bounding box rejection: 94% of boxes are rejected using ||s_i||∞ < T, leaving 6% for DeepID-Net processing and focusing the model on hard examples.
- 4.4. Bounding box rejection: The remaining 6% of boxes retain 84.4% ground-truth recall at 0.5 IoU, a 7.2% drop from processing all boxes.
- 4.4. Bounding box rejection: 1.18 seconds per image is required for remaining-box feature extraction versus 10.24 seconds for all boxes, while rejection improves mean AP by around 1%.
5. Bounding box classification by DeepID-Net
DeepID-Net combines a baseline CNN with multi-stage classifiers, deformation-aware part processing, and whole-image context to produce refined object-detection scores. Its training strategy addresses the feature mismatch between image classification and tightly cropped object detection.
- Architecture: DeepID-Net processes candidate image regions through a baseline CNN, multi-stage fully connected layers, and variable-size filters with def-pooling to produce 200-class detection scores.The candidate region is warped to 227×227, and the def-pooling layer learns deformation constraints for part filters.
- Architecture: Whole-image 1000-class classification scores are combined with 200-class bounding-box scores to refine the final 200-class detection scores.The whole image supplies contextual information for refining scores of candidate bounding boxes.
- Pretraining strategy: RCNN pretraining creates a feature mismatch because image classification uses whole images and requires scale and translation robustness, whereas detection uses tight bounding-box regions.The paper identifies this task mismatch as a source of mismatched learned representations.
- Pretraining strategy: DeepID-Net bridges the annotation gap by pretraining with object-level annotations from 1,000 ImageNet classes before fine-tuning on 200 detection classes.Scheme 1 inserts object-level 1,000-class fine-tuning after image-level pretraining, while Scheme 2 directly uses object-level 1,000-class annotations before 200-class detection fine-tuning.
- Multi-stage training: Multiple classifiers are jointly trained across stages so that different stages process samples at different difficulty levels.The stage-by-stage procedure updates stage-specific parameters and then jointly updates the baseline and preceding classifier parameters through back-propagation.
7 end
DeepID-Net combines multi-stage classifiers with deformation-constrained pooling to model difficult samples and deformable object parts. Its sub-box feature aggregation also improves detection accuracy while reusing computed features.
- Multi-stage training: Unlike cascaded classifiers that pass one score, DeepID-Net transfers information through multiple hidden nodes across stages.Previous-stage and current-stage classifiers jointly process misclassified samples.
- Multi-stage training: Multi-stage training adds classifiers progressively and jointly optimizes existing classifiers to handle misclassified samples at different difficulty levels.The baseline model is trained first, then extra classifiers are introduced stage-by-stage.
- Deformation-constrained pooling: Def-pooling applies variable-size filters to conv5 features, learns deformation constraints, and produces part-aware outputs for classification.Filters of sizes 3 × 3, 5 × 5, and 9 × 9 generate 128-channel part detection maps before def-pooling and subsequent 1×1 convolutions.
- Deformation-constrained pooling: Def-pooling sums part detection responses with deformation penalties and performs block-wise max pooling, generalizing max-pooling with learned deformation constraints.When the penalty coefficient is zero, def-pooling degenerates to max-pooling; with one output location, it degenerates to the earlier deformation layer.
- Deformation-constrained pooling: The model supports multiple part responses and pooling at different abstraction levels, unlike an earlier deformation layer restricted to one output and one part response.This design accommodates objects with multiple instances of a part, such as windows or light bulbs.
- Sub-box features: Sub-box features reuse selective-search computations, concatenate root, max-pooled, and average-pooled features, and improve mAP by 0.5%.Selected sub-boxes may move, improving robustness to translation of object parts while avoiding repeated feature computation.
6. Combining models with high diversity
DeepID-Net combines diverse models that differ in architecture, training, losses, and components, improving detection through complementary model averaging. Class-specific model selection further raises validation performance.
- Model diversity: The 10 averaged models vary in net structure, pretraining, loss functions, def-pooling, multi-stage training, sub-box features, and bounding box rejection.The models were selected by greedy search based on val2 performance.
- Model diversity: 40.9% mAP on val2 and 40.7% mAP on ILSVRC2014 test data ranked the model-averaging system #2 in the challenge.
- Model averaging results: 42.4% mAP was obtained on val2 after post-challenge improvements to the model-averaging results.
- Class-specific selection: 45% mAP on val2 was achieved by selecting model combinations separately for each object category.Different models were observed to vary substantially in effectiveness across categories.
7. Experimental Results
Experiments evaluate the pipeline against RCNN and analyze pretraining, architectures, and component additions on ImageNet detection data. The results show gains from object-level pretraining, deformable pooling, context, bounding-box processing, and model averaging.
- Experimental setup: 29.9 mAP on val2 was obtained without bounding box regression, close to RCNN's reported 29.7 mAP baseline.ImageNet Det val2 evaluates components, while test data evaluates overall performance.
- Pretraining: Object-level pretraining without image-level pretraining outperformed the two-stage alternative by 2.6% mAP.The paper concludes that image-level annotation is unnecessary when object-level annotation is available.
- Network structures: Multi-stage training improved mAP by 1.5% over the C-net baseline in the investigated DeepID-Net configuration.
- Pipeline ablation: Replacing image-level with object-level pretraining increased mAP by about 4%, while def-pooling added 2.5%.These are step-by-step contributions in the overall pipeline analysis.
- Pipeline ablation: Contextual information, bounding box regression, and model averaging contributed about 1%, about 1%, and the best 45% mAP result, respectively.
8. Appedix A: Relationship between the deformation layer and the DPM in [20]
The appendix relates the deformation layer to DPM-style quadratic deformation costs. Learned parameters control deformation penalties and part centers, producing a final deformation-adjusted response.
- DPM relationship: The quadratic deformation constraint assigns a cost to displacement from a predefined part anchor location.
- DPM relationship: Parameters c1 and c2 control deformation cost: zero permits unconstrained movement, whereas infinity prevents parts from moving.
- Parameterization: The anchor location and learned offsets jointly determine the center of the part under the quadratic constraint.
- Output: The final output is computed from the deformation-adjusted matrix, with c1, c2, c3, and c4 learned and c5 fixed across locations.
9. Conclusion
DeepID-Net learns feature extraction, deformation handling, context modeling, and classification within a generic object-detection pipeline. The component ablations summarize how these design choices contribute to the final system.
- Conclusion: DeepID-Net learns feature extraction, deformation handling, context modeling, and classification for generic object detection.
- Conclusion: The pipeline ablations report mean AP after adding key components step by step and summarize each component's contribution.