Source-linked AI summary
Deformable Part Models are Convolutional Neural Networks
Ross Girshick, Forrest Iandola, Trevor Darrell, Jitendra Malik
TL;DR
The paper addresses the perceived divide between DPMs and CNNs by formulating DPM inference as an equivalent CNN and using learned CNN features in DeepPyramid DPM. On PASCAL VOC, the resulting detector significantly outperforms HOG-based DPMs, slightly exceeds comparable R-CNN performance, and runs about 20x faster.
Problem
DPMs and CNNs are typically viewed as distinct approaches to visual recognition, despite both being influential model classes.
Method
The paper unrolls DPM inference into equivalent CNN layers and applies the resulting DPM-CNN to a feature pyramid computed by another CNN, yielding DeepPyramid DPM.
Results
DeepPyramid DPM significantly outperforms equivalent HOG-based detectors and slightly outperforms comparable R-CNN while running about 20x faster.
Takeaways & Limitations
The construction shows that DPMs can be expressed as CNNs and supports replacing HOG templates with learned features in visual recognition systems.
Takeaways & Limitations
The construction is limited to models with exact inference using a non-iterative algorithm; loopy models requiring iterative approximate inference cannot be converted to equivalent fixed-depth CNNs.
Abstract
from arXiv · showhide
Deformable part models (DPMs) and convolutional neural networks (CNNs) are two widely used tools for visual recognition. They are typically viewed as distinct approaches: DPMs are graphical models (Markov random fields), while CNNs are "black-box" non-linear classifiers. In this paper, we show that a DPM can be formulated as a CNN, thus providing a novel synthesis of the two ideas. Our construction involves unrolling the DPM inference algorithm and mapping each step to an equivalent (and at times novel) CNN layer. From this perspective, it becomes natural to replace the standard image features used in DPM with a learned feature extractor. We call the resulting model DeepPyramid DPM and experimentally validate it on PASCAL VOC. DeepPyramid DPM significantly outperforms DPMs based on histograms of oriented gradients features (HOG) and slightly outperforms a comparable version of the recently introduced R-CNN detection system, while running an order of magnitude faster.
1. Introduction
The paper reframes DPMs and CNNs as compatible approaches by expressing a DPM as an equivalent CNN and replacing HOG features with learned CNN features. DeepPyramid DPM outperforms HOG-based DPMs and slightly exceeds comparable R-CNN performance while running about 20x faster.
- DPMs and CNNs are typically viewed as distinct visual-recognition approaches, with DPMs modeled graphically and CNNs treated as black-box nonlinear classifiers.
- The paper formulates a DPM as an equivalent CNN, providing a synthesis of the two approaches.
- DeepPyramid DPM replaces HOG features with a learned CNN feature pyramid and forms a single unified CNN through function composition.
- DeepPyramid DPM slightly outperforms comparable R-CNN on object-detection mean average precision while running about 20x faster.The comparison uses R-CNN on the same conv5 features without fine-tuning.
- Sliding-window detectors on deep feature pyramids significantly outperform equivalent models using HOG features.
2. DeepPyramid DPM
DeepPyramid DPM formulates DPM inference as a CNN by unrolling detection into convolutional, distance-transform pooling, geometry, and maxout layers, then applying it to CNN-derived feature pyramids.
- DeepPyramid DPM: DeepPyramid DPM maps an image pyramid to a feature pyramid with a first CNN, then applies a DPM-CNN to produce detection-score pyramids.The two networks compose into a single unified CNN.
- DPM-CNN construction: DT pooling generalizes max pooling by replacing a fixed pooling window with a learnable distance penalty over the feature-map domain.For DPM, the penalty is a convex quadratic with learnable parameters for each part.
- DPM-CNN construction: A single-component DPM-CNN convolves root and part filters, applies DT pooling to part maps, and combines them with an object geometry filter.The resulting map contains one DPM component score per location.
- Object geometry filters: Making object geometry explicit reveals a second implicit convolutional layer whose filter could, in principle, be trained discriminatively.The standard construction uses a sparse binary pattern for this filter.
- Combining mixture components with maxout: Multi-component DPM-CNNs combine one component network per mixture component with a maxout layer that selects the highest component score at each location.The max operation has a direct interpretation as switching among model components.
- Generalizations: The construction extends to models with exact non-iterative inference, but loopy graphical models requiring iterative approximate inference cannot become equivalent fixed-depth CNNs.The stated boundary excludes models such as hierarchical poselets with loopy structures.
3. Implementation details
The implementation uses a truncated SuperVision CNN to compute conv5 feature pyramids and trains the feature extractor and DPM in separate stages.
- Implementation: The system uses Caffe and modifies the DPM voc-release5 code for its experiments.The complete system’s source code was planned for release.
- CNN feature extraction: The feature extractor uses publicly available R-CNN network weights trained for ILSVRC 2012 classification without detection fine-tuning.These weights enable a direct comparison with R-CNN.
- CNN feature extraction: Removing pool5 and fully connected layers leaves the fifth convolutional layer, conv5, as the output with 256 feature channels.The network also uses zero-padding before convolutional and max-pooling layers.
- Image pyramid: The implementation embeds seven image-pyramid levels in a 1713 × 1713 pixel canvas and processes them through the CNN.For PASCAL VOC, the original image is upsampled by 3.4× on average at the first level.
- Training: DeepPyramid DPM could be trained end-to-end, but the experiments instead train it stage-wise in two stages.The first stage uses the pretrained CNN; the second freezes it and trains a DPM on conv5 feature pyramids.
- Training: Relative to HOG-based DPM training, the implementation removes mirrored mixture-component pairs and uses R-CNN’s intersection-over-union-based non-maximum suppression.These changes were found necessary for the deep-feature setting.
4. Experiments
Experiments on PASCAL VOC show that DeepPyramid DPM benefits from conv5-based representations and implicit parts, outperforming HOG-based DPMs and remaining competitive with comparable R-CNN variants.
- Model design: 0.9 percentage points: adding four 3 × 3 parts to a three-component DPM increases mAP, improving 16 of 20 classes with p = 0.016.The measured effect is small but was judged statistically significant using a paired-sample permutation test.
- Feature analysis: Conv5 features describe large 163 × 163 pixel patches and respond selectively to faces over a narrow scale range, unlike scale-invariant HOG statistics.In the visualized face pyramid, conv5 activity peaks when the face reaches its optimal scale.
- Model design: A 3 × 3 max filter over conv5 followed by a three-component root-only DPM improves mAP and training and testing speed over direct conv5 DPM variants.The max-filtered representation, called max5, treats conv5 channels as shared parts without explicit DPM parts.
- Comparison with other methods: 45.2% mAP: the max5 DeepPyramid DPM exceeds the 33.7% standard six-component, eight-part HOG-DPM baseline.Removing parts from the HOG-DPM baseline lowers performance to 25.2%.
- Comparison with other methods: DeepPyramid DPM slightly outperforms comparable R-CNN using the same conv5 features without fine-tuning while running about 20x faster.Performance varies by class: R-CNN is stronger for aeroplane and cat, whereas DeepPyramid DPM is stronger for bottle, chair, and person.
- Results on PASCAL VOC 2010-2012: On VOC 2010, DeepPyramid DPM outperforms recent methods other than fine-tuned R-CNN, with 41.6% mAP on VOC 2011 and 2012.Its performance advantage over HOG-DPM is especially strong, and it performs better than R-CNN FT fc7 without bounding-box regression for bottle and person.
5. Conclusion
The paper presents a synthesis in which deformable part models become equivalent CNNs, yielding DeepPyramid DPM for multi-scale detection and learned feature pyramids.
- Any DPM can be expressed as an equivalent CNN using distance transform pooling layers and maxout units.Distance transform pooling generalizes max pooling and connects deformable parts with CNN pooling.
- DeepPyramid DPM runs a DPM-CNN on a feature pyramid constructed by another CNN.This replaces HOG-based input features with features learned by a CNN.
- DeepPyramid DPM is a single CNN that maps an image pyramid to a detection score pyramid for multi-scale object detection.
- The paper shows the potential for replacing HOG templates in a wide range of visual recognition systems.