PaperlayerView exact revision on arXiv

Source-linked AI summary

Rich feature hierarchies for accurate object detection and semantic segmentation

Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik

arXiv:1311.2524v5cs.CV

TL;DR

Object detection had stagnated, with leading systems relying on complex ensembles and uncertainty about whether CNN classification results would transfer to detection. R-CNN applies CNNs to region proposals and uses supervised pre-training followed by fine-tuning, achieving a 30% relative improvement on PASCAL VOC 2012 and outperforming OverFeat on ILSVRC2013 detection.

  • Problem

    Object detection progress had stagnated, while whether CNN classification results generalize to PASCAL VOC detection remained unresolved.

  • Method

    R-CNN applies a CNN to category-independent region proposals and uses supervised auxiliary-task pre-training followed by domain-specific fine-tuning.

  • Results

    30% relative improvement over previous PASCAL VOC 2012 results, with 31.4% mAP versus OverFeat’s 24.3% on ILSVRC2013 detection.

  • Takeaways & Limitations

    Supervised pre-training followed by domain-specific fine-tuning is highly effective for learning high-capacity CNNs when labeled detection data is scarce.

  • Takeaways & Limitations

    R-CNN has a significant speed disadvantage relative to OverFeat, which is about 9x faster.

Abstract

from arXiv · show

Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012---achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also compare R-CNN to OverFeat, a recently proposed sliding-window detector based on a similar CNN architecture. We find that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset. Source code for the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn.

1. Introduction

The introduction frames stagnant PASCAL VOC detection progress as motivating CNN-based region recognition, and presents R-CNN’s localization, data-scarcity, comparative, efficiency, error-analysis, and segmentation results.

  • PASCAL VOC detection progress was slow during 2010–2012, with only small gains from ensemble systems and minor method variants.
  • R-CNN bridges image classification and detection by applying CNN features to around 2000 region proposals and classifying them with category-specific linear SVMs.The method addresses localizing objects with a deep network while retaining the recognition-using-regions paradigm.
  • 31.4% versus 24.3%: R-CNN significantly outperformed OverFeat on the 200-class ILSVRC2013 detection dataset.
  • Supervised pre-training on ILSVRC followed by PASCAL fine-tuning effectively trains high-capacity CNNs when annotated detection data is scarce.
  • Two orders of magnitude lower-dimensional features shared across categories make class-specific computation limited to a small matrix-vector product and greedy non-maximum suppression.
  • 47.9%: R-CNN achieved average segmentation accuracy on the VOC 2011 test set, while bounding-box regression reduced mislocalizations, its dominant error mode.

2. Object detection with R-CNN

R-CNN combines category-independent region proposals, shared CNN feature extraction, and class-specific linear SVMs for object detection. Its design supports efficient large-scale detection while achieving strong VOC and ILSVRC performance.

  • System design: R-CNN uses three modules: category-independent region proposals, a CNN that extracts fixed-length region features, and class-specific linear SVMs.The proposals define candidate detections, while the CNN and SVMs classify them.
  • Feature extraction: 4096-dimensional feature vectors are extracted by forward propagating warped, mean-subtracted 227 × 227 RGB proposal images through five convolutional and two fully connected layers.Regardless of region size or aspect ratio, pixels in each tight bounding box are warped to the CNN’s required input size.
  • Test-time detection: At test time, selective search produces around 2000 proposals per image, which are scored with class-specific SVMs and filtered using class-independent greedy non-maximum suppression.CNN computation is shared across categories; only SVM dot products and suppression remain class-specific.
  • Run-time analysis: 10 seconds is sufficient for the matrix multiplication with 100k classes on a modern multi-core CPU, enabling scaling to thousands of object classes without approximation.R-CNN’s shared, low-dimensional features avoid the much larger memory and computation costs of high-dimensional alternatives.
  • Results: 53.7% mAP is achieved versus 35.1% for the prior multi-feature method, while R-CNN is much faster and obtains 53.3% mAP on VOC 2011/12 test.On ILSVRC 2013 detection, R-CNN reaches 31.4% mAP versus 24.3% for OverFeat.

3. Visualization, ablation, and modes of error

The analysis visualizes what CNN units detect and shows that fine-tuning, architecture choice, and bounding-box regression substantially affect detection performance. Without fine-tuning, higher-layer features can be pruned without degrading mAP, while fine-tuning yields a large gain.

  • Visualization: Pool5 units capture both semantic concepts, such as people and text, and texture or material properties, such as dot arrays and specular reflections.The visualizations show representative units firing on dog faces, red blobs, human faces, and abstract patterns.
  • Ablation: 29% of CNN parameters, or about 16.8 million, can be removed by dropping fc7 without degrading mAP when the network is pretrained only on ILSVRC 2012.Features from fc7 generalize worse than fc6, while removing both fully connected layers still produces good results using pool5 features.
  • Ablation: 8.0 percentage points: fine-tuning on VOC 2007 raises mAP to 54.2%, with larger gains for fc6 and fc7 than for pool5.This suggests ImageNet-trained pool5 features are general, while much of the improvement comes from domain-specific nonlinear classifiers.
  • Architecture: 66.0% mAP: replacing T-Net with the 16-layer O-Net increases R-CNN performance from 58.5% to 66.0%.O-Net was fine-tuned with the same protocol, using smaller minibatches to fit GPU memory.
  • Modes of error: 3 to 4 mAP points: bounding-box regression reduces localization errors by predicting a new detection window from pool5 features for each selective-search proposal.The method is a linear regression model inspired by bounding-box regression in DPM.

4. The ILSVRC2013 detection dataset

The ILSVRC2013 detection dataset uses distinct train, validation, and test distributions, requiring careful choices about training data and evaluation splits. R-CNN therefore relies heavily on a balanced val1/val2 split, supplements val1 with selected train positives, and evaluates data-use choices through ablation.

  • Dataset splits: ILSVRC2013 detection is split into train (395,918), val (20,121), and test (40,152) images.The counts are given in parentheses for each split.
  • Dataset splits: The val and test sets share a scene-like distribution and exhaustive bounding-box annotations for all instances from 200 classes, unlike train.Train images come from the classification distribution and have more variable complexity, while val and test resemble PASCAL VOC images.
  • Training-data choices: Because train annotations are not exhaustive and its distribution differs from val and test, the authors rely heavily on val and use train only as an auxiliary source of positives.The split characteristics make train unsuitable for hard negative mining.
  • Validation split: 11% maximum relative imbalance and 4% median relative imbalance characterize the selected val1/val2 split.The split was chosen from candidate partitions generated by clustering val images by class counts and applying randomized local search.
  • Training-data choices: R-CNN training data combines all selective-search and ground-truth boxes from val1 with up to N train ground-truth boxes per class, forming val1+trainN.The ablation evaluates N ∈ {0, 500, 1000} using mAP on val2.
  • Ablation study: 20.9% mAP is the initial ablation result using an ILSVRC2012-pretrained CNN without fine-tuning and with the small val1 training set.The ablation also reports that val2 mAP closely matches test mAP, supporting val2 as an indicator of test performance.

5. Semantic segmentation

The paper applies R-CNN to semantic segmentation using CNN features computed on CPMC regions, comparing full, foreground-masked, and combined feature strategies. On VOC 2011, full+fg achieves 47.9% average accuracy, while the CNN system outperforms R&P and roughly matches O2P without fine-tuning.

  • Method: R-CNN is evaluated for semantic segmentation within O2P’s open-source framework using CPMC region proposals.O2P generates 150 region proposals per image and predicts each region’s class-specific quality with SVR.
  • CNN features for segmentation: Three feature strategies warp each region’s bounding window to 227 × 227: full, foreground-mask-only (fg), and their combination (full+fg).The full strategy ignores region shape, fg replaces background with the mean input, and full+fg combines both representations.
  • Results on VOC 2011: 47.9% average accuracy is achieved by full+fg on the VOC 2011 validation set, the best result among the evaluated strategies.Within each strategy, fc6 outperforms fc7; fg slightly outperforms full, indicating that masked region shape provides a stronger signal.
  • Results on VOC 2011: Without fine-tuning, the CNN achieves top segmentation performance on the VOC 2011 test set, outperforming R&P and roughly matching O2P.The comparison uses the Regions and Parts (R&P) and second-order pooling (O2P) methods as strong baselines.

6. Conclusion · Appendix

The paper presents a simple, scalable detector that improves prior PASCAL VOC 2012 results by 30% relative, combining CNNs with bottom-up region proposals. Its training strategy uses supervised pre-training on an auxiliary classification task before fine-tuning for detection when labeled target data is scarce.

  • 6. Conclusion: 30% relative improvement over the best previous results was achieved on PASCAL VOC 2012.
  • 6. Conclusion: The method addresses stagnating object detection performance with a simple and scalable algorithm.Earlier leading systems were complex ensembles combining low-level image features with high-level context.
  • 6. Conclusion: High-capacity convolutional neural networks are applied to bottom-up region proposals to localize and segment objects.
  • 6. Conclusion: Supervised pre-training on an auxiliary task with abundant data is followed by fine-tuning for a target task with scarce labeled data.The auxiliary task is image classification, while the target task is detection.
  • 6. Conclusion: The results demonstrate that classical computer vision tools and deep learning are natural partners rather than opposing approaches.The cited combination is bottom-up region proposals with convolutional neural networks.
  • 6. Conclusion: The research received support from DARPA, NSF, MURI, and Toyota, while NVIDIA donated the GPUs used.

A. Object proposal transformations

Object proposals are converted from arbitrary image rectangles into the CNN’s fixed 227 × 227-pixel input using square-based or warp transformations. Each transformation can include configurable surrounding context, with out-of-image regions replaced by the image mean.

  • Input transformation: The CNN requires object proposals, represented as arbitrary image rectangles, to be transformed into fixed-size 227 × 227-pixel inputs.Two transformation approaches were evaluated for producing valid CNN inputs.
  • Square transformations: The tightest-square transformations isotropically scale proposals, either retaining surrounding context or excluding image content outside the original proposal.These variants correspond to the tightest square with context and without context.
  • Warp transformation: The warp transformation anisotropically scales each object proposal directly to the CNN input size.This transformation is illustrated in Figure 7 column (D).
  • Context padding: Context padding p adds a border around each proposal in transformed-input coordinates, with Figure 7 illustrating p = 0 and p = 16 pixels.When the source rectangle extends beyond the image, missing data is replaced with the image mean.

B. Positive vs. negative examples and softmax

R-CNN uses different positive/negative definitions for CNN fine-tuning and SVM training, with fine-tuning positives including proposals overlapping ground truth by at least 0.5. Replacing post-fine-tuning SVMs with the network’s 21-way softmax reduced VOC 2007 performance from 54.2% to 50.9% mAP, though the gap may be closable with further fine-tuning changes.

  • Positive and negative definitions: Fine-tuning labels proposals matched to a ground-truth instance with IoU at least 0.5 as positives, while all other proposals are background negatives.The matched ground-truth class determines the positive label.
  • Positive and negative definitions: SVM training uses a different labeling scheme because it was established before CNN fine-tuning and empirically performed best among evaluated alternatives.The authors initially trained SVMs on features from an ImageNet-pre-trained CNN.
  • Positive and negative definitions: Approximately 30x more positive examples arise from jittered proposals with overlap between 0.5 and 1, which the authors conjecture helps prevent overfitting when fine-tuning the entire network.The authors attribute this design primarily to limited fine-tuning data.
  • Softmax versus SVMs: 54.2% to 50.9% mAP: using the fine-tuned network’s 21-way softmax detector instead of SVMs reduced VOC 2007 performance.The proposed explanation includes fine-tuning positives that do not emphasize precise localization.
  • Softmax versus SVMs: Additional fine-tuning tweaks might close the remaining performance gap, simplifying and speeding R-CNN training without sacrificing detection performance.The authors present this as a conjecture rather than an established result.

C. Bounding-box regression

R-CNN improves localization with a class-specific bounding-box regressor applied after proposal scoring, using CNN features to predict box transformations. The method uses regularized least squares with selective training-pair criteria, and iterative refinement does not improve results.

  • C. Bounding-box regression: A class-specific bounding-box regressor predicts a new detection box after each selective-search proposal is scored by a class-specific detection SVM.The regressor uses CNN-computed features rather than geometric features, distinguishing it from bounding-box regression in deformable part models.
  • C. Bounding-box regression: The transformation predicts scale-invariant center translations and log-space width and height translations from each proposal.The predicted coordinates are obtained by translating the proposal center and exponentiating the predicted width and height offsets.
  • C. Bounding-box regression: The four transformation functions are linear models of pool5 CNN features, learned with regularized least squares and solved efficiently in closed form.Each function has learnable parameters and is optimized using ridge regression.
  • C. Bounding-box regression: λ = 1000 was selected using a validation set, and training excludes proposals too far from all ground-truth boxes.The authors identify regularization and training-pair selection as important implementation issues.
  • C. Bounding-box regression: Iterating box prediction and rescoring does not improve results, so each proposal is refined only once at test time.The system predicts a new detection window once after scoring each proposal.

D. Additional feature visualizations

The section visualizes how selected pool5 units respond across VOC 2007 test region proposals and identifies each unit by its feature-map position and channel.

  • D. Additional feature visualizations: 24 region proposals maximally activate each of 20 visualized pool5 units out of approximately 10 million VOC 2007 test regions.Figure 12 presents these top-activating proposals for each unit.
  • D. Additional feature visualizations: Each pool5 unit is labeled by its (y, x, channel) position in the 6 × 6 × 256 feature map.Within a channel, changing (y, x) changes only the receptive field, while the CNN computes the same input-region function.

E. Per-category segmentation results

Table 7 reports per-category segmentation accuracy on the VOC 2011 validation set for six proposed methods and O2P, identifying the strongest methods across 20 PASCAL classes and background.

  • E. Per-category segmentation results: Table 7 compares per-category segmentation accuracy on VOC 2011 val across six segmentation methods and O2P.The comparison covers each of the 20 PASCAL classes plus the background class.

F. Analysis of cross-dataset redundancy · G. Document changelog

The paper investigates cross-dataset redundancy between PASCAL test images and ILSVRC 2012 training data, finding limited overlap, while the changelog records major methodological, evaluation, correction, and architecture updates.

  • F. Analysis of cross-dataset redundancy: The analysis checks whether auxiliary-dataset training could introduce redundancy between PASCAL test images and ILSVRC 2012 training and validation images.The authors note that object detection and whole-image classification are substantially different tasks, reducing concern about cross-set redundancy.
  • F. Analysis of cross-dataset redundancy: 0.63% of images matched by exact Flickr image IDs, comprising 31 matches out of 4952.The check used IDs available in VOC 2007 test annotations; later PASCAL test-set IDs were intentionally secret.
  • F. Analysis of cross-dataset redundancy: GIST descriptor matching compared warped 32 × 32 pixel versions of all ILSVRC 2012 trainval and PASCAL 2007 test images.The method followed prior work demonstrating strong near-duplicate detection performance in collections larger than 1 million images.
  • F. Analysis of cross-dataset redundancy: 38 near-duplicate images were found, including all 31 Flickr-ID matches, showing overlap below 1% for the examined sets.Differences were mainly in JPEG compression and resolution, with smaller effects from cropping.
  • G. Document changelog: v2 improved detection performance through a 0.001 rather than 0.0001 fine-tuning learning rate, context padding, and bounding-box regression.The revision was the CVPR 2014 camera-ready version and describes bounding-box regression as correcting localization errors.
  • G. Document changelog: v3 added ILSVRC2013 detection results and an OverFeat comparison to several sections, primarily Sections 2 and 4.These additions documented evaluation on the ILSVRC2013 detection dataset.
  • G. Document changelog: v4 corrected an error in Appendix B’s softmax-versus-SVM results, and v5 added results from Simonyan and Zisserman’s 16-layer network to Section 3.3 and Table 3.The v4 revision acknowledged assistance in identifying the error; v5 incorporated the newer architecture results.
Loading 1311.2524v5…