PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Detection Networks on Convolutional Feature Maps

Shaoqing Ren, Kaiming He, Ross Girshick, Xiangyu Zhang, Jian Sun

arXiv:1504.06066v2cs.CV

TL;DR

Object detection has advanced its shared feature extractors more than its region-wise classifiers, which are often simple MLPs. This paper introduces NoCs—region-wise classifier networks on shared convolutional features—and studies MLP, convolutional, and maxout designs. It finds that deep, convolutional NoCs are essential for strong detection, including when using ResNets with Faster R-CNN.

  • Problem

    The paper addresses limited attention to object-classifier design despite rapid progress in deep convolutional feature extractors and widespread use of simple region-wise MLPs.

  • Method

    The authors investigate region-wise NoC architectures on shared, region-independent convolutional features, including MLPs, ConvNets, and maxout ConvNets.

  • Results

    Deep region-wise classifiers improve detection accuracy, and a deep, convolutional NoC is essential for outstanding performance with Faster R-CNN and fully convolutional backbones.

  • Takeaways & Limitations

    Region-wise classifier design is an essential element of high-performing object detectors, alongside shared features, Faster R-CNN, and deep backbones.

  • Takeaways & Limitations

    The design challenge is especially relevant to fully convolutional networks without hidden fully connected layers, while the ablations use frozen features, precomputed proposals, and post-hoc SVMs.

Abstract

from arXiv · show

Most object detectors contain two important components: a feature extractor and an object classifier. The feature extractor has rapidly evolved with significant research efforts leading to better deep convolutional architectures. The object classifier, however, has not received much attention and many recent systems (like SPPnet and Fast/Faster R-CNN) use simple multi-layer perceptrons. This paper demonstrates that carefully designing deep networks for object classification is just as important. We experiment with region-wise classifier networks that use shared, region-independent convolutional features. We call them "Networks on Convolutional feature maps" (NoCs). We discover that aside from deep feature maps, a deep and convolutional per-region classifier is of particular importance for object detection, whereas latest superior image classification models (such as ResNets and GoogLeNets) do not directly lead to good detection accuracy without using such a per-region classifier. We show by experiments that despite the effective ResNets and Faster R-CNN systems, the design of NoCs is an essential element for the 1st-place winning entries in ImageNet and MS COCO challenges 2015.

1 INTRODUCTION

Object detectors divide work between shared feature extraction and object classification, but classifier design has received less attention. The paper introduces NoCs to study region-wise classifiers on shared convolutional features and finds that carefully designed deep classifiers improve detection.

  • Motivation: Traditional detectors combine a feature extractor with an object classifier, while recent convolutional systems commonly use shared features followed by region-wise MLPs.Earlier systems used hand-engineered features with linear, boosted, or kernel classifiers; SPPnet and its successors shared convolutional features across regions.
  • NoC framework: The paper defines NoCs as region-wise classifier architectures built on shared, region-independent convolutional features.The study examines classifier design separately from feature design to provide insights into object detection systems.
  • Main observation: Carefully designed region-wise classifiers improve detection accuracy over typical MLP classifiers.The paper studies MLPs, ConvNets, and ConvNets with maxout for latent scale selection.
  • Main observation: A deep region-wise classifier is important for detection accuracy in addition to deep shared features, and region-wise convolutional layers complement shared-image convolutional features.These conclusions are supported by ablation experiments and results on MS COCO.
  • Implications: Superior image-classification backbones such as ResNets and GoogLeNets do not directly produce better detection accuracy without a deep, convolutional NoC.The paper presents a way to integrate fully convolutional image classifiers into Faster R-CNN.

2 RELATED WORK

Prior detection systems progressively improved shared convolutional features and region-wise classification, but retained MLP classifiers. This design fits networks with hidden fully connected layers and is nontrivial to apply effectively to fully convolutional models.

  • Traditional and ConvNet-based detection: Traditional object detectors paired hand-crafted features with boosted or SVM-based classifiers, while ConvNet-based methods introduced convolutional feature maps for detection.Examples include Haar-like features with boosted classifiers, HOG with linear SVMs, and DPMs with latent SVMs.
  • SPPnet and successors: SPPnet pooled proposal features from shared convolutional maps and classified regions with an MLP, forming a bridge between hybrid detectors and R-CNN.Fast R-CNN and Faster R-CNN developed this shared-feature and region-wise-classifier strategy further.
  • SPPnet and successors: Fast R-CNN and Faster R-CNN inherited a logical division of shared convolutional features and region-wise MLP classifiers.Faster R-CNN also reused shared features for region proposals, reducing proposal-related computational burden.
  • Architectural scope: These systems naturally support AlexNet-, ZF-, OverFeat-, and VGG-style networks with multiple fine-tunable fully connected layers.Their common classifier design is tied to the architectures used in those systems.
  • Architectural scope: Fast/Faster R-CNN do not achieve good accuracy straightforwardly with fully convolutional networks such as ResNet and GoogleNet, which lack hidden fully connected layers.The paper identifies this compatibility issue as a motivation for designing NoCs.

3 ABLATION EXPERIMENTS

The ablation study isolates NoC classifier effects by using frozen shared feature maps and precomputed proposals, then trains and evaluates varied classifiers on RoI-pooled regions. NoCs treat fixed-resolution region features as new inputs for learned classification networks.

  • Experimental setup: The ablations use frozen shared feature maps, Selective Search proposals, and post-hoc SVMs to focus on classifier effects.The authors state that the observations generally remain valid when these restrictions are relaxed in Faster R-CNN.
  • Experimental setup: Experiments use PASCAL VOC 2007, covering 20 categories and evaluating mAP on a 5k-image test set.Training uses either the original 5k-image trainval set or an augmented 16k-image set including VOC 2012 trainval images.
  • Feature extraction: The study uses ImageNet-pretrained ZF and VGG models to generate shared convolutional feature maps, which remain frozen during these experiments.The models provide different convolutional depths and fixed-resolution feature-map inputs for NoC analysis.
  • Region features: Approximately 2,000 Selective Search regions are RoI-pooled into fixed-resolution m × m feature maps that function as small multichannel images.The pooled resolution replaces the pre-trained model’s final pooling-layer resolution for each region.
  • NoC training and inference: NoCs classify RoI-pooled features through multilayer networks ending in an (n+1)-way softmax, with training by backpropagation and SGD.The second-to-last fully connected layer supplies region features for category-specific linear SVMs during comparison.
  • NoC training and inference: During inference, RoI-pooled features pass through the NoC to the second-to-last fully connected layer, after which SVM scores are followed by non-maximum suppression.This pipeline scores individual regions and removes overlapping detections.

3.1 Using MLP as NoC

The paper first studies MLPs as NoCs by varying their depth while keeping the RoI features fixed. Deeper MLP classifiers improve detection accuracy over a single-layer SVM baseline.

  • MLP design: A simple NoC uses only fully connected layers, forming an MLP with 2 to 4 layers.Hidden layers are 4,096-dimensional with ReLU, while the final layer predicts object categories plus background.
  • Results: 7.8% higher mAP is achieved by the 4-fc NoC than by an SVM on the same RoI features.The NoC weights are randomly initialized, so the reported gain is attributed to the MLP classifier rather than pre-training.
  • Results: NoC accuracy increases as the MLP depth grows from 2 to 4 fully connected layers.The 3-fc configuration resembles the region-wise classifiers used in SPPnet and Fast/Faster R-CNN.

3.2 Using ConvNet as NoC

The paper adds region-aware convolutional layers to NoC classifiers, addressing the limitation of shared features that ignore regions of interest. These layers improve detection when sufficient training data is available, but deeper models can overfit small datasets.

  • Motivation: Shared convolutional features are region-independent, so NoCs add convolutional layers that learn features tailored to regions of interest.The added layers operate within the region-wise classifier rather than the full-image shared feature extractor.
  • Architecture: The evaluated ConvNet NoCs add one to three 3×3 convolutional layers before three fully connected classifier layers.The spatial resolution remains m × m, with 256 filters for ZF and 512 for VGG.
  • Results: On VOC 07 trainval, additional convolutional layers do not improve mAP and deeper models degrade because the training set is too small and overfitting occurs.The comparison uses randomly initialized NoCs without bounding-box regression.
  • Results: 56.5% baseline mAP rises to 58.9% with the advanced 2conv3fc NoC when trained on VOC 07+12 trainval.The reported gain supports using additional convolutional layers with the augmented training set.
  • Implication: ConvNet NoCs are also reported as central for Faster R-CNN with ResNets and other fully convolutional pretrained architectures.The paper presents the design as applicable beyond the ZF and VGG experiments.

3.3 Maxout for Scale Selection

The paper uses maxout to combine adjacent feature-pyramid scales within NoCs, improving scale selection. Maxout variants outperform their non-maxout counterparts, while their placement has only marginal effect.

  • Scale selection: For each proposal region, two adjacent feature-pyramid scales are RoI-pooled and supplied as competing NoC inputs.This design targets improved scale invariance.
  • Maxout design: Maxout merges the two scale-specific feature maps by an element-wise maximum while sharing corresponding pathway weights.The merged feature retains the same dimensionality, and the total number of weights is unchanged.
  • Results: Maxout NoCs improve mAP over non-maxout counterparts by up to 1.8% mAP.The gains occur across all tested maxout placements.
  • Results: Differences among the four maxout variants are marginal.The main reported benefit is associated with using maxout rather than with a particular insertion point.

3.4 Fine-tuning NoC

The paper compares randomly initialized and fine-tuned NoCs to assess how classifier initialization affects detection. Fine-tuning improves results, but randomly initialized deep NoCs already obtain most of the classifier-side accuracy gain.

  • Experimental setup: The study compares random initialization with transferring pretrained weights and fine-tuning the NoC layers.The comparison is intended to examine the impact of the established fine-tuning strategy on classifiers.
  • Initialization: Fine-tuning the two fully connected layers makes the 3-fc NoC equivalent to the SPPnet object detection system.Additional convolutional layers are initialized to identity mappings in the fine-tuned versions.
  • Results: 60.7% mAP from a randomly initialized NoC exceeds the 47.7% SVM counterpart by 13.0% using the same RoI-pooled features.The comparison isolates the classifier design while keeping the RoI-pooled features consistent.
  • Results: 62.9% mAP from the fine-tuned NoC adds 2.2% over the randomly initialized NoC.The result indicates that training a deep classifier on detection data supplies most of the classifier-side accuracy.

3.5 Deep Features vs. Deep Classifiers

The experiments separate the contributions of shared features and region-wise classifiers. Both depths matter, but deeper region-wise classifiers produce the larger reported mAP gain and substantially reduce localization error.

  • Deep features: Increasing shared-feature depth from VGG-16 to VGG-19 raises mAP from 64.6% to 65.1%, a 0.5% increase.The classifier remains the baseline 3fc NoC in this comparison.
  • Deep classifiers: Increasing region-aware classifier depth with VGG-16 features raises mAP from 64.6% to 66.1%, a 1.5% increase.This gain is larger than the reported gain from increasing shared-feature depth.
  • Overall result: The best VGG-16 NoC reaches 68.8% mAP.The paper analyzes feature and classifier effects separately using ZF versus VGG-16 features and 3fc versus maxout 2conv3fc NoCs.
  • Error analysis: With ZF features, the stronger maxout 2conv3fc NoC reduces localization error from 28.1% to 22.6%.The comparison uses the same ZF feature set and contrasts the stronger NoC with the 3fc baseline.
  • Error analysis: With VGG-16 features, the stronger NoC reduces localization error from 24.8% to 20.1%.The paper attributes the stronger NoC’s main effect to localization, while VGG-16 generally has lower recognition error than ZF with the same classifier.

3.7 Comparisons of Results

The method achieves 71.6% mAP on PASCAL VOC 2007, exceeding Fast R-CNN with Selective Search proposals but remaining below Faster R-CNN with learned proposals. NoC effects are orthogonal to the 3-fc region-wise classifiers used by Fast/Faster R-CNN.

  • 71.6% mAP is achieved on the PASCAL VOC 2007 test set.
  • The method outperforms Fast R-CNN using Selective Search proposals but underperforms Faster R-CNN using learned proposals.The compared systems otherwise differ in proposal generation as described for these results.
  • NoC effects are orthogonal to the 3-fc region-wise classifier structure used by Fast/Faster R-CNN.The paper notes that this effect is particularly prominent with ResNets.

3.8 Summary of Observations

The experiments identify two largely separate design dimensions for detection: region-wise classifier depth and classifier convolutionality. Deeper region-wise classifiers help independently of deeper feature maps, while convolutional classifiers outperform MLP-based classifiers, with support from MS COCO experiments.

  • A deeper region-wise classifier is useful and generally orthogonal to deeper feature maps.
  • A convolutional region-wise classifier is more effective than an MLP-based region-wise classifier.
  • The observations are strongly supported by experiments on the more challenging MS COCO dataset.

4 NOC FOR FASTER R-CNN WITH RESNET

For fully convolutional backbones in Faster R-CNN, NoC design is needed because these networks lack hidden fully connected layers for region-wise classification. Experiments compare VGG-16, GoogleNet, and ResNet-101 under Faster R-CNN settings and show that naive classifier choices can outweigh image-classification strength.

  • NoC design is essential for Faster R-CNN to achieve superior results with fully convolutional models such as GoogleNet and ResNet.These models have no hidden fully connected layers for constructing region-wise classifiers.
  • The MS COCO experiment trains on 80k images and evaluates on 40k validation images using COCO-style AP, AP@0.5, and AP@0.75.
  • The Faster R-CNN comparison covers VGG-16, GoogleNet, and ResNet-101.The reported ImageNet center-crop top-1 errors are 28.5% for VGG-16, 26.4% for reproduced GoogleNet, and 23.6% for ResNet-101.
  • The Faster R-CNN setup fine-tunes the full network end-to-end, learns proposals with an RPN, and jointly learns softmax classification and bounding-box regression.These changes do not affect the NoC design.
  • The naive Faster R-CNN configuration applies RoI pooling after the final convolutional layer and follows it with an 81-d classifier equivalent to a 1fc NoC.
  • 21.2% AP for VGG-16 exceeds 15.2% for GoogleNet and 16.9% for ResNet under naive Faster R-CNN.With stride reduced from 32 to 16, naive Faster R-CNN reaches 18.6% AP for GoogleNet and 21.3% for ResNet.
  • The paper attributes the naive comparison partly to VGG-16 using a 3fc NoC while GoogleNet and ResNet use a 1fc NoC.This supports the importance of deeper region-wise NoCs despite deeper feature maps.
Loading 1504.06066v2…