PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

arXiv:1506.01497v3cs.CV

TL;DR

Region proposal computation had become the test-time bottleneck in state-of-the-art object detection. Faster R-CNN introduces a shared-feature Region Proposal Network that unifies proposal generation with Fast R-CNN, achieving 5fps including all steps on a GPU while improving detection accuracy over Selective Search with Fast R-CNN.

  • Problem

    Region proposal computation had become a test-time bottleneck because it remained as costly as, or slower than, efficient detection networks.

  • Method

    Faster R-CNN unifies a fully convolutional Region Proposal Network with Fast R-CNN by sharing convolutional features.

  • Results

    5fps (including all steps) on a GPU, while RPN with Fast R-CNN produced detection accuracy better than Selective Search with Fast R-CNN.

  • Takeaways & Limitations

    The method provides a practical near-real-time object detection system with nearly cost-free proposals and improved proposal quality and overall detection accuracy.

  • Takeaways & Limitations

    The approximate joint-training procedure ignores gradients with respect to predicted box coordinates because differentiable RoI pooling is beyond the paper’s scope.

Abstract

from arXiv · show

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.

1 INTRODUCTION

The paper addresses region proposal computation as the remaining bottleneck in fast object detection by introducing RPNs that share convolutional features with detection networks. RPNs generate proposals efficiently through fully convolutional prediction and anchor boxes, while unified training enables strong accuracy with low test-time cost.

  • Motivation: 2 seconds per image is Selective Search’s CPU runtime, versus 0.2 seconds per image for EdgeBoxes, making proposal computation costly relative to efficient detection networks.These methods use inexpensive features and economical inference schemes, but remain slower than detection networks.
  • Region Proposal Networks: RPNs use a deep convolutional network to generate proposals while sharing convolutional layers with the detection network.At each regular-grid location, additional convolutional layers simultaneously predict region bounds and objectness scores.
  • Region Proposal Networks: Anchor boxes provide regression references across multiple scales and aspect ratios without enumerating image or filter pyramids.The scheme works with single-scale images, benefiting running speed.
  • Unified Training: An alternating fine-tuning scheme unifies RPNs with Fast R-CNN by sharing convolutional features between proposal generation and object detection.The scheme alternates between proposal-task and detection-task fine-tuning while keeping proposals fixed, and converges quickly.
  • Results: 10 milliseconds is the effective test-time running cost for proposals, while the complete detection method reaches 5fps on a GPU with very deep models.On PASCAL VOC, RPNs with Fast R-CNNs achieve better detection accuracy than Selective Search with Fast R-CNNs.
  • Results: Faster R-CNN and RPN formed the basis of several 1st-place entries across ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.RPNs learn to propose regions from data and can benefit from deeper, more expressive features such as 101-layer residual networks.

2 RELATED WORK

Prior object detectors commonly relied on external proposal modules, while deep-network approaches explored proposal generation, bounding-box prediction, and shared convolutional computation for efficiency. Faster R-CNN builds on these directions by using a fully convolutional proposal network within a unified detection architecture.

  • Object Proposals: Object proposal methods include super-pixel grouping and sliding-window approaches, often used as external modules independent of detectors.Examples include Selective Search, CPMC, MCG, objectness in windows, and EdgeBoxes.
  • Deep Networks for Object Detection: R-CNN classifies proposal regions as objects or background, but its accuracy depends on the region proposal module.Bounding-box regression refines proposals rather than replacing the proposal process.
  • Unified Architecture: Faster R-CNN uses a single unified network in which the RPN module serves as the network’s attention mechanism.The RPN identifies where the unified detector should look.
  • Deep Networks for Object Detection: MultiBox methods generate class-agnostic box proposals from networks predicting multiple boxes, using a single crop or multiple large image crops.These proposals are used for R-CNN, unlike Faster R-CNN’s fully convolutional proposal approach.
  • Shared Computation: Shared convolutional computation and adaptive pooling improve the efficiency of region-based object detection while retaining shared feature maps.Related systems include OverFeat, SPP, and Fast R-CNN.

3 FASTER R-CNN

Faster R-CNN is a unified object detection network comprising a fully convolutional region proposal module and a Fast R-CNN detector. The RPN guides the detector’s attention by telling it where to look, while both modules share features.

  • 3 FASTER R-CNN: Faster R-CNN combines a deep fully convolutional region proposal network with a Fast R-CNN detector in one unified object detection system.The RPN proposes regions that the Fast R-CNN module uses for detection.
  • 3 FASTER R-CNN: The RPN acts as an attention mechanism by telling the Fast R-CNN module where to look.The paper develops the region proposal network’s design and shared-feature training algorithms in Sections 3.1 and 3.2.

3.1 Region Proposal Networks

The Region Proposal Network is a fully convolutional network that maps an image to rectangular proposals with objectness scores while sharing convolutional layers with Fast R-CNN. It generates proposals by sliding a small network over shared convolutional features and jointly predicting box coordinates and object-versus-background scores.

  • Region Proposal Networks: An RPN takes an image of any size and outputs rectangular object proposals, each paired with an objectness score.Objectness measures membership in a set of object classes versus background.
  • Region Proposal Networks: The RPN slides a small network over the last shared convolutional feature map, producing a 256-d feature for ZF or 512-d for VGG before ReLU.The sliding network uses an n × n spatial window and feeds the resulting feature to sibling regression and classification layers.
  • Region Proposal Networks: At each sliding-window location, the network predicts k proposals with 4k box-regression outputs and 2k object-versus-background classification scores.The classification layer is implemented as a two-class softmax, though logistic regression can alternatively produce k scores.

Translation-Invariant Anchors

The anchor and proposal functions are translation invariant, so translated objects yield translated proposals through the same prediction function. This property also substantially reduces the proposal output layer’s parameter count compared with MultiBox.

  • Translation-Invariant Anchors: The method guarantees translation invariance for both anchors and the functions predicting proposals relative to them.If an object translates within an image, the proposal translates and the same function predicts it at either location.
  • Translation-Invariant Anchors: 2.8 × 10^4 parameters is the output-layer size for the method with k = 9 anchors, versus 6.1 × 10^6 for MultiBox.The method uses a (4 + 2) × 9-dimensional convolutional output layer, while MultiBox uses a (4 + 1) × 800-dimensional fully connected output layer.
  • Translation-Invariant Anchors: Translation invariance holds up to the network’s total stride, as in fully convolutional networks.This qualification specifies the spatial resolution at which the invariance is guaranteed.

Multi-Scale Anchors as Regression References

The RPN addresses multiple scales and aspect ratios with a pyramid of anchor boxes while computing features from a single-scale image. Anchors serve as regression references, enabling cost-efficient multi-scale prediction and feature sharing without extra scale-specific convolutional computation.

  • Multi-Scale Anchors: The anchor-based design replaces image/feature pyramids with a more cost-efficient pyramid of anchors spanning multiple scales and aspect ratios.Image/feature pyramids require resized images and feature maps at multiple scales, whereas anchors use a single image and feature-map scale.
  • Multi-Scale Anchors: Anchors let the RPN classify and regress bounding boxes relative to reference boxes using single-scale features and filters of one size.This design addresses multiple scales and sizes without computing separate feature maps for each image scale.
  • Feature Sharing: Multiscale anchors enable convolutional features from a single-scale image to be shared with Fast R-CNN without extra cost for scale handling.The paper identifies this feature-sharing property as a key component of the anchor design.
  • Anchor Regression: Bounding-box regression is formulated from each anchor box to a nearby ground-truth box, using four parameterized coordinates and a smooth L1 regression loss for positive anchors.Positive anchors are assigned from highest-overlap or IoU > 0.7 matches, while regression is inactive for negative anchors.

3.2 Sharing Features for RPN and Fast R-CNN

The section develops a unified RPN–Fast R-CNN network by sharing convolutional layers, addressing the fact that independently trained networks modify those layers differently. It considers alternating, approximate joint, and non-approximate joint training, adopting a pragmatic 4-step alternating procedure for experiments.

  • Motivation: RPN and Fast R-CNN must share convolutional layers because independent training modifies those layers differently.The proposed unified network replaces learning two separate networks.
  • Training strategies: Three feature-sharing strategies are described: alternating training, approximate joint training, and non-approximate joint training.Alternating training iteratively trains RPN and Fast R-CNN; approximate joint training combines shared-layer gradients from both losses.
  • 4-Step Alternating Training: The experiments use a pragmatic 4-step alternating algorithm to learn shared features via alternating optimization.The procedure trains RPN, trains a separate Fast R-CNN detector, initializes RPN from the detector while fixing shared layers, then fine-tunes Fast R-CNN’s unique layers.
  • Training strategies: Non-approximate joint training requires gradients with respect to RPN-predicted box coordinates because RoI pooling consumes both convolutional features and predicted boxes.Approximate joint training ignores these coordinate gradients.
  • 4-Step Alternating Training: The final networks share convolutional layers and form a unified network, while additional alternating iterations yield negligible improvements.The shared layers are fixed while the layers unique to RPN and Fast R-CNN are fine-tuned.

3.3 Implementation Details

The implementation uses single-scale 600-pixel inputs, multi-scale anchors, boundary-aware training, and NMS to produce efficient region proposals. These choices avoid image pyramids while reducing proposal redundancy before Fast R-CNN detection.

  • Image scaling: Images are resized so their shorter side is s = 600 pixels, using single-scale training and testing rather than an image pyramid.The last convolutional layer has a total stride of 16 pixels for both ZF and VGG nets.
  • Anchor configuration: Anchors use 3 box scales—128^2, 256^2, and 512^2 pixels—and 3 aspect ratios: 1:1, 1:2, and 2:1.These hyper-parameters are not carefully chosen for a particular dataset, and the method predicts multiple region scales without an image or filter pyramid.
  • Boundary handling: 6,000 anchors per image remain for training after cross-boundary anchors are ignored, from roughly 20,000 anchors in a typical 1000 × 600 image.Ignoring boundary-crossing anchors prevents their large, difficult-to-correct error terms from contributing to the loss.
  • Proposal processing: 0.7 IoU NMS reduces redundant RPN proposals to about 2,000 regions per image before selecting the top-N proposals for detection.NMS is applied using proposal classification scores and is reported not to harm ultimate detection accuracy.

4 EXPERIMENTS

Experiments show that RPN-based Faster R-CNN matches or exceeds proposal baselines with far fewer proposals and substantially lower runtime. Its gains persist across architectures and datasets, with feature sharing, box regression, classification scores, and stronger training data contributing to performance.

  • PASCAL VOC: RPN with Fast R-CNN reaches 59.9% mAP using up to 300 proposals, versus 58.7% for Selective Search and 58.6% for EdgeBoxes.Shared convolutional features and fewer proposals also make the RPN system faster.
  • RPN Ablations: Removing RPN classification scores reduces mAP from 55.8% with 1000 random proposals to 44.6% with 100, while removing regression yields 52.1% mAP.These ablations indicate that classification scores select accurate top proposals and regression produces high-quality box bounds.
  • VGG-16: 69.9% mAP is achieved with feature-shared RPN+VGG, exceeding the strong Selective Search baseline while providing nearly cost-free proposals.The unshared RPN+VGG variant reaches 68.5% mAP.
  • Runtime: 198ms is the total VGG-16 runtime for proposal generation and detection, compared with 1.5s average for Selective Search and 320ms for Fast R-CNN on 2000 proposals.With shared features, the RPN’s additional layers take only 10ms.
  • COCO and Deeper Features: On COCO test-dev, Faster R-CNN reaches 42.1% mAP@0.5 and 21.5% mAP@[.5, .95], improving over Fast R-CNN by 2.8% and 2.2%, respectively.Using ResNet-101 instead of VGG-16 further raises COCO results from 41.5%/21.2% to 48.4%/27.2%.
  • Cross-Dataset Training: 78.8% mAP is obtained on PASCAL VOC 2007 after fine-tuning a COCO-pretrained detection model, a 5.6% gain from the extra COCO data.Without VOC fine-tuning, the directly evaluated COCO model obtains 76.1% mAP.

5 CONCLUSION

Faster R-CNN introduces RPNs that share convolutional features with the downstream detector, making region proposals nearly cost-free while improving proposal quality and overall accuracy. This unified deep-learning system runs at near real-time frame rates.

  • 5 CONCLUSION: RPNs share convolutional features with the downstream detection network, making region proposal generation nearly cost-free.This integration enables a unified object detection system.
  • 5 CONCLUSION: Learned RPNs improve region proposal quality and overall object detection accuracy.
  • 5 CONCLUSION: The unified deep-learning-based object detection system runs at near real-time frame rates.
Loading 1506.01497v3…