PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Pyramid Networks for Object Detection

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie

arXiv:1612.03144v2cs.CV

TL;DR

Object detection needs multi-scale representations, but image pyramids impose substantial inference-time and memory costs. FPN reuses a ConvNet’s pyramidal hierarchy, combining top-down and lateral pathways to create semantically strong features at all scales. It improves detection and segmentation systems, achieving state-of-the-art single-model COCO results with a basic Faster R-CNN and without increasing testing time over the single-scale baseline.

  • Problem

    Image pyramids improve multi-scale recognition but considerably increase inference time, make end-to-end training memory-infeasible, and create train/test inconsistency.

  • Method

    FPN builds an in-network feature pyramid by combining low-resolution semantic features with high-resolution localized features through top-down and lateral connections.

  • Results

    FPN achieves state-of-the-art single-model COCO detection results with a basic Faster R-CNN and improves Average Recall by 8.0 points, COCO-style AP by 2.3 points, and PASCAL-style AP by 3.8 points over a strong single-scale baseline.

  • Takeaways & Limitations

    FPN provides a practical feature-pyramid solution for research and applications without computing image pyramids, while retaining multi-scale representational power.

  • Takeaways & Limitations

    The reported system omits several complementary improvements, including iterative regression, hard negative mining, context modeling, and stronger data augmentation.

Abstract

from arXiv · show

Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

1. Introduction

Detecting objects across large scale changes requires multi-scale feature pyramids, but image pyramids are costly in inference time and memory. FPN reuses a ConvNet’s hierarchy with top-down and lateral connections to provide semantically strong features at every scale.

  • Motivation: Feature pyramids support scale-invariant detection by scanning across positions and pyramid levels.An object’s scale change is offset by shifting its level in the pyramid.
  • Limitations of image pyramids: Featurizing every image-pyramid level increases inference time considerably and makes end-to-end training memory-infeasible.These limitations create inconsistency between training and testing, leading Fast and Faster R-CNN to omit image pyramids by default.
  • Existing alternatives: ConvNets provide an inherent multi-scale hierarchy, but high-resolution maps have weaker semantics that harm object-recognition capacity.SSD-style approaches reuse this hierarchy incompletely by starting the pyramid high in the network and adding new layers.
  • FPN: FPN combines low-resolution strong features with high-resolution weak features through a top-down pathway and lateral connections.The resulting pyramid is built from a single input scale and supplies rich semantics at all levels.
  • FPN: FPN makes predictions independently at every pyramid level, unlike related top-down architectures that predict only on the finest level.This design echoes a featurized image pyramid while retaining the architecture’s multi-scale structure.
  • Results: 8.0 points higher Average Recall, 2.3 points higher COCO-style AP, and 3.8 points higher PASCAL-style AP were obtained over a strong single-scale Faster R-CNN baseline.The method also achieved a state-of-the-art single-model COCO result using a basic Faster R-CNN detector.

2. Related Work

Prior work explored image pyramids, multi-layer features, and lateral connections for recognition, detection, and segmentation. FPN differs by using a pyramidal architecture as a feature pyramid with independent predictions at all levels.

  • Image pyramids: Hand-engineered HOG and SIFT features were commonly computed densely over image pyramids for classification, detection, and pose estimation.Earlier systems also investigated faster ways to compute featurized image pyramids.
  • Deep ConvNet detectors: Deep ConvNet detectors improved accuracy using strategies including image-pyramid sliding windows, scale-normalized region classification, and spatial pyramid pooling.These approaches established deep features as a major alternative to hand-engineered representations.
  • Multiple layers: Methods such as FCN, Hypercolumns, HyperNet, ParseNet, and ION combine features from multiple ConvNet layers for segmentation or detection.Their combinations include summing partial scores or concatenating transformed multi-layer features.
  • Lateral and skip connections: U-Net, SharpMask, Recombinator, Hourglass, and Laplacian-pyramid methods use lateral or skip connections but generally produce a single prediction feature map.FPN instead makes predictions independently at all pyramid levels, matching the role of a featurized image pyramid.

3. Feature Pyramid Networks

FPN builds a multi-scale feature pyramid inside a ConvNet by enriching backbone stage outputs through top-down upsampling and lateral connections. The construction is fully convolutional, supports multiple backbone architectures, and is adapted to detection systems with minimal changes.

  • Architecture: FPN uses a bottom-up backbone hierarchy, a top-down pathway, and lateral connections to construct feature maps at multiple levels.The method is general-purpose and is applied to region proposal, object detection, and instance-segmentation proposals.
  • Implementation: FPN takes a single-scale arbitrary-size image and outputs proportionally sized feature maps at multiple levels in a fully convolutional fashion.The construction is independent of the backbone architecture, and the paper reports experiments using ResNets.
  • Bottom-up pathway: The bottom-up pathway supplies one pyramid level per backbone stage, using each stage’s deepest output because it has the strongest features.The hierarchy contains feature maps at several scales with a scaling step of 2.
  • Top-down pathway: The top-down pathway upsamples semantically strong coarse maps and merges them with same-size bottom-up maps through lateral connections.Bottom-up maps provide more accurate localization because they undergo fewer subsampling operations.
  • Feature construction: Each merged map is produced by element-wise addition after 2× nearest-neighbor upsampling and a 1×1 lateral convolution.A final 3×3 convolution generates the feature map and reduces aliasing from upsampling.
  • Applications: The method uses simple design choices and requires only minimal modifications when adapted to RPN and Fast R-CNN.More sophisticated connection blocks produced only marginally better results and were outside the paper’s focus.

4. Applications

The paper adapts FPN to RPN and Fast R-CNN with minimal changes, using pyramid-level heads and scale-aware feature assignment while sharing parameters across levels.

  • Integration: Both RPN and Fast R-CNN can be trained and tested on the feature pyramid using the original systems’ procedures with minimal architectural adaptations.P6 supports the largest RPN anchor scale but is not used by Fast R-CNN.
  • RPN: FPN replaces RPN’s single-scale feature map with a pyramid and attaches the same 3×3-convolutional, dual-1×1-convolution head to every level.Dense evaluation across all pyramid levels removes the need for multi-scale anchors on each individual level.
  • RPN: RPN assigns anchor areas of {32^2, 64^2, 128^2, 256^2, 512^2} pixels to P2 through P6, with three aspect ratios per level.This yields 15 anchors across the pyramid.
  • RPN: RPN labels anchors positive when they have the highest IoU for a ground-truth box or IoU above 0.7, and negative when all IoUs are below 0.3.Ground-truth scale is not explicitly used to assign boxes to pyramid levels.
  • Fast R-CNN: Fast R-CNN assigns each RoI to a pyramid level using its width and height relative to a canonical 224^2 input-region scale, with k0 = 4.An RoI half the canonical scale is mapped to a finer-resolution level, such as k = 3.
  • Fast R-CNN: Fast R-CNN applies shared class-specific classification and bounding-box regression heads to RoIs at every pyramid level after 7×7 RoI pooling.The method uses two hidden 1,024-dimensional fully connected layers rather than the ResNet conv5 head described in the comparison system.

5. Experiments on Object Detection

Experiments show that FPN improves object proposals and detection across RPN, Fast R-CNN, and Faster R-CNN while retaining practical efficiency. Its gains arise from combining multi-scale pyramid levels with top-down and lateral connections, and it surpasses strong single-model COCO competitors without extensive engineering.

  • RPN ablations: 56.3 AR1k is achieved by FPN in RPN, an 8.0-point increase over the single-scale baseline.Small-object AR1k also increases by 12.9 points.
  • RPN ablations: Removing top-down or lateral connections produces inferior proposal results, with lateral connections adding 10 AR1k points over the top-down-only variant.The top-down ablation suggests that high-resolution maps otherwise retain harmful low-level features, while lateral connections provide more precise locations.
  • Fast R-CNN: FPN improves Fast R-CNN AP by 2.0 points over the standard baseline and by 5.1 points over a comparable two-fully-connected-layer baseline.Small-object AP improves by 2.1 points over the standard baseline.
  • Pyramid ablations: Using only the finest P2 feature map reaches 33.4 AP versus 33.9 AP with all pyramid levels, while its 750k anchors do not by themselves improve accuracy.For sliding-window RPN, scanning pyramid levels improves robustness to scale variation beyond simply increasing anchor count.
  • Faster R-CNN: Under controlled Faster R-CNN settings, FPN improves AP by 2.3 points and AP@0.5 by 3.8 points over a strong baseline.The FPN-based system runs at 0.148 seconds per image with ResNet-50, compared with 0.32 seconds for the single-scale baseline.
  • COCO benchmark: On COCO test-dev, FPN reaches 36.2 AP and 59.1 AP@0.5, exceeding existing best results by 0.5 and 3.4 points respectively.The single-model result uses one input image scale, does not rely on image pyramids, and surpasses heavily engineered competition winners without many common enhancements.

6. Extensions: Segmentation Proposals

FPN replaces costly image pyramids for multi-scale segmentation proposals by predicting masks from a feature pyramid. It improves proposal accuracy, especially for small objects, while running substantially faster than prior methods.

  • FPN generates segmentation proposals fully convolutionally from a feature pyramid instead of densely sampled image pyramids.The approach extends FPN beyond object detection to mask proposal generation.
  • 45.7 AR is achieved by combining 5×5 and 7×7 MLPs, compared with 43.4 AR for the single-MLP baseline.Increasing the mask output size to 28×28 adds another point, and doubling training iterations raises AR to 48.1.
  • Over 8.3 points AR improvement is reported over DeepMask, SharpMask, and InstanceFCN, with nearly doubled accuracy on small objects.
  • 6 to 7 FPS makes FPN substantially faster than computationally expensive image-pyramid-based mask proposal methods.The results support using FPN as a generic feature extractor for other multi-scale detection problems.

7. Conclusion

The paper presents a simple in-network framework for feature pyramids that improves strong baselines and competition winners without computing image pyramids. It concludes that explicit pyramid representations remain important for multi-scale recognition despite deep ConvNets' scale robustness.

  • FPN builds feature pyramids inside ConvNets and provides a practical solution without computing image pyramids.
  • The study reports significant improvements over strong baselines and competition winners.
  • The findings suggest that explicit pyramid representations remain critical for multi-scale problems despite deep ConvNets' representational power and scale robustness.

A. Implementation of Segmentation Proposals

The segmentation-proposal implementation assigns object scales to feature-pyramid levels and predicts masks and scores fully convolutionally. It uses separate MLP input sizes to cover canonical and half-octave scales.

  • The feature pyramid uses levels P2–P6, mapping mask scales {32, 64, 128, 256, 512} pixels to those levels.Objects at intermediate scales are assigned to the nearest scale in log space.
  • A 5×5 MLP handles canonical scales, while a 7×7 MLP handles half-octave scales.The half-octave branch follows the use of half-octave image pyramids in DeepMask and SharpMask.
  • Each pyramid level predicts masks at locations whose centers fall within 2^k pixels of the corresponding feature-map position.Locations without an object center train only the score branch, not the mask branch.
  • The prediction head applies a 5×5 kernel with 512 outputs, followed by fully convolutional 1×1-convolution replacements for fully connected mask and score layers.The standard branch outputs a 14×14 mask and one object score.
  • Training samples 2048 examples per mini-batch, with 128 examples per image from 16 images.
Loading 1612.03144v2…