PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Feature Pyramid Reconfiguration for Object Detection

Tao Kong, Fuchun Sun, Wenbing Huang, Huaping Liu

arXiv:1808.07993v1cs.CV

TL;DR

Object detectors need efficient feature pyramids that integrate semantic information across scales. The paper reformulates pyramid construction as nonlinear feature reconfiguration using global attention and local reconfiguration. In SSD, the method achieves state-of-the-art benchmark results without losing real-time processing speed.

  • Problem

    Feature pyramid designs remain inefficient at integrating semantic information across scales, while shallow SSD features lack deeper semantics for small-object detection.

  • Method

    The paper treats pyramid construction as nonlinear feature reconfiguration, combining global attention with local reconfiguration through lightweight, end-to-end trainable components.

  • Results

    The method achieves state-of-the-art results on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO without losing real-time processing speed.

  • Takeaways & Limitations

    Global-local reconfiguration provides a flexible way to spread strong semantic information across pyramid scales while preserving efficient detection.

  • Takeaways & Limitations

    The formulation assumes spatial consistency and shared learnable parameters across spatial locations, and includes a VGG-16-specific setting with P = 23.

Abstract

from arXiv · show

State-of-the-art object detectors usually learn multi-scale representations to get better results by employing feature pyramids. However, the current designs for feature pyramids are still inefficient to integrate the semantic information over different scales. In this paper, we begin by investigating current feature pyramids solutions, and then reformulate the feature pyramid construction as the feature reconfiguration process. Finally, we propose a novel reconfiguration architecture to combine low-level representations with high-level semantic features in a highly-nonlinear yet efficient way. In particular, our architecture which consists of global attention and local reconfigurations, is able to gather task-oriented features across different spatial locations and scales, globally and locally. Both the global attention and local reconfiguration are lightweight, in-place, and end-to-end trainable. Using this method in the basic SSD system, our models achieve consistent and significant boosts compared with the original model and its other variations, without losing real-time processing speed.

1 Introduction

Feature pyramids improve multi-scale object detection, but existing designs integrate semantics across scales inefficiently or too simply. The paper reframes pyramid construction as nonlinear feature reconfiguration using global attention and local transformations, improving SSD performance while retaining real-time speed.

  • Detecting objects at vastly different scales is a fundamental computer-vision challenge, motivating multi-scale feature pyramids.
  • SSD combines predictions from multiple resolutions but shallow-layer features lack deep semantics, limiting small-object detection.
  • FPN-style lateral connections strengthen shallow features through top-down semantic propagation, but their feature integration is equivalent to a linear combination.
  • The proposed architecture uses global attention followed by local reconfiguration to combine feature information across scales and spatial locations nonlinearly.Both components are designed as lightweight operations that can be embedded into ConvNets and trained end-to-end.
  • Experiments compare feature-pyramid strategies within SSD and show the proposed reconfiguration is more competitive in accuracy and efficiency.
  • The method achieves state-of-the-art results on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO without losing real-time processing speed.

2 Related work

Deep object detectors increasingly use feature pyramids to represent objects at multiple scales, moving from costly image pyramids toward reusable features within a single ConvNet. Recent lateral connections enrich shallow layers, while this paper proposes global-local integration across multiple layers.

  • Hand-crafted features such as HOG and SIFT used image pyramids to achieve scale-invariant extraction before deep convolutional networks became widespread.
  • Image-pyramid detection is time-consuming because features must be computed independently at each image scale and cannot be reused.
  • SSD combines predictions from multiple-resolution layers in one ConvNet, using default boxes at different scales to detect varied object sizes.
  • Lateral connections enrich shallow-layer semantics through a top-down pathway, whereas this paper integrates semantic knowledge from multiple layers globally and locally.

3 Method

The method reframes feature-pyramid construction as nonlinear feature reconfiguration, combining hierarchy information globally and locally to generate scale-specific detection features.

  • SSD and Existing Pyramids: SSD detects large instances with deeper layers and small instances with shallow, high-resolution layers, whose limited semantics constrain recognition.The method identifies missing reuse of deeper semantic information in shallow layers as SSD’s key bottleneck.
  • SSD and Existing Pyramids: FPN-like methods enrich shallow features by passing semantically strong, low-resolution features through top-down lateral connections.These connections enhance shallow-layer detection ability by transferring semantic information one level at a time.
  • Deep Feature Reconfiguration: The paper models each detector-level feature as a nonlinear transformation of the ConvNet feature hierarchy rather than a simple linear combination.The authors argue that detection features often lie on a nonlinear manifold, limiting linear reconfiguration power.
  • Deep Feature Reconfiguration: The proposed architecture first combines multiple feature maps, generates features at a target level, and then detects objects at multiple scales.Its global attention and local reconfiguration operations are lightweight, complementary, and trainable end-to-end within ConvNets.
  • Global Attention: Global attention emphasizes informative hierarchy features and suppresses less useful ones for a specific scale through channel-wise recalibration.The implementation uses squeeze-and-excitation, with global pooling followed by excitation and rescaling.
  • Local Reconfiguration: Local reconfiguration maps feature-hierarchy patches to output patches using a shared residual learning block across local receptive fields.The residual block is described as a universal function approximator trained by back-propagation; pyramid levels use shared operations and 256-channel outputs.

4 Experiments

Experiments evaluate the method on VOC and COCO using reimplemented SSD, Faster R-CNN, and FPN frameworks, with specified training settings and baselines.

  • Experiments use PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, with ImageNet1k-pretrained VGG-16 and ResNet backbones.
  • The study reimplements SSD, Faster R-CNN, and Feature Pyramid Networks using PyTorch.
  • SSD experiments train on VOC 2007 and VOC 2012 trainval sets and test on VOC 2007 test, using VGG-16 for ablations.
  • The comparison includes original SSD and SSD with feature lateral connections as baselines.
  • Original SSD scores 77.5%, while adding lateral connections improves the result to 78.5%.

Baselines

Ablations within SSD300 compare the proposed reconfiguration with lateral connections and isolate the contribution of global attention.

  • The table evaluates the effectiveness of various designs within SSD300.
  • Removing global attention reduces performance to 79.0% mAP, a 0.6% decrease.
  • 79.6% is achieved by global and local reconfiguration, compared with 78.5% for SSD with lateral connections.

Only use the term R(·)

Using only the term R(·) and restricting feature use to deeper layers affect accuracy and the efficiency comparison with lateral connections.

  • Using only R(·) lowers the result by 0.4% compared with the residual learn block.
  • Using only deeper features lowers accuracy by 0.2% relative to using the total feature hierarchy.
  • The model reports 2.7% accuracy gain with 39.5 fps and exceeds the lateral-connection SSD in both accuracy and speed.

Accuracy vs. Speed

The design generates final pyramid maps simultaneously and also improves performance when replacing lateral connections in a two-stage detector.

  • Final pyramid maps are generated simultaneously, making the design more efficient than serial pyramid construction.
  • In Faster R-CNN, global-local transformation reaches 80.6% versus 79.8% with lateral connections.
  • The global-local transformation improves Faster R-CNN with ResNet-101 from 78.9% mAP to 80.6%.

Comparison with other state-of-the-arts

On PASCAL VOC 2007, the proposed model improves SSD-based detection and remains competitive across feature-pyramid and detector comparisons.

  • 79.6% mAP is achieved by the 300 × 300 model, compared with 77.5% for SSD300 and on par with SSD512.
  • 82.4% mAP is achieved with ResNet-101 and 512 × 512 input, exceeding the reported DSSD and R-FCN results.
  • Table 2 evaluates the effectiveness of alternative designs within the Faster R-CNN framework.
  • The model’s recall exceeds 90% under the detection-analysis evaluation, with higher recall under the weak 0.1 Jaccard-overlap criterion.

4.2 PASCAL VOC 2012

On PASCAL VOC 2012, the proposed models follow the VOC2007 trend and outperform SSD variants, with stronger results using ResNet-101.

  • +2.7% with 300 × 300 is reported over SSD and other variants on the VOC2012 task.
  • The VGG-16 model obtains results similar to DSSD with a ResNet-101 backbone.
  • 81.1% is achieved by the final ResNet-101 model, exceeding the reported state-of-the-art methods.
  • Table 4 reports PASCAL VOC 2012 test detection results trained with 07++12.

4.3 MS COCO

On MS COCO, the proposed method improves SSD across input sizes and backbones, with gains reported for overall and multi-scale detection.

  • MS COCO evaluation uses average mAP over IoU thresholds from 0.5 to 0.95 as the overall performance metric.
  • 31.3% AP is achieved with the ResNet-101 backbone, exceeding DSSD321 by +3.3%.
  • 34.6% accuracy is achieved with 512 × 512 input, exceeding the reported RetinaNet result.
  • The method achieves better detection accuracies than SSD and DSSD for small, medium, and large objects.
  • Table 6 reports multi-scale results under SSD with a ResNet-101 backbone.

5 Conclusions

The paper frames feature-pyramid construction as feature reconfiguration and concludes that global-and-local transformations provide room to improve multiscale detection.

  • The paper identifies reconfiguring and reusing the feature hierarchy as a key issue in constructing ConvNet feature pyramids.
  • Global-and-local transformations explicitly model feature reconfiguration for object-specific scales.
  • Extensive experiments compare the proposed method with other feature-pyramid variations.
  • The qualitative examples compare SSD300 at 77.5% mAP with Ours-300 at 79.6% mAP on VOC2007.
Loading 1808.07993v1…