PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable ConvNets v2: More Deformable, Better Results

Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai

arXiv:1811.11168v2cs.CV

TL;DR

Deformable ConvNets adapt to geometric variation but may still draw features from irrelevant regions. This paper introduces DCNv2 with broader and modulated deformation plus R-CNN feature mimicking, yielding significant COCO gains over DCNv1 in detection and instance segmentation.

  • Problem

    Deformable ConvNets can adapt to object geometry, yet their spatial support may include irrelevant background content.

  • Method

    DCNv2 expands deformable convolution, adds learned modulation, and uses R-CNN feature mimicking to guide training toward object-focused features.

  • Results

    DCNv2 achieves significant performance gains over DCNv1 on COCO object detection and instance segmentation.

  • Takeaways & Limitations

    The reformulated deformable modules improve focus on pertinent image regions while remaining lightweight and incorporable into existing architectures.

  • Takeaways & Limitations

    The feature mimicking module is not used in the semantic segmentation experiments.

Abstract

from arXiv · show

The superior performance of Deformable Convolutional Networks arises from its ability to adapt to the geometric variations of objects. Through an examination of its adaptive behavior, we observe that while the spatial support for its neural features conforms more closely than regular ConvNets to object structure, this support may nevertheless extend well beyond the region of interest, causing features to be influenced by irrelevant image content. To address this problem, we present a reformulation of Deformable ConvNets that improves its ability to focus on pertinent image regions, through increased modeling power and stronger training. The modeling power is enhanced through a more comprehensive integration of deformable convolution within the network, and by introducing a modulation mechanism that expands the scope of deformation modeling. To effectively harness this enriched modeling capability, we guide network training via a proposed feature mimicking scheme that helps the network to learn features that reflect the object focus and classification power of R-CNN features. With the proposed contributions, this new version of Deformable ConvNets yields significant performance gains over the original model and produces leading results on the COCO benchmark for object detection and instance segmentation.

1. Introduction

DCNv1 adapts convolutional sampling and pooling to object geometry, but its support can include irrelevant content. DCNv2 expands deformable modeling and trains it with R-CNN feature guidance, improving COCO detection and segmentation.

  • Geometric variation from scale, pose, viewpoint, and part deformation challenges object recognition and detection.
  • DCNv1 adapts feature representations by learning offsets for convolution sampling locations and RoIpooling bin positions.
  • DCNv1 features conform more closely to object structure, yet their spatial support can spread beyond the region of interest, especially on COCO.
  • DCNv2 expands deformable convolution across more layers and adds modulation to control both sample locations and relative sample influence.
  • R-CNN feature mimicking guides DCNv2 toward features less affected by content outside the region of interest.
  • Extensive COCO experiments show significant improvement over DCNv1 for object detection and instance segmentation.

2. Analysis of Deformable ConvNet Behavior

The analysis compares complementary measures of spatial support and finds that deformable networks adapt more strongly to image content than regular ConvNets. However, their support can still include irrelevant context, motivating DCNv2’s stronger focus on pertinent regions.

  • Spatial Support Visualization: Effective receptive fields, effective sampling locations, and error-bounded saliency regions provide complementary views of image regions influencing a node.
  • Spatial Support Visualization: Error-bounded saliency regions identify the smallest image region preserving a node’s response within a small error bound.
  • Spatial Support of Deformable ConvNets: Deformable convolution considerably enhances geometric modeling, with foreground support covering objects and background nodes encompassing greater context.
  • Spatial Support of Deformable ConvNets: Foreground nodes can still include background areas irrelevant for detection, making deformable spatial support inexact.
  • Spatial Support of Deformable ConvNets: Sampling locations alone can mislead because node responses jointly depend on learned offsets and network weights.
  • Spatial Support of Deformable ConvNets: Deformable RoIpooling places a larger proportion of bins over object foreground than aligned RoIpooling, while foreground bins receive greater classification-branch gradients.

3. More Deformable ConvNets

DCNv2 increases deformable modeling across the network, adds modulation to control feature amplitudes, and uses R-CNN feature mimicking to learn more focused representations.

  • More deformable convolution: DCNv2 replaces more regular convolutional layers with deformable counterparts to strengthen geometric-transformation modeling across feature levels.In ResNet-50, deformable convolutions are applied to all 3 × 3 layers in conv3, conv4, and conv5, totaling 12 layers.
  • Modulated deformable modules: Modulation lets each sampled location or RoI bin receive both a learned offset and a scalar that adjusts its feature amplitude.The scalar lies in [0, 1], allowing the module to reduce or eliminate contributions from selected spatial locations.
  • Modulated deformable modules: The modulated convolution and RoIpooling modules jointly give the network additional freedom to adjust its spatial support and exclude redundant context.The modulation mechanism can set a location or bin’s feature amplitude to zero, substantially reducing its influence on the output.
  • Motivation: The proposed changes target a central limitation of deformable features: their saliency regions can extend beyond the RoI and incorporate irrelevant image content.This outside-RoI content may affect extracted features and degrade object-detection results.
  • R-CNN feature mimicking: Because standard Faster R-CNN training does not learn the enriched representations well, DCNv2 adds feature mimicking from R-CNN features extracted on cropped image content.The auxiliary objective encourages per-RoI Faster R-CNN features to resemble R-CNN features that are less affected by information outside the RoI.

4. Related Work

The paper situates DCNv2 within efforts to model geometric variation and interpret effective spatial support. It extends deformable convolution with stronger modeling and feature mimicking to improve object-focused representations.

  • Deformation Modeling: Handcrafted features and global affine transformations have limited representation power for the complex geometric variations in vision tasks.Deformable ConvNets instead use local, dense sampling through learnable offsets.
  • Deformation Modeling: Deformable ConvNets model geometric transformations through learnable offsets in deformable convolution and deformable RoIpooling.These modules adapt sampling and pooling patterns to object structure.
  • Network Mimicking: DCNv2 extends Deformable ConvNets by enhancing modeling power and facilitating training through R-CNN feature mimicking.The mimic loss targets features reflecting R-CNN’s object focus and classification power.
  • Attention and Relation Modules: Relation and attention modules can incur quadratic computation, whereas the paper’s sparse deformable mechanism has linear computational overhead.Learned offsets select a sparse set of attended elements, while modulation controls their aggregation weights.
  • Effective Support: Effective receptive-field and salient-region analyses show that only a small proportion of theoretically receptive-field pixels contribute strongly to predictions.These analyses motivate examining the image regions that actually support network responses.

5. Experiments

Experiments evaluate DCNv2 against regular ConvNets and DCNv1 on COCO detection and instance segmentation. Enriched deformation modeling and R-CNN feature mimicking improve accuracy, spatial focus, and performance across backbones.

  • Experimental Setup: COCO experiments use Faster R-CNN and Mask R-CNN baselines with ImageNet-pretrained ResNet-50 backbones and mean average precision at box and mask IoUs.Training uses 118k COCO 2017 train images, with validation and test-dev evaluation sets.
  • Enriched Deformation Modeling: 38.0% APbbox for Faster R-CNN and 40.4% APbbox with 35.3% APmask for Mask R-CNN are obtained by the DCNv1 baselines.Regular CNN baselines obtain 34.7% APbbox, and 36.6% APbbox with 32.2% APmask, respectively.
  • Enriched Deformation Modeling: 2.0%–3.0% gains in APbbox and APmask arise when deformable convolution replaces layers in conv3–conv5, while modulation adds 0.3%–0.7%.Replacing additional conv2 layers yields no further COCO improvement.
  • Spatial Support: Enriched deformable modeling produces spatial support that adapts better to image content than DCNv1.This comparison is reported with reference to Figure 1.
  • R-CNN Feature Mimicking: R-CNN feature mimicking improves APbbox and APmask by about 1%–1.4% in Faster R-CNN and Mask R-CNN with enriched deformation modeling.Mimicking positive-box foreground features is particularly effective, whereas negative boxes benefit from context.
  • R-CNN Feature Mimicking: Feature mimicking provides almost no accuracy gain for regular ConvNets, whose representation capacity does not focus features on object foreground.The visualized support regions remain unfocused even with the auxiliary mimic loss.
  • Application on Stronger Backbones: DCNv2 considerably outperforms regular ConvNets and DCNv1 across ResNet-101 and ResNeXt-101 backbones on COCO test-dev.The DCNv2 configurations use modulated deformable convolution in conv3–conv5, modulated deformable RoIpooling, and R-CNN feature mimic supervision.

6. Conclusion

DCNv2 reformulates Deformable ConvNets to focus more effectively on pertinent image regions by increasing modeling power and strengthening training. It achieves significant gains on COCO for object detection and instance segmentation.

  • DCNv2 addresses spatial support extending beyond the region of interest and exposing features to irrelevant image content.
  • The reformulation improves the network’s ability to focus on pertinent image regions through increased modeling power and stronger training.
  • DCNv2 produces significant performance gains on the COCO benchmark for object detection and instance segmentation.

A1. Error-bounded Image Saliency

The appendix defines visual support regions through error-bounded saliency and uses a heuristic two-step procedure to make their optimization tractable. These visualizations support comparisons of network spatial support across resolutions and architectures.

  • A binary mask preserves pixels where M(p)=1 and sets pixels where M(p)=0 to zero before measuring the network response.
  • The visual support region is the set of pixels retained in the optimized mask under a strictly bounded reconstruction loss.
  • The optimization is made tractable by constraining support to a centered rectangle that starts at area zero and enlarges gradually.
  • The procedure visualizes both whole-image feature-map nodes and the per-RoI 2fc node in Faster R-CNN.
  • Figure 4 reports APbbox across input resolutions, while Figure 5 compares effective receptive fields and error-bounded saliency regions.

A2. DCNv2 with Various Image Resolution

DCNv2 outperforms regular ConvNets across tested input resolutions and adapts its spatial support more effectively, particularly when high-resolution images contain large and medium objects. Multi-scale testing further improves its APbbox score over the best single-scale setting.

  • DCNv2 outperforms regular ConvNets at every tested input resolution from 400 to 1400 pixels on COCO 2017 test-dev.
  • DCNv2 obtains its highest APbbox scores with input images whose shorter side is 1,000 pixels.
  • At resolutions above 1,000 pixels, regular ConvNet APbbox decreases noticeably while DCNv2 scores remain almost unchanged.
  • DCNv2 adapts its spatial support to objects at various resolutions, whereas regular ConvNets may cover only a small portion of large objects at high resolution.
  • 1.2%: multi-scale testing improves DCNv2 APbbox over its best single-scale setting using ResNet-101.

A3. ImageNet Pre-Trained DCNv2

ImageNet pre-training gives DCNv2 noticeable classification improvements over regular and DCNv1 baselines with minor additional computation overhead. Fine-tuning pretrained offsets and modulation scalars helps on PASCAL VOC, while the effect on COCO detection is minor.

  • DCNv2 replaces deformable convolutions in DCNv1’s conv5 stage with modulated deformable convolutions across the conv3–conv5 stages.
  • DCNv2 achieves noticeable improvements over regular and DCNv1 baselines on ImageNet classification with minor additional computation overhead.
  • Pre-training learnable offsets and modulation scalars yields noticeable accuracy improvements for PASCAL VOC object detection and semantic segmentation.
  • The effect of offset and modulation pre-training on COCO detection is minor.
  • The mimicking module is not used in the semantic segmentation experiments.
Loading 1811.11168v2…