PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable Convolutional Networks

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei

arXiv:1703.06211v3cs.CV

TL;DR

CNNs struggle to model unknown and complex geometric transformations because their sampling structures are fixed. Deformable ConvNets learn offsets for convolution and RoI pooling, providing a simple end-to-end solution shown effective for object detection and semantic segmentation.

  • Problem

    Fixed transformation assumptions and handcrafted invariant features limit CNNs’ ability to generalize to unknown or overly complex geometric transformations.

  • Method

    Deformable ConvNets augment convolutional and RoI pooling sampling locations with learned offsets, replacing standard modules and training end-to-end.

  • Results

    Experiments show that learning dense spatial transformations in CNNs is feasible and effective for object detection and semantic segmentation.

  • Takeaways & Limitations

    Deformable ConvNets offer a simple, efficient, end-to-end approach for modeling dense spatial transformations in sophisticated vision tasks.

  • Takeaways & Limitations

    The presented modules operate on the 2D spatial domain, while extension to 3D is described as straightforward rather than evaluated here.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation. The code would be released.

1. Introduction

The introduction identifies fixed geometric structures as a key limitation of CNNs for modeling large, unknown transformations. It presents deformable convolution and deformable RoI pooling, which learn input-conditioned offsets and can be integrated into CNNs end-to-end.

  • Motivation: Geometric transformations in scale, pose, viewpoint, and part deformation remain a central challenge in visual recognition.Existing strategies rely on augmented training data or hand-crafted invariant features and algorithms.
  • Problem: CNNs are inherently limited because their convolution, pooling, and RoI pooling modules use fixed spatial sampling structures.This leaves them without internal mechanisms for handling large, unknown geometric transformations.
  • Deformable convolution: Deformable convolution adds learned 2D offsets to regular convolution grids, enabling local, dense, adaptive, free-form deformation of sampling locations.The offsets are predicted from preceding feature maps through additional convolutional layers.
  • Deformable RoI pooling: Deformable RoI pooling learns offsets for individual bins, enabling adaptive localization of object parts with different shapes.The offsets depend on preceding feature maps and the RoIs.
  • Deformable ConvNets: Both modules add small amounts of parameters and computation, replace plain CNN counterparts, and support end-to-end training with standard back-propagation.CNNs using these modules are called deformable convolutional networks, or deformable ConvNets.

2. Deformable Convolutional Networks

Deformable convolution and deformable RoI pooling augment regular spatial sampling or binning locations with learned offsets, enabling CNN modules to adapt their receptive fields and pooling regions. The modules preserve the input-output interfaces of their standard counterparts and can be trained end-to-end through bilinear interpolation.

  • Deformable Convolution: Deformable convolution augments a regular grid R with learned offsets ∆p_n, producing sampling at irregular locations p_n+∆p_n.Fractional offsets are implemented with bilinear interpolation.
  • Deformable Convolution: An offset-generating convolution uses the same spatial resolution and dilation as the current convolution, with 2N output channels representing N two-dimensional offsets.The offset fields are generated from the same input feature map.
  • Deformable RoI Pooling: Deformable RoI pooling adds offsets to spatial binning positions and uses bilinear interpolation when those offsets are fractional.The RoI is divided into k × k bins before offset-based pooling.
  • Deformable RoI Pooling: RoI-pooling offsets are predicted as normalized values, scaled by the RoI width and height, with γ empirically set to 0.1.A fully connected layer generates the normalized offsets, which are transformed element-wise into real offsets.
  • Training and Integration: Both deformable modules retain the input-output interfaces of their plain counterparts, so they can replace them in existing CNNs and train end-to-end by back propagation.Offset-learning layers are zero-initialized and gradients pass through bilinear interpolation; the resulting models are called deformable ConvNets.

3. Understanding Deformable ConvNets

Deformable convolution and deformable RoI pooling learn task-specific spatial deformations through adaptive sampling, improving receptive-field adaptation and localization. Compared with related methods, the modules are lightweight, local, dense, and end-to-end trainable for object detection and semantic segmentation.

  • Deformable modules: Stacked deformable convolutions adapt receptive fields and sampling locations to object scale and shape.The paper presents quantitative evidence of this adaptive deformation in Table 2.
  • Deformable modules: Deformable RoI pooling moves parts beyond regular bins toward foreground regions, enhancing localization especially for non-rigid objects.Its grid structure is therefore no longer regular, as illustrated in Figure 7.
  • Related work: Unlike STN, deformable convolution uses local dense sampling and weighted summation rather than global parametric warping.The offset learning is described as a lightweight spatial transformer that generates new feature maps through weighted summation.
  • Related work: Deformable convolution integrates into CNN architectures, trains easily, and supports dense or semi-dense prediction tasks that challenge STN.The paper specifically identifies semantic segmentation and object detection as effective application settings.
  • Related work: Unlike Active Convolution’s shared static offsets, deformable convolution predicts dynamic offsets that vary by image location.These offsets model dense spatial transformations and support object detection and semantic segmentation.
  • Related work: Deformable RoI pooling is the first method described here to learn pooling regions end-to-end in CNNs.The current regions have the same size, while extension to multiple sizes is described as straightforward.

4. Experiments

Experiments show that deformable convolution and deformable RoI pooling improve vision-task accuracy, while adding only small parameter and computation overhead. Ablations indicate adaptive receptive fields and task-dependent dilation contribute to these gains.

  • Deformable Convolution: Accuracy improves as more deformable convolution layers are added, saturating at 3 layers for DeepLab and 6 layers for other evaluated networks.The remaining experiments use 3 deformable layers in feature extraction networks.
  • Deformable Convolution: Learned deformable-convolution offsets adapt to image content, with effective receptive-field sizes correlated with object sizes.Effective dilation is defined as the mean distance between adjacent sampling locations and serves as a rough receptive-field measure.
  • Deformable Convolution: Larger atrous-convolution dilation improves accuracy across tasks, but optimal dilation is task-dependent, such as 6 for DeepLab and 4 for Faster R-CNN.These findings indicate that default networks have overly small receptive fields, while deformable convolution further improves performance.
  • Deformable RoI Pooling: Deformable RoI pooling alone yields noticeable gains, especially at strict mAP@0.7, and combining it with deformable convolution produces significant accuracy improvements.The module applies to Faster RCNN and R-FCN.
  • Model Complexity and Runtime: Deformable ConvNets add only small overhead in model parameters and computation, indicating that their performance gains arise from modeling geometric transformations rather than enlarging the models.Runtime comparisons include image resizing, network forward, and postprocessing.
  • Object Detection: 25.8%, 33.1%, and 34.5% mAP@[0.5:0.95] are achieved by deformable class-aware RPN, Faster R-CNN, and R-FCN, respectively, on COCO test-dev.These scores are 11%, 13%, and 12% relatively higher than the corresponding plain-ConvNet counterparts.

5. Conclusion

The paper presents deformable ConvNets as a simple, efficient, deep, end-to-end solution for modeling dense spatial transformations. It demonstrates that CNNs can feasibly and effectively learn such transformations for object detection and semantic segmentation.

  • Deformable ConvNets provide a simple, efficient, deep, and end-to-end approach to modeling dense spatial transformations.The conclusion characterizes the approach using all four properties.
  • The work demonstrates that learning dense spatial transformations in CNNs is feasible and effective.
  • The approach is validated on sophisticated vision tasks including object detection and semantic segmentation.

A. Deformable Convolution/RoI Pooling Back-propagation

The section derives back-propagation gradients for deformable convolution and deformable RoI pooling, including 2D offsets and normalized RoI offsets. The normalized-offset gradient follows by differentiating the offset parameterization.

  • Deformable Convolution: Deformable convolution computes gradients with respect to the offset ∆p_n.The gradient is given for the offset used in deformable convolution Eq. (2).
  • Deformable Convolution: The offset ∆p_n is two-dimensional, with ∂∆p_n denoting the derivative with respect to ∂∆p_x.The derivative can be derived from Eq. (4).
  • Deformable RoI Pooling: Deformable RoI pooling similarly computes gradients for its offset parameters.The section introduces the corresponding gradient derivation for the deformable RoI pooling module.
  • Deformable RoI Pooling: Gradients with respect to normalized offsets ∆b_pij follow by differentiating ∆p_ij = γ · ∆b_pij ◦ (w, h).The normalized-offset derivative is obtained through the stated offset transformation.

B. Details of Aligned-Inception-ResNet

Aligned-Inception-ResNet modifies the original architecture to address feature misalignment in dense prediction tasks. It uses proper padding and stride-2 1 × 1 convolutions when feature dimensions change, and is pre-trained on ImageNet-1K classification.

  • Motivation: Valid convolution and pooling in the original Inception-ResNet create feature alignment issues for dense prediction tasks.Projected feature-cell locations can differ from receptive-field centers, conflicting with task-specific alignment assumptions such as those in FCN semantic segmentation.
  • Architecture: Aligned-Inception-ResNet removes the feature alignment problem through proper padding in convolutional and pooling layers.The modified architecture is presented as an alternative to the original Inception-ResNet.
  • Architecture: A stride-2 1 × 1 convolution is used when the feature dimension changes.
  • Training and evaluation: The Aligned-Inception-ResNet model is pre-trained on ImageNet-1K classification, with Table 7 reporting model complexity and top-1 and top-5 classification errors.The training procedure follows [22].
Loading 1703.06211v3…