PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable Convolutional Networks

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei

arXiv:1703.06211v3cs.CV

TL;DR

CNNs have difficulty modeling geometric transformations when their modules use fixed spatial structures. Deformable ConvNets learn offsets to adapt spatial sampling, and the paper reports this is feasible and effective for object detection and semantic segmentation.

  • Problem

    Existing approaches assume known geometric transformations or rely on hand-crafted invariance, limiting their handling of unknown or overly complex transformations.

    Source: §1 · p. 1

  • Method

    Deformable convolution and RoI pooling augment spatial sampling locations with learned offsets, forming end-to-end trainable modules that replace their plain counterparts.

    Source: §2.1 · p. 2, §1 · p. 2, §2.2 · p. 3

  • Results

    The paper shows dense spatial transformation learning in CNNs is feasible and effective for sophisticated vision tasks, including object detection and semantic segmentation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Deformable ConvNets provide a simple, efficient, end-to-end solution for modeling dense spatial transformations in CNNs.

    Source: §5 · p. 9

    The modules operate on the 2D spatial domain; extension to 3D is described as straightforward.

    Source: §2 · p. 2

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation. The code would be released.

1. Introduction

The paper addresses CNNs’ limited ability to model large, unknown geometric transformations by introducing deformable convolution and deformable RoI pooling, which learn adaptive offsets from features. The lightweight modules replace standard counterparts and train end-to-end with standard back-propagation.

Introduction

  • Deformable convolution adds learned 2D offsets to standard convolution’s sampling grid, enabling free-form deformation conditioned locally and adaptively on preceding feature maps.The offsets are learned through additional convolutional layers.Source: §1 · pp. 1–2
  • Deformable RoI pooling learns offsets for each bin from preceding feature maps and RoIs, enabling adaptive part localization for objects with different shapes.Source: §1 · p. 2
  • Both modules add few parameters and computation, replace plain CNN counterparts, and support end-to-end training with standard back-propagation.CNNs using them are called deformable convolutional networks, or deformable ConvNets.Source: §1 · p. 2
  • Unlike spatial transform networks [26] and deformable part models, deformable ConvNets handle dense spatial transformations in a simple, efficient, deep, and end-to-end manner.Source: §1 · p. 2

2. Deformable Convolutional Networks

The section introduces deformable convolution and deformable RoI pooling, which learn offsets to adapt spatial sampling while retaining the same operation across channels. It describes the modules in 2D and notes that extension to 3D is straightforward.

2.1. Deformable Convolution

  • Deformable convolution adds learned offsets to regular-grid kernel locations, using bilinear interpolation for fractional samples and a convolutional layer to predict offsets.The offset-prediction layer operates on the same input feature map and learns simultaneously with the feature-producing convolution.Source: §2.1 · pp. 2–3

2.2. Deformable RoI Pooling

  • Deformable RoI pooling offsets each bin, predicts normalized offsets from pooled features, scales them by RoI width and height (γ = 0.1), and uses bilinear interpolation.For position-sensitive pooling, the input is the bin-specific score map x_i,j, and a convolutional branch plus PS pooling produces normalized offsets.Source: §2.2 · p. 3

2.3. Deformable ConvNets

  • Both modules preserve their plain versions’ inputs and outputs and can replace them; offset layers start at zero and learn through backpropagation, with β = 0.01 for Faster R-CNN’s fc layer.The default offset-layer learning rate is β = 1 times the existing layers’ rate; the Faster R-CNN fc layer uses β = 0.01.Source: §2.3 · pp. 3–4
  • Feature extraction uses ImageNet-pretrained ResNet-101 or modified Aligned-Inception-ResNet, with the last block’s stride reduced and convolution dilation increased to preserve resolution.The effective stride changes from 32 to 16 pixels; stride becomes 1 and dilation 2 in the last block.Source: §2.3 · p. 4
  • Task heads use DeepLab for per-pixel segmentation and adapt Faster R-CNN or R-FCN RoI pooling optionally, while Category-Aware RPN uses a C + 1-class classifier.Deformable convolution can be applied to the last few convolutional layers; three layers were found to be a good trade-off across tasks.Source: §2.3 · p. 4

3. Understanding Deformable ConvNets

Deformable ConvNets adapt convolutional sampling locations and RoI parts to image content, enabling adaptive receptive fields and improved localization of non-rigid objects. The related-work discussion distinguishes this approach from methods that transform features, constrain filters, or use predefined pooling layouts.

3. Understanding Deformable ConvNets

  • Stacked deformable convolutions adapt sampling locations to object scale and shape, producing receptive fields that vary across the feature map.Figure 6 illustrates sampling locations for background, small-object, and large-object activations.Source: §3 · p. 5, Figure 6 · p. 5
  • Deformable RoI pooling moves parts beyond regular RoI bins onto nearby foreground regions, enhancing localization, especially for non-rigid objects.Figure 7 illustrates offset parts covering non-rigid objects.Source: §3 · p. 5, Figure 7 · p. 5

3.1. In Context of Related Works

  • Unlike STN’s global parametric warping, deformable convolution samples locally and densely, then combines samples by weighted summation; it is easy to train and integrate for dense or semi-dense predictions.The paper describes the offsets as an extremely lightweight spatial transformer [26]; such prediction tasks are difficult, if not infeasible, for STN [26] [37].Source: §3.1 · p. 5
  • Unlike Active Convolution’s spatially shared, static offsets [27], deformable convolution predicts offsets that vary by image location for (semi-)dense prediction.Active Convolution also augments sampling locations and learns offsets end-to-end by back-propagation.Source: §3.1 · pp. 5–6
  • Deformable convolution learns receptive fields adaptively, addressing the limited effective receptive field [43]; it generalizes atrous convolution by adapting sampling locations.The paper notes that effective receptive field size increases more slowly than theoretical receptive field size as layers are added.Source: §3.1 · p. 6
  • Deformable RoI pooling learns pooling regions end-to-end in CNNs, unlike hand-crafted spatial pyramid regions or sparse layouts learned from engineered sets [28].Its part deformation is similar in spirit to DPM, but simpler because it does not model spatial relations between parts.Source: §3.1 · p. 6
  • Unlike approaches that encode transformations known a priori, deformable modules learn transformation invariance from the target task and generalize across transformations.The paper contrasts this with hand-crafted feature-extraction structures based on known transformation parameterizations.Source: §3.1 · p. 6
  • Dynamic filters condition on input features but learn filter weights rather than sampling locations, which deformable convolution learns.Source: §3.1 · p. 6

4. Experiments

Across segmentation and detection experiments, deformable convolution and deformable RoI pooling improve performance, while deformable ConvNets add only small model and computation overhead. On COCO, deformable detection models outperform their plain counterparts, with further gains from multi-scale testing and iterative box averaging.

4.1. Experiment Setup and Implementation

  • Experiments assess segmentation on PASCAL VOC and Cityscapes using mIoU, and detection on PASCAL VOC and COCO using standard mAP protocols.Detection reports VOC mAP at IoU thresholds 0.5 and 0.7, and COCO mAP@[0.5:0.95] and mAP@0.5.Source: §4.1 · p. 7

4.2. Ablation Study

  • Deformable RoI pooling alone yields noticeable gains, especially at mAP@0.7; combining it with deformable convolution produces significant accuracy improvements.The module applies to Faster R-CNN and R-FCN.Source: Table 3 · p. 8, §4.2 · p. 8
  • Larger atrous dilation improves accuracy across tasks, but optimal dilation varies by task—6 for DeepLab and 4 for Faster R-CNN—and deformable convolution performs best.The tested atrous dilation values were 4, 6, and 8, compared with the default value of 2.Source: Table 3 · p. 8, §4.2 · p. 8
  • Deformable ConvNets add only small parameter and computation overhead, so the reported significant performance improvement is attributed to modeling geometric transformations rather than increasing parameters.The runtime comparison includes image resizing, network forward computation, and postprocessing.Source: Table 4 · p. 8, §4.2 · pp. 8–9
  • Accuracy rises as more deformable convolution layers are added, especially for DeepLab and class-aware RPN, then saturates at 3 layers for DeepLab and 6 for others.The remaining experiments use 3 deformable layers in feature extraction networks.Source: §4.2 · p. 8
  • Effective dilation measurements show deformable filter receptive fields correlate with object size, while background filter sizes fall between those for medium and large objects.These observations are consistent across layers; effective dilation is the mean distance between adjacent sampling locations and roughly measures receptive-field size.Source: §4.2 · p. 8

4.3. Object Detection on COCO

  • On COCO test-dev, deformable class-aware RPN/Faster R-CNN/R-FCN score mAP@[0.5:0.95] 25.8%/33.1%/34.5%, respectively, 11%/13%/12% relatively above plain counterparts.The reported relative improvements correspond, in order, to the three detection models.Source: Table 5 · p. 9, §4.3 · p. 9
  • With multi-scale testing and iterative bounding box average, deformable R-FCN reaches mAP@[0.5:0.95] 37.5%, and deformable ConvNet gains are complementary to these additions.The tested image shorter-side scales were 480, 576, 688, 864, 1200, and 1400 pixels.Source: Table 5 · p. 9, §4.3 · p. 9

5. Conclusion

Deformable ConvNets provide a simple, efficient, deep, end-to-end solution for modeling dense spatial transformations. The paper shows that CNNs can feasibly and effectively learn these transformations for sophisticated vision tasks.

  • Deformable ConvNets offer a simple, efficient, deep, end-to-end solution for modeling dense spatial transformations.Source: §5 · p. 9
  • The paper shows for the first time that learning dense spatial transformations in CNNs is feasible and effective for sophisticated vision tasks.Source: §5 · p. 9
  • The demonstrated tasks include object detection and semantic segmentation.Source: §5 · p. 9

A. Deformable Convolution/RoI Pooling Back-propagation

The section addresses offset-gradient back-propagation for deformable convolution and RoI pooling, noting that convolution offsets are two-dimensional.

A. Deformable Convolution/RoI Pooling Back-propagation

B. Details of Aligned-Inception-ResNet

Aligned-Inception-ResNet modifies Inception-ResNet to avoid feature-alignment problems in dense prediction while using a simpler repetitive-module design. It is pretrained on ImageNet-1K, with model complexity and classification errors reported.

B. Details of Aligned-Inception-ResNet

  • Original Inception-ResNet [51] uses multiple valid convolution and pooling layers, causing feature-alignment issues for dense prediction.A feature cell near the output can project to an image location different from its receptive-field center, despite task-specific networks such as FCNs assuming alignment.Source: B. Details of Aligned-Inception-ResNet · p. 10
  • Aligned-Inception-ResNet uses proper convolution and pooling padding to avoid feature-alignment problems, repetitive modules for simpler design, and stride-2 1 × 1 convolutions when feature dimensions change.Figure 8 depicts Inception Residual Blocks for the conv2, conv3, conv4, and conv5 stages.Source: B. Details of Aligned-Inception-ResNet · p. 10, Figure 8 · p. 10
  • The model is pretrained on ImageNet-1K classification using the training procedure in [22]; Table 7 reports model complexity and top-1 and top-5 classification errors.Source: B. Details of Aligned-Inception-ResNet · pp. 10–11

Related papers

Loading 1703.06211v3…