PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhanced Deformable Convolution with Center-invariant Offset and Edge-aware Mask

Yixiao Li, Xiaoyuan Yang, Jin Jiang, Minghao Zou, Guanghui Yue, Baoquan Zhao, Jun Liu, Wei Zhou

arXiv:2609.10387v1cs.CV

TL;DR

Existing deformable convolutions are limited by dense offsets, restricted long-range dependencies, and difficulty scaling to larger kernels. The paper proposes EDCN, whose decoder EDC combines center-invariant large-kernel offsets with edge-aware selective deformation. EDC surpasses deformable-convolution counterparts across four semantic segmentation datasets, while the paper also reports limitations of existing methods in larger-kernel and plug-and-play settings.

  • Problem

    Existing deformable convolutions have dense offsets, limited long-range dependencies, and difficulty extending effectively to larger kernels for semantic segmentation.

  • Method

    EDCN uses a decoder Enhanced Deformable Convolution that integrates large-kernel Center-invariant Offset and edge-aware masking modules as a plug-and-play layer.

  • Results

    EDCN retains deformable-convolution benefits while surpassing competing variants across four semantic segmentation datasets.

  • Takeaways & Limitations

    EDC provides an efficient, high-performing plug-and-play choice that refines deformation toward target-object edges and supports analysis across kernel sizes.

  • Takeaways & Limitations

    Existing deep-stacking approaches are not fully efficient for plug-and-play segmentation, and existing deformable variants struggle with larger kernels.

Abstract

from arXiv · show

Deformable convolution networks have recently become popular for many computer vision tasks, especially for semantic segmentation, because of their exceptional capabilities in dynamic spatial modeling. However, due to the dense deformable offsets and the lack of longer-range dependencies, they can not fully adopt proper and precise deformations for feature representations. To tackle the issues, in this paper, we propose Enhanced Deformable ConvNets (EDCN) for semantic segmentation. Specifically, a novel Enhanced Deformable Convolution (EDC) is exploited in the decoder, which integrates the Center-invariant Offset Module (COM) and Edge-aware Mask Module (EMM). The COM employs larger kernels and eliminates deformations at the kernel center, obtaining offsets that are more in line with the target from richer spatial information. Concurrently, the EMM obtains the significance of image content via Sobel edge detection, then selectively applies deformations based on the content significance, minimizing unnecessary deformations associated with relatively less important information, thereby avoiding impact from less informative regions. Experiments show that EDC outperforms state-of-the-art deformable convolution variants, including Deformable ConvNets V1-V4 and Entire Deformable ConvNets, across mainstream segmentation datasets with various decoder settings. Moreover, ablation studies confirm the effectiveness of each component. In addition, visualizations illustrate that EDC enhances spatial adaptation and target focus. We further analyze the extendibility of EDC to larger kernels on the image classification benchmark. Code will be publicly released.

1 Introduction

Existing deformable convolutions adapt sampling for segmentation but remain limited by restricted receptive fields, dense offsets, and poor scalability to larger kernels. EDC addresses these issues with larger-kernel, center-invariant offsets and edge-guided selective deformation, and EDCN surpasses deformable-convolution counterparts across four segmentation datasets.

  • Motivation: Restricted effective receptive fields limit long-range dependencies, while dense offset sampling constrains deformable convolutions’ use with larger kernels.Larger receptive fields benefit dense prediction, but existing approaches require deep stacking and remain difficult to extend efficiently.
  • EDCN architecture: EDC replaces standard decoder convolutions as a plug-and-play layer in a general CNN-based semantic-segmentation framework.The framework accommodates different placements of standard convolutions through Front SSM and Back SSM components.
  • Enhanced Deformable Convolution: EDC integrates a Center-invariant Offset Module and an Edge-aware Mask Module to improve deformable offset sampling.The contribution combines large-kernel spatial information with edge awareness.
  • EDCN architecture: EDCN retains deformable-convolution benefits while surpassing competing variants on four semantic segmentation datasets.The proposed EDC is evaluated as a plug-and-play replacement within a unified encoder-decoder architecture.
  • Experiments: Visualizations show finer target-object edges, while ablations analyze kernel size, edge operators, thresholds, complexity, and inference speed.The study also examines center-to-border ratios and binary logistic regression.

2 Related Work

Deformable convolution variants differ in how they sample, shift, decompose, and aggregate features, balancing adaptive deformation against efficiency and interference. Extending these variants to larger kernels remains challenging.

  • Deformable convolution variants: Deformable convolution variants include DCN V1, DCN V2, Entire Deformable ConvNets, DCN V3, and DCN V4, with differing offset-sampling strategies.DCN V1 uses learnable horizontal and vertical offsets at each regular-grid location, while DCN V2 refines this approach.
  • Deformable convolution variants: Entire Deformable ConvNets shifts the whole kernel, reducing mutual offset interference but sacrificing adaptive shape and dynamic sampling.
  • Deformable convolution variants: DCN V3 decomposes convolution into depth-wise and point-wise components with grouping, improving computational and memory efficiency while retaining deformability.The passage notes that this approach is less effective with larger kernel sizes.
  • Open challenge: Expanding existing deformable convolution variants to larger kernel sizes remains challenging.

3 Proposed Method

EDCN replaces decoder convolutions with EDC, combining center-invariant offsets and edge-aware masks to obtain selective, spatially adaptive deformations. Its design targets dense-offset limitations, weak large-kernel scalability, and unnecessary deformation in less informative regions.

  • Motivation: Existing deformable convolution variants introduce dense offsets at every sampling point but face difficulty extending to large kernels and deforming every input region appropriately.These limitations motivate EDC’s selective sampling strategy and center-invariant treatment of larger kernels.
  • Enhanced Deformable ConvNets: EDCN replaces decoder standard convolutions with EDC within a general CNN-based segmentation framework, with front and back spatial semantic mixers determined by convolution placement.PSPNet uses FSSM, ANN uses both FSSM and BSSM, and ISANet uses BSSM according to the position of its replaced convolution.
  • Edge-aware Mask Module: The Edge-aware Mask Module selectively applies deformable convolution to crucial or salient regions and standard convolution to less informative regions.This dual-branch strategy is intended to reduce the impact of less important information while retaining richer semantic information from large kernels.
  • Center-invariant Offset Module: Center-invariant offsets keep center sampling locations unchanged while deforming border locations, enabling larger kernels with limited additional parameters.The design addresses local-information limitations and avoids unnecessary offset variables that may introduce noise or confusion as kernels grow.
  • Enhanced Deformable Convolution: EDC integrates Center-invariant Offset and Edge-aware Mask Modules to provide more precise, selectively adaptive deformations focused on target information.The dual-branch design combines center-invariant sampling with content-dependent use of deformable or standard convolution.

4 Experiments

Experiments compare EDCN with deformable convolution variants across segmentation datasets, decoders, kernel settings, and ablations. EDCN generally outperforms counterparts while balancing performance, complexity, and inference speed.

  • Semantic Segmentation Results: EDCN outperforms other deformable convolution counterparts in most comparisons across MS COCO, ADE-20K, Cityscapes, and PASCAL VOC.The comparisons cover PSPNet, ANN, and ISANet decoders under the reported experimental settings.
  • ISANet Analysis: EDC reduces unnecessary offsets and addresses failure issues associated with More Deformable ConvNets in the shallow ISANet setting.The paper attributes the issue to dense offsets introduced early without adequate long-range information.
  • Complexity and Inference Speed: EDCN achieves a better trade-off between segmentation performance, parameter size, computational cost, and inference speed than several deformable variants.Segformer has fewer parameters and FLOPs, but the authors describe EDCN as an efficient high-performing plug-and-play choice.
  • Decoder Settings: EDC alone achieves good performance with only two standard convolution layers in the decoder.This result is reported for the decoder configuration discussed in the experiments.
  • Ablation Studies: Ablations identify COM and EMM as important components, supporting center-invariant kernels and edge-aware masks for adaptive deformation.The supplied passages report the importance of both components and describe edge-region deformation with standard kernels elsewhere.

4.5 Ablation Study on Kernel Size

The kernel-size analysis evaluates deformable variants from standard segmentation kernels through larger settings and visualizes their spatial behavior. EDCN is generally strongest at larger segmentation kernels, while ultra-large-kernel scaling remains difficult.

  • Kernel-Size Performance: EDCN and DCN V3 achieve the best and second-best performances in most three-kernel-size comparisons on ADE-20K and MS COCO.Entire Deformable ConvNets and DCN V2 suffer significant performance drops as kernel size increases.
  • Complexity and Speed: EDCN and DCN V3 provide the best trade-off between model complexity and inference speed in the larger-kernel comparison.The comparison covers the three kernel sizes evaluated on ADE-20K and MS COCO.
  • Variant Limitations: DCN V4 is limited to a 3 × 3 kernel and cannot be directly extended to 5 × 5 or 7 × 7 settings.The paper attributes this limitation to DCN V4’s design and separately reports its 3 × 3 performance.
  • Ultra-Large Kernels: On ultra-large kernels up to 17 × 17, EDC shows a slight advantage at K=9 but struggles at K=15 and K=17 with higher complexity.The paper identifies extending deformable convolution to ultra-large kernels as an unresolved challenge.
  • Redundant Deformations: EDC focuses receptive-field responses more precisely on target objects, whereas other variants cover more irrelevant regions.The comparison identifies redundant coverage around table, pillow, and glass handrail regions.
  • Long-Range Dependencies: EDC produces a larger effective receptive field than several deformable counterparts on CIFAR-100, supporting stronger long-range dependency modeling.The accompanying quantitative analysis is reported to show the best performance for ResNet34 with EDC.

5 Conclusion

The conclusion presents EDCN as a unified semantic-segmentation architecture built around EDC, which combines center-invariant offsets with edge-aware deformation. The reported experiments and visualizations indicate improved performance across kernel sizes and stronger deformable-convolution effectiveness for larger kernels.

  • Architecture: EDCN uses EDC, a dual-branch convolution combining the Center-invariant Offset Module and Edge-aware Mask Module.The architecture is introduced for semantic segmentation.
  • Design: EDC leverages a larger effective receptive field while selectively applying deformations at edge regions.This design is presented as the mechanism underlying the reported improvements over other deformable convolution counterparts.
  • Evidence: Ablation studies and visualizations report superior performance across various kernel sizes.The conclusion uses these findings to characterize the effectiveness of EDC for larger-kernel models.
  • Conclusion: The findings suggest that EDC enhances deformable convolutions in models with larger kernels.

A Online Appendix

The online appendix adds experimental analyses beyond the main text, including component settings and effective-receptive-field quantification.

  • Additional Ablations: The appendix reports additional ablations on edge-detection operators, COM ratios, logistic regression, and thresholds.
  • ERF Analysis: The appendix includes effective-receptive-field quantification on ResNet34.

A.1 Ablation Study on Edge Detection Operators

The EMM compares Sobel, Laplace, and Prewitt for edge detection, with Sobel achieving the best segmentation performance.

  • A.1 Ablation Study on Edge Detection Operators: Sobel achieves optimal performance among Sobel, Laplace, and Prewitt for edge detection in the EMM.The comparison is conducted to assess each operator’s influence on segmentation performance.
  • A.1 Ablation Study on Edge Detection Operators: Sobel is less noise-sensitive than Laplace because it uses first-derivative approximations with smoothing.Laplace’s second-derivative operation can introduce more false edges in noisy images.
  • A.1 Ablation Study on Edge Detection Operators: Sobel also localizes edges using gradient calculations across both horizontal and vertical directions within a 3 × 3 kernel.The passage contrasts this localization behavior with Prewitt’s 3 × 3 kernel.

A.2 Ablation Study on the “Ratio” in COM

The COM ablation finds that increasing the center-area proportion improves performance, supporting the K × K/(K−2) × (K−2) ratio used throughout the paper.

  • A.2 Ablation Study on the “Ratio” in COM: Testing performance rises steadily as the COM center-area proportion increases.The study evaluates K values of 7 and 9 with k values of 0, 1, 3, 5, and 7.
  • A.2 Ablation Study on the “Ratio” in COM: 19.2% higher Top1 accuracy is achieved by ‘7 × 7/5 × 5’ than by ‘7 × 7’.The same configuration is also 1.61% and 3.89% higher than ‘7 × 7/1 × 1’ and ‘7 × 7/3 × 3’, respectively.
  • A.2 Ablation Study on the “Ratio” in COM: The K × K/(K−2) × (K−2) ratio significantly improves performance and is therefore applied to all variants.The ratio defines the kernel size relative to the center area.

A.3 Ablation Study on Logistic Regression in EMM

The EMM ablation shows that sigmoid masking and binary logistic regression behave differently across kernel sizes, with binary regression becoming more effective as kernels grow.

  • A.3 Ablation Study on Logistic Regression in EMM: Sigmoid-only masking is more precise for ‘5 × 5/3 × 3’, whereas binary logistic regression performs better for ‘7 × 7/5 × 5’ and ‘9 × 9/7 × 7’.The reported reversal is attributed to inappropriate content awareness without binary logistic regression at larger kernels.
  • A.3 Ablation Study on Logistic Regression in EMM: Using sigmoid scores directly leaves near-zero-score offsets active, allowing less important information to influence deformations.Binary logistic regression removes these offsets through its activation behavior.
  • A.3 Ablation Study on Logistic Regression in EMM: Binary logistic regression becomes more effective as kernel size increases.The masks act as offset activations rather than merely reducing offset amplitudes.

A.4 Ablation Study on Threshold of Logistic Regression

The threshold ablation finds similar training performance across settings but identifies 0.5 as the empirically optimal threshold, while extreme values may overfit.

  • A.4 Ablation Study on Threshold of Logistic Regression: 92%–93% Top-1 accuracy is achieved by all threshold counterparts during training.The variants exhibit similar performance across the threshold sweep.
  • A.4 Ablation Study on Threshold of Logistic Regression: 0.5 is the empirically optimal threshold for binary logistic regression in the EMM.The analysis evaluates thresholds from 0.0 to 1.0.
  • A.4 Ablation Study on Threshold of Logistic Regression: Extremely high or low thresholds may lead to overfitting.At threshold 0.0 all image contents are treated as essential, whereas threshold 1.0 removes deformation entirely.

A.5 Ablation Study on ERF Quantification with ResNet34

The study quantifies effective receptive field visualizations on CIFAR-100 using ResNet34 with deformable variants substituted in stage 3. EDC achieves the best accuracy among the evaluated variants.

  • A.5 Ablation Study on ERF Quantification with ResNet34: Table 16 reports quantitative CIFAR-100 results supporting the effective receptive field visualizations.
  • A.5 Ablation Study on ERF Quantification with ResNet34: EDC achieves the best accuracy on the CIFAR-100 classification task with ResNet34.The experiment replaces 10 layers in stage 3 with deformable variants.
  • A.5 Ablation Study on ERF Quantification with ResNet34: The ablation compares five deformable convolution variants based on ResNet34, including DCN V1, DCN V2, DCN V3, Entire DCN, and EDC.DCN V1 and V2 denote Deformable ConvNets and More Deformable ConvNets, respectively; Entire DCN denotes Entire Deformable ConvNets.
Loading 2609.10387v1…