PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency-Adaptive Dilated Convolution for Semantic Segmentation

Linwei Chen, Lin Gu, Ying Fu

arXiv:2403.05369v7cs.CV

TL;DR

Dilated convolution must balance receptive-field expansion against loss of effective bandwidth for high-frequency content. FADC addresses this by adapting dilation to local frequency and adding modules that tune kernel and feature frequencies; experiments report consistent gains across segmentation and detection, including 81.0 mIoU at 37.7 FPS with PIDNet-M.

  • Problem

    Increasing dilation expands the receptive field but reduces bandwidth and can limit high-frequency information capture.

  • Method

    FADC combines frequency-based spatial dilation adaptation with AdaKern’s per-channel kernel-frequency adjustment and FreqSelect’s spatial feature-frequency reweighting.

  • Results

    FADC consistently improves segmentation and object detection performance, including 81.0 mIoU at 37.7 FPS with PIDNet-M.

  • Takeaways & Limitations

    Frequency-aware control can jointly improve effective bandwidth and receptive-field use across segmentation, detection, deformable convolution, and dilated attention.

Abstract

from arXiv · show

Dilated convolution, which expands the receptive field by inserting gaps between its consecutive elements, is widely employed in computer vision. In this study, we propose three strategies to improve individual phases of dilated convolution from the view of spectrum analysis. Departing from the conventional practice of fixing a global dilation rate as a hyperparameter, we introduce Frequency-Adaptive Dilated Convolution (FADC), which dynamically adjusts dilation rates spatially based on local frequency components. Subsequently, we design two plug-in modules to directly enhance effective bandwidth and receptive field size. The Adaptive Kernel (AdaKern) module decomposes convolution weights into low-frequency and high-frequency components, dynamically adjusting the ratio between these components on a per-channel basis. By increasing the high-frequency part of convolution weights, AdaKern captures more high-frequency components, thereby improving effective bandwidth. The Frequency Selection (FreqSelect) module optimally balances high- and low-frequency components in feature representations through spatially variant reweighting. It suppresses high frequencies in the background to encourage FADC to learn a larger dilation, thereby increasing the receptive field for an expanded scope. Extensive experiments on segmentation and object detection consistently validate the efficacy of our approach. The code is publicly available at https://github.com/Linwei-Chen/FADC.

1. Introduction

The paper reframes dilated convolution as a frequency-dependent trade-off between effective bandwidth and receptive field, then proposes FADC to adapt this trade-off spatially. FADC combines AdaDR, AdaKern, and FreqSelect to adjust dilation, kernel frequency response, and feature frequency balance.

  • 1. Introduction: Large dilation rates expand the receptive field but reduce bandwidth, limiting high-frequency capture and potentially causing gridding artifacts.Increasing dilation from 1 to D scales the kernel and bandwidth by 1/D.
  • 1. Introduction: FADC replaces globally fixed dilation with spatially adaptive rates based on local frequency components.AdaDR uses small dilation for high-frequency regions and larger dilation for low-frequency regions.
  • 1. Introduction: AdaKern decomposes convolution weights into low- and high-frequency components and dynamically adjusts their per-channel ratio to enhance effective bandwidth.Increasing the high-frequency component strengthens high-frequency responses.
  • 1. Introduction: FreqSelect spatially reweights feature frequencies, suppressing high-frequency background content to encourage larger dilations and expand the receptive field.The module balances high- and low-frequency components before dilated convolution.
  • 1. Introduction: With PIDNet, FADC achieves 81.0 mIoU at 37.7 FPS on Cityscapes while also improving integrated deformable-convolution and dilated-attention models.The reported results support gains across segmentation and object detection settings.

2. Related work

Related work adapts convolution through content-dependent weights or sampling grids, while prior adaptive dilation methods rely on intuitive assumptions. The paper instead motivates dilation adaptation through local frequency analysis and distinguishes its approach from deformable convolution’s spatial offsets.

  • 2. Related work: Content-adaptive vision methods adjust weights or attention values based on the input and often target large receptive fields.The passage situates input-adaptive weighting and attention within broader vision architectures.
  • 2. Related work: Deformable convolution modifies sampling grids with learned offsets, which can introduce spatial deviations in extracted features.The cited description contrasts offset-based sampling with dilation-based approaches.
  • 2. Related work: Prior adaptive dilation methods discard globally fixed dilation but use learned weights or assumptions about inter-layer patterns and object scale.The paper contrasts these assumptions with its frequency-based rationale.
  • 2. Related work: Research on aliasing artifacts extends from insufficient downsampling to applications including vision transformers, tiny-object detection, and image generation.The related-work discussion places dilated-convolution aliasing within a broader anti-aliasing literature.

3. Frequency Adaptive Dilated Convolution

FADC analyzes dilated convolution in the frequency domain and adapts its dilation, kernel spectrum, and feature frequencies to balance effective bandwidth with receptive field size. Its three strategies respectively assign dilation spatially, adjust kernel frequency components, and reweight feature bands.

  • Frequency analysis: Dilated convolution enlarges receptive fields, but increasing dilation reduces frequency response and bandwidth, limiting high-frequency capture above the Nyquist frequency.The resulting trade-off can produce gridding artifacts when feature frequencies exceed the sampling rate.
  • Adaptive Dilation Rate: AdaDR assigns dilation rates spatially according to local frequency content, using smaller rates for high-frequency regions and larger rates for smoother regions.The strategy directly balances effective bandwidth against receptive field size for spatially variant feature maps.
  • Adaptive Kernel: AdaKern decomposes convolution weights into low-frequency averages and high-frequency residuals, then dynamically adjusts their channel-wise ratio.The low-frequency component is associated with a mean filter, while the residual captures local differences and high-frequency components.
  • Frequency Selection: FreqSelect decomposes features into Fourier-domain frequency bands and spatially reweights those bands to produce frequency-balanced representations.The implementation uses predefined thresholds and four octave-wise bands spanning [0, 1/16), [1/16, 1/8), [1/8, 1/4), and [1/4, 1/2].

4. Experiments

Experiments evaluate FADC across semantic segmentation, real-time segmentation, object detection, and related architectures. The reported results show improvements across these settings, including a strong Cityscapes speed–accuracy result and gains when integrating the plug-in strategies.

  • Real-time Semantic Segmentation: 81.0 mIoU at 37.7 FPS establishes FADC-enhanced PIDNet-M as faster than PIDNet-L while achieving the reported accuracy.The comparison reports 37.7 versus 31.1 FPS for PIDNet-M and PIDNet-L, respectively.
  • Integration with Related Architectures: AdaKern and FreqSelect improve DCNv2, InternImage, and DiNAT across object detection and segmentation evaluations.The reported gains include +0.9 box AP for DCNv2, +0.8 on ADE20K for InternImage, and +0.6 mask AP on COCO for DiNAT.
  • Visualized Results: Visualizations show FADC capturing thin poles and responding more uniformly to large trucks than standard dilated convolution.The comparison attributes the baseline failures to missing high-frequency detail and an insufficient receptive field, respectively.

5. Analysis and Disccusion

The analysis links FADC’s behavior to frequency-aware dilation, receptive-field expansion, attention reweighting, and high-frequency feature extraction. AdaDR adapts sampling to local frequency content, while FreqSelect and AdaKern respectively encourage larger receptive fields and stronger bandwidth.

  • Analysis of AdaDR: AdaDR assigns small dilation rates to high-frequency boundaries and larger rates to smoother regions, while avoiding the spatial deviation reported for deformable convolution.This preserves bandwidth for fine details and expands the receptive field in low-frequency areas.
  • Receptive Field: FreqSelect suppresses high frequencies in backgrounds and object centers, encouraging higher dilation rates and an enlarged receptive field.The analysis reports that spatially variant frequency weighting contributes to increased average dilation rates.
  • Bandwidth: FADC extracts more high-frequency information than dilated convolution, and AdaKern further increases power in the [1/4, 1/2] frequency band.The reported frequency analysis uses extracted feature power as a direct assessment of bandwidth-related behavior.
  • Aliasing Artifacts: Aliasing artifacts arise when feature frequencies exceed dilated convolution’s effective bandwidth, impairing fine-detail representation.The analysis connects this failure mode to gridding artifacts in dilated convolution.

6. Conclusion

The conclusion presents FADC as a frequency-based redesign of dilated convolution using adaptive dilation, adaptive kernels, and frequency selection. The reported scope includes segmentation, object detection, deformable convolution, and dilated attention, while future work targets quantitative analysis and architecture design.

  • Conclusion: FADC combines AdaDR, AdaKern, and FreqSelect to adapt dilation, convolution weights, and feature frequencies.AdaDR adjusts dilation spatially, AdaKern changes low- and high-frequency kernel components per channel, and FreqSelect reweights feature frequencies spatially.
  • Conclusion: The reported experiments validate FADC and its plug-in strategies across segmentation, object detection, deformable convolution, and dilated attention.The conclusion states that the strategies can replace standard convolution layers in existing architectures.
  • Conclusion: Future work will extend quantitative frequency analysis to deformable and dilated attention and design architectures specifically for FADC.These directions are stated as planned extensions beyond the current work.
Loading 2403.05369v7…