PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency Dynamic Convolution for Dense Image Prediction

Linwei Chen, Lin Gu, Liang Li, Chenggang Yan, Ying Fu

arXiv:2503.18783v2cs.CVcs.AI

TL;DR

Dynamic convolution can incur high parameter costs while its parallel weights retain limited frequency diversity, restricting frequency-adaptive representation. FDConv addresses this gap with Fourier-disjoint weights plus spatial and frequency-band modulation. Across detection, segmentation, and classification, it outperforms prior methods with modest parameter growth and broad architectural compatibility.

  • Problem

    Dynamic convolution’s parallel weights often have similar frequency responses despite increased parameter costs, limiting adaptive capture of different frequency information.

  • Method

    FDConv divides a fixed Fourier-domain parameter budget into disjoint frequency groups and adds Kernel Spatial Modulation and Frequency Band Modulation.

  • Results

    +3.6M parameters yields superior ResNet-50 performance, while experiments across detection, segmentation, and classification outperform prior dynamic-convolution methods with higher overhead.

  • Takeaways & Limitations

    FDConv provides a frequency-diverse dynamic-convolution design that integrates into ConvNets and vision transformers for a range of vision tasks.

  • Takeaways & Limitations

    Isolating specific spatial-domain frequency bands can require infinitely large kernels because an ideal low-pass filter has infinite spatial support.

Abstract

from arXiv · show

While Dynamic Convolution (DY-Conv) has shown promising performance by enabling adaptive weight selection through multiple parallel weights combined with an attention mechanism, the frequency response of these weights tends to exhibit high similarity, resulting in high parameter costs but limited adaptability. In this work, we introduce Frequency Dynamic Convolution (FDConv), a novel approach that mitigates these limitations by learning a fixed parameter budget in the Fourier domain. FDConv divides this budget into frequency-based groups with disjoint Fourier indices, enabling the construction of frequency-diverse weights without increasing the parameter cost. To further enhance adaptability, we propose Kernel Spatial Modulation (KSM) and Frequency Band Modulation (FBM). KSM dynamically adjusts the frequency response of each filter at the spatial level, while FBM decomposes weights into distinct frequency bands in the frequency domain and modulates them dynamically based on local content. Extensive experiments on object detection, segmentation, and classification validate the effectiveness of FDConv. We demonstrate that when applied to ResNet-50, FDConv achieves superior performance with a modest increase of +3.6M parameters, outperforming previous methods that require substantial increases in parameter budgets (e.g., CondConv +90M, KW +76.5M). Moreover, FDConv seamlessly integrates into a variety of architectures, including ConvNeXt, Swin-Transformer, offering a flexible and efficient solution for modern vision tasks. The code is made publicly available at https://github.com/Linwei-Chen/FDConv.

1. Introduction

FDConv addresses limited frequency diversity and high parameter costs in dynamic convolution by constructing frequency-diverse weights in the Fourier domain and adding spatial and frequency-band modulation. Experiments report stronger performance across vision tasks with modest parameter growth and integration across ConvNet and transformer architectures.

  • Motivation: Dynamic convolution uses multiple attention-mixed weights for sample-specific adaptation, but its parallel weights can have highly similar frequency responses despite increased parameters.This limits adaptive capture of low-frequency noise-suppressing components and high-frequency details or boundaries.
  • FDConv: Kernel Spatial Modulation adjusts each filter element using local and global channel information to fine-tune frequency responses across the kernel.KSM generates a dense matrix of modulation values for element-wise control.
  • FDConv: Frequency Band Modulation decomposes weights into frequency bands and modulates them independently across spatial locations based on local content.This allows different regions to emphasize or suppress selected frequency bands.
  • FDConv: FDConv uses Fourier Disjoint Weight to divide Fourier-domain parameters into disjoint frequency groups, producing many frequency-diverse kernels under a fixed parameter budget.An inverse discrete Fourier transform converts grouped spectral coefficients into spatial weights.
  • Results: +3.6M parameters accompanies FDConv’s superior ResNet-50 performance, compared with +90M for CondConv and +76.5M for KW.The method also integrates into ConvNeXt and Swin Transformer architectures.
  • Results: FDConv is presented as an integrated approach combining FDW, KSM, and FBM, with experiments across detection, instance segmentation, semantic segmentation, and classification.The contribution claims improved performance with only a minor parameter increase.

2. Related Work

Prior work adapts convolution through feature or weight attention, decomposition, and frequency-domain analysis. FDConv instead targets dynamic convolution’s parameter cost and limited weight diversity from a frequency-based perspective.

  • Feature Recalibration: Feature recalibration methods adaptively emphasize or suppress features across channel and spatial dimensions, whereas FDConv recalibrates convolution weights specifically by frequency.The related methods include RAN, SE, CBAM, GE, SRM, ECA, and SimAtt.
  • Dynamic Convolution: Dynamic convolution learns multiple parallel weights and mixes them with attention, while CondConv uses sigmoid fusion and DY-Conv uses softmax fusion.ODConv further predicts channel-wise, filter-wise, and spatial attention factors.
  • Efficient Dynamic Convolution: DCD and PEDConv reduce overhead through low-rank matrix decomposition, while KW reconstructs dynamic kernels from smaller shareable units across stages and layers.These approaches address parameter or computational costs through decomposition and sharing.
  • Frequency Domain Learning: Frequency-domain techniques have been applied to optimization, robustness, generalization, and feature-dependent dilation, while FDConv applies frequency analysis to diversify dynamic-convolution weights.FDConv is positioned as a frequency-based solution for capturing a wider range of frequency information.

3. Method

FDConv builds frequency-diverse dynamic weights with a fixed parameter budget, then adds fine-grained kernel and spatial frequency modulation for greater adaptability.

  • Fourier Disjoint Weight: FDW transforms each disjoint Fourier group with iDFT, crops the spatial result into k × k patches, and reassembles them into k × k × Cin × Cout tensors.The construction supports a large number of diversified weights, with n > 10 possible without increasing the parameter cost.
  • Fourier Disjoint Weight: Dynamic convolution combines parallel weights using input-dependent attention, but its learned weights can have highly similar frequency responses despite increased parameter costs.This frequency redundancy limits adaptive capture of features across different frequency bands.
  • Fourier Disjoint Weight: FDW divides a fixed parameter budget into groups with disjoint Fourier indices, producing multiple weights with distinct frequency responses without increasing parameter cost.Each group is transformed to spatial weights with iDFT and reassembled into convolution kernels.
  • Architecture Integration: FDW can also replace linear layers in Transformer architectures because they are equivalent to convolutions with kernel size 1.The method is presented as applicable across convolutional and modern vision architectures.
  • Kernel Spatial Modulation: KSM predicts a dense modulation matrix for each kernel by fusing local channel information from a lightweight 1-D convolution with global channel information from a fully connected layer.This replaces coarse weight-wise adjustment with element-level modulation across kernel spatial and channel dimensions.
  • Frequency Band Modulation: FBM decomposes weights into frequency bands and dynamically modulates those bands at each spatial location, enabling selective emphasis or suppression based on local content.Binary frequency masks isolate bands, while the equivalent implementation can instead filter features into sub-bands before applying the full kernel.

4. Experiment

The evaluation uses standard benchmarks and established model settings to assess FDConv across semantic segmentation, object detection, and instance segmentation.

  • FDConv is evaluated on Cityscapes and ADE20K for semantic segmentation, using mIoU as the metric.
  • Object detection and instance segmentation are evaluated with Average Precision (AP).
  • Experiments follow the original settings for UPerNet, Mask2Former, MaskDINO, Swin Transformer, and ConvNeXt.

5. Main Results

FDConv is tested across detection and segmentation benchmarks and integrates with convolutional and transformer architectures. It delivers strong performance while requiring substantially less parameter overhead than competing dynamic convolution methods.

  • FDConv is evaluated on object detection, instance segmentation, and semantic segmentation using COCO, ADE20K, and Cityscapes.
  • FDConv achieves the highest performance across detection and segmentation tasks while substantially reducing parameter overhead.
  • Object Detection: +3.6M parameters and +1.8G FLOPs yield an APbox of 39.4 for Faster R-CNN with FDConv.
  • Instance Segmentation: FDConv achieves an APbox of 42.4 and APmask of 38.6 with Mask R-CNN, while adding only 3.6M parameters.
  • Semantic Segmentation: FDConv reaches 43.8 single-scale mIoU with UPerNet while using 70M total parameters, compared with 131M for ODConv and 141M for KW.
  • Advanced Architectures: FDConv improves mIoU by +1.0 on Cityscapes with Mask2Former-ResNet-50 and by +1.0 on ADE20K with Mask2Former-Swin-B.

6. Analyses and Discussion

Analysis shows that FDConv produces more diverse kernels and frequency responses than existing dynamic convolution methods. Its frequency-band modulation concentrates higher-frequency emphasis near boundaries while suppressing unnecessary high-frequency content elsewhere.

  • Weight Similarity: FDConv kernels exhibit zero cosine similarity, indicating unique and complementary features compared with highly similar ODConv kernels.
  • Weight Frequency Analysis: FDConv demonstrates more diversified frequency responses across stages, capturing a broader range of frequency characteristics.
  • Feature Visualization for FBM: Higher frequency-band modulation values concentrate around object boundaries and become more pronounced in higher frequency bands.
  • Feature Visualization for FBM: Selective modulation suppresses high frequencies in backgrounds and object centers, reducing high-frequency noise while enhancing foreground features.

7. Conclusion

FDConv addresses restricted frequency diversity and high parameter costs in dynamic convolution through FDW, KSM, and FBM. Experiments show greater frequency diversity, strong results across vision tasks, and compatibility with convolutional and transformer architectures.

  • FDConv combines FDW, KSM, and FBM to enhance frequency adaptability without increasing parameter overhead.
  • FDConv achieves greater frequency diversity and better feature capture across spatial and frequency domains.
  • FDConv outperforms prior state-of-the-art methods on object detection, segmentation, and classification with only a modest parameter increase.
  • FDConv integrates into ConvNets and vision transformers for a wide range of computer vision tasks.
Loading 2503.18783v2…