Source-linked AI summary
Decoupled Dynamic Filter Networks
Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, Ming-Hsuan Yang
TL;DR
Standard convolution is content-agnostic and computationally costly, while dynamic filters adapt to content at greater computational overhead. DDF decouples spatial and channel dynamic filters to provide a lightweight content-adaptive alternative. The paper reports improved classification performance and stronger upsampling performance for DDF-Up across detection and joint upsampling tasks.
Problem
Standard convolution is content-agnostic and costly, while dynamic filters add computation and depth-wise convolution can reduce performance or require more channels.
Source: §1 · pp. 1–2Method
Results
DDF consistently improves ResNet performance while reducing parameters and computational costs, and DDF-Up boosts performance in detection and joint depth upsampling.
Source: §7 · p. 8Takeaways & Limitations
DDF can replace standard convolution layers, while DDF-Up serves as a more computationally efficient upsampling alternative to specialized content-adaptive layers.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Convolution is one of the basic building blocks of CNN architectures. Despite its common use, standard convolution has two main shortcomings: Content-agnostic and Computation-heavy. Dynamic filters are content-adaptive, while further increasing the computational overhead. Depth-wise convolution is a lightweight variant, but it usually leads to a drop in CNN performance or requires a larger number of channels. In this work, we propose the Decoupled Dynamic Filter (DDF) that can simultaneously tackle both of these shortcomings. Inspired by recent advances in attention, DDF decouples a depth-wise dynamic filter into spatial and channel dynamic filters. This decomposition considerably reduces the number of parameters and limits computational costs to the same level as depth-wise convolution. Meanwhile, we observe a significant boost in performance when replacing standard convolution with DDF in classification networks. ResNet50 / 101 get improved by 1.9% and 1.3% on the top-1 accuracy, while their computational costs are reduced by nearly half. Experiments on the detection and joint upsampling networks also demonstrate the superior performance of the DDF upsampling variant (DDF-Up) in comparison with standard convolution and specialized content-adaptive layers.
1. Introduction
Standard convolution is content-agnostic and computationally expensive, while existing dynamic filters add substantial costs. DDF decouples spatial and channel filtering to combine content adaptation with lightweight computation, and the paper reports gains across classification and upsampling tasks.
1. Introduction
- Standard convolution shares filters across image pixels and images, while dynamic filters can be compute-intensive, memory-intensive, or require specialized processing units.These costs limit many dynamic filters to a few CNN layers, tiny architectures, or specific scenarios such as upsampling.Source: §1 · pp. 1–2
- Depth-wise and grouped convolutions can reduce computation, but direct replacement of standard convolution usually lowers performance or requires more channels and memory.DDF is presented as a lightweight drop-in alternative to standard convolution.Source: §1 · p. 2
- DDF predicts separate spatial and channel dynamic filters, combining them into a filter at each pixel with computational costs similar to depth-wise convolution.Unlike existing dynamic filters, DDF can replace all k × k convolutions in a CNN.Source: §1 · p. 2
- DDF consistently improves classification performance while reducing computational costs; DDF-Up outperforms standard convolution and specialized content-adaptive layers in upsampling tasks.The paper reports these findings for convolution-replacement experiments and for detection and joint upsampling networks.Source: §1 · p. 2
- DDF provides spatially varying filtering that adapts filters to image content.The paper also characterizes DDF as faster than standard convolution and dynamic filters, with lower memory consumption than dynamic filters.Source: §1 · p. 2
2. Related Work
Prior lightweight convolutions trade computation for performance, while many dynamic filters are costly or restricted in where they can be used. DDF draws on spatial and channel-wise attention to decouple its dynamic filters.
Lightweight convolutions
- Depth-wise and grouped convolutions reduce parameters and computation, but direct replacement of standard convolution usually lowers performance unless the model is widened.The paper positions DDF as a lightweight drop-in replacement for standard convolution.Source: §2 · p. 2
Dynamic filters
- Dynamic-filter methods vary filter neighborhoods or values; full per-pixel filters can be too costly for large networks, while some methods are spatially shared or task-specific.For example, CARAFE predicts channel-wise shared 2D filters for upsampling, whereas several full dynamic filters are limited to a few layers or small networks.Source: §2 · p. 2
Attention mechanisms
- Spatial and channel-wise attention are established design patterns, and DDF applies their decoupled structure to predict spatial and channel dynamic filters.The cited examples include parallel spatial and channel attention in BAM and sequential attention in CBAM.Source: §2 · p. 3
3. Preliminaries
The preliminaries contrast standard convolution’s shared, input-agnostic filters with dynamic filters that predict content-adaptive filters at each pixel. The latter’s large per-pixel filter predictions impose compute and memory costs.
Standard convolution
- Standard convolution computes each output pixel as a linear combination of features in its k × k neighborhood using a filter shared across pixels.The input feature map has c channels and n pixels, and the output has c′ channels.Source: §3 · p. 3
- In standard convolution, the same filter is used at every pixel and its weights do not depend on input features.The filter weights are indexed by the offset between the output pixel and each neighboring input pixel.Source: §3 · p. 3
Dynamic filters
- Dynamic filters replace the shared filter with a spatially varying filter predicted at each pixel, enabling content-adaptive feature embeddings.Predicting the resulting large number of pixel-wise filter values requires heavy side networks and makes these models compute- and memory-intensive.Source: §3 · p. 3
4. Decoupled Dynamic Filter
DDF makes filtering content-adaptive by decoupling dynamic filters into spatial and channel filters. This design keeps computational complexity near depth-wise convolution while reducing generated-filter memory relative to full dynamic filters.
4. Decoupled Dynamic Filter
- DDF decouples dynamic filtering into spatial and channel filters predicted from the input, reducing a full per-pixel filter to smaller spatial and channel filter tensors.The full dynamic filter has size nc′ck^2, compared with nk^2 spatial and ck^2 channel filters for DDF.Source: §4 · pp. 3–4
- DDF’s CUDA implementation avoids storing intermediate multiplied filters during training and inference.This implementation is described as alleviating the need to save those intermediate filters.Source: §4 · p. 4
- DDF filtering is equivalent to applying attention to unfolded features, where neighboring feature values become separate channels and generated filters reweight them.The spatial branch uses a 1 × 1 convolution, while the channel branch uses global average pooling and a squeeze-and-excitation structure.Source: §4 · p. 4
4.1. Computational Complexity.
- DDF has time complexity O(nck^2), similar to depth-wise convolution and better than standard convolution, while its filter-space complexity is O((n+c)k^2).Its generated-filter space can be hundreds or thousands of times smaller than full dynamic filters at commonly encountered channel or pixel counts.Source: §4.1 · p. 5
5. DDF Networks for Image Classification
The section evaluates DDF as a drop-in replacement in ResNet image-classification networks, retaining the original hyperparameters. Ablations examine its filter branches, normalization, and squeeze ratio, while comparisons assess accuracy and efficiency against related filters and ResNet variants.
5. DDF Networks for Image Classification
- DDF replaces the 3 × 3 spatial-embedding convolutions across stacked ResNet blocks while retaining the original hyperparameters, especially channel counts.DDF-ResNets are evaluated on ImageNet.Source: §5 · p. 5, Figure 4 · p. 5
- Replacing the proposed filter normalization with batch normalization or sigmoid activation leads to considerable accuracy drops.The authors suggest sigmoid may miss correlations between filter values, while batch normalization may weaken filter dynamics across samples.Source: §5 · p. 5
- Replacing convolution with channel-only filters improves top-1 accuracy by 1.6%, and full DDF by 1.9%; spatial-only filters cause a significant performance drop.The ablations use ResNet50 as the base network.Source: §5 · p. 5, Table 3 · p. 6
- Against related filters, DDF boosts base-model performance while reducing parameters and FLOPs; DwWeightNet performs worse than DDF and channel-only DDF.Adaptive convolution is compared only in ResNet18 because of its large running-memory consumption.Source: Table 4 · p. 6, §5 · p. 6
- Higher squeeze ratios substantially increase parameters for marginal performance gains; the default squeeze ratio is 0.2, while FLOPs remain low.The channel-filter branch has minimal computational cost.Source: §5 · p. 6
- DDF-ResNets achieve the best performance with the lowest number of parameters and FLOPs among compared ResNet variants; DDF-ResNet50 reaches 81.3% top-1 accuracy with training and evaluation tricks.The comparison includes ResNet50 and ResNet101 variants on ImageNet.Source: Table 5 · p. 6, §5 · p. 6
- DDF is proposed as a new fundamental building block for neural architecture search spaces.The paper presents this as a possible contribution to NAS methods.Source: §5 · p. 6
6. DDF as Upsampling Module
DDF-Up extends decoupled dynamic filtering to typical and guidance-based upsampling, and can be embedded in detection and joint depth upsampling networks. It improves reported results over common alternatives while reducing computation relative to specialized content-adaptive filtering.
6. DDF as Upsampling Module
- DDF-Up predicts spatial dynamic filters from separate guidance features, stacks and pixel-shuffles the resulting features, and uses depth-wise convolution to predict guidance for typical upsampling.The number of DDF operations is set according to the upsampling factor.Source: §6 · p. 7, Figure 5 · p. 7
- DDF-Up replaces nearest-neighbor upsampling in Faster R-CNN with FPN, embedding the module in FPN’s top-down feature-fusion pathway.The evaluation uses COCO minival and standard COCO metrics for small, medium, large, and all-scale objects.Source: §6 · p. 7
- FPN with DDF-Up improves all-scale mAp by 1.2% over nearest-neighbor interpolation and is on par with CARAFE using one-third of its FLOPs.It also improves on static-filtering Deconvolution and Pixel Shuffle upsamplers.Source: §6 · p. 7, Table 6 · p. 7
- For joint depth upsampling, DDF-Up combines low-resolution depth features with guidance features encoded from a high-resolution RGB image.Each module performs 2× upsampling, and k sequential modules yield a 2^k upsampling factor.Source: §6 · p. 8
- DDF-Up-Net performs better than state-of-the-art techniques at 4×, 8×, and 16× depth upsampling, while reducing computation by an order of magnitude versus PAC.At 16×, sample results show more recovered depth details than PAC-Net and other techniques.Source: §6 · p. 8, Table 7 · p. 8, Figure 7 · p. 8
7. Conclusion
The conclusion presents DDF as a lightweight content-adaptive filter based on decoupled spatial and channel dynamic filters. It reports improved ResNet performance with lower costs, and gains from DDF-Up in detection and joint depth upsampling.
7. Conclusion
- DDF predicts decoupled spatial and channel dynamic filters to provide lightweight content-adaptive filtering.The conclusion reports that it can replace standard convolution layers, improving ResNet performance while reducing parameters and computational costs.Source: §7 · p. 8