Source-linked AI summary
Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications
Yuwen Xiong, Zhiqi Li, Yuntao Chen, Feng Wang, Xizhou Zhu, Jiapeng Luo, Wenhai Wang, Tong Lu, Hongsheng Li, Yu Qiao, Lewei Lu, Jie Zhou, Jifeng Dai
TL;DR
DCNv4 addresses DCNv3’s speed and convergence limitations by removing softmax normalization and optimizing redundant memory access. It converges faster, exceeds 3× forward-speed acceleration, and improves speed and performance across backbones and vision tasks, including generative models.
Problem
DCNv3’s sparse deformable operator suffers from high memory-access overhead and unexpectedly slow initial convergence compared with other operators.
Method
DCNv4 combines unbounded dynamic aggregation weights obtained by removing softmax normalization with optimized memory access and kernel implementation.
Results
DCNv4 converges significantly faster than DCNv3 and accelerates forward speed by more than 3×, while FlashInternImage gains 50 ∼80% speed without additional modifications.
Takeaways & Limitations
DCNv4 functions as a fast, effective operator across vision backbones and tasks, including ConvNeXt, ViT, and latent diffusion models.
Takeaways & Limitations
The speed analysis assumes batch size one and channel-last memory layout.
Abstract
from arXiv · showhide
We introduce Deformable Convolution v4 (DCNv4), a highly efficient and effective operator designed for a broad spectrum of vision applications. DCNv4 addresses the limitations of its predecessor, DCNv3, with two key enhancements: 1. removing softmax normalization in spatial aggregation to enhance its dynamic property and expressive power and 2. optimizing memory access to minimize redundant operations for speedup. These improvements result in a significantly faster convergence compared to DCNv3 and a substantial increase in processing speed, with DCNv4 achieving more than three times the forward speed. DCNv4 demonstrates exceptional performance across various tasks, including image classification, instance and semantic segmentation, and notably, image generation. When integrated into generative models like U-Net in the latent diffusion model, DCNv4 outperforms its baseline, underscoring its possibility to enhance generative models. In practical applications, replacing DCNv3 with DCNv4 in the InternImage model to create FlashInternImage results in up to 80% speed increase and further performance improvement without further modifications. The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.
1. Introduction
DCNv4 rethinks deformable convolution to address DCNv3’s speed and convergence limitations, combining implementation optimizations with removal of softmax normalization. It delivers faster convergence and substantially higher speed while extending effectively across vision architectures and tasks.
- Motivation: DCNv3 can be slower than optimized dense global attention and converges more slowly initially, despite its sparse convolutional design.The analysis identifies extra overhead from sampling non-nearby locations as a major speed bottleneck.
- Design: DCNv4 optimizes redundant memory accesses after profiling found that memory access accounts for 99% of DCNv3’s cost.The profiling reports computation below 1%, motivating implementation-level optimization.
- Design: Removing softmax normalization gives spatial aggregation weights an unbounded range, enhancing DCNv4’s dynamic property and expressive power.The change avoids the bounded 0–1 range imposed by softmax normalization.
- Results: More than 3× forward-speed acceleration and significantly faster convergence distinguish DCNv4 from DCNv3.The improvement is presented as enabling DCNv4 to leverage its sparse structure more fully.
- Results: 50 ∼80% speed increase over InternImage is achieved by FlashInternImage after replacing DCNv3 with DCNv4 without additional modifications.FlashInternImage also improves convergence speed and downstream-task performance.
- Applications: DCNv4 performs on par while being much faster when replacing depthwise convolution or dense self-attention in ConvNeXt and ViT without hyperparameter adjustments.The paper also applies DCNv4 in generative models, including latent diffusion models.
2. Related Work
Related work positions convolution and its variants as central vision operators while emphasizing that practical latency depends strongly on memory access costs, not FLOPs alone.
- Core operators in vision models: Standard convolution remains the predominant operator underlying most computer vision architectures.Depthwise separable convolution separates spatial and channel operations and has supported lightweight, efficient models.
- Memory access cost: Memory access costs can determine practical model speed and latency beyond what FLOPs indicate.FlashAttention is cited as faster than vanilla attention despite higher FLOPs because it reduces high-bandwidth-memory accesses.
3. Method
DCNv4 redesigns deformable convolution by removing softmax normalization to allow unbounded dynamic weights and by reducing redundant memory access. Its implementation exploits shared group offsets and weights to improve GPU efficiency while preserving sparse spatial aggregation.
- 3.1. Rethinking the Dynamic Property in Deformable Convolution: DCNv3 applies softmax-normalized spatial aggregation weights to input-dependent sampling points within grouped deformable windows.Each group processes C′ = C/G channels, while offsets correspond to predefined grid locations.
- 3.1. Rethinking the Dynamic Property in Deformable Convolution: Removing softmax gives DCNv4 unbounded dynamic aggregation weights, increasing expressive power for operators with dedicated windows.The paper contrasts this with attention and DCNv3, whose dynamic weights are bounded from 0 to 1.
- 3.1. Rethinking the Dynamic Property in Deformable Convolution: ConvNeXt experiments show that softmax-normalizing depthwise-convolution weights causes a marked decline in performance and convergence speed.This supports using unbounded aggregation weights for convolutional and deformable-convolution operators with location-specific windows.
- 3.2. Speeding up DCN: DCNv3’s theoretical cost is 36HWC FLOPs, while its ideal and cache-free memory-access costs are approximately 3.7HWC and 64HWC, respectively.The cache-free estimate is 17 times larger than the ideal case, exposing substantial memory-access overhead.
- 3.2. Speeding up DCN: DCNv4 assigns one thread to multiple same-group channels sharing offsets and weights, reducing repeated memory reads and bilinear-interpolation coefficient computation.Vectorized loads and writes merge memory instructions when channel values are contiguous, improving memory-bandwidth utilization.
- 3.2. Speeding up DCN: DCNv4 surpasses commonly used operators across input resolutions in the op-level benchmark.The benchmark reports FP32 and FP16 results where implementations are available and varies the downsample rate.
4. Experiments
Experiments evaluate DCNv4 at operator and system levels across classification, segmentation, detection, and diffusion settings. DCNv4 consistently improves speed while maintaining or improving performance over relevant baselines.
- 4.1. Speed Benchmark for Operators: More than 3× speedup over DCNv3 demonstrates that DCNv4 substantially reduces operator runtime.The benchmark covers standard- and high-resolution inputs and reports FP32 and FP16 results.
- 4.2. Image Classification: 50% ∼80% higher throughput than InternImage, with slightly improved performance, makes FlashInternImage competitive with ConvNeXt at higher accuracy.FlashInternImage-S reaches 84.4% versus ConvNeXt-B’s 83.8% while remaining faster.
- 4.3. Downstream Tasks with High-Resolution Input: 80% −90% faster than larger InternImage-S/B models, FlashInternImage-T/S achieves a higher speed-accuracy tradeoff in COCO instance segmentation.The comparison holds across Mask R-CNN and Cascade Mask R-CNN under 1× and 3× schedules.
- 4.3. Downstream Tasks with High-Resolution Input: FlashInternImage improves semantic segmentation performance and speed across model scales, resulting in a new state-of-the-art.The evaluation uses UperNet on ADE20K.
- 4.3. Downstream Tasks with High-Resolution Input: 50% −90% faster than InternImage baselines, or 200% −300% faster for the backbone alone, FlashInternImage maintains on-par NDS and higher mAP on nuScenes.The overall measurements include an underoptimized BEVFormer v2 head.
- 4.4. Generative Models: Replacing attention or regular 3 × 3 convolution in a latent diffusion U-Net, DCNv4 achieves better FID/Throughput with fewer parameters.This demonstrates applicability beyond perception tasks.
5. Conclusion
The paper presents DCNv4 as an efficient dynamic and sparse operator that improves speed and effectiveness over DCNv3. Its integration into multiple architectures and latent diffusion models supports broad applicability across vision tasks.
- 5. Conclusion: DCNv4 rethinks deformable convolution’s dynamic property and streamlines memory access to improve speed and effectiveness over DCNv3.The operator is presented as dynamic and sparse.
- 5. Conclusion: FlashInternImage improves speed and performance across various vision tasks when DCNv4 replaces DCNv3 in InternImage.The conclusion also reports improved throughput and accuracy in ConvNeXt and ViT integrations.
- 5. Conclusion: DCNv4 works well in latent diffusion models, indicating potential for enhancing generative models.The conclusion identifies generative modeling as an additional application domain.
A. Implementation Details
The implementation details specify hardware, software, training schedules, datasets, and task-specific optimization settings used in the experiments.
- Environment: Throughput benchmarks use an NVIDIA A100 80GB SXM GPU with PyTorch 1.13, CUDA 11.7, and cuDNN 8.5.FlashAttention uses version 2.3.1, and Window Attention uses timm 0.9.7.
- 2D object detection on COCO: COCO detection experiments use Mask R-CNN and Cascade Mask R-CNN with 1× and 3× schedules of 12 and 36 epochs.The base learning rate is 1e-4 with batch size 16 and AdamW optimization.
- 2D semantic detection on ADE20K: ADE20K semantic segmentation uses UperNet with batch size 16 and crop sizes of 512 for FlashInternImage-T/S/B and 640 for FlashInternImage-L.Learning rates are 6e-5 for T/S/B and 2e-5 for L.
- 3D object detection on nuScenes: nuScenes 3D detection uses BEVFormerV2 with COCO-pretrained backbones, batch size 16, learning rate 4e-4, and 24 training epochs.The data spans 8 seconds of past and future information.
B. Additional Experimental Results
Additional experiments test advanced task heads, alternative backbones, operator configurations, and image-generation applications. These results show competitive or improved accuracy alongside faster inference, including in diffusion models.
- Downstream results with advanced headers: With advanced DINO and Mask2Former heads, FlashInternImage maintains a significant accuracy advantage while offering competitive inference speed.The experiments use advanced downstream headers to further validate the backbone.
- Downstream results for other backbones: Replacing DWConv in ConvNeXt with DCNv4 increases inference speed and performance, while replacing ViT attention increases speed at comparable performance.These are drop-in substitutions without changing the surrounding architecture and hyperparameters.
- Drop-in replacement in diffusion model: DCNv4 works well for image generation and achieves better FID/Throughput with fewer parameters than regular convolution in a latent diffusion U-Net.Qualitative latent diffusion results are also presented.
- Module-level speed benchmark: The operator benchmark compares DCNv4 with attention, window attention, depthwise convolution, and regular convolution while accounting for projection-layer variants.The lightweight DCNv4 variant is used for ConvNeXt experiments, while the projected variant is used elsewhere.