Source-linked AI summary
Learning to Upsample by Learning to Sample
Wenze Liu, Hao Lu, Hongtao Fu, Zhiguo Cao
TL;DR
Existing dynamic upsamplers can impose substantial computational workload and may depend on high-resolution guidance. DySample reformulates upsampling as content-aware point sampling implemented with PyTorch operations, and it reports better performance with lower resource use across five dense prediction tasks.
Problem
Kernel-based dynamic upsamplers can require time-consuming dynamic convolution, extra kernel-generation sub-networks, and sometimes high-resolution feature guidance.
Method
DySample bypasses dynamic convolution by generating content-aware sampling offsets and re-sampling a bilinearly interpolated continuous feature map with PyTorch's grid sample function.
Results
DySample reports better performance than other upsamplers across semantic segmentation, object detection, instance segmentation, panoptic segmentation, and monocular depth estimation, with lower computational costs.
Takeaways & Limitations
DySample is presented as a fast, effective, and universal dynamic upsampler that can replace nearest-neighbor or bilinear interpolation in existing dense prediction models.
Takeaways & Limitations
Strictly constraining offset scope to [−0.25, 0.25] with tanh performed worse and limited representation power in cases requiring larger shifts.
Abstract
from arXiv · showhide
We present DySample, an ultra-lightweight and effective dynamic upsampler. While impressive performance gains have been witnessed from recent kernel-based dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much workload, mostly due to the time-consuming dynamic convolution and the additional sub-network used to generate dynamic kernels. Further, the need for high-res feature guidance of FADE and SAPA somehow limits their application scenarios. To address these concerns, we bypass dynamic convolution and formulate upsampling from the perspective of point sampling, which is more resource-efficient and can be easily implemented with the standard built-in function in PyTorch. We first showcase a naive design, and then demonstrate how to strengthen its upsampling behavior step by step towards our new upsampler, DySample. Compared with former kernel-based dynamic upsamplers, DySample requires no customized CUDA package and has much fewer parameters, FLOPs, GPU memory, and latency. Besides the light-weight characteristics, DySample outperforms other upsamplers across five dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, and monocular depth estimation. Code is available at https://github.com/tiny-smart/dysample.
1. Introduction
Feature upsampling is essential in dense prediction, but existing dynamic approaches can be computationally heavy and may require high-resolution guidance. DySample reformulates upsampling as efficient content-aware point sampling and reports strong performance with low resource use.
- Motivation: Dynamic upsamplers improve flexibility over fixed interpolation but can involve dynamic convolution, complicated structures, customized CUDA, and high-resolution guidance.FADE and SAPA additionally require higher-resolution features, narrowing their application scenarios.
- Method: DySample bypasses dynamic convolution by interpolating features continuously and re-sampling them at content-aware points generated from learned offsets.The offsets are produced by linear projection and applied with PyTorch's grid sample function.
- Results: 46% more performance improvement than CARAFE is reported for DySample with MaskFormer-SwinB, while using 3% of CARAFE's parameters and 20% of its FLOPs.Inference takes 6.2 ms versus 1.6 ms for bilinear interpolation on a 256 × 120 × 120 feature map.
- Conclusion: DySample is presented as a lightweight replacement for nearest-neighbor or bilinear interpolation in existing dense prediction models.The stated rationale combines effectiveness with efficiency.
2. Related Work
Related work spans dense prediction tasks, feature upsampling operators, and dynamic sampling methods. DySample follows the point-sampling perspective while targeting a simpler and more efficient dynamic upsampler than kernel-based alternatives.
- Dense Prediction Tasks: Dense prediction tasks require point-wise label prediction, including semantic, instance, and panoptic segmentation, object detection, and monocular depth estimation.These tasks have distinct challenges, such as preserving smooth interiors while representing sharp edges.
- Feature Upsampling: Upsampling is essential because low-resolution multi-scale features must be restored to higher resolution in dense prediction models.A lightweight, effective upsampler can therefore benefit multiple dense prediction architectures and tasks.
- Feature Upsampling: Nearest-neighbor and bilinear interpolation use fixed rules, whereas learnable operators add parameters and dynamic methods make upsampling content-aware.CARAFE generates dynamic convolution kernels, while FADE and SAPA use high- and low-resolution features to generate kernels.
- Dynamic Sampling: Dynamic sampling replaces standard grid sampling with dynamically selected points to model geometric information.Related examples include deformable convolution, deformable attention, and content-aware image resizing.
- DySample: DySample interprets upsampling as point re-sampling and uses simple designs to achieve an efficient dynamic upsampler.This perspective distinguishes it from recent kernel-based upsamplers.
3. Learning to Sample and Upsample
DySample reformulates dynamic upsampling as content-aware point sampling, then strengthens a naive design through better initialization, constrained offsets, grouping, and scope modulation. The resulting variants improve prediction performance while reducing computational and memory costs.
- Preliminary: DySample re-samples a bilinearly interpolated feature map at content-aware points generated by learned offsets and the original sampling grid.The sampling set is formed by adding offsets to grid positions, then passed to PyTorch grid sampling.
- Preliminary: 37.9 AP and 41.9 mIoU were obtained by the naive design on Faster R-CNN object detection and SegFormer-B1 semantic segmentation, respectively.The corresponding CARAFE results were 38.6 AP and 42.8 mIoU.
- DySample: Upsampling by Dynamic Sampling: Bilinear initialization improves performance to 38.1 (+0.2) AP and 42.1 (+0.2) mIoU by distributing the s^2 initial sampling positions evenly.Nearest initialization shares one starting position and ignores relations among neighboring upsampled points.
- DySample: Upsampling by Dynamic Sampling: A 0.25 static scope factor constrains local offset movement and improves performance to 38.3 (+0.2) AP and 42.4 (+0.3) mIoU.The constraint addresses overlapping offset scopes, which can disorder boundary values and propagate prediction artifacts.
- DySample: Upsampling by Dynamic Sampling: Dynamic scope modulation further raises performance to 38.7 (+0.1) AP and 43.3 (+0.1) mIoU, while grouping reaches 38.6 (+0.3) AP and 43.2 (+0.8) mIoU at g = 4.The dynamic scope is generated point-wise and modulates offsets; grouping shares sampling sets within feature groups.
- Complexity Analysis: DySample variants require 6.2 ∼7.6 ms to upsample a 256 × 120 × 120 feature map, while PL can reduce parameters to 1/s^4 under fixed hyperparameters.The series also has the least reported latency, memory, training time, GFLOPs, and parameters among tested strong dynamic upsamplers.
- DySample: Upsampling by Dynamic Sampling: DySample uses one dynamically sampled point per upsampled position when the s^2 upsampled points are dynamically divided.This design supports a lightweight point-sampling formulation rather than dynamic convolution.
4. Applications
DySample is evaluated across five dense prediction tasks, where it generally improves task performance while differing in boundary behavior for semantic segmentation. The experiments use established task-specific baselines and modify only their upsampling stages.
- Semantic Segmentation: DySample achieves the best mIoU of 43.58 on SegFormer-B1, although guided upsamplers obtain higher bIoU for boundary quality.The results suggest DySample mainly improves interior regions, whereas guided upsamplers mainly improve boundary quality.
- Semantic Segmentation: DySample improves MaskFormer mIoU from 52.70 to 53.91 (+1.21) with Swin-B and from 54.10 to 54.90 (+0.80) with Swin-L.
- Object Detection and Instance Segmentation: With R50, DySample achieves the best performance among tested upsamplers, with Faster R-CNN box AP gains of +1.2 for R50 and +1.1 for R101.On Mask R-CNN, mask AP gains are +1.0 for R50 and +0.8 for R101.
- Panoptic Segmentation: DySample provides consistent panoptic segmentation gains, improving PQ by 1.2 with an R50 backbone and 0.8 with R101.
- Monocular Depth Estimation: DySample+ improves depth estimation over bilinear upsampling by 0.05 in δ < 1.25 accuracy, while reducing Abs Rel by 0.04 and RMS by 0.09.The qualitative comparison also reports accurate, consistent depth for a chair.
5. Conclusion
The paper concludes that DySample is a fast, effective, universal dynamic upsampler based on point sampling rather than common kernel-based dynamic upsampling. It reports strong performance with low computational resource use and identifies low-level tasks as future work.
- DySample reformulates dynamic upsampling from point sampling and is developed progressively from a naive design.
- DySample reports the best performance while using no customised CUDA packages and the least resources across latency, memory, training time, GFLOPs, and parameters.
- Future work will apply DySample to low-level tasks and study joint modeling of upsampling and downsampling.