PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Upsample by Learning to Sample

Wenze Liu, Hao Lu, Hongtao Fu, Zhiguo Cao

arXiv:2308.15085v1cs.CV

TL;DR

Kernel-based dynamic upsamplers can require complex structures and high-resolution guidance, limiting their efficiency and application scenarios. DySample uses point sampling instead, and the paper reports best performance with the least computational resources among prior dynamic upsamplers.

  • Problem

    Kernel-based dynamic upsamplers can require complex structures and high-resolution guidance, which narrows their application scenarios.

    Source: §1 · pp. 1–2

  • Method

    DySample formulates dynamic upsampling as point sampling and develops the approach using PyTorch’s built-in function.

    Source: §3.1 · p. 3, §5 · p. 9

  • Results

    DySample reports the best performance and least computational resources among prior dynamic upsamplers, with inference taking 6.2–7.6 ms per 256 × 120 × 120 feature map.

    Source: §3.4 · p. 5, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude DySample can replace nearest-neighbor or bilinear interpolation in existing dense prediction models.

    Source: §1 · p. 2

    The soft offset-scope factor does not fully solve the scope problem, while strict tanh constraints performed worse and may limit representation power.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We present DySample, an ultra-lightweight and effective dynamic upsampler. While impressive performance gains have been witnessed from recent kernel-based dynamic upsamplers such as CARAFE, FADE, and SAPA, they introduce much workload, mostly due to the time-consuming dynamic convolution and the additional sub-network used to generate dynamic kernels. Further, the need for high-res feature guidance of FADE and SAPA somehow limits their application scenarios. To address these concerns, we bypass dynamic convolution and formulate upsampling from the perspective of point sampling, which is more resource-efficient and can be easily implemented with the standard built-in function in PyTorch. We first showcase a naive design, and then demonstrate how to strengthen its upsampling behavior step by step towards our new upsampler, DySample. Compared with former kernel-based dynamic upsamplers, DySample requires no customized CUDA package and has much fewer parameters, FLOPs, GPU memory, and latency. Besides the light-weight characteristics, DySample outperforms other upsamplers across five dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, and monocular depth estimation. Code is available at https://github.com/tiny-smart/dysample.

1. Introduction

DySample reformulates dynamic upsampling as content-aware point sampling, avoiding dynamic convolution and high-resolution guidance. It is presented as an efficient, effective alternative that outperforms other upsamplers across five dense prediction tasks.

1. Introduction

  • DySample outperforms other upsamplers across five dense prediction tasks while using 3% of CARAFE’s parameters and 20% of its FLOPs.On semantic segmentation with MaskFormer-SwinB as the baseline, it reports 46% more performance improvement than CARAFE. Its inference time is 6.2 ms versus 1.6 ms for bilinear interpolation when upsampling a 256 × 120 × 120 feature map.Source: Figure 1 · p. 1, §1 · p. 2
  • Kernel-based dynamic upsamplers such as CARAFE, FADE, and SAPA add computational workload, customized CUDA requirements, and inference time relative to bilinear interpolation.FADE and SAPA also use high-resolution guiding features, which add workload and restrict use to scenarios where those features are available.Source: §1 · pp. 1–2
  • Because modern architectures often combine higher-resolution features after upsampling, the paper argues that a single-input dynamic upsampler can be sufficient.DySample requires neither high-resolution guiding features nor extra CUDA packages beyond PyTorch.Source: §1 · p. 2
  • DySample generates content-aware point offsets and uses PyTorch grid sampling to resample a bilinearly interpolated feature map, refining the design through three stepwise tweaks.The tweaks control the initial sampling position, adjust the offsets’ moving scope, and divide upsampling into independent groups.Source: §1 · p. 2

2. Related Work

The section reviews dense prediction tasks and existing feature-upsampling and dynamic-sampling approaches. It frames DySample’s direction as point re-sampling for efficient dynamic upsampling.

Dense Prediction Tasks

  • Dense prediction tasks require point-wise labels and include segmentation [39] [40] [8] [7] [11] [16] [19], detection [33] [4] [24] [36], and depth estimation [38] [18] [3] [21], each with distinct challenges.Examples include predicting smooth interiors and sharp edges, distinguishing objects, and handling differing relationships between semantic meaning and depth.Source: §2 · p. 2
  • Because dense-prediction backbones produce multi-scale features, low-resolution features need upsampling, making lightweight, effective operators useful across tasks.The paper reports consistent gains with negligible workload across segmentation, detection, instance and panoptic segmentation, and monocular depth estimation models.Source: §2 · p. 2

Feature Upsampling

  • Nearest-neighbor and bilinear interpolation use fixed rules and ignore feature semantics; SegNet’s max unpooling preserves edges but can disrupt smooth regions with noise and zero filling.Learnable alternatives include deconvolution and Pixel Shuffle [34], which increases channels with convolution before reshaping features to higher resolution.Source: §2 · p. 2
  • CARAFE generates content-aware kernels with a sub-network, while FADE [29] and SAPA [30] use high-resolution guidance, adding complexity and limiting their use where such features are unavailable.SAPA also computes similarity-aware kernels between high- and low-resolution features using point affiliation.Source: §2 · p. 2

Dynamic Sampling

  • Prior geometric-sampling work includes deformable convolution [9] [43], deformable attention in Deformable DETR, and learned downsampling for content-aware image resizing [41] [15]; DySample instead treats upsampling as point re-sampling.This motivates simple designs for dynamic upsampling that preserve effectiveness while improving efficiency.Source: §2 · p. 3

3. Learning to Sample and Upsample

DySample reframes upsampling as content-aware point sampling: it predicts sampling positions and uses PyTorch grid sampling to resample features. The section develops this design from a naive implementation through refinements and compares its behavior, complexity, and relation to other methods.

3.1. Preliminary

  • The naive design predicts offsets from features, adds them to the original grid, and applies PyTorch grid sampling to produce the upsampled map.A linear layer generates offsets that are reshaped by Pixel Shuffling; grid sampling uses the resulting positions to resample the feature map.Source: §3.1 · p. 3

3.2. DySample: Upsampling by Dynamic Sampling

  • Setting the static scope factor to 0.25 improves performance to 38.3 (+0.2) AP and 42.4 (+0.3) mIoU; a strict tanh constraint on offset scope works worse.Source: §3.2 · p. 4
  • With four feature groups, performance reaches 38.6 (+0.3) AP and 43.2 (+0.8) mIoU; dynamic scope factors further reach 38.7 (+0.1) AP and 43.3 (+0.1) mIoU.Dynamic scope factors are predicted point-wise and modulate offsets within [0, 0.5], centered at 0.25.Source: Figure 5 · p. 4, Table 3 · p. 4, §3.2 · p. 5
  • The PL offset-generation style reduces parameters to 1/s^4 of LP, works better on SegFormer and MaskFormer, but slightly worse on other tested models.PL uses pixel reshuffling before linear projection; the authors set group counts to 4 for LP and 8 for PL.Source: §3.2 · p. 5, Figure 6 · p. 5

3.3. How DySample works

  • DySample dynamically divides low-resolution points into upsampled points, using offsets that can assign neighboring samples across feature boundaries.The illustrated example splits an edge point into four, with one sample assigned to the sky and three to the house.Source: §3.3 · p. 5

3.4. Complexity Analysis

  • DySample series achieve the best performance and lowest latency, memory, training time, GFLOPs, and parameter count among the compared dynamic upsamplers.On a 256 × 120 × 120 feature map, inference takes 6.2–7.6 ms, compared with 1.6 ms for bilinear interpolation; the authors report negligible added training time.Source: §3.4 · p. 5, Figure 8 · p. 6

3.5. Discussion on Related Work

  • Unlike CARAFE’s learned kernels, DySample predicts positions; its 2 × 2 bilinear-kernel view requires fewer channels and at least 2.25 times fewer GFLOPs than CARAFE.DySample also avoids SAPA’s guidance map, making it more efficient and easier to use.Source: §3.5 · pp. 5–6
  • Deformable attention aggregates many sampled points to enhance features, whereas DySample samples one point per upsampled position to divide each input point into s^2 outputs.The authors argue that a single sample per output position suffices when the s^2 upsampled points can be dynamically divided.Source: §3.5 · p. 6

4. Applications

The paper evaluates DySample across five dense prediction tasks and reports improvements in semantic, detection, instance, panoptic, and depth-estimation settings.

4. Applications

  • The application study covers semantic segmentation, object detection, instance segmentation, panoptic segmentation, and depth estimation, comparing DySample with established upsamplers under task-specific settings.The competitors include bilinear interpolation, deconvolution, pixel shuffle, CARAFE, IndexNet, A2U, FADE, and SAPA-B.Source: §4 · p. 6

4.1. Semantic Segmentation

  • SegFormer-B1 mIoU is best at 43.58 (bIoU trails FADE/SAPA); MaskFormer mIoU rises 52.70→53.91 (+1.21) with Swin-B and 54.10→54.90 (+0.80) with Swin-L.The authors attribute DySample’s gains mainly to interior regions, while guided upsamplers primarily improve boundary quality.Source: Table 4 · p. 7, §4.1 · p. 7, Table 5 · p. 8

4.2. Object Detection and Instance Segmentation

  • DySample outperforms all tested upsamplers; with stronger backbones, Faster R-CNN box AP gains are +1.2 (R50) vs +1.1 (R101), and Mask R-CNN mask AP gains are +1.0 vs +0.8.With R50, DySample achieves the best performance among tested upsamplers.Source: Table 6 · p. 8, Table 7 · p. 8, §4.2 · p. 8

4.3. Panoptic Segmentation

  • DySample yields consistent PQ improvements of 1.2 with an R50 backbone and 0.8 with R101.The evaluation uses Panoptic FPN on MS COCO and modifies its three FPN upsampling stages.Source: Table 8 · p. 8, §4.3 · p. 9

4.4. Monocular Depth Estimation

  • Compared with bilinear upsampling, DySample+ increases δ < 1.25 accuracy by 0.05 and decreases Abs Rel by 0.04 and RMS by 0.09.It achieves the best performance among the tested upsamplers; the qualitative comparison shows an accurate, consistent chair depth map.Source: Table 9 · p. 9, §4.4 · p. 9

5. Conclusion

DySample is a point-sampling dynamic upsampler that achieves the best performance among the compared dynamic upsamplers while using the least computational resources. The authors plan to apply it to low-level tasks and study joint upsampling and downsampling.

5. Conclusion

  • DySample reports the best performance among other dynamic upsamplers while using the least resources across latency, training memory, training time, GFLOPs, and parameters.It also requires no customised CUDA packages.Source: §5 · p. 9
  • Unlike common kernel-based dynamic upsampling, DySample is designed from the perspective of point sampling and is improved step by step from a naive design.The authors describe these improvements as guided by their insight into upsampling.Source: §5 · p. 9
  • Future work will apply DySample to low-level tasks and study joint modeling of upsampling and downsampling.Source: §5 · p. 9

Related papers

Loading 2308.15085v1…