Source-linked AI summary
DPT: Deformable Patch-based Transformer for Visual Recognition
Zhiyang Chen, Yousong Zhu, Chaoyang Zhao, Guosheng Hu, Wei Zeng, Jinqiao Wang, Ming Tang
TL;DR
Fixed-size patch embedding can disrupt object-related local structures and produce inconsistent information for the same object across images. DePatch adapts patch positions and scales from input content within a plug-and-play transformer module. DPT reports improvements over PVT-Tiny on ImageNet classification and MSCOCO object detection.
Problem
Fixed-size patch embeddings assume one image split suits all inputs, though object scale and geometric variation can make patches miss local structures or capture inconsistent information for the same object.
Source: §1 · pp. 1–2Method
DePatch predicts patch offsets and scales from input features, enabling adaptive patch positions and sizes in a plug-and-play transformer module.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
2.3% improvement on ImageNet classification and 2.8%/3.5% box mAP improvements with RetinaNet/Mask R-CNN are reported over PVT-Tiny.
Source: §1 · p. 2Takeaways & Limitations
DPT is presented as a way to preserve patch semantics while improving performance across image classification and object detection.
Source: §1 · p. 2With k=2, sampling 2×2 points cannot represent larger regions, limiting capacity for images with heavy geometric deformation; more points may add cost.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Transformer has achieved great success in computer vision, while how to split patches in an image remains a problem. Existing methods usually use a fixed-size patch embedding which might destroy the semantics of objects. To address this problem, we propose a new Deformable Patch (DePatch) module which learns to adaptively split the images into patches with different positions and scales in a data-driven way rather than using predefined fixed patches. In this way, our method can well preserve the semantics in patches. The DePatch module can work as a plug-and-play module, which can easily be incorporated into different transformers to achieve an end-to-end training. We term this DePatch-embedded transformer as Deformable Patch-based Transformer (DPT) and conduct extensive evaluations of DPT on image classification and object detection. Results show DPT can achieve 81.9% top-1 accuracy on ImageNet classification, and 43.7% box mAP with RetinaNet, 44.3% with Mask R-CNN on MSCOCO object detection. Code has been made available at: https://github.com/CASIA-IVA-Lab/DPT .
1 INTRODUCTION
Fixed-size image patches can disrupt local object structures and capture inconsistent semantics across images. The paper introduces DePatch, which adapts patch positions and scales, and reports gains over PVT-Tiny on ImageNet and MSCOCO detection.
1 INTRODUCTION
- Fixed-size patch embeddings assume one split suits all images, although they can miss complete object-related local structures and capture inconsistent information for the same object across images.These problems may destroy semantic information and degrade performance.Source: §1 · p. 2
- 2.3% improvement on ImageNet classification and 2.8%/3.5% box mAP improvements with RetinaNet/Mask R-CNN are reported over PVT-Tiny.The detector comparisons are for MSCOCO.Source: §1 · p. 2
- DePatch predicts each patch's position and scale from the input, aiming to preserve patch semantics while reducing semantic destruction from image splitting.It is lightweight and can be incorporated as a plug-and-play module into different transformers; the paper integrates it into PVT as DPT.Source: §1 · p. 2
- The work evaluates DPT on image classification and object detection, reporting gains for both task types.The introduction identifies detection as a task that highly depends on local features.Source: §1 · p. 2
2 RELATED WORK
The related work covers vision transformers and adaptive methods for geometric variation. DPT is distinguished by adjusting patch position and scale in a data-specific way within a pure vision transformer.
2.1 Vision Transformer
- DPT adaptively modifies patch position and scale according to input content, which the authors identify as data-specific patch splitting in a vision transformer.Earlier approaches often combine CNN and transformer components, retain high-resolution feature maps, add locality biases, or redesign patch embedding [3] [22], [10] [17] [26], [6] [9], [30].Source: §2.1 · p. 2
2.2 Deformable-Related Work
- Unlike deformable convolution [5] [34], which offsets each convolution-kernel pixel, DPT uses regular patches; unlike Deformable-DETR [35], it applies deformable operations in a pure vision transformer.The module is described as plug-and-play across vision transformer architectures.Source: §2.2 · pp. 2–3
3 METHOD
The method reviews how vision transformers embed image patches and aggregate them with self-attention, then introduces DePatch to adapt patch positions and scales. DPT replaces PVT’s patch embeddings at stages 2, 3, and 4 with DePatch.
3.1 Preliminaries: Vision Transformer
- Vision transformers convert images into patch tokens, then use multi-head self-attention to aggregate information across the sequence into new representations.The patch embedding module uses fixed-size, fixed-position regions and a linear layer; self-attention gives each token a global view.Source: §3.1 · p. 3
3.2 DePatch Module
- DePatch predicts input-dependent offsets and scales for each patch, allowing its rectangular region to vary even among patches in one image.A new branch predicts these parameters from the input feature map before patch embeddings are formed.Source: §3.2 · p. 3, Figure 2 · p. 3
- DePatch uniformly samples k × k points inside each predicted region and uses bilinear interpolation to obtain features at fractional coordinates before forming patch embeddings.The interpolation kernel is nonzero only at the four integral locations nearest each sample, enabling computation with few multiply-adds.Source: §3.2 · pp. 3–4
3.3 Overall Architecture
- DPT incorporates DePatch into PVT by replacing the patch embedding modules at the beginnings of stages 2, 3, and 4 while leaving other configurations unchanged.DePatch is described as a self-adaptive, plug-and-play module that can be incorporated into different vision transformers.Source: §3.3 · p. 4
4 EXPERIMENTS
Experiments evaluate DPT on ImageNet classification and COCO detection, then examine design choices, convergence, and learned patch geometry. The reported results show gains across tasks and faster early convergence.
4.1 Image Classification
- 81.9% top-1 accuracy is achieved by DPT-Medium on ImageNet, outperforming PVT-Large and matching DeiT-Base.DPT-Tiny reaches 77.4% top-1 accuracy, 2.3% above the corresponding PVT model.Source: §4.1 · p. 4, Table 1 · p. 5
4.2 Object Detection
- With 3× multi-scale training, RetinaNet+DPT-Medium reaches 43.7% mAP, while Mask R-CNN+DPT-Small reaches 44.4% box mAP and 41.0% mask mAP.The RetinaNet result outperforms PVT-Large at more than 20% lower cost.Source: §4.2 · p. 5, Table 2 · p. 6, Table 3 · p. 6
- 37.7% box mAP is achieved by DPT-Small with DETR after 50 epochs, exceeding PVT-Small by 3.0% and ResNet50 by 5.4%.The authors report this as evidence that DPT is compatible with transformer-based detectors.Source: §4.2 · p. 5, Table 4 · p. 6
4.3 Ablation Studies
- Replacing patch embeddings in stages 2, 3, and 4 yields the best tested placement, outperforming baseline PVT by 1.5% with 0.86M additional parameters.The gains for replacements at stages 2, 3, and 4 individually are 0.3%, 1.0%, and 1.5%.Source: §4.3 · p. 5, Table 5 · p. 6
- Increasing sampling from 2 × 2 to 3 × 3 adds 0.8%, versus 0.2% for 4 × 4; k=3 is default, and learned regions expand beyond initialization.The scale statistics cover stages 2, 3, and 4, with scale measured as region edge size.Source: §4.3 · p. 5, Figure 4 · p. 5, Table 6 · p. 6
- Offsets alone improve 1.5% over baseline, and predicting scales adds another 0.8%, supporting the use of both adaptive controls.Offsets shift patches toward important regions, while scales help maintain local structures.Source: Table 7 · p. 6, §4.3 · p. 7
- At 150 epochs, DPT-Tiny beats fully trained PVT-Tiny by 1.1%, with only 1.2% degradation; zero initialization has little effect, and early convergence is faster.The training loss and test accuracy converge faster during the first few epochs.Source: §4.3 · p. 7, Table 8 · p. 7, Table 9 · p. 7
4.4 Visualization
- Visualizations show offsets shifting toward foreground objects, especially at edges and toward nearby objects, while scales shrink for subtle details and expand when more context is needed.The authors note that patches can orient toward separate objects in multi-object images.Source: §4.4 · p. 7
5 CONCLUSION
The paper introduces DePatch, which splits images into deformable patches to focus on object-related regions and reduce sensitivity to geometric deformation. DPT incorporates DePatch into a transformer, and experiments on image classification and object detection report stronger feature extraction and performance than CNN-based models and other vision transformers.
5 CONCLUSION
- DePatch splits images into deformable patches, encouraging the network to extract patch information from object-related regions and become less sensitive to geometric deformation.The paper describes DePatch as a module for splitting patches.Source: §5 · p. 7
- DPT incorporates DePatch into a transformer, and the module is described as plug-and-play for use in other vision transformers.The paper also proposes applying DePatch to other downstream tasks to improve performance.Source: §5 · p. 7
- Experiments on image classification and object detection indicate that DPT extracts better features and outperforms CNN-based models and other vision transformers.The paper characterizes DePatch as a data-dependent modification to vision transformers and hopes it will support future studies.Source: §5 · p. 7