Source-linked AI summary
HRFormer: High-Resolution Transformer for Dense Prediction
Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, Jingdong Wang
TL;DR
Vision Transformer representations are too low-resolution and single-scale for dense prediction, while full self-attention is computationally expensive. HRFormer combines HRNet-style parallel multi-resolution streams with local-window attention and depth-wise convolution in the FFN. It achieves competitive results across classification, pose estimation, and semantic segmentation, including improved accuracy with lower model complexity in reported comparisons.
Problem
ViT loses fine-grained spatial details, outputs only a single-scale representation, and has high memory and computation costs for dense prediction.
Method
HRFormer combines HRNet’s multi-resolution parallel design with local-window self-attention and a 3 × 3 depth-wise convolution inside the FFN.
Results
HRFormer achieves competitive performance across image classification, pose estimation, and semantic segmentation, with improved accuracy and lower complexity in reported comparisons.
Takeaways & Limitations
High-resolution multi-resolution representations and efficient local attention provide an effective transformer architecture for dense prediction tasks.
Abstract
from arXiv · showhide
We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by $1.3$ AP on COCO pose estimation with $50\%$ fewer parameters and $30\%$ fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.
1 Introduction
HRFormer addresses ViT’s low-resolution, single-scale representations for dense prediction by combining high-resolution multi-resolution processing with efficient local attention and cross-window convolution. It achieves competitive results across classification, pose estimation, and segmentation benchmarks.
- ViT loses fine-grained spatial details and lacks multi-scale representations needed for accurate dense prediction.
- HRFormer maintains a high-resolution stream alongside parallel medium- and low-resolution streams to model multi-scale variation.
- Local-window self-attention reduces memory and computation complexity from quadratic to linear with spatial size.Non-overlapping windows isolate attention, while a 3 × 3 depth-wise convolution reconnects them and expands the receptive field.
- +1.0% top-1 accuracy on ImageNet over DeiT-B with 40% fewer parameters and 20% fewer FLOPs.
- 0.9% AP over HRNet-W48 on COCO with 32% fewer parameters and 19% fewer FLOPs; segmentation gains reach +1.2% and +2.0% mIoU on two benchmarks.
2 Related work
Prior transformer vision models add multi-scale hierarchies or convolutional locality, while dense prediction work highlights the need for higher-resolution representations. HRFormer instead uses HRNet’s multi-resolution parallel transformer design.
- MViT, PVT, and Swin introduce multi-scale feature hierarchies following typical convolutional architectures.
- CvT, CeiT, and LocalViT enhance transformer locality by inserting depth-wise convolutions into self-attention or the FFN.
- Earlier local-attention schemes used overlapped windows after strided convolution, incurring heavy computation; HRFormer uses independent non-overlapping windows for efficiency.
- Dense prediction studies show that increasing ViT output resolution is important for semantic segmentation.
- HRFormer takes a different path to ViT’s low-resolution problem through a multi-resolution parallel transformer scheme inspired by HRNet.
3 High-Resolution Transformer
HRFormer combines HRNet-style parallel multi-resolution streams with local-window self-attention and an FFN containing depth-wise convolution. This design preserves high-resolution features while exchanging information within and across windows and resolutions.
- Multi-resolution parallel transformer: The architecture begins with a high-resolution convolution stem and adds high-to-low resolution streams across successive stages.Streams remain parallel, with transformer blocks updating each resolution and cross-resolution fusion exchanging information.
- Local-window self-attention: Local-window self-attention partitions feature maps into non-overlapping K × K windows and applies multi-head attention independently within each window.
- Architecture configurations: Table 1 specifies stage-wise modules, blocks, windows, heads, and MLP expansion ratios; Table 2 defines tiny, small, and base model variants.
- FFN with depth-wise convolution: The FFN inserts a 3 × 3 depth-wise convolution between two point-wise MLPs to exchange information across otherwise disconnected windows.
- Representation head designs: The four-resolution output feature maps support task-specific heads, including concatenation after upsampling for semantic segmentation.
- Analysis: The 3 × 3 depth-wise convolution both enhances locality and enables interactions across windows, expanding the receptive field.
4 Experiments
Experiments evaluate HRFormer on pose estimation, semantic segmentation, and ImageNet classification, including efficiency comparisons and ablations of its architectural choices.
- Human Pose Estimation: COCO pose estimation experiments compare HRFormer with convolutional and transformer baselines under ImageNet pretraining.HRFormer-B gains 0.9% AP over HRNet-W48 with 32% fewer parameters and 19% fewer FLOPs on COCO val.
- Semantic Segmentation: HRFormer-B + OCR achieves competitive semantic segmentation performance across Cityscapes, PASCAL-Context, and COCO-Stuff.It matches SETR-PUP on Cityscapes while saving 70% parameters and 50% FLOPs, gains 1.1% and 1.5% over two baselines on PASCAL-Context, and nearly 2% over HRNet-W48 + OCR on COCO-Stuff.
- Image Classification: HRFormer-B gains 1.0% top-1 accuracy over DeiT-B while saving nearly 40% parameters and 20% FLOPs on ImageNet-1K.All compared models are trained on ImageNet-1K only.
- Ablation Experiments: With 3×3 depth-wise convolution in the FFN, the intra-window transformer outperforms Swin-T, and the convolution scheme outperforms shifted windows across tasks.These comparisons evaluate alternatives to shifted-window attention and support the FFN convolution as an effective cross-window design choice.
- Cross-Model Comparisons: HRFormer-B achieves better pose-estimation performance than ViT-Large, DeiT-B, and Swin-B with fewer parameters and FLOPs.Against HRNet, HRFormer-T improves by 2.0%, 1.5%, and 1.6% on three tasks while requiring around 50% of the parameters and FLOPs.
5 Conclusion
The conclusion presents HRFormer as a transformer architecture for dense prediction that combines high-resolution multi-resolution processing with efficient local attention and convolutional information exchange.
- Conclusion: HRFormer targets dense prediction tasks including human pose estimation and semantic segmentation.The paper empirically verifies the architecture on both task types.
- Conclusion: HRFormer combines HRNet-inspired multi-resolution parallel design with local-window self-attention and depth-wise convolution in the FFN.The architecture also uses early-stage convolution and multi-scale fusion to mix short-range and long-range attention.
- Conclusion: The reported experiments support HRFormer’s effectiveness for dense prediction while improving memory and computation efficiency.The efficiency claim is tied to the local-window attention and depth-wise-convolution design described in the conclusion.
6 Appendix
The appendix adds visualizations and examines how local-window size affects semantic segmentation performance.
- More Visualization Results: Additional visualizations show HRFormer pose-estimation results on COCO val and semantic-segmentation results across three datasets.The segmentation visualizations cover Cityscapes val, PASCAL-Context test, and COCO-Stuff test.
- Ablation of Window Sizes: The window-size ablation uses W1, W2, W3, and W4 for feature-map strides 4, 8, 16, and 32.The chosen configuration uses larger windows for higher-resolution branches, so W1 > W2 > W3 > W4.
- Ablation of Window Sizes: Applying larger local windows improves semantic-segmentation performance in the reported window-size experiments.The ablation associates W1, W2, W3, and W4 with progressively lower-resolution feature maps.