Source-linked AI summary
High-Resolution Representations for Labeling Pixels and Regions
Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, Jingdong Wang
TL;DR
High-resolution representations are essential for many vision tasks, but their learning remains an open problem. The paper modifies HRNet by aggregating upsampled representations from all parallel convolutions and constructs a multi-level representation for object detection. The modified networks achieve strong results across semantic segmentation, facial landmark detection, and object detection.
Problem
High-resolution representations are essential for pixel- and region-labeling tasks, but learning such representations remains an open problem.
Method
The paper augments HRNet by aggregating upsampled representations from all parallel convolutions and constructs a multi-level representation for object detection.
Results
The modified networks achieve top results in semantic segmentation and facial landmark detection and outperform existing single-model methods on COCO object detection.
Takeaways & Limitations
Strong high-resolution and multi-level representations are effective across semantic segmentation, facial landmark detection, and object detection.
Abstract
from arXiv · showhide
High-resolution representation learning plays an essential role in many vision problems, e.g., pose estimation and semantic segmentation. The high-resolution network (HRNet)~\cite{SunXLW19}, recently developed for human pose estimation, maintains high-resolution representations through the whole process by connecting high-to-low resolution convolutions in \emph{parallel} and produces strong high-resolution representations by repeatedly conducting fusions across parallel convolutions. In this paper, we conduct a further study on high-resolution representations by introducing a simple yet effective modification and apply it to a wide range of vision tasks. We augment the high-resolution representation by aggregating the (upsampled) representations from all the parallel convolutions rather than only the representation from the high-resolution convolution as done in~\cite{SunXLW19}. This simple modification leads to stronger representations, evidenced by superior results. We show top results in semantic segmentation on Cityscapes, LIP, and PASCAL Context, and facial landmark detection on AFLW, COFW, $300$W, and WFLW. In addition, we build a multi-level representation from the high-resolution representation and apply it to the Faster R-CNN object detection framework and the extended frameworks. The proposed approach achieves superior results to existing single-model networks on COCO object detection. The code and models have been publicly available at \url{https://github.com/HRNet}.
1. Introduction
High-resolution representations are important for pixel- and region-labeling tasks, but their learning remains an open problem. The paper studies HRNet and modifies its output representations, achieving strong results across segmentation, landmark detection, and object detection.
- High-resolution representations are essential for semantic segmentation, object detection, human pose estimation, and related vision problems.
- Existing approaches either recover high-resolution features from lower-resolution network outputs or maintain them using parallel multi-resolution convolutions.
- HRNet maintains high-resolution representations through parallel high-to-low resolution convolutions and repeated multi-scale fusions.
- HRNetV2 aggregates representations from all parallel convolutions instead of only the high-resolution convolution, adding small overhead and producing stronger representations.
- The approach achieves state-of-the-art semantic segmentation on PASCAL Context, Cityscapes, and LIP, and overall best facial landmark results on AFLW, COFW, 300W, and WFLW.
- A multi-level representation applied to Faster R-CNN and related frameworks improves COCO object detection, especially for small objects, and surpasses existing single-model methods.
2. Related Work
Prior work either recovers high-resolution representations after low-resolution processing or maintains multiple resolutions through parallel information exchange. The paper positions HRNet within the latter line and contrasts it with earlier architectures and related multi-scale methods.
- Semantic segmentation research has developed low-resolution, recovered high-resolution, and maintained high-resolution representations.
- Dilated convolutions extend fully convolutional networks toward medium-resolution and multi-scale contextual representations for objects at multiple scales.
- Decoder-style methods recover high-resolution features from low-resolution outputs using symmetric or asymmetric upsampling subnetworks and skip connections.
- Maintaining high-resolution representations typically uses parallel convolutions at multiple resolutions with repeated information exchange.
- Earlier convolutional neural fabrics and interlinked CNNs lacked careful stream scheduling and exchange design and omitted batch normalization and residual connections.
- GridNet uses two asymmetric information-exchange stages, which limits its segmentation quality.
3. Learning High-Resolution Representations
HRNet uses staged, parallel multi-resolution processing, while HRNetV2 aggregates upsampled features from every resolution into a high-resolution representation. HRNetV2p derives a multi-level representation for object detection from that aggregate.
- Architecture: HRNet has four stages, with the second through fourth repeating modularized multi-resolution blocks built from group and multi-resolution convolutions.
- Architecture: A multi-resolution convolution divides input and output channels into resolution-specific subsets and connects them in a fully-connected fashion.
- Modification: In HRNetV1, only high-resolution convolution outputs are used, leaving subsets from low-resolution convolutions unused.
- Modification: HRNetV2 exploits the additional low-resolution channel subsets, fully exploring multi-resolution convolution capacity with small parameter and computation overhead.
- Modification: Low-resolution representations are bilinearly upsampled and concatenated with high-resolution features to form the representation used for segmentation and landmark heatmaps.
- Modification: For object detection, HRNetV2p downsamples the high-resolution representation by average pooling to construct multiple representation levels.
4. Experiments
Experiments evaluate HRNetV2 across semantic segmentation, facial landmark detection, and COCO object detection, reporting strong accuracy alongside model-size and computation comparisons.
- Semantic segmentation: Cityscapes validation gains reached 5.6 points over UNet++, 2.6 over DeepLabv3, and about 1.4 over PSPNet and DeepLabv3+ for HRNetV2-W48.HRNetV2-W40 also exceeded UNet++, DeepLabv3, PSPNet, and DeepLabv3+ with similar model size and lower computation complexity.
- Semantic segmentation: 1 point improvement over the previous state of the art was achieved by HRNetV2-W48 on the Cityscapes test set without coarse data.This held for models trained on both the train set and the train-plus-validation set, using six scales and flipping.
- Semantic segmentation: HRNetV2-W48 achieved the best overall LIP performance with fewer parameters and lower computation cost, without pose or edge information.LIP evaluation covers 20 categories, including 19 human-part labels and background.
- COCO object detection: With 2× training, HRNetV2p-W18 and HRNetV2p-W32 outperformed ResNet-50-FPN and ResNet-101-FPN, respectively, on COCO validation.With 1× training, HRNetV2p-W32 exceeded ResNet-101-FPN, while HRNetV2p-W18 underperformed ResNet-50-FPN.
- COCO object detection: HRNetV2p outperformed similarly sized and similarly complex ResNet-based detectors on COCO test-dev without multi-scale training or testing.The comparisons covered Faster R-CNN with W32, W40, and W48, plus Cascade R-CNN with HRNetV2p-W32.
- Facial landmark detection: HRNetV2 achieved the best or near-best facial landmark results across reported datasets, often surpassing methods using extra boundary, 3D, or stronger augmentation information.HRNetV2-W18 used 9.3M parameters and 4.3 GFLOPs, compared with 25.0M/3.8G for ResNet-50 and 25.1M/19.1G for Hourglass.
5. Conclusions
The paper studies a simple modification to high-resolution representations and evaluates modified networks across semantic segmentation, facial landmark detection, and object detection.
- The modified networks demonstrate effective high-resolution and multi-level representations across semantic segmentation, facial landmark detection, and object detection.
Appendix: Network Pretraining
The appendix describes ImageNet pretraining for HRNet classification, including a four-resolution classification head, training protocol, and comparisons with ResNets and alternative schemes.
- The classification head receives representations at four resolutions before transforming and combining them for ImageNet classification.Figure 5 identifies the four-resolution representations as the box input.
- Training uses 100 epochs, batch size 256, SGD with Nesterov momentum, and learning-rate drops at epochs 30, 60, and 90.
- ImageNet classification results compare HRNet-Wx-C with ResNets under matched input-stem conventions.Table 14 is identified as the comparison between HRNet and ResNets.
- The proposed HRNet-Wx-C scheme is superior to the HRNet-Wx-Ci and HRNet-Wx-Cii alternatives in the ablation study.Table 15 compares the proposed scheme with separately pooled and residual-unit alternatives.