PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep High-Resolution Representation Learning for Visual Recognition

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, Bin Xiao

arXiv:1908.07919v2cs.CV

TL;DR

Position-sensitive vision tasks need high-resolution representations, but prevailing networks recover them from low-resolution features. HRNet maintains high resolution throughout, connects resolutions in parallel, and repeatedly fuses them, delivering superior results across pose estimation, segmentation, and detection, especially for small objects.

  • Problem

    Position-sensitive tasks need high-resolution representations, while prior methods typically recover them from low-resolution representations.

  • Method

    HRNet maintains high resolution throughout by connecting parallel multi-resolution streams and repeatedly fusing their representations.

  • Results

    HRNet achieves superior results across visual recognition tasks, with particularly dramatic improvement for small-object detection.

  • Takeaways & Limitations

    HRNet is suggested as a stronger backbone for computer vision problems across a wide range of visual recognition tasks.

  • Takeaways & Limitations

    Training memory cost for object detection is a little larger than for state-of-the-art methods, although memory costs are otherwise comparable.

Abstract

from arXiv · show

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions \emph{in series} (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams \emph{in parallel}; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at~{\url{https://github.com/HRNet}}.

1 INTRODUCTION

HRNet maintains high-resolution representations throughout the network by connecting parallel multi-resolution streams and repeatedly exchanging information across them. Its variants support strong performance in human pose estimation, semantic segmentation, and object detection.

  • Motivation: Conventional classification networks progressively reduce feature-map resolution through high-to-low convolutions connected in series, producing low-resolution representations.This design follows the rule established by LeNet-5 and used by networks including AlexNet, VGGNet, GoogleNet, and ResNet.
  • Motivation: Position-sensitive tasks require high-resolution representations, so prior methods recover resolution from low-resolution outputs using encoder-decoder architectures.Examples include Hourglass, SegNet, DeconvNet, U-Net, SimpleBaseline, and encoder-decoder methods.
  • HRNet Architecture: HRNet maintains high-resolution representations throughout the process by adding lower-resolution streams progressively and connecting multi-resolution streams in parallel.The network has four stages in this paper; stage n contains n streams corresponding to n resolutions.
  • HRNet Architecture: Repeated multi-resolution fusion exchanges information across parallel streams, producing representations that are both semantically strong and spatially precise.The design maintains high resolution directly rather than recovering it from low resolution.
  • Applications and Results: HRNetV1 targets human pose estimation, HRNetV2 targets semantic segmentation, and HRNetV2p supports detection frameworks with improved detection performance.HRNetV1 shows superior COCO keypoint detection performance; HRNetV2 achieves state-of-the-art results on PASCAL-Context, Cityscapes, and LIP, while HRNetV1 and HRNetV2 perform similarly on COCO pose estimation.

2 RELATED WORK

The related work spans low-resolution representation learning, high-resolution recovery and maintenance, and multi-scale fusion for pose estimation, segmentation, and detection. HRNet distinguishes itself by connecting high-to-low-resolution streams in parallel and repeatedly fusing multiple resolutions.

  • Low-resolution representation learning: Prior methods learn low-resolution representations with fully convolutional networks, while dilated convolutions produce medium-resolution and multi-scale contextual representations.Feature pyramids augment representations for segmenting objects at multiple scales.
  • Recovering high-resolution representations: High-resolution recovery methods upsample low-resolution representations through symmetric subnetworks, skip connections, copied feature maps, or an added full-resolution stream.Examples include VGGNet, SegNet, DeconvNet, U-Net, Hourglass, and full-resolution residual networks.
  • Maintaining high-resolution representations: Several architectures maintain high-resolution representations, but early designs lacked careful choices for starting parallel streams and exchanging information, as well as batch normalization and residual connections.The cited works include convolutional neural fabrics, interlinked CNNs, GridNet, and multi-scale DenseNet.
  • Multi-scale fusion: Multi-scale fusion combines representations across resolutions through separate networks, progressive skip connections, pyramid pooling, atrous spatial pyramid pooling, and related mechanisms.The paper notes that its fusion module outputs four-resolution representations and repeats fusion modules several times.
  • Our approach: HRNet connects high-to-low convolution streams in parallel and repeatedly exchanges information across resolutions, while extending the earlier HRNetV1 into HRNetV2 and HRNetV2p.The paper identifies these extensions as part of its technical novelties over the previous conference paper.

3 HIGH-RESOLUTION NETWORKS

HRNet maintains a high-resolution representation by connecting multi-resolution convolution streams in parallel and repeatedly exchanging information across resolutions. Its main body combines parallel convolutions, repeated fusion modules, and configurable representation heads.

  • Network structure: The main body comprises parallel multi-resolution convolutions, repeated multi-resolution fusions, and a representation head.The stem uses two stride-2 3 × 3 convolutions to reduce resolution to 1/4 before the main body.
  • Parallel multi-resolution convolutions: The network begins with a high-resolution stream, progressively adds lower-resolution streams, and connects all streams in parallel across stages.Each later stage retains the previous resolutions and adds one extra lower resolution.
  • Repeated multi-resolution fusions: Fusion modules exchange information across resolutions by repeatedly transforming and summing representations from multiple input streams.They are repeated several times, for example every 4 residual units; transformations use identity mappings, stride-2 convolutions, or bilinear upsampling according to resolution changes.
  • Representation heads: The representation heads either retain the high-resolution stream or upsample and concatenate representations from all resolutions.The paper applies HRNetV1 to human pose estimation, HRNetV2 to semantic segmentation, and HRNetV2p to object detection.
  • Modularized block: Multi-resolution parallel convolution separates channels across spatial resolutions, while multi-resolution fusion connects input and output channel subsets in a fully connected fashion.This makes the parallel convolution resemble group convolution and the fusion unit resemble a multi-branch full-connection form of regular convolution.

4 HUMAN POSE ESTIMATION

For human pose estimation, HRNet regresses keypoint heatmaps from high-resolution representations using HRNetV1 and mean squared error loss. On COCO, HRNet achieves strong validation and test-dev performance across model sizes and input resolutions, with gains over competing top-down methods.

  • Method: Pose estimation is formulated as predicting K keypoint heatmaps, each encoding the location confidence of one keypoint.The heatmaps have size W/4 × H/4.
  • Method: The method regresses heatmaps from HRNetV1 high-resolution representations and trains with mean squared error against Gaussian groundtruth heatmaps.HRNetV1 and HRNetV2 perform almost identically, so HRNetV1 is selected for its slightly lower computation complexity.
  • Results on the val set: HRNetV1-W32 trained from scratch at 256 × 192 achieves AP 73.4 on COCO val, outperforming other methods using the same input size.Compared to Hourglass, it improves AP by 6.5 points with much lower GFLOPs and a similar parameter count.
  • Results on the val set: At 384 × 288, HRNetV1-W32 and HRNetV1-W48 achieve AP 75.8 and 76.3, gaining 1.4 and 1.2 over 256 × 192.Against SimpleBaseline with ResNet-152, they gain 1.5 and 2.0 AP at 45% and 92.4% computational cost, respectively.
  • Results on the test-dev set: On COCO test-dev, HRNetV1-W32 achieves AP 74.9 and HRNetV1-W48 achieves the highest AP score 75.5, outperforming top-down approaches.Compared with SimpleBaseline at the same input size, the small and big networks gain 1.2 and 1.8; extra AI Challenger data raises the big model to AP 77.0.

5 SEMANTIC SEGMENTATION

HRNetV2 applies a high-resolution representation with a linear classifier for semantic segmentation, upsampling predictions four times to the input size. Across Cityscapes, PASCAL-Context, and LIP, HRNetV2-W48 achieves strong or state-of-the-art results with favorable complexity.

  • Method: HRNetV2 feeds a 15C-dimensional representation at each position to a linear softmax classifier and upsamples segmentation maps four times by bilinear interpolation.This procedure is used during both training and testing.
  • Cityscapes: On Cityscapes validation, HRNetV2-W40 gains 4.7 points over UNet++, 1.7 points over DeepLabv3, and about 0.5 points over PSPNet and DeepLabv3+.HRNetV2-W40 has similar model size to DeepLabv3+ but much lower computation complexity.
  • Cityscapes: On Cityscapes test, HRNetV2-W48 achieves superior performance in both evaluations without coarse data, using models trained on train and train+val sets.All reported results use six scales and flipping.
  • PASCAL-Context: On PASCAL-Context, HRNetV2-W48 achieves state-of-the-art results for both mIoU over 59 classes and mIoU over 60 classes.The only exception is a result from, which exceeds HRNetV2-W48 without using the OCR scheme.
  • LIP: On LIP, HRNetV2-W48 performs best overall with fewer parameters and lighter computation cost, without using extra information such as pose or edge.Evaluation uses 20 categories and averages segmentation maps from original and horizontally flipped images.

6 COCO OBJECT DETECTION

HRNetV2p applies multi-level representations to single-scale COCO object detection and is evaluated across anchor-based, anchor-free, and joint detection–instance-segmentation frameworks. Across comparable settings, HRNetV2 generally outperforms ResNet and ResNeXt, while some smaller configurations underperform ResNet baselines.

  • Evaluation setup: HRNetV2p uses five-level multi-level representations, horizontal-flip augmentation, 800-pixel shorter-edge resizing, and single-scale inference.The five-level design follows FPN.
  • Detection frameworks: The evaluation covers Faster R-CNN, Cascade R-CNN, FCOS, and CenterNet for COCO validation detection performance.Faster R-CNN and Cascade R-CNN are anchor-based, whereas FCOS and CenterNet are anchor-free.
  • Joint detection and segmentation: Joint detection and instance segmentation are evaluated with Mask R-CNN, Cascade Mask R-CNN, and Hybrid Task Cascade, with overall performance better than ResNet and ResNeXt.The results are reported in Table 10 from MMDetection.
  • COCO validation results: HRNetV2 achieves better overall object detection performance than ResNet under similar model size and computation complexity.This comparison is reported for Tables 8 and 9.
  • COCO validation results: HRNetV2p-W18 can perform worse than ResNet-50-FPN in some 1× settings, possibly because of insufficient optimization iterations.The passage identifies this as an exception to the broader comparison.
  • COCO test-dev results: On COCO test-dev, HRNetV2p is compared with similarly complex ResNet and X-101-64 × 4d FPN detectors without multi-scale training or testing.The reported Faster R-CNN comparisons include HRNetV2p-W32 versus ResNet-101-FPN, HRNetV2p-W40 versus ResNet-152-FPN, and HRNetV2p-W48 versus X-101-64 × 4d-FPN.

7 ABLATION STUDY

The ablations show that higher-resolution representations, repeated multi-resolution fusion, and the proposed fusion operations improve performance across pose estimation and semantic segmentation. HRNetV2 substantially benefits segmentation and detection, while pose performance remains similar to HRNetV1.

  • Representations of different resolutions: The lowest-resolution response map produces an AP score below 10, while the other three maps perform better, showing that resolution affects keypoint prediction quality.The comparison uses heatmaps estimated from feature maps at four resolutions.
  • Repeated multi-resolution fusion: More multi-resolution fusion improves both COCO pose AP and Cityscapes segmentation mIoU, with the proposed eight-fusion design outperforming fewer-fusion variants.The variants compare one, three, and eight total fusion units.
  • Fusion design: Replacing strided convolutions with bilinear downsampling reduces pose AP to 72.6 and segmentation mIoU to 74.2; replacing summation with multiplication further reduces them to 54.7 and 66.0.The paper attributes the first drop to weaker volume-size reduction and the second to increased training difficulty.
  • Resolution maintenance: A variant adding all four streams at the beginning achieves 72.5 AP versus 73.4 for HRNetV1-W32 and 75.7 mIoU versus 76.4 for HRNetV2-W48.The authors suggest early low-resolution streams provide less helpful low-level features.
  • V1 vs. V2: HRNetV2-W32 reaches 73.6 pose AP versus 73.4 for HRNetV1-W32, while HRNetV2 significantly outperforms HRNetV1 on segmentation and object detection except for a minor large-model segmentation gain.The comparison also includes HRNetV2p and the HRNetV1h output-dimension alignment variant.

8 CONCLUSIONS · APPENDIX A NETWORK INSTANTIATION

HRNet maintains high-resolution representations through parallel multi-resolution streams and repeated fusion, achieving strong results across visual recognition tasks while keeping memory and runtime generally comparable to prior state-of-the-art methods. Its four-stage architecture provides a modular implementation pattern, and follow-up work extends it to higher resolutions and additional applications.

  • 8 CONCLUSIONS: HRNet connects high- and low-resolution convolutions in parallel, maintains high resolution throughout, and repeatedly fuses multi-resolution representations.These design choices distinguish it from series-connected low-resolution networks and methods that recover high resolution afterward.
  • 8 CONCLUSIONS: Superior results across human pose estimation, semantic segmentation, and object detection support HRNet as a stronger computer-vision backbone.The paper also encourages designing architectures directly for specific vision problems.
  • 8 CONCLUSIONS: HRNet memory cost is comparable to state-of-the-art methods across the three applications, except for slightly larger training memory in object detection.The discussion addresses the concern that maintaining higher resolution necessarily makes HRNet memory-intensive.
  • 8 CONCLUSIONS: HRNet training and inference time is generally comparable to prior state-of-the-art methods, with faster segmentation inference and slightly slower pose-estimation training.On MXNet 1.5.1, pose-estimation cost is similar to SimpleBaseline because static-graph inference benefits the multi-branch convolutions.
  • 8 CONCLUSIONS: Follow-up work combines HRNet with OCR for segmentation and plans to increase representation resolution to 1/2 or full resolution.The stated future applications include semantic and instance segmentation, with broader suitability for position-sensitive vision tasks.
  • 8 CONCLUSIONS: HRNet has been adopted in winning or leading results for Mapillary panoptic segmentation, COCO DensePose, COCO keypoint detection, and OpenImage instance segmentation.The cited examples include a slightly modified HRNet with ASPP for Mapillary panoptic segmentation.
  • APPENDIX A NETWORK INSTANTIATION: The main HRNet body contains four stages repeated 1, 1, 4, and 3 times, with 1, 2, 3, and 4 resolution branches respectively.Each branch contains four residual units and one multi-resolution fusion unit.

APPENDIX B NETWORK PRETRAINING

HRNet is pretrained on ImageNet with a four-resolution classification head that expands channels and progressively fuses representations. Under similar parameter counts and GFLOPs, HRNet classification results are comparable to and slightly better than ResNets, with alternative aggregation schemes also evaluated.

  • Classification head: The ImageNet classification head bottlenecks four-resolution feature maps, expands them to 128, 256, 512, and 1024 channels, and fuses resolutions by downsampling and addition.The high-resolution representation is downsampled with a 2-strided 3 × 3 convolution producing 256 channels before addition to the second-high-resolution representation.
  • Training and evaluation: Training uses 100 epochs, batch size 256, an initial learning rate of 0.1, 10× reductions at epochs 30, 60, and 90, SGD, weight decay 0.0001, and Nesterov momentum 0.9.Evaluation uses standard single-crop testing with 224 × 224-pixel crops, reporting top-1 and top-5 validation error.
  • Results: Under similar #parameters and GFLOPs, HRNet classification results are comparable to and slightly better than ResNets.The comparison uses ResNet variants with bottleneck units or two 3 × 3 convolutions and an input stem adapted to the HRNet resolution schedule.
  • Ablation study: Two alternatives are evaluated: separately pooled and concatenated multi-resolution features, and residual processing that increases dimensions before concatenation and average pooling.The alternatives are named HRNet-Wx-Ci and HRNet-Wx-Cii, alongside HRNet-Wx-C in the ImageNet ablation study.

APPENDIX C TRAINING/INFERENCE COST

The appendix compares HRNet’s training and inference memory and runtime costs with standard networks across human pose estimation, semantic segmentation, and object detection. HRNet generally achieves similar or lower memory use at comparable parameter complexity, with task-dependent runtime advantages and trade-offs.

  • Memory cost: HRNet has similar or lower training and inference memory costs than pose-estimation state-of-the-art networks at similar parameter complexity.For semantic segmentation, memory costs are also similar to competing methods, while object detection uses slightly more training memory but similar inference memory for comparable parameter counts.
  • Runtime cost: For semantic segmentation, HRNet’s training time is slightly smaller and inference time significantly smaller than PSPNet and DeepLabv3.The table observations characterize both training and inference time costs as much smaller for HRNet.
  • Runtime cost: For object detection, HRNet training is slower than ResNet-based networks but faster than ResNeXt-based networks, while inference is faster at similar GFLOPs.The comparison uses COCO 2017val models trained with a 2× learning schedule.

APPENDIX D FACIAL LANDMARK DETECTION

HRNetV2 is evaluated for facial landmark detection across WFLW, AFLW, COFW, and 300W using normalized mean error alongside AUC and failure rates. HRNetV2-W18 achieves strong accuracy across these benchmarks while using a compact model with 9.3M parameters and 4.3G GFLOPs.

  • Evaluation setup: Evaluation covers WFLW, AFLW, COFW, and 300W, primarily using normalized mean error with inter-ocular or bounding-box normalization.Area-under-the-curve scores and failure rates are also reported.
  • Model efficiency: HRNetV2-W18 uses 9.3M parameters and 4.3G GFLOPs at 256 × 256 input resolution.Its parameter and computation costs are similar to or smaller than widely used ResNet-50 and Hourglass backbones.
  • WFLW: On WFLW, the approach is significantly better than other methods on the test set and every evaluated subset.This includes methods using extra boundary information or stronger data augmentation.
  • AFLW: On AFLW, the approach performs best among methods without extra information or stronger augmentation and outperforms DCFE with extra 3D information.It performs slightly worse than LAB and PDB, which use extra boundary information and stronger data augmentation, respectively.
  • COFW: On COFW, HRNetV2 outperforms other methods by a large margin, including LAB and PDB.LAB uses extra boundary information, while PDB uses stronger data augmentation.

APPENDIX E MORE OBJECT DETECTION AND INSTANCE RESULTS ON COCO V A L2017

This appendix presents additional COCO val results for object detection and instance segmentation, reporting box and mask mean average precision and noting the provenance of the compared results.

  • Results scope: Table 25 reports additional object detection and instance segmentation results on COCO val.The table compares results from prior work with HRNet results obtained by running the MMDetection code.
  • Metrics: APb denotes box mAP, while APm denotes mask mAP.These metrics distinguish detection performance from instance-segmentation performance.
  • Evaluation provenance: Most reported results are taken from reference [17], whereas HRNet results are obtained using code from the OpenMMLab MMDetection repository.The appendix identifies the source or evaluation procedure for the compared methods.
Loading 1908.07919v2…