PaperlayerView exact revision on arXiv

Source-linked AI summary

The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation

Junjie Huang, Zheng Zhu, Feng Guo, Guan Huang, Dalong Du

arXiv:1911.07524v2cs.CV

TL;DR

The paper addresses biased data processing in human pose estimation, where coordinate transformations and keypoint encoding or decoding can introduce systematic errors. It proposes UDP, combining unbiased coordinate and keypoint-format transformations, and reports improved accuracy and efficiency across established methods. UDP consequently provides a higher and more reliable baseline, although parts of the analysis assume perfect network learning and interpolation remains irreversible and cumulative.

  • Problem

    Data processing has not been systematically considered, while biased coordinate and keypoint-format transformations can degrade pose-estimation performance and encourage unreported remedies.

  • Method

    UDP combines unbiased coordinate system transformations with unbiased keypoint format transformations, defining coordinate transformations through continuous-space principles.

  • Results

    UDP improves established pose-estimation methods across top-down and bottom-up settings, including 1.5 AP for SimpleBaseline ResNet50-256×192 and 2.7 AP plus 6.1-times acceleration for HigherHRNet HRNet-W32-512×512.

  • Takeaways & Limitations

    UDP pushes the performance boundary and provides a higher, more reliable baseline for human pose estimation research.

  • Takeaways & Limitations

    The unbiased coordinate-transformation target assumes perfect network learning, while interpolation precision degradation is irreversible and cumulative.

Abstract

from arXiv · show

Being a fundamental component in training and inference, data processing has not been systematically considered in human pose estimation community, to the best of our knowledge. In this paper, we focus on this problem and find that the devil of human pose estimation evolution is in the biased data processing. Specifically, by investigating the standard data processing in state-of-the-art approaches mainly including coordinate system transformation and keypoint format transformation (i.e., encoding and decoding), we find that the results obtained by common flipping strategy are unaligned with the original ones in inference. Moreover, there is a statistical error in some keypoint format transformation methods. Two problems couple together, significantly degrade the pose estimation performance and thus lay a trap for the research community. This trap has given bone to many suboptimal remedies, which are always unreported, confusing but influential. By causing failure in reproduction and unfair in comparison, the unreported remedies seriously impedes the technological development. To tackle this dilemma from the source, we propose Unbiased Data Processing (UDP) consist of two technique aspect for the two aforementioned problems respectively (i.e., unbiased coordinate system transformation and unbiased keypoint format transformation). As a model-agnostic approach and a superior solution, UDP successfully pushes the performance boundary of human pose estimation and offers a higher and more reliable baseline for research community. Code is public available in https://github.com/HuangJunJie2017/UDP-Pose

1 INTRODUCTION

The paper argues that biased data processing is an overlooked source of error in human pose estimation and introduces UDP to address it. UDP improves established methods across top-down and bottom-up settings without extra cost, while also supporting a higher, more reliable baseline.

  • Data processing has not been systematically considered in human pose estimation despite being fundamental to training and inference.
  • Flipping introduces coordinate misalignment, while defective encoding and decoding add precision degradation; together, these problems reduce performance and encourage unreported remedies.
  • UDP addresses both issues through unbiased coordinate system transformations and unbiased keypoint format transformations.
  • 1.5 AP improvement raises SimpleBaseline ResNet50-256×192 from 70.2 to 71.7 without extra latency.
  • 2.7 AP improvement and 6.1-times acceleration are reported for HigherHRNet HRNet-W32-512×512 in the bottom-up setting.
  • UDP is extended to bottom-up methods and CrowdPose, and serves as a baseline for UDP++ in COCO and LVIS 2020 competitions.

2 RELATED WORK

Related work is organized around top-down and bottom-up multi-person pose estimation, then identifies data-processing biases across coordinate and keypoint-format transformations. Existing methods often rely on empirical corrections or specialized unbiased transformations.

  • Multi-person pose estimation is commonly categorized into bottom-up and top-down approaches.
  • Bottom-up methods detect identity-free joints and group them into person instances, with detection and grouping efficiency as central concerns.
  • Top-down methods first detect person bounding boxes, then estimate keypoints within fixed-scale patches.
  • Pixel-based resizing creates flipping misalignment and has prompted empirical shifts, higher output resolution, and other often-unreported remedies.
  • Prior work also includes mathematically error-free or distribution-aware keypoint-format transformations intended to align decoding with encoding.

3 UNBIASED DATA PROCESSING FOR HUMAN POSE ESTIMATION

UDP treats human-pose data processing as transformations between coordinate systems and between keypoint formats. The section develops unbiased methods for these two aspects.

  • Human-pose data processing includes coordinate system transformation and keypoint format transformation.

3.1 Unbiased Coordinate System Transformation

UDP defines coordinate transformations in continuous space to preserve semantic alignment and eliminate systematic error across cropping, resizing, rotating, and flipping. The resulting unbiased pipeline aligns predictions with ground truth and avoids the degradation associated with biased transformations.

  • Unified definition of data in continuous space: UDP constructs coordinate transformations from a continuous-space view, treating image matrices as discrete samples while keypoint coordinates remain continuous.This unified treatment is intended to prevent precision degradation when transforming images and keypoints.
  • Concept of coordinate system transformation: Coordinate system transformation maps keypoint coordinates and image contents between source and destination systems while preserving their semantic alignment.Image values at transformed positions are obtained through inverse mapping and bilinear interpolation from valid surrounding samples.
  • Concept of coordinate system transformation: Unbiased transformation requires predictions to align with ground truth under perfect learning, leaving network learning as the only source of precision degradation.The design therefore seeks to prevent defects in the transformation pipeline from introducing systematic error.
  • Coordinate system transformation in elementary operations: Cropping translates the coordinate-system origin to the upper-left corner of a region of interest, while resizing changes sampling and unit length without changing image semantics.These operations use transformation matrices designed from the continuous-space formulation.
  • Diagnosis of the biased coordinate system transformation: Under the proposed pipeline, source-space predictions equal ground truth and flipped-image results align with original-image results without systematic error.This establishes the unbiased property of the coordinate transformation pipeline.
  • Diagnosis of the biased coordinate system transformation: Existing biased transformations accumulate prediction error, making higher input or output resolution useful for suppressing error but potentially costly in latency.UDP offers similar improvement without requiring high output resolution, while existing top-down methods lose more accuracy at lower input resolution.

3.2 Unbiased Keypoint Format Transformation

The paper targets precision loss in keypoint encoding and decoding, proposing unbiased transformations that avoid systematic error. It also analyzes how biased coordinate and format transformations combine to enlarge errors.

  • 3.2.1 The Concept of Unbiased Keypoint Format Transformation.: Keypoint format transformation maps coordinates to heatmaps during encoding and reverses this mapping during decoding.
  • 3.2.1 The Concept of Unbiased Keypoint Format Transformation.: Unbiased transformation is designed to avoid precision degeneration and systematic error throughout the encoding-decoding pipeline.
  • 3.2.2 Unbiased Keypoint Format Transformation.: The proposed unbiased paradigms achieve the target that no systematic error enters the keypoint format transformation pipeline.
  • 3.2.2 Unbiased Keypoint Format Transformation.: Combined classification-regression format uses a classification heatmap to localize keypoints and X,Y offset heatmaps to preserve residual location information.
  • 3.2.2 Unbiased Keypoint Format Transformation.: Combined-format decoding first selects the highest-response heatmap position, then updates it with predicted offsets.
  • 3.2.2 Unbiased Keypoint Format Transformation.: Classification-format decoding with DARK searches for the Gaussian center where the first derivative is zero; its Taylor-approximation precision degradation is theoretically negligible.
  • 3.2.3 Analysis of Biased Keypoint Format Transformation.: 0.125 unit length is the expected per-direction error of the biased decoding method under uniformly distributed keypoints, with variance approximately 0.0052.
  • 3.2.3 Analysis of Biased Keypoint Format Transformation.: Higher output resolution benefits methods with biased format transformations because it reduces the analyzed coordinate errors.

4.1 Result on COCO dataset

On COCO, UDP is evaluated by changing the data-processing pipeline while preserving baseline training settings, improving pose accuracy across top-down and bottom-up configurations and generalizing to test-dev.

  • Experimental setup: UDP is applied to SimpleBaseline and HRNet top-down baselines while preserving all training settings except the data-processing pipeline.Unbiased keypoint format transformation is used by default in the top-down paradigm.
  • Experimental setup: UDP is applied to HigherHRNet-based bottom-up experiments, replacing output resizing and biased decoding with the unbiased decoding method.Both single-scale and multi-scale testing are used.
  • COCO val results: +1.6 AP and +1.4 AP are obtained for SimpleBaseline with ResNet-50 and ResNet-152 backbones, respectively, on COCO val.Higher-resolution configurations also receive +0.8 AP and +0.9 AP improvements.
  • COCO val results: +1.2 AP and +1.3 AP are obtained for HRNet-W32 and HRNet-W48, respectively, on COCO val.The reported gains compare 75.6 to 76.8 AP and 75.9 to 77.2 AP.
  • Bottom-up results: UDP raises HRNet-W32-512×512 from 64.4 AP to 67.0 AP and increases inference speed from 0.8 IPS to 4.9 IPS relative to biased processing.The reported accuracy improvement is 2.6 AP.
  • COCO test-dev results: On COCO test-dev, UDP improves SimpleBaseline by 1.5 AP and 1.0 AP and HRNet by 1.7 AP and 1.4 AP across the reported configurations.The gains are reported for ResNet50-256×192, ResNet152-256×192, W32-256×192, and W48-256×192, respectively.

4.2 Results on CrowdPose dataset

On CrowdPose, UDP is evaluated under the same experimental configuration as COCO and improves accuracy across all reported configurations while also accelerating inference.

  • Experimental setup: CrowdPose experiments use HigherHRNet as the baseline, with models trained on train and val sets and tested on the test set.The experimental configurations are kept the same as those used for COCO.
  • Results: UDP improves AP for all reported CrowdPose configurations and also speeds up inference.The passage attributes these results to experiments using HigherHRNet as the baseline.

4.3 Ablation Study on Top-down Paradigm

Top-down ablations show that unbiased coordinate and keypoint format transformations are jointly necessary for accurate pose estimation, while empirical compensation only partially addresses coordinate bias.

  • Coordinate-system transformation: UCST raises configuration D to 75.7 AP from configuration B’s 74.4 AP when flipping testing is used.Without flipping testing, configurations A and B perform similarly at 74.5 AP and 74.4 AP.
  • Coordinate-system transformation: Flipping testing lowers configuration C from 74.5 AP to 73.3 AP, a 1.2 AP drop caused by biased coordinate transformation.The result demonstrates that flipping can amplify the coordinate-system error rather than improve performance.
  • Coordinate-system transformation: SNOOP reaches 75.6 AP, but extra compensation adds only 0.2 AP, from 75.6 to 75.8 AP.The residual coordinate error therefore has limited measured impact, while SNOOP remains a less interpretable and less accurate remedy than unbiased transformation.
  • Keypoint format transformation: UKFT-CCRF and UKFT-CF each reach 76.8 AP, improving 1.1 AP over configuration D’s 75.7 AP.Applying UKFT without UCST instead degrades performance by 2.3 AP to 74.5 AP, showing the two defects have accumulative effects.
  • Combined effect: Both UCST and UKFT are required for accurate top-down prediction because each corrects a distinct data-processing defect.The ablation results link coordinate-system bias and keypoint format bias to separate performance losses.

4.4 Ablation Study on Bottom-up Paradigm

Bottom-up ablations confirm that unbiased processing improves HigherHRNet while reducing reliance on costly output resizing, although gains depend on the configuration and output-resolution setting.

  • Unbiased processing: Removing RNO and applying UCST improves the baseline from 64.4 AP at 0.8 IPS to 65.9 AP at 4.9 IPS.This corresponds to a 1.5 AP improvement and 5.9-times faster inference.
  • Unbiased processing: Adding UKFT-CF raises configuration C to 67.0 AP at the same 4.9 IPS inference speed.UCST and UKFT-CF are both effective in the bottom-up paradigm.
  • Resize the Network Output: RNO provides only 0.2 AP improvement while imposing 2.6-times latency relative to the referenced HigherHRNet setting.The referenced configuration F achieves 67.1 AP at 1.1 IPS.
  • Unbiased processing: UCST and UKFT-CF incrementally raise configuration E to 67.3 AP and then 67.8 AP while maintaining 2.9 IPS.The gains are smaller than those from configurations B and C, consistent with higher output resolution suppressing part of the systemic error.
  • Resize the Network Output: With UDP, RNO changes performance by 0 AP at 6.1-times latency and by -0.9 AP at 2.6-times latency, making it unnecessary in the tested bottom-up settings.The degradation is attributed to distribution variation caused by resizing, which destroys the precondition for UKFT-CF.

5 CONCLUSION

The paper concludes that coupled biases in coordinate and keypoint format transformations significantly degrade pose estimation across top-down and bottom-up paradigms. UDP addresses both through principled unbiased transformations and provides a more reliable baseline.

  • Conclusion: Systematic errors in coordinate and keypoint format transformations couple together and significantly degrade human pose estimators in both paradigms.The conclusion identifies these coupled errors as the source of a trap involving suboptimal remedies.
  • Conclusion: UDP combines unbiased coordinate system transformation with unbiased keypoint format transformation to solve the defective processing pipeline.The strategy is presented as a principled formulation of unbiased data processing.
  • Conclusion: UDP pushes the performance boundary and provides a reliable baseline by removing the trap created by biased data processing.This conclusion summarizes the paper’s claimed practical contribution to pose-estimation research.
Loading 1911.07524v2…