PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Pose Regression with Residual Log-likelihood Estimation

Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, Cewu Lu

arXiv:2107.11291v3cs.CVcs.LG

TL;DR

Heatmap-based pose estimation performs well but demands more computation and storage, while regression is efficient yet suffers from inferior performance. The paper introduces RLE, which models residual distribution changes with maximum likelihood and normalizing flows. Across pose-estimation tasks, regression with RLE surpasses heatmap-based methods and improves conventional regression by 12.4 mAP on MSCOCO without test-time overhead.

  • Problem

    Heatmap methods are robust to ambiguous labels but computationally and storage intensive, whereas regression methods are efficient yet have inferior performance.

  • Method

    RLE estimates residual log-likelihood relative to a tractable preset density using normalizing flows and a reparameterization strategy.

  • Results

    Regression with RLE achieves superior performance to heatmap-based methods across the evaluated pose-estimation settings.

  • Takeaways & Limitations

    RLE provides an efficient and flexible regression paradigm applicable to various human pose-estimation algorithms and tasks.

  • Takeaways & Limitations

    The desired distribution concerns output deviations from ground truth, but the formulation must avoid learning the ground-truth distribution across all images.

Abstract

from arXiv · show

Heatmap-based methods dominate in the field of human pose estimation by modelling the output distribution through likelihood heatmaps. In contrast, regression-based methods are more efficient but suffer from inferior performance. In this work, we explore maximum likelihood estimation (MLE) to develop an efficient and effective regression-based methods. From the perspective of MLE, adopting different regression losses is making different assumptions about the output density function. A density function closer to the true distribution leads to a better regression performance. In light of this, we propose a novel regression paradigm with Residual Log-likelihood Estimation (RLE) to capture the underlying output distribution. Concretely, RLE learns the change of the distribution instead of the unreferenced underlying distribution to facilitate the training process. With the proposed reparameterization design, our method is compatible with off-the-shelf flow models. The proposed method is effective, efficient and flexible. We show its potential in various human pose estimation tasks with comprehensive experiments. Compared to the conventional regression paradigm, regression with RLE bring 12.4 mAP improvement on MSCOCO without any test-time overhead. Moreover, for the first time, especially on multi-person pose estimation, our regression method is superior to the heatmap-based methods. Our code is available at https://github.com/Jeff-sjtu/res-loglikelihood-regression

1. Introduction

The paper targets the performance gap between efficient regression-based pose estimation and dominant heatmap-based methods by modeling ambiguous outputs with maximum likelihood. Its Residual Log-likelihood Estimation paradigm improves regression across pose-estimation settings without test-time overhead.

  • Motivation: Heatmap methods are robust to ambiguous labels, whereas regression methods are efficient but vulnerable to occlusions, motion blur, truncations, and noisy annotations.A standard heatmap head costs 1.4× the FLOPs of a ResNet-50 backbone, while a regression head costs 1/20000 as much.
  • Motivation: MLE frames regression losses as assumptions about the output density, motivating likelihood functions that better match the true underlying distribution.The paper characterizes ℓ1 and ℓ2 losses as Laplace- and Gaussian-distribution assumptions with constant variance.
  • Method: RLE estimates residual log-likelihood relative to a tractable preset density, using normalizing flows and reparameterization to learn distribution changes efficiently.The reparameterization strategy supports off-the-shelf flow models without requiring a sophisticated network architecture.
  • Method: The RLE module is optimized jointly during training but omitted during inference, improving the regression model without test-time overhead.Both the regression model and RLE module are trained through maximum likelihood estimation.
  • Results: RLE applies across multiple pose-estimation algorithms and single- and multi-person 2D/3D tasks, improving conventional regression by 12.4 mAP on MSCOCO.The paper reports superior performance to heatmap-based methods and evaluates MPII, MSCOCO, and Human3.6M.
  • Results: Regression with RLE achieves superior performance to heatmap-based methods while remaining more computation- and storage-efficient.The paper presents this as the first such result for regression-based human pose estimation.

2. Related Work

Prior work largely centers on likelihood heatmaps, while regression methods emphasize direct, flexible prediction and efficiency. The paper positions RLE as a regression approach that reaches comparable performance and applies across human pose-estimation algorithms.

  • Heatmap-based Pose Estimation: Heatmap-based approaches dominate 2D human pose estimation and have been extended from single-person to top-down and bottom-up multi-person settings.These methods represent joint locations with likelihood heatmaps.
  • Regression-based Pose Estimation: Earlier regression-based methods directly predict poses or displacements, but reported limitations include vulnerability to occlusions and restricted performance.The related methods include iterative error feedback, one-stage direct regression, and accumulative displacement regression.
  • Regression-based Pose Estimation: The paper reports improving regression-based approaches to a comparable level of heatmap-based approaches for the first time.It also describes the method as flexible across various human pose-estimation algorithms.
  • Normalizing Flow in Human Pose Estimation: Prior normalizing-flow work in pose estimation mainly builds priors for 3D human pose, shape, or joint angles, or samples outputs from ambiguous images.The paper distinguishes its use of flows for residual log-likelihood estimation in regression.
  • Adaptive Loss Function: Related adaptive-loss methods learn or tune loss functions through histograms, parameterized function families, or teacher models.The paper relates RLE to this broader line of learnable loss-function research.

3. Method

The method reformulates human pose regression as maximum likelihood estimation and uses normalizing flows with reparameterization to model output distributions. Residual Log-likelihood Estimation learns distributional deviations, simplifying training while preserving efficient inference.

  • General Formulation of Regression: Regression losses correspond to assumptions about the output probability distribution, so better density estimates can improve regression performance.Constant-variance Gaussian and Laplace assumptions recover standard ℓ2 and ℓ1 losses, respectively.
  • Basic Design: Normalizing flows transform a simple initial density into a flexible learned distribution conditioned on the input image.The distribution depends jointly on regression parameters and flow parameters and can approximate arbitrary densities given sufficient flow complexity.
  • Residual Log-likelihood Estimation: RLE estimates residual log-likelihood relative to a tractable preset distribution instead of directly fitting the unknown underlying distribution.The preset distribution provides an approximation, while the residual compensates for its mismatch.
  • Reparameterization: Reparameterization shifts and rescales a zero-mean flow distribution using predicted location and scale, allowing the flow to learn normalized output deviations.The transformed target is ¯µg = (µg − ˆµ)/ˆσ.
  • Training and Inference: The regression model and RLE module are optimized jointly during training, while RLE is removed during inference and adds no test-time overhead.The approach is presented as a regression paradigm compatible with off-the-shelf flow models.
  • Residual Log-likelihood Estimation: The residual formulation reduces training difficulty because pushing residual log-probability toward zero is easier than learning an identity mapping through invertible transformations.The paper states that this effectiveness is validated experimentally.

4. Experiments on COCO

On COCO, RLE substantially improves regression-based pose estimation while preserving low test-time cost, outperforming or matching heatmap-based approaches across several settings. Experiments also show stronger confidence calibration, robustness to occlusion, and stability across architectures and initial density assumptions.

  • Comparison with conventional regression: 12.4 mAP improvement is achieved over conventional regression with the same test-time computational complexity.The flow model is excluded during inference, so no extra testing computation is introduced.
  • Comparison with heatmap-based methods: RLE surpasses heatmap-based methods by 7.5 mAP against Integral Pose and 0.3 mAP against heatmap-supervised SimplePose.With HRNet, integral regression with RLE also surpasses conventional heatmaps by 0.2 mAP.
  • Robustness and flexibility: RLE improves occlusion robustness for all joints and extends across top-down, one-stage, and multi-person pose estimation settings.Its flexibility is demonstrated through embeddings in different approaches and backbone networks.
  • Main results: 7.0 mAP improvement over other regression-based methods on COCO test-dev accompanies state-of-the-art performance.RLE is also 1.4 mAP higher than the same-backbone heatmap method with ResNet-152 and 0.2 mAP higher with HRNet-W48.
  • Confidence estimation: RLE provides a confidence score more strongly correlated with OKS than heatmap confidence, yielding a relative 15.2% improvement.The predicted standard deviation is larger for more uncertain results and supplies the regression model’s confidence measure.
  • Computation complexity: Total FLOPs decrease by 58.8% and parameters by 30.6% versus heatmap-based methods, while the regression head costs only 1/28500 of the heatmap head.Final AP remains comparable to heatmap-based methods under this lower computational complexity.
  • Ablation and robustness: RLE results remain stable across RealNVP architectures and different initial density assumptions.Learning the residual log-likelihood reduces the performance gap between Gaussian and Laplace initial densities.

5. Experiments on Human3.6M

Human3.6M experiments evaluate RLE for 3D pose estimation against regression and heatmap-based baselines, emphasizing accuracy and computational efficiency. RLE improves regression performance while reducing computational and parameter costs.

  • Experimental setup: Human3.6M is an indoor benchmark for 3D pose estimation evaluated with MPJPE and PA-MPJPE.Training uses S1, S5, S6, S7, and S8; evaluation uses S9 and S11.
  • Ablation study: 1.5 mm lower error is achieved than the single-stage regression baseline, while reducing FLOPs by 61.7% and parameters by 30.6%.This comparison excludes 3D heatmaps.
  • State-of-the-art comparison: The two-stage Human3.6M result is based on SRNet with RLE.The table caption identifies the model used for the reported result.
  • Ablation study: 2.7 mm improvement over the two-stage regression baseline is obtained without any test-time overhead.The result is reported for the two-stage approach.
  • State-of-the-art comparison: Single-stage RLE achieves comparable performance to state-of-the-art methods while reducing FLOPs by 86.7%.The comparison uses the reported Human3.6M single-stage results.

6. Conclusion

The paper concludes that RLE is an MLE-based regression paradigm using normalizing flows to learn residual log-likelihood. Experiments support its efficacy, efficiency, and superiority over heatmap-based methods.

  • Conclusion: RLE uses normalizing flows to learn residual log-likelihood relative to a tractable initial density function.The flow model transforms a simple distribution through invertible mappings, and multiple mappings can be composed.
  • Experiments: Comprehensive experiments validate the efficacy of the proposed regression paradigm.Supplementary materials include additional experiments, ablations, visualizations, and pseudocode.
  • Conclusion: Regression-based methods achieve superior performance to heatmap-based methods in the paper’s reported comparisons.The conclusion presents this as the first such result for the authors’ method.
  • Conclusion: The method is presented as efficient and flexible across human pose estimation settings.The conclusion frames regression’s efficiency and flexibility as motivating properties.
  • Efficiency: The RealNVP flow model has negligible training overhead and is described as computationally and storage efficient.Its training-time computation complexity and parameter count are reported in Table 11.

B. Experiments on MPII

MPII experiments compare heatmaps, direct regression, and RLE for single-person 2D pose estimation. RLE improves direct regression and performs competitively, especially for high-precision localization.

  • Experimental setup: PCK and AUC are used to evaluate the proposed regression paradigm on MPII.The experiment uses the same ResNet-50 + FC model for single-person 2D pose estimation.
  • Ablation study: 13.1% relative improvement on PCKh@0.1 is achieved over direct regression on MPII.MPII is described as less challenging than COCO, but the improvement remains significant under a high localization-accuracy requirement.
  • Ablation study: Compared with the heatmap-based method, RLE is 5.1% higher on PCKh@0.1, 1.8% lower on PCKh@0.5, and 1.9% lower on AUC.The pre-trained model achieves the best PCKh@0.1 results.
  • Experimental setup: The MPII validation comparison is summarized in Table 12, while MPII and Human3.6M comparisons between DLE and RLE are summarized in Table 13.These tables organize the reported ablation comparisons.
  • Ablation study: RLE shows superiority in high-precision localization on MPII.The conclusion follows the stronger PCKh@0.1 result reported in the comparison.

C. Ablation Study

The ablation studies examine RLE against direct likelihood estimation, auxiliary heatmap losses, and heatmap-based methods, including robustness and learned-distribution analyses. Results consistently support RLE’s gains and distributional flexibility.

  • DLE versus RLE: RLE shows consistent improvements over direct likelihood estimation on MPII and Human3.6M.The comparison is reported in Table 13, extending the paper’s COCO analysis.
  • Auxiliary heatmap loss: The auxiliary heatmap-loss experiment adds parallel deconvolutional heatmap layers to a ResNet-50 + FC regression model.The setup predicts both heatmaps and regressed coordinates.
  • Robustness to occlusion: Occlusion robustness is evaluated by masking areas around various joints and measuring effects on other joints.Results are reported separately for Integral Pose and RLE in Tables 15 and 16.
  • Robustness to truncation: Regression-based methods can infer joints outside the input image under truncation, whereas heatmap-based methods fail in that setting.This property is linked to robustness in crowded cases where human detection can fail.
  • Learned distribution: RLE learns a distribution with a sharper peak than Gaussian and smoother edges than Laplace.The visualization compares the learned distribution with Gaussian and Laplace distributions.

E. Derivation of s in RLE

The normalization factor s is approximated numerically, but the implementation omits log s because it adds computation without materially affecting results. Training uses the regression model and flow model to estimate residual log-probabilities, while inference excludes the flow model.

  • s normalizes Pφ(¯x) by enforcing that its integral equals one.
  • The integral over [a, b] is approximated with a Riemann sum using subinterval width ∆x = (b−a)/N.N denotes the total number of subintervals, and sufficiently large N provides a small step ∆x.
  • In practice, the interval can be set to [−5, 5] because Q(¯x) is close to zero outside it.
  • The implementation drops log s because it requires N flow-model evaluations and barely affects results; RLE’s effectiveness comes from the gradient shortcut in Q(¯x).The paper reports that removing log s saves computation resources.
  • The flow model does not participate in inference, so the proposed method adds no test-time overhead.
  • During training, the regression model predicts hat_mu and hat_sigma, then forms the normalized deviation bar_mu = (gt_mu − hat_mu) / hat_sigma.

G. Qualitative Results

The paper presents additional qualitative results on COCO, MPII, and Human3.6M, alongside a table evaluating RLE on the DME dataset.

  • Additional qualitative results on COCO, MPII, and Human3.6M are shown in Figures 6, 7, and 8.
  • Table 18 reports the effect of Residual Log-likelihood Estimation on the DME dataset.

H. Experiments on Retina Segmentation

The paper tests RLE beyond pose estimation on retinal boundary regression using OCT scans from patients with severe diabetic macular edema. On the DME dataset, RLE significantly reduces regression error.

  • RLE is evaluated for boundary regression in retinal segmentation from optical coherence tomography.
  • The evaluation uses the publicly available DME dataset containing 110 Bscans from 10 patients with severe DME pathology.
  • The experiment replaces the previous method’s output layer with a fully connected regression layer, using Adam for 200 epochs with batch size 2.
  • RLE significantly reduces regression error on the DME dataset.The paper reports the quantitative results in Table 18.
Loading 2107.11291v3…