PaperlayerView exact revision on arXiv

Source-linked AI summary

MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation

István Sárándi, Timm Linder, Kai O. Arras, Bastian Leibe

arXiv:2007.07227v2cs.CV

TL;DR

2.5D heatmaps require separate scale recovery and cannot localize joints beyond image boundaries. MeTRo predicts heatmaps in fixed metric space, and MeTRAbs combines them with 2D heatmaps for absolute pose estimation; the approaches achieve state-of-the-art results on Human3.6M, MPI-INF-3DHP, and MuPoTS-3D.

  • Problem

    2.5D heatmaps need post-processing to resolve scale ambiguity and cannot localize joints outside image boundaries.

    Source: §1 · p. 1

  • Method

    MeTRo predicts volumetric heatmaps whose dimensions have fixed metric extents, while MeTRAbs combines them differentiably with 2D image-space heatmaps for absolute pose estimation.

    Source: §1 · p. 2, §8 · p. 11

  • Results

    MeTRo and MeTRAbs achieve state-of-the-art results on Human3.6M, MPI-INF-3DHP, and MuPoTS-3D.

    Source: §1 · p. 2, §7 · p. 9, §8 · p. 11

  • Takeaways & Limitations

    MeTRo predicts complete skeletons when body parts are outside the image and can learn scale cues that outperform fixed bone-length recovery on most test scenarios.

    Source: §8 · p. 11

    Absolute pose estimation assumes known camera intrinsics; root-relative estimation does not require intrinsic calibration.

    Source: §4 · p. 6

Abstract

from arXiv · show

Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust (MeTRo) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D. Our code will be made publicly available to facilitate further research.

1 INTRODUCTION

The paper targets scale ambiguity and incomplete poses in 2.5D heatmaps, which require post-processing for metric scale and cannot localize joints beyond image boundaries. It introduces metric-space heatmaps and a differentiable absolute-pose extension, reporting state-of-the-art benchmark results.

1 INTRODUCTION

  • 2.5D heatmaps require separate scale recovery to obtain metric-scale poses and cannot estimate joints outside image boundaries.Bone-length and skeleton-length priors can struggle when subjects vary in height, while truncated images may omit parts of the person.Source: §1 · pp. 1–2
  • MeTRo defines all heatmap dimensions in fixed metric space, enabling a fully-convolutional network to predict metric-scale poses without image-aligned heatmap axes.The authors report that a standard modern FCN can learn this looser input-pixel-to-output-metric correspondence effectively.Source: §1 · p. 2
  • Because prediction-volume bounds do not match image bounds, MeTRo can represent out-of-image joints without a separate scale-recovery step or test-time root distance and focal length.These test-time requirements apply to root-relative prediction.Source: §1 · p. 2
  • MeTRAbs combines root-relative 3D metric heatmaps with 2D image-space heatmaps and differentiably reconstructs absolute 3D pose for end-to-end supervision.The paper describes this as a two-headed CNN and applies supervision to the heads and final combined output.Source: §1 · p. 2
  • For 2D-labeled examples, an alignment-layer loss enables weak supervision while retaining MeTRo's ability to infer joints outside image boundaries.The alignment is used only for training loss computation.Source: §1 · p. 2
  • MeTRo/MeTRAbs achieves state-of-the-art results on Human3.6M, MPI-INF-3DHP, and MuPoTS-3D; the paper also reports state-of-the-art results on 3DPW.The approach additionally placed first in the 2020 ECCV 3D Poses in the Wild Challenge.Source: §1 · p. 2

2 RELATED WORK

Prior work spans coordinate regression, heatmap-based 3D estimation, scale recovery, and approaches to truncated images. The related work highlights the costs of staged 2D-to-3D estimation and the unresolved challenges of metric scale and out-of-frame joints.

2.1 Deep 3D Human Pose Estimation

  • Heatmap methods extend 2D confidence maps into 3D, while soft-argmax provides differentiable coordinate decoding and can reduce the need for dense high-resolution outputs.Other approaches lift 2D poses numerically or combine heatmaps with metric location maps, each with noted limitations.Source: §2.1 · p. 3

2.2 Scale and Distance Estimation

  • Metric-scale estimation must resolve the image-projection ambiguity between person size and distance, often through known camera information or anthropometric priors.The paper uses bone-length-based scale recovery as its main baseline and notes other absolute-pose approaches, including weak-perspective reconstruction and separate root-depth prediction.Source: §2.2 · p. 3

2.3 Truncated Pose Estimation

  • Truncation is a practical concern because detector crops can hide body parts, and existing methods can perform poorly even on visible joints in truncated images.The paper reports that its approach performs significantly better in the truncated setting.Source: §2.3 · pp. 3–4

3 SINGLE-PERSON ROOT-RELATIVE APPROACH

The single-person approach predicts root-relative metric-scale poses directly from volumetric heatmaps, with fixed metric dimensions rather than image-aligned axes. Its design also addresses coarse heatmap resolution, scale-agnostic 2D supervision, and truncated inputs.

3.1 Metric-Space Volumetric Heatmap Representation

  • Metric-space heatmaps decode joint coordinates with fixed scaling factors, and subtracting the predicted root yields root-relative poses without camera back-projection or bone-length rescaling.The prediction volume has fixed metric extents of 2.2 meters in this work, and the root’s location within it is not explicitly supervised.Source: §3.1 · p. 4

3.2 Architecture

  • A ResNet-50 directly predicts spatial heatmaps without additional learned layers; dense prediction raises compute requirements but improves accuracy while retaining real-time execution.Training uses 8×8 spatial heatmaps, while single-person testing can use stride 4 to produce 64×64 heatmaps.Source: §3.2 · pp. 4–5

3.3 Centered Striding

  • Centered striding reverses the last layer’s stride selection to distribute denser receptive-field samples evenly, avoiding the border-skew caused by ordinary striding.The figure contrasts stride 32 and stride 16 sampling on a 256×256 image.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Scale and Translation Agnostic 2D Loss

  • For 2D-labeled examples, a differentiable alignment layer translates and uniformly scales the orthographically projected 3D skeleton to the least-squares fit before computing loss.This makes the 2D loss invariant to prediction scale and translation.Source: §3.4 · p. 5

3.5 Truncated Pose Estimation

  • Truncation robustness is evaluated by randomly cropping Human3.6M inputs while retaining at least one quarter of the person bounding-square area, with crops used at test time alone or in both training and testing.The metric-space heatmap boundary is decoupled from the image boundary, permitting joints outside the frame to be predicted.Source: §3.5 · p. 5

3.6 Training

  • The training objective combines annotated 3D and 2D losses with a fixed λ = 0.1, tuned on a separate Human3.6M validation set.Experiments use seeded randomness and report mean and standard deviation across five runs.Source: §3.6 · p. 5

3.7 Intuition

  • Soft-argmax uses the heatmap’s center of mass, so even distant values can shift predictions; the figure shows approximate invariance to image scale and truncation.The authors describe zero-padding as a positional cue that can help the network shift a full skeleton within the volume when image borders truncate it.Source: §3.7 · pp. 5–6, Figure 4 · p. 6

4 MULTI-PERSON ABSOLUTE POSE APPROACH

MeTRAbs combines metric-scale 3D root-relative heatmaps with 2D image-space heatmaps and differentiably reconstructs absolute pose. It uses full-perspective reconstruction and absolute-pose supervision, with known camera intrinsics assumed for absolute estimation.

4 MULTI-PERSON ABSOLUTE POSE APPROACH

  • MeTRAbs combines complete metric-scale 3D root-relative pose estimates with 2D image-space pose estimates to reconstruct absolute 3D pose in a top-down multi-person pipeline.For joints within the image it uses the 2D-head back-projection; for truncated joints it uses the reconstructed offset added to the root-relative output.Source: §4 · p. 6
  • Absolute-pose estimation assumes known camera intrinsics, while root-relative estimation does not require intrinsic calibration.For truncated images, joints predicted near the image border are excluded from the reconstruction optimization.Source: §4 · p. 6
  • The method derives root position under a full-perspective pinhole camera model and solves the noisy, overdetermined equations by differentiable linear least squares.Backpropagation through this reconstruction permits loss supervision on final absolute 3D pose, which the authors report is crucial for accurate distance estimation.Source: §4 · p. 6
  • The absolute loss is activated only after 5,000 update steps because incompatible early outputs from the two heads can make reconstruction ill-conditioned.To retain real-time multi-person inference, MeTRAbs uses coarse 8×8 heatmaps rather than dense prediction.Source: §4 · p. 7

5 2.5D BASELINE

The 2.5D baseline uses image-space heatmaps and soft-argmax, then recovers root depth through test-time bone-length optimization. It does not supervise absolute pose during training or backpropagate through the depth optimization.

5 2.5D BASELINE

  • The baseline predicts 2.5D heatmaps through a final 1×1 convolution, applies soft-argmax, and computes L1 loss on the resulting coordinates.It uses a backbone architecture matching the paper’s comparison setup.Source: §5 · p. 7
  • At test time, root depth is recovered by choosing the depth that minimizes squared discrepancies between back-projected bone lengths and average training bone lengths.The optimization uses known camera intrinsics and only bones whose endpoints lie more than one stride length inside the image.Source: §5 · p. 7
  • The baseline solves the convex nonlinear depth optimization with Levenberg–Marquardt initialized at 2 m, without absolute-pose training supervision or backpropagation through the optimization.The authors note differentiable convex optimization layers could in principle enable backpropagation through such a solution.Source: §5 · p. 7

6 DATASETS, PREPROCESSING, EVALUATION

The evaluation spans single-person Human3.6M and MPI-INF-3DHP, plus multi-person MuPoTS-3D, with training and test setups tailored to each benchmark. Metrics cover relative and absolute pose accuracy, and truncation robustness is examined on Human3.6M.

6 DATASETS, PREPROCESSING, EVALUATION

  • Human3.6M uses subjects 1, 5, 6, 7, 8 for Protocol 1 training and 9 and 11 for testing; Protocol 2 trains on subjects 1, 5, 6, 7, 8, 9 and tests on 11.Protocol 1 has root alignment without Procrustes alignment; Protocol 2 uses Procrustes alignment, and every 64th frame is evaluated.Source: §6 · p. 7
  • MPII supplies 25,000 2D-labeled training images for weak supervision, while MuCo provides 150,000 synthetic four-person images for training and MuPoTS supplies 20 mixed indoor/outdoor test sequences.Single-person MPII use is restricted to instances marked well-separated; multi-person boxes are obtained with YOLOv3.Source: §6 · pp. 7–8
  • MPI-INF-3DHP testing covers green-screen studio, non-green-screen studio, and outdoor scenes, but uses only chest-height cameras and examples with all joints inside the image.The official 3DHP ground truth changed twice; the first update changed scores by 1–3%, while the second changed them by 0.1%, within experimental fluctuation.Source: §6 · pp. 7–8
  • Images are cropped to a 256 × 256 px person-centered square, with geometric and color augmentation; synthetic occlusion is used with 70% probability for single-person data and 30% for MuCo.Background augmentation is also applied with 70% probability on 3DHP and MuCo.Source: §6 · p. 8
  • PCK counts joints within 150 mm, AUC integrates the PCK curve over thresholds from 0 to 150 mm, and Human3.6M uses MPJPE.3DHP and MuPoTS evaluate 14 non-root joints, while Human3.6M uses 17 joints including the root; MuPoTS also reports absolute versions of metrics.Source: §6 · p. 8
  • On Human3.6M, average performance remains relatively stable when up to 7 joints are truncated.The figure analyzes robustness to truncation.Source: Figure 6 · p. 8

7 RESULTS

Across Human3.6M, MPI-INF-3DHP, and MuPoTS-3D, the experiments report strong results for MeTRo and MeTRAbs, including under truncation and multi-person evaluation. The paper also reports real-time throughput and winning the 2020 3DPW challenge, while noting a training-data comparability caveat for that challenge.

7 RESULTS

  • On H3.6M, MeTRo achieves 49.3 mm MPJPE without ground-truth depth or scale information, within the margin of error of Xu et al. at 49.2 mm, and outperforms the 2.5D scale-recovery baseline under equal conditions.Protocol 2's Procrustes alignment masks scale-recovery quality, so the method performs about equally to the 2.5D variant there.Source: §7 · p. 8, Table 1 · p. 9, Table 2 · p. 9
  • On 3DHP, MeTRo outperforms prior work by a large margin and beats the 2.5D baseline on green-screen studio images for both universal and metric-scale skeletons, while bone-length recovery does better outdoors.MeTRo's scale recovery beats the average-bone-length baseline on H3.6M and is comparable on 3DHP; on H3.6M it also outperforms the 2.5D variant using per-frame ground-truth bone lengths.Source: §7 · pp. 8–9, Table 3 · p. 10, Table 4 · p. 10
  • On truncated H3.6M crops, MeTRo far outperforms prior approaches and remains robust to truncation of 7 or 8 of 17 joints before performance substantially degrades.Training on truncated images improves performance substantially, and truncated joints can be estimated with about 100 mm average error.Source: §7 · p. 9, Table 5 · p. 10
  • On unnormalized MuPoTS-3D poses without bone rescaling, absolute MPJPE improves from 328.8 mm to 248.2 mm and absolute PCK from 36.7% to 40.2% with absolute-loss supervision.MeTRAbs reports state-of-the-art multi-person results; absolute PCK has high variance, with standard deviations around 1.4–3.2%.Source: §7 · p. 9, Table 8 · p. 10
  • Training with full-perspective reconstruction improves absolute results on MuCo, but weak-perspective reconstruction performs better when testing on MuPoTS.The median and 90th-percentile farthest-to-closest joint depth ratios are 1.22 and 1.41 on MuCo versus 1.16 and 1.26 on MuPoTS, confirming stronger perspective effects in MuCo.Source: §7 · pp. 9–10, Table 9 · p. 11

7.1 Inference Speed

  • At stride 32, the root-relative architecture processes 511 crops per second on an RTX 2080 Ti, while MeTRAbs processes 128, 118, 98, 67, and 41 frames per second for 1, 2, 4, 8, and 16 people.Crop availability is assumed and detector time is excluded; denser heatmaps offer diminishing returns in accuracy.Source: §7.1 · pp. 10–11, Table 6 · p. 10

7.2 ECCV 3DPW Challenge

  • MeTRAbs won the 2020 3DPW challenge, and the reported results show strong performance in in-the-wild scenarios despite methods using different training data.The challenge model used H3.6M, MuCo, SURREAL, SAIL-VOS, and CMU-Panoptic, with ResNet-101, truncation augmentation, and five-crop test averaging.Source: §7.2 · p. 11, Table 10 · p. 11

8 CONCLUSION

The conclusion presents MeTRo heatmaps as a metric-space representation for root-relative and absolute 3D pose, reporting state-of-the-art results across single-person and multi-person benchmarks. It also highlights complete-skeleton prediction from partial images and identifies future directions for learning scale cues and evaluating height-diverse subjects.

8 CONCLUSION

  • MeTRo heatmaps represent metric space around the person rather than image space, supporting root-relative and absolute pose estimation and complete-skeleton prediction from partial-body images.They can be predicted with any standard fully-convolutional network.Source: §8 · p. 11
  • MeTRo outperformed a fixed bone-length heuristic in most test scenarios, suggesting it can implicitly discover scale cues from data.The conclusion identifies learning similar cues from large-scale outdoor data and evaluating people with widely differing heights as future directions, if such data becomes available at scale.Source: §8 · p. 11
  • MeTRo achieved state-of-the-art results on Human3.6M and MPI-INF-3DHP, while MeTRAbs achieved state-of-the-art results on MuPoTS-3D in a real-time-capable top-down multi-person setting.For Human3.6M and MPI-INF-3DHP, the conclusion also cites modified weak supervision for 2D labels, stride alignment, and strong data augmentation.Source: §8 · p. 11
  • MeTRAbs combines MeTRo heatmaps with 2D heatmap prediction for multi-person absolute pose, and the conclusion reports the importance of end-to-end absolute-pose supervision.Weak- and full-perspective geometry alternatives performed similarly in the experiments.Source: §8 · p. 11
  • The conclusion frames heatmap estimation as a way to estimate absolute 3D pose by encoding quantities such as coordinates or sizes as activation locations rather than activation values.It concludes that heatmap estimation is a versatile paradigm for this task.Source: §8 · p. 11

H3.6M (partial body)

The supplied figure captions describe qualitative pose visualizations, including predictions on truncated images and MuPoTS-3D. They specify the visual encodings but do not report quantitative results.

H3.6M (partial body)

  • Fig. 7 marks predictions within 150 mm of ground truth with green spheres and those beyond that threshold with red cubes.Predictions are shown in color and ground truth in gray, except for MPII, where ground truth is unavailable.Source: Figure 7 · p. 13
  • Fig. 7 reports that the method performs well on truncated, partial-body images in its second row.Source: Figure 7 · p. 13
  • Fig. 8 shows qualitative MuPoTS-3D results with predictions in blue-yellow and ground truth in red.Source: Figure 8 · p. 13

Related papers

Loading 2007.07227v2…