PaperlayerView exact revision on arXiv

Source-linked AI summary

PARE: Part Attention Regressor for 3D Human Body Estimation

Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, Michael J. Black

arXiv:2104.08527v2cs.CV

TL;DR

State-of-the-art 3D human pose and shape regressors can fail dramatically under small partial occlusions because they rely on global representations. PARE introduces body-part-guided soft attention that uses visible parts to reason about occluded ones, achieving more accurate and robust reconstruction on occlusion-specific and standard benchmarks.

  • Problem

    State-of-the-art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce implausible predictions despite substantial visible body evidence.

  • Method

    PARE jointly regresses 3D body parameters and body-part attention weights using pixel-aligned features, with early guidance from part segmentation labels.

  • Results

    PARE achieves consistently lower error than state-of-the-art methods on 3DPW, 3DOH, and 3DPW-OCC, covering occluded and non-occluded cases.

  • Takeaways & Limitations

    PARE’s body-part-driven attention uses visibility cues and aggregated attended features to improve robustness across scene, self, and frame occlusions.

Abstract

from arXiv · show

Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable. To address this, we introduce a soft attention mechanism, called the Part Attention REgressor (PARE), that learns to predict body-part-guided attention masks. We observe that state-of-the-art methods rely on global feature representations, making them sensitive to even small occlusions. In contrast, PARE's part-guided attention mechanism overcomes these issues by exploiting information about the visibility of individual body parts while leveraging information from neighboring body-parts to predict occluded parts. We show qualitatively that PARE learns sensible attention masks, and quantitative evaluation confirms that PARE achieves more accurate and robust reconstruction results than existing approaches on both occlusion-specific and standard benchmarks. The code and data are available for research purposes at {\small \url{https://pare.is.tue.mpg.de/}}

1. Introduction

PARE addresses the fragility of direct 3D human pose and shape regression under partial occlusion with body-part-driven attention. It uses visible body-part features to reason about occluded parts and reports lower error than state-of-the-art methods on occluded and non-occluded benchmarks.

  • In-the-wild 3D human pose and shape estimation must handle self-occlusion, person-person occlusion, and scene-object occlusion.
  • Small occlusions, including a single joint, can cause state-of-the-art methods to produce entirely implausible pose predictions.
  • PARE learns body-part attention weights alongside 3D body parameters using separate pixel-aligned feature branches.
  • Part segmentation labels guide the attention branch early in training, after which PARE trains end-to-end using pose supervision alone.
  • PARE leverages visible body-part features to reason about occluded parts and achieves consistently lower error than state-of-the-art methods across occluded and non-occluded datasets.
  • The paper contributes occlusion sensitivity analysis, a body-part-driven attention framework, visibility-based reasoning, and state-of-the-art results on occluded and standard benchmarks.

2. Related Work

Prior 3D pose and shape methods use keypoints, silhouettes, global CNN features, visibility information, or saliency and UV representations. Their occlusion handling is limited by manual inputs, weak spatial structure, restricted occlusion types, or dependence on accurate masks.

  • Early 3D body estimation approaches using keypoints and silhouettes can require manual input, additional data, or fail to generalize to in-the-wild images.
  • Current direct regression methods globally pool the image into one feature, making body-part locations and visibility difficult to represent explicitly.
  • Data augmentation is commonly used to improve occlusion robustness in globally pooled regression frameworks.
  • Cylinder-based visibility modeling handles only self-occlusion and relies on an unrealistic body approximation.
  • Multi-person methods address person-person occlusion through collision, depth, or simultaneous-person modeling, whereas PARE focuses attention on individual body parts.
  • Saliency-mask and UV-map approaches require accurate saliency maps and can produce mesh artifacts, limiting their use on in-the-wild images.

3. Occlusion Sensitivity Analysis

The occlusion sensitivity analysis measures how localized image occlusions affect joint errors in globally pooled regression models. It reveals that errors depend on occluder location and can propagate across body parts, motivating PARE’s pixel-aligned attention design.

  • The analysis slides a square occluder across an image and records each joint’s Euclidean 3D error at every occluder location.
  • Per-joint heatmaps encode the error caused by occlusion at each image pixel, while aggregate maps show effects on average joint error.
  • The SPIN heatmaps show low background error and high body-region error, indicating that the model attends to meaningful image regions.
  • Occluding a visible joint produces high error for that joint, while naturally occluded joints depend on other image regions.
  • Across 3DPW, occlusion sensitivity is transferred from image pixels to visible SMPL vertices and pooled into one sensitivity mesh per joint.
  • The left elbow is sensitive to occlusion around the face, left shoulder, and left upper arm, demonstrating dependencies between body parts.

4. Method

PARE uses pixel-aligned, body-part-driven attention to aggregate localized image features for 3D pose, shape, and camera regression. Its hybrid supervision allows attention to visible parts and other informative regions when parts are occluded.

  • Architecture: PARE extracts separate pixel-aligned part-attention and body-regression feature volumes from CNN features.The part branch predicts J body-part masks plus one background mask, while the regression branch stores C-dimensional features at the same H × W locations.
  • Attention aggregation: Spatially softmax-normalized part masks weight corresponding regression features to produce one feature representation per body part and channel.Each pixel contributes proportionally to the final feature tensor, so higher-attention regions contribute more strongly.
  • Training strategy: PARE initially supervises attention with part labels, then removes that supervision so occluded-part estimation can use other informative image regions.This hybrid scheme avoids forcing occluded parts to receive zero attention while retaining flexibility beyond the body itself.
  • Parameter regression: The resulting feature tensor regresses body shape and weak-perspective camera parameters, while each part feature row predicts its corresponding joint rotation.The camera includes scale and translation, and joint rotations are represented as 6D vectors during regression and loss computation.
  • Losses: The method combines 3D joint, 2D joint, SMPL-parameter, and part-label losses, with 2D projections computed from SMPL joints and a weak-perspective camera.The part-label loss is applied only during the initial training stages.

5. Experiments

Experiments evaluate PARE across standard and occlusion-focused benchmarks, compare architectural and augmentation choices, and analyze the role of part attention. PARE consistently improves robustness and reconstruction accuracy over existing approaches.

  • Experimental setup: PARE is trained on COCO, MPII, LSPET, MPI-INF-3DHP, and Human3.6M, with evaluation on 3DPW, 3DPW-OCC, and 3DOH.Part labels are obtained by rendering segmented SMPL meshes, with 24 parts corresponding to SMPL joints.
  • Comparison to state of the art: PARE improves PA-MPJPE by 10% compared to HMR-EFT, one of the best-performing recent methods.Results are reported with both ResNet-50 and HRNet-W32 backbones.
  • Occlusion benchmarks: PARE consistently outperforms HMR-EFT on occlusion-specific datasets, despite both methods using the same augmentation and training data.The comparison covers 3DPW-OCC and 3DOH under matched training conditions.
  • Occlusion robustness: Occlusion sensitivity analysis finds PARE more robust than SPIN and HMR-EFT, with lower overall and per-joint error under simulated occlusions.SPIN is especially sensitive to upper-body occlusions around the head and back, while augmentation alone is insufficient in qualitative comparisons.
  • Qualitative analysis: PARE attention maps expand toward larger, more distant image regions for occluded joints, such as the right hand.Qualitative comparisons include input images and SPIN, HMR-EFT, and PARE results on COCO and 3DPW.
  • Part-attention ablations: Part-segmentation supervision combined with soft attention works significantly better than sparse keypoint sampling, unsupervised attention, or segmentation-only alternatives.A two-sided t-test reports p<0.01 for comparisons between rows (c) and (d), (d) and (e), and (b) and (d).
  • Part-attention ablations: Mean part-segmentation IoU is 1%, 85%, and 74% for unsupervised, parts, and parts/unsupervised variants, while lower segmentation accuracy does not hurt reconstruction.The reported values are measured on the 3DPW test set.
  • Augmentation and backbones: Synthetic occlusion improves performance on both 3DPW and 3DPW-OCC, whereas early random cropping hurts performance; HRNet-W32 performs better than ResNet-50.PARE improves over HMR-EFT with both backbones, and crop augmentation is applied only after 175K training steps.

6. Conclusion

PARE addresses occlusion in 3D human pose and shape estimation by using body-part-driven soft attention and visibility cues. The method aggregates information from visible regions to reason about occluded joints and improves robustness across occlusion types.

  • PARE regresses 3D human pose and shape by exploiting the visibility of individual body parts to gain robustness to occlusion.
  • Its attention mechanism uses part segmentation masks as visibility cues and aggregates features from attended regions to reason about occluded joints.The mechanism is designed around dependencies between visible and occluded body parts.
  • PARE improves robustness to scene, self, and frame occlusion, with ablations showing how design choices contribute to state-of-the-art benchmark performance.

Supplementary Material

The supplementary material contains additional documentation and visualizations accompanying the method and results.

  • The supplementary material includes a document and video with acknowledgements, disclosure, additional information, and visualizations.

A. Methods

Implementation details specify initialization, training schedules, loss weights, body-part supervision, augmentation, and runtime. PARE runs at 14.8 ms per image, only 1 ms slower than HMR.

  • Implementation details: MPII-pretrained weights initialize both ResNet-50 and HRNet-W32 because they converge faster than ImageNet-pretrained weights.
  • Implementation details: PARE and baselines are trained on COCO for 175K steps for ablations, then evaluated on 3DPW and 3DPW-OCC.The full training data are incorporated for the state-of-the-art comparison.
  • Loss: The loss uses coefficients λ3D = 300, λ2D = 300, λSMPL = 60, and λP = 60.
  • Body-part supervision: SMPL annotations provide body-part supervision by rendering segmented meshes, with one corresponding label for each joint in the SMPL kinematic tree.
  • Augmentation: Synthetic occlusion and random-crop augmentations are used to improve robustness during training.
  • Runtime: 14.8 ms/image is PARE’s runtime on a GTX2080Ti, only 1 ms/image slower than HMR.

B.1. Training Datasets

PARE is evaluated using established datasets and comparisons spanning standard, occlusion-specific, 2D, and temporal-model settings. The supplementary material also documents training data composition and representative failure cases.

  • Training Datasets: Training uses MPI-INF-3DHP, Human3.6M, and in-the-wild COCO, MPII, and LSPET data, following prior work.The datasets provide 3D pose annotations, 2D keypoints, and pseudo-SMPL annotations from EFT.
  • Failure Cases: Reported failure scenarios include crowded crops, rarely seen extreme poses, children whose shapes exceed SMPL’s explanation, and extreme occlusion.
  • Evaluation: PARE outperforms video-based methods on 3DPW despite being run independently on each tracked frame without temporal smoothing.The comparison uses average joint errors in millimeters.
  • Evaluation: The supplementary comparisons include 2D keypoint projection accuracy on the 3DPW test set using percentage of correct keypoints.

C. More on visualizing attention of networks

This section introduces occlusion sensitivity maps and part attention maps to visualize how PARE responds to image perturbations and attends to body parts. These visualizations show greater robustness to localized occlusion and attention that broadens toward visible regions when parts are occluded.

  • Occlusion Sensitivity: The proposed occlusion sensitivity map replaces classification score with an MPJPE-based error measure to study how input perturbations affect pose output.The analysis uses MPJPE without Procrustes Alignment and is applied to both PARE and SPIN.
  • Occlusion Sensitivity: PARE is more robust to localized part occlusion than SPIN, whose warmer sensitivity regions indicate larger MPJPE increases when those pixels are occluded.The maps compare pixel occlusion effects relative to ground-truth body-pose error.
  • Occlusion Sensitivity: Occlusion sensitivity meshes aggregate per-pixel errors onto SMPL vertices and show SPIN's localized sensitivity, whereas PARE has more consistent cold colors across the body.For SPIN, right-arm or face occlusions can produce high right-wrist error; PARE's mesh again indicates greater robustness to partial occlusion.
  • Part Attention: PARE's warm attention regions resemble part-segmentation labels for visible parts, while attention widens over other parts and the scene for naturally occluded parts.The attention maps are visualized before softmax and suggest that PARE uses visibility information when attending to body parts.
  • Part Attention: Removing part supervision late in training decreases part-segmentation IoU but increases body-reconstruction accuracy, especially under occlusion.The comparison contrasts full part-segmentation supervision with supervision applied only during initial training stages.
  • Qualitative Comparisons: The section also presents qualitative comparisons with SPIN, HMR-EFT, Zhang et al., and VIBE, plus attention-map progression showing occluded-part maps focusing on visible parents.VIBE is video-based, whereas PARE is run independently on each video frame.
Loading 2104.08527v2…