PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation

Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, Youliang Yan

arXiv:2208.00571v2cs.CV

TL;DR

Cropping in top-down estimation discards location information needed for accurate global rotation in the original camera coordinate system. CLIFF restores that information through bounding-box inputs and full-frame reprojection, and uses a CLIFF-based annotator to produce pseudo-GT for in-the-wild data. It outperforms prior methods across reported benchmarks and reaches first place on the AGORA SMPL-Algorithms leaderboard, while using an approximate focal length when ground truth is unavailable.

  • Problem

    Cropping discards full-frame location information, limiting accurate global-rotation estimation, while 3D supervision for diverse in-the-wild data is scarce and costly.

  • Method

    CLIFF concatenates cropped-image features with bounding-box information, applies full-frame 2D reprojection supervision, and supports a pseudo-GT annotator trained with pretrained and target-dataset supervision.

  • Results

    CLIFF outperforms prior methods by significant margins on several metrics and reaches first place on the AGORA SMPL-Algorithms leaderboard.

  • Takeaways & Limitations

    Global-location-aware information improves global rotation and articulated pose estimation while enabling high-quality 3D annotation for in-the-wild 2D datasets.

  • Takeaways & Limitations

    When camera focal length is unknown, CLIFF approximately estimates it using the full image dimensions and a 55° diagonal field of view.

Abstract

from arXiv · show

Top-down methods dominate the field of 3D human pose and shape estimation, because they are decoupled from human detection and allow researchers to focus on the core problem. However, cropping, their first step, discards the location information from the very beginning, which makes themselves unable to accurately predict the global rotation in the original camera coordinate system. To address this problem, we propose to Carry Location Information in Full Frames (CLIFF) into this task. Specifically, we feed more holistic features to CLIFF by concatenating the cropped-image feature with its bounding box information. We calculate the 2D reprojection loss with a broader view of the full frame, taking a projection process similar to that of the person projected in the image. Fed and supervised by global-location-aware information, CLIFF directly predicts the global rotation along with more accurate articulated poses. Besides, we propose a pseudo-ground-truth annotator based on CLIFF, which provides high-quality 3D annotations for in-the-wild 2D datasets and offers crucial full supervision for regression-based methods. Extensive experiments on popular benchmarks show that CLIFF outperforms prior arts by a significant margin, and reaches the first place on the AGORA leaderboard (the SMPL-Algorithms track). The code and data are available at https://github.com/huawei-noah/noah-research/tree/master/CLIFF.

1 Introduction

Top-down 3D human pose and shape estimation benefits from cropped-image processing but loses full-frame location information needed for accurate global rotation. CLIFF restores this information through bounding-box inputs and full-frame reprojection, and extends it to pseudo-GT annotation for in-the-wild data.

  • Single-image 3D human pose and shape estimation reconstructs human meshes despite depth ambiguity, supporting applications such as AR/VR, telepresence, and action analysis.
  • Top-down methods dominate because they decouple human detection from pose estimation and provide high recall and precision.
  • Cropping discards full-frame location information, making accurate global-rotation estimation difficult and rendering cropped-image reprojection supervision improper.
  • CLIFF concatenates cropped-image features with bounding-box information and computes 2D reprojection loss by projecting predicted joints onto the full image.
  • The CLIFF-based pseudo-GT annotator produces high-quality 3D annotations for in-the-wild images, addressing scarce and costly SMPL supervision.
  • 5.7mm MPJPE and 6.5mm PVE on 3DPW accompany significant-margin improvements and first place on the AGORA SMPL-Algorithms leaderboard.

2 Related Work

CLIFF extends top-down HMR-based estimation by retaining person location and supervising reprojection in the full image, addressing global-rotation errors caused by cropping. It also supports pseudo-ground-truth annotation for in-the-wild datasets using only 2D keypoint supervision.

  • CLIFF Model: CLIFF adds bounding-box information to cropped-image features, encoding the crop’s location and size so the regressor can estimate global rotation relative to the full camera.The bounding-box representation includes normalized center coordinates and crop size; its first terms encode the transformation angle between cropped and full-camera coordinates.
  • CLIFF Model: CLIFF transforms root translation from the crop camera to the full camera and projects predicted 3D joints onto the full image plane for reprojection supervision.This replaces the cropped-image reprojection used by HMR and previous top-down methods.
  • CLIFF Annotator: The CLIFF annotator pretrains with available 3D ground truth, predicts target-dataset SMPL parameters, fine-tunes with 2D keypoints and parameter regularization, then outputs pseudo-ground truth.The annotator is designed for in-the-wild datasets where 3D annotations are scarce and costly, while 2D keypoints are easier to obtain.
  • CLIFF Annotator: The annotator uses global-location-aware inputs and supervision to produce better global-rotation and articulated-pose annotations, without requiring an extra AMASS-trained prior model.The paper reports that these annotations improve training performance.

4 Experiments and Results

CLIFF is evaluated against prior methods, through ablations, pseudo-GT comparisons, and qualitative results across multiple datasets. It improves global rotation estimation and produces useful pseudo-GT annotations for in-the-wild images.

  • Evaluation setup: CLIFF is evaluated on 3DPW, Human3.6M, and AGORA using standard Euclidean-distance metrics in millimeters.The metrics include MPJPE, PA-MPJPE, and PVE.
  • Comparison with prior methods: CLIFF outperforms prior methods by significant margins across all metrics on the three evaluation datasets.With a ResNet-50 backbone, it reduces MPJPE and PVE by more than 13mm versus HMR-EFT.
  • Comparison with prior methods: CLIFF reaches first place on the AGORA leaderboard in the SMPL-Algorithms track.The comparison includes methods with similar image backbones and transformer-based architectures.
  • Qualitative comparison: CLIFF produces better global alignment than PARE on 3DPW, with less pixel misalignment and closer novel-view overlays to ground truth.The articulated poses can be accurate for both methods, while CLIFF’s advantage is attributed to more accurate global rotation estimation.
  • Ablation study: Removing bounding-box input worsens MPJPE, and additionally removing full-frame reprojection supervision causes larger errors in the Human3.6M ablation.The ablation associates bounding-box information with improved global rotation estimation and full-frame supervision with better predictions.
  • Pseudo-GT annotation: The CLIFF annotator outperforms other pseudo-GT annotators on 3DPW and improves downstream training with pseudo-GT from in-the-wild 2D images.It reduces MPJPE and PVE by 12.3mm and 20.5mm versus Pose2Mesh, and exceeds SPIN and EFT by more than 13mm on both metrics.
  • Pseudo-GT annotation: CLIFF annotator meshes are pixel-aligned to image evidence and perceptually realistic from side views when 2D keypoints are good.The qualitative results attribute plausibility to the annotator’s strong priors.

5 Discussion

The CLIFF idea can extend beyond its HMR-based implementation to other regression-based top-down methods and potentially to bottom-up methods with alternative location encodings.

  • Applicability: CLIFF can benefit regression-based top-down methods using cropped-region image, keypoint, edge, or silhouette features.For bottom-up methods, the discussion suggests encoding location information with alternatives such as a location map.

6 Conclusion

CLIFF addresses the loss of full-frame location information by adding bounding-box input and full-image reprojection supervision, and it supports pseudo-GT annotation for in-the-wild 2D datasets. Experiments report significant gains over prior methods and first place on AGORA’s SMPL-Algorithms track.

  • Problem: Cropping discards full-frame location information, limiting accurate global-rotation inference from cropped images.The paper identifies full-frame global location as important for 3D human pose and shape estimation.
  • Method: CLIFF combines cropped-image features with bounding-box information and computes 2D reprojection loss in the full image.The predicted 3D joints are projected in a way similar to the person’s projection in the image.
  • Method: A CLIFF-based pseudo-GT annotator generates high-quality 3D annotations for in-the-wild 2D datasets to support regression-based training.The annotator is presented as providing full supervision where 3D annotations are unavailable.
  • Results: CLIFF outperforms state-of-the-art methods by significant margins and ranks first on the AGORA leaderboard’s SMPL-Algorithms track.This conclusion summarizes the reported benchmark outcome.

Supplementary Materials

The supplementary materials derive a main-paper equation, discuss CLIFF’s inputs and video use, and provide further annotator details.

  • Supplementary contents: The supplement derives Equation 7 from the main paper.It begins by presenting the derivation before discussing video application and annotator details.
  • Supplementary contents: The supplement discusses CLIFF input design and its performance when applied frame by frame to videos.It also covers additional details about the CLIFF annotator.

1 Derivation of Equation 7

CLIFF converts weak-perspective parameters and crop geometry into full-frame root translation, then uses perspective projection for reprojection supervision.

  • 1 Derivation of Equation 7: Equation 7 converts weak-perspective scale and translations, crop location, crop size, and focal length into full-frame translation parameters.The resulting translation is expressed in the original camera coordinate system.
  • 1 Derivation of Equation 7: The horizontal and vertical translations add crop-center offsets, 2 · c_x / (b · s) and 2 · c_y / (b · s), to weak-perspective translations.c_x and c_y specify crop location relative to the full-image center.
  • 1 Derivation of Equation 7: The full-frame depth is t_Z^full = 2 · f_CLIFF / (b · s), derived by projecting a 2m × 2m body-enclosing box.Here b and f_CLIFF are measured in pixels, while t_Z^full is measured in meters.
  • 1 Derivation of Equation 7: CLIFF uses perspective reprojection with an appropriate focal length because the original image is captured by perspective projection.The model still predicts weak-perspective projection parameters for most cases.

2 Impact of the BBox Quality

CLIFF remains effective with imperfect bounding boxes, although bounding-box quality remains important as in other top-down methods.

  • 2 Impact of the BBox Quality: CLIFF reaches first place on the AGORA leaderboard using Mask R-CNN boxes trained on COCO without AGORA fine-tuning.AGORA includes crowded and severely occluded scenes.
  • 2 Impact of the BBox Quality: Random scaling and cropping augmentations make CLIFF robust to inaccurate bounding-box detection.The method remains effective when large body parts are missing in bounding boxes.

3 Impact of the Focal Length as Part of the Input

CLIFF is robust to substantial focal-length estimation errors within a broad range, while the focal length is often available in practical applications.

  • 3 Impact of the Focal Length as Part of the Input: Less than 5% error increase occurs when the estimated focal length lies in [0.4f_GT, 3f_GT].This result comes from perturbing the ground-truth focal length on the 3DPW test set.
  • 3 Impact of the Focal Length as Part of the Input: The CLIFF focal-length estimate f_CLIFF = √(w^2 + h^2) falls within this robust range for most cases except super telephotos.The passage also states that ground-truth focal length is often known in practical applications.

4 Smoothness Comparison with Video-Based Methods

Frame-by-frame CLIFF can achieve smoothness comparable to video-based methods, and OneEuro filtering improves smoothness with slightly larger pose errors.

  • 4 Smoothness Comparison with Video-Based Methods: CLIFF processes video frame by frame and can apply temporal smoothing such as OneEuro filtering.Video-based methods instead process additional adjacent frames and therefore require more computation.
  • 4 Smoothness Comparison with Video-Based Methods: The CLIFF annotator's evaluation errors do not diverge during long training, so a generic stopping criterion need not be selected carefully.
  • 4 Smoothness Comparison with Video-Based Methods: CLIFF has comparable smoothness performance to video-based methods before post-processing.
  • 4 Smoothness Comparison with Video-Based Methods: OneEuro filtering significantly improves smoothness with negligible extra computation and slightly larger pose errors.The resulting pose errors remain much smaller than those of the competitors.

5 CLIFF Annotator Training

The CLIFF annotator remains effective across base-model comparisons and optimization, while qualitative results show pseudo-GT that improves alignment without sacrificing plausibility.

  • The CLIFF annotator is robust during optimization because its priors prevent overfitting to 2D keypoints and implausible poses.On 3DPW, evaluation errors do not diverge through 120 training epochs.
  • Switching the base model from CLIFF to HMR increases errors, but the HMR-based annotator remains better than other SOTA methods.
  • The final pseudo-GT achieves better pixel alignment while maintaining plausibility with the proposed priors.Figure 5 compares 2D annotations, explicit-prior views, and pseudo-GT views from front and side perspectives.
Loading 2208.00571v2…