PaperlayerView exact revision on arXiv

Source-linked AI summary

AGORA: Avatars in Geography Optimized for Regression Analysis

Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, Michael J. Black

arXiv:2104.14643v1cs.CV

TL;DR

AGORA addresses the gap between improving benchmark performance and failures in complex real-world scenes by introducing a realistic synthetic dataset with accurate 3D ground truth. It fits SMPL-X to diverse scans, evaluates pose, shape, hands, face, occlusion, and detection, and reports improved performance after fine-tuning, including on 3DPW.

  • Problem

    Existing 3DHPS benchmarks simplify clothing, environments, subjects, occlusion, and annotations, limiting evaluation of robust methods in complex scenes.

  • Method

    AGORA renders diverse scanned people in realistic scenes and fits SMPL-X models to provide body, hand, face, and shape ground truth, including a child shape extension.

  • Results

    AGORA reveals limitations of current methods, while fine-tuning on the synthetic dataset improves a state-of-the-art method on natural 3DPW data.

  • Takeaways & Limitations

    AGORA provides a challenging benchmark, training data, and evaluation protocol for advancing 3D human pose and shape estimation.

  • Takeaways & Limitations

    The fitting pipeline sometimes fails for hands and faces, especially when hands grasp objects, so evaluation uses body-only subsets where necessary.

Abstract

from arXiv · show

While the accuracy of 3D human pose estimation from images has steadily improved on benchmark datasets, the best methods still fail in many real-world scenarios. This suggests that there is a domain gap between current datasets and common scenes containing people. To obtain ground-truth 3D pose, current datasets limit the complexity of clothing, environmental conditions, number of subjects, and occlusion. Moreover, current datasets evaluate sparse 3D joint locations corresponding to the major joints of the body, ignoring the hand pose and the face shape. To evaluate the current state-of-the-art methods on more challenging images, and to drive the field to address new problems, we introduce AGORA, a synthetic dataset with high realism and highly accurate ground truth. Here we use 4240 commercially-available, high-quality, textured human scans in diverse poses and natural clothing; this includes 257 scans of children. We create reference 3D poses and body shapes by fitting the SMPL-X body model (with face and hands) to the 3D scans, taking into account clothing. We create around 14K training and 3K test images by rendering between 5 and 15 people per image using either image-based lighting or rendered 3D environments, taking care to make the images physically plausible and photoreal. In total, AGORA consists of 173K individual person crops. We evaluate existing state-of-the-art methods for 3D human pose estimation on this dataset and find that most methods perform poorly on images of children. Hence, we extend the SMPL-X model to better capture the shape of children. Additionally, we fine-tune methods on AGORA and show improved performance on both AGORA and 3DPW, confirming the realism of the dataset. We provide all the registered 3D reference training data, rendered images, and a web-based evaluation site at https://agora.is.tue.mpg.de/.

1. Introduction

AGORA addresses limitations of saturated 3DHPS benchmarks by providing realistic, challenging images with accurate 3D ground truth. It also introduces child-specific modeling and evaluation resources for advancing and assessing current methods.

  • Existing 3DHPS datasets simplify clothing, occlusion, scenes, demographics, subjects, and evaluated body structure, limiting robust evaluation.The benchmarks often use laboratory environments, single subjects, and sparse major-joint annotations.
  • AGORA provides high-resolution images with accurate 3D bodies in varied clothing, poses, ages, ethnicities, scenes, lighting, occlusion, crowds, and spatial scales.It includes face and hand pose, children, and 4K imagery.
  • 4240 textured scans, including 257 child scans, are fitted with SMPL-X to capture body shape, pose, hands, and facial shape under clothing.The resulting fits have an average error of 5mm, and SMPL ground truth is also provided for backward compatibility.
  • AGORA extends SMPL-X with an interpolated adult-infant shape dimension because existing methods perform poorly on children.The added parameter interpolates between an adult SMPL-X template and an infant SMIL template.
  • The release includes registered training and validation ground truth, test images, evaluation code, and a web evaluation server.Commercial scans are not released, but a shopping list enables others to purchase them for related applications.
  • The evaluation protocol measures joints, vertices, body, hands, face, occlusion, and detection accuracy for multi-person scenes.It introduces an error measure that accounts for both missed and falsely detected people.

2. Related Work

Prior 3DHPS datasets obtain 3D annotations through controlled capture or synthetic generation, but each leaves important realism or coverage gaps. AGORA is positioned as a benchmark combining realistic complexity with high-quality 3D ground truth.

  • Real-image datasets use mocap, markerless capture, or IMUs, but remain limited in clothing, occlusion, scene variety, ethnicity, and face-hand coverage.Markerless systems are typically less accurate than marker-based systems, while IMUs are affected by yaw drift.
  • Synthetic datasets scale ground-truth generation through parametric human models, but must achieve realism in bodies, motion, clothing, textures, and environments.Several approaches composite rendered people onto background images, while others position models in 3D scenes.
  • Learned human-synthesis methods can place people consistently with scene geometry and lighting, but generative artifacts make their outputs unsuitable as ground truth.
  • Datasets without 3D ground truth address selected 2D concerns such as occlusion, landmarks, silhouettes, or large-scale training data.
  • AGORA combines realistic textures, complex clothing and scenes, varied lighting and occlusion, 4K imagery, and high-quality 3D ground truth.The benchmark is intended both to reveal current-method limitations and to provide training data for multiple applications.

3. Method: Obtaining reference data

The reference-data pipeline fits SMPL-X models to textured scans using pose, shape, expression, semantic, and geometric information. A child-specific template interpolation addresses distortions that adult models produce on child scans.

  • Obtaining reference data: AGORA selects 4240 textured scans and represents each fitted person with SMPL-X parameters for pose, shape, facial expression, and hand pose.Hand pose is represented through a six-dimensional PCA latent vector.
  • Obtaining reference data: Fitting optimizes SMPL-X parameters so the model resembles each scan while accounting for clothing and enforcing consistent shape across scans of one identity.
  • Obtaining reference data: The pipeline initializes fits from multi-view SMPLify-X and refines pose and shape against the 3D scan surface.It combines 2D landmarks with skin and clothing geometry because landmarks alone provide little body-shape information.
  • Obtaining reference data: Skin and clothing energy terms bring the model toward the scan while discouraging clothing penetration, supplemented by landmark and regularization terms.
  • Fitting child scans: Direct SMPL-X fitting distorts child scans because the adult model cannot represent children adequately.
  • Fitting child scans: Interpolating adult and converted infant templates approximately captures children’s body shapes better than direct fitting or naive adult scaling.The interpolation is expressed as αTA + (1 − α)TC.

4. AGORA Dataset

AGORA combines carefully curated SMPL-X fits with challenging multi-person imagery and evaluation metrics that assess pose, shape, detection, and detailed body regions. Its ground truth is accurate enough for benchmarking, while explicit scan subsets and evaluation choices define the supported scope.

  • Dataset curation: 4,240 scans are partitioned into 3,161 BFH scans with aligned body, face, and hands and 1,079 B scans with aligned bodies only.Body-only evaluation is used for B scans, while body, hand, and face evaluation is used for BFH scans.
  • Dataset splits: AGORA provides approximately 3,387 withheld test images, plus 14,529 training and 1,225 validation images with released SMPL-X parameters and masks.Test fits are withheld, while training and validation ground truth are released for training and evaluation.
  • Fitting accuracy: SMPL-X fits achieve approximately 4.73mm average skin error, with 16% of clothing vertices penetrating the body at an average distance of 4.63mm.The authors regard the approximately 5mm fitting error as valid pseudo ground truth for benchmarking.
  • Evaluation metrics: AGORA evaluates pose, shape, hands, and face while retaining absolute scene arrangement through pelvis alignment instead of Procrustes alignment.SMPL-X methods receive separate body, hand, face, and weighted full-body scores; vertex metrics additionally evaluate shape.
  • Evaluation metrics: NMJE and NMVE normalize joint and vertex errors by F1 score, penalizing missed and false-positive detections in images containing 5–15 people.This requires methods to detect every person, avoid spurious detections, and estimate accurate poses and shapes for matched people.

5. Experiments

AGORA evaluates 3D human pose and shape methods under multi-person, occlusion, child-shape, and perspective-related challenges. Fine-tuning SPIN on AGORA improves performance on both AGORA and 3DPW.

  • Baseline Evaluation: AGORA evaluates existing methods using metrics that account for pose, body, hands, face, occlusion, missed detections, and false positives.The protocol evaluates both reconstruction accuracy and detection performance in multi-person images.
  • Baseline Evaluation: SPIN fine-tuned on AGORA outperforms other SMPL-based methods on MPJPE and MVE, but its NMJE and NMVE worsen because of misses and false positives.This demonstrates that MPJPE alone does not capture multi-person detection failures.
  • Baseline Evaluation: Performance degrades with increasing occlusion, while CenterHMR handles high occlusion better but suffers many misses, particularly for small people.FrankMocap and SPIN are highly sensitive to occlusion; AGORA fine-tuning improves SPIN in high-occlusion cases.
  • Baseline Evaluation: Error consistently increases as people move farther from the image center, reflecting the limits of weak-perspective camera assumptions.AGORA’s large field of view enables this analysis of off-center subjects.
  • Baseline Evaluation: SOTA methods perform significantly worse on child shape than adult shape, motivating the child-shape representation introduced with AGORA.The comparison uses body joint and vertex error on the AGORA kids subset.
  • Baseline Improvement: AGORA fine-tuning improves MPJPE by ∼12% on 3DPW and ∼13% on AGORA.The comparison evaluates pretrained SPIN against SPIN fine-tuned with AGORA and EFT.

6. Conclusions and Future Work

AGORA is a challenging dataset designed to expose limitations in 3D human pose and shape estimation. Despite being synthetic, it improves a state-of-the-art method on natural 3DPW data and introduces evaluation and child-body modeling advances.

  • Conclusions: AGORA includes environmental and person-person occlusion, scale variation, children, crowds, and other challenging cases absent from simpler benchmarks.The authors identify additional future extensions including varied camera heights, indoor scenes, multi-view images, larger crowds, animals, and movement.
  • Conclusions: Fine-tuning on synthetic AGORA data improves a state-of-the-art method on the natural 3DPW dataset.The conclusion presents this as evidence that AGORA can improve performance beyond its synthetic setting.
  • Conclusions: The paper introduces a metric that penalizes misses and false positives, plus a child body model and improved 3D ground truth for images with children.These additions target evaluation and modeling limitations in multi-person and child-pose settings.

A. Method

The method initializes SMPL-X by extending single-view fitting to multiple rendered views with known camera parameters, then refines the fit using regularized optimization. The objective combines landmark agreement with priors on pose, hands, shape, and facial expression.

  • A.1. Multi-view Initialization: Multi-view fitting initializes SMPL-X parameters from landmarks detected across rendered views of each scan.The approach extends single-view SMPLify-X and uses predefined virtual cameras.
  • A.1. Multi-view Initialization: The optimization adapts SMPLify-X to multiple views by adding per-camera landmark terms to the regularized objective.Known camera intrinsics and extrinsics remove the need to estimate camera translation first.
  • A.1. Multi-view Initialization: Regularization includes L2 priors for body pose, hand pose, body shape, and facial expression.Additional terms penalize strong elbow and knee bending and prevent mesh intersections.

A.2. 2D+3D Refinement

The refinement fits SMPL-X to scan geometry while separating skin and clothing, using robust, probability-weighted terms and special handling for clothing penetration and loose garments. The appendix also documents child-template interpolation, dataset attributes, occlusion masks, and refinement-error comparisons.

  • A.2. 2D+3D Refinement: Skin and clothing scan vertices are identified from vendor masks or Graphonomy labels aggregated across rendered views.Projected visible vertices receive labels that are combined into skin, clothing, or other probabilities.
  • A.2. 2D+3D Refinement: The skin term minimizes robust point-to-surface distances weighted by each scan vertex’s skin probability.The Geman-McClure function reduces the influence of outliers.
  • A.2. 2D+3D Refinement: The clothing term penalizes garment points penetrating the body while keeping the model close to loose clothing.Penetrating and outside points receive different treatments, including robust handling for garments such as skirts and robes.
  • A.2. 2D+3D Refinement: The alternative probability placement yields 4.73mm skin error and 4.42mm cloth error, versus 4.82mm and 4.51mm for Eqs. 3 and 4.The reported difference is negligible.
  • A.2. 2D+3D Refinement: The child-body template interpolates between adult SMPL-X and the SMIL infant template to fit 257 child scans.Varying the interpolation coefficient produces approximate templates for different age groups.
  • AGORA Statistics: AGORA includes varied age, ethnicity, and gender distributions, with masks used to measure person occlusion.The dataset also contains challenging environmental occlusion in rendered scenes.

C. Additional Analysis on Baseline Experiment

This section provides detailed evaluation-protocol information and additional analysis of the baseline experiments, complementing the main paper’s shorter description and visualizations.

  • C. Additional Analysis on Baseline Experiment: The section expands the evaluation protocol beyond the main paper’s less detailed description.It provides additional procedural detail for interpreting the experiments.
  • C. Additional Analysis on Baseline Experiment: The section includes additional analysis of the baseline experiments.This analysis accompanies the protocol details.

C.1. Evaluation Protocol.

The evaluation protocol covers detection, 3D pose and shape estimation, and prediction-to-ground-truth matching across several state-of-the-art methods. It also specifies constrained OpenPose settings, occlusion-mask examples, an IoU-based false-positive rule, and a SPIN training comparison.

  • C.1. Evaluation Protocol: The protocol evaluates ExPose, FrankMocap, SMPLify-X, HMR, SPIN, EFT, and CenterHMR across detection, estimation, and matching stages.These methods are selected to represent the current state of the art.
  • C.1. Evaluation Protocol: OpenPose processing scales the larger image side to 272 pixels and uses two body-keypoint scales separated by 0.25.The settings are modified to run on a common GPU with 12 GB of memory.
  • C.1. Evaluation Protocol: Person masks are illustrated as color images, full masks, and individual masks rendered without occlusions.These masks support measuring how much a person is occluded.
  • C.1. Evaluation Protocol: Table 5 compares training SPIN from scratch with Human3.6M against EFT and AGORA.The table organizes an additional training-source comparison for SPIN.
  • C.1. Evaluation Protocol: Predictions with IoU below 0.1 are treated as false positives when no other possible match exists.The IoU is computed from 2D axis-aligned bounding boxes derived from keypoints.

C.2. Evaluation of Methods

The evaluation compares qualitative outputs from several 3D human pose-and-shape methods and analyzes how errors vary with body orientation. Errors increase as subjects turn away from the camera, peak near 180°, and then decrease.

  • Orientation: Errors grow with yaw angle, peak around 180°, then decrease as the subject turns further.Here, 0° denotes facing the camera.
  • Orientation: The methods perform worst when subjects are not facing the camera.
  • Figure 13 compares RGB inputs with outputs from FrankMocap, ExPose, CenterHMR, HMR, SMPLify-X, and SPIN.
Loading 2104.14643v1…