PaperlayerView exact revision on arXiv

Source-linked AI summary

Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation

Hanbyul Joo, Natalia Neverova, Andrea Vedaldi

arXiv:2004.03686v3cs.CV

TL;DR

Large-scale 3D human pose annotations are scarce for realistic in-the-wild images. The paper introduces EFT to generate pseudo-3D annotations from existing 2D datasets, and shows that they support state-of-the-art regressors, simpler training, and benchmarks for difficult real-world conditions.

  • Problem

    Realistic in-the-wild images lack large-scale 3D ground-truth annotations, while existing 3D datasets are often indoor or laboratory-based.

  • Method

    EFT fine-tunes a pretrained image-based 3D pose regressor to fit 2D annotations, using its implicit pose prior as a pose parameterization.

  • Results

    The EFT pseudo-annotations are sufficient to train state-of-the-art 3D pose regressors that outperform previous methods on in-the-wild benchmarks.

  • Takeaways & Limitations

    Pseudo-annotated 2D datasets enable straightforward 3D regressor training and support extreme-crop, auxiliary-input, and real-world benchmark improvements.

  • Takeaways & Limitations

    Without an explicit pose-prior term, EFT can overfit a single sample and potentially cause the implicit prior to drift.

Abstract

from arXiv · show

Differently from 2D image datasets such as COCO, large-scale human datasets with 3D ground-truth annotations are very difficult to obtain in the wild. In this paper, we address this problem by augmenting existing 2D datasets with high-quality 3D pose fits. Remarkably, the resulting annotations are sufficient to train from scratch 3D pose regressor networks that outperform the current state-of-the-art on in-the-wild benchmarks such as 3DPW. Additionally, training on our augmented data is straightforward as it does not require to mix multiple and incompatible 2D and 3D datasets or to use complicated network architectures and training procedures. This simplified pipeline affords additional improvements, including injecting extreme crop augmentations to better reconstruct highly truncated people, and incorporating auxiliary inputs to improve 3D pose estimation accuracy. It also reduces the dependency on 3D datasets such as H36M that have restrictive licenses. We also use our method to introduce new benchmarks for the study of real-world challenges such as occlusions, truncations, and rare body poses. In order to obtain such high quality 3D pseudo-annotations, inspired by progress in internal learning, we introduce Exemplar Fine-Tuning (EFT). EFT combines the re-projection accuracy of fitting methods like SMPLify with a 3D pose prior implicitly captured by a pre-trained 3D pose regressor network. We show that EFT produces 3D annotations that result in better downstream performance and are qualitatively preferable in an extensive human-based assessment.

1. Introduction

The paper addresses the scarcity and limited realism of 3D training data by augmenting large-scale 2D datasets with pseudo-ground-truth 3D annotations. EFT enables these annotations, which support simpler training, improved robustness, and new real-world benchmarks.

  • Motivation: Existing 3D annotations are difficult to obtain in everyday scenarios and are concentrated in indoor or laboratory datasets.Prior methods often combine these datasets with in-the-wild 2D data using complicated training procedures.
  • Approach: The authors augment COCO, MPII, PoseTrack, and LSPet with pseudo-ground-truth 3D annotations.The annotations are inferred from single images but are reported to have surprisingly high quality.
  • Results: The pseudo-annotated data is sufficient to train state-of-the-art 3D pose regressors without combining incompatible 2D and 3D datasets.The resulting training pipeline is straightforward and outperforms prior methods on challenging in-the-wild benchmarks such as 3DPW.
  • Approach: EFT fine-tunes a pretrained image-based 3D pose regressor to fit 2D annotations while retaining its implicit pose prior.It replaces direct optimization of body parameters with optimization of the regressor weights.
  • Extensions: Extreme crop augmentation improves reconstruction of truncated people, while auxiliary inputs improve 3D pose estimation accuracy.The paper evaluates color-coded segmentation maps and DensePose IUV encodings as auxiliary inputs.
  • Benchmarks: The authors introduce benchmarks covering in-the-wild scenes, gymnastic poses, severe occlusions, and truncation.EFT-generated annotations are assessed through a large human study, and the benchmarks complement 3DPW with greater subject diversity.

2. Related Work

Prior single-view 3D human pose methods either fit parametric body models to image cues or regress model parameters directly. They differ in their use of pose priors, input cues, and predicted representations.

  • Fitting-based methods: Fitting-based methods optimize parametric body models such as SMPL against 2D joints, silhouettes, or part labels.SMPLify automated this fitting process and later work incorporated silhouettes and multiple people.
  • Regression-based methods: Regression-based methods predict 3D pose or body-model parameters directly from image-based cues using learned mappings.Inputs can include raw RGB values, sparse keypoints, or dense keypoints.
  • Generalization: Because indoor 3D datasets may not generalize to unconstrained images, several approaches combine them with in-the-wild datasets carrying 2D annotations.These combinations address the mismatch between available 3D supervision and realistic imagery.

3. Exemplar Fine Tuning

EFT reformulates 3D body fitting by optimizing the weights of a pretrained image-conditioned regressor rather than directly optimizing pose parameters. This combines image-conditioned pose structure with explicit 2D reprojection fitting.

  • Model: The SMPL model represents body shape and motion through pose parameters θ and shape parameters β.The pose controls rotations of 24 joints, while the shape controls body-shape variation through 10 parameters.
  • Classical fitting: Classical fitting jointly optimizes body and camera parameters to minimize 2D reprojection error while regularizing pose and shape.The pose prior addresses the ambiguity of inferring 3D pose from 2D keypoints, but initialization and loss balancing strongly affect optimization.
  • Regression prior: Regression networks learn an image-conditioned mapping from input cues to body-model parameters and implicitly capture a pose prior.Unlike unconditional fitting priors, this learned prior is conditioned on the image.
  • EFT: EFT treats the pretrained network as a re-parameterization of the body model and fine-tunes its weights on one image to improve the fit.After the individual update, the adapted weights are discarded, leaving the fitted body parameters for that example.
  • EFT: EFT drops the explicit pose-prior term and relies on the pretrained network's implicit prior while retaining image information during fitting.A large explicit prior weight in traditional fitting can favor a mean pose, whereas EFT falls back toward the pretrained regressor's image-conditioned estimate.

4. The EFT Training and Validation Datasets

EFT augments existing 2D pose datasets with 3D pseudo-ground-truth annotations for training and validation. The resulting datasets apply sanity checks, keypoint-based retention rules, and, for selected validation sets, strict human filtering.

  • Dataset construction: EFT augments COCO, MPII, LSPet, PoseTrack, and OCHuman with 3D pseudo-ground-truth annotations for downstream training and validation.The resulting datasets are denoted with the [·]EFT notation and are publicly available.
  • Dataset construction: COCO training samples are retained when they have at least 6 annotated keypoints, while COCO-Part requires 12 keypoints with all limbs present.
  • Validation: COCO-Val, LSPet, and OCHuman receive additional manual filtering before use for validation or benchmarking.
  • Validation: ≈20% of manually reviewed samples are rejected under the strict criterion that all three annotators must accept the estimated 3D annotation.The study uses three annotators per sample and accepts a sample only with unanimous agreement.
  • Crop evaluation: Figure 2 precomputes eight intermediate bounding-box levels, while yellow levels 1, 2, and 4 are used to evaluate truncation robustness in table 4.The boxes support crop augmentation and robustness testing on truncated bodies.

5. Training Pose Regressors with EFT datasets

EFT datasets enable training 3D pose regressors from scratch with a straightforward pipeline. The paper explores extreme crop augmentation and auxiliary inputs as performance-improving extensions.

  • Training pipeline: EFT datasets allow 3D pose regressors to be trained from scratch with a simple pipeline using HMR without its discriminator.The training uses the same hyperparameters as SPIN.
  • Augmentation: Extreme crop augmentation is introduced to improve regressor performance on highly truncated human bodies.
  • Auxiliary inputs: Auxiliary inputs are incorporated by concatenating RGB images with color-coded inputs, requiring a six-channel first HMR layer.The first ResNet50 layer is modified by duplicating its initial weights.

6. Results

EFT-generated pseudo-annotations support straightforward training of 3D pose regressors and improve performance across in-the-wild, truncated, occluded, and auxiliary-input settings. EFT also serves as a post-processing method, while overfitting generally preserves regression performance.

  • EFT Datasets for Learning Models: EFT pseudo-annotations are sufficient to train state-of-the-art 3D pose regressors from scratch, outperforming previous methods on in-the-wild benchmarks.The evaluation uses straight 3D supervision on EFT-lifted datasets and compares performance on 3DPW and H36M.
  • EFT Datasets for Learning Models: 57.5 mm PA-MPJPE is achieved on 3DPW by training with [COCO-Train]EFT, while [COCO-Train]EFT + H36M + MPI-INF-3DHP reaches 54.7 mm and adding 3DPW training data reaches 51.6 mm.[COCO-Part]EFT is already comparable to SPIN, and combining EFT and 3D datasets improves performance further.
  • EFT Datasets for Learning Models: Models trained with EFT annotations outperform models trained with SMPLify-based pseudo-annotations, indicating higher pseudo-ground-truth quality for downstream training.The comparison includes pseudo-annotations generated using EFT, SPIN, SMPLify, and SMPLify-X with manual filtering.
  • Learning Models with Auxiliary Inputs: DensePose auxiliary inputs improve in-the-wild accuracy, with [COCO-Train]EFT reaching 56.1 mm and outperforming the video-based VIBE result of 56.5 mm.Segmentation encodings do not noticeably improve performance in the reported comparison.
  • New 3D Human Pose Benchmarks: Extreme crop augmentation improves performance under severe truncation without degrading performance on the original 3DPW benchmark.[COCO-Train]EFT already performs better than several baselines without crop augmentation, and crop augmentation further improves extreme crop levels.
  • EFT as A Post-processing Method: EFT improves test-time performance as a post-processing step across initialization settings, whereas SMPLify can degrade accuracy when initialization is already good.The comparison is conducted on 3DPW using identical 2D keypoint annotations and varying fitting settings.
  • Overfitting Analysis of EFT: Overfitting regressors to individual samples with EFT usually has a small effect on overall 3DPW regression performance, although strongly affected samples can contain occlusions or annotation errors.The analysis evaluates 500 samples with 20 and 100 EFT iterations.

7. Discussion

EFT provides large-scale, high-quality pseudo-ground-truth 3D annotations that can train state-of-the-art regressors while simplifying preprocessing and enabling links to other computer-vision tasks.

  • EFT datasets provide high-quality pseudo-GT 3D pose annotations sufficient to train state-of-the-art regressors.The annotations are generated for popular large-scale 2D datasets such as COCO.
  • EFT removes the need for complicated preprocessing or balancing techniques when training 3D pose regressors.
  • EFT annotations create opportunities to relate 3D human pose estimation to other computer-vision tasks.

Supplementary Material

The supplementary material documents gold-standard SMPLify and EFT implementations, their optimization choices, and why their learning rates cannot be directly compared.

  • The supplement provides implementation details for gold-standard SMPLify and EFT settings, plus additional experiments and benchmark evaluations.
  • Implementation details: SMPLify uses pixel-space reprojection with robust and prior terms, while EFT uses normalized-space L2 reprojection and additional lower-leg orientation supervision.Both methods optimize with Adam and stop when the average 2D reprojection error reaches 3 pixels or after 50 iterations.
  • Implementation details: EFT and SMPLify use different learning rates because they optimize neural-network weights versus a direct 85-dimensional SMPL parameter vector.EFT optimizes roughly 27 million network parameters through nonlinear computations, so the same learning rate is inappropriate.
  • The supplement discusses SMPLify’s gold-standard comparison setting and directs readers to the main manuscript for the corresponding results.

B. Comparison SMPLify and EFT in the Same Data Term Setting (main paper sec. 6.5)

With identical data terms, the comparison shows that SMPLify is sensitive to initialization and prior weighting, whereas EFT consistently improves accuracy and is less sensitive to its regularizer.

  • The comparison uses the same reprojection data term for EFT and SMPLify while varying each method’s prior or regularizer weight.
  • Results: SMPLify’s post-processing produces mixed results that depend on initialization quality and pose-prior weighting.Prior terms can help poor initializations but degrade results when the regressor initialization is already accurate.
  • Results: 51.72mm to 57.64mm changes in the final row show that small weight adjustments can materially affect SMPLify performance.An extremely strong γ = 100 setting converges to an accuracy around 86mm under fixed 100-iteration optimization.
  • Results: EFT improves accuracy in all tested initialization cases and remains insensitive to changes in its regularizer weight.Even an extremely large γ = 100 does not degenerate the regressor’s initial accuracy in the reported experiments.
  • Scope: The experiment is intended to expose the methods’ major difference, not to identify universally optimal settings for each method.The authors note that SMPLify’s best setting on 3DPW may not transfer to other cases.

C. Further Evaluation on Standard Benchmarks (main paper sec. 6.2)

Further evaluations show that EFT-trained models remain competitive across standard benchmarks, auxiliary inputs can improve performance, and human judgments favor EFT meshes in many cases.

  • PA-MPJPE after rigid alignment is used to evaluate models on standard benchmarks, with additional Per-Vertex Error reported.
  • MPI-INF-3DHP: EFT-trained models are competitive on MPI-INF-3DHP, and combining EFT annotations with 3D datasets improves performance further.
  • Baselines: 2D-loss-only models show very poor 3D pose accuracy, despite retaining informative 2D projections of their estimated keypoints.
  • Auxiliary inputs: Auxiliary inputs including DensePose and segmentation maps are evaluated alongside RGB-only models on 3DPW.
  • EFT behavior: Overfitting individual exemplars usually has a small effect on overall regression performance, although effects vary for occluded or incorrectly annotated samples.
  • Human evaluation: 61.8% of AMT evaluations preferred EFT, while 132 of 500 samples were unanimously favored by all three annotators.SMPLify was unanimously preferred on 47 of 500 samples, without an obvious failure pattern.
  • Runtime: EFT takes about 0.82 seconds per sample with 20 iterations, compared with about 1.5 seconds for SMPLify with 50 iterations.
  • 2D localization: Hips and ankles are harder to localize because of loose clothing and occlusions, with higher COCO OKS sigma values than shoulders and wrists.The reported OKS sigma values are 1.07 for hips, 0.89 for ankles, 0.79 for shoulders, and 0.62 for wrists.

I. Statistics of our EFT datasets

The EFT datasets expose distinct 3D pose distributions across in-the-wild sources, while fitting analyses identify annotation and association errors that affect pose quality.

  • Pose-parameter statistics compare COCO, MPII, and LSPet using joint-wise angle-axis standard deviations and hip-angle histograms.
  • LSPet contains more large angle-axis values than COCO, indicating more gymnastic poses, whereas COCO concentrates near standing-pose values.
  • SPIN fitting can incorrectly associate OpenPose detections from other individuals with the target person.
  • The authors therefore exclude OpenPose estimates and use only ground-truth annotations during their SMPLify process.
  • Examples of EFT overfitting failures include annotations on occluded body parts and left-right swaps.
  • DensePose annotations can be automatically generated from pseudo-ground-truth data.

L. Potential Applications for EFT Datasets

EFT datasets support additional applications beyond regressor training, including derived DensePose annotations and searches for similar 3D poses in outdoor images.

  • The EFT datasets are presented as a source of high-quality pseudo-ground-truth 3D pose data for new research opportunities.
  • EFT pseudo-ground-truth data can automatically produce DensePose annotations for potential training use.
  • Nearest-neighbor search can identify similar 3D human poses in the COCO EFT dataset.
  • Human evaluations compare EFT and SMPLify meshes, with separate figures showing cases favored by all three annotators.
Loading 2004.03686v3…