PaperlayerView exact revision on arXiv

Source-linked AI summary

GFPose: Learning 3D Human Pose Prior with Gradient Fields

Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, Yizhou Wang

arXiv:2212.08641v1cs.CV

TL;DR

Pose priors can help distinguish feasible 3D human poses and recover poses from ambiguous or partial observations. GFPose learns task-conditional pose scores with hierarchical condition masking, then reports strong results across estimation and other pose tasks. Its reverse sampling process, however, takes time proportional to the number of sampling steps.

  • Problem

    Biomechanical constraints make natural poses lie on a low-dimensional manifold, while pose recovery may rely on ambiguous, occluded, noisy, or partial observations.

    Source: §1 · p. 1

  • Method

    GFPose learns a time-dependent score for task-conditional 3D pose distributions and uses hierarchical condition masking to support different pose tasks.

    Source: §1 · p. 2

  • Results

    GFPose outperforms state-of-the-art methods in multi- and single-hypothesis pose estimation, shows stronger robustness to severe occlusions in completion, and generates diverse, realistic pose samples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors demonstrate GFPose as a unified framework for multiple pose tasks, including estimation, denoising, completion, and generation.

    Source: §1 · p. 2, §6 · p. 8

    Inference time grows with the number of sampling steps, which may limit very large deep networks in real-time scenarios.

    Source: §6 · p. 8

Abstract

from arXiv · show

Learning 3D human pose prior is essential to human-centered AI. Here, we present GFPose, a versatile framework to model plausible 3D human poses for various applications. At the core of GFPose is a time-dependent score network, which estimates the gradient on each body joint and progressively denoises the perturbed 3D human pose to match a given task specification. During the denoising process, GFPose implicitly incorporates pose priors in gradients and unifies various discriminative and generative tasks in an elegant framework. Despite the simplicity, GFPose demonstrates great potential in several downstream tasks. Our experiments empirically show that 1) as a multi-hypothesis pose estimator, GFPose outperforms existing SOTAs by 20% on Human3.6M dataset. 2) as a single-hypothesis pose estimator, GFPose achieves comparable results to deterministic SOTAs, even with a vanilla backbone. 3) GFPose is able to produce diverse and realistic samples in pose denoising, completion and generation tasks. Project page https://sites.google.com/view/gfpose/

1. Introduction

Valid 3D human poses follow biomechanical constraints, making learned pose priors useful for distinguishing infeasible poses and recovering poses from incomplete or ambiguous observations. GFPose learns task-conditional scores and uses hierarchical condition masking to support multiple pose tasks in one framework.

1. Introduction

  • Biomechanical constraints place natural human postures on a low-dimensional manifold, and pose priors help distinguish infeasible poses and recover 3D poses from ambiguous or partial inputs.The passage identifies monocular images with depth ambiguities and occlusions, noisy IMU signals, and partial sensor inputs as examples.Source: §1 · p. 1
  • GFPose represents pose priors as gradient fields for multi-hypothesis estimation, pose correction, missing-joint completion, and generation from noise.The time-dependent score network is trained with denoising score matching and hierarchical masks at human, body-part, and joint levels.Source: Figure 1 · p. 1, §1 · p. 2
  • Unlike methods that learn an unconditional pose distribution or a prior for one given task, GFPose estimates the score of a task-conditional pose distribution.Its condition-masking strategy varies task specifications during training to support different pose-related tasks in one framework.Source: §1 · pp. 1–2
  • GFPose outperforms state-of-the-art methods across multiple pose tasks, including multi- and single-hypothesis estimation, and produces diverse, realistic samples for generation.The evaluated tasks also include pose denoising and completion; the introduction reports stronger robustness to severe occlusions.Source: §1 · p. 2

2. Related Work

Prior work models pose plausibility with task-independent or task-aware priors, while monocular 3D estimation must address ambiguity and score-based generative models offer a framework for learning pose priors. GFPose brings these threads together with a score-based approach for multiple pose tasks.

2.1. Human Pose Priors

  • Task-independent priors model plausible-pose distributions but typically need extra optimization for task constraints, whereas task-aware priors are often limited to a given task.The paper places joint-angle limits, GMMs, VAEs, GANs, and neural implicit functions in the first group, and conditional models in the second.Source: §2.1 · p. 2

2.2. 3D Human Pose Estimation

  • Monocular 2D-to-3D estimation is ill-posed, motivating multiple plausible hypotheses; GFPose uses score gradients to guide poses toward validity and task fidelity.The cited pipeline starts from noise and uses a reverse-time SDE with a Predictor-Corrector sampler.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3. Score-Based Generative Model

  • Score-based generative models estimate log-likelihood gradients, and GFPose explores this framework for 3D human pose priors.The paper describes its approach as the first exploration of score-based generative models for learning 3D human pose priors.Source: §2.3 · p. 3

3. Revisiting Denoising Score Matching

Score-based generative models learn a score function approximating the gradient of the log data distribution and use Langevin Dynamics to generate samples. Denoising Score Matching makes score training tractable by learning to denoise perturbed data.

3. Revisiting Denoising Score Matching

  • A score-based generative model learns a network to approximate the gradient of the log data distribution.Source: §3 · p. 3
  • Langevin Dynamics generates a new sample by iteratively updating an initial point using a step size and Gaussian noise.Source: §3 · p. 3
  • As the step size approaches zero and the number of iterations grows, the sample approaches an exact data-distribution sample under regularity conditions.Source: §3 · p. 3
  • DSM replaces the intractable score-matching objective with a tractable denoising objective using a prespecified noise distribution.For a Gaussian kernel, the perturbation score is tractable, and the optimal score network recovers the data-distribution score almost everywhere.Source: §3 · p. 3

4. Method

GFPose models task-conditional pose priors by estimating a score, then uses diffusion and reverse-time sampling to produce plausible poses. Hierarchical condition masking supports different task conditions and occlusions.

4.1. Problem Statement

  • GFPose estimates the conditional pose-distribution score, using 2D poses for monocular estimation, 3D poses for completion, and no condition for generation or denoising.The formulation also allows other observation types and 3D pose representations.Source: §4.1 · p. 4

4.2. Learning Pose Prior with Gradient Fields

  • The forward diffusion process progressively perturbs data poses toward a Gaussian prior, while reverse sampling transforms prior samples into plausible poses.Source: §4.2 · p. 4
  • A time-dependent score network estimates the conditional score across diffusion times; training uses a tractable objective with a weighting term and perturbation kernel.The reverse-time SDE uses negative time steps, and a Predictor-Corrector sampler improves sample quality.Source: §4.2 · p. 4

4.3. Masked Condition for Versatility

  • Hierarchical masking randomly removes conditions at the whole-human, body-part, or joint level to support unconditional priors and recovery from occlusions.The body-part level treats humans as two legs, two arms, and a torso.Source: §4.3 · p. 4

5. Experiments

Experiments evaluate GFPose across pose estimation, completion, denoising, and generation, alongside ablations of model design. Results report performance across these settings, including cross-dataset testing and synthetic-data augmentation.

5.1. Datasets and Evaluation Metrics; 5.2. Implementation Details

  • H3.6M uses standard subject splits and root-aligned or rigid-aligned MPJPE; 3DHP tests transfer with PCK, while experiments use a vanilla fully connected score backbone.Main results use independently trained task models; a mixed-condition model is also possible but performs slightly worse on each task.Source: §5.1 · p. 5, §5.2 · p. 5

5.3.1 Multi-Hypothesis

5.3.2 Single-Hypothesis

  • With one hypothesis, GFPose beats probabilistic SOTAs and matches deterministic SOTA [68] despite using a plain fully connected network.The table reports H3.6M MPJPE under Protocol #1.Source: §5.3.2 · pp. 5–6, Table 3 · p. 6

5.4. Pose Completion (Incomplete 2D/3D →3D)

  • On partial 2D inputs, GFPose outperforms [28], with error drops of 1.8mm versus 5.8mm as missing joints increase.Two masking strategies address random missing joints and body parts; 200 samples are evaluated under Protocol #1.Source: Table 5 · p. 7, §5.4 · p. 7
  • GFPose completes partial 3D poses; legs are harder to recover than arms.The table reports minMPJPE under Protocol #1, with S denoting the number of samples.Source: Table 6 · p. 7, §5.4 · p. 7

5.5. Denoise MoCap Data (Noisy 3D →Clean 3D)

  • GFPose improves MPJPE by about 24% for Gaussian variance 25 and 49% for variance 400, but adds moderate noise to clean poses.The evaluation uses Gaussian and uniform noise at different intensities on H3.6M.Source: §5.5 · p. 7, Table 7 · p. 8

5.6. Pose Generation (Noise →3D)

  • 177,200 synthetic poses alone yield moderate estimator performance; synthetic augmentation also helps with 2,438 or 1,559,752 real poses.The estimator is evaluated on H3.6M; the two real-data amounts represent a small and full training set.Source: Table 8 · p. 8, §5.6 · p. 8

5.7. Ablation Study

  • More sampling steps or deeper or wider networks did not improve reconstruction accuracy and substantially reduced inference speed.The ablation reports minMPJPE and single-sample inference speed on an NVIDIA 2080Ti GPU.Source: Table 9 · p. 8, §5.7 · p. 8

6. Conclusion and Discussion

GFPose models 3D human pose priors via denoising score matching, incorporating task conditions into gradient fields and using condition masking for varied applications. It is validated across downstream tasks, though repeated reverse sampling may limit real-time use of very large networks.

6. Conclusion and Discussion

  • GFPose incorporates pose priors and task-specific conditions into gradient fields for varied applications.The framework uses denoising score matching and proposes condition masking to enhance its versatility.Source: §6 · p. 8
  • The authors validate GFPose on various downstream tasks and report compelling results.The passage does not specify individual task results.Source: §6 · p. 8
  • Repeated model inferences make inference time proportional to sampling steps, potentially limiting very large networks in real-time scenarios.This limitation concerns the reverse sampling process.Source: §6 · p. 8

A. Details of subVP SDE

The appendix specifies a subVP SDE for perturbing 3D poses and a closed-form loss based on its transition kernel. Training perturbs poses with sampled noise and time values before using the resulting loss to train the score network.

A. Details of subVP SDE

  • The subVP SDE perturbs 3D poses, with a linearly scheduled noise scale set from 0.1 to 20.0 over continuous time t ∈ [0, 1].The drift coefficient is affine in the pose and proportional to the noise scale.Source: A. Details of subVP SDE · p. 12

B. Closed Form of Loss Function

  • Because the drift is affine, the transition kernel used in the loss is Gaussian with a closed-form mean and variance.The loss uses the weighting function λ(t) = σ(t)^2.Source: B. Closed Form of Loss Function · p. 12
  • Training samples t uniformly from [0, 1] and standard-normal noise z, then perturbs the ground-truth pose to obtain x(t).The sampled noise and the score network are identified in the loss description.Source: B. Closed Form of Loss Function · p. 12
  • The score network is trained by computing the loss on the perturbed pose.The passage points to Eq. 10 for the loss computation.Source: B. Closed Form of Loss Function · p. 12

C. Network Architecture

GFPose uses a fully connected score network with residual blocks and specified normalization, activation, and dropout settings. Pose sampling simulates the reverse-time SDE with a Predictor-Corrector sampler.

C. Network Architecture

  • The score network is fully connected with two residual blocks, 1024-dimensional hidden layers, group normalization, SiLU activation, and 0.25 dropout.The number of normalization groups is 32.Source: C. Network Architecture · p. 12

D. Pose Sampling

  • Pose sampling uses Euler–Maruyama as the Predictor-Corrector sampler’s predictor and Identity as its corrector, with 1000 steps from T = 1.0 to ε = 1e-3.The sampler simulates the reverse-time SDE.Source: D. Pose Sampling · p. 12

E. Detailed Settings for Different Tasks

Task-specific conditions, masking strategies, and sampling procedures distinguish GFPose’s implementations across pose estimation, completion, denoising, and generation. The appendix also identifies the scope and metric of a condition-masking comparison for monocular pose estimation.

Monocular 3D Human Pose Estimation

Pose Completion (2D →3D)

  • For 2D-to-3D completion, training uses HPJ-001 for missing joints and HPJ-020 for missing parts, with the combined sampling approach selected as best.The passage describes the combined approach as combining standard conditional inference and imputation.Source: E. Detailed Settings for Different Tasks · pp. 12–13

Pose Completion (3D →3D)

Denoising Mocap Data

  • For mocap denoising, GFPose uses an always-active human-level mask and starts reverse sampling from a noisy pose at T = 0.05 or 0.1.The condition is zero, and the start time is smaller than the default 1.0.Source: E. Detailed Settings for Different Tasks · p. 13

Pose Generation

  • For generation, GFPose conditions on Ø and adjusts standard-normal noise to generate 3D poses.Training uses masking strategy HPJ-T00; generated poses are described as realistic and diverse.Source: E. Detailed Settings for Different Tasks · p. 13

F. Train A Unified Model for Various Tasks

U3D unifies pose estimation, completion, denoising, and generation with only a small overall performance loss. Its task-specific results vary: it benefits estimation, remains competitive for completion, and splits denoising and generation results against specialized models.

F. Train A Unified Model for Various Tasks

  • The unified model activates all three condition-mask levels; U2D conditions on detected 2D poses, whereas U3D adds 3D poses with 0.1 probability.U2D uses only detected 2D poses; U3D conditions on detected 2D and 3D poses with probabilities 0.9 and 0.1, respectively.Source: F. Train A Unified Model for Various… · p. 13
  • U3D benefits pose estimation, while U2D's mixed masking slightly reduces reconstruction accuracy and human- or part-level masks boost it by about 0.5 mm.The section reports U3D benefits but gives no numeric magnitude for that benefit.Source: F. Train A Unified Model for Various… · p. 14
  • Unified models trail specifically trained models slightly for occluded body-part recovery, but are competitive for random-joint recovery and significantly outperform SOTA; U3D also performs well on partial 3D observations.U2D cannot directly handle partial 3D completion without the imputation technique introduced later.Source: F. Train A Unified Model for Various… · p. 14, Table 11 · p. 14, Table 12 · p. 14
  • The unified model performs slightly better at low noise, while specialized HPJ-T00 performs slightly better at high noise.The comparison concerns denoising on H3.6M; Table 13 reports MPJPE under Protocol #2.Source: F. Train A Unified Model for Various… · p. 14, Table 13 · p. 14
  • HPJ-T00 generates the highest-quality poses, while unified models also achieve competitive results.Table 14 evaluates generated poses using deterministic estimators on the H3.6M test set.Source: F. Train A Unified Model for Various… · p. 14, Table 14 · p. 14
  • A unified U3D model trades very little performance loss for versatility across tasks.The section summary characterizes the overall trade-off against specifically trained models.Source: F. Train A Unified Model for Various… · p. 14

G. Different Sampling Approaches

The section compares conditional inference, imputation, and their combination for pose estimation and completion. Conditional inference generally performs better than imputation, while combining them improves pose completion results.

G. Different Sampling Approaches

  • The study treats pose estimation and completion as imputation tasks by imputing 2D-pose depth or missing body joints.Imputation repeatedly replaces part of the sampled data with observed data during sampling.Source: G. Different Sampling Approaches · p. 14
  • Conditional inference performs best on the 3D human pose estimation task, while imputation works better with unconditional models (HPJ-T00); pose-completion comparisons include partial 2D observations (2 legs) and partial 3D observations (rigth leg).Source: G. Different Sampling Approaches · p. 15, Table 16 · p. 15, Table 17 · p. 15
  • On the 3D human pose estimation task, conditional inference performs best, while imputation works better with unconditional models (HPJ-T00).HPJ-T00 means the human-level mask is always activated and an unconditional score model is learned.Source: G. Different Sampling Approaches · p. 15, Table 15 · p. 15

H. More Qualitative Results

The qualitative examples illustrate U3D outputs for multi-hypothesis estimation, pose completion, denoising, and generation. They show plausible alternatives and completions, correction of noisy poses, and diverse generated poses.

H. More Qualitative Results

  • Five sampled hypotheses illustrate multi-hypothesis pose estimation from four viewpoints.The viewpoints rotate the plausible poses 90 degrees clockwise around the z-axis from left to right.Source: H. More Qualitative Results · p. 15, Figure 5 · p. 16
  • Completion examples show five plausible poses for partial 2D inputs and one plausible completion per partial 3D input.The examples include invisible left-side or lower-body regions in 2D, and missing limbs or upper bodies in 3D.Source: H. More Qualitative Results · p. 15, Figure 6 · p. 17
  • After Gaussian noise ∼N(0, 100) is added to ground-truth poses, GFPose corrects unreasonable poses and also generates diverse, realistic poses.The figure distinguishes noisy and denoised poses by color.Source: H. More Qualitative Results · p. 15, Figure 7 · p. 18

Related papers

Loading 2212.08641v1…