PaperlayerView exact revision on arXiv

Source-linked AI summary

Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields

Garvita Tiwari, Dimitrije Antic, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, Gerard Pons-Moll

arXiv:2207.13807v1cs.CV

TL;DR

Noisy or partial human-pose observations can produce unrealistic poses, while VAE-based priors may fail to preserve pose-space distances. Pose-NDF learns a differentiable distance field for the plausible-pose manifold and projects poses onto it. The authors report improved results across several downstream tasks and more diverse generated poses than VAE-based methods.

  • Problem

    Noisy or occluded capture can produce unrealistic poses, and VAE-based priors do not preserve pose-space distances and can yield implausible samples in outer regions.

    Source: §1 · pp. 1–2

  • Method

    Pose-NDF models plausible poses as the zero level set of a differentiable neural unsigned distance field over quaternion-based SO(3)^K pose space.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 5

  • Results

    Pose-NDF improves state-of-the-art results in pose estimation from images and motion denoising and generates more diverse samples than VAE-based methods.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The learned distance and gradient can project arbitrary poses onto the plausible-pose manifold, supporting its use as a prior in downstream tasks.

    Source: §1 · p. 2, §5 · p. 14

    The method assumes realistic and plausible human poses lie on a manifold embedded in SO(3)^K.

    Source: §3 · p. 4

Abstract

from arXiv · show

We present Pose-NDF, a continuous model for plausible human poses based on neural distance fields (NDFs). Pose or motion priors are important for generating realistic new poses and for reconstructing accurate poses from noisy or partial observations. Pose-NDF learns a manifold of plausible poses as the zero level set of a neural implicit function, extending the idea of modeling implicit surfaces in 3D to the high-dimensional domain SO(3)^K, where a human pose is defined by a single data point, represented by K quaternions. The resulting high-dimensional implicit function can be differentiated with respect to the input poses and thus can be used to project arbitrary poses onto the manifold by using gradient descent on the set of 3-dimensional hyperspheres. In contrast to previous VAE-based human pose priors, which transform the pose space into a Gaussian distribution, we model the actual pose manifold, preserving the distances between poses. We demonstrate that PoseNDF outperforms existing state-of-the-art methods as a prior in various downstream tasks, ranging from denoising real-world human mocap data, pose recovery from occluded data to 3D pose reconstruction from images. Furthermore, we show that it can be used to generate more diverse poses by random sampling and projection than VAE-based methods.

1 Introduction

Pose-NDF addresses unrealistic poses from noisy or occluded capture and limitations of VAE priors by modeling the plausible-pose manifold directly. Its differentiable distance field supports projection onto that manifold and is reported to improve several downstream tasks while generating more diverse samples.

1 Introduction

  • Noisy or occluded motion-capture and reconstruction systems can produce skating, self-intersections, jitters, and unrealistic poses, often requiring extensive cleaning.The motivating applications include 3D scene understanding, motion generation, and AR/VR.Source: §1 · p. 1
  • VAE-based priors can favor poses near a Gaussian mean, fail to preserve pose-space distances, and expose data-free outer regions that yield implausible samples.The paper contrasts these properties with modeling the pose manifold directly.Source: §1 · p. 2
  • Pose-NDF represents plausible poses as a manifold in high-dimensional pose space, preserving pose distances and projecting arbitrary poses toward the closest manifold point by gradient descent.The method extends neural distance fields to SO(3)^K, with plausible poses at zero distance and implausible poses outside the manifold.Source: §1 · p. 2
  • Pose-NDF is reported to improve human-body fitting from images and motion denoising, run as fast or faster than current methods, and generate more diverse samples than Gaussian-assumption methods.The introduction also reports improvements over VPoser and HuMoR on motion denoising.Source: §1 · p. 3

2 Related Work

Prior human-pose research spans joint constraints, statistical and adversarial models, VAEs, and implicit functions. Pose-NDF differs by directly modeling the plausible-pose manifold in high-dimensional space.

2 Related Work

  • Earlier priors modeled joint limits, fitted Gaussian mixtures or PCA, and used adversarial training, while some task-specific models require images or previous-frame poses and cannot serve as general priors.The cited examples include HMR, which models p(θ|I), and HP-GAN, which models p(θt|θt−1).Source: §2 · p. 3
  • VAE pose and motion priors support pose generation and reconstruction, while Pose-NDF instead models the plausible-pose manifold directly and preserves pose-space distances.The related work describes conditional motion VAEs and action-conditioned VAE-Transformers alongside pose VAEs.Source: §2 · p. 4
  • Earlier implicit pose functions covered only one or two joints and had representation ambiguities or lacked differentiability, whereas Pose-NDF uses all joints and quaternion components.The paper places its approach among neural fields for shapes and other domains, extending them to non-Euclidean SO(3)^K pose space.Source: §2 · p. 4

3 Method

Pose-NDF learns an unsigned distance field over the SO(3)^K pose space, treating plausible poses as its zero level set. It uses quaternion-based poses, a hierarchy-aware network, and gradient-based projection onto the learned manifold.

3 Method

  • Plausible poses are modeled as a zero level set in SO(3)^K, with the field value representing unsigned distance to that manifold.For the SMPL body model, the paper uses K = 21 joints.Source: §3 · p. 4

3.1 Quaternions as Representation of SO(3)

  • Unit quaternions represent each joint rotation because they support continuous parameterization, efficient geodesic-distance computation, and gradient descent with reprojection by normalization.The pose is represented by K quaternions, one for each joint relative to its parent; joint weights in the pose-distance metric are higher for earlier joints in the kinematic chain, and sign-flip augmentation handles quaternion double cover.Source: §3.1 · p. 5

3.2 Hierarchical Implicit Neural Function

  • A hierarchical encoder conditions each joint’s pose representation on its parent because joint plausibility depends on ancestor rotations; concatenated features feed a distance-prediction MLP.Together, the encoder and distance MLP form Pose-NDF.Source: §3.2 · pp. 5–6

3.3 Loss functions

  • Training uses pose-distance pairs with a distance-prediction loss and an Eikonal regularizer that encourages unit-norm gradients outside the manifold.Further training-data and architecture details are deferred to supplementary material.Source: §3.3 · p. 6

3.4 Projection Algorithm

  • Projection uses the field’s distance and gradient to move a pose toward the nearest point on the manifold, with reprojection to the 3-sphere by vector normalization.The stated convergence guarantee is to local minima on the sphere; assuming a correctly learned distance function, this is the nearest point on the pose manifold.Source: §3.4 · p. 6

4 Experiments and Results

The experiments evaluate Pose-NDF as a prior for motion denoising, image-based pose estimation, pose generation, and interpolation. Across these applications, the results highlight denoising accuracy, realistic and diverse pose generation, and smoother interpolation than VPoser.

4.1 Experimental Setup

  • The experiments use AMASS training data, assigning zero distance to its poses and generating non-manifold training samples by adding noise.Training proceeds from samples far from the manifold toward samples nearer to it.Source: §4.1 · p. 7

4.2 Denoising Mocap Data

  • Pose-NDF achieves the lowest per-vertex error across HPS, AMASS, and noisy AMASS motion-denoising settings.For already realistic HPS and AMASS poses, it also makes only small changes to the input.Source: Table 1 · p. 8, §4.2 · p. 9
  • Pose-NDF performs better than VPoser and HuMoR for leg and arm/hand occlusions, while HuMoR leads for shoulder occlusions.Pose-NDF's results depend on the initialization of the occluded joint.Source: Table 2 · p. 10, §4.2 · p. 10

4.3 3D pose Estimation from Images

  • For image-based pose and shape estimation, Pose-NDF slightly improves on VPoser and GAN-S, and its prior improves ExPose through optimization-based refinement.ExPose outperforms the optimization-based results before refinement; with no prior, refinement produces unrealistic poses.Source: §4.3 · pp. 11–12, Table 3 · p. 11

4.4 Pose Generation

  • Pose-NDF generates plausible poses with greater diversity than VPoser, with APD values of 32.31 cm versus 23.13 cm, respectively.The reported self-intersecting-face percentages are 0.89% for Pose-NDF, 1.43% for GAN-S, and 2.10% for VPoser.Source: §4.4 · p. 12, Figure 5 · p. 13

4.5 Pose Interpolation

  • Pose-NDF and GAN-S produce smooth interpolations, while VPoser interpolation exhibits large jumps.Mean per-vertex distances are 2.72 ± 2.16 for Pose-NDF, 2.71 ± 2.45 for GAN-S, and 2.53 ± 4.62 for VPoser.Source: §4.5 · p. 13, Figure 6 · p. 14

4.6 Pose-NDF vs. Gaussian Assumption models

  • Unlike VAE-based priors that assume a Gaussian latent space, Pose-NDF models the pose manifold directly and preserves distances between poses.The reported cumulative errors for Pose-NDF remain 8.18, 8.20, and 8.21 cm across σ, 2σ, and 3σ samples; VPoser and HuMoR errors increase farther from the mean.Source: §4.6 · pp. 13–14

5 Conclusion

Pose-NDF represents plausible human poses as the zero level set of a scalar neural distance field in SO(3)^K. Evaluations cover pose sampling, image-based pose estimation, and motion denoising, with greater pose diversity than prior VAE-based works and improved state-of-the-art results in reconstruction and motion estimation.

5 Conclusion

  • Pose-NDF represents the plausible-pose manifold as the zero level set of a scalar neural distance field in SO(3)^K.The field maps quaternion-based poses to unsigned distances from the manifold and can project arbitrary poses onto it.Source: §5 · p. 14
  • The method is evaluated on diverse pose sampling, image-based pose estimation, and motion denoising.Source: §5 · p. 14
  • Pose-NDF generates more diverse poses than prior VAE-based works and improves state-of-the-art results in image reconstruction and motion estimation.Source: §5 · p. 14

Related papers

Loading 2207.13807v1…