PaperlayerView exact revision on arXiv

Source-linked AI summary

imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose

Thiemo Alldieck, Hongyi Xu, Cristian Sminchisescu

arXiv:2108.10842v1cs.CVcs.LG

TL;DR

The paper addresses how to build an implicit human model without losing the detail and articulation that make human modeling challenging. imGHUM represents pose and shape as a template-free signed distance function with attached semantics, and reports better shape-completion metrics than task-trained IF-Net.

  • Problem

    Implicit representations avoid fixed mesh topology and resolution, but prior implicit networks tend to produce overly smooth shapes and can fail on articulated humans.

    Source: §1 · p. 1

  • Method

    imGHUM learns a template-free signed distance function for GHUM-conditioned human pose and shape, with a multi-part architecture and attached spatial semantics.

    Source: §1 · pp. 1–2, §1.1 · p. 2

  • Results

    0.103 × 10−3 vs. 0.315×10−3 Chamfer distance (↓), and 0.962 vs. 0.936 NC (↑), favor imGHUM over task-trained IF-Net on completion.

    Source: §3.3 · p. 8

  • Takeaways & Limitations

    The SDF neighborhood supports collision tests, and the template-free model is presented as a path toward diverse, inclusive human models.

    Source: §4 · p. 8

    Very extreme configurations absent from training can expose limitations, including self-intersections or joint bending beyond anatomical range.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

We present imGHUM, the first holistic generative model of 3D human shape and articulated pose, represented as a signed distance function. In contrast to prior work, we model the full human body implicitly as a function zero-level-set and without the use of an explicit template mesh. We propose a novel network architecture and a learning paradigm, which make it possible to learn a detailed implicit generative model of human pose, shape, and semantics, on par with state-of-the-art mesh-based models. Our model features desired detail for human models, such as articulated pose including hand motion and facial expressions, a broad spectrum of shape variations, and can be queried at arbitrary resolutions and spatial locations. Additionally, our model has attached spatial semantics making it straightforward to establish correspondences between different shape instances, thus enabling applications that are difficult to tackle using classical implicit representations. In extensive experiments, we demonstrate the model accuracy and its applicability to current research problems.

1. Introduction

The paper addresses whether data-driven human body models can be represented implicitly without losing the pose, shape, and detail of explicit models. imGHUM combines a template-free signed-distance representation with articulated, detailed body modeling and spatial semantics.

1. Introduction

  • Implicit zero-level-set representations avoid fixed mesh topology and resolution, while imGHUM combines the favorable properties compared in Table 1.The table contrasts imGHUM with mesh-based GHUM, interpolation-only IGR, and occupancy-only NASA.Source: §1 · p. 1, Table 1 · p. 2
  • imGHUM uses a novel architecture and learning paradigm to model human pose, shape, and semantics as a detailed implicit signed distance function.Its target detail includes hands and faces.Source: §1 · pp. 1–2

1.1. Related Work

  • Earlier implicit networks tend to produce overly smooth shapes and can fail on articulated humans [8], motivating imGHUM’s focus on hands and faces.The related-work discussion also describes initial implicit methods as struggling with complex shapes and detail.Source: §1 · p. 1, §1.1 · p. 2
  • GHUM supplies imGHUM’s latent pose and shape representation and generates supervised training data from posed meshes.GHUM represents shape with VAEs and pose with normalizing flows.Source: §1.1 · p. 2
  • Concurrent LEAP models human shape and pose implicitly but lacks hand poses, expressions, and semantics; imGHUM targets these capabilities.imGHUM also aims to model a broad range of body shapes and poses.Source: §1.1 · p. 3

2. Methodology

imGHUM represents a posed human body as the zero set of a signed distance function conditioned on GHUM’s latent pose and shape code. Its training and multi-part design target geometric detail, while a jointly predicted semantic function provides correspondences to a canonical representation.

2. Methodology

  • The model shares GHUM’s shape, expression, and pose latents, and learns a signed distance field whose zero set preserves detail in the input meshes.Training targets may come from raw scans, registrations, or GHUM latent-space samples.Source: §2 · p. 3, §2.1 · p. 3

2.1. Models and Training

  • A single-part MLP learns the SDF directly, using surface, off-surface, and Eikonal losses rather than precomputed ground-truth distances.Surface samples provide normals; off-surface samples provide inside/outside labels, which are easier and faster to obtain than ground-truth SDF values.Source: §2.1 · pp. 3–4
  • Fourier coordinate features address spectral bias, while the training set includes surface and off-surface samples with labels for the SDF losses.Samples are unposed and normalized into a shared bounding box before encoding.Source: §2.1 · pp. 4–5
  • Four local networks model the head, hands, and remaining body, and an MLP combines their outputs into a consistent full-body SDF.The multi-part design reduces local spectral variation to help capture facial expressions and articulated hand poses.Source: §2.1 · p. 4
  • The jointly trained semantic function returns canonical-mesh correspondences near the surface, supporting texture, segmentation-label, and landmark transfer.The semantic mapping is smooth in the spatial domain, unlike encodings with distortion and boundary discontinuities.Source: §2.1 · pp. 4–5
  • Training combines 75K sampled GHUM poses with 35K registered human scans, and fine-tunes on registrations to capture detail that GHUM does not represent well.The scan collection includes CAESAR data and full-body, head, and hand scans.Source: §2.1 · p. 5

3. Experiments

The experiments assess imGHUM’s representational power, compare it with baselines, and test it on reconstruction, completion, and inclusive modeling tasks. Results include quantitative comparisons and qualitative examples, with limitations for extreme poses outside the training distribution.

3. Experiments

  • Evaluation uses Chamfer-L2 for surface accuracy and completeness, Normal Consistency for normals, and IoU for volume; Marching Cubes is primarily needed for final mesh visualization.The authors caution that metrics do not always correlate with perceived reconstruction quality.Source: §3 · pp. 5–6

3.1. Representational Power

  • imGHUM reconstructs motion with consistent body shape across poses and supports realistic generation and smooth shape and expression latent-space interpolation.The experiments report generalization to novel body shapes, expressions, and poses, with interpretable and decoupled latent representations.Source: §3.1 · p. 6, Figure 3 · p. 6
  • For extreme pose configurations not covered by training, outputs can include distorted or incomplete geometry and inaccurate semantics.The passage cites anthropometrically invalid poses, including self-intersections and joints bent beyond their anatomical range.Source: §3.1 · p. 6

3.2. Baseline Experiments

  • Only imGHUM consistently produces high-quality results for hands and the face, while Fourier input mapping improves results across tested variants.The multi-part architecture improves head and hand reconstruction accuracy; single-part models show artifacts, especially in the hands.Source: Table 4 · p. 6, §3.2 · pp. 6–7

3.3. Applications

  • IoU (↑) 0.962 (ours) vs. 0.839 (theirs), Ch. (↓) 0.068×10−3 (ours) vs. 3.53 × 10−3 (theirs), and NC (↑) 0.985 (ours) vs. 0.903 (theirs) against NASA.The comparison uses a single-subject multi-pose dataset; imGHUM also uses a smaller network in this experiment.Source: Figure 5 · p. 7, §3.2 · p. 7
  • Chamfer distance (↓) 0.156 × 10−3 (ours) vs. 0.844 × 10−3 (IF-Net), and NC (↑) 0.954 (ours) vs. 0.914 (IF-Net) on triangle-set reconstruction.The figure also shows imGHUM fitting scans and IF-Net inference.Source: Figure 6 · p. 8
  • Chamfer distance (↓) 0.103 × 10−3 (ours) vs. 0.315×10−3 (theirs), and NC (↑) 0.962 (ours) vs. 0.936 (theirs) for partial point cloud completion.The comparison is qualified because IF-Net reconstructs a surface, whereas imGHUM recovers a parametrized body model.Source: §3.3 · p. 8, Figure 6 · p. 8
  • The inclusive-modeling examples show imGHUM with residual SDF layers modeling non-standard body topologies and additional geometric detail beyond a fitted template mesh.The layered representation can be reposed by changing the underlying imGHUM parameterization.Source: Figure 7 · p. 8, §3.3 · p. 8

4. Discussion and Conclusion

imGHUM is a controllable 3D human body model represented as an implicit signed distance function, with representation power comparable to state-of-the-art mesh-based models. It captures variation in pose, shape, facial expression, and semantics, while also representing the body’s surrounding space.

4. Discussion and Conclusion

  • imGHUM represents controllable human pose and shape as an implicit signed distance function, without limiting its representation to the body surface.Its neighborhood representation can support collision tests with other objects and efficient distance losses.Source: §4 · p. 8
  • The model represents significant variation in body pose, shape, and facial expressions, with precise underlying semantics.Source: §4 · p. 8
  • The authors present imGHUM as a building block for diverse human models, including people who may not match a standard template.They identify clothing modeling, virtual apparel try-on, and free-viewpoint photorealistic visualization as possible applications.Source: §4 · p. 8

Supplementary Material

The supplementary material details imGHUM’s implementation and architectures, adds results, ablations, and experiment comparisons, and demonstrates differentiable rendering.

Supplementary Material

A. Implementation Details

The supplement specifies training settings, baseline architectures, mesh-extraction timing, and additional numerical comparisons across training-data distributions.

A. Implementation Details

  • Loss weights are λ_o1=1, λ_o2=1, λ_e=0.1, λ_l=0.1, and λ_s=0.5; empirically increasing λ_o1 to 50 over 100K iterations gives perceptually better results.Training uses Adam with a learning rate of 0.2×10^-3, exponentially decaying by 0.9 over 100K iterations; batches contain 32 instances with on-surface, near-surface, and uniform samples.Source: A. Implementation Details · p. 11
  • The single-part network has 2.01M parameters; the deeper single-part network has 2.53M, and the PointNet++ autoencoder has 3.91M.The autoencoder uses a PointNet++ encoder and the single-part decoder.Source: A. Implementation Details · p. 11
  • 1.08 s is the average time to reconstruct a mesh at maximum grid resolution 256^3 on an NVIDIA Tesla V100.The resulting meshes have approximately 100K vertices and 200K facets; imGHUM can also be queried at arbitrary resolutions without generating an explicit mesh.Source: A. Implementation Details · p. 11
  • Tables 5 and 6 compare imGHUM trained with different data distributions on the registration and GHUM-samples test sets, respectively.Source: Table 5 · p. 11, Table 6 · p. 11

B. Results

Supplementary results show diverse imGHUM instances and additional application experiments. The model supports queries at arbitrary resolutions and spatial locations and represents space around the person, not only the surface.

B. Results

  • Samples from imGHUM’s latent space show variety in pose, shape, hand pose, and facial expression, with detail and generative power described as on par with state-of-the-art mesh-based models.Source: B. Results · p. 11
  • imGHUM can be queried at arbitrary resolutions and spatial locations, and models space around the person as well as the surface.Source: B. Results · p. 11
  • The supplement presents additional results for application experiments in Figures 11, 12, 14, and 15.Source: B. Results · p. 11

C. Ablations

The ablations examine the training data and joint-rotation representation. Scan data add detail, while training only on scans does not generalize sufficiently to poses outside the limited training set.

Dataset

  • Training on scans alone represents their distribution but does not generalize sufficiently to poses absent from the limited training set.The main paper reports benefits from combining GHUM samples with scan registrations; fine-tuning with scan data increases facial detail and soft-tissue deformation.Source: C. Ablations · p. 11
  • The reconstruction examples show imGHUM meshes at resolutions 512^3, 256^3, 128^3, and 64^3 alongside the ground-truth shape.The figure presents these resolutions from left to right.Source: Figure 9 · p. 12

Rotation Representations

  • Different joint-rotation representations yield only minor qualitative and quantitative differences, so the work uses compact Euler angles.The alternatives include Euler angles, Fourier mapping, and the 6D representation [57].Source: C. Ablations · pp. 11–12

D. Applications

The applications section introduces three uses: triangle-set surface reconstruction, dressed and inclusive human modeling, and silhouette-based pose estimation with differentiable rendering.

D.1. Triangle Set Surface Reconstruction

  • The triangle-set surface reconstruction experiment uses a weighted combination of losses.The following subsection explains the losses used in this experiment.Source: D. Applications · p. 12

D.2. Dressed and Inclusive Human Modeling

  • The dressed and inclusive modeling experiment uses a residual model.The section outlines this application after the triangle-set experiment.Source: D. Applications · p. 12

D.3. Differentiable Rendering

  • The section also introduces pose estimation from silhouettes using differentiable rendering.This is presented as an additional application.Source: D. Applications · p. 12

D.1. Triangle Set Surface Reconstruction

The triangle-set experiment fits imGHUM using surface, sign, joint-landmark, and semantic surface-landmark losses. Its differential losses use more of the triangle set than ICP and avoid ICP's initialization-sensitive nearest-point correspondences.

D.1. Triangle Set Surface Reconstruction

  • The surface sample loss encourages samples onto the zero-level set with SDF gradients matching surface normals, while sign losses classify points sampled along and opposite the normals.The sign-loss samples use Gaussian distances γ_i ∈ [0, 0.05].Source: §D.1 · p. 12
  • Joint landmarks align transformed joint centers, while semantic surface landmarks match queried correspondences to landmarks on the canonical mesh.The surface semantics are conditioned on α.Source: §D.1 · p. 12
  • imGHUM's differential losses exploit surface normals and sign labels beyond landmark losses, whereas ICP's nearest-point correspondences are non-differentiable and initialization-sensitive.The authors report equivalent performance to mesh-based representations on landmark losses; numerical comparisons are in the main paper's §3.3.Source: §D.1 · p. 12

D.2. Dressed and Inclusive Human Modeling

The applications model dressed and structurally different bodies with a learned SDF residual, and fit imGHUM to image silhouettes through differentiable sphere tracing.

D.2. Dressed and Inclusive Human Modeling

  • A four-layer MLP residual modulates the body's signed distance field using both the body's distance and semantic fields, capturing clothing, hair, apparel, or other departures from the standard template.The residual uses 256-dimensional layers, Swish activations, and a skip connection to the middle layer.Source: §D.2 · pp. 12, 14
  • The examples include textured random full-body and part instances, triangle-set reconstructions showing facial expressions and hand poses, and partial point-cloud completions.The figure captions identify the inputs and outputs for the reconstruction and completion examples.Source: Figure 10 · p. 13, Figure 11 · p. 13, Figure 12 · p. 13
  • The method first fits imGHUM to a scan, then trains only the residual on samples from that personalized scan while keeping imGHUM fixed.A separate residual instance is trained for each scan; a combined auto-decoder model is beyond the work's scope.Source: §D.2 · p. 14
  • For limb differences, Chamfer distance was 0.014 × 10−3 (ours, limb differences) versus 1.393 × 10−3 (GHUM ACAP, limb differences), and Normal Consistency was 0.993 (ours, limb differences) versus 0.984 (GHUM ACAP, limb differences).Across ten limb-difference scans and 30 dressed scans, imGHUM+residual also achieved lower Chamfer distance and higher Normal Consistency than GHUM ACAP for dressed scans.Source: §D.2 · p. 14

D.3. Differentiable Rendering

  • Sphere tracing uses the SDF's distance to the surface to set ray steps, enabling depth, normal, and semantic rendering and differentiable silhouette fitting.The approximate renderer takes 15 fixed steps per camera ray.Source: §D.3 · p. 14
  • Silhouette fitting combines a standard silhouette-overlap loss with a sparse 2D joint-landmark loss, using differentiable binarized pixels computed from the final ray points.The pixel value is differentiable with respect to α.Source: §D.3 · p. 14

E. Details on Compared Methods

The section details the architectures and training setups used for method comparisons, alongside visual examples of fine-tuning, dressed and inclusive modeling, and silhouette-based reconstruction.

E. Details on Compared Methods

  • For the 63-part GHUM task, NASA uses eight 64-dimensional layers per part (1.92M parameters), while smaller imGHUM uses half as many layers and half dimensionality (0.64M) yet performs significantly better.NASA's original SMPL-based model used four 40-dimensional layers per part and 0.38M parameters.Source: E. Details on Compared Methods · p. 14
  • IF-Net uses 128^3-resolution point clouds and is trained separately for full and partial point-cloud tasks, whereas the same imGHUM model is used across comparisons.The IF-Net setup follows the original source code and sampling and resizing strategy, with input point clouds scaled to a maximum side length of one unit.Source: E. Details on Compared Methods · p. 14
  • Fine-tuning on the registration dataset adds visible detail to imGHUM reconstructions, including faces, body shapes, and soft-tissue deformations.The figure compares before and after fine-tuning using error heat maps, where red indicates errors of at least 2 cm.Source: Figure 13 · p. 15
  • imGHUM+residual explains scan detail in dressed humans and supports repose and reshaping through the underlying imGHUM parameterization, while GHUM ACAP struggles with layered structures.The color scale represents imGHUM semantics and correspondences between instances.Source: Figure 14 · p. 15
  • imGHUM+residual explains body shapes that do not match the standard template, whereas GHUM ACAP registrations fail to explain those shapes.Missing limbs are deformed but remain present in the reconstructions.Source: Figure 15 · p. 15
  • Differentiable silhouette and landmark losses are used for visual 3D reconstruction, and the silhouette loss supports accurate body-shape reconstruction.Source: Figure 16 · p. 15
Loading 2108.10842v1…