PaperlayerView exact revision on arXiv

Source-linked AI summary

NPMs: Neural Parametric Models for 3D Deformable Shapes

Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, Angela Dai

arXiv:2104.00702v2cs.CVcs.GR

TL;DR

Traditional parametric 3D models can require substantial manual intervention and struggle with details such as clothing. NPMs learn disentangled implicit shape and pose spaces that can be optimized to fit observations, and the paper reports better reconstruction and tracking than parametric and implicit 4D approaches.

  • Problem

    Traditional parametric models require manual intervention and object-specific constraints, and can struggle to represent details such as clothing, while dynamic deforming surfaces remain challenging to represent.

    Source: §1 · p. 1

  • Method

    NPMs learn disentangled shape and pose representations using implicit functions, then jointly optimize the learned spaces to fit new observations.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    NPMs capture higher-quality reconstructions with finer detail and more accurate non-rigid tracking than state-of-the-art parametric 3D models and implicit 4D representations.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    NPMs support shape and pose transfer, smooth interpolation across new shapes and poses, and demonstrated applicability to a hands dataset.

    Source: §5 · p. 8, §4.3 · p. 8

    High-frequency details remain challenging, and the learned spaces do not model deformation physics.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Parametric 3D models have enabled a wide variety of tasks in computer graphics and vision, such as modeling human bodies, faces, and hands. However, the construction of these parametric models is often tedious, as it requires heavy manual tweaking, and they struggle to represent additional complexity and details such as wrinkles or clothing. To this end, we propose Neural Parametric Models (NPMs), a novel, learned alternative to traditional, parametric 3D models, which does not require hand-crafted, object-specific constraints. In particular, we learn to disentangle 4D dynamics into latent-space representations of shape and pose, leveraging the flexibility of recent developments in learned implicit functions. Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit to new observations, similar to the fitting of a traditional parametric model, e.g., SMPL. This enables NPMs to achieve a significantly more accurate and detailed representation of observed deformable sequences. We show that NPMs improve notably over both parametric and non-parametric state of the art in reconstruction and tracking of monocular depth sequences of clothed humans and hands. Latent-space interpolation as well as shape/pose transfer experiments further demonstrate the usefulness of NPMs. Code is publicly available at https://pablopalafox.github.io/npms.

1. Introduction

NPMs learn disentangled shape and pose spaces as an alternative to hand-crafted parametric models, then optimize those spaces to fit new observations. The paper reports improved reconstruction and tracking of monocular depth sequences against parametric and implicit 4D approaches.

1. Introduction

  • Traditional parametric models require substantial manual intervention and object-specific constraints, and can struggle with added complexity such as clothing and hair.These models have been used for human bodies, hands, animals, and faces.Source: §1 · p. 1
  • NPMs learn disentangled shape and pose representations from data using implicit functions, without requiring object-specific assumptions such as a kinematic chain, part count, or skeleton.Training requires the same identity or shape to appear in different poses, including a canonical pose.Source: §1 · pp. 1–2
  • NPMs require no surface registration across identities, enabling training on a wider variety of data.The training data may include different shape identities and topologies, with deformations of each identity.Source: §1 · p. 2
  • At test time, optimizing regularized shape and pose spaces fits observations while retaining data detail and supporting globally consistent shapes and temporally consistent poses.The implicit representations also support arbitrary connectivity and topology and finer-scale detail.Source: §1 · p. 2
  • NPMs capture higher-quality reconstructions with finer detail and more accurate non-rigid tracking than state-of-the-art parametric 3D models and implicit 4D representations.The reported evaluation includes reconstruction and tracking of monocular depth sequences.Source: §1 · p. 2

2. Related Work

Prior approaches to deformable shapes include domain-specific parametric models, implicit representations, and learned deformable models. NPMs combine learned implicit functions with disentangled shape and pose spaces, without requiring a template or registration across identities.

2. Related Work

  • Traditional parametric models often require domain-specific annotations, while vertex-based models such as SMPL and GHUM can struggle with complex surface features.NPMs are learned without expert knowledge or manual intervention and use implicit functions to capture intricate surface detail.Source: §2 · p. 2
  • Continuous implicit-function methods remove the explicit-grid limitation, but cited approaches produce static, uncontrollable surfaces because shape and pose share a latent code.NPMs instead learn separate implicit spaces for shape and pose to support controllable models for fitting dynamic data or generating posed shapes through interpolation.Source: §2 · p. 2
  • Unlike the registered-mesh approach of Zhou et al. [52], NPMs require neither a template nor registration of dataset identities to each other.The paper connects this flexibility to representing complex detail and broader shape variety, including clothed bodies, and to fitting sparse observations at test time.Source: §2 · pp. 2–3
  • OFlow learns continuous motion fields for 4D reconstruction but remains limited to very short sequences; IP-Net [3] combines implicit functions with a traditional parametric model for controllable humans.NPMs instead learn disentangled shape and pose latent spaces and fit by optimizing over them jointly.Source: §2 · p. 3

3. Method

NPMs learn separate implicit shape and pose spaces from posed identities, then optimize their latent codes to fit new depth sequences. The method represents canonical geometry with an SDF and deformations with shape-conditioned flows.

3.1. Neural Parametric Models

  • NPMs use separate implicit shape and pose representations, learned from data without object-specific constraints, and jointly optimize their latent codes to fit new observations.The shape representation encodes an SDF; the pose representation encodes flow from canonical to deformed shapes.Source: §3 · p. 3, §3.1 · p. 3

3.2. Learned Shape Space

  • The shape MLP learns canonical identities by mapping points and identity codes to SDF values, with each identity code optimized directly in an auto-decoder setup.Canonical training shapes are normalized and made watertight; the shape is the SDF zero-level surface.Source: §3.2 · p. 3

3.3. Learned Pose Space

  • The pose MLP predicts canonical-to-posed flow conditioned on both shape and pose codes, learning from each identity’s available deformations without requiring identical poses across identities.The pose space is optimized over pose codes; conditioning on shape codes is required to disentangle pose from shape.Source: §3.3 · p. 4

3.4. Inference-time Optimization

  • At test time, NPMs fit one shape code and per-frame pose codes to a depth sequence, using reconstruction, code-regularization, temporal, and ICP-like losses.The optimized canonical mesh is extracted once and deformed into each frame using the pose MLP.Source: §3.4 · p. 5

3.4.1 Predicting Shape and Pose Initializations

  • Two 3D convolutional encoders predict initial shape and pose codes from partial voxel grids, using the learned training-set codes as targets.The authors report that this initialization provides robust initial estimates for reconstruction and tracking.Source: §3.4.1 · p. 5

4. Experiments

Experiments evaluate monocular-depth fitting on real and synthetic humans, hand registration, shape and pose transfer, and latent-space interpolation. They report improved reconstruction and tracking, demonstrate transfer and interpolation, and identify remaining representation limits.

4.1. Model Fitting to Monocular Depth Sequences

  • Across real CAPE and synthetic DeformingThings4D, NPM fitting improves reconstruction and tracking; on D-FAUST completion, it improves over OFlow despite more partial input.The D-FAUST comparison uses monocular depth sequences for NPM fitting, while OFlow receives denser point-cloud trajectories.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • Hand registration captures global structure and knuckle creases, with IoU of 0.86, Chamfer-ℓ2 of 1.39 × 10−5, and EPE of 5.89 × 10−3.Source: §4.1 · p. 7

4.2. Shape and Pose Transfer

  • Shape transfer applies one identity to a posed shape, while pose transfer reposes a target identity using poses from a source identity.The experiments show these transfers through novel combinations of shape and pose latent codes.Source: §4.2 · p. 7

4.3. Latent-Space Interpolation

  • Interpolation through the learned shape and pose spaces suggests that both latent spaces are continuous.Source: §4.3 · pp. 7–8
  • NPMs can struggle with very flat surfaces and high-frequency details, and their learned spaces do not model deformation physics.The paper gives clothing boundaries as fine-scale details they can capture, but a tie outline as a challenging high-frequency detail.Source: §4.3 · p. 8

5. Conclusion

The conclusion presents NPMs as learned parametric models with disentangled shape and pose representations for dynamic objects. It also highlights their reconstruction, transfer, interpolation, and broader applicability, while the appendix outlines additional technical material.

5. Conclusion

  • NPMs use disentangled learned shape and pose representations to accurately represent 4D sequences of dynamic objects.The conclusion contrasts their learned implicit functions with traditional parametric models.Source: §5 · p. 8
  • NPMs capture local detail and fit observed details in monocular depth sequences, outperforming parametric and learned 4D representations.Source: §5 · p. 8
  • The learned spaces support shape and pose transfer and smooth interpolation across new shapes and poses.Source: §5 · p. 8
  • The paper demonstrates NPMs on a hands dataset and identifies potential applicability to other spatio-temporal modeling domains.Source: §5 · p. 8

Appendix

  • The appendix covers data preparation, network architectures, training and inference-time optimization, and additional qualitative evaluations.Additional qualitative results are shown in the supplemental video.Source: Appendix · p. 9

A. Data Preparation

Canonical shapes are made watertight through a multiview reconstruction pipeline, which retains original mesh detail but is not guaranteed to succeed on meshes with large holes.

A. Data Preparation

  • The pipeline renders depth maps from a virtual multiview setup, reconstructs partial surface meshes, then applies Poisson reconstruction to the merged meshes.Source: §A.1 · p. 9
  • The approach is not guaranteed to produce watertight meshes, especially when the original meshes have large holes.Source: §A.1 · p. 9
  • The authors found the procedure worked well on their canonical shapes while retaining details from the original non-watertight meshes.Source: §A.1 · p. 9

A.2. Train Samples

Training samples are drawn from canonical shapes and corresponding posed shapes to supervise shape geometry and near-surface deformation. The section gives sampling distances and correspondence counts for these data.

A.2. Train Samples

  • Each canonical shape contributes SDF samples from near its surface and from uniformly sampled points within its unit bounding box.The stated experimental setting uses N_ns_s = 100k near-surface samples and N_s = 400k total samples.Source: §A.2 · p. 9
  • Near-surface shape samples lie within 0.05 of the shape surface, while Figure 8 visualizes inside and outside samples by SDF sign.Red points are outside with positive SDF values; blue points are inside with negative SDF values.Source: §A.2 · p. 9, Figure 8 · p. 9
  • Pose training samples pair canonical surface points with corresponding posed-shape points, defining a near-surface deformation field.The corresponding points use the same barycentric weights and normal displacement.Source: §A.2 · p. 9
  • The experiments precompute 200k correspondences per posed shape, sampling half with σ = 0.01 and half with σ = 0.002 in normalized coordinates.Source: §A.2 · p. 9

B. Network Architecture

The shape and pose auto-decoders are feed-forward MLPs, while separate 3D convolutional encoders predict latent-code initializations from partial voxel grids. The appendix details their architectures and training targets.

B.1. Shape and Pose Auto-decoders

  • Both auto-decoder MLPs have eight fully connected layers, weight normalization, ReLU intermediate activations, and a fourth-layer skip connection.The shape MLP outputs an SDF through tanh; the pose MLP outputs a 3D flow vector.Source: §B.1 · p. 9, Figure 9 · p. 10, Figure 10 · p. 10
  • Human and hand models use different feature and latent dimensions, and both MLPs use eight-band positional encoding for query points.Human shape uses Fs = 512 and Ds = 256; hand shape uses Fs = 64 and Ds = 16.Source: §B.1 · pp. 9–10

B.2. Shape and Pose Encoders for Initialization

  • Two 3D convolutional encoders map partial voxel grids to initial shape and pose codes, trained against the corresponding learned codes.The shape encoder targets an identity's shape code; the pose encoder targets the posed shape's pose code.Source: §B.2 · p. 10, Figure 11 · p. 10

C. Training Details

Training samples canonical shapes and posed-shape correspondences, and trains separate shape and pose MLPs to convergence.

C. Training Details

  • The shape MLP trains for 4,000 epochs and the pose MLP for 150, with approximately 8 days of total human-model training on one GeForce RTX 3090.The authors note that parallelizing across multiple GPUs could accelerate training.Source: C. Training Details · p. 11
  • For each identity, shape training randomly samples 50,000 points, with 70% drawn from near-surface points and 30% from another available point set.The supplied passage does not identify the latter set beyond its notation.Source: C. Training Details · p. 11
  • For each posed shape in a mini-batch, pose training randomly subsamples 50,000 available flow correspondences.Source: C. Training Details · p. 11

D. Inference-time Optimization Details

At inference, the method optimizes a shared shape code and per-frame pose codes against the input sequence. It initializes those codes with learned encoders and uses an ICP-like term early in optimization.

D. Inference-time Optimization Details

E. Ablation Studies

The ablation study examines network choices and test-time consistency and alignment components on CAPE.

E. Ablation Studies

F. Robustness to Initialization

The experiments report robustness to initialization and self-occlusion, near-identical shape estimates across consecutive sub-sequences, and examples involving complex topology and clothing. The paper also compares NPMs with Zhou et al. [52] for pose transfer.

F. Robustness to Initialization

  • Even without encoder-based initial code estimates, NPMs achieve competitive performance; their fitting is also robust to rotation-induced self-occlusions.The comparison notes that SMPL fitting becomes ill-posed for monocular depth fitting without OpenPose joint predictions.Source: F. Robustness to Initialization · pp. 11–12, Figure 12 · p. 12

G. Consistency of Shape Fitting

H. Additional Examples

I. Comparison with Zhou et al. [52]

  • Unlike Zhou et al. [52], whose mesh auto-encoder is limited to fixed topology, NPMs are presented as handling complex topology in the pose-transfer comparison.The paper frames its focus as disentangled spaces for fitting new observations, and reports training NPMs on the same human-body dataset as [52].Source: §I · p. 12

Related papers

Loading 2104.00702v2…