PaperlayerView exact revision on arXiv

Source-linked AI summary

LEAP: Learning Articulated Occupancy of People

Marko Mihajlovic, Yan Zhang, Michael J. Black, Siyu Tang

arXiv:2104.06849v1cs.CVcs.AI

TL;DR

Neural implicit representations have limited generalization to articulated human shapes, and prior occupancy models may require per-subject training. LEAP maps query points to canonical space with learned LBS and models identity- and pose-dependent occupancy, outperforming NASA in generalization and fidelity across settings.

  • Problem

    Neural implicit representations have limited generalization to deformable bodies, while NASA’s occupancy representation is trained per subject and does not generalize well across body shapes.

    Source: §1 · pp. 1–2

  • Method

    LEAP learns inverse and forward LBS functions to map posed-space query points into canonical space, where an occupancy network models identity- and pose-dependent body deformations.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    LEAP outperforms NASA in generalization ability and fidelity in all settings; for unseen poses, its IOU improves from 73.69% to 98.39%.

    Source: §7 · p. 8, §6.3 · p. 8

  • Takeaways & Limitations

    LEAP provides a fully differentiable volumetric body representation for efficient occupancy checks and point-based collision resolution with point-cloud objects.

    Source: §7 · p. 8

    When inverse LBS is poorly trained and body parts are close or self-intersecting, query points may map incorrectly and distort occupancy; explicit self-contact modeling remains future work.

    Source: E. Limitations · p. 10

Abstract

from arXiv · show

Substantial progress has been made on modeling rigid 3D objects using deep implicit representations. Yet, extending these methods to learn neural models of human shape is still in its infancy. Human bodies are complex and the key challenge is to learn a representation that generalizes such that it can express body shape deformations for unseen subjects in unseen, highly-articulated, poses. To address this challenge, we introduce LEAP (LEarning Articulated occupancy of People), a novel neural occupancy representation of the human body. Given a set of bone transformations (i.e. joint locations and rotations) and a query point in space, LEAP first maps the query point to a canonical space via learned linear blend skinning (LBS) functions and then efficiently queries the occupancy value via an occupancy network that models accurate identity- and pose-dependent deformations in the canonical space. Experiments show that our canonicalized occupancy estimation with the learned LBS functions greatly improves the generalization capability of the learned occupancy representation across various human shapes and poses, outperforming existing solutions in all settings.

1. Introduction

LEAP addresses limited generalization of neural occupancy models to varied human shapes and poses by mapping posed-space queries into a canonical space and modeling identity- and pose-dependent occupancy there.

1. Introduction

  • Prior neural implicit methods are limited on deformable objects, while NASA requires per-subject training and does not generalize well across body shapes.Source: §1 · pp. 1–2
  • LEAP maps posed-space query points into canonical space with learned inverse LBS, then estimates occupancy using a network that models differentiable body deformations.Source: §1 · p. 2
  • Because conventional LBS weights exist only on body surfaces, LEAP learns forward and inverse LBS for spatial queries and uses cycle distance to model their consistency.Source: §1 · p. 2
  • LEAP encodes bone transformations using kinematic structure and plausible body shapes, with locally aware features that capture detailed pose- and shape-dependent deformations.Source: §1 · p. 2
  • For 3D scene placement, LEAP prevents person-person and person-scene interpenetration and outperforms the recent baseline [72].Source: §1 · p. 2
  • LEAP introduces a neural occupancy representation and canonicalized estimation framework that generalize across body shapes and poses, learning forward and inverse LBS weights throughout 3D space.Source: §1 · p. 2

2. Related work

Prior work offers mesh-based, implicit, and articulated human representations, but LEAP combines differentiable occupancy with generalization to unseen subjects without per-subject training.

Articulated mesh representations

  • Mesh representations support rendering and animation but make differentiable inside/outside tests difficult because triangle indexing is a preprocessing step and is non-differentiable.Source: §2 · p. 2

Learning-based articulated representations

  • Unlike mesh-based or per-subject approaches, LEAP approximates forward and inverse LBS with neural networks and generalizes to unseen subjects, outperforming NASA across settings.Source: §2 · p. 3

Learning-based implicit representations

  • Learning-based implicit representations provide continuous occupancy or distance queries and can use local features, but existing methods primarily address static shapes rather than articulated objects.Source: §2 · p. 3

Application: Placing people in 3D scenes

  • Earlier scene-generation methods require scene meshes and SDFs for collision tests and can produce interpenetrating people; LEAP instead supports point-based collision losses for point-cloud objects.Source: §2 · p. 3

Structure-aware representations

  • LEAP’s structure-aware encoding uses forward kinematics to derive structural features from bone transformations, complementing prior methods that use kinematic structure for prediction.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe SMPL’s canonical template, shape and pose offsets, joint regression, and LBS deformation as background for LEAP’s articulated occupancy model.

SMPL and its canonicalized shape correctives

  • SMPL adds shape- and pose-dependent vertex offsets to a canonical template mesh, with shape and pose blend-shape functions controlling the respective deformations.Source: §3 · pp. 3–4

Regressing body vertices from joints

  • SMPL regresses rest-pose joint locations from body vertices using a learned matrix, and LEAP uses this relation to derive shape features from joint displacements.Source: §3 · p. 4

Linear Blend Skinning (LBS)

  • Linear blend skinning deforms each modified vertex using blend weights and bone transformations, while SMPL defines those weights only for mesh vertices.Source: §3 · p. 4

4. LEAP: Learning occupancy of people

LEAP predicts occupancy by mapping posed-space query points into canonical space, then evaluating them with an occupancy network conditioned on local bone features and a cycle-distance cue. Its encoders represent shape, kinematic structure, and pose, while learned inverse and forward LBS networks support the mapping.

4. LEAP: Learning occupancy of people

  • LEAP encodes bone transformations, estimates inverse-LBS weights to map each query point into canonical space, and predicts occupancy from the canonical point, local code, and cycle-distance feature.The forward LBS network estimates weights for the return projection used to form the cycle-distance feature.Source: §4 · p. 4

4.1. Encoders

  • Three encoders exploit kinematic structure and represent shape- and pose-dependent deformations from the input bone transformations.The global feature is customized into a per-bone code through a learned projection module.Source: §4 · p. 4, §4.1 · p. 5, Figure 2 · p. 5

4.1.1 Shape encoder

  • The shape encoder inverts SMPL from bone transformations to estimate shape coefficients and canonical vertices, then uses PointNet features for canonical and posed bodies.The passage states that the required operations are differentiable and can be efficiently implemented using SMPL model parameters.Source: §4.1.1 · p. 5

4.1.2 Structure encoder

  • The structure encoder models joint dependencies with a hierarchical network whose per-bone MLPs use joint location, bone length, relative rotation, and parent-bone features.The per-bone outputs are concatenated into a structure feature.Source: §4.1.2 · p. 5

4.1.3 Pose encoder

  • The pose encoder projects the skeleton root location into each bone's local frame and concatenates the projections into a pose feature vector.Source: §4.1.3 · p. 5

4.2. Learning linear blend skinning

  • Because conventional LBS weights exist only on body surfaces, LEAP learns inverse weights for every spatial query and adds a forward-LBS cycle-distance cue for ambiguous mappings.The cycle cue can flag a posed-space point that maps inside the canonical body but corresponds to a different body part.Source: §4.2 · pp. 5–6

4.3. Training

  • Training first fits the forward and inverse LBS networks independently, then freezes them while optimizing the occupancy network against ground-truth occupancy values.The LBS networks are trained using pseudo-ground-truth weights assigned from the closest human mesh vertex and its SMPL skinning weights.Source: §4.3 · p. 6

5. Application: Placing people in scenes

LEAP is used to optimize human placement in scenes by resolving collisions against scene geometry and other people represented as point clouds. Its point-based loss avoids reliance on precomputed scene and human signed distance fields.

5. Application: Placing people in scenes

  • The approach addresses limitations of prior scene-placement methods that require high-quality scene signed distance fields and can leave collisions between generated people.The cited prior methods also do not account for previously generated people when generating new bodies.Source: §5 · p. 6
  • LEAP resolves collisions with scene geometry and other people represented as point clouds by optimizing its input parameters with a point-based loss.LEAP’s parameters remain fixed during optimization; the representation is used as a differentiable module with respect to its input.Source: §5 · p. 6
  • For scene meshes, sampling points along the direction opposite mesh vertex normals supplies an oriented volumetric error signal to avoid human-scene interpenetrations.A simpler option is to treat mesh vertices as a point cloud and apply the same point-based loss.Source: §5 · pp. 6–7

6. Experiments

Experiments evaluate feature encoders, multi-person occupancy, generalization to unseen poses and subjects, and collision resolution in scenes. LEAP improves generalization and representation quality, and its point-based optimization reduces collisions with scenes and other people.

6. Experiments

  • Experiments use DFaust and MoVi data, with 200k sampled query points per pose and a 52-bone articulated body model.The models are trained in two stages, and the reported metrics are IOU and scaled Chamfer distance.Source: §6 · p. 7

6.1. The impact of feature encoders

  • The structure encoder has the largest impact on performance, while combining all three encoders gives the best results.The encoder ablation evaluates held-out sequences from 10 DFaust subjects.Source: §6.1 · p. 7

6.2. Multi-person occupancy

  • LEAP has higher representation power than NASA, preserving high-frequency details and smoother connections between adjacent bones; the cycle-distance feature further improves results.Figure 4 highlights small details and cycle-distance improvements in close-ups.Source: §6.2 · p. 7, Figure 4 · p. 7

6.3. Generalization

  • 98.39% IOU versus 73.69% for unseen poses; LEAP also improves results for unseen subjects.The evaluation includes actions not observed during training for the unseen-pose setting and held-out subjects for the unseen-subject setting.Source: §6.3 · pp. 7–8, Table 2 · p. 7

6.4. Placing people in 3D scenes

  • LEAP optimization reduces collisions with scene meshes and other people represented by point clouds, with human-scene scores affected by noisy scene SDFs.The experiment optimizes one person’s global position while holding other pose parameters fixed; it evaluates 151 initially colliding person pairs.Source: §6.4 · p. 8, Table 3 · p. 8

7. Conclusion

The conclusion presents LEAP as an articulated occupancy representation that generalizes across human shapes and poses and outperforms NASA in generalization and fidelity. It also identifies learning from imagery and extending to clothed bodies as future work.

7. Conclusion

  • LEAP generalizes across human shapes and poses and outperforms NASA in generalization ability and fidelity across the reported settings.The representation performs efficient occupancy checks from bone transformations and 3D query points, yielding a fully differentiable volumetric representation.Source: §7 · p. 8
  • The point-based loss is presented as an efficient way to resolve collisions between humans and objects represented by point clouds.Source: §7 · p. 8
  • Learning LEAP from imagery and extending it to articulated clothed bodies are future directions; raw-scan results are preliminary.The preliminary raw-scan results show realistic surface details.Source: §7 · p. 8

A. Overview

  • The supplementary document covers network modules, training, additional results, limitations, and notation.Source: A. Overview · p. 9

B. Network architecture

The network architecture combines structure, shape, and pose encoders into a global feature vector, then uses occupancy and LBS networks for point and skinning-weight predictions.

B. Network architecture

  • A global feature vector combines structure, shape, and pose features, with dimensions 128, 312, and 156, respectively.The concatenated vector has 596 dimensions.Source: B. Network architecture · p. 9

B.1. Structure encoder

  • The structure encoder uses 52 MLPs in a skeleton tree, with each joint producing a compact six-dimensional bone code.Each MLP takes parent features, rotation, bone length, and joint location as input; a separate linear layer handles the root.Source: §B.1 · p. 9

B.2. PointNet encoder

  • PointNet encodes point clouds into fixed-size features: 128 dimensions for shape and 100 dimensions for each LBS network.The structure encoder’s MLPs contain 500 trainable parameters.Source: §B.2 · p. 9, Table B.1 · p. 9

B.3. Bone projection layers

  • Bone projection layers map the 596-dimensional global feature into compact 12-dimensional per-bone features using grouped 1D convolutions.Source: §B.3 · p. 9

B.5. Linear blend skinning networks

  • Forward and inverse LBS networks use similar MLP architectures to regress skinning weights, conditioned on latent features.The inverse network’s 280-dimensional feature combines the forward feature with pose and joint-location features; the forward feature is 200-dimensional.Source: §B.5 · pp. 9–10, Figure B.3 · p. 10

B.4. ONet

  • The occupancy network is an MLP with five ONet blocks; its conditional batch-normalization blocks use local point and cycle-distance features.The conditional blocks are followed by ReLU activations.Source: Figure B.2 · p. 9, §B.4 · p. 9

C. Training

Training uses separate procedures for the LBS networks and occupancy modules, with the LBS networks trained first and then held as differentiable functions during occupancy training.

C. Training

  • The inverse and forward LBS networks are trained first, then used as deterministic differentiable functions during occupancy training.Source: C. Training · p. 10
  • Occupancy training jointly trains all modules except the LBS networks, using about 1.5M trainable parameters and 3,072 sampled points per batch.The batch includes 1,536 uniform points and 1,536 near-surface points: 1,024 posed and 512 canonical.Source: C. Training · p. 10
  • The inverse LBS network has about 1.5M parameters and uses 1,024 uniform plus 1,024 near-surface points per batch; the forward network has about 1.2M parameters.Forward-network batches include 1,024 canonical-space points and points sampled for inverse-network training, mapped with pseudo ground-truth weights.Source: C. Training · p. 10

D. Additional experiments and results

The appendix supplements the experiments on generalization, LBS learning, and placing people in scenes.

D. Additional experiments and results

D.1. Evaluation of linear blend skinning networks

The additional experiments report improved collision scores over the baseline for scene placement, with a caveat for the human-scene score, and compare forward and inverse LBS performance across settings.

D.1. Evaluation of linear blend skinning networks

  • The forward LBS network consistently outperforms the inverse network across settings, while inverse-LBS performance worsens on unseen subjects.The reported metric is the l1 distance between predicted and pseudo ground-truth weights; the comparison spans DFaust multi-person and unseen-pose settings and MoVi unseen subjects.Source: §D.1 · p. 10

D.2. Generating people in scenes

  • The human-scene collision score uses a noisy SDF, retained to make the comparison with PLACE [72] fair.Additional qualitative views show person-person collision resolution, and the noisy SDF used for scoring is visualized.Source: §D.2 · p. 10
  • Collision scores improve over PLACE [72], except that the human-scene score is unreliable because of noisy SDF; deep scene interpenetrations are also resolved qualitatively.The larger-room experiment optimizes 60 interpenetrating body pairs sampled from 50 people.Source: §D.2 · p. 10

E. Limitations

The limitation is inaccurate inverse-LBS mapping in challenging close-contact or self-intersecting configurations; the appendix also supplies a notation summary.

E. Limitations

  • The inverse LBS network may mis-map query points to canonical space when poorly trained and body parts are close or self-intersect, distorting posed-space occupancy.The authors identify explicit self-contact modeling as a promising future direction; Figure E.8 illustrates self-intersecting hands on an unseen MoVi subject.Source: E. Limitations · p. 10, Figure E.8 · p. 14
  • The generalization figure covers unseen DFaust poses and unseen MoVi subjects, while the LBS evaluation table compares forward and inverse networks across experiment settings.The table caption identifies l1 distance as the metric and specifies the datasets for the multi-person, unseen-pose, and unseen-subject settings.Source: Figure D.4 · p. 11, Table D.2 · p. 11

F. Notation

Related papers

Loading 2104.06849v1…