PaperlayerView exact revision on arXiv

Source-linked AI summary

PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction

Zerong Zheng, Tao Yu, Yebin Liu, Qionghai Dai

arXiv:2007.03858v2cs.CV

TL;DR

Single-image 3D human reconstruction must handle missing depth, difficult poses, and varied clothing while remaining geometrically detailed. PaMIR combines SMPL-conditioned implicit fields with depth-ambiguity-aware training and body-reference optimization. The framework reports improved robustness and surface details, extends to multi-image inputs without calibration or synchronization, and outperforms state-of-the-art methods within the reported settings.

  • Problem

    Single-image 3D human reconstruction is ill-posed because depth is unavailable and poses, body shapes, clothing, and lighting vary.

  • Method

    PaMIR combines a parametric SMPL model with a free-form implicit surface function, depth-ambiguity-aware reconstruction loss, and body-reference optimization.

  • Results

    The method outperforms state-of-the-art methods in robustness and surface details and supports multi-image inputs without camera calibration or synchronization.

  • Takeaways & Limitations

    SMPL semantic conditioning and mutual optimization between the body template and outer surface support reconstruction across challenging poses, clothing types, and input setups.

  • Takeaways & Limitations

    The multi-image extension neglects surface-detail deformation across frames and is trained only on multi-view renderings without pose deviations.

Abstract

from arXiv · show

Modeling 3D humans accurately and robustly from a single image is very challenging, and the key for such an ill-posed problem is the 3D representation of the human models. To overcome the limitations of regular 3D representations, we propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. In our PaMIR-based reconstruction framework, a novel deep neural network is proposed to regularize the free-form deep implicit function using the semantic features of the parametric model, which improves the generalization ability under the scenarios of challenging poses and various clothing topologies. Moreover, a novel depth-ambiguity-aware training loss is further integrated to resolve depth ambiguities and enable successful surface detail reconstruction with imperfect body reference. Finally, we propose a body reference optimization method to improve the parametric model estimation accuracy and to enhance the consistency between the parametric model and the implicit function. With the PaMIR representation, our framework can be easily extended to multi-image input scenarios without the need of multi-camera calibration and pose synchronization. Experimental results demonstrate that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.

1 INTRODUCTION

Single-image 3D human reconstruction is ill-posed because images lack depth and vary in pose, shape, clothing, and lighting. PaMIR combines SMPL-conditioned implicit representation with depth-aware training and body-reference optimization to reconstruct detailed humans robustly.

  • Single RGB images make 3D human recovery difficult because depth is missing and body shapes, poses, clothing, and lighting vary.
  • DeepHuman improves robustness by conditioning volumetric reconstruction on SMPL, but regular occupancy volumes limit high-quality geometric detail.
  • PaMIR combines the parametric SMPL model with a free-form implicit surface function in a unified learning and optimization framework.
  • The PaMIR network converts image features and an SMPL-derived feature volume into an implicit surface representation that provides soft semantic constraints for challenging poses and self-occlusions.
  • A depth-ambiguity-aware reconstruction loss adapts supervision to predicted SMPL estimates and 2D observations, improving robustness to inaccurate depth alignment.
  • Body-reference optimization refines the initial SMPL estimate using the network and its outputs, while the framework supports multi-image inputs without calibration or synchronization.

2 RELATED WORK

Related work spans multi-view reconstruction, parametric body estimation, non-parametric single-image methods, and implicit representations. These approaches trade setup requirements, body-model scope, geometric detail, and flexibility in different ways.

  • Multi-view methods integrate silhouette, stereo, and shading cues, while sparse-view systems use about 4 camera views for coarse surface-detail capture.
  • Parametric body-estimation methods fit or regress statistical models such as SMPL to recover shape and pose from single images.
  • Parametric body models capture minimally clothed shape and pose but lack the ability to represent general clothing layers.
  • Implicit representations avoid point-cloud structure loss, voxel-grid memory demands, and mesh topology constraints while flexibly representing arbitrary surfaces.
  • PIFu and PIFuHD use image features for high-resolution human reconstruction and local-detail capture, whereas the proposed method also handles human-object interaction and multi-image inputs.

3 SURFACE REPRESENTATION

The section combines pixel-aligned image features with SMPL-derived volumetric features in PaMIR, using a conditioned implicit function to reconstruct human surfaces. SMPL provides pose- and shape-aware regularization, while depth-aware supervision registers the implicit field with predicted poses.

  • Deep Implicit Function: Deep implicit functions represent surfaces as level sets of an occupancy probability field conditioned on object information.PIFu samples pixel-aligned image features and point depth to estimate occupancy along camera rays.
  • Deep Implicit Function: Pixel-only implicit reconstruction can fail under self-occlusion and large pose variation, producing inconsistent surfaces or incomplete and asymmetric bodies.The stated failure modes include multiple occupancy peaks along viewing rays, broken arms, and unequal legs.
  • Parametric Body Model: SMPL maps pose and shape parameters to a posed mesh, providing a pose- and shape-aware geometry prior for reconstruction.The framework uses GCMR for initial body-shape inference but is designed to handle inaccurate body initialization through body reference optimization.
  • PaMIR: Parametric Model-Conditioned Implicit Representation: PaMIR conditions the implicit function on both pixel-aligned image features and voxel-aligned features encoded from a voxelized SMPL mesh.The feature volume is produced by voxelizing the SMPL mesh and encoding the resulting occupancy volume with a 3D encoder.
  • PaMIR: Parametric Model-Conditioned Implicit Representation: SMPL regularization supplies an initial pose-aware guess that factors out pose changes and helps the network focus on surface detail reconstruction.The paper states that this representation improves robustness especially under challenging poses.
  • PaMIR: Parametric Model-Conditioned Implicit Representation: The depth-ambiguity-aware loss shifts point samples according to the depth difference between ground-truth and predicted SMPL models, teaching registration to the predicted 3D pose.This guides the network toward plausible surfaces corresponding to the predicted SMPL model rather than exact ground-truth occupancy alignment.

4 METHOD

The method combines image and SMPL-derived features in a PaMIR implicit representation, then addresses depth misalignment through specialized training and body-reference optimization. Texture inference predicts whole-surface color and blends it with observed image colors.

  • PaMIR: PaMIR encodes image features and an SMPL-derived occupancy volume into a unified implicit surface representation.The SMPL prediction is converted into an occupancy volume and encoded as a volumetric feature.
  • Depth-ambiguity-aware reconstruction loss: Training uses predicted SMPL models and a depth-ambiguity-aware loss to accommodate depth-axis misalignment with ground-truth scans.The loss compensates sampled points along the z-axis using correspondences between predicted and ground-truth SMPL models.
  • Body reference optimization: Body reference optimization iteratively updates SMPL parameters by minimizing a fitting loss between the predicted implicit function and SMPL mesh.The fitting loss penalizes SMPL vertices outside the predicted surface while allowing shrinkage into loose clothing surfaces.
  • Body reference optimization: The optimization directly fits the SMPL model to the implicit surface rather than relying on sparse or dense keypoint observations.Additional constraints, such as 2D keypoints, can still be incorporated.
  • Texture inference: Texture inference predicts an RGBα field, using the alpha channel to blend network-predicted colors with image observations.This allows the network to infer colors across the whole surface while sampling observed colors for visible regions.

5 EXTENSION TO MULTI-IMAGE SETUP

PaMIR extends to multiple images by using SMPL-based correspondences to align features across frames, without requiring camera calibration or synchronization. Additional frames recover details invisible in a single image, although pose deviations and changing surface details can cause artifacts.

  • Multi-image feature aggregation: SMPL-based correspondences establish shared model-space relationships across views, enabling multi-image reconstruction.The multi-image network decomposes feature processing into embedding and occupancy-reasoning components, then aggregates frame features.
  • Training: The multi-image network is fine-tuned from the single-image network using multi-view renderings of 3D human models.
  • Multi-image input: The method supports multi-view or video inputs without camera calibration or synchronization, including images with non-identical poses.The authors describe this as enabling video-based personalized avatar creation.
  • Multi-image results: Adding four video frames recovers back-surface details invisible in the first frame, whereas the single-image result over-smooths that area.The comparison is presented between single-image and multi-image reconstruction results.
  • Limitations: Unmodeled wrinkle changes, inconsistent cross-frame geometry, and challenging pose deviations can cause inaccurate feature fusion and reconstruction artifacts.The multi-image network is trained only with rendered views without pose deviations; the authors suggest 4D data as future work.

6 EXPERIMENTS

The experiments evaluate reconstruction primarily through geometry performance, covering challenging poses, comparisons with state-of-the-art methods, and qualitative and quantitative assessments.

  • Evaluation focus: The evaluation focuses mainly on geometry reconstruction performance.The paper identifies geometry reconstruction as its main evaluation focus.
  • Evaluation design: Experiments cover challenging poses, state-of-the-art comparisons, and qualitative and quantitative evaluations of the proposed contributions.Quantitative results are reported in Table 3.

6.1 Implementation Details

The implementation uses image and volumetric feature extractors, rendered and scanned human data, predicted and ground-truth SMPL models during training, and iterative optimization at test time. Reconstruction requires substantially more time for geometry than texture.

  • Network architecture: The image encoder takes 512×512 images and outputs 256-channel 128×128 feature maps, while a 3D convolutional network extracts volumetric features.The volumetric network uses two convolution layers and three residual blocks.
  • Training data: Training data include 1000 textured human scans with varied clothing, shapes, and poses, split into 900 training and 100 testing scans.An additional 600 models are sampled from the DeepHuman dataset to augment pose variety.
  • Training and testing: Training combines predicted and ground-truth SMPL models, while testing uses RGB images to produce the parametric model and textured reconstructed surface.Body reference optimization is run for all results unless otherwise stated.
  • Runtime: Fifty optimization iterations take about 40 seconds, and full reconstruction takes about 50 seconds for geometry plus 1 second for surface color.The reported full reconstruction timing includes body reference optimization.
  • Network complexity: The implementation reports module parameter counts and execution times in Table 1.

6.2 Evaluation Dataset

The evaluation uses real-world full-body images for single-image reconstruction and VideoAvatar RGB sequences for multi-image reconstruction.

  • Single-image evaluation uses full-body images from DeepFashion and the Internet, with backgrounds removed by neural segmentation and Grabcut refinement.
  • Multi-image evaluation uses 24 VideoAvatar RGB sequences showing different subjects turning 360 degrees in a rough A-pose.

6.3 Results

Qualitative results cover diverse poses and clothing styles, showing that the method reconstructs high-quality 3D humans robustly across these variations and supports multi-image reconstruction with high-resolution details.

  • The single-image examples span dancing, Kungfu, sitting, and running poses, plus loose pants, skirts, sports suits, and casual clothes.
  • The results demonstrate high-quality 3D human reconstruction and robust performance across varied human poses and clothing styles.
  • Using five sampled frames per VideoAvatar sequence, the method reconstructs full-body models with high-resolution details despite differing poses and unavailable camera extrinsics.
  • The multi-image results support the representation's generalization capability for reconstruction from video inputs.

6.4 Comparison

The method is compared qualitatively and quantitatively with parametric, volumetric, depth-map, and implicit-function baselines, outperforming them in reconstruction quality and robustness across settings.

  • Comparison setup: Compared methods represent humans through SMPL deformation, implicit geometry, paired depth maps, or SMPL-conditioned volumetric occupancy.
  • Single-image comparison: Our method reconstructs plausible models under challenging poses and varied clothing, while baselines struggle with loose garments, self-occlusion, broken parts, or high-frequency details.
  • Quantitative comparison: Quantitative evaluation on Twindom and BUFF uses point-to-surface error and Chamfer distance, with results reported in Table 2.
  • Quantitative comparison: Our method outperforms the compared state-of-the-art methods in surface reconstruction accuracy and is reported as more general, robust, and accurate.
  • Multi-image comparison: In multi-image comparisons, all methods recover overall shapes, but ours recovers more surface details because its non-parametric representation permits more flexible reconstruction.

6.5 Evaluation

The evaluation examines multi-modal outputs, depth-aware loss, training with imperfect body references, body-reference optimization, and multi-view detail fusion.

  • Multi-modal outputs: PaMIR produces multiple human models corresponding to different body-pose hypotheses, with reconstructed parts changing consistently with the adjusted SMPL input.
  • Training loss: Depth-ambiguity-aware loss yields pose-consistent reconstructions with fewer artifacts than a traditional-loss baseline under estimated SMPL inputs.
  • Training scheme: Under inaccurate SMPL estimation, the proposed training scheme reconstructs plausible complete bodies where a ground-truth-SMPL-trained baseline can fail to reconstruct full-body models.
  • Reference optimization: Body reference optimization further registers SMPL models to image observations and improves body-pose estimation, with quantitative results reported in Table 4.
  • Multi-view fusion: Fusing information from more input views recovers more geometric detail, evaluated using normal reprojection error from eight uniform viewpoints.

7 DISCUSSION

The framework combines PaMIR, depth-ambiguity-aware reconstruction, and body reference optimization to improve robustness and surface detail in single-image human reconstruction. Its scope remains limited by costly, static-pose scan data, with failures on extremely challenging poses.

  • Contributions: The method combines SMPL with a non-parametric deep implicit function, depth-ambiguity-aware reconstruction loss, and body reference optimization.Together, these components target robustness, surface details, and consistency between the parametric model and implicit function.
  • Results: The framework outperforms state-of-the-art methods in both robustness and surface details.The conclusion reports this outcome for single-image 3D human model reconstruction.
  • Applications: Applying the method independently to video frames produces temporally consistent reconstruction results.The conclusion describes this result as demonstrated in the supplementary video.
  • Limitations and Future Work: High-quality human scans are costly and time-consuming to obtain at large scale, constraining the training data available to the method.The paper identifies scan acquisition as a limitation and future work explores large-scale image and video datasets for unsupervised training.
  • Limitations and Future Work: Static-pose scanning biases training toward simple standing poses, and the method fails on extremely challenging poses.Current scanners require sophisticated capture environments and subjects to remain static, limiting coverage of real-world motions.
Loading 2007.03858v2…