Source-linked AI summary
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
Tong He, John Collomosse, Hailin Jin, Stefano Soatto
TL;DR
Single-view clothed-human reconstruction must resolve ambiguities and maintain plausible global shape from incomplete visual evidence. Geo-PIFu combines geometry- and pixel-aligned implicit features with latent voxel shape regularization, and reports lower reconstruction errors than PIFu on a much larger benchmark. The method still fails when both its 2D and 3D feature streams produce incorrect meshes.
Problem
Single-view reconstruction is under-constrained, and prior approaches lack either dedicated local shape encoding or global shape consistency.
Method
Geo-PIFu fuses geometry-aligned features from latent voxel representations with pixel-aligned features and uses the voxel representation as a coarse shape proxy.
Results
42.7% reduction in Chamfer and Point-to-Surface Distances and 19.4% reduction in normal estimation errors are reported against PIFu.
Takeaways & Limitations
The fused representation supports clothed-human meshes with improved local surface details, global topology regularities, and reduced shape and pose artifacts.
Takeaways & Limitations
When both the 2D and 3D feature streams fail to generate correct meshes, fused features still produce failures.
Abstract
from arXiv · showhide
We propose Geo-PIFu, a method to recover a 3D mesh from a monocular color image of a clothed person. Our method is based on a deep implicit function-based representation to learn latent voxel features using a structure-aware 3D U-Net, to constrain the model in two ways: first, to resolve feature ambiguities in query point encoding, second, to serve as a coarse human shape proxy to regularize the high-resolution mesh and encourage global shape regularity. We show that, by both encoding query points and constraining global shape using latent voxel features, the reconstruction we obtain for clothed human meshes exhibits less shape distortion and improved surface details compared to competing methods. We evaluate Geo-PIFu on a recent human mesh public dataset that is $10 \times$ larger than the private commercial dataset used in PIFu and previous derivative work. On average, we exceed the state of the art by $42.7\%$ reduction in Chamfer and Point-to-Surface Distances, and $19.4\%$ reduction in normal estimation errors.
1 Introduction
Geo-PIFu addresses ambiguities and weak global consistency in single-view clothed-human reconstruction by combining geometry- and pixel-aligned features with a coarse shape proxy. It improves reconstruction quality on a benchmark substantially larger than prior PIFu evaluations.
- Motivation: Single-view reconstruction is under-constrained, while prior global or pixel-aligned approaches can miss local shape patterns and global consistency.These limitations can produce unnatural shapes or poses and loss of high-frequency surface details.
- Method: Geo-PIFu fuses 3D geometry-aligned and 2D pixel-aligned features to resolve query-point ambiguities and recover richer surface details.Geometry features distinguish points that project to similar image coordinates, including front- and back-facing points along similar camera rays.
- Method: A latent voxel representation serves as a coarse human shape proxy that regularizes reconstruction and encourages plausible global shapes and poses.The proxy is especially relevant when the unobserved rear of the person must be inferred, reducing artifacts such as distorted hands and feet.
- Evaluation: 10 times larger than PIFu’s private commercial dataset, the DeepHuman benchmark contains 5436 training meshes and 1359 test meshes.No deep implicit function method had previously been benchmarked on this dataset.
- Results: 42.7% reduction in Chamfer and Point-to-Surface Distances and 19.4% reduction in normal estimation errors are reported against PIFu.The paper associates these gains with improved local surface details and global mesh-topology regularities.
2 Related Work
Related work spans global implicit representations, pixel-aligned implicit functions, and explicit voxel, point-cloud, or parametric-mesh models for single-view reconstruction. Pixel alignment improved local detail, but prior PIFu-based methods retained ambiguity and lacked global shape robustness.
- Implicit representations: Global implicit methods learn surface functions from whole-image representations and achieved strong results on ShapeNet rigid objects with mostly flat surfaces.Examples include Occupancy Networks, DeepSDF, LIF, and DIST.
- Pixel-aligned methods: PIFu and DISN encode query points with locally interpolated pixel-aligned 2D features derived through projection onto the image plane.PIFu demonstrated clothed-human reconstruction with detailed clothing wrinkles.
- Explicit representations: Explicit shape models represent 3D form as voxels, point clouds, or parametric meshes, with representative human-reconstruction systems estimating low-resolution body volumes.BodyNet and VRN illustrate voxel-based approaches reviewed in this section.
3 Method
Geo-PIFu reconstructs clothed human meshes by estimating continuous occupancy from fused geometry-aligned 3D and pixel-aligned 2D features. Latent voxel features provide query-point shape information and coarse global shape regularization, while sparse query supervision learns high-resolution occupancy.
- Geo-PIFu: Geo-PIFu encodes each query point with geometry-aligned 3D features, pixel-aligned 2D features, and its depth for occupancy estimation.The implicit surface function is implemented as a multilayer perceptron.
- Geometry-aligned Features: 3D U-Nets lift the input image into latent voxel features, which are sampled at query points using multi-scale trilinear interpolation.The interpolation uses query-point xyz coordinates and provides dedicated representations for local shape patterns.
- Geometry-aligned Features: Geometry-aligned features resolve ambiguity between query points sharing similar camera-ray projections, improving reconstruction of local surface details.Front- and back-side points can otherwise receive similar pixel-aligned features after projection.
- Coarse occupancy volume loss: A low-resolution occupancy volume is decoded from latent voxel features during training to provide coarse human-shape supervision.The decoder and sigmoid estimate the occupancy volume, while latent voxel features are retained for inference-time trilinear interpolation.
- High-resolution query point loss: Sparse query-point training learns high-resolution continuous occupancy fields because directly estimating dense continuous fields is difficult.Geo-PIFu uses groups of sampled query points to compute occupancy losses, combining coarse-volume and query losses in staged training.
- Results: The resulting meshes show fewer shape artifacts and distortions than PIFu, with improved global regularity and clearer local surface details than DeepHuman and PIFu.DeepHuman’s voxel resolution limits rich surface detail, while PIFu is less robust for complex poses and large self-occlusion.
4 Experiments
Experiments evaluate Geo-PIFu on the public DeepHuman benchmark using global-topology and local-detail metrics, comparisons with competing methods, and feature ablations. Geo-PIFu improves both global alignment and local surface detail, while failures remain when both feature types are incorrect.
- Dataset: 5,436 training and 1,359 test meshes comprise the public DeepHuman benchmark, substantially larger than PIFu’s private dataset.The PIFu dataset contains 442 training and 54 test meshes, mostly of upstanding poses.
- Competing Methods: Geo-PIFu has 30,616,954 parameters, reported as 12 times smaller than PIFuHD in the pretrained-weight comparison.Table 2 evaluates the two models using pretrained weights supplied by their authors.
- Global Topology Results: 42.7% relative error reduction in CD/PSD over PIFu indicates stronger global topology regularity and ground-truth alignment.Reconstructions show fewer topology artifacts and distortions than PIFu and better alignment than DeepHuman.
- Geometry vs. Pixel Aligned Features: Geometry-only features improve global topology, pixel-only features preserve more local details, and their joint use achieves the best global/local performance.The ablation compares individual representations and multiple fusion architectures.
- Local Details: 19.4% relative error reduction in normal-estimation errors over PIFu demonstrates improved local surface details, including clothing wrinkles.Cosine and L2 distances between reconstructed and ground-truth input-view normals measure these details.
- Failure Cases and Adversarial Training: When both geometry and pixel features independently produce severe artifacts, fused features still fail to correct all shape distortions.Adding coarse occupancy and 3D-GAN losses to latent voxel training yields reported CD 2.464, PSD 3.372, Cosine 0.1298, and L2 0.4148.
5 Conclusion
Geo-PIFu fuses geometry- and pixel-aligned query features for single-view clothed-human reconstruction. Latent voxel features resolve query-point ambiguity and regularize global shape, while experiments show improved global and local performance but also identify failure cases and preliminary 3D-GAN results.
- 5 Conclusion: Geo-PIFu fuses geometry- and pixel-aligned query point features for deep implicit single-view reconstruction of clothed human meshes.The representation provides dedicated information for each 3D query point.
- 5 Conclusion: Latent voxel features resolve feature ambiguity and regularize reconstructions, reducing shape, pose, and mesh-distortion artifacts.They are generated by structure-aware 3D U-Nets and supervised with coarse occupancy volume losses.
- 5 Conclusion: The large-scale benchmark and ablation studies demonstrate improved global and local performance compared with competing methods.The conclusion also discusses typical failures and preliminary 3D-GAN training as directions for future work.
Broader Impact
Geo-PIFu may benefit VR/AR and graphics applications by reconstructing clothed human meshes from single-view images, while raising workforce and reliability concerns. The dataset may still contain long-tail biases in human poses and clothing.
- Potential benefits: Single-view mesh reconstruction may support VR/AR applications involving customized virtual human avatars.Examples include conference systems and role-playing games.
- Potential disadvantages: Some entry-level graphics artists and designers might be affected as the industries move toward automatic content generation.The passage frames these techniques as productivity aids rather than replacements for highly skilled workers.
- Failure consequences: Failed reconstructions might produce an unpleasant user experience.The paper also discusses typical failure cases and possible solutions.
- Data considerations: Human poses and clothes may exhibit long-tail dataset biases despite the dataset being 10× larger than competitors’ datasets.The collection procedure can be expanded with additional meshes covering different shapes, poses, and clothes.
A Implementation Details
Implementation resources, including architectures, scripts, pretrained models, and evaluation protocols, are available in the Geo-PIFu GitHub repository.
- Implementation details: The Geo-PIFu repository provides network architectures, training and test scripts, pretrained models, and evaluation protocols.The README includes step-by-step instructions.
B More Results
Qualitative results show that Geo-PIFu produces clothed human meshes with fewer distortions and artifacts than PIFu, while preserving stronger global regularity and local detail.
- Qualitative comparisons: Geo-PIFu results have fewer shape artifacts and distortions than PIFu in single-view clothed human reconstruction.
- Qualitative comparisons: Geo-PIFu improves global regularity and ground-truth alignment while producing more accurate, clearer local surface details than PIFu.The comparison specifically highlights invisible-side meshes for complex poses and large self-occlusion.
- Additional results: Figure 7 presents additional single-view clothed human mesh reconstructions from Geo-PIFu.