Source-linked AI summary
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, Hao Li
TL;DR
Detailed clothed-human digitization from sparse images is difficult because existing representations limit resolution, alignment, topology, or visibility of unseen regions. PIFu uses pixel-aligned implicit functions to infer geometry and texture from a single image while extending naturally to multiple views. The method produces high-resolution, plausible surfaces and textures, including largely unseen regions, but remains limited by unresolved scale inference and unoccluded-subject assumptions.
Problem
Existing 3D representations limit detailed clothed-human digitization by restricting resolution, spatial alignment, or topology, while unseen regions remain difficult to infer.
Method
PIFu combines a fully convolutional image encoder with a continuous implicit function that aligns pixel features with global 3D context for surface and texture inference.
Results
PIFu infers high-resolution, plausible clothed-human geometry and complete texture from a single image, including largely unseen regions, and extends to multiple views.
Takeaways & Limitations
Pixel-aligned implicit representation supports detailed clothed-human digitization with arbitrary topology and surface-aligned texture inference from partial observations.
Takeaways & Limitations
Scale-factor inference remains open, and the demonstrated examples exclude subjects occluded by other objects or scene elements.
Abstract
from arXiv · showhide
We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu can produce high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.
1. Introduction
PIFu addresses detailed clothed-human digitization from one or more images by aligning pixel features with global 3D context. It produces high-resolution geometry and texture while preserving visible details and inferring plausible unseen regions.
- Single-image 3D inference remains limited in resolution and accuracy because existing representations are ineffective for detailed domain-specific modeling.
- PIFu represents clothed-human surfaces with pixel-aligned implicit functions that connect local image features to global 3D context.The method learns per-pixel features and evaluates whether queried points along camera rays lie inside or outside the surface.
- The representation preserves input-image details while inferring plausible geometry in largely unseen regions.
- PIFu produces high-resolution models of complex hairstyles and clothing, including skirts, scarves, high-heels, and wrinkles matching the image at pixel level.
- Regressing RGB values at queried surface points extends PIFu to complete texture inference, while multi-view constraints incorporate additional information from arbitrary numbers of views.
- The method is evaluated on challenging real-world clothed-human images and high-end photogrammetry scan datasets, including dynamic reconstructions from video.
2. Related Work
Prior clothed-human digitization methods rely on parametric bodies, voxels, sparse-view reconstruction, or view synthesis, each leaving important clothing, topology, detail, or visibility challenges.
- Parametric human models capture body measurements and motion but generally produce naked bodies, ignoring clothing, hair, and accessories.
- Displacement-based clothing models handle some skin-tight garments but fail on complex topologies such as dresses, skirts, and long hair.
- Voxel-based methods directly generate 3D people from images, but their memory requirements limit fine-scale detail.
- Sparse-view reconstruction methods combine multiple views but also rely on memory-intensive voxel representations, with some requiring at least two views.
- View-synthesis approaches can predict back views for texturing, but they cannot handle self-occluding regions and side views.
3. PIFu: Pixel-Aligned Implicit Function
PIFu represents clothed humans as continuous, pixel-aligned implicit fields, enabling detailed geometry and texture inference from single or multiple images. Its feature aggregation extends the framework naturally to arbitrary numbers of views.
- PIFu Representation: PIFu combines a fully convolutional image encoder with a continuous MLP-based implicit function whose surface is a level set.The representation avoids explicitly storing the embedded 3D space.
- PIFu Representation: Pixel-aligned image features preserve local image detail while supporting memory-efficient geometry with arbitrary topology.The continuous representation can generate detailed geometry without voxel discretization.
- Digitization Pipeline: Surface PIFu predicts a continuous inside/outside probability field, from which an iso-surface is extracted at threshold 0.5.Inference densely samples the field and uses Marching Cubes to extract the implicit surface.
- Texture Inference: Tex-PIFu extends the implicit output from a scalar to an RGB vector field, directly predicting colors on surface geometry with arbitrary topology and self-occlusion.Texture training samples colors around surface points using normal offsets, rather than restricting color prediction to the exact surface.
- Multi-View PIFu: Multi-view PIFu decomposes f into embedding and reasoning networks, aggregates view-specific embeddings in 3D world coordinates, and supports more than three views.The aggregated features inform surface and texture predictions from arbitrary available views.
4. Experiments
The experiments evaluate PIFu across datasets, reconstruction metrics, qualitative clothing and detail fidelity, texture inference, and multi-view reconstruction. Results show high-resolution, detailed reconstructions and direct surface texture inference across challenging clothed-human inputs.
- Datasets: PIFu is evaluated on RenderPeople, BUFF, and DeepFashion, including ground-truth measurements and diverse complex clothing.The experiments use multiple datasets to assess reconstruction performance across varied settings.
- Metrics: The evaluation measures P2S distance, Chamfer distance, and normal reprojection error between reconstructed and ground-truth surfaces.P2S is measured in centimeters from reconstructed vertices to the ground-truth surface.
- Quantitative Results: PIFu produces pixel-aligned high-resolution surfaces that capture hairstyles and clothing wrinkles, with higher fidelity than voxel-based representations under the same encoder architecture.The comparison attributes the fidelity difference to the greater expressiveness of implicit representations.
- Texture and Multi-View Reconstruction: PIFu predicts textures directly on the surface mesh, avoiding the projection distortion and silhouette artifacts reported for SiCloPe.The multi-view experiments compare PIFu with LSM and Huang et al. using three-view inputs and examine explicit depth conditioning.
- Qualitative Results: On DeepFashion images, PIFu handles skirts, jackets, and dresses while recovering high-resolution local details, plausible unseen surfaces, and complete single-image textures.The reconstructed textures support viewing the models from 360 degrees, including dynamic clothed-human performances from a single 2D input video.
5. Discussion
PIFu aligns image pixels with 3D shape to reconstruct detailed clothed humans, including unseen regions, geometry, and texture. The method supports single- and multi-view inputs while remaining memory efficient and topology-free, but scale inference and occlusion handling remain open issues.
- PIFu spatially aligns pixel-level image information with 3D shape for learning-based geometry and texture inference of clothed humans.
- PIFu infers plausible geometry in largely unseen regions while preserving high-frequency details present in the image.
- PIFu directly predicts surface textures, avoiding projection artifacts and enabling texture completion in unseen, concave, and side regions.
- PIFu handles additional views and improves surface and texture predictions as more views are added.
- PIFu produces high-resolution output without voxel representations’ high memory requirements and supports arbitrary-topology textured surfaces.
- The authors identify higher-resolution appearance, scale-factor inference, and reconstruction under occlusion or partial framing as future work.
Appendix I. Implementation Details
The implementation combines photogrammetry-based training data and physically informed rendering with a flexible fully convolutional encoder and sequential surface-to-texture training.
- Experimental Setup: The High-Fidelity Clothed Human Dataset contains 491 textured meshes, split into 442 training subjects and 49 test subjects.Each mesh contains about 100,000 triangles and spans varied clothing, shapes, and poses.
- Experimental Setup: Precomputed radiance transfer simulates global and local light transport, including ambient occlusion, for higher-fidelity real-image reconstructions.Visibility is precomputed on each surface and represented with spherical harmonics.
- Network Architecture: PIFu permits any fully convolutional image encoder; the final surface model adapts a stacked hourglass network and uses group normalization for small-batch training.The architecture is not tied to a specific encoder.
- Network Architecture: Surface reconstruction uses a six-layer MLP with skip connections from the image feature and depth, while multi-view features are aggregated by average pooling.The surface MLP ends with sigmoid activation after leaky-ReLU layers.
- Training Procedure: Surface reconstruction is trained before texture inference, which conditions the texture module on learned surface-reconstruction image features.RMSProp trains surface reconstruction, while Adam trains texture inference.
Appendix II. Additional Evaluations
Additional evaluations examine sampling and encoder choices, then compare PIFu with template-based reconstruction under different input-view requirements.
- Spatial Sampling: Combining adaptive sampling with σ = 5 cm and uniform sampling produces the best qualitative and quantitative surface-reconstruction results.Uniform sampling alone blurs the decision boundary, whereas smaller surface offsets sharpen it but increase artifacts.
- Network Architecture: The appendix compares stacked hourglass, ResNet, and VGG-based encoder architectures on the evaluation data.The associated figures visualize reconstructed geometry and point-to-surface error.
- Network Architecture: ResNet performs slightly better in-domain but overfits and generalizes poorly to BUFF and DeepFashion, motivating the stacked-hourglass final model.The comparison uses three surface-reconstruction architectures.
Appendix III. Additional Results
Additional results compare PIFu with voxel and template-based methods, probe general-object transfer, and assess reconstruction from video sequences.
- Voxel Comparison: PIFu aligns reconstructions with image pixels at higher resolution than voxel regression and supports texturing arbitrary-topology, self-occluding shapes.The voxel representation suffers misalignment from limited precision.
- General Objects: Preliminary class-agnostic ShapeNet experiments reveal new challenges when extending PIFu from clothed humans to general objects.The authors speculate that greater shape variety makes globally coherent reconstruction difficult from pixel-level features alone.
- Video Sequences: PIFu reconstructs video frames independently yet produces reasonably temporally coherent results after fixed cropping and scaling per sequence.The video ground truth uses eight views with controlled lighting, while PIFu uses a single view.