Source-linked AI summary
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion
Julian Chibane, Thiemo Alldieck, Gerard Pons-Moll
TL;DR
The paper addresses deficient 3D inputs for continuous surface reconstruction and completion, especially the loss of input detail and poor articulated-human reconstruction. It proposes IF-Nets, which classify features from an aligned multi-scale 3D tensor at continuous query points, and reports stronger reconstruction across ShapeNet and human experiments.
Problem
Existing learning-based 3D reconstruction methods often fail to retain input detail and reconstruct articulated humans from deficient 3D inputs.
Method
IF-Nets encode shapes with an aligned multi-scale 3D deep-feature tensor and classify features sampled at continuous query points.
Results
IF-Nets outperform state-of-the-art baselines across the evaluated tasks, preserving input detail while completing missing structure, including articulated human shapes.
Takeaways & Limitations
The approach combines continuous outputs and multiple topologies with detail preservation and articulated-human reconstruction from sparse or incomplete 3D inputs.
Abstract
from arXiv · showhide
While many works focus on 3D reconstruction from images, in this paper, we focus on 3D shape reconstruction and completion from a variety of 3D inputs, which are deficient in some respect: low and high resolution voxels, sparse and dense point clouds, complete or incomplete. Processing of such 3D inputs is an increasingly important problem as they are the output of 3D scanners, which are becoming more accessible, and are the intermediate output of 3D computer vision algorithms. Recently, learned implicit functions have shown great promise as they produce continuous reconstructions. However, we identified two limitations in reconstruction from 3D inputs: 1) details present in the input data are not retained, and 2) poor reconstruction of articulated humans. To solve this, we propose Implicit Feature Networks (IF-Nets), which deliver continuous outputs, can handle multiple topologies, and complete shapes for missing or sparse input data retaining the nice properties of recent learned implicit functions, but critically they can also retain detail when it is present in the input data, and can reconstruct articulated humans. Our work differs from prior work in two crucial aspects. First, instead of using a single vector to encode a 3D shape, we extract a learnable 3-dimensional multi-scale tensor of deep features, which is aligned with the original Euclidean space embedding the shape. Second, instead of classifying x-y-z point coordinates directly, we classify deep features extracted from the tensor at a continuous query point. We show that this forces our model to make decisions based on global and local shape structure, as opposed to point coordinates, which are arbitrary under Euclidean transformations. Experiments demonstrate that IF-Nets clearly outperform prior work in 3D object reconstruction in ShapeNet, and obtain significantly more accurate 3D human reconstructions.
1. Introduction
The paper targets continuous 3D surface reconstruction and completion from deficient voxel and point-cloud inputs. IF-Nets use aligned multi-scale feature tensors to preserve detail, support varied topologies, and reconstruct articulated humans.
- 1. Introduction: Deficient 3D inputs include low- and high-resolution voxels, sparse and dense point clouds, and complete or incomplete observations.These inputs arise from increasingly accessible 3D scanners and computer-vision pipelines.
- 1. Introduction: Existing representations trade off continuity, topology, resolution, rendering, detail retention, or articulated-shape reconstruction.Implicit functions provide continuous arbitrary-resolution outputs and multiple topologies, but prior methods often lose input detail and miss articulated limbs.
- 1. Introduction: IF-Nets replace a single shape vector with an aligned 3-dimensional multi-scale tensor of deep features.The tensor preserves spatial structure in the Euclidean embedding of the shape.
- 1. Introduction: IF-Nets classify features sampled at continuous query points rather than classifying x-y-z coordinates directly.This makes predictions depend on local and global shape structure instead of memorized coordinate locations.
- 1. Introduction: IF-Nets reconstruct rigid ShapeNet objects more accurately and articulated humans without missing limbs while recovering details such as cloth wrinkles.Experiments compare sparse and dense inputs and report advantages in both global consistency and fine-scale detail.
2. Related Work
Related work organizes 3D reconstruction by representation and object type. Prior voxel, mesh, point-cloud, and implicit approaches each impose limitations involving discretization, topology, rendering, computational cost, or human-shape detail.
- Voxels for rigid objects: Voxel methods provide natural 3D-grid processing but memory grows cubically with resolution, limiting resolution, batch size, or training speed.Higher-resolution variants can remain computationally costly, and grid-based TSDF representations remain resolution-bounded.
- Implicit functions: A hierarchical latent code addresses limitations of one-dimensional shape vectors, whereas IF-Nets use a simpler aligned three-dimensional multi-scale feature tensor.The tensor is aligned with the original Euclidean space embedding the shape.
- Meshes: Mesh-based rigid-object methods commonly deform templates, restricting them to a single topology, while direct mesh prediction remains expensive and may not guarantee clean meshes.Human mesh methods based on SMPL or templates also struggle to model clothing detail and can produce oversmoothed results.
- Point clouds: Point clouds are efficient and sensor-relevant, but obtaining renderable surfaces requires nontrivial classical post-processing.This limitation also helps explain why relatively few methods represent humans directly as point clouds.
- Implicit functions: Learned implicit functions produce continuous representations but prior methods can struggle with complex human poses and retaining input detail.An image-based implicit human method produces compelling detail for simple poses but struggles with more complex poses and lacks multi-scale 3D shape representation.
3. Method
IF-Nets replace a single shape vector and coordinate-based occupancy classifier with aligned multi-scale 3D features queried continuously. This design supports continuous reconstruction and completion from incomplete or sparse inputs while preserving detail and modeling global structure.
- Implicit-function baseline: Prior learned implicit functions produce continuous, arbitrary-resolution surfaces and support multiple topologies, but often miss articulated limbs and input detail.They encode a shape as a latent vector and classify query coordinates as inside or outside the surface.
- Shape encoding: IF-Nets encode 3D inputs with recursively convolved, downscaled feature grids whose early scales capture detail and late scales capture global structure.The resulting multi-scale tensor remains aligned with the original 3D space and supports reasoning about sparse or missing data.
- Shape decoding: At continuous query points, IF-Nets use trilinear interpolation to extract features from each grid and feed them to a point-wise fully connected decoder.The decoder predicts whether each queried point lies inside or outside the shape.
- Shape decoding: Neighborhood-aware point encodings sample the query location and surrounding Cartesian-axis offsets to incorporate local context.The offsets are defined as p + a · e_i · d, with a in {1, 0, −1}, where d is the distance from the center point.
- Method training: Training jointly optimizes the multi-scale encoder and decoder end-to-end with cross-entropy over sampled occupancy points from watertight ground-truth surfaces.Half the samples are drawn near the surface with small σ1 and half from farther surroundings with larger σ2.
- Inference: At test time, the encoder constructs feature grids, the decoder predicts continuous occupancies on a chosen-resolution grid, and marching cubes converts them into a mesh.This produces a continuous and complete representation from a discrete, potentially incomplete 3D input.
4. Experiments
Experiments evaluate IF-Nets on point-cloud completion, voxel super-resolution, and single-view human reconstruction across ShapeNet and Humans data. Across these settings, IF-Nets preserve input detail while completing missing structure and outperform baselines qualitatively and quantitatively.
- Experimental setup: IF-Nets are compared with Occupancy Networks, Point Set Generation Networks, Deep Marching Cubes, and IMNET using IoU, Chamfer-L2, and normal consistency.Higher IoU and normal consistency are better, while lower Chamfer-L2 is better.
- Point Cloud Completion: 3000-point completion preserves local detail and recovers global structure, while IF-Nets outperform all baselines across metrics.Only IF-Nets reconstruct features such as car rear-view mirrors and an additional wardrobe shelf; competitors have larger Chamfer distance than the input.
- Voxel Super-Resolution: For voxel super-resolution, IF-Nets consistently produce accurate, highly detailed results, improving over baselines in all reported metrics.Most baseline methods either hallucinate structure or fail in the qualitative comparisons.
- Voxel Super-Resolution: On Humans voxel super-resolution, IF-Nets reconstruct all limbs and detailed structures, whereas implicit baselines produce truncated or missing limbs.IF-Nets outperform all baselines quantitatively and show greater gains with dense inputs than competing methods.
- Single-View Human Reconstruction: For single-view human reconstruction, IF-Nets infer plausible occluded surfaces from approximately 5000 visible-side points and achieve IoU 0.86, Chamfer-L2 0.011 × 10^-2, and Normal-Consistency 0.90.The results fall between reconstruction quality from 32^3 and 128^3 full-subject voxel inputs, supporting completion of single-view data.
5. Discussion and Conclusion
The paper introduces IF-Nets for reconstruction and completion from deficient 3D inputs, using 3D multi-scale deep-feature tensors and continuous feature queries. Experiments report continuous, topology-flexible reconstructions that preserve input detail and complete occluded regions, while future work targets generative and image-based extensions.
- IF-Nets reconstruct and complete shapes from deficient 3D inputs.
- Their encoding uses a 3D multi-scale tensor aligned with the shape's Euclidean space, while queries classify deep features at continuous locations.
- Experiments report continuous outputs, multiple reconstructed topologies, and large-margin improvements over state-of-the-art baselines across all tasks.
- Single-view point-cloud reconstruction preserves visible details while completing occluded regions, including for articulated shapes.
- Future work will explore generative sampling conditioned on partial input and a two-stage image-based pipeline using predicted depth maps.
- The authors position IF-Nets as a potential building block for reconstructing and completing increasingly prevalent deficient 3D data.