PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Deep Implicit Functions for 3D Shape

Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, Thomas Funkhouser

arXiv:1912.06126v2cs.CVcs.GR

TL;DR

3D shape representations must balance accurate reconstruction, compact storage, efficient computation, consistency, generalization, and depth-based inference. LDIF addresses this by decomposing space into structured local implicit functions inferred from meshes or posed depth images. It improves reconstruction, depth completion, and unseen-class generalization while using far fewer parameters than the state of the art.

  • Problem

    Existing explicit representations preserve detail but lack shape priors, while learned global representations struggle with complex details and novel shape classes.

  • Method

    LDIF decomposes a shape into Gaussian-arranged local regions, each paired with a latent code decoded by a local deep implicit function.

  • Results

    LDIF improves F-Score by 10.3 points for trained-class autoencoding, 17.8 points for unseen classes, and 15.8 percentage points for depth completion, while using approximately 0.4% of OccNet’s parameters.

  • Takeaways & Limitations

    The representation supports accurate reconstruction, depth completion, generalization, compact storage, and structured decompositions across diverse shape collections.

  • Takeaways & Limitations

    LDIF uses flat local regions, assumes known camera poses for depth reconstruction, and estimates a constant number of regions.

Abstract

from arXiv · show

The goal of this project is to learn a 3D shape representation that enables accurate surface reconstruction, compact storage, efficient computation, consistency for similar shapes, generalization across diverse shape categories, and inference from depth camera observations. Towards this end, we introduce Local Deep Implicit Functions (LDIF), a 3D shape representation that decomposes space into a structured set of learned implicit functions. We provide networks that infer the space decomposition and local deep implicit functions from a 3D mesh or posed depth image. During experiments, we find that it provides 10.3 points higher surface reconstruction accuracy (F-Score) than the state-of-the-art (OccNet), while requiring fewer than 1 percent of the network parameters. Experiments on posed depth image completion and generalization to unseen classes show 15.8 and 17.8 point improvements over the state-of-the-art, while producing a structured 3D representation for each input with consistency across diverse shape collections.

1. Introduction

LDIF addresses the tension between explicit representations that preserve detail and learned representations that provide compact shape priors but struggle with complex shapes and novel classes. It decomposes a global implicit representation into locally decoded functions, supporting multiple reconstruction tasks with improved scalability and efficiency.

  • Existing representations trade off detail and flexibility against compactness and shape priors: explicit methods lack learned priors, while learned global vectors struggle with complex details and unseen classes.No current representation is described as satisfying all desired properties simultaneously.
  • Deep implicit functions represent shapes with a single fixed-length latent vector and repeatedly evaluate a complex network, limiting shape complexity, generality, and computational efficiency.
  • LDIF combines SIF-style spatial decomposition with local deep implicit functions, associating latent codes with Gaussian-defined regions to recover finer geometric detail.The representation is both structured like SIF and implicitly decoded like DIF.
  • 10–15 points better F-Score performance than the state-of-the-art is reported for shape reconstruction benchmarks, with fewer than 1% of the network parameters.The system design supports 3D autoencoding, depth image completion, and partial surface completion.
  • The pipeline improves local prediction by modeling DIF outputs as residuals to Gaussian functions, focusing encoding on local regions, and simplifying the decoders.The design also investigates rotational degrees of freedom, symmetry constraints, smaller latent codes, and fewer layers.

2. Related Work

Related work spans traditional explicit and implicit representations, learned global feature vectors, and structured decompositions. LDIF builds on these lines by combining learned shape priors with local structure intended to improve scalability and efficiency.

  • Traditional Shape Representations: Traditional meshes, point clouds, voxel grids, and implicit surfaces are simple and hardware-efficient but do not leverage shape priors and can be inefficient in expressiveness.These limitations make traditional reconstruction pipelines memory- and computation-intensive and poor at completion.
  • Learned Shape Representations: Learned implicit representations such as OccNet [24], ImNet [7], DeepSDF [27], and DISN [45] improve reconstruction and completion but rely on a single global feature vector.The global representation limits their scalability and generalization.
  • Structured Shape Representations: Structured representations compose simpler elements to encode repeated and hierarchical shape structure, including scene graphs, CSG trees, partition of unity implicits, and learned structured models.

3. Local Deep Implicit Functions

LDIF represents a 3D shape as overlapping local regions, combining analytic Gaussian structure with learned local implicit functions for finer detail. This decomposed representation replaces one global latent code with local parameter-code pairs.

  • LDIF represents a shape using N local elements, each parameterized by analytic variables θi and latent variables zi.The representation supports inside/outside classification for query points x.
  • Analytic and deep shape functions: Each local element combines a Gaussian function g(x, θi), which defines coarse shape and influence, with a deep function f(x, zi) for details.The deep function modulates the local analytic function rather than representing the entire shape globally.
  • The global implicit function is decomposed into a sum of local implicit functions using short latent codes and analytic parameters instead of one potentially long code.This is the key representational difference from typical deep implicit functions.
  • Analytic shape function: The analytic component uses an oriented, anisotropic 3D Gaussian with ten parameters: scale, center, radii, and Euler angles.The transformation induced by the last nine variables maps points into each element’s local coordinate frame.
  • Deep shape function: The local decoder transforms x into the element’s local frame and uses fewer layers than the original OccNet-style network because it predicts only local modulation.The cited passage states that the local decoder has 9 layers versus 33.
  • Symmetry constraints: For man-made shapes, symmetry constraints evaluate selected elements at both x and a reflected point, increasing effective elements without extra stored parameters.Half of the shape elements can be constrained to share this symmetry behavior.

4. Processing Pipeline

The LDIF pipeline predicts local-region parameters and latent codes from meshes or posed depth images, then decodes them into an implicit function for reconstruction. Training combines pointwise inside/outside supervision with a conservative center-placement loss.

  • Pipeline: At inference, the system outputs parameters Θ and latent codes Z for N overlapping regions that can classify any query location as inside or outside.Complete surfaces are reconstructed by evaluating LDIF on a regular grid and applying Marching Cubes.
  • Mesh input: For meshes, the encoder renders 20 depth images and uses ResNet50 to regress Θ, while sampled points and normals provide local inputs for latent-code estimation.The mesh is represented through views at 137 x 137 resolution and 10K sampled points with normals.
  • Depth-image input: For posed depth images, XYZ coordinate channels are fed to ResNet50 for Θ prediction, while latent codes are regressed using the same local processing as meshes.The input is a 3-channel stack of 224 x 224 images in object coordinates.
  • Training losses: The training objective combines point-sample loss LP with shape-element center loss LC.The total loss weights these two terms separately.
  • Point Sample Loss: Point-sample loss uses 1024 near-surface and 1024 uniformly sampled points to supervise predicted inside/outside values against the ground-truth indicator.The decoded value is scaled, passed through a sigmoid, and compared with an L2 loss.
  • Shape Element Center Loss: Center loss penalizes predicted element centers only when they are definitely outside the target shape, using a threshold that accounts for the coarse 32x32x32 signed-distance grid.Centers inside the shape are not penalized; centers outside the grid use an additional bounding-box loss.

5. Experimental Setup

The experiments evaluate LDIF on ShapeNet using standard reconstruction metrics and compare it with leading implicit, structural, and explicit mesh baselines. Unless otherwise specified, models use 32 shape elements and 32-dimensional latent vectors.

  • Configuration: Experiments generally use N = 32 shape elements and M = 32-dimensional latent vectors.These settings apply except where otherwise noted.
  • Datasets: The default dataset is ShapeNet with the 3D-R2N2 train/test splits and an additional 85%, 5%, 10% train/validation/test subdivision.Shapes are made watertight using the Occupancy Networks depth-fusion pipeline.
  • Metrics: Reconstruction is evaluated with mean IoU, mean Chamfer distance, and mean F-Score at τ = 0.01, with discussion focused mainly on F-Score.The paper notes that IoU is difficult to interpret at low values and Chamfer distance is sensitive to outliers.
  • Baselines: The main comparisons use Occupancy Networks [24] and Structured Implicit Functions [13], with AtlasNet as an explicit-mesh baseline.OccNet is identified as the state-of-the-art deep implicit-function baseline, while SIF is the structural-decomposition baseline.

6. Experimental Evaluation

LDIF improves reconstruction accuracy and efficiency over established baselines, while also supporting consistent decompositions, unseen-class generalization, and non-rigid human-body shapes.

  • Accuracy: LDIF reaches a mean F-Score of 92.2, exceeding OccNet by 10.3 points, AtlasNet by 20.0, and SIF by 33.2.It improves on OccNet for 93% of test shapes, with better geometric details and robustness to unusual part placements.
  • Efficiency: LDIF’s N = 32, M = 32 decoder is 0.004× the size of OccNet while providing 1.13× better F-Score.At equal shape-vector dimensionality, LDIF performs comparably to OccNet, outperforms SIF, and scales to larger embeddings.
  • Consistency: LDIF decomposes shapes into consistently indexed elements across object categories, with slightly better qualitative consistency than SIF.Element symmetries and rotations provide degrees of freedom for decomposing shapes with fewer elements.
  • Generalizability: On unseen ShapeNet classes, LDIF achieves a mean F-Score of 84.4, 17.8 points above OccNet and 41.4 points above SIF.It is more accurate on 91% of novel examples without fine-tuning.
  • Domain-independence: LDIF reconstructs human bodies across large non-rigid pose and shape variation, achieving 93% mIOU versus 85% for SIF.Despite no pose or subject-alignment supervision, it produces surfaces close to the originals and establishes coarse correspondences.

7. Applications

LDIF extends to completion from partial observations, outperforming an apples-to-apples OccNet* depth baseline and supporting unsupervised completion of human scans without domain-specific templates.

  • 3D Completion from a Single Depth Image: The depth-completion pipeline uses a single 512 × 512 depth image and is trained from scratch on synthetic views of 3D-R2N2 shapes.OccNet* is trained on the same data with an XYZ-image encoder to isolate representation-decoding differences.
  • 3D Completion from a Single Depth Image: LDIF’s depth-completion F-Score is 78.2 versus 62.4 for OccNet*, a 15.8-point improvement.Qualitative examples show better details in both observed and unobserved regions, especially on hard examples.
  • Ablation study: Local PointNet encoding, XYZ inputs, and partial symmetry each improve depth completion, with removing local PointNet encoding reducing F-Score by 11.4 points.The ablation identifies local encoding of shape-element latents as the largest reported effect.
  • Human-scan completion: For incomplete CAESAR human scans, LDIF captures high-frequency details missing from SIF without requiring a human-body template or manual supervision.S-SCAPE performs better but requires a domain-specific template, landmarks, and bootstrapping; LDIF is domain-independent and unsupervised.

8. Conclusion

The paper introduces LDIF as a structured implicit representation built from local functions and demonstrates improvements in reconstruction accuracy, generalization, efficiency, and completion. It also identifies limitations involving spatial hierarchy, camera poses, region counts, and structured-implicit combinations.

  • 8. Conclusion: LDIF represents shapes as local 3D functions whose Gaussian-weighted residuals are decoded from latent vectors.The method infers structured shape elements and encodes points within each element using PointNet and a residual decoder.
  • 8. Conclusion: 10.3 points higher F-Score than [24] is reported for autoencoding test models from trained classes, while unseen classes improve by 17.8 points.
  • 8. Conclusion: 15.8 percentage points higher depth-completion results than [24] are reported, and the local decoder uses approximately 0.4% of [24]'s parameters.
  • 8. Conclusion: LDIF also reconstructs complete human-body surfaces from partial scans without changing the representation.The supplied conclusion reports this application but does not provide a numerical result.
  • 8. Conclusion: Future work includes multiresolution spatial hierarchies, camera-pose estimation, dynamically varying numbers of local regions, and deeper combinations of structured and implicit representations.

A. Hyperparameters

The appendix specifies training hyperparameters, network choices, point-sampling procedures, activation functions, and metric normalization details for the autoencoder experiments.

  • A. Hyperparameters: The depth-image encoder uses a ResNet50 V2 trained from scratch, with 20 depth images concatenated channelwise before encoding.
  • A. Hyperparameters: The PointNet encoder removes its 64x64 orthogonal transformation to improve speed and reduce memory requirements.
  • A. Hyperparameters: The local decoder follows OccNet's overall structure but reduces residual blocks from five to one and decreases latent-layer widths proportionally.
  • A. Hyperparameters: Local encoding samples up to 1,000 points within threshold r = 4.0 after transforming 10,000 points into each local frame.
  • A. Hyperparameters: Evaluation uses normalized-coordinate metrics against watertight ground-truth meshes, including F-Score at τ = 0.01 and Chamfer distance with a scaling factor of 100.
  • A. Hyperparameters: Training hyperparameters and optimization details for the autoencoder are provided in Table 5.
Loading 1912.06126v2…