PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner

arXiv:1612.00101v2cs.CV

TL;DR

Partial 3D scans leave missing global structure that local geometry processing cannot reliably reconstruct. The paper combines a semantic 3D-EPN for coarse completion with database-guided shape synthesis for local detail, and reports significant outperformance of current state-of-the-art methods across its experiments. The approach is bounded by extreme partial inputs and the limited geometric variety of available 3D training data.

  • Problem

    Incomplete scans make reconstructing high-level structures such as chair legs or airplane wings impractical for traditional geometry-processing methods.

  • Method

    A volumetric 3D-EPN predicts coarse complete shapes from partial scans, then correlated database geometry is used in iterative synthesis to add high-resolution local detail.

  • Results

    The combined method significantly outperforms current state-of-the-art methods in all reported experiments on partial-shape completion.

  • Takeaways & Limitations

    Combining deep learning for global-structure inference with traditional synthesis for local improvements is supported as a promising direction for mesh completion.

  • Takeaways & Limitations

    The method remains limited by extreme partial inputs and by ShapeNet training data that do not reflect all real-world scene geometries.

Abstract

from arXiv · show

We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution -- but complete -- output. To this end, we introduce a 3D-Encoder-Predictor Network (3D-EPN) which is composed of 3D convolutional layers. The network is trained to predict and fill in missing data, and operates on an implicit surface representation that encodes both known and unknown space. This allows us to predict global structure in unknown areas at high accuracy. We then correlate these intermediary results with 3D geometry from a shape database at test time. In a final pass, we propose a patch-based 3D shape synthesis method that imposes the 3D geometry from these retrieved shapes as constraints on the coarsely-completed mesh. This synthesis process enables us to reconstruct fine-scale detail and generate high-resolution output while respecting the global mesh structure obtained by the 3D-EPN. Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis. In our results, we show extensive evaluations on a newly-introduced shape completion benchmark for both real-world and synthetic data.

1. Introduction

The paper addresses incomplete 3D scans by combining a semantic 3D-Encoder-Predictor Network with database-guided shape synthesis. The network infers global structure, while synthesis restores local detail and produces high-resolution meshes.

  • Motivation: Commodity RGB-D scans are often incomplete because occlusions and sensor restrictions leave surface regions unobserved.These gaps include difficult areas such as spaces behind shelves and fine chair structure.
  • Motivation: Traditional geometry-processing methods can fill small holes but cannot practically reconstruct high-level structures such as chair legs or airplane wings.The limitation motivates data-driven completion based on learned shape priors.
  • Method: 3D-EPNs use volumetric CNNs and semantic classification features to map partial implicit distance fields to complete shapes.The classification network’s probability class vector is provided to the 3D-EPN latent space.
  • Method: The method correlates coarse predictions with CAD database geometry and iteratively synthesizes high-resolution local detail from retrieved shapes.This combines learned global prediction with database-guided 3D shape synthesis.
  • Results: The end-to-end method reports compelling qualitative and quantitative results on synthetic and real-world scanning data, favorably comparing with state-of-the-art methods.Its listed contributions include semantic 3D-EPN completion, high-resolution mesh synthesis, and their combination.

2. Previous Work

Previous work spans local geometric hole filling, handcrafted regularity detection, database retrieval, and fully data-driven learning. The paper positions its method as combining learned global structure prediction with synthesis for local detail.

  • Geometry processing: Geometry-processing methods primarily address small holes using surface primitives, smoothing, or Poisson reconstruction.These approaches are rooted in local surface completion and continuous energy minimization.
  • Regularity-based completion: Symmetry-based methods fill missing data effectively but constrain the shape space to predefined, hand-crafted regularities.Their generality is limited by the regularities they explicitly encode.
  • Database priors: Database retrieval methods can align identical or similar CAD models, but depend on finding sufficiently similar shapes and therefore generalize poorly to new shapes.The paper also uses database geometry, but restricts it to local-detail synthesis rather than global-structure inference.
  • Learning-based completion: Fully data-driven methods aim to generalize across new shapes, while generative models face practical resolution limitations.The paper instead directly predicts missing shape regions and follows prediction with shape synthesis.

3. Method Overview

The method maps partial scans to coarse complete distance fields with a 3D network, then uses learned database correlations and patch synthesis to refine them. This separates global structure inference from high-resolution detail generation.

  • 3D-EPN prediction: Partial and complete models are represented as 3D voxel-grid distance fields, and the network learns to reconstruct complete meshes while respecting known surfaces and free space.Unlike an ordinary autoencoder, the network fills missing data rather than reconstructing its input.
  • 3D-EPN prediction: The 3D-EPN uses semantic class probabilities and a mostly continuous distance-field representation to generalize geometric structure under missing data.The semantic vector enters the network’s latent space, while the distance field supports a task-specific loss.
  • Coarse prediction: The network operates at relatively low resolution, such as 32^3 voxel volumes, to make prediction tractable while recovering missing global structures.Higher-resolution 3D regression is computationally challenging because of high dimensionality.
  • Shape prior correlation: A learned correlation links coarse 3D-EPN outputs to geometrically similar ShapeNet models used as a test-time geometric prior.The database prior supplies geometry while the learned prediction establishes high-level structure.
  • Shape synthesis: Patch-based synthesis copies compatible volumetric patches from nearest neighbors and hierarchically refines the prediction to a 128^3 distance field.The synthesis step transfers local detail while preserving the coarse prediction’s structure.

4. Training Data Generation

Training pairs are generated by virtually scanning ShapeNet objects and computing complete distance fields. The pipeline uses partial trajectories and separate distance-field information to train completion at 32^3 resolution.

  • Dataset construction: Training uses 25,590 instances from eight ShapeNet categories, with 5,384 models in the test set.The data are generated from a ShapeNet model database and divided into training and test objects.
  • Representation: The signed distance field encodes known-empty space, surface location, and unknown values through its sign.The sign channel is separated from absolute distance values before entering the 3D-EPN.
  • Dataset construction: Virtual scanning generates partial reconstructions, while a 3D scanline method produces the complete ground-truth distance field.The resulting pairs model partial observations and their corresponding complete geometry.
  • Resolution: Training pairs use TSDF and DF representations at 32^3 resolution, while final synthesis produces a 128^3 implicit distance field.The two resolutions correspond to coarse network prediction and high-resolution shape synthesis.

5. 3D Encoder-Predictor Network (3D-EPN) for Shape Completion

The 3D-EPN predicts missing distance-field values from partial scans using volumetric convolutions, global semantic context, and local skip connections. It represents scans with distance and known/unknown-space channels and trains with a masked loss focused on unknown regions.

  • Input representation: The network consumes partial scans as volumetric grids with separate distance-field and known/unknown-space channels.The known/unknown channel indicates which regions are missing and provides additional input information.
  • Encoder: A 3D encoder compresses observed distances, empty space, and semantic class predictions into a hidden volume.The semantic predictions come from a 3D-CNN shape classifier and are concatenated into the latent representation.
  • Predictor: A predictor network uses 3D up-convolutions to expand the hidden volume into a 32^3 output of estimated distance-field values.The predictor infers missing values from the global context summarized by the encoder.
  • Skip connections: Skip connections concatenate encoder features with up-convolution outputs to preserve local input structure in the predictions.The connections double the feature-map size at corresponding layers.
  • Training: Training uses a masked L1 loss that counts errors only in unknown regions while enforcing known occupied and empty voxels to match the input.The model is optimized with ADAM using a 0.001 learning rate and 0.9 momentum.

6. Shape Prior Correlation

The method correlates coarse 3D-EPN predictions with a learned shape embedding and retrieves geometrically similar CAD models as a high-resolution geometric prior. Test-benchmark models are excluded to target previously unseen shapes.

  • Learned shape embedding: A 3D-CNN trained on all 55 ShapeNet classes provides a feature descriptor whose embedding reflects shape similarity.The descriptor is learned as a byproduct of discriminative object classification.
  • Database retrieval: The method retrieves the three closest database models to each 3D-EPN output using a k-nearest-neighbor query in feature space.The retrieved models supply full-resolution geometry for later completion.
  • Geometric prior: The method assumes access to full-resolution ShapeNet training meshes and uses them as a geometric prior rather than encoding all fine-scale detail in the deep network.This separates coarse prediction from high-resolution geometric retrieval.
  • Evaluation scope: The 5397 test-benchmark models are excluded from retrieval and feature learning to support generalization to previously unseen shapes.The paper notes that identical physical and virtual objects are a valid real-world scenario but does not explore it.

7. Shape Synthesis and Mesh Generation

Shape synthesis refines the coarse 3D-EPN prediction by copying compatible volumetric patches from nearest shape neighbors across multiple resolutions. The resulting distance field reaches 128^3 voxels before mesh extraction.

  • Synthesis inputs: The synthesis process takes the 3D-EPN prediction and nearest shape neighbors as inputs, then copy-pastes neighbor voxels to construct a high-resolution output.It uses iterative synthesis to add local geometric detail while starting from the coarse prediction.
  • Patch matching: Multi-scale synthesis searches for similar volumetric patches in the k nearest neighbors and updates each predicted voxel with a matched neighbor value.Synthesis proceeds from coarse to fine resolution.
  • Surface restriction: Synthesis considers only voxels near the implicit surface, whose neighborhoods contain at least one isosurface voxel.This restricts refinement to surface-relevant regions.
  • Mesh generation: The hierarchy produces a 128^3 voxel distance field, which is converted into a mesh with Matlab’s isosurface function.The distance field is the representation used before final mesh extraction.

8. Results

The method combines 3D-EPN predictions with shape synthesis to complete partial scans, yielding global structure and local detail. Experiments evaluate completion, classification, and retrieval on synthetic and real-world data, while revealing resolution and input-coverage limitations.

  • The combined 3D-EPN and synthesis pipeline provides both completed global structure and locally accurate geometry.3D-EPN alone completes structure at low resolution, while synthesis alone cannot complete missing structure.
  • The final method outputs a distance field at 128^3 voxels by combining 3D-EPN predictions with 3D shape synthesis.The synthesis stage provides higher-resolution output than the low-resolution network prediction.
  • The method favorably compares with state-of-the-art completion methods on synthetic and real-world scanning data.Quantitative completion results use ℓ1 error on unknown regions against ground-truth distance fields, with a truncation distance of 2.5 voxels.
  • Using 3D-EPN predictions as an intermediate result significantly improves both classification and shape retrieval on the same partial test inputs.The comparison trains classifiers on partial versus complete ShapeNet models, and retrieval accuracy uses the top 3 neighbors’ classes.
  • The 3D-EPN fails to infer smaller components when detail falls below voxel size, and extreme partial inputs may lack enough context for plausible completion.The training pairs use a 32^2 resolution representation, limiting recovery of fine-scale structure.

9. Conclusion and Future Work

The method combines 3D deep learning with 3D shape synthesis to complete partial scans, while future work targets broader scene scale and richer generative or visual conditioning.

  • 9. Conclusion and Future Work: The approach combines a new 3D deep learning architecture with 3D shape synthesis for partially scanned shapes.The authors characterize this combination as an efficient completion method.
  • 9. Conclusion and Future Work: The authors identify combining deep learning for global structure with traditional synthesis for local improvements as a promising direction.This conclusion links the predictor's structural inference with synthesis-based refinement.
  • 9. Conclusion and Future Work: A practical next step is scaling the approach from isolated objects to room-scale scenes such as ScanNet data.The paper also mentions conditioned generative models and RGB-based shading constraints as possible directions.

A. Additional Results on Synthetic Scans

Additional synthetic-scan evaluations examine architecture, representation, training strategy, and real-world transfer, with signed-distance inputs and class-aware multi-class training performing strongly.

  • A. Additional Results on Synthetic Scans: The 3D-EPN with skip connections and a class vector performs best on partial scans with varying trajectory sizes.This configuration also informs the best subsequent shape-synthesis results.
  • A. Additional Results on Synthetic Scans: On real-world PrimeSense scans, the method produces faithful completions even for highly partial input, though quality is slightly worse than on synthetic data.The authors attribute this gap to differences between PrimeSense sensor characteristics and synthetic training data, with no sensor noise currently simulated.
  • A. Additional Results on Synthetic Scans: The signed distance field representation provides the lowest error among the evaluated volumetric representations.It jointly represents known-free space, surface voxels, and unknown space as a smooth function.
  • A. Additional Results on Synthetic Scans: A single network trained across all classes outperforms separate class-specific networks, and class predictions provide the best performance.The authors attribute the multi-class advantage to the larger amount of training data.

E. Evaluation on Different Degrees of Incompleteness

The evaluations show relatively stable completion under increasing missing data, stronger global-structure inference in difficult cases, and latent organization by geometric similarity.

  • E. Evaluation on Different Degrees of Incompleteness: The method maintains relatively low and stable completion errors as partial input ranges from 20% to 70%.Under highly partial input, its error remains comparatively stable while 3D ShapeNets becomes more unstable.
  • E. Evaluation on Different Degrees of Incompleteness: For a chair missing its seat and front back, the method more faithfully infers global structure than the compared approach.These missing regions are identified as difficult for structural inference.
  • E. Evaluation on Different Degrees of Incompleteness: The latent-vector visualization places shapes with similar geometry near one another despite varying degrees of occlusion.The visualization uses 512-dimensional vectors reduced to two dimensions with t-SNE.
  • E. Evaluation on Different Degrees of Incompleteness: Real-world examples compare partial input, 3D-EPN predictions, retrieved shapes, and final synthesized outputs.Retrieved models can be clean yet differ from the input in global structure, motivating synthesis rather than direct replacement.
Loading 1612.00101v2…