Source-linked AI summary
High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, Yizhou Yu
TL;DR
Recovering large missing regions in 3D shapes requires both global structural inference and detailed local geometry. The paper jointly trains a multi-view and volumetric global network with a patch-level refinement network, and reports high-quality completion that outperforms existing state-of-the-art methods.
Problem
Large missing regions and low-resolution volumetric representations make detailed completion of incomplete 3D shapes difficult.
Method
A jointly trained architecture combines global structure inference from multi-view depth and volumetric inputs with patch-level local geometry refinement.
Results
Qualitative and quantitative evaluations on six object categories show that the method outperforms existing state-of-the-art shape-completion methods.
Takeaways & Limitations
Jointly training global and local networks provides a supported direction for high-resolution completion of partial 3D shapes.
Abstract
from arXiv · showhide
We propose a data-driven method for recovering miss-ing parts of 3D shapes. Our method is based on a new deep learning architecture consisting of two sub-networks: a global structure inference network and a local geometry refinement network. The global structure inference network incorporates a long short-term memorized context fusion module (LSTM-CF) that infers the global structure of the shape based on multi-view depth information provided as part of the input. It also includes a 3D fully convolutional (3DFCN) module that further enriches the global structure representation according to volumetric information in the input. Under the guidance of the global structure network, the local geometry refinement network takes as input lo-cal 3D patches around missing regions, and progressively produces a high-resolution, complete surface through a volumetric encoder-decoder architecture. Our method jointly trains the global structure inference and local geometry refinement networks in an end-to-end manner. We perform qualitative and quantitative evaluations on six object categories, demonstrating that our method outperforms existing state-of-the-art work on shape completion.
1. Introduction
The paper addresses high-resolution completion of large missing regions in 3D shapes, where conventional geometry-based and low-resolution learning methods are inadequate. It proposes jointly trained global-structure and local-geometry networks for detailed shape recovery.
- Motivation: Large gaps caused by occlusions, reflectance, and poor lighting remain difficult for complete 3D shape reconstruction.Traditional geometry-based methods generally handle only relatively small gaps, motivating learning-based shape priors.
- Motivation: Volumetric convolutional methods are constrained by computation and memory, typically producing coarse 32^3 representations that lose surface detail.Patch synthesis can refine outputs, but it begins from a low-resolution intermediate shape.
- Method: The proposed framework jointly infers global structure and local geometry through a global network and a patch-level refinement network.The global network combines 3DFCN and LSTM-CF modules, while the local encoder-decoder synthesizes missing surface regions under global guidance.
- Results: Qualitative and quantitative evaluations on six object categories show high-quality completion and improvement over existing state-of-the-art methods.The method converts incomplete point clouds into complete 3D shapes.
- Method: The global structure network maps incomplete inputs to a representation encoding a complete global structure.Its architecture is based on a 3D fully convolutional network and LSTM.
- Method: The patch-level 3D CNN synthesizes detailed surfaces from low-resolution voxel representations under guidance from the global structure network.This design targets local geometric detail while preserving the recovered overall structure.
3. Overview
The pipeline completes partial shapes patch by patch, using a voxelized global representation and local surface inference. Global structure is inferred from multi-view depth and volumetric features before local patches progressively fill missing boundaries.
- Overview: The method iteratively extends surfaces from missing-region boundaries and updates those boundaries until the regions are filled.This patch-by-patch process is analogous at a high level to PatchMatch-based image completion.
- Overview: The input point cloud is voxelized into a 256^3 grid, and 32^3 patches are extracted along missing-region boundaries.The local network maps a volumetric distance field to an implicit complete-shape representation, with 0 inside and 1 outside.
- Overview: Global structure inference uses projected depth images of size 128^2 together with a down-sampled voxelized point cloud D32 at 32^3 resolution.The figure defines S as feature stacking and C as concatenation.
- Overview: Six depth-image streams use 2D convolutional and LSTM layers, then assemble their features into a 32^3 representation alongside volumetric 3D-convolution features.The two branches are concatenated to form the global structure representation.
4. Network Architecture
The architecture combines global structure inference from down-sampled volumetric and six-view depth inputs with local high-resolution geometry refinement guided by the global prediction. Its two subnetworks interact through joint training to improve shape completion.
- Global Structure Inference: The global structure subnetwork processes D32 together with six orthographic 128^2 depth images to infer a complete 3D structural representation.D32 is used for volumetric information, while the six views compensate for its low resolution.
- Global Structure Inference: Six 2D BLSTM output maps are projected onto voxel locations, concatenated, and fused with a parallel 3D-convolutional feature map for voxel-wise prediction.The two branches produce 3D feature maps at size 32^3 before concatenation and final 1 × 1 × 1 convolution.
- Local Geometry Refinement: The local refinement subnetwork infers high-resolution geometry within 32^3 patches cropped from D256 along missing-region boundaries.An encoder-decoder with 3D convolutions, pooling, fully connected layers, and deconvolutions produces voxel-wise binary predictions.
- Local Geometry Refinement: Global guidance enters local refinement through cropped patches from S32 at both the encoding and decoding stages.This connects each local patch P32 to the inferred global structure representation.
- Network Training: The networks are trained in two phases: global inference is trained first, then local refinement is trained while the global network is fine-tuned jointly.The combined objective balances local and global losses with an L2 parameter regularizer.
5. Training Data Generation
The training data cover six object categories and simulate incomplete scans with partial views, holes, noise, volumetric representations, and boundary patches.
- The dataset contains chairs, cars, guitars, sofas, guns, and self-collected animals, processed separately by category.ShapeNet supplies five categories; animal models were manually aligned to the ShapeNet coordinate system.
- Incomplete point clouds are generated by fusing multiple simulated partial scans with missing regions.Each created point cloud is represented using a volumetric distance field.
- Fifty boundary patches are sampled per incomplete model, then clustered so only cluster centers become training patches.Clustering reduces repeated local geometries and increases training-sample diversity.
- Virtual scanning uses 20 camera viewpoints, randomly selects 3–5 views, and adds holes and noise to emulate limited-view acquisition.
- Each point cloud combines colored signed-distance information with a binary surface occupancy channel.The colored signed-distance field distinguishes negative and positive distances, while the binary surface preserves input-point occupancy.
- Six orthographic depth images from the bounding cube’s faces provide the global network’s second input stream.The depth images use jet color mapping.
6. Shape Completion
At test time, the method first infers a coarse complete structure, then iteratively fills missing regions with globally guided local patches.
- The global structure inference network converts an incomplete point cloud into a complete but coarse structure.
- Boundary detection identifies missing-region borders, from which completion proceeds iteratively until the regions are filled.
- Overlapping local 3D patches cover boundary points, and the refinement network predicts voxel-wise interior probabilities under global guidance.Overlapping predictions are combined for patch coverage.
7. Experimental Results
Experiments evaluate qualitative results, benchmark comparisons, and component ablations across six categories, with the method outperforming existing coarse-resolution approaches.
- Qualitative Results: The evaluation covers six object categories and compares incomplete inputs with repaired point clouds from multiple views.Figure 5 presents two models per category.
- Evaluation Metrics: The method is evaluated using normalized distance and completeness to measure repaired-shape accuracy and coverage.Normalized distance is computed from repaired points to the ground truth and normalized by category-wide maximum shape diameter.
- Comparisons with Existing Methods: The proposed method outperforms existing 32^3-level methods even without local geometry inference, with results significantly better than comparison baselines.The comparison includes retrained competing networks and a downsampled ground-truth baseline.
- Comparisons with Existing Methods: The local refinement network uses high-resolution input information, distinguishing it from post-processing patch synthesis that starts from low-resolution outputs.
- Ablation Study: Ablation performance drops to 0.912 without global guidance, while removing BLSTM context modeling and AUC loss reduces performance to 0.896 and 0.904.
8. Conclusion
The paper presents a 3D-CNN framework for completing partial shapes and reports improved performance over existing state-of-the-art methods.
- The framework completes partial shapes through 3D convolutional neural networks.
- Reported results significantly improve the performance of existing state-of-the-art methods.
- Jointly training the global and local networks is identified as a promising direction.
1. Visualization of Input
The visualizations represent input point clouds using CSDF, BSurf, and projected depth images, alongside global-structure predictions and ground truth across object categories.
- Input representations: Each point cloud is converted into a signed distance field and represented as Colored SDF and binary surface channels.The representation is illustrated using cross sections from a 256^3 voxelized representation.
- Input representations: Two sampled projected depth images are shown as part of the input representation.
- Global structure visualization: The global structure inference network outputs are displayed with inputs and ground-truth structures for two models from each object category.
3. Sampled Outputs from Ablation Study of Global Structure Prediction
The ablation study compares global structure predictions under changes to losses, input channels, depth images, context modeling, and network architecture, while additional figures compare completion with Poisson reconstruction across categories.
- Global structure ablations: The global structure ablation removes AUC loss, projected depth images, or global context modeling to test their effects.
- Global structure ablations: Additional ablations isolate CSDF and BSurf inputs and replace the global network’s 1x1x1 convolution with an encoder-decoder.
- Comparison results: Poisson reconstruction is compared with the method’s high-resolution completion and ground truth for five sampled models.
- Category galleries: Additional high-resolution completion galleries present one figure per object category, including chair, car, guitar, and other categories.
6. Shape Completion for larger missing regions
The method is evaluated on chair point clouds with substantially larger missing regions, including cases where an entire side is absent, and reports completion quality using completeness, normalized distance, and F1-score.
- Experimental setting: The large-missing-region setting uses one or two nearby viewpoints, producing inputs with an average completeness of 55.68%.
- Completion results: 91.67% average completeness and 0.00459 normalized dist are reported for completed models.
- Completion results: The global and local refinement networks achieve F1-scores of 0.895 and 0.951, respectively.Both scores are described as only slightly lower than those obtained from point clouds scanned from 3–5 viewpoints.
- Qualitative results: Two sampled completion results are shown for models with large missing regions, including cases with one entire side missing.