Source-linked AI summary
ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, Matthias Nießner
TL;DR
Incomplete indoor 3D scans leave missing geometry and semantic information, while volumetric data growth limits high-resolution processing of large scenes. ScanComplete uses a size-invariant fully convolutional network with coarse-to-fine autoregressive inference to complete geometry and labels at large spatial extents. The authors report unprecedented scene-scale processing and significant improvements in completion and semantic labeling over existing methods.
Problem
Indoor scans suffer from occlusions and sensor limitations, while volumetric data grows cubically with scene size, making large, high-resolution completion difficult.
Method
ScanComplete trains a fully convolutional 3D network on scene subvolumes and combines coarse-to-fine hierarchical prediction with autoregressive volumetric inference.
Results
The method achieves unprecedented scene completion and semantic labeling at large spatial extents, outperforming existing completion methods and obtaining significantly higher semantic-labeling accuracy.
Takeaways & Limitations
A single approach predicts completed geometry and semantic voxel labels for entire scenes while decoupling training and testing resolutions.
Takeaways & Limitations
The approximately 5cm3 final voxel resolution remains insufficient for resolving fine-scale objects such as cups.
Abstract
from arXiv · showhide
We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels. The key contribution of our method is its ability to handle large scenes with varying spatial extent, managing the cubic growth in data size as scene size increases. To this end, we devise a fully-convolutional generative 3D CNN model whose filter kernels are invariant to the overall scene size. The model can be trained on scene subvolumes but deployed on arbitrarily large scenes at test time. In addition, we propose a coarse-to-fine inference strategy in order to produce high-resolution output while also leveraging large input context sizes. In an extensive series of experiments, we carefully evaluate different model design choices, considering both deterministic and probabilistic models for completion and semantic inference. Our results show that we outperform other methods not only in the size of the environments handled and processing efficiency, but also with regard to completion quality and semantic segmentation performance by a significant margin.
1. Introduction
ScanComplete addresses incomplete, low-quality indoor 3D reconstructions by completing missing geometry and semantic labels across large scenes. Its fully convolutional, coarse-to-fine design handles variable spatial extents while balancing global context and local detail.
- Motivation: RGB-D scans remain incomplete because occlusions and range-sensor limitations leave holes that hinder downstream processing.Traditional methods can fill small holes, but high-level geometry such as walls and chair legs is more challenging.
- Motivation: Volumetric representations grow cubically with scene size, limiting resolution, while indoor scenes are large and irregularly shaped.
- Approach: ScanComplete uses fully convolutional networks trained on subvolumes and applied to arbitrarily sized environments for scene completion and per-voxel semantic inference.The paper reports examples reaching 1480×1230×64 voxels, approximately 70×60×3m.
- Approach: A coarse-to-fine hierarchy combines low-resolution predictions with large context and higher-resolution predictions with smaller context.
- Results: Evaluations show completion and semantic labeling at unprecedented spatial extents, including transfer from synthetic training data to real commodity RGB-D scans.
- Results: The contributions comprise arbitrary-extent fully convolutional completion, coarse-to-fine prediction, and scene completion plus semantic labeling outperforming existing methods by significant margins.
2. Related Work
Prior work spans local geometric hole filling, database and structural priors, classical structured prediction, and deep 3D generative models. These approaches establish useful foundations but commonly target objects, fixed volumes, or limited completion settings.
- Geometric completion: Traditional geometry-processing methods fill small holes using local surface primitives or continuous energy minimization.Poisson Surface Reconstruction is an example of fitting a surface to point-cloud observations.
- Shape priors: Other methods exploit mesh or point-cloud symmetries, part-based priors, or aligned CAD models retrieved from databases.Database-retrieval approaches assume exact object matches, although nonrigid registration can relax that assumption.
- Data-driven prediction: Voxlets applies data-driven structured prediction with a random decision forest to predict unknown voxel neighborhoods.
- Deep learning: Deep 3D methods use convolutional or belief-network models for shape generation, completion, classification, and broken-mesh repair.
- Deep learning: Hierarchical CNNs and dilated convolutions extend 3D processing toward higher resolution, larger context, missing-voxel prediction, and semantic inference.The cited approaches nonetheless operate on a fixed-size setting in the supplied passage.
3. Method Overview
ScanComplete converts partial volumetric scans into complete geometry and semantic labels using a size-invariant fully convolutional network. A coarse-to-fine autoregressive hierarchy combines large context with high-resolution local refinement.
- Inputs and outputs: The method inputs a partial TSDF and predicts a complete TDF, trained against a TDF generated from a complete ground-truth mesh.
- Network design: Its fully convolutional architecture is invariant to spatial extent, allowing training on random crops and testing on different scene sizes.
- Coarse-to-fine inference: The hierarchy first predicts low-resolution geometry and semantics with large context, then refines them at higher resolution using smaller contexts and previous outputs.Three levels range from several meters of coarse context toward fine-scale voxel resolution.
- Autoregressive prediction: At each level, an autoregressive model predicts eight interleaved voxel groups sequentially, conditioning each group on preceding predictions.Voxels within a group do not neighbor one another, and the model uses separate networks for the groups.
- Training objectives: The method evaluates deterministic ℓ1 and probabilistic classification losses for predicting completion fields.The deterministic formulation focuses on a single mode when the partial scan supports one explanation.
- Semantic inference: The model jointly predicts semantic labels with completion, using the same fully convolutional autoregressive and coarse-to-fine architecture.The results demonstrate that completion greatly helps semantic inference.
4. Data Generation
Training data pairs partial TSDF scans with complete TDFs and semantic labels generated from synthetic scenes. Virtual scanning trajectories, volumetric fusion, and multilevel subvolume sampling approximate commodity RGB-D acquisition.
- Synthetic scanning: Training pairs are generated from SUNCG scenes by virtually scanning depth along trajectories modeled from ScanNet statistics.The dataset uses 5359 training scenes and 155 test scenes from a prior train-test split.
- Trajectory generation: Camera candidates are sampled by height and angle, then selected within room regions using depth-histogram similarity and object coverage.Similarity is measured with Earth Mover’s Distance to the average ScanNet depth histogram.
- Volumetric representation: Rendered depth views are volumetrically fused into dense grids storing truncated signed distances, repeated independently at three voxel sizes.The voxel sizes are 4.7cm3, 9.4cm3, and 18.8cm3, with truncation set to three times voxel size.
- Ground truth generation: Complete SUNCG meshes produce target TDFs containing truncated unsigned distances and the closest object’s semantic label at each voxel.
- Training samples: Subvolumes are sampled every 3m, retaining regions with non-structural objects and usually discarding structural-only regions.This produces 225,414 training subvolumes across the three hierarchy levels.
- Field conventions: Input TSDF signs encode known and unknown space, whereas unsigned TDF targets are used because all ground-truth voxels are known.
5. ScanComplete Network Architecture
ScanComplete uses a fully convolutional 3D architecture that accepts varying spatial extents, combines hierarchical context with autoregressive voxel-group prediction, and jointly predicts geometry and semantics.
- The fully convolutional 3D architecture is invariant to input size, enabling training on random crops and testing on different spatial extents.
- At each hierarchy level, the network combines the partial TSDF scan with previous low-resolution and voxel-group predictions before producing geometry and semantic outputs.
- Training uses ground-truth previous voxel-group inputs but predicted previous-hierarchy outputs, avoiding over-smoothed final volumes observed with ground-truth hierarchy inputs.
- The implementation trains 24 networks—eight voxel groups across three hierarchy levels—with parallel training within each level taking approximately three days overall.
6. Results and Evaluation
Experiments evaluate architectural variants, competing completion methods, real-scan transfer, and semantic labeling. The full autoregressive hierarchy achieves the strongest completion accuracy, while joint completion improves semantic inference and supports efficient large-scene processing.
- Completion Evaluation on SUNCG: Reducing quantization bins from 256 to 32 improves completion, while further reduction weakens the discrete approximation of the continuous distance field.
- Completion Evaluation on SUNCG: An autoregressive, deterministic three-level hierarchy with the largest spatial context provides the highest completion accuracy.
- Completion Evaluation on SUNCG: ScanComplete achieves lower reconstruction error than Poisson Surface Reconstruction, 3D-EPN, and SSCNet on synthetic SUNCG data.
- Completion Evaluation on SUNCG: O(1) forward passes at test time make ScanComplete more efficient than fixed-subvolume methods that typically require O(wd) predictions for a w × h × d scene.
- Completion Results on ScanNet (real data): A model trained only on synthetic data generalizes to qualitative completion of real RGB-D scans from ScanNet.
- Semantic Inference on SUNCG: Joint completion and semantic prediction significantly outperform semantic-only and current state-of-the-art methods for SUNCG voxel labeling.
- Semantic Inference on SUNCG: Processing entire scenes and jointly completing geometry produces more globally consistent and accurate voxel labels than fixed-subvolume alternatives.
7. Conclusion and Future Work
ScanComplete predicts completed geometry and semantic voxel labels for entire scenes using fully convolutional, coarse-to-fine inference. The authors report strong completion and segmentation results, while identifying output resolution as an important remaining limitation.
- ScanComplete predicts completed geometry and semantic voxel labels for an entire scene at once.
- A fully convolutional network decouples training and testing resolutions, enabling variably sized test scenes with unbounded spatial extents.
- Coarse-to-fine prediction with a volumetric autoregressive network combines large spatial context with local detail.
- The final output resolution of ∼5cm3 voxels remains insufficient to resolve fine-scale objects such as cups.
A. Qualitative Evaluation Real Data
The qualitative real-data evaluation applies a network trained only on synthetic SUNCG to complete real RGB-D scans and infer semantic labels.
- A network trained only on synthetic SUNCG infers missing geometry and per-voxel semantics in real-world ScanNet RGB-D scans.
B. Quantitative Evaluation on Real Data
The real-data evaluation compares semantic predictions on ScanNet and Matterport3D and examines completion artifacts from independently processed blocks. Results emphasize real-data benchmarking and the benefit of whole-scene processing for avoiding seams.
- Semantic predictions are comprehensively compared on ScanNet and Matterport3D using ground-truth per-voxel annotations.
- Independent fixed-size block predictions can create seams at block boundaries because neighboring-block information is not considered.
- Boundary artifacts remain problematic under visual inspection even when a baseline’s quantitative error metrics are not too bad.
- Whole-scene processing effectively alleviates seams caused by inconsistent independent subvolume predictions.
D. Training Block Pairs
The method trains fully convolutional networks on randomly sampled subvolume pairs across three hierarchy levels, supporting varying test extents and whole-scene processing. Runtime comparisons highlight the efficiency of processing complete scenes in each pass rather than voxel columns or independent blocks.
- Training Block Pairs: Randomly selected subvolume ground-truth pairs across three hierarchy levels train the network to generalize to varying spatial extents at test time.
- Training Block Pairs: The fully convolutional architecture processes arbitrarily sized 3D environments in a single test pass.
- Timings: The method requires 3 × 8 forward passes, each run over the entire scene, because it uses three hierarchy levels and eight autoregressive voxel groups.
- Timings: ScanNet’s semantic voxel labeling method runs on a per-voxel-column basis, making runtime significantly slower despite using a less powerful network.
- Quantitative Evaluation on Real Data: Table 4 reports per-voxel class accuracies on Matterport3D and ScanNet test scenes, with significant improvement in average class accuracy on Matterport3D.
- Timings: The adapted 3D-EPN baseline processes independent 32 × 32 × 64 blocks and requires as many forward passes as voxel blocks.
F. Additional Results on Completion and Semantics on SUNCG
This section presents additional qualitative completion and semantic prediction results on SUNCG, including whole scenes and close-ups across challenging scenarios.
- SUNCG results include both complete-scene views and close-ups covering a variety of challenging scenarios.
- Table 5 reports test-scene evaluation time in seconds for scenes of various sizes on a GTX 1080.
- Table 6 reports SUNCG semantic-labeling performance as per-class IOU over the visible surface of partial test scans.