Source-linked AI summary
SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
Angela Dai, Christian Diller, Matthias Nießner
TL;DR
RGB-D scans are inherently incomplete, while existing scene-completion methods depend on fully complete synthetic ground truth. SG-NN learns completion self-supervised from incomplete real-world scans using a sparse generative network, outperforming supervised state-of-the-art methods at 2cm resolution.
Problem
RGB-D scans remain incomplete because occlusions leave regions unobserved, while supervised completion methods require complete synthetic ground truth and face real-world domain-transfer limitations.
Method
SG-NN removes frames from incomplete scans to create self-supervision and uses a progressively growing sparse generative network to predict geometry coarse-to-fine.
Results
2cm resolution enables SG-NN to outperform state-of-the-art fully supervised scan-completion methods while training only on incomplete real-world scans.
Takeaways & Limitations
Self-supervised sparse generation supports high-resolution scene completion from real-world observations without requiring fully complete training scans.
Takeaways & Limitations
Existing supervised scene-completion methods require fully complete synthetic ground truth, limiting their transfer to real-world scans.
Abstract
from arXiv · showhide
We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry. Our approach is fully self-supervised and can hence be trained solely on real-world, incomplete scans. To achieve self-supervision, we remove frames from a given (incomplete) 3D scan in order to make it even more incomplete; self-supervision is then formulated by correlating the two levels of partialness of the same scan while masking out regions that have never been observed. Through generalization across a large training set, we can then predict 3D scene completion without ever seeing any 3D scan of entirely complete geometry. Combined with a new 3D sparse generative neural network architecture, our method is able to predict highly-detailed surfaces in a coarse-to-fine hierarchical fashion, generating 3D scenes at 2cm resolution, more than twice the resolution of existing state-of-the-art methods as well as outperforming them by a significant margin in reconstruction quality.
1. Introduction
RGB-D scans remain incomplete because physical occlusions limit the scanning process, while existing data-driven completion methods require fully complete ground-truth models and supervised datasets. The paper addresses these limitations with self-supervised learning from partial real-world scans and a sparse, progressively growing architecture for high-resolution scene completion.
- Motivation: Physical occlusions leave RGB-D reconstructions incomplete despite progress in indoor tracking and reconstruction.The limitation arises from the physical constraints of the scanning process, even during careful scans.
- Limitations of Existing Methods: Existing data-driven scene completion methods require fully complete ground-truth 3D models and large-scale synthetic datasets for supervised training.The passage identifies ShapeNet and SUNCG as examples of such synthetic datasets.
- Self-Supervised Completion: The proposed self-supervised formulation trains only on partial real-world scans by removing frames to create a more incomplete input and masking unknown regions.The original scan serves as the target while the frame-removed scan serves as input.
- Self-Supervised Completion: The method learns to predict geometry more complete than any geometry seen during training by leveraging common patterns in incompleteness differences.This is enabled by correlating different levels of partialness within the same scan.
- Sparse Generative Architecture: A sparse generative neural network progressively generates high-resolution geometry with sparse convolutions in a fully convolutional, coarse-to-fine hierarchy.The formulation produces a sparse truncated signed distance function representation and increases surface resolution during training.
2. Related Work
Prior RGB-D reconstruction methods achieve strong tracking and scalability but remain limited by incomplete scans, while learned completion methods increasingly address larger missing regions using 3D representations whose scalability remains challenging.
- RGB-D Reconstruction: Commodity range-sensor reconstruction builds on volumetric fusion with truncated signed distance fields, supporting strong tracking and scalable 3D reconstruction.These methods established a foundation for real-time reconstruction from RGB-D scans.
- RGB-D Reconstruction: Incomplete scans remain a major limitation because occlusions and unobserved regions prevent reconstructions from matching manually created 3D assets.The resulting 3D representations cannot fully recover the quality of complete geometry.
- Deep Learning on 3D Scans: Deep learning methods apply convolutional operators to regular 3D grids, while sparse representations reduce the computational and memory costs of dense volumetric processing.Common grid representations include distance fields and occupancy grids, and sparse approaches have been used for discriminative 3D tasks.
- Shape and Scene Completion: Traditional completion methods fit surfaces locally and work well for small missing regions, whereas deep networks learn global shape structure for larger object gaps.Learned approaches have also extended completion to scene-scale scans, including rooms and building floors.
- Shape and Scene Completion: SSCNet and ScanComplete perform large-scale scene completion but operate on dense volumetric grids, while single implicit latent spaces remain difficult to scale to varying-sized environments.DeepSDF and Occupancy Networks exemplify the single-latent-space alternative.
3. Method Overview
SG-NN performs self-supervised completion of incomplete RGB-D scans by learning from deliberately reduced observations and generating sparse, high-resolution scene geometry.
- Representation: The method represents input and completed scenes as sparse truncated signed distance fields (TSDFs), from which a mesh can be extracted.Each TSDF stores voxel locations within truncation and corresponding distance values.
- Self-supervision: Self-supervision trains the model to map existing incomplete scans to more incomplete observations created by removing frames.This avoids requiring fully complete ground-truth scenes and reflects real-world incompleteness from occlusions and sensor limitations.
- Self-supervision: The training objective learns completion where scan geometry is known while masking regions of unobserved space.Across data, the generative model can produce more complete scenes than any individual target sample contains.
- SG-NN architecture: Sparse Generative Neural Networks generate new, unseen 3D geometry as a sparse surface representation using sparse convolutions.Unlike discriminative 3D tasks that receive geometric structure as input, SG-NN is designed to generate it.
4. Self-Supervised Completion
Self-supervision trains completion from a more incomplete scan to a less incomplete target while masking unobserved target regions. The method uses 2cm TSDF voxels, randomly removes about 50% of frames, and trains efficiently on cropped scan views.
- Self-supervision: Self-supervision removes depth frames from an incomplete RGB-D scan, using the resulting more incomplete scan as input and the original scan as target.The paired scans preserve real-world scanning characteristics while representing different levels of partialness.
- Self-supervision: The completion loss is evaluated only where Starget(v) > −τ, preventing unobserved target regions from producing contradictory supervision.This masking enables learning geometry beyond the completeness of the target scan, including occluded regions.
- Implementation: Target scans use volumetric fusion with 2cm voxels and truncation of 3 voxels, while approximately 50% of frames are randomly removed to form inputs.These settings construct the target and input TSDFs from the same RGB-D scan at different completeness levels.
- Implementation: Training uses random 64×64×128-voxel crops, while the fully convolutional model supports inference on full scenes of varying sizes.Experiments use real-world Matterport3D RGB-D scans with annotated room regions.
5. Generating a Sparse 3D Scene Representation
SG-NN hierarchically generates sparse TSDF scene representations from partial scans using sparse encoding, dense coarse prediction, and coarse-to-fine refinement. Sparse skip connections, progressive training, and multilevel losses support the completion process.
- Sparse 3D Scene Representation: SG-NN hierarchically generates a sparse, truncated signed distance field representation, from which the final mesh is extracted as an isosurface.The architecture targets the sparsity of 3D scene geometry.
- Network Architecture: Sparse 3D convolutions encode TSDF values only at locations within truncation distance, compressing spatial dimensions by a factor of two per convolution set.The encoded features are converted into a low-resolution dense grid for full-scene geometry prediction.
- Coarse-to-Fine Generation: The model converts coarse predictions back to sparse representations and progressively upsamples them by a factor of two to generate higher-resolution geometry.Each hierarchy level predicts occupancy and TSDF values for the next level.
- Sparse Skip Connections: Sparse skip connections concatenate encoder and decoder features at shared sparse locations, using zero features where destination locations lack corresponding sources.This adapts UNet-style connections to sparse feature maps with different active locations.
- Loss: Training uses a log-transformed TSDF L1 loss at final predicted locations, supplemented by proxy losses for occupancy and TSDF outputs at each hierarchy level.The log transform emphasizes accurate prediction near surface geometry.
6. Results and Evaluation
SG-NN is evaluated on real-world and synthetic RGB-D scan completion, where it outperforms prior methods in both self-supervised and fully supervised settings. Ablations show benefits from masked self-supervision and TSDF representations, while performance remains robust to less-complete training data.
- Comparison to state of the art: SG-NN outperforms fully supervised 3D-EPN and ScanComplete on real-world Matterport3D scans despite training solely on real-world data.The baselines require fully complete ground truth and are therefore trained on synthetic scans.
- Synthetic scan completion: SG-NN also outperforms state of the art in fully supervised synthetic scan completion.Synthetic evaluation uses complete ground truth and compares against Poisson Surface Reconstruction, SSCNet, 3D-EPN, and ScanComplete at 5cm resolution.
- Self-supervision ablation: Masked self-supervision enables completion in commonly occluded regions, whereas removing the masking often fails there.Without masking, completion remains effective mainly in regions commonly observed in target scans.
- Representation ablation: TSDF input and output representations improve performance over point-cloud input and occupancy output because they describe surfaces and neighboring regions more expressively.The comparison appears in the real-world ablation study at 2cm voxel resolution.
- Training-data completeness: 30%, 40%, 50%, 60%, and 100% frame-based training targets leave the self-supervised approach robust when evaluated on 50% incomplete scans.Training configurations include 50%−all, 40%−60%, and 30%−50% input-target completeness.
- Limitations: SG-NN predicts geometry but not full scene appearance or semantic object decomposition.Color generation and inpainting remain necessary for complete appearance, while semantic decomposition is identified as future work.
7. Conclusion
The paper presents a self-supervised method that learns generative geometric completion solely from incomplete, real-world RGB-D scans. Its sparse representation enables higher-resolution geometry and suggests broader real-world-observation-based 3D modeling without extensive synthetic data or domain transfer.
- Self-supervised completion: The method trains solely on incomplete, real-world scans while learning a generative process for RGB-D scan geometry completion.It predicts 3D scene geometry more complete than any single target scene observed during training.
- Generative completion: The generative completion process predicts 3D scene geometry more complete than any single target scene seen during training.This capability follows from learning across incomplete real-world scans rather than complete geometric targets.
- Sparse representation: The sparse generative approach produces sparse TSDF representations at much higher output geometric resolution than previous methods on large-scale 3D scenes.The representation is positioned as a basis for higher-resolution scene completion.
- Future directions: Real-world-only self-supervision opens possibilities for generative 3D modeling from observations, potentially reducing reliance on extensive synthetic data generation or domain transfer.The authors identify this direction as promising future research.
A. SG-NN Architecture Details
The SG-NN architecture specification defines convolution parameters and distinguishes concatenation from addition, with batch normalization and ReLU following each convolution except the last.
- A. SG-NN Architecture Details: Convolution parameters are specified as (nf in, nf out, kernel size, stride, padding), with stride and padding defaulting to 1 and 0.Arrows indicate concatenation, while L indicates addition.
- A. SG-NN Architecture Details: Each convolution except the last is followed by batch normalization and a ReLU.
B. Varying Target Data Incompleteness
The evaluation tests self-supervised completion as the target scans used during training become less complete. Across varying frame-retention levels, the approach maintains robust geometric completion when training data contains sufficient variety in completion patterns.
- B. Varying Target Data Incompleteness: The study evaluates self-supervision under decreasing target-data completeness, relying on variety in training completion patterns to learn highly complete scene geometry.Target scans are generated from Matterport3D room scenes with varying completeness.
- B. Varying Target Data Incompleteness: Target scans retain ≈50%, 60%, or 100% of room-scene frames, while corresponding input scans retain ≈30%, 40%, or 50%.The evaluation includes quantitative results in the main paper and qualitative results in Figure 9.
- B. Varying Target Data Incompleteness: Even as target-data completeness decreases, the approach maintains robustness in scene completion.Figure 9 evaluates 50% incomplete scans after training with incomplete versions retaining ≈30%, 40%, 50%, 60%, and 100% of frames.