Source-linked AI summary
OctNetFusion: Learning Depth Fusion from Data
Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, Andreas Geiger
TL;DR
Depth fusion must produce complete 3D geometry despite sensor noise, outliers, and occluded or missing surfaces. OctNetFusion learns an implicit surface and output octree from depth images, and experiments report stronger volumetric fusion than TSDF and TV-L1 plus state-of-the-art shape completion.
Problem
TSDF fusion needs many frames to suppress noise and outliers and cannot reconstruct occluded regions or large holes.
Method
OctNetFusion is a 3D CNN that predicts truncated signed distance fields and the output octree structure from one or more depth images.
Results
Experiments show improved volumetric fusion over vanilla TSDF and TV-L1, including noise and outlier reduction, missing-surface completion, and stronger single-view shape completion.
Takeaways & Limitations
Learning geometric structure from 3D data enables the method to reduce reconstruction artifacts and fill missing surfaces.
Takeaways & Limitations
The paper focuses on depth-map fusion; extending the model to RGB-based reconstruction is left for future work.
Abstract
from arXiv · showhide
In this paper, we present a learning based approach to depth fusion, i.e., dense 3D reconstruction from multiple depth images. The most common approach to depth fusion is based on averaging truncated signed distance functions, which was originally proposed by Curless and Levoy in 1996. While this method is simple and provides great results, it is not able to reconstruct (partially) occluded surfaces and requires a large number frames to filter out sensor noise and outliers. Motivated by the availability of large 3D model repositories and recent advances in deep learning, we present a novel 3D CNN architecture that learns to predict an implicit surface representation from the input depth maps. Our learning based method significantly outperforms the traditional volumetric fusion approach in terms of noise reduction and outlier suppression. By learning the structure of real world 3D objects and scenes, our approach is further able to reconstruct occluded regions and to fill in gaps in the reconstruction. We demonstrate that our learning based approach outperforms both vanilla TSDF fusion as well as TV-L1 fusion on the task of volumetric fusion. Further, we demonstrate state-of-the-art 3D shape completion results.
1. Introduction
OctNetFusion addresses the noise, outliers, missing data, and occlusions that limit conventional volumetric depth fusion. It learns both the reconstruction and its output space partitioning, improving fusion and completion results.
- 1. Introduction: Four uniformly spaced views illustrate noisy TSDF results, detail loss from TV-L1 smoothing, and improved noise reduction and surface completion from the proposed approach.The comparison highlights the trade-off between smoothing and preserving thin geometric details.
- 1. Introduction: TSDF fusion requires many frames to suppress sensor noise and outliers and cannot reconstruct occluded regions or large holes.These limitations motivate a learned alternative to conventional volumetric fusion.
- 1. Introduction: OctNetFusion learns volumetric fusion from noisy depth observations using large 3D model datasets and high-capacity 3D CNNs.The model is designed to smooth noise, handle outliers, and complete missing geometry.
- 1. Introduction: OctNetFusion estimates both the 3D reconstruction and its output octree structure, whose surface location is unknown from noisy input observations.It predicts truncated signed distance fields that can be meshed using standard techniques.
- 1. Introduction: Experiments on synthetic and real-world datasets show reduced noise and outliers, completed missing surfaces, and improved volumetric fusion over TSDF and TV-L1.The method also improves single-view volumetric shape completion over the reported state of the art.
2. Related Work
Prior fusion methods rely on averaging, local smoothness, or probabilistic constraints, while shape-completion methods use model fitting or voxel-level prediction. OctNetFusion instead learns large-resolution 3D structure directly from data and supports arbitrary input views.
- 2. Related Work: Traditional TSDF fusion averages truncated signed distance functions across viewpoints and is widely used because of its simplicity.Variational methods add local smoothness but are typically slow and unable to handle missing data.
- 2. Related Work: Ray-consistency methods model free-space and visibility constraints probabilistically but do not learn object and scene geometry from data.They rely on hand-crafted priors rather than learned geometric structure.
- 2. Related Work: Existing shape-completion methods fit known 3D models or predict voxel-level outputs, whereas this paper learns 3D structure end to end without requiring exact model priors.The proposed approach targets general scenes rather than only objects with known 3D shapes.
- 2. Related Work: OctNetFusion provides large-resolution reasoning, sub-voxel surface estimates, and support for an arbitrary number of input views.These properties distinguish it from voxel-level and single-image reconstruction approaches.
3. Method
OctNetFusion jointly learns 3D reconstruction and the output octree structure through a coarse-to-fine 3D CNN. It uses structure manipulation, multi-resolution losses, and depth-derived representations to reconstruct sparse, incomplete geometry.
- OctNet representation: The model represents inputs and outputs with grid-octrees, exploiting sparse 3D structure to reduce the memory burden of dense voxel CNNs.OctNets operate on grid-octrees whose cells can contain variable numbers of voxels, while conventional 3D CNN memory grows cubically with resolution.
- OctNetFusion: OctNetFusion jointly estimates the reconstruction quantity and its output space partitioning instead of requiring the octree structure in advance.The architecture predicts truncated signed distance fields or occupancy while adapting the grid-octree structure during inference.
- Network architecture: The coarse-to-fine network combines encoder-decoder modules, structure modules, and losses at every pyramid level as resolution doubles between levels.Encoder-decoder modules increase contextual information, while the structure module doubles feature-map resolution and supports intermediate reconstruction supervision.
- Structure module: Each structure module predicts a split mask that subdivides octree cells near the reconstructed surface and passes the resulting high-resolution structure to the next level.For TSDF regression, proximity is determined using the zero-level set and the truncation threshold; occupancy uses a tunable distance parameter.
- Input encoding: The input can be built from occupancy, traditional TSDF fusion, or a 10D distance histogram that preserves multimodal depth measurements and supports sub-voxel surface estimation.Occupancy is vulnerable to outliers and discards distances; TSDF preserves distances but omits uncertainty, while the histogram distributes observations across neighboring bins.
4. Evaluation
The evaluation compares OctNetFusion with volumetric baselines across synthetic and real scans, input encodings, view counts, noise levels, category generalization, runtime, and shape completion. Across these tests, the learned approach reduces reconstruction errors, handles missing geometry, and remains effective on unseen categories and real Kinect data.
- Evaluation scope: Experiments cover volumetric fusion on ModelNet40 and real Kinect object scans, plus single-image volumetric shape completion.The evaluation varies input and output representations, view counts, noise, category familiarity, and runtime.
- Input encodings: The TSDF histogram encoding yields the best results among the evaluated input encodings.OctNetFusion outperforms traditional volumetric fusion and TV-L1 by a large margin, especially at high resolutions.
- Number of views: With four input views, OctNetFusion reduces errors by a factor of 2 to 3 relative to TSDF and TV-L1 fusion.The largest baseline difference occurs with one view, indicating learned completion of missing geometry; performance improves as viewpoints increase.
- Input noise: MAD rises from 0.274 mm without noise to 0.374 mm under severe noise (σ = 0.03), while TSDF increases by more than 0.5 mm.TV-L1 fusion increases by more than 0.2 mm under the same comparison.
- Generalization: Performance decreases slightly on unseen categories but remains substantially better than TSDF and TV-L1 fusion.The unseen-category test trains on eight of ten categories and evaluates on the remaining two.
- Real scans and runtime: On real Kinect scans, OctNetFusion is consistently more accurate than traditional fusion and TV-L1 across resolutions and input-view counts.The relative advantage over traditional fusion grows at finer resolutions, while TV-L1 reduces errors more than vanilla volumetric fusion.
- Volumetric completion: For single-image volumetric completion, OctNetFusion improves IoU, precision, and recall over Voxlets and Zheng et al.The model is modified to predict binary occupancy maps for this task.
5. Conclusion
OctNetFusion is a deep 3D CNN for fusing depth information into accurate and complete reconstructions. Experiments show strong performance on synthetic and real Kinect data, while extending the model to RGB-based reconstruction remains future work.
- Conclusion: OctNetFusion fuses depth information from multiple viewpoints to produce accurate and complete 3D reconstructions.The method is a deep 3D convolutional neural network.
- Conclusion: The experiments show advantages over traditional fusion and compelling results on novel object categories in synthetic and real Kinect data.The conclusion summarizes the method's generalization and evaluation outcomes.
- Future work: The paper focuses on fusing depth maps, leaving reconstruction from RGB images as a direction for future work.The proposed extension would learn 3D representations jointly with 2D image representations end to end.