Source-linked AI summary
Occupancy Networks: Learning 3D Reconstruction in Function Space
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger
TL;DR
Existing 3D reconstruction representations struggle to combine memory efficiency, arbitrary topology, and high-resolution geometry. Occupancy Networks learn a continuous occupancy function whose decision boundary defines the surface, achieving high-quality meshes across varied inputs and tasks.
Problem
Existing 3D representations lack a consensus solution that is memory efficient, data-inferable, high-resolution, and capable of arbitrary topology.
Method
Occupancy Networks learn a continuous occupancy function with a neural classifier, whose decision boundary implicitly represents the 3D surface at arbitrary resolution.
Results
The approach generates high-quality meshes, compares favorably with state-of-the-art methods, and works effectively for supervised and unsupervised learning across varied 3D inputs.
Takeaways & Limitations
Occupancy Networks provide an expressive representation for realistic high-resolution meshes and a useful tool across a wide variety of 3D tasks.
Takeaways & Limitations
Inference requires an initial occupancy-grid resolution high enough to contain points from every connected interior and exterior mesh component.
Abstract
from arXiv · showhide
With the advent of deep neural networks, learning-based approaches for 3D reconstruction have gained popularity. However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometry of arbitrary topology. Many of the state-of-the-art learning-based 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricted domain. In this paper, we propose Occupancy Networks, a new representation for learning-based 3D reconstruction methods. Occupancy networks implicitly represent the 3D surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches, our representation encodes a description of the 3D output at infinite resolution without excessive memory footprint. We validate that our representation can efficiently encode 3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both qualitatively and quantitatively, for the challenging tasks of 3D reconstruction from single images, noisy point clouds and coarse discrete voxel grids. We believe that occupancy networks will become a useful tool in a wide variety of learning-based 3D tasks.
1. Introduction
The introduction identifies the lack of a memory-efficient, efficiently inferable 3D representation that supports arbitrary topology and high-resolution geometry. It proposes learning a continuous occupancy function with a neural network, enabling arbitrary-resolution evaluation and mesh extraction for multiple input types.
- Motivation: Learning-based 3D reconstruction is increasingly popular, but the field lacks a memory-efficient representation that can be efficiently inferred from data.Learned models encode prior information about 3D shapes, helping resolve ambiguities in input.
- Limitations of existing representations: Template-based mesh representations do not support arbitrary topologies, while point- and mesh-based approaches limit the number of reliably predicted points or vertices.Existing representations include voxel-, point-, and mesh-based categories.
- Limitations of existing representations: Voxel representations have cubic memory growth, limiting naive implementations to 32^3 or 64^3 voxels and data-adaptive methods to relatively small 256^3 grids.Octrees reduce memory but introduce implementation complexity.
- Proposed approach: The paper introduces continuous 3D occupancy learning with a neural network fθ, replacing fixed-resolution voxel prediction with an occupancy function evaluable at arbitrary resolution.This approach reduces the memory footprint during training.
- Contributions: The representation supports 3D reconstruction from various input types and is experimentally shown to generate high-quality meshes that compare favorably with the state of the art.At inference, meshes are extracted from the learned model using multi-resolution isosurface extraction.
2. Related Work
Related 3D reconstruction methods are organized around voxel, point-cloud, and mesh output representations. The paper contrasts their computational or geometric limitations with an approach producing expressive, high-resolution closed surfaces through deep learning.
- Representation categories: Learning-based 3D reconstruction methods broadly use voxel-based, point-based, or mesh-based output representations.These categories organize the related approaches discussed in the paper.
- Voxel representations: Voxel methods are simple and widely used, but memory demands restrict resolution and can cause slow training.Earlier methods were limited to 32^3 grids; newer methods reached 128^3 only with shallow networks and small batch sizes, while multi-resolution methods remained limited to 256^3 grids.
- Point representations: Point clouds provide an alternative representation widely used in robotics and computer graphics, with permutation invariance achieved through point-wise processing and global pooling.Qi et al. pioneered point clouds for discriminative deep learning tasks using this architecture.
- Mesh representations: Mesh-based reconstruction methods can generate self-intersections, support only simple topology, require class-specific templates, or fail to guarantee closed surfaces.These limitations affect different existing mesh approaches rather than necessarily all methods simultaneously.
- Occupancy Networks: The proposed approach produces high-resolution closed surfaces without self-intersections or class-specific template meshes, using deep learning rather than solving a differential equation.The representation is related to classical level-set approaches while integrating naturally into an end-to-end learning pipeline.
3. Method
The method represents 3D geometry as a continuous occupancy function modeled by a conditioned neural network, whose decision boundary implicitly defines the object surface. It trains this representation from sampled 3D points and extracts high-resolution meshes efficiently with hierarchical isosurface extraction and refinement.
- Occupancy representation: Occupancy Networks model occupancy at every 3D point p ∈ R3 with a neural network that outputs probabilities, using its binary-classification decision boundary as the implicit surface.The network is conditioned on an observation x by taking (p, x) as input and is called the Occupancy Network.
- Learning: Training samples K points within each object’s 3D bounding volume and minimizes cross-entropy between predicted and true occupancies.Uniform sampling inside the padded bounding box performed best in practice.
- Mesh extraction: MISE incrementally builds an octree, evaluates occupancy only where neighboring grid predictions differ, and applies Marching Cubes to extract the isosurface.This avoids densely evaluating every point in a high-dimensional occupancy grid.
- Mesh extraction: MISE converges to the correct mesh when the initial grid samples every connected interior and exterior component, with an initial resolution of 32^3 sufficient in almost all cases.The occupancy threshold τ controls extracted surface thickness and is cross-validated on a validation set.
- Mesh refinement: The extracted mesh is simplified and gradient-refined to remove Marching Cubes discretization artifacts, while vertex normals are obtained by backpropagation through the occupancy network.Inference requires 3s per mesh, and the refinement is unavailable for directly predicted voxel representations.
4. Experiments
The experiments evaluate occupancy networks’ representation power and reconstruction from images, noisy point clouds, and coarse voxel inputs using IoU, Chamfer-L1 distance, and normal consistency. Across these settings, the method represents detailed, complex-topology geometry efficiently and improves over coarse or competing representations.
- Experimental design: Experiments assess representation power, image-conditioned reconstruction, noisy point-cloud reconstruction, and reconstruction from low-resolution voxel representations.The representation-power experiment examines reconstruction from learned latent embeddings as an upper bound for conditioning on additional inputs.
- Evaluation: Evaluations use volumetric IoU, Chamfer-L1 distance, and normal consistency, with mesh comparisons based on watertight ground-truth meshes.IoU uses 100k sampled points, while Chamfer-L1 uses samples from both meshes and KD-tree nearest-neighbor estimates.
- Representation power: 0.89 mean IoU: the occupancy network faithfully represents the entire dataset while encoding all 4746 training samples with as little as 6M parameters.The comparison is against voxel representations at various resolutions, which cannot accurately represent the meshes at low resolution.
- Single-image reconstruction: For single-image reconstruction, the method captures complex topologies, produces closed meshes, preserves details, and achieves the highest IoU and normal consistency.Competing methods exhibit coarse geometry, missing connectivity, or missed holes; PSGN and AtlasNet do not yield watertight meshes, preventing IoU evaluation.
- Point-cloud reconstruction: For noisy point clouds, the method achieves the highest IoU and normal consistency and the lowest Chamfer-L1 distance.All metrics are significantly better than in single-image reconstruction because point-cloud inputs reduce ambiguity and require the model to fill gaps.
- Coarse-input reconstruction and analysis: Against coarse input meshes, the model considerably improves IoU, Chamfer-L1 distance, and normal consistency.The paper also reports compelling new models from latent-space sampling, while uniform sampling performs best and reducing samples from 2048 to 64 retains good performance.
5. Conclusion
The paper introduced occupancy networks as a representation for 3D geometry that avoids 3D-space discretization and supports realistic high-resolution meshes. Experiments show that they are expressive, effective for supervised and unsupervised learning, and applicable to diverse 3D tasks.
- 5. Conclusion: Occupancy networks represent 3D geometry without being constrained by 3D-space discretization, enabling realistic high-resolution meshes.They are presented as a new representation for 3D geometry.
- 5. Conclusion: Experiments demonstrate that occupancy networks are expressive and effective for both supervised and unsupervised learning.The authors therefore view them as useful for a wide variety of 3D tasks.