Source-linked AI summary
Convolutional Occupancy Networks
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, Andreas Geiger
TL;DR
Existing implicit 3D reconstruction methods are limited in handling local information and scaling beyond simple single objects. This paper introduces Convolutional Occupancy Networks, combining convolutional encoders with implicit occupancy decoders. The resulting representation supports fine-grained object reconstruction, large indoor scenes, and generalization across synthetic and real data.
Problem
Existing implicit methods are limited to simple single-object geometry and large-scale scenes because fully connected architectures lack local feature integration and translation-equivariant inductive biases.
Method
Convolutional Occupancy Networks combine convolutional encoders with implicit occupancy decoders to integrate local and global 3D information.
Results
The representation enables fine-grained reconstruction of objects, scales to large indoor scenes, and generalizes from synthetic to real data and novel categories and layouts.
Takeaways & Limitations
Convolutional feature representations make implicit 3D reconstruction applicable from single objects to large-scale indoor spaces.
Takeaways & Limitations
The method is not rotation equivariant, is translation equivariant only for translations that are multiples of the voxel size, and retains a synthetic-to-real performance gap.
Abstract
from arXiv · showhide
Recently, implicit neural representations have gained popularity for learning-based 3D reconstruction. While demonstrating promising results, most implicit approaches are limited to comparably simple geometry of single objects and do not scale to more complicated or large-scale scenes. The key limiting factor of implicit methods is their simple fully-connected network architecture which does not allow for integrating local information in the observations or incorporating inductive biases such as translational equivariance. In this paper, we propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We investigate the effectiveness of the proposed representation by reconstructing complex geometry from noisy point clouds and low-resolution voxel representations. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
1 Introduction
Existing implicit methods represent geometry continuously but remain limited by fully connected architectures that lack local observation features and translation-equivariant inductive biases. Convolutional Occupancy Networks combine convolutional features with implicit decoding to reconstruct detailed objects and large indoor scenes.
- 3D reconstruction representations should capture arbitrary geometries and topologies, scale to large scenes, integrate local and global information, and remain computationally tractable.
- Volumetric, point-cloud, and mesh representations each face limitations involving memory, topology, point capacity, or neural-network prediction.
- Implicit representations provide continuous geometry without topology restrictions, but existing approaches largely remain limited to single objects and do not scale to large scenes.
- Fully connected implicit models cannot integrate local observations or incorporate translation equivariance, limiting structured reasoning and producing overly smooth reconstructions.
- Convolutional Occupancy Networks combine convolutional neural networks with implicit representations to support accurate, large-scale 3D reconstruction using local and global information.
- The model enables reconstruction from object to scene level and generalizes from synthetic to real data, unseen object categories, novel room layouts, and scenes.
2 Related Work
Prior 3D reconstruction methods use voxel, point-cloud, mesh, or implicit representations, each with distinct limitations. Convolutional Occupancy Networks move feature aggregation into physical 3D space to support viewpoint-independent scene-level reconstruction.
- Voxels: Voxel methods face cubic memory and computation costs, even when using multiscale or octree data structures.
- Point Clouds: Point clouds are lightweight but limited in the number of points they handle and cannot represent topological relations.
- Meshes: Mesh-based neural reconstruction can require fixed-topology templates or produce non-watertight, self-intersecting meshes.
- Implicit Representations: Implicit models represent shapes continuously, naturally handle complicated topologies, and predict occupancy or distance values at arbitrary 3D points.
- Implicit Representations: Existing implicit approaches remain limited to simple single-object geometry and large-scene scaling because fully connected networks lack local feature integration and translation-equivariant inductive biases.
- Convolutional Occupancy Networks: Unlike image-based methods restricted to single objects, this approach aggregates features in physical 3D space using 2D and 3D convolutions, independent of viewpoint and input representation.
3 Method
The method encodes 3D inputs into planar or volumetric feature grids, processes them with convolutional U-Nets, and uses interpolated features to predict occupancy at query points. Planar encodings offer higher resolution, while volumetric encodings represent 3D structure more directly.
- Encoder: The encoder converts point clouds or coarse voxel grids into 2D or 3D feature grids using task-specific networks and pooling.
- Occupancy Prediction: For a query point p, bilinear or trilinear interpolation obtains ψ(x, p), which a fully connected network maps to an occupancy probability.
- Encoder: Planar encoding orthographically projects input points onto canonical planes and averages features within H × W cells.
- Encoder: The model evaluates ground-plane, three-plane, volumetric, and combined representations; three planes recover richer z-dimensional structure but require more computation.
- Decoder: 2D and 3D U-Nets process feature planes or volumes with downsampling, upsampling, and skip connections to integrate local and global information.
- Decoder: Convolutional processing makes output features translation equivariant and can inpaint sparse inputs while preserving global information.
- Training and Inference: Training samples query points uniformly and minimizes binary cross-entropy between predicted and true occupancy values; inference extracts meshes with MISE.
- Training and Inference: The fully convolutional design supports sliding-window inference for reconstructing entire apartments and other large scenes.
4 Experiments
Experiments evaluate Convolutional Occupancy Networks on object- and scene-level reconstruction from noisy point clouds and coarse voxel inputs. The method achieves strong reconstruction quality, supports large indoor scenes, and generalizes from synthetic training data to real scenes.
- Experimental Setup: The experiments cover ShapeNet object reconstruction, synthetic indoor scene reconstruction, and synthetic-to-real evaluation on ScanNet and Matterport3D.Inputs include noisy point clouds and low-resolution voxel grids; baselines include ONet, PointConv, and SPSR.
- Object-Level Reconstruction: Convolutional variants match or outperform baselines on all three ShapeNet point-cloud metrics and converge faster during training.Projecting point features to planes or volumes followed by CNN processing is more effective than direct point-feature aggregation in PointConv.
- Object-Level Reconstruction: Three-plane features achieve comparable voxel super-resolution to volumetric features while using only 37% of the GPU memory.The single-plane variant fails because one plane may not resolve ambiguities in coarse, regularly structured voxel inputs.
- Scene-Level Reconstruction: On synthetic indoor scenes, all proposed variants reconstruct geometric details smoothly, while ONet and PointConv have low accuracy and SPSR produces noisy surfaces.Higher-resolution planes capture finer details but are more noise-prone; lower-resolution volumes are more robust but smoother.
- Scene-Level Reconstruction: Combining plane and volumetric features improves isolated-feature results, while volumetric features at 64^3 achieve the best result in this setting.The findings support complementarity between plane-based detail capture and volumetric noise robustness.
- Ablation and Real-World Evaluation: Multi-plane aggregation slightly outperforms single-plane and volumetric aggregation at similar GPU memory, and bilinear interpolation benefits the multi-plane variant.On ScanNet v2, volumetric models perform best, all variants outperform learning-based baselines, and the method reconstructs detailed scenes without additional hyperparameters.
5 Conclusion
Convolutional Occupancy Networks combine convolutional and implicit representations, with different feature layouts trading memory efficiency against real-world performance. The method remains limited in its equivariance properties and has a performance gap between synthetic and real data.
- Convolutional Occupancy Networks combine convolutional neural networks with implicit representations for 3D shape reconstruction.
- The 3-plane model is memory efficient, performs well on synthetic scenes, and supports larger feature resolutions.
- The volumetric model outperforms other variants on real-world scenarios but consumes more memory.
- The method is not rotation equivariant and is translation equivariant only for translations that are multiples of the defined voxel size.
- A performance gap remains between synthetic and real data, while future work targets implicit appearance modeling and 4D reconstruction.