Source-linked AI summary
Local Implicit Grid Representations for 3D Scenes
Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser
TL;DR
Indoor scenes lack scalable, general shape priors because conventional 3D autoencoders struggle with their scale, complexity, and diversity. The paper learns local part-scale priors and places their decoded representations on an overlapping latent grid optimized against observations, achieving substantially better sparse-point reconstruction than alternatives.
Problem
Indoor scenes lack learned shape priors because conventional 3D autoencoders cannot handle their scale, complexity, or diversity.
Method
LIG learns an autoencoder for part-sized local regions and optimizes implicit feature codes on an overlapping regular grid to match partial or noisy observations.
Results
0.889 versus 0.455 F-Score is reported for Matterport reconstruction from 100/m2 input points, compared with the state-of-the-art approach.
Takeaways & Limitations
LIG generalizes learned object-part priors to unseen object classes and scalable scene reconstruction without scene-level training data.
Takeaways & Limitations
Assuming unoccupied grid cells are exterior works well without large enclosed volumes but creates artificial back-faces inside enclosed interiors.
Abstract
from arXiv · showhide
Shape priors learned from data are commonly used to reconstruct 3D objects from partial or noisy data. Yet no such shape priors are available for indoor scenes, since typical 3D autoencoders cannot handle their scale, complexity, or diversity. In this paper, we introduce Local Implicit Grid Representations, a new 3D shape representation designed for scalability and generality. The motivating idea is that most 3D surfaces share geometric details at some scale -- i.e., at a scale smaller than an entire object and larger than a small patch. We train an autoencoder to learn an embedding of local crops of 3D shapes at that size. Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation. We demonstrate the value of this proposed approach for 3D surface reconstruction from sparse point observations, showing significantly better results than alternative approaches.
1. Introduction
The paper introduces Local Implicit Grid representations, which learn geometric priors at an intermediate part scale to represent and reconstruct arbitrary scenes. The approach generalizes across object categories and improves sparse-point scene reconstruction over state-of-the-art methods.
- Motivation: An intermediate part scale provides shared geometric abstractions across globally different shapes while avoiding micro-scale planar approximations.Examples include curved sofa seats and car windshields, or thin sharp edges on tabletops and airplane wings.
- Representation: LIG represents scenes as regular grids of overlapping part-sized local regions encoded by implicit feature vectors.The representation uses learned part-level geometric features and associated latent-grid optimization methods.
- Method: The method learns a part autoencoder on 13 ShapeNet object categories, then optimizes overlapping latent-grid codes to match partial or noisy scene observations.The decoder is reused during reconstruction, avoiding scene-level training data.
- Generalization: LIG reconstructs shapes from novel object classes and entire scenes after training on chairs or ShapeNet parts, respectively.This demonstrates improved generalizability over methods that learn priors for entire objects.
- Results: 0.889 versus 0.455 F-Score is reported for Matterport reconstruction from 100/m2 input points, outperforming the state-of-the-art approach.The comparison concerns scene reconstruction from sparse point samples.
2. Related Work
Prior geometric representations and neural methods provide flexibility, implicit detail, or learned priors, but global and object-specific approaches struggle with localization, scalability, and transfer to scenes. LIG addresses these gaps with learned local part representations.
- Existing representations: Traditional point clouds, meshes, and voxel grids have been augmented with neural architectures to leverage data priors.These representations support geometric analysis or generation but do not by themselves resolve broad scene generalization.
- Implicit representations: Neural implicit methods represent continuous surfaces with spatial functions but typically encode an entire geometry using a global latent code.Their expressive capacity supports fine geometric details.
- Limitations of global codes: Global latent representations fail to capture localized details and scale poorly as scenes become larger and more complex.Hybrid local-global approaches have not yet demonstrated scalability beyond single objects.
- Local representations: Existing local patch methods either do not learn shape priors from 3D datasets or limit shape expressibility by using patches extracted from examples.These alternatives motivate learned local representations.
- Scene reconstruction: Most deep-learning reconstruction studies target objects, while scene-level methods require reliable, high-quality, domain-specific scene data that can be costly to acquire.LIG instead generalizes object-dataset shape priors to scenes without additional scene-level training.
3. Methods
The method learns local part embeddings with an implicit autoencoder, then arranges their latent codes in overlapping grids and optimizes them to reconstruct observed geometry. Sparse oriented samples supervise latent optimization, while interpolation and zero-contour extraction produce continuous surfaces.
- Learning a latent embedding for parts: The part autoencoder learns fixed-scale geometric embeddings from ShapeNet parts using a 3D CNN encoder and implicit decoder.Training uses local TSDF crops and sampled interior/exterior points; the dataset contains 20 million parts from 13 ShapeNet categories.
- Local implicit grids: Local Implicit Grids represent objects and scenes with sparse, overlapping grid cells whose latent codes decode local implicit surfaces.Each cell stores a local latent code, allowing the learned part representation to be reused beyond individual objects.
- Local implicit grids: Overlapping neighboring cells by half the part scale and trilinearly interpolating their independent decoder queries yields a C0-continuous implicit function.In 3D, each query uses the eight overlapping neighboring cells, with interpolation weights determined by position.
- Geometric encoding via latent optimization: At inference, latent grid vectors are optimized by minimizing classification loss on sparse interior/exterior observations, starting from random normal values with standard deviation 10^-2.The optimization updates latent codes associated with the effective grid cells.
- Geometric encoding via latent optimization: For sparse oriented samples, signed points are sampled along both sides of each normal with Gaussian falloff, then the optimized grid is densely queried to extract its zero-contour surface.Grid cells without input points are treated as empty exterior volume.
4. Experiments
The experiments evaluate LIG for part-level generalization, scene-scale representation, and reconstruction from sparse oriented points. Across these settings, LIG transfers better across categories, captures scene details more effectively, and improves sparse-scene reconstruction over baselines, with a finite-thickness limitation.
- Experimental setup: The experiments assess learned-part generalization, scene scalability, and reconstruction from sparse oriented point samples.Geometric quality is evaluated with Chamfer Distance, Normal Alignment, and F-Score.
- Generalization of learned part representation: Training on chairs, LIG transfers across all 13 object categories, whereas the IM-NET object representation does not generalize beyond the source class.The comparison combines quantitative results from Table 1 with qualitative results in Figure 5.
- Method: The representation is designed to reconstruct scenes at high fidelity by combining part-level geometric features with overlapping latent grids and latent-grid optimization.The method optimizes a scene representation against observations using a pretrained decoder.
- Scalability of scene representational power: LIG captures both global scene structure and local object details, while IM-NET reconstructs smooth walls and floors but misses fine details when representing entire scenes.The scene comparison uses Matterport regions and one SceneNet example.
- Scene reconstruction from sparse points: For sparse scene reconstruction, LIG leverages geometric priors to outperform PSR at low point densities and reconstructs thin structures more effectively.The evaluation uses SceneNet and Matterport after watertight preprocessing and compares against traditional and learned baselines.
5. Ablation Study
The ablation study examines part scale, overlapping latent grids, and latent code length. Overlapping grids substantially improve reconstruction quality, while smaller latent codes reduce expressivity and part scale has a comparatively modest effect.
- Ablation setup: The ablation varies latent code length, part scale, and overlapping latent-grid design on SceneNet reconstruction from 100 points/m2.The study uses this setting as a representative reconstruction case.
- Part scale: Reconstruction results are affected by part scale, although they are not very heavily influenced by that choice.The paper contrasts this modest effect with the stronger effect of overlapping latent grids.
- Reconstruction quality: Sparse-point reconstruction is significantly better than baseline methods, especially for sharp edges and thin structures.The qualitative comparison highlights the same geometry types examined in the ablation context.
- Grid design: Overlapping latent grids significantly improve the quality of the overall reconstruction.The result is reported in the ablation comparison.
- Latent code length: A latent code size of 8 slightly deteriorates performance because it provides more limited expressivity for part geometries.The ablation identifies latent-code capacity as a constraint on part representation.
6. Discussion and Future Work
LIG reconstructs unseen object classes and scales to scene reconstruction from sparse point sets, while suggesting several directions for future work.
- LIG reconstructs 3D surfaces of object classes that were unseen during training.
- LIG is presented as the first learned 3D representation, to the authors’ knowledge, for scalable scene reconstruction from sparse point sets.
- Future work includes constraining optimized latent codes near training examples and exploring alternate implicit function representations such as OccNet.
- The authors also propose investigating LIG for 3D reconstruction from images.
Appendix
The part autoencoder encodes normalized TSDF crops with a residual 3D CNN and uses an efficient implicit decoder for local geometry.
- Encoder architecture: The latent codes are decoded into implicit surfaces by an implicit network decoder.
- Encoder architecture: The encoder maps normalized TSDF crops into 32-dimensional latent codes using 3D CNNs with residual bottleneck blocks.
- Decoder architecture: The implicit decoder preserves IM-NET’s architecture while reducing its latent dimension from 128 to 32.
- Decoder architecture: The decoder also reduces each model layer’s hidden-layer count to one quarter of the original for improved efficiency.
A.2. Part autoencoder training
The part autoencoder is trained with sampled geometric points, empty-volume examples, latent regularization, and fixed optimization settings.
- 2048 point samples are drawn for each shape during part autoencoder training.
- The model uses batch size 32, latent penalty λ = 10^-2, and learning rate 10^-3.
- Empty volumes are sampled with probability 10^-3 to embed empty space.
- The part autoencoder is trained for 10^7 steps.
- Training geometry reconstruction samples 10 points along each point normal with standard deviation 1cm.
A.4. Postprocessing algorithm
The postprocessing algorithm removes back-face artifacts by filtering reconstructed faces using normal alignment and component area.
- Back-face artifacts arise when empty LIG grid cells are treated as exterior space, enclosing back-faces within large volumes.
- The algorithm computes each reconstructed face’s centroid and normal, then compares the centroid with nearby oriented input points.
- The original mesh contains a visible dark-blue back-face artifact that is effectively removed in the postprocessed mesh.
- Faces below a normal-alignment threshold and disconnected components below an area threshold are discarded.
- The reported settings are k = 3, n = -0.75, λ = 0.5, i = 50, and a = 1.
B. Additional ablation studies
The ablation studies how latent code length affects reconstruction quality and identifies 32 latent channels as a practical balance between performance and efficiency.
- Increasing latent channels improves reconstruction performance, but with diminishing marginal improvement.The study evaluates latent code length using SceneNet reconstruction from 100 point samples / m2.
- 32 latent channels provide the chosen balance between reconstruction performance and efficiency.
- Figure 10 plots Chamfer Distance, Normal Alignment, and F-Score against latent code length.
C. Additional visual results
The visual results compare ground-truth surfaces and point samples with reconstructions produced by the proposed method and Screened PSR on Matterport and SceneNet scenes.
- For Matterport, the figure compares the ground-truth mesh, input point samples, the proposed reconstruction, and Screened PSR reconstruction.The input density is 500 points / m2.
- For SceneNet, the figure compares the ground-truth mesh, input point samples, the proposed reconstruction, and Screened PSR reconstruction.The input density is 500 points / m2.