Source-linked AI summary
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction
Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, Richard Newcombe
TL;DR
Complex 3D reconstruction must preserve detail and completeness without the memory demands of dense SDF volumes or the scene-scaling limits of object-centric neural representations. DeepLS addresses this with locally learned continuous SDFs stored as independent latent codes over spatial regions. It reports efficient, accurate, compressed representations with local shape completion and finer detail.
Problem
Dense voxel SDFs require substantial memory and can miss fine details or unobserved surfaces, while object-centric neural SDFs do not trivially scale to detailed scenes or novel classes.
Method
DeepLS divides space into local voxel regions, assigns each a latent code, and uses learned local SDF priors to compose complex shapes and scenes.
Results
DeepLS improves reconstruction accuracy and efficiency relative to the cited alternatives while preserving finer detail, completing unobserved surfaces, and using less memory.
Takeaways & Limitations
Decomposing surfaces into local shapes enables high-quality scene reconstruction with scalable representations and efficient training and inference.
Takeaways & Limitations
Enforcing border consistency through partition-of-unity or trilinear interpolation increases SDF-query decoder evaluations by a factor of eight.
Abstract
from arXiv · showhide
Efficiently reconstructing complex and intricate surfaces at scale is a long-standing goal in machine perception. To address this problem we introduce Deep Local Shapes (DeepLS), a deep shape representation that enables encoding and reconstruction of high-quality 3D shapes without prohibitive memory requirements. DeepLS replaces the dense volumetric signed distance function (SDF) representation used in traditional surface reconstruction systems with a set of locally learned continuous SDFs defined by a neural network, inspired by recent work such as DeepSDF. Unlike DeepSDF, which represents an object-level SDF with a neural network and a single latent code, we store a grid of independent latent codes, each responsible for storing information about surfaces in a small local neighborhood. This decomposition of scenes into local shapes simplifies the prior distribution that the network must learn, and also enables efficient inference. We demonstrate the effectiveness and generalization power of DeepLS by showing object shape encoding and reconstructions of full scenes, where DeepLS delivers high compression, accuracy, and local shape completion.
1 Introduction
DeepLS addresses the memory, detail, completeness, and scene-scaling limitations of voxelized SDFs and object-centric neural SDFs by using learned local shape representations. It reports finer, faster, and more complete reconstruction across varied inputs.
- Limitations of Existing Representations: Dense voxel SDFs consume substantial memory, limiting representable resolution and spatial extent.These constraints can cause missing thin structures and fine surface details.
- Limitations of Existing Representations: Because SDF fusion infers only directly observed surfaces, reconstructions are often incomplete when range sensors miss difficult surfaces.
- Limitations of Existing Representations: Object-centric neural SDFs do not trivially scale to detailed scenes and generalize poorly to novel classes or objects.Their shared network and parameterization create global co-dependence across spatial locations.
- DeepLS: DeepLS divides space into grid voxels with local latent codes, combining learned local shape priors with an adaptable representation for complex scenes.The voxels can be larger than typical fusion voxels without sacrificing representable surface detail.
- DeepLS: DeepLS generalizes from local shape patches and provides finer reconstruction, faster inference, stronger completion, and high compression across input types.
2 Related Work
Related work spans traditional local and global surface representations, learned point, voxel, mesh, and continuous implicit methods, and local shape priors. DeepLS uses data-driven local SDF priors to improve completion, memory use, smoothness, and scene scalability.
- Traditional Shape Representations: Traditional surface representations include local implicit primitives and global volumetric approaches for reconstructing geometry from observations.
- Traditional Shape Representations: DeepLS uses data-driven local SDF priors rather than quadratic local functions, with border consistency reducing the need for blending evaluations.The method is designed to tolerate noise and locally complete supported surfaces.
- Learned Shape Representations: Point-based methods regress 3D point coordinates, while voxel-based methods store occupancy or SDF values but retain high memory requirements.Octrees relax dense-voxel limitations to some degree.
- Learned Shape Representations: Mesh methods trade arbitrary topology against smooth, connected surfaces, with sphere deformation limiting topology and multi-plane warping producing disconnected surfaces.
- Learned Shape Representations: Continuous implicit methods learn occupancy or signed-distance functions, and DeepLS adopts DeepSDF as its local shape-network backbone.
- Local Shape Priors: Object-level learned representations can complete known-class shapes but generalize beyond training distributions poorly and do not trivially scale to full scenes.DeepLS instead maintains separate representations for small spatial regions.
3 Review of DeepSDF
DeepSDF models a family of shapes with a shared neural decoder conditioned on an optimized shape code. A shape is represented as the network's zero-level set.
- DeepSDF Model: DeepSDF models a signed surface distance function with a fully connected network parameterized by θ and conditioned on shape code z.
- DeepSDF Model: DeepSDF uses global shape codes, whereas DeepLS applies the same idea to local shape codes arranged across space.
- DeepSDF Model: A shape S is defined as the set of points x where fθ(x, z) equals zero.
- DeepSDF Model: The network parameters θ are shared across shapes, while a separate latent code z is optimized for each shape.
4 Deep Local Shapes
DeepLS represents complex shapes and scenes as compositions of locally decoded SDFs, using sparse voxel-associated codes and shared network parameters. Its design addresses border consistency while supporting code optimization from partial observations.
- Representation: DeepLS composes complex shapes and scenes from an adaptable collection of simpler local shapes rather than a single fixed-length global code.Each local volume has an associated code, allowing the global representation to scale with scene complexity.
- Representation: Each local shape is decoded from a code z_i in local coordinates, with a shared autodecoder network f_θ and a weighting operation combining local contributions.The transformation T_i maps global locations into local coordinate systems, while w and L govern combination.
- Voxel-based construction: The basic implementation partitions space into sparsely allocated voxels, subtracting each voxel center x_i to obtain local coordinates and using an indicator weighting function.This voxel-based construction defines the global surface from local decoder evaluations within voxel volumes.
- Border consistency: Disjoint local voxels can produce inconsistent surface estimates at voxel boundaries, while trilinear interpolation increases SDF query decoder evaluations eightfold.The paper therefore treats border consistency as an accuracy–efficiency trade-off.
- Border consistency: Training extends each local shape beyond its voxel using neighboring-voxel samples; an L∞ radius of 1.5 voxel side-lengths provides a reported accuracy–efficiency trade-off.The extended receptive field makes boundary SDF values computable from abutting local shapes.
- Training and inference: DeepLS jointly optimizes shared decoder parameters and local codes during training, then fixes decoder weights and optimizes local codes for new shapes from partial observations.Training uses sampled SDF pairs, while encoding performs maximum-a-posteriori code estimation.
5 Experiments
Experiments evaluate DeepLS on object and scene reconstruction against learned and volumetric baselines. Results indicate finer detail, faster inference, stronger completion, and high compression, with caveats about unequal representations and benchmark metrics.
- Object Reconstruction: DeepLS improves 3D Warehouse Chamfer distance error by approximately one order of magnitude over related shape-learning approaches.The comparison is not between equal methods because competing approaches use global object-level representations and different parameter distributions.
- Object Reconstruction: DeepLS reconstructs much finer surface details than DeepSDF on 3D Warehouse shapes.The qualitative comparison is reported in Fig. 4.
- Object Reconstruction: 1 minute of training and inference produces a Stanford Bunny reconstruction with 0.03% RMSE, while DeepSDF requires over 8 days for the same surface error.DeepLS was trained on randomly oriented primitive shapes and generalized to the novel bunny shape.
- Scene Reconstruction: DeepLS reconstructs more complete surfaces at the same accuracy level as TSDF Fusion on synthetic ICL-NUIM scenes.The evaluation covers all four ICL-NUIM sequences and examines the accuracy-completeness tradeoff.
- Scene Reconstruction: At the highest compression rate, DeepLS maintains completeness using 4.4K parameters for the full scene, whereas TSDF Fusion drops to about 50% less completeness.The benchmark can report low surface error for missing surfaces because its metric does not strongly penalize incompleteness.
- Scene Reconstruction: DeepLS is evaluated on real 3D Scene Dataset scans, including scenes with thin objects, using sparse depth frames sampled every tenth capture frame.The reported comparisons include TSDF Fusion and volumetric-fusion ground truth.
6 Conclusion
The conclusion presents DeepLS as a local-shape combination of volumetric fusion and deep priors. It reports improved detail, accuracy, completion, memory use, and efficiency relative to the relevant baselines.
- Conclusion: DeepLS combines volumetric fusion and deep shape priors for 3D surface reconstruction from depth observations.Its central design decomposes large surfaces into local shapes.
- Conclusion: Local-shape decomposition enables higher accuracy, finer detail, unobserved-surface completion, and lower memory use than storing a full SDF volume.These benefits are stated for the presented surface-reconstruction approach.
- Conclusion: Compared with object-centric shape-learning approaches, DeepLS improves surface reconstruction accuracy by an order of magnitude and increases training and inference efficiency.The conclusion attributes these differences to local shape decomposition.
Supplementary Material
The supplementary material adds experimental details, local-shape analyses, metrics, comparisons, and results. It also includes a video showing reconstructions in motion.
- Supplementary Material: The supplement provides detailed experimental setup information, additional 3D Warehouse metrics, and further experiment results and comparisons.These materials are organized across Sections B, E, and F.
- Supplementary Material: Figure 11 includes a latent-code-dimension study and a 200-primitive training scene with instantiated local shape blocks.The latent-code study measures Chamfer-distance test error on the airplane class.
B Experimental Setup
The supplementary setup describes the local-shape decoder, truncated-SDF training, sparse block allocation, and training data used to learn local priors. It also examines latent interpolation and border consistency.
- Autodecoder Network: The DeepLS autodecoder is a four-layer fully connected network with 128 outputs per layer and produces scaled truncated SDF values.The network uses leaky ReLUs and a final tanh, with a selected latent size of 125.
- Training: Training applies tanh to scaled ground-truth SDF values before optimizing latent codes and network parameters with Adam.The scaling makes the post-tanh interval [−0.9, 0.9] cover approximately two blocks.
- Local Shape Blocks: Scene local shape blocks are allocated sparsely based on available depth data, supporting scalability to real-world inputs.Figure 12 shows the instantiated blocks in a scene.
- Latent Priors: Latent-space interpolation sequences illustrate transitions between local surfaces such as a flat surface and a pole.The paper notes that the possible function space within a voxel is generally much larger.
- Training Data: Training data combines randomly posed primitive shapes, 3D Warehouse meshes, and additional model categories for learning local shape priors.The 3D Warehouse subset contains 200 models from each of five classes, split into 32×32×32 local blocks.
- Local Priors: Local-prior training restricts voxel solutions to reasonable local SDF surfaces, while allocated blocks together reconstruct the whole scene.The accompanying analyses also examine border consistency using simple 2D shapes and sampled SDF measurements.
E 3D Warehouse Comparison - Additional Metric
DeepLS is evaluated on unknown 3D Warehouse shapes using both Chamfer distance and mesh accuracy, with all metrics favoring DeepLS over related object-level representations.
- Mesh accuracy measures the maximum distance within which 90% of generated points lie from the ground-truth mesh.
- DeepLS achieves substantially higher accuracy than related object-level representations on the 3D Warehouse test shapes.
- Chamfer distance and mesh accuracy exhibit the same comparative trend across the evaluated representations.
F Scene Experiments
Scene experiments evaluate DeepLS from depth-derived SDF samples under varying reconstruction settings and noisy observations, showing smoother and quantitatively improved reconstruction over TSDF Fusion.
- Table 4 reports Chamfer distance and mesh accuracy for unknown 3D Warehouse test shapes, with lower values better for all metrics.
- Depth scans produce zero-SDF samples at observed points, signed samples displaced ±1.5 cm along surface normals, and additional free-space ray samples.
- DeepLS uses a parameter controlling the region around depth samples where marching cubes is performed, while receptive-field size affects SDF reconstruction quality.
- Under artificial noise of σ = 0.015, learned local shape priors produce smoother surfaces than TSDF Fusion.
- 6.41 mm surface error and 71.04 % completion are achieved by DeepLS, compared with 7.29 mm and 68.53 % for TSDF Fusion on the whole ICL-NUIM benchmark scene.
F.3 Qualitative Results
On real structured-light scans, DeepLS preserves small details and produces sharper corners and more complete surfaces than TSDF Fusion.
- DeepLS represents small details such as chair bars that TSDF Fusion tends to lose.
- DeepLS produces sharper corners in magnified regions of the real-scanned comparisons.
- DeepLS provides more complete surfaces than TSDF Fusion on partially scanned real scenes.