DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors

1. Introduction

• Conventional depth-fusion representations often require substantial memory and can yield incomplete or unsatisfactory geometry under sensor noise or occlusion.Prior systems commonly use SDF integration on sparse voxel sets.Source: §1 · p. 1
• Deep implicit functions represent continuous geometry, support extraction at arbitrary resolution, and encode priors that can support high-quality reconstruction.Local implicit fields extend these capabilities to scene-level reconstruction without relying on scene semantics.Source: §1 · p. 1
• Online use of deep priors requires explicit uncertainty modeling, accurate camera tracking, and efficient incremental integration of new observations.DI-Fusion introduces PLIVox, which encodes geometry and uncertainty in one neural network, and develops tracking and mapping methods around that representation.Source: §1 · pp. 1–2

2. Related Works

• Online reconstruction systems have used voxel or surfel data structures for scale, then added bundle adjustment or sub-map registration to improve model quality and global consistency.Examples include OctreeFusion, VoxelHashing, Scalable-VoxelHashing, Elastic-Fusion, and bundle-adjustment approaches.Source: §2 · p. 2
• Geometry deep learning has led reconstruction and tracking methods to use neural networks for sparse features, monocular depth observations, and object instances [33, 47]; stochastic sensor noise motivates probabilistic geometry, including hand-crafted models.Source: §2 · p. 2
• Discrete implicit maps can limit surface quality and storage, while deep implicit representations have been used in learning and offline reconstruction; DI-Fusion applies them to online fusion.Earlier incremental alternatives include Gaussian Process maps.Source: §2 · p. 2

3. Method

• Figure 3 depicts encoding and decoding within one PLIVox, with an incremental latent-vector update when a previous-frame latent vector is available.Figure 4 compares convergence for different camera-tracking optimization strategies; Figure 5 illustrates incremental integration and encodes surface uncertainty as mesh color.Source: Figure 3 · p. 3, Figure 4 · p. 4, Figure 5 · p. 5

3.1. PLIVox Representation

• Each PLIVox uses a shared encoder-decoder network to represent a probabilistic SDF, encoding local geometry and uncertainty as Gaussian mean and standard deviation.The encoder aggregates local-coordinate and normal features from observed points; the decoder combines a local coordinate with the geometry latent vector to predict the distribution.Source: §3.1 · p. 3

3.2. Camera Tracking

• Camera tracking minimizes an objective combining probabilistic SDF alignment and an intensity term, using an approximate gradient that holds σD constant during local linearization.The intensity term is intended to help when the SDF term fails in areas with few geometric details, such as walls or floors.Source: §3.2 · p. 4

3.3. Surface Mapping

• Mapping updates each PLIVox’s geometry latent vector from new observations, rather than updating individual voxel SDFs, and integrates only sparse frames.The method can extract a mesh at a desired resolution by decoding SDFs on grids and applying Marching Cubes, with overlapping PLIVox domains used to maintain continuity.Source: §3.3 · p. 5

4.1. System Implementation

• ShapeNet training uses six categories with 100 shapes each, while each PLIVox contributes 4,096 decoder samples and 16–128 encoder samples.Position jitter and normal perturbation augment the training data; the latent vector length is 29.Source: §4.1 · p. 5

4.2. Quantitative Evaluations

• DI-Fusion achieves lower or comparable ATE and state-of-the-art surface error against TSDF or surfel representations, with fewer parameters.The evaluation uses ICL-NUIM; Table 1 reports ATE and Table 2 reports surface error, both in centimeters.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.3. Qualitative Results

• 91.3% and 95.8% fewer parameters than TSDF High-res and surfel tracking, respectively, accompany a more complete reconstruction on ICL-NUIM.At the same memory use, TSDF Low-res hampers both camera tracking and reconstruction quality.Source: §4.3 · p. 6, Figure 6 · p. 7
• ScanNet reconstructions are reported as preferable to TSDF and surfel baselines, using ShapeNet-trained encoder-decoder weights without ScanNet fine-tuning.The authors describe this as evidence of generalization across different scene types.Source: §4.3 · p. 6, Figure 7 · p. 7, Figure 8 · p. 7

4.4. System Ablations and Analysis

• Assuming constant standard deviation 1.0 worsens trajectory estimation and surface error relative to probabilistic tracking, which downweights erroneous fitting.This ablation removes the decoder branch that outputs σD.Source: §4.4 · pp. 6–7
• Intensity weights between 500 and 2000 perform best; pure SDF tracking can fail with few geometric features, while weights above 4000 increase drift.The max latent-vector update also yields consistently worse average ATE and surface error than the mean update, with possible sensitivity to sensor noise.Source: Figure 9 · p. 7, §4.4 · p. 8, Figure 10 · p. 8
• Larger frame-integration intervals increase average ATE, while overly frequent integration introduces sensor noise that can deteriorate mapping quality.Integration is performed on sparse frames sampled every N incoming frames.Source: Figure 9 · p. 7, §4.4 · p. 8
• Shrinking PLIVox size improves reconstruction detail but raises runtime and memory needs; the system runs online at 10–12 Hz on most tested modern GPUs.The timing figure breaks down component costs, and texturing takes approximately 50 ms per integrated frame.Source: §4.4 · p. 8, Figure 11 · p. 8, Figure 12 · p. 8

• The learned prior may not fit all local geometries, especially overcomplicated objects.Source: §5 · p. 8
• Independent PLIVoxs do not model relationships between neighbors, so spatial continuity is not guaranteed, and the system lacks loop closure for global consistency.Source: §5 · p. 8
• DI-Fusion incorporates geometry and uncertainty priors into joint camera tracking and surface mapping, achieving more accurate camera pose estimation and higher-quality 3D reconstruction.Source: §5 · p. 8