Source-linked AI summary
Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation
Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, Guofeng Zhang
TL;DR
Dense tracking and mapping systems struggle to represent unknown, varied scenes while supporting accurate, complete, and efficient reconstruction. Vox-Fusion combines voxel-based neural implicit surfaces with an expandable octree and reports higher accuracy, smaller memory use, and faster speed.
Problem
Existing mapping approaches have limited novel-view rendering, memory efficiency, global consistency, or generalization across different scene types.
Method
Vox-Fusion combines voxel embeddings and a neural implicit SDF network within a dynamically expandable sparse octree for dense tracking and mapping.
Results
Experiments show higher accuracy, smaller memory use, and faster speed than prior systems, with better reconstruction and trajectory estimation across evaluated synthetic and real scenes.
Takeaways & Limitations
Vox-Fusion supports practical dense mapping of unexplored scenes and applications requiring real-time tracking, including augmented and virtual reality.
Abstract
from arXiv · showhide
In this work, we present a dense tracking and mapping system named Vox-Fusion, which seamlessly fuses neural implicit representations with traditional volumetric fusion methods. Our approach is inspired by the recently developed implicit mapping and positioning system and further extends the idea so that it can be freely applied to practical scenarios. Specifically, we leverage a voxel-based neural implicit surface representation to encode and optimize the scene inside each voxel. Furthermore, we adopt an octree-based structure to divide the scene and support dynamic expansion, enabling our system to track and map arbitrary scenes without knowing the environment like in previous works. Moreover, we proposed a high-performance multi-process framework to speed up the method, thus supporting some applications that require real-time performance. The evaluation results show that our methods can achieve better accuracy and completeness than previous methods. We also show that our Vox-Fusion can be used in augmented reality and virtual reality applications. Our source code is publicly available at https://github.com/zju3dv/Vox-Fusion.
1 INTRODUCTION
Vox-Fusion targets scalable, detailed implicit tracking and mapping of unknown scenes by combining sparse voxel embeddings, an explicit octree, and a neural implicit network. The system directly renders signed distance volumes for improved tracking and reconstruction without performance overhead, and supports large maps through efficient keyframe selection.
- Motivation: The method addresses the difficulty of representing geometric details and scaling larger environments with a single global MLP by using a hybrid sparse voxel structure.The introduction identifies single-MLP representation as difficult for geometric detail and larger environments.
- Proposed system: Vox-Fusion combines voxel embeddings indexed by an explicit octree with a neural implicit network for scalable implicit scene reconstruction with sufficient detail.The system uses a sparse octree with Morton coding for fast voxel allocation and retrieval.
- Proposed system: Directly rendering signed distance volumes improves tracking accuracy and reconstruction quality over current SOTA systems without performance overhead.This claim is presented as a contribution of the proposed fusion system.
- Proposed system: A keyframe selection strategy based on ratio testing and information gain is designed to maintain large-sized maps efficiently.The strategy is described as fast and efficient and more suitable for large-sized maps.
- Evaluation and applications: Extensive experiments on synthetic and real-world scenes demonstrate high-quality 3D reconstructions that can directly benefit AR applications.Dense SLAM supports AR and VR functions such as real-time occlusion effects and collision detection when tracking is accurate and surfaces are complete.
2 RELATED WORK
Prior dense SLAM and neural implicit methods provide important foundations but face limitations in scene scale, reconstruction quality, robustness, or practical deployment. Vox-Fusion builds on voxel-based neural implicit representations while targeting simultaneous RGB-D SLAM with dynamically expandable scene structure.
- Traditional SLAM Methods: Traditional dense SLAM methods use photometric or RGB-D constraints for tracking and reconstruction, but early systems are limited in applicable scene size.DTAM uses every-pixel photometric consistency and a cost-volume scene model, while KinectFusion exploits commodity RGB-D depth acquisition.
- Learning-based Methods: Learning-based methods encode geometric or depth information into low-dimensional representations, but learned priors can fail in complex areas and reduce robustness.DI-Fusion decodes latent codes into SDF values, whereas CodeSLAM jointly optimizes keyframes and camera poses using encoded depth maps and a pretrained decoder.
- Comparison with Concurrent Work: NICE-SLAM addresses scalability with hierarchical voxel grids, but preallocating an entire scene is unsuitable when scene bounds are unknown, motivating dynamic allocation in Vox-Fusion.The passage states that Vox-Fusion and NICE-SLAM both use voxel features rather than encoding 3D coordinates, while differing in scene allocation.
- Neural Implicit Representations: Neural rendering methods improve scene representation efficiency through voxel features or embeddings, but photorealistic rendering does not guarantee accurate surface reconstruction.NSVF embeds local information in a voxel feature grid, and Plenoxels use spherical harmonics as voxel embeddings without a neural network.
- Voxel-based Neural Implicit Representations: Vox-Surf shares Vox-Fusion’s voxel-based neural implicit structure but is an offline system requiring posed images and preallocating all voxels, whereas Vox-Fusion consumes consecutive RGB-D frames for simultaneous SLAM.The comparison directly distinguishes the systems’ operating mode, input data, and voxel allocation requirements.
3 SYSTEM OVERVIEW
Vox-Fusion processes continuous RGB-D frames with separate tracking and mapping processes. Tracking estimates each camera pose relative to a fixed implicit scene network, while mapping constructs and optimizes the global map.
- System inputs: The system receives continuous RGB-D frames containing RGB images and depth maps, using a known intrinsic matrix and the pinhole camera model.The RGB images are denoted I_i ∈ R^3, depth maps D_i ∈ R, and camera intrinsics K ∈ R^3×3.
- Tracking: Tracking estimates each frame’s 6-DoF camera pose relative to the fixed implicit scene network through differentiable volume rendering.The pose is represented as T_i ∈ SE(3).
- Mapping: Mapping initializes the global map from the first frame, then receives tracked frames and estimated poses to construct the global map by allocating new voxels.New voxels are allocated from back-projected frame data.
4 METHOD
Vox-Fusion represents observed scene surfaces with voxel embeddings and an implicit SDF network, rendering RGB-D observations through interpolated voxel features. It combines pose tracking, joint mapping, explicit keyframe selection, and octree-based voxel allocation for efficient online reconstruction.
- Voxel-based representation: Voxel embeddings attached to shared voxel vertices encode the scene and reduce border artifacts between neighboring voxels.The embeddings are optimized together with the implicit SDF decoder parameters.
- Implicit surface rendering: The system directly regresses SDF values and renders color and depth by applying volume rendering to interpolated voxel embeddings.This uses voxel embeddings rather than global 3D coordinates as the network input.
- Optimization: Four losses supervise sampled points: RGB, depth, free-space, and SDF losses.Points behind the truncation region are masked during rendering, avoiding surface-intersection ambiguities and accelerating processing.
- Tracking and mapping: Tracking fixes voxel embeddings and network parameters while optimizing only the current frame’s 6-DoF pose in SE(3).Pose updates are represented in the tangent space as ξ ∈ se(3), under a zero-motion assumption between nearby frames.
- Key-frame selection: Keyframes are inserted when the observed-to-unobserved voxel ratio exceeds a threshold, with a maximum frame interval preventing missing constraints during long loops.The explicit voxel structure supports intersection-based selection rather than only heuristic metrics or fixed intervals.
- Scene representation and mapping: An octree partitions the scene into axis-aligned voxels, allocating new voxels for back-projected depth points and enabling fast retrieval through Morton-coded coordinates.Joint mapping optimizes scene geometry and camera poses over randomly sampled keyframes and rays.
5 EXPERIMENTS
Experiments evaluate Vox-Fusion on synthetic, real-world, and outdoor RGB-D data using reconstruction and trajectory metrics. The results show improved reconstruction quality, detailed surfaces, robust real-scene reconstruction, and efficient memory use, with demonstrations in augmented reality.
- Datasets: Experiments use Replica, ScanNet, and RGB-D sequences captured by iOS devices with range sensors.Replica provides synthetic scenes, ScanNet provides captured sequences and ground-truth poses, and iOS devices supply real indoor and outdoor data.
- Evaluation Metrics: Reconstruction quality is measured by mesh accuracy and completion, while pose estimation uses absolute trajectory error.Accuracy and completion are based on bidirectional distances between reconstructed and ground-truth geometry; ATE measures absolute translational pose differences.
- Synthetic Scenes: On Replica, Vox-Fusion produces better maps than iMap and NICE-SLAM, with detailed surfaces such as table legs and flowers recovered despite a single voxel-grid level.The explicit voxel map limits hallucinated surfaces to visible sparse voxels, while the signed-distance representation supports detailed reconstruction.
- Real-World Reconstruction: On noisy ScanNet scans, Vox-Fusion achieves competitive reconstruction despite the challenge of erroneous real-world measurements.The evaluation compares against DI-Fusion, iMap*, and NICE-SLAM using selected sequences consistent with prior work.
- Efficiency and Applications: Vox-Fusion reconstructs outdoor iOS RGB-D sequences without prior scene knowledge, while sparse voxel processing and implicit features reduce computational and memory demands.The system uses parallel tracking and mapping, profiles runtime components on an RTX 3090, and reports better reconstruction accuracy with significantly less memory than NICE-SLAM.
6 APPLICATIONS
Vox-Fusion supports AR and VR applications by combining accurate camera tracking, detailed depth estimation, and realistic differentiable rendering. Its voxel-based representation also enables scene editing, fast collision detection, and controllable levels of detail.
- Augmented reality: Vox-Fusion places arbitrary virtual objects into reconstructed scenes and represents real–virtual occlusions using rendered depth maps.These capabilities support augmented reality applications in cluttered scenes.
- Virtual reality: Vox-Fusion enables free-view virtual scene traveling through accurate camera tracking and realistic rendering.This supports virtual reality applications.
- Virtual reality: Deleting supporting voxels and associated feature embeddings makes scene editing easier.The voxel-based representation allows parts of the scene to be removed directly.
- Virtual reality: The explicit voxel representation supports fast collision detection and controllable levels of detail through splitting and refining feature embeddings.These operations provide additional control for VR applications.
7 CONCLUSION
Vox-Fusion is a voxel-based implicit-surface system for dense tracking and mapping that supports dynamic voxel creation and improved performance. Experiments report higher accuracy with less memory and faster speed, while dynamic-object handling remains non-robust.
- Vox-Fusion is a dense tracking and mapping system built on voxel-based implicit surface representation.
- Dynamic voxel creation makes Vox-Fusion more suitable for practical scenes.
- A multi-process architecture and corresponding strategies improve the system’s performance.
- Experiments show higher accuracy, smaller memory usage, and faster speed than the prior system configuration.
- Vox-Fusion cannot yet robustly handle dynamic objects.