PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniTAM v3: A Framework for Large-Scale 3D Reconstruction with Loop Closure

Victor Adrian Prisacariu, Olaf Kähler, Stuart Golodetz, Michael Sapienza, Tommaso Cavallari, Philip H S Torr, David W Murray

arXiv:1708.00783v1cs.CV

TL;DR

Although voxel hashing enables much larger maps than dense volumes, off-the-shelf video memory holds roughly one room at 4mm voxel resolution; inactive regions can be swapped to host memory. InfiniTAM v3 extends its reconstruction framework with robust tracking, keyframe-based random-ferns relocalisation, rigid-submap TSDF reconstruction, and surfel reconstruction. The report says low-level code enhancements significantly improve camera tracking performance and describes four added reconstruction and tracking features.

  • Problem

    Although voxel hashing enables much larger maps than dense volumes, off-the-shelf video memory holds roughly one room at 4mm voxel resolution; inactive regions can be swapped to host memory.

    Source: §5 · p. 11

  • Method

    InfiniTAM v3 extends its reconstruction framework with robust tracking, keyframe-based random-ferns relocalisation, rigid-submap TSDF reconstruction, and surfel reconstruction.

    Source: §1.1 · p. 2

  • Results

    The report says low-level code enhancements significantly improve camera tracking performance and describes four added reconstruction and tracking features.

    Source: §1.1 · p. 2

  • Takeaways & Limitations

    The surfel implementation is designed to let users explore its trade-offs within the same framework as TSDF reconstruction.

    Source: §1.1 · p. 2

    Keyframe-based random ferns tend to relocalise poorly from novel poses, though they perform well for poses corresponding to stored keyframes.

    Source: §7.1 · p. 16

Abstract

from arXiv · show

Volumetric models have become a popular representation for 3D scenes in recent years. One breakthrough leading to their popularity was KinectFusion, which focuses on 3D reconstruction using RGB-D sensors. However, monocular SLAM has since also been tackled with very similar approaches. Representing the reconstruction volumetrically as a TSDF leads to most of the simplicity and efficiency that can be achieved with GPU implementations of these systems. However, this representation is memory-intensive and limits applicability to small-scale reconstructions. Several avenues have been explored to overcome this. With the aim of summarizing them and providing for a fast, flexible 3D reconstruction pipeline, we propose a new, unifying framework called InfiniTAM. The idea is that steps like camera tracking, scene representation and integration of new data can easily be replaced and adapted to the user's needs. This report describes the technical implementation details of InfiniTAM v3, the third version of our InfiniTAM system. We have added various new features, as well as making numerous enhancements to the low-level code that significantly improve our camera tracking performance. The new features that we expect to be of most interest are (i) a robust camera tracking module; (ii) an implementation of Glocker et al.'s keyframe-based random ferns camera relocaliser; (iii) a novel approach to globally-consistent TSDF-based reconstruction, based on dividing the scene into rigid submaps and optimising the relative poses between them; and (iv) an implementation of Keller et al.'s surfel-based reconstruction approach.

1 Introduction

InfiniTAM v3 is a technical implementation report that adds reconstruction and relocalisation capabilities while improving camera tracking. Its new features include robust tracking, random-ferns relocalisation, globally consistent TSDF reconstruction, and surfel-based reconstruction.

1.1 What’s New

  • InfiniTAM v3 adds robust tracking, random-ferns relocalisation, globally consistent TSDF reconstruction, and surfel-based reconstruction.The report also says low-level code enhancements significantly improve camera tracking performance.Source: §1.1 · p. 2
  • The tracker aligns depth images with a scene raycast and consecutive RGB frames, and detects tracking failures to trigger relocalisation.RGB alignment is intended to improve pose estimation on geometrically poor surfaces.Source: §1.1 · p. 2
  • Glocker et al.’s keyframe-based random ferns camera relocaliser recovers camera pose when tracking fails.The implementation is cited as.Source: §1.1 · p. 2
  • The globally consistent TSDF approach divides the scene into rigid submaps and optimises their relative poses using accumulated inter-submap constraints.The approach is cited as.Source: §1.1 · p. 2
  • The framework adds Keller et al.’s surfel-based reconstruction to let users explore its trade-offs alongside TSDF reconstruction.Surfels can support dynamic scenes, while visibility determination and collision detection can be more costly than with a TSDF.Source: §1.1 · p. 2

2 Cross-Device Implementation Architecture

InfiniTAM organizes processing as engines that pass data structures through a pipeline. Each engine separates an abstract interface from device-specific and device-agnostic implementation layers.

2 Cross-Device Implementation Architecture

  • InfiniTAM uses a chain-of-responsibility design in which engines process and pass data structures through the pipeline.Engines are intended to be stateless and responsible for one aspect of processing; ITMMainEngine holds the engines and objects and controls information flow.Source: §2 · pp. 2–3
  • Each engine has an abstract interface, a device-specific implementation, and a device-agnostic layer containing inline C code.The abstract layer may share common code, while device-specific implementations can differ between CPU and GPU.Source: §2 · p. 3
  • For tracking, generic error-function optimization can sit in the abstract layer, per-pixel evaluation in a loop or CUDA kernel, and single-pixel evaluation in an inline C function.The source files are organized into separate modules, each split into abstract, specific, and agnostic folders.Source: §2 · p. 3

3 Volumetric Representation with Hashes

InfiniTAM uses a hash lookup to support volumetric reconstruction at large scale, storing voxel data in blocks and locating those blocks through a hash table. Hash-table operations search collision chains to retrieve voxels or allocate blocks for insertion.

3 Volumetric Representation with Hashes

  • A hash lookup is the key component for scaling KinectFusion to large 3D environments.The representation is implemented with a voxel block array, a hash table and hashing function, and insertion, retrieval, and removal operations.Source: §3 · p. 3

3.1 The Voxel Block Array

  • Each ITMVoxel stores an SDF value and depth-observation weight; depending on depth representation, the SDF is float initialized to 1.0 or short initialized to 32767, and variants may include color or confidence.Voxels are grouped into 8 × 8 × 8 blocks stored contiguously in the voxel block array.Source: §3.1 · pp. 3–4

3.2 The Hash Table and The Hashing Function

  • The hash table locates voxel blocks using their corner coordinates; collisions use a one-entry ordered bucket and an unordered excess list.Each entry stores an offset into the voxel block array, and the excess list is linked by entry offsets.Source: §3.2 · p. 4

3.3 Hash Table Operations

  • The current version supports insertion and retrieval, but does not implement removal because it is not currently required.The code discourages or disallows direct memory access outside the hash-table operations.Source: §3.3 · pp. 4, 6
  • Retrieval searches for a target block and returns its voxel, while insertion reserves a voxel block and records its address and world location.When the ordered bucket is exhausted, search continues through the linked excess list; GPU reserve operations use atomic operations.Source: §3.3 · pp. 5–6

4 Method Stages

InfiniTAM processes each camera frame through six stages, from tracking and voxel allocation to integration, rendering, and memory swapping. Its standard fusion and globally consistent pipelines differ in which reconstruction they support.

4 Method Stages

  • Each ProcessFrame call runs six stages: tracking, allocation, swapping in, integration, raycast, and swapping out.ITMBasicEngine provides standard fusion, while ITMMultiEngine supports globally consistent reconstructions.Source: §4 · pp. 6–7
  • Tracking fits the current depth image, and optionally color, to the previous frame’s projection of the world model.The ITMTracker family includes depth, color, and extended trackers.Source: §4 · p. 6
  • Allocation creates voxel blocks as needed from the depth image and builds a list of visible blocks.The voxel and surfel maps use their respective scene-reconstruction engines.Source: §4 · p. 7
  • Swapping moves map data between host and device memory, but the current implementation supports it only for voxel-based ITMBasicEngine.Both swapping in and swapping out have this scope boundary.Source: §4 · p. 7
  • Integration incorporates current depth and color frames into the voxel or surfel map, while raycasting renders the model for display and next-frame tracking.Raycasting uses the current pose.Source: §4 · p. 7
  • Supporting engines also handle low-level processing, image preparation, meshing, sensor input, and relocalization.Mesh generation uses Marching Cubes [7]; the relocalizer is in FernRelocLib.Source: §4 · p. 7

4.1 Tracking

InfiniTAM v3 estimates camera pose using depth, color, or a combined extended tracker. Its extended tracker adds robust depth alignment, optional photometric alignment, and tracking-quality assessment to the tracking pipeline.

4.1 Tracking

  • The revised ITMExtendedTracker estimates pose from the world model and has CPU and CUDA implementations.It can replace ITMDepthTracker in its default configuration and typically provides better tracking accuracy.Source: §4.1 · p. 7, §4.1.3 · p. 8

4.1.1 ITMDepthTracker

  • ITMDepthTracker aligns depth points to rendered surface points and normals, minimizing planar distances through iterative pose updates.A depth-image resolution hierarchy improves convergence; the alignment follows.Source: §4.1.1 · p. 7

4.1.2 ITMColorTracker

  • ITMColorTracker minimizes color differences between projected rendered surface points and the current image using Levenberg-Marquardt optimization.It uses a color-image resolution hierarchy and subsamples the surface-point list by a factor of 4.Source: §4.1.2 · p. 8

4.1.3 ITMExtendedTracker

  • ITMExtendedTracker combines robust, depth-weighted geometric alignment with optional frame-to-frame photometric alignment and uses an SVM to assess tracking quality.It uses a Huber norm, rejects points beyond a configurable geometric threshold, and activates the relocalizer on tracking failure; the combined geometric and photometric error is optimized with Levenberg-Marquardt.Source: §4.1.3 · pp. 8–9

4.1.4 Configuration

  • A settings string selects the tracker and its configuration; the default is ITMExtendedTracker with color tracking disabled.This permits depth-only sensors; device code computes function, gradient, and Hessian values, while optimization remains in the abstract class.Source: §4.1.4 · p. 9

4.2 Allocation

Voxel allocation is divided into three stages that identify candidate blocks, allocate storage, and assemble the visible-block list. The design aims to minimize blocking operations and avoid critical sections.

4.2 Allocation

  • The allocation stage is split into three parts to minimize blocking operations and avoid critical sections.Source: §4.2 · p. 9
  • For each depth pixel, the system backprojects a line from d − µ to d + µ and identifies voxel blocks intersected in world coordinates.It searches the hash table for each block and records allocation and visibility information; intra-frame collisions allocate only one block, with the artifact fixed automatically at the next frame.Source: §4.2 · p. 9
  • The system allocates voxel blocks for nonzero allocation and visibility entries using atomic subtraction on a stack of free block indices.Source: §4.2 · p. 9
  • It then builds a list of live hash entries that project inside the visible frustum for later integration and swapping.Source: §4.2 · p. 9

4.3 Integration

Integration updates voxels in currently visible blocks using observations from the latest depth image. Voxels behind the observed surface receive no new information; observations close to or in front of the surface contribute to the accumulated values.

4.3 Integration

  • The update projects each voxel into the depth image and returns without updating when the voxel is behind the observed surface.The implementation also returns for points behind the camera, outside the image, or without a valid depth measurement.Source: §4.3 · pp. 9–10
  • For voxels close to or in front of the observed surface, the measured observation is added to the accumulated TSDF value and weight.The update computes a truncated SDF observation and increments the accumulated weight, capped at maxW.Source: §4.3 · pp. 9–10
  • The updated TSDF value and depth weight are written back to the voxel.Source: §4.3 · p. 10
  • Integration calls the voxel-update function only for voxels in visible blocks, rather than updating every voxel in a fixed volume.Source: §4.3 · pp. 9–10

4.4 Raycast

Raycasting produces a depth image for tracking and visualization by finding surface intersections along camera rays. InfiniTAM accelerates this search with TSDF-guided steps and a staged process that handles unallocated blocks and bounds the plausible depth range.

4.4 Raycast

  • The final depth image is computed from the updated 3D model and a camera pose and intrinsics, supplying the next frame’s tracking input and supporting visualization.Raycasting checks TSDF values along each pixel ray until a zero-crossing is found.Source: §4.4 · p. 10
  • TSDF values guide conservative ray steps, while a five-state machine handles unallocated blocks and identifies surface crossings without treating empty-block boundaries as surfaces.The search progresses from coarse block steps to fine signed-distance steps; two trilinear interpolation steps locate the surface after a crossing.Source: §4.4 · pp. 10–11
  • A projected depth range from visible voxel blocks narrows the raycasting search for each pixel.The CPU computes projected block bounding boxes and fills them with minimum and maximum depths, handling overlaps.Source: §4.4 · p. 11
  • GPU depth-range estimation projects blocks into 16 × 16 pixel fragments, then uses atomic updates to set per-pixel minimum and maximum depths.Fragmenting reduces update collisions to typically a few hundreds or thousands of pixels in a 640×480 image.Source: §4.4 · p. 11

5 Swapping

InfiniTAM swaps inactive voxel blocks from video-card memory to host memory to extend maps beyond the active GPU capacity. Its fixed-size transfers, lightweight host-side checks, and post-swap fusion address transfer bounds, host processing, and variable storage speeds.

5 Swapping

  • Voxel hashing permits larger maps than dense KinectFusion volumes, but an off-the-shelf video card holds roughly one room at 4 mm voxel resolution; inactive map parts are swapped to host memory.Only map parts inside or near the current view frustum remain in video-card memory.Source: §5 · p. 11
  • The swapping design targets bounded, minimized host/device transfers, minimal host processing, and no assumptions about host-memory type or speed.Transfers use fixed-size buffers; host storage mirrors the hash table’s capacity so indices can be checked without a separate host hash table.Source: §5 · p. 11
  • After swapping, secondary integration fuses host voxel data with newly integrated device-map data, preserving both while accommodating slow host storage.The device continues integrating live depth data while visible blocks are being swapped in, so some blocks may already contain new data when they arrive.Source: §5 · p. 12
  • Allocation identifies blocks to swap in, while integration marks blocks to swap out; blocks move in near live-frame boundaries and out after depth data is integrated.Swapping in transfers hash indices and then blocks through host/device buffers; swapping out transfers indices and blocks to host voxel memory.Source: §5 · p. 12

6 UI, Usage and Examples

InfiniTAM’s UI presents live reconstruction views and processing information, with controls for frame-by-frame or full-video processing. Running the system requires calibrated depth input, and available data sources determine which tracking modes can be used.

6 UI, Usage and Examples

  • The UI shows a live raycasted reconstruction, depth and colour images, per-frame processing time, and available keyboard shortcuts.Users can process one frame at a time or the whole input video; the UI also offers reset, model export, arbitrary-view raycasting, and rendering styles.Source: §6 · pp. 12, 14
  • InfiniTAM requires calibrated depth input, with an optional image; calibration includes camera image parameters, depth-to-RGB extrinsics, and depth calibration.The calibration is specified through an external file.Source: §6 · p. 14
  • Supported data sources include OpenNI and PPM/PGM images; depth-only input, such as from the Structure Sensor, works only with the ICP depth-based tracker.The tested OpenNI sources include PrimeSense-based long-range cameras, Kinect, RealSense, and Structure Sensor.Source: §6 · p. 14

7 New Features in InfiniTAM v3

InfiniTAM v3 adds keyframe-based random-ferns relocalisation, globally consistent reconstruction from optimised rigid submaps, and a surfel-based reconstruction implementation. These features support camera recovery and mapping while retaining distinct scope and implementation limits.

7.1 Random Ferns Relocaliser

  • Keyframe-based random ferns retrieve similar stored keyframes to restart tracking, and can also detect loop closures; they relocalise poorly from novel poses.The system encodes images with binary fern codes, looks up similar keyframes, and uses their stored poses; correspondence-based methods can improve recovery from novel poses.Source: §7.1 · p. 16

7.2 Globally-Consistent Reconstruction

  • Rigid submaps are tracked as active or retained as passive, while accumulated inter-submap constraints drive background optimisation of their relative poses.Loop-closure detection adds constraints between the primary and passive submaps and reactivates the involved submaps. The optimised map is rendered through a combined TSDF view, without explicitly constructing that map.Source: §7.2 · p. 17

7.3 Surfel-Based Reconstruction

  • The surfel implementation incrementally matches projected scene surfels to incoming points and updates matches by weighted averaging; it is still beta and omits hierarchical region growing.Surfel support is capped at 5 million points, is less optimised than voxel reconstruction, and supports neither mesh generation nor loop closure.Source: §7.3 · pp. 17–18
Loading 1708.00783v1…