PaperlayerView exact revision on arXiv

Source-linked AI summary

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

arXiv:1604.01093v3cs.GRcs.CV

TL;DR

Large-scale RGB-D scanning needs real-time operation without sacrificing global consistency, robust tracking, or dense surface quality. BundleFusion globally optimizes poses from the complete RGB-D history with hierarchical sparse-to-dense alignment and continuously re-integrates the model. The resulting system supports real-time correction and relocalization, with quality and completeness reported as competitive with offline methods and online systems at higher speed.

  • Problem

    Large-scale RGB-D reconstruction is challenged by pose drift, slow online correction, brittle tracking, and point-based representations that limit scan quality and applicability.

  • Method

    BundleFusion hierarchically globally optimizes camera poses from the complete RGB-D history using sparse features plus dense geometric and photometric alignment, then updates the reconstruction continuously.

  • Results

    The framework provides globally consistent real-time reconstruction with robust tracking, relocalization, and quality and completeness reported as on par with offline methods.

  • Takeaways & Limitations

    BundleFusion offers a comprehensive real-time scanning solution for large indoor environments that can be operated with little expertise.

  • Takeaways & Limitations

    Longer sequences would require more than two hierarchy levels or spatial clustering to keep optimization real time.

Abstract

from arXiv · show

Real-time, high-quality, 3D scanning of large-scale scenes is key to mixed reality and robotic applications. However, scalability brings challenges of drift in pose estimation, introducing significant errors in the accumulated model. Approaches often require hours of offline processing to globally correct model errors. Recent online methods demonstrate compelling results, but suffer from: (1) needing minutes to perform online correction preventing true real-time use; (2) brittle frame-to-frame (or frame-to-model) pose estimation resulting in many tracking failures; or (3) supporting only unstructured point-based representations, which limit scan quality and applicability. We systematically address these issues with a novel, real-time, end-to-end reconstruction framework. At its core is a robust pose estimation strategy, optimizing per frame for a global set of camera poses by considering the complete history of RGB-D input with an efficient hierarchical approach. We remove the heavy reliance on temporal tracking, and continually localize to the globally optimized frames instead. We contribute a parallelizable optimization framework, which employs correspondences based on sparse features and dense geometric and photometric matching. Our approach estimates globally optimized (i.e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i.e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework. Our approach outperforms state-of-the-art online systems with quality on par to offline methods, but with unprecedented speed and scan completeness. Our framework leads to a comprehensive online scanning solution for large indoor environments, enabling ease of use and high-quality results.

1 INTRODUCTION

BundleFusion targets accessible, large-scale real-time scanning by combining globally consistent pose alignment with dense volumetric reconstruction and continuous model updates. Its framework addresses scalability, tracking robustness, global consistency, and high-quality surface modeling in one end-to-end system.

  • 1 INTRODUCTION: Large-scale real-time scanning must provide high-quality surfaces, scalability, global model consistency, robust tracking, and on-the-fly model updates.The requirements include continuous-surface representations, room-scale acquisition, drift correction, relocalization, and updating models after pose refinement.
  • 1 INTRODUCTION: BundleFusion addresses the lack of a single real-time approach covering these requirements through an end-to-end framework using the complete RGB-D history.Camera trajectories are globally optimized with an efficient local-to-global hierarchical strategy.
  • 1 INTRODUCTION: Figure 1 presents real-time global pose alignment with dense volumetric reconstructions whose quality and completeness were previously attainable only offline.The figure caption characterizes the approach as both real-time and dense rather than point-based.
  • 1 INTRODUCTION: Global correlation handles loop closure implicitly, supports instantaneous relocalization after failures, and allows repeated revisits while reconstruction quality improves.The authors report robustness to interrupted scans, different viewpoints, and novice large-scale scanning.
  • 1 INTRODUCTION: Sparse RGB features provide coarse global pose estimates, followed by dense photometric and geometric matching for fine-scale alignment.The fully parallelizable sparse-then-dense framework preserves global structure while improving local reconstruction accuracy.
  • 1 INTRODUCTION: The system continuously updates the volumetric model from refined global poses, enabling real-time correction and revisiting of scanned areas.Its RGB-D re-integration strategy supports on-the-fly model updates when optimized poses become available.

2 RELATED WORK

Prior reconstruction systems trade off scalable volumetric modeling, global consistency, real-time operation, tracking robustness, and dense surface quality. BundleFusion’s pose optimization combines RGB-D correspondences and local/global alignment to address these gaps without the principal limitations of earlier online methods.

  • 2 RELATED WORK: Volumetric implicit methods support continuous surfaces but traditionally scale poorly because they rely on uniform grids.Sparse TSDF-based spatial subdivision improves volumetric fusion at scale, but pose drift can still distort the model.
  • 2 RELATED WORK: Offline globally consistent RGB-D reconstruction optimizes entire pose trajectories but requires minutes or hours, preventing real-time revisiting or refinement.These methods require access to all input frames before correction.
  • 2 RELATED WORK: BundleFusion’s global pose optimization uses RGB-D input, pairwise correspondences, and local and global alignment steps to compute per-frame poses.Its stated strategy combines sparse and dense correspondences for global alignment at real-time rates.
  • 2 RELATED WORK: Monocular SLAM methods achieve real-time drift-free pose estimation but do not generate the detailed dense 3D models targeted here.Their optimization commonly minimizes reprojection error through pose-graph optimization or bundle adjustment.
  • 2 RELATED WORK: Online RGB-D systems improve pose and model correction but may be slow, trajectory-constrained, fusion-order constrained, or limited to point-based representations.The reported limitations include seconds or minutes of optimization, explicit loop-closure assumptions, inability to refine models afterward, and reduced applicability of point representations.

3 METHOD OVERVIEW

BundleFusion combines efficient global pose optimization with large-scale real-time reconstruction, continuously updating the model as pose estimates improve. It avoids strict temporal dependence to support relocalization, free-form motion, and revisiting.

  • 3 METHOD OVERVIEW: The framework jointly performs global pose optimization and large-scale real-time 3D reconstruction.The two components operate in unison, with reconstruction updates driven by newly computed pose estimates.
  • 3 METHOD OVERVIEW: Sparse feature correspondences provide coarse global alignment, followed by dense photometric and geometric optimization for refinement.Sparse features support loop closure detection and relocalization before dense alignment improves pose accuracy.
  • 3 METHOD OVERVIEW: A hierarchical local-to-global optimization reduces the number of unknowns per step and enables scaling to large scenes.The strategy optimizes within chunks and then aligns chunks globally.
  • 3 METHOD OVERVIEW: A data-parallel GPU solver addresses the nonlinear alignment problem using the framework’s sparse structure.The solver is designed for the parallelizable optimization framework.
  • 3 METHOD OVERVIEW: Continuous pose changes trigger symmetric on-the-fly reintegration of RGB-D frames into the volumetric scene representation.Frames are removed at old poses and reintegrated at improved poses to update the reconstruction.

4 GLOBAL POSE ALIGNMENT

Global pose alignment estimates camera transforms from sparse frame correspondences and dense image constraints, using filtering and hierarchical nonlinear optimization. The design supports global structure, fine-scale alignment, and robustness to invalid matches.

  • 4 GLOBAL POSE ALIGNMENT: The system estimates rigid camera transforms that align all RGB-D frames in a globally consistent coordinate system.The first frame defines the world coordinate system, and each transform maps camera coordinates into world space.
  • 4 GLOBAL POSE ALIGNMENT: Sparse correspondences are filtered using geometric, photometric, stability, and surface-area checks before pose optimization.Invalid matches are removed when transforms are unstable, reprojection errors are high, or correspondence coverage is insufficient.
  • 4 GLOBAL POSE ALIGNMENT: Hierarchical optimization divides long RGB-D sequences into overlapping 11-frame chunks, then globally aligns representative keyframes.This local-to-global structure is designed to run at real-time rates on sequences containing tens of thousands of frames.
  • 4 GLOBAL POSE ALIGNMENT: The alignment objective combines sparse feature, dense photometric, and dense geometric constraints in a nonlinear least-squares formulation.The sparse term establishes global structure, while dense terms refine local alignment.
  • 4 GLOBAL POSE ALIGNMENT: A GPU-based Gauss-Newton solver optimizes camera parameters for long sequences with more than twenty thousand frames.The solver uses first-order derivatives and is tailored to the sparsity pattern of the global alignment objective.
  • 4 GLOBAL POSE ALIGNMENT: Verification and optimization pruning filter out the vast majority of false loop closures and remove the remainder.Table 2 reports the effects of these filtering steps.

5 DYNAMIC 3D RECONSTRUCTION

Dynamic reconstruction keeps the volumetric model synchronized with continually changing optimized poses. Frames can be de-integrated from old poses and reintegrated at refined poses to correct accumulated drift while preserving rapid feedback.

  • 5 DYNAMIC 3D RECONSTRUCTION: The 3D model is updated from newly optimized camera poses through continuous integration and de-integration of frames.This corrects volumetric errors from accumulated drift or dead reckoning as better poses become available.
  • 5 DYNAMIC 3D RECONSTRUCTION: Scene geometry is represented as a sparse voxel-hashed TSDF that scales to large-scale surface reconstruction.Empty space does not need to be represented or processed in the sparse volumetric representation.
  • 5 DYNAMIC 3D RECONSTRUCTION: Depth integration updates each voxel’s signed distance and weight using the incoming frame’s projective distance and integration weight.The update combines the existing voxel state with measurements from the new depth frame.
  • 5 DYNAMIC 3D RECONSTRUCTION: De-integrating a frame reverses its contribution so the frame can be reintegrated using a new pose.This operation adapts existing surface measurements to continuously changing pose estimates.
  • 5 DYNAMIC 3D RECONSTRUCTION: Each frame stores depth and color data together with its integrated pose and continually changing optimized pose.The integrated pose controls the current reconstruction, while the optimized pose records the latest optimization result.
  • 5 DYNAMIC 3D RECONSTRUCTION: Incoming frames receive an initial transform by combining frame-to-frame correspondence information with the newest available optimized pose.This provides rapid integration feedback even when global optimization has not yet produced a pose for the current frame.

6 RESULTS

The system achieves real-time, globally consistent reconstruction across large indoor scenes, combining robust tracking and model updates with high-quality geometry and texture. Evaluations show strong accuracy, completeness, recovery from failures, and performance comparable to offline reconstruction.

  • Qualitative Comparison: Compared with offline Redwood, the method delivers comparable or better reconstruction quality and more small-scale detail at real-time rates instead of hours-long processing.Redwood requires 2.3h–13.2h per sequence in the reported comparisons.
  • Performance and Convergence: Well beyond 30Hz throughput is achieved for all shown test sequences, using two GPUs for volumetric reconstruction, correspondence search, and global pose optimization.The global dense optimization runs in under 500ms at sequence end, and only a few iterations are needed after adding a global keyframe.
  • Recovery from Tracking Failure: Global matching detects tracking loss and enables recovery after sensor occlusion, featureless regions, interrupted scans, or continuation from a completely different location.When alignment fails, surface measurements are not integrated; the user is shown a gray overlay and can recover by moving to previously scanned areas.
  • Loop Closure Detection and Handling: Continuous integration and de-integration of surface measurements handles loop closures transparently while incrementally correcting the volumetric model.This avoids requiring explicit loop-closure detection and supports repeated revisiting of scene regions.
  • Quantitative Comparison: The method outperforms state-of-the-art online systems in trajectory accuracy, scan completeness, and alignment, while producing a high-quality surface mesh rather than only a point cloud.On the evaluated ICL-NUIM and RGB-D benchmarks, the full sparse-and-dense approach achieves the highest accuracy among the tested design variants.
  • Limitations: The current implementation is limited to scans of up to 25,000 RGB-D frames and requires two GPUs, while larger sequences would need additional hierarchy levels or spatial clustering.At 30Hz, the frame limit corresponds to approximately 14 minutes of continuous scanning.

7 ADDITIONAL EVALUATION

Additional evaluations cover qualitative reconstruction, quantitative tracking and surface accuracy, frame registration, and SIFT runtime across indoor and synthetic datasets.

  • Additional Qualitative Results: Our method maintains real-time rates, while Redwood requires 2.6–13.2 hours across listed scenes and cannot handle Apt 2 relocalization.ElasticFusion lacks some results because dense frame-to-model methods cannot handle occasional frame jumps in the streaming setup.
  • Additional Qualitative Results: The evaluation includes SUN3D and NYU2 indoor-scene reconstructions, including all 464 NYU2 scenes and SUN3D scenes without annotations.SUN3D comparisons use SUN3Dsfm and the proposed approach; NYU2 results are shown for several reconstructions.
  • Additional Quantitative Results: On augmented ICL-NUIM, trajectory performance is on par with or better than existing state of the art, while loop closures improve stability except in Office 1.Office 1 is difficult because part of the loop crosses a wall with little to no color features.
  • Additional Quantitative Results: The method registers the vast majority of frames, with rejected frames mainly associated with conservative correspondence filtering, sensor occlusions, or untextured walls.Apt 2 has more unregistered frames because the sensor is occluded during relocalization demonstrations.
  • Additional Quantitative Results: GPU SIFT processing detects about 150 features per frame and 250 per keyframe, with detection time around 6.4ms at 1296 × 968 resolution.Detection time includes descriptor computation; matching time is reported per image pair and parallelized.

8 CONCLUSION

The paper presents an online real-time reconstruction system that globally optimizes camera poses and updates the model through dynamic integration and de-integration. It reports robust tracking, implicit loop closure, and reconstruction quality and completeness previously associated with offline approaches.

  • 8 CONCLUSION: The framework globally optimizes the trajectory for every captured frame, providing robust tracking and implicitly solving loop closure.Its pose optimization combines online SIFT processing with sparse and dense correspondences in a parallel nonlinear framework.
  • 8 CONCLUSION: Dynamic integration and de-integration update the reconstruction as optimized pose estimates continuously change.The reconstruction is updated from the continuously changing stream of optimized poses.
  • 8 CONCLUSION: The demonstrated reconstructions achieve quality and completeness previously possible only with offline approaches and tedious capture sessions.The conclusion connects global accurate tracking to immersive AR/VR and high-fidelity handheld 3D reconstruction.
Loading 1604.01093v3…