Source-linked AI summary
DeepFactors: Real-Time Probabilistic Dense Monocular SLAM
Jan Czarnowski, Tristan Laidlow, Ronald Clark, Andrew J. Davison
TL;DR
Monocular SLAM must estimate camera motion and rich geometry despite differing map representations, consistency metrics, and uses of learned priors. DeepFactors addresses this with a learned compact depth representation and photometric, reprojection, and geometric factors in a probabilistic factor-graph system. Combining all three factors achieves the best trajectory and reconstruction results while maintaining real-time performance.
Problem
Monocular SLAM approaches differ in geometry representation, consistency metrics, and learned-prior use, motivating a unified framework for estimating camera motion and dense geometry.
Method
DeepFactors jointly optimises learned compact depth codes and camera poses in a probabilistic factor graph using photometric, reprojection, and geometric consistency factors.
Results
Combining all three factors achieves the best trajectory and reconstruction results, while the system outperforms CodeSLAM on all sequences and CNN-SLAM on all but one.
Takeaways & Limitations
A standard factor-graph framework supports dense probabilistic SLAM and straightforward integration of different sensor modalities while retaining real-time performance.
Abstract
from arXiv · showhide
The ability to estimate rich geometry and camera motion from monocular imagery is fundamental to future interactive robotics and augmented reality applications. Different approaches have been proposed that vary in scene geometry representation (sparse landmarks, dense maps), the consistency metric used for optimising the multi-view problem, and the use of learned priors. We present a SLAM system that unifies these methods in a probabilistic framework while still maintaining real-time performance. This is achieved through the use of a learned compact depth map representation and reformulating three different types of errors: photometric, reprojection and geometric, which we make use of within standard factor graph software. We evaluate our system on trajectory estimation and depth reconstruction on real-world sequences and present various examples of estimated dense geometry.
I. INTRODUCTION
DeepFactors targets the limitations of monocular SLAM by combining dense reconstruction, probabilistic inference, and learned geometric priors in a real-time system.
- Dense SLAM uses all image pixels and can improve robustness to motion blur, but photometric errors remain fragile when brightness constancy is violated.
- Learned monocular depth priors have been integrated into SLAM systems ranging from model-based pipelines with neural subcomponents to end-to-end prediction.
- DeepFactors builds and maintains a dense reconstruction while supporting probabilistic inference and combining advantages of different SLAM paradigms.
- DeepFactors is presented as the first real-time probabilistic dense SLAM system.
- The system integrates learned geometry priors with classical SLAM formulations through a probabilistic factor-graph formulation.
II. RELATED WORK
Prior systems explored learned compact representations, learned priors, or dense reconstruction, but often lacked full-SLAM capabilities, real-time operation, or generalisation. DeepFactors builds a unified framework combining learned and model-based methods with dense and sparse localisation and mapping.
- Learned representations: CodeSLAM introduced compact optimisable representations for dense structure-from-motion, but its system lacked full SLAM, real-time performance, and generalisation to handheld scenes.Its feasibility was demonstrated through windowed 3D reconstruction and visual odometry.
- DeepFactors: DeepFactors extends the CodeSLAM idea with a new mapping backend, sparse geometric and reprojection errors, and SLAM design choices including keyframing and map maintenance.The system also adds local and global loop closure and relocalisation.
- Comparable systems: CNN-SLAM provides a directly comparable real-time full-SLAM baseline by combining LSD-SLAM with learned priors and supporting large-scale mapping and loop closures.Its operating principle differs substantially from DeepFactors.
- Learned reconstruction: Other learned systems focus on multi-frame dense reconstruction rather than fully featured SLAM, while BA-Net optimises predicted basis depth maps within bundle adjustment.BA-Net includes the optimisation in end-to-end training, according to the related-work discussion.
- Learned tracking and mapping: DeepTAM replaces DTAM's TV-L1 optimisation and camera tracking with a deep convolutional network, but retains the tracking-mapping split and is not real-time.The passage also notes that DeepTAM addressed generalisation but relied on seeing all required scene content.
- Problem formulation: The DeepFactors formulation represents each frame with a 6DoF pose and compact code, then minimises pairwise consistency losses to estimate dense geometry and camera poses.The scene geometry is formed from depth maps decoded from the per-frame codes.
- Unifying framework: The paper aims to unify learned and model-based methods with dense and sparse localisation and mapping in a single SLAM framework.It evaluates the system on real-world sequences and presents estimated dense geometry.
III. CODE BASED OPTIMIZATION
DeepFactors represents each frame’s dense depth map with a learned, image-conditioned compact code, making joint geometry optimization more tractable. The resulting consistency objectives become pairwise constraints in a factor graph for estimating scene geometry and camera motion.
- III. CODE BASED OPTIMIZATION: Depth maps are parameterized by compact codes rather than independently optimized pixels, reducing parameters and ill-posedness.The reconstructed geometry is a set of frame-specific depth maps, while code optimization correlates groups of depth pixels.
- III. CODE BASED OPTIMIZATION: The depth map is modeled as Di = f(ci, Ii) = D0_i + J(Ii)ci, a linear relation between code, image, and depth.D0_i is the depth decoded from an all-zero code, and J(Ii) is the image-conditioned Jacobian.
- III. CODE BASED OPTIMIZATION: Perturbing latent-code elements affects different depth-image regions, producing structured correlations across pixels.Figure 3 visualizes the pixels affected by perturbations of different code elements.
- III. CODE BASED OPTIMIZATION: Photometric, reprojection, and geometric objectives provide pairwise factor-graph constraints for jointly estimating geometry and camera motion.The objectives enforce consistency between overlapping camera frames.
A. Photometric Factor
The system uses image warping to compare observations across frames through photometric and reprojection consistency. These factors use decoded depth, relative camera motion, and matched image features to constrain optimization.
- A. Photometric Factor: Photometric error compares source-frame intensities with target-frame intensities after warping into the source frame.The warp uses depth and the relative transformation between the two frames.
- A. Photometric Factor: The warp maps pixel coordinates using projection, reprojection, decoded depth Di(x), and the relative 6DoF transformation Tji.The transformation is defined from frame i to frame j.
- A. Photometric Factor: Indirect reprojection error measures differences between observed and hypothesized locations of matched landmark observations.Matched features are represented by Mij, and BRISK detects and describes the image key points.
- A. Photometric Factor: A Cauchy robust cost gives mismatched features a constant response to outliers.The reprojection factor uses salient image features matched between frames.
C. Sparse Geometric Factor
The sparse geometric factor compares one frame’s depth map with another frame’s depth after warping, while the learned network supplies a compact, image-conditioned depth representation and uncertainty.
- C. Sparse Geometric Factor: Geometric consistency compares depth map Dj with depth map Di warped into frame j.The formulation uses the z component of the transformed 3D point at the warped pixel location.
- C. Sparse Geometric Factor: The geometric loss uses a Huber robust cost and evaluates only a sparse, uniformly sampled set of pixels to save computation.Different pixel subsets may be sampled across iterations to stochastically cover the whole image.
- C. Sparse Geometric Factor: The depth network uses a feature-conditioned variational auto-encoder to learn an optimizable compact representation.Its depth decoder omits nonlinear activations so reconstructed depth remains linear in the code.
- C. Sparse Geometric Factor: A separate encoder predicts an optimal code from the input image, improving initial depth predictions in real-camera experiments.The predicted code is constrained to the same space as the learned code.
- C. Sparse Geometric Factor: The Feature Network predicts uncertainty for a multi-resolution negative-log Laplacian loss, while predicted depth is supervised with an L1 loss.These losses train the depth representation and its uncertainty estimate.
V. SYSTEM
The system maintains a keyframe map and incrementally optimizes a batch MAP factor graph as new observations arrive. One-way frames add many views cheaply, supporting refinement while tracking keeps pace with incoming images.
- V. SYSTEM: Incoming images are corrected and tracked against the nearest keyframe, with new keyframes added when baseline and other criteria are sufficient.Each new keyframe receives an estimated pose and an initial code prediction.
- V. SYSTEM: The batch MAP factor graph connects each new keyframe to the last N keyframes and is solved incrementally for real-time performance.Optimization occurs whenever new observations are introduced.
- V. SYSTEM: Tracking and joint mapping alternate after keyframe creation, with interleaved optimization steps helping the system keep up with incoming frames.The graph is optimized for a set number of iterations or until convergence.
- V. SYSTEM: One-way frames provide low- and high-baseline views to refine the latest keyframe without attaching depth to every frame.They are marginalized and removed after a set number of optimization steps, keeping computation inexpensive.
- V. SYSTEM: The system’s components are described in the subsequent sections.
A. Camera Tracking
DeepFactors tracks each frame against a keyframe and jointly optimises keyframe poses and compact depth codes in a factor graph. New keyframes receive network-predicted depth initialisation, with multi-frame initialisation available when single-frame prediction fails.
- A. Camera Tracking: Each camera frame is tracked against the closest keyframe using direct whole-image SE3 Lucas-Kanade.
- A. Camera Tracking: Keyframes contain pose and code variables connected by photometric, reprojection, or geometric pairwise consistency factors.
- A. Camera Tracking: Batch optimisation updates all keyframes using factor-graph software and incremental iSAM2 relinearisation.
- A. Camera Tracking: One-way frames are made leaves in the clique tree so they can be marginalised.
- A. Camera Tracking: An explicit code-prediction network supplies each new keyframe's initial code, which the depth decoder converts into an initial depth map.
- A. Camera Tracking: When single-image prediction fails, the system can create one keyframe per input frame and optimise the initial graph before starting.
D. Loop Closure
DeepFactors detects local and global loops using pose criteria and bag-of-words candidates, then adds constraints to improve graph consistency. Its ScanNet training uses merged sensor and rendered depths across about 1.4M images.
- D. Loop Closure: Within the last 10 keyframes, pose-based loop detection adds pairwise constraints and tightens local reconstructions.
- D. Loop Closure: Global loop candidates outside the active window are found with bag of words, then filtered by tracking inliers and pose distance.
- D. Loop Closure: The network was trained on about 1.4M ScanNet images using merged raw sensor depths and rendered depths from full-scene models.
- D. Loop Closure: Training used 13 epochs, learning rate 0.0001, image size 256 × 192, and code size 32.
B. Ablation Studies
The ablation compares photometric, geometric, and reprojection factors for trajectory and reconstruction quality. Adding either auxiliary factor improves both objectives, while combining all three performs best.
- B. Ablation Studies: The evaluation compares photometric, geometric, and reprojection factors across trajectory and reconstruction errors.
- B. Ablation Studies: Adding either geometric or reprojection factors reduces trajectory error and increases reconstruction accuracy.
- B. Ablation Studies: Reprojection factors typically influence trajectory error more strongly, whereas geometric factors influence reconstruction accuracy more strongly.
- B. Ablation Studies: Geometric error pins separate depth maps together in textureless areas by imposing a single-surface prior.
- B. Ablation Studies: Combining photometric, geometric, and reprojection factors achieves the best trajectory and reconstruction results.
- B. Ablation Studies: Depth accuracy is evaluated as the percentage of pixels within 10% of ground truth on ICL-NUIM and TUM sequences.
D. Trajectory Estimation
DeepFactors evaluates dense reconstruction and trajectory estimation against established systems on real-world sequences. It reports strong comparisons while maintaining real-time operation, with performance depending on configuration and scene motion.
- D. Trajectory Estimation: Trajectory evaluation uses ATE RMSE against CNN-SLAM, CodeSLAM, and DeepTAM, with some reported baselines taken from prior papers.
- D. Trajectory Estimation: DeepFactors outperforms CodeSLAM on every evaluated sequence and CNN-SLAM on all but one, while usually matching DeepTAM with real-time performance.
- D. Trajectory Estimation: Incoming images are tracked at around 250 Hz while whole-map mapping is interleaved with tracking.
- D. Trajectory Estimation: Interactive speed varies with graph connectivity, enabled factors, iSAM2 relinearisation, loop closures, and camera motion.
VIII. CONCLUSIONS
DeepFactors is presented as a real-time probabilistic dense SLAM system using a learned compact depth representation and combining classical SLAM with learned priors. The authors identify future improvements in code training, camera tracking, and implementation efficiency.
- DeepFactors combines classical SLAM with learned data priors in a standard probabilistic factor-graph framework.
- The system uses a learned compact depth map representation to support real-time probabilistic dense SLAM.
- Future work includes integrating structure-from-motion optimization into training of the compact depth code.
- The authors also propose unsupervised intensity-based code learning and a relative-pose prediction network to improve tracking robustness.
- Further performance work targets faster network-Jacobian computation and improved GPU implementation of the geometric factor.