Source-linked AI summary
DeepTAM: Deep Tracking and Mapping
Huizhong Zhou, Benjamin Ummenhofer, Thomas Brox
TL;DR
DeepTAM addresses limited and dataset-sensitive learning-based solutions for camera tracking and dense mapping. It learns incremental keyframe tracking with synthetic viewpoints and combines cost-volume measurements with image-based priors for mapping. The system generalizes to new datasets and compares favorably with strong classical and deep-learning algorithms across tracking and dense mapping.
Problem
Learning-based tracking and mapping remain limited by suitable datasets, whose missing motion modes can cause overfitting and poor generalization to new data.
Method
DeepTAM uses incremental image-to-keyframe pose estimation with synthetic viewpoints and a cost-volume mapping network that combines depth measurements with image-based priors.
Results
DeepTAM generalizes to new datasets and compares favorably with state-of-the-art RGB-D tracking and strong classic and deep learning dense-depth algorithms.
Takeaways & Limitations
The approach provides learned full 6 DOF keyframe pose tracking and dense mapping that can process arbitrary many frames at interactive frame rates.
Takeaways & Limitations
Generalization depends on diverse training data and architecture choices designed to prevent overfitting; evaluation includes datasets not used for training and no-finetuning transfer.
Abstract
from arXiv · showhide
We present a system for keyframe-based dense camera tracking and depth map estimation that is entirely learned. For tracking, we estimate small pose increments between the current camera image and a synthetic viewpoint. This significantly simplifies the learning problem and alleviates the dataset bias for camera motions. Further, we show that generating a large number of pose hypotheses leads to more accurate predictions. For mapping, we accumulate information in a cost volume centered at the current depth estimate. The mapping network then combines the cost volume and the keyframe image to update the depth prediction, thereby effectively making use of depth measurements and image-based priors. Our approach yields state-of-the-art results with few images and is robust with respect to noisy camera poses. We demonstrate that the performance of our 6 DOF tracking competes with RGB-D tracking algorithms. We compare favorably against strong classic and deep learning powered dense depth algorithms.
1 Introduction
DeepTAM extends learning-based visual SLAM to camera pose tracking and dense mapping, addressing dataset bias through incremental tracking and combining depth measurements with image-based priors. The learned system generalizes to new datasets and compares favorably with strong classical and learned methods.
- DeepTAM applies deep learning to the two central visual SLAM components: camera pose tracking and dense mapping.
- Generating multiple pose hypotheses improves camera pose accuracy over direct pose prediction.
- Its tracking network incrementally aligns the current image with a keyframe through coarse-to-fine pose refinement and virtual keyframe updates.The incremental formulation simplifies learning and reduces dataset-bias effects.
- The mapping network combines multi-image cost-volume measurements with image-based priors to estimate and iteratively refine dense depth.Refinement uses a narrow band around the previous surface estimate.
- DeepTAM is designed to avoid overfitting while integrating diverse visual cues, contrasting with classical methods based on handcrafted features and photoconsistency.
- The system generalizes to new datasets, compares favorably with state-of-the-art RGB-D tracking, uses less data, and supports arbitrary frame counts at interactive rates.
2 Related work
DeepTAM builds on dense keyframe tracking and mapping ideas while replacing handcrafted or classical components with learned networks. Its tracking formulation targets full 6 DOF motion and addresses dataset-motion bias through incremental estimation and rendered virtual viewpoints.
- DeepTAM implements both tracking and mapping with deep networks that learn the task directly from data, following the generic dense keyframe idea of DTAM.
- Unlike DeMoN, DeepTAM processes more than two images, uses keyframes to avoid drift, and refines depth as additional frames arrive.
- Prior learning-based work often focuses on KITTI's predominantly planar 3 DOF driving motion, whereas DeepTAM performs full 6 DOF tracking and evaluates on a 6 DOF benchmark.
- DeepTAM reduces motion-learning difficulty by predicting small pose increments relative to rendered virtual keyframes generated from pose guesses.
- The pose branch generates multiple hypotheses, while an auxiliary optical-flow task encourages features that exploit the relationship between the current image and virtual keyframe.
- The tracking network aligns the current image with a keyframe's color and depth using a coarse-to-fine stack that incrementally refines pose estimates.
4 Mapping
DeepTAM’s mapping network combines multi-image photoconsistency costs with image-based priors to estimate dense keyframe depth. A fixed-band stage provides coarse estimates, while an adaptive narrow band refines them with higher sampling density and regularization.
- Cost volume construction: DeepTAM accumulates photoconsistency information from multiple images in a cost volume and uses a convolutional network to extract the depth map.The costs compare 3×3 keyframe patches with warped images using relative poses and candidate depths.
- Cost volume construction: The network combines cost-volume measurements with image-based scene priors to produce more accurate and robust depth estimates than direct minimum-cost selection.This learned alternative replaces handcrafted regularization and optimization techniques used in classic methods.
- Narrow-band refinement: The adaptive narrow band centers depth labels at the previous estimate, increasing sampling density while keeping the number of labels constant.Its width is controlled by σ_nb, and the cost volume is recomputed for selected frames to search for a better estimate.
- Narrow-band refinement: The narrow-band stage recovers finer depth details but requires good initialization and regularization to remain stable.A refinement network using the previous depth estimate produces better-regularized and more stable predictions than measurement-only updates.
- Fixed-band estimation: The fixed-band module samples 32 depth labels across the full range [0.01, 2.5] and processes the keyframe image with its cost volume.The fixed-band cost volume uses evenly spaced labels over the whole depth range.
- Network architecture: The mapping architecture uses multiple encoder-decoder networks and can be interpreted as alternating data and smoothness updates.Training proceeds progressively from the fixed-band module to jointly trained unrolled narrow-band iterations.
- Training: Training uses diverse indoor, outdoor, synthetic, and reconstructed datasets, with dataset poses used to construct the cost volume.The objective combines L1 inverse-depth loss with a scale-invariant gradient loss emphasizing depth discontinuities.
5 Experiments
Experiments evaluate DeepTAM's tracking and mapping against RGB-D, classical, and learned baselines, including ablations, generalization, and pose-noise robustness. DeepTAM performs strongly with few frames, improves through multi-frame and iterative depth refinement, and remains robust to noisy poses.
- Tracking evaluation: DeepTAM outperforms a state-of-the-art RGB-D method on most benchmark sequences despite using less information.Multiple pose hypotheses further reduce translational drift on most sequences, and no benchmark sequences are used for training or finetuning.
- Mapping evaluation: Accumulating information from multiple frames improves fixed-band depth estimates, with performance saturating after six or more frames.The largest qualitative improvement occurs between two and six frames.
- Mapping evaluation: Iterative narrow-band refinement improves depth estimates, converges after about three iterations, and can worsen SUN3D results beyond three iterations.The narrow band increases sampling density around the previous depth estimate.
- Mapping evaluation: DeepTAM achieves the best performance on all mapping metrics and test sets against DeMoN, DTAM, and SGM.The comparison uses sequences of length 10, while DeMoN receives only the first and last frames.
- Mapping evaluation: DeepTAM's combined tracking and mapping generalizes without benchmark training and is more robust and accurate than CNN-SLAM on most sequences.It performs clearly worse on fr1/plant but seldom fails and yields more reliable trajectories overall.
- Mapping evaluation: Qualitative comparisons show DeepTAM producing high-quality depth maps with few frames and capturing more details than competing methods.DTAM and SGM struggle with short sequences or textureless scenes, while DeMoN misses many details.
6 Conclusions
DeepTAM presents learned architectures for real-time dense tracking and mapping. Its tracking uses synthetic viewpoints for incremental keyframe alignment, while its mapping combines cost-volume information with image-based priors to produce accurate and robust depth estimates.
- 6 Conclusions: DeepTAM proposes a novel deep learning architecture for real-time dense mapping and tracking.The system targets the two principal components of visual SLAM: camera pose tracking and dense mapping.
- 6 Conclusions: Synthetic viewpoints enable incremental tracking with respect to a keyframe.The approach estimates camera pose through incremental alignment rather than directly learning unrestricted camera motion.
- 6 Conclusions: Combining cost-volume information with image-based priors yields accurate and robust dense depth estimates.The paper reports that the methods outperform strong classic and deep learning algorithms.
1 Tracking Network Implementation Details
The tracking implementation feeds the current image and a rendered virtual keyframe into convolutional networks. Rendering visible and occluded content with complementary depth tests supplies the network with additional image and depth information.
- 1 Tracking Network Implementation Details: The tracking ConvNets take the current camera image and a rendered virtual keyframe as input.The virtual keyframe contains rendered image and depth information.
- 1 Tracking Network Implementation Details: Two rendered image-depth pairs expose both visible content and occluded parts to the tracking network.The GREATER depth test produces the visible image and inverse-depth map, while LESS reveals occluded content.
2 Mapping Network Implementation
The mapping module is implemented with convolutional networks whose operations and parameters are summarized for the mapping architecture.
- 2 Mapping Network Implementation: Figure 3 documents the operations and parameters of the mapping module's ConvNets.The mapping architecture includes fixed-band and narrow-band processing stages elsewhere described in the paper.
3 Runtimes
The system uses a multi-level tracking network and two-stage mapping architecture. Tracking runs in real time, while mapping modules are invoked after sufficient frames accumulate.
- Runtime: Tracking runs in real time, whereas fixed-band and narrow-band mapping modules are invoked only after enough frames have been collected.The cost volume continues collecting information from incoming frames between mapping updates.
- Tracking: Tracking uses a shared encoder-decoder configuration across three resolution levels, with level-specific pose-hypothesis parameters.The listed resolution levels are 60 × 80, 120 × 160, and 240 × 320.
- Mapping: The fixed-band mapping module combines a keyframe image with a 32-label cost volume to predict an interpolation factor for depth estimation.The depth labels span [0.01, 2.5] and are equally spaced.
- Mapping: The narrow-band module refines depth using the current depth estimate and keyframe image after processing narrow-band matching information.A second encoder-decoder pair produces the refined depth Dnb2.
- Runtime: Table 1 reports component runtimes, including tracking, cost-volume computation, and mapping-module execution.The tracking statistic excludes outliers, and isolated tracking runs at about 44 Hz.
4 Results
The results section evaluates both tracking and mapping on the TUM RGB-D benchmark and provides qualitative comparisons with established dense-depth methods.
- Benchmark evaluation: The evaluation includes freiburg1 sequences with public ground truth and validation sets with secret ground truth.The extended evaluation is reported on the TUM RGB-D benchmark.
- Benchmark evaluation: The benchmark evaluation covers both the tracking component and the combined mapping-and-tracking system.These are presented as separate portions of Table 2.
5 Generalization
The authors address overfitting through diverse training and testing data and evaluate generalization on TUM RGB-D and KITTI. KITTI differs substantially from the training data and uses a distinct evaluation setup.
- Generalization strategy: Training and testing use diverse indoor, outdoor, realistic, and artificial datasets to reduce overfitting.The network architecture is also designed carefully to avoid overfitting.
- TUM RGB-D: TUM RGB-D was not included in the training data and is used to evaluate generalization on the benchmark.Table 2 reports results for this unseen dataset.
- Evaluation metrics: Table 2 evaluates tracking and combined mapping-and-tracking using translational RMSE in [m/s].The tracking comparison includes the RGB-D SLAM method of Kerl et al., using frame-to-keyframe odometry results.
- Evaluation setup: Figure 3 compares qualitative predictions from 10-frame sequences, while DeMoN uses only the first and last frame.The comparison concerns the mapping component.
- KITTI: KITTI provides a substantially different generalization setting because it contains urban scenes captured with a wide-angle camera.The setup uses five left-camera color frames for SGM, DTAM, and the proposed method, but only the first and last frame for DeMoN.