Source-linked AI summary
CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, Andrew J. Davison
TL;DR
Real-time monocular SLAM needs dense geometric detail without the computational and inference burden of high-dimensional maps. The paper learns an intensity-conditioned depth representation with compact optimisable codes, demonstrating joint optimisation of motion and dense geometry in a targeted real-time monocular system.
Problem
Dense maps provide complete geometry but are costly for storage, processing, and rigorous probabilistic inference, while sparse representations capture only partial scene information.
Method
The paper conditions a depth auto-encoder on intensity images so compact codes represent dense scene geometry and can be jointly optimised with camera poses and overlapping keyframe codes.
Results
The paper implements a real-time targeted monocular system that achieves tight joint optimisation of motion and dense geometry.
Takeaways & Limitations
Image-conditioned learned codes provide generic, detailed dense scene information while allowing efficient probabilistic joint optimisation with camera poses.
Takeaways & Limitations
The demonstrated representations remain tied to single images within a keyframe-based approach, with broader learned compact 3D geometry left for future work.
Abstract
from arXiv · showhide
The representation of geometry in real-time 3D perception systems continues to be a critical research issue. Dense maps capture complete surface shape and can be augmented with semantic labels, but their high dimensionality makes them computationally costly to store and process, and unsuitable for rigorous probabilistic inference. Sparse feature-based representations avoid these problems, but capture only partial scene information and are mainly useful for localisation only. We present a new compact but dense representation of scene geometry which is conditioned on the intensity data from a single image and generated from a code consisting of a small number of parameters. We are inspired by work both on learned depth from images, and auto-encoders. Our approach is suitable for use in a keyframe-based monocular dense SLAM system: While each keyframe with a code can produce a depth map, the code can be optimised efficiently jointly with pose variables and together with the codes of overlapping keyframes to attain global consistency. Conditioning the depth map on the image allows the code to only represent aspects of the local geometry which cannot directly be predicted from the image. We explain how to learn our code representation, and demonstrate its advantageous properties in monocular SLAM.
1. Introduction
The paper addresses the tension between dense geometric detail and tractable probabilistic SLAM by learning a compact, image-conditioned representation that supports joint optimisation of motion and dense structure.
- Motivation: Dense maps capture more complete geometry but make storage, processing, and rigorous uncertainty inference costly, whereas sparse SLAM supports probabilistic inference with partial scene coverage.Dense or semi-dense systems often approximate inference by discarding cross-correlations and alternating pose and map optimisation.
- Motivation: Natural-scene geometry is highly structured, so depth can potentially be represented compactly rather than with a parameter for every element.Neighbouring depth values are correlated, while manually designed object and plane abstractions remain limited for natural, cluttered scenes.
- Approach: The proposed representation conditions a depth auto-encoder on intensity images, allowing compact codes to preserve reconstruction detail for generic dense scene geometry.The image supplies local details, while the learned code represents additional global shape parameters.
- Approach: A keyframe depth map becomes a function of its intensity image and compact code, enabling joint estimation of camera poses and dense depth across overlapping keyframes.The approach targets the keyframe-based SLAM paradigm and supports global consistency through code optimisation.
- Contribution: The paper combines learned depth representation with estimation-based SLAM to achieve tight joint optimisation of motion and dense geometry in a real-time monocular system.The stated contributions are the compact optimisable representation and its first real-time targeted monocular system implementation.
2. Intensity Conditioned Depth Auto-Encoding
The paper uses an intensity-conditioned variational auto-encoder to represent dense depth with a compact code while preserving reconstruction detail. The architecture predicts depth and uncertainty across multiple resolutions and is trained with a differentiable proximity parametrisation.
- Representation: A compact geometry representation is obtained by conditioning a depth auto-encoder on intensity images.The intensity image supplies information that would otherwise be lost through the auto-encoder bottleneck.
- Representation: The depth map is modeled as D(I, c), where image I provides local detail and code c represents remaining geometry.The variational design encourages smooth changes in depth when the code changes.
- Network architecture: The network combines an intensity U-Net with a variational depth auto-encoder whose features are concatenated at corresponding resolutions.The bottleneck uses fully connected layers and Gaussian latent sampling; the decoder outputs predicted depth mean and uncertainty.
- Training objective: Pixel-wise uncertainty is learned from intensity only and enters a Laplace negative log-likelihood that attenuates difficult regions.The objective balances the reconstruction error |d̃ − µ| against a log(b) regularisation term.
- Training objective: The loss is evaluated over a four-level depth pyramid, with each level weighted by a factor of 4 to account for lower pixel counts.This multi-resolution objective follows the depth-learning setup described in the paper.
- Training setup: The proximity parametrisation maps depths to [0, 1] using average depth a, treating near and far ranges with depth-like and inverse-depth-like mappings.It is differentiable and intended to better reflect the observable quantity.
- Training setup: Training uses photorealistic randomized indoor scenes from SceneNet RGB-D, with colour and depth images, ADAM, and six epochs.The learning rate starts at 10^-4 and is reduced to 10^-6.
3. Dense Warping
Dense warping uses a keyframe’s intensity image, estimated code, depth, and relative pose to establish pixel correspondences between views. The resulting expressions are differentiable, enabling photometric errors and Jacobians for joint SLAM optimisation.
- Correspondence: For each pixel in view A, the method projects its depth into view B using the relative transformation between the views.The correspondence uses projection and inverse-projection operators together with the depth map D(I_A, c_A).
- Photometric alignment: The warped correspondence can be applied to intensity images to derive a photometric error between views.The formulation uses the estimated depth of view A and the relative pose to sample view B.
- Jacobians: The warping expressions are differentiable with respect to their inputs, allowing the corresponding Jacobians to be computed.These derivatives include image gradients, projection differentials, transformation terms, and the depth derivative with respect to the code.
- Jacobians: The derivative of depth with respect to the code can cost up to 1 sec to evaluate, depending on network size.A linear decoder permits pre-computation of this term and can significantly accelerate evaluation.
4. Inference Framework
The framework jointly estimates dense geometry codes and camera motion from overlapping frames, using photometric and geometric residuals with iterative optimisation. It also supports tracking and keyframe-based SLAM, while implementation speed remains a limitation.
- N-Frame Structure from Motion: Each frame receives an unknown geometry code and pose, and overlapping frames generate photometric and geometric residuals.The residuals are used to connect frame appearances and estimated depths.
- Residual weighting: The loss functions mask invalid correspondences and robustly reweight errors, including geometric, photometric, slanted-surface, and occlusion effects.Occlusion downweighting applies only to the photometric loss.
- Joint optimisation: Jacobians are computed with respect to all codes and poses, then a damped Gauss-Newton algorithm jointly optimises them.Linear decoding permits computing each keyframe decoder Jacobian with respect to its code only once.
- Tracking: Tracking pairs the current frame with the last keyframe and estimates relative pose from cost minimisation without a geometric cost.The vision-only setup lacks the current frame’s depth image beyond a rough guess.
- SLAM system: The preliminary SLAM system alternates tracking and mapping, adds keyframes after sufficient baseline, globally optimises them, and marginalises old keyframes when necessary.Initialisation jointly optimises the relative pose and codes of two images.
- Training validation: Figure 5 compares validation losses across code sizes and decoder or image variants, with larger codes lowering loss and nonlinear decoding or colour having little effect.The reference setup uses greyscale images and a linear decoder.
- Implementation limitation: The implementation uses TensorFlow image warping and could be accelerated through more targeted warping and optimisation in future work.
5. Experimental Evaluation and Discussion
Experiments show that image-conditioned depth codes preserve scene detail, support refinement across overlapping keyframes, and generalise to real-world monocular SLAM settings.
- Image-Conditioned Depth Encoding: The reference network uses a 128-dimensional code, while colour inputs and nonlinear decoding do not significantly change reconstruction accuracy.Code size is evaluated alongside RGB information and nonlinear depth decoding.
- Image-Conditioned Depth Encoding: Zero-code reconstructions capture some geometric detail but miss the full scene, whereas encoded and optimised codes produce similar detailed depth estimates.The similarity between encoder-derived and optimised codes indicates effective encoding; uncertainty is higher near depth discontinuities and shiny regions.
- Image-Conditioned Depth Encoding: Individual code entries influence specific image regions, partly respect intensity boundaries, and show some consistency across similar images.The final reconstruction combines the effects of all code entries linearly.
- Structure from Motion: Adding overlapping keyframes continuously reduces depth reconstruction error, with refinement depending on new-view information and baseline.This low-dimensional optimisation space enables probabilistic joint inference over the connected frames.
- SLAM System: The system runs through most of EuRoC MH02 despite domain differences and reflections, using joint motion-geometry optimisation without IMU data.The evaluation uses a sliding-window visual odometry setup trained on SceneNet RGB-D.
6. Conclusions
The learned, image-conditioned depth representation provides generic and detailed dense geometry while supporting efficient joint optimisation with camera poses. The authors identify a full real-time keyframe SLAM system, real-data training, and representations beyond single-image keyframes as future directions.
- Image-conditioned learned depth codes represent generic, detailed dense scene geometry while remaining suitable for efficient joint optimisation with camera poses.
- A full real-time keyframe-based SLAM system remains future work using the demonstrated components.
- Longer-term work aims to move beyond representations tied to single images toward compact optimisable representations for general 3D geometry and object recognition.