Source-linked AI summary
iMAP: Implicit Mapping and Positioning in Real-Time
Edgar Sucar, Shikun Liu, Joseph Ortiz, Andrew J. Davison
TL;DR
Real-time dense SLAM needs a representation that is precise, efficient, predictive for unobserved regions, and flexible without prior training data. iMAP trains a single MLP online as the scene model while using it for tracking, and achieves real-time operation with compact, adaptive, complete room-scale reconstructions. The system also reports more complete reconstructions than standard dense SLAM methods and competitive tracking performance.
Problem
Real-time SLAM needs a precise, efficient, predictive, and flexible 3D representation that can operate without substantial prior training data.
Method
iMAP trains a randomly initialized MLP online as the sole occupancy-and-colour scene representation, using parallel tracking and mapping, keyframes, and information-guided pixel sampling.
Results
iMAP produces more complete scene reconstructions with a significantly smaller memory footprint than standard dense SLAM systems and competitive tracking performance on TUM RGB-D.
Takeaways & Limitations
An MLP trained from scratch can serve as the only scene representation in a real-time handheld RGB-D SLAM system while producing adaptive, watertight room-scale models.
Abstract
from arXiv · showhide
We show for the first time that a multilayer perceptron (MLP) can serve as the only scene representation in a real-time SLAM system for a handheld RGB-D camera. Our network is trained in live operation without prior data, building a dense, scene-specific implicit 3D model of occupancy and colour which is also immediately used for tracking. Achieving real-time SLAM via continual training of a neural network against a live image stream requires significant innovation. Our iMAP algorithm uses a keyframe structure and multi-processing computation flow, with dynamic information-guided pixel sampling for speed, with tracking at 10 Hz and global map updating at 2 Hz. The advantages of an implicit MLP over standard dense SLAM techniques include efficient geometry representation with automatic detail control and smooth, plausible filling-in of unobserved regions such as the back surfaces of objects.
1. Introduction
iMAP uses a randomly initialized MLP as the sole scene representation for real-time handheld RGB-D SLAM, training it online without prior data. Keyframes, parallel tracking and mapping, and information-guided sampling enable efficient room-scale reconstruction and tracking.
- iMAP is presented as the first real-time SLAM system using an MLP as its only scene representation with a handheld RGB-D camera.
- The MLP is trained from scratch during live operation to represent scene occupancy and colour while supporting camera tracking.
- Tracking runs at over 10 Hz while a parallel mapping process continually trains the MLP and jointly optimises historic keyframe poses.
- With around 10 - 20 keyframes and an MLP containing only 1 MB of parameters, iMAP maps whole rooms using continuous, adaptive resolution.
- Across eight Replica scenes, iMAP produces more complete reconstructions with a significantly smaller memory footprint and competitive TUM RGB-D tracking performance.
- Automated keyframe selection and loss-guided sparse active sampling enable incremental real-time training of the implicit scene network.
2. Related Work
Related work contrasts iMAP with layered, fixed-resolution, and offline approaches to dense scene representation. iMAP instead treats real-time SLAM with one MLP as an online continual-learning problem using replay through keyframes.
- Dense SLAM uses a unified scene representation for both camera tracking and detailed world modelling, avoiding separate sparse localisation layers.
- Fixed-resolution volumetric occupancy or signed-distance representations are memory-intensive, while hierarchical alternatives are more complex and provide limited levels of detail.
- Earlier learning-based dense reconstruction methods either use depth-map views, require an existing scan, or depend on previous training data.
- Prior implicit MLP methods addressed reconstruction, compression, view synthesis, scene completion, or pose optimisation mainly as offline processes requiring hours, days, or weeks.
- iMAP frames real-time SLAM as continual learning, balancing plasticity for new observations with stability against catastrophic forgetting.
- The method uses automatically selected keyframes as compressed replay memories and loss-guided sampling to reinforce previously observed regions.
3. iMAP: A Real-Time Implicit SLAM System
iMAP represents a volumetric scene with an MLP and jointly optimizes scene parameters and camera poses from actively sampled keyframe measurements. Its real-time pipeline combines differentiable rendering, adaptive keyframe selection, and loss-guided sampling to limit computation while retaining informative observations.
- System representation: The scene network maps 3D coordinates to colour and volume density, which are rendered along camera rays into predicted depth and colour.The MLP uses positional embedding and separate colour and density outputs; rendering queries samples along back-projected rays.
- Concurrent optimization: Tracking optimizes the latest camera pose against a fixed scene network, while mapping jointly optimizes network parameters and selected keyframe poses.The two processes run concurrently and use differentiable rendered measurements with geometric and photometric losses.
- Keyframe selection: Keyframes are added when the map explains less than 65% of a frame, using normalized depth error to measure newly observed regions.The adaptive rule places keyframes more closely when the camera is nearer to objects.
- Image active sampling: Joint optimization uses sparse pixel samples and reallocates samples toward image regions with higher loss, while tracking uses uniform sampling.Images are divided into an 8×8 grid, and active samples are allocated according to normalized regional loss.
- Keyframe active sampling: Keyframe sampling allocates more pixels to high-loss frames, and bounded selection samples three additional keyframes while always including the last keyframe and current live frame.This forms a constantly changing optimization window of W = 5 frames.
4. Experimental Results
Experiments show that iMAP reconstructs room-scale scenes with strong completion, low memory use, competitive tracking, and real-time operation. Ablations identify active sampling, bounded keyframes, and continual training choices that balance speed, accuracy, and detail.
- Scene reconstruction: iMAP produces globally coherent reconstructions across Replica rooms and real-world recordings, spanning whole rooms, small objects, and thin structures.The evaluation includes eight Replica scenes, handheld Azure Kinect sequences, and TUM RGB-D tracking sequences.
- Scene reconstruction: 4% higher Completion Ratio across all 8 Replica scenes, including an 11% improvement in office-3, demonstrates more complete reconstructions than TSDF fusion.The comparison focuses on completion while using iMAP-produced camera tracking for TSDF fusion.
- Memory efficiency: 60× less memory usage than default TSDF fusion achieves similar reconstruction accuracy.The comparison uses 256^3 voxel resolution for TSDF fusion and network width 256 for iMAP.
- Tracking: 2–6 cm tracking errors on TUM RGB-D make iMAP competitive with BAD-SLAM, Kintinuous, and ORB-SLAM2, though it does not outperform them in pose accuracy.The reported metric is tracking ATE RMSE in centimeters.
- Real-time operation: 10 Hz tracking and 2 Hz mapping are enabled by selected keyframes, concurrent processes, and randomized information-guided sampling.The default timing configuration uses a network width of 256, window size 5, and 200 samples per keyframe.
- Ablative analysis: 13 keyframes on average reconstruct the eight Replica rooms, while active sampling converges faster and reaches higher completion than random sampling.The network first forms coarse geometry within seconds and adds fine detail over minutes as training and camera motion continue.
5. Conclusions
iMAP reframes dense SLAM as real-time continual learning, training an MLP from scratch as the sole scene representation in a live system. Its design combines parallel tracking and mapping, loss-guided pixel sampling, and intelligent keyframe replay.
- iMAP trains an MLP from scratch as the only scene representation in a live dense SLAM system.
- The system constructs and tracks against a complete, accurate volumetric model of room-scale scenes using an RGB-D camera.
- Parallel tracking and mapping, loss-guided pixel sampling, and intelligent keyframe selection support real-time, long-term SLAM performance.Keyframe selection serves as replay to avoid network forgetting.