Source-linked AI summary
Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras
Huajian Huang, Longwei Li, Hui Cheng, Sai-Kit Yeung
TL;DR
Existing neural-rendering SLAM methods are resource-intensive and difficult to deploy on portable devices. Photo-SLAM addresses this with a hyper primitives map combining explicit geometry for localization and implicit photometric features for texture, plus geometry-based densification and Gaussian-Pyramid learning. It reports state-of-the-art online photorealistic mapping and real-time execution on embedded hardware.
Problem
Existing neural-rendering SLAM methods rely heavily on implicit representations, making them computationally intensive and unsuitable for portable, resource-constrained devices.
Method
Photo-SLAM combines explicit geometric features for localization with implicit photometric features in a hyper primitives map, using geometry-based densification and Gaussian-Pyramid-based progressive learning.
Results
Photo-SLAM significantly outperforms existing state-of-the-art SLAM systems for online photorealistic mapping and achieves real-time performance on embedded platforms.
Takeaways & Limitations
The system supports photorealistic mapping with monocular, stereo, and RGB-D cameras while maintaining real-time operation on embedded devices.
Abstract
from arXiv · showhide
The integration of neural rendering and the SLAM system recently showed promising results in joint localization and photorealistic view reconstruction. However, existing methods, fully relying on implicit representations, are so resource-hungry that they cannot run on portable devices, which deviates from the original intention of SLAM. In this paper, we present Photo-SLAM, a novel SLAM framework with a hyper primitives map. Specifically, we simultaneously exploit explicit geometric features for localization and learn implicit photometric features to represent the texture information of the observed environment. In addition to actively densifying hyper primitives based on geometric features, we further introduce a Gaussian-Pyramid-based training method to progressively learn multi-level features, enhancing photorealistic mapping performance. The extensive experiments with monocular, stereo, and RGB-D datasets prove that our proposed system Photo-SLAM significantly outperforms current state-of-the-art SLAM systems for online photorealistic mapping, e.g., PSNR is 30% higher and rendering speed is hundreds of times faster in the Replica dataset. Moreover, the Photo-SLAM can run at real-time speed using an embedded platform such as Jetson AGX Orin, showing the potential of robotics applications.
1. Introduction
Photo-SLAM targets resource-efficient simultaneous localization and photorealistic mapping by combining geometric and photometric representations in a hyper primitives map. It supports multiple camera types and reports state-of-the-art mapping, localization, and rendering performance, including real-time embedded execution.
- Motivation: Neural-rendering SLAM methods rely heavily on implicit representations, making them computationally intensive and unsuitable for resource-constrained devices.These methods may also require depth information and carefully bounded ray sampling, limiting scalability.
- Method: Geometry-based densification and Gaussian-Pyramid-based learning progressively acquire multi-level features for high-quality online photorealistic mapping.These strategies reduce reliance on dense depth information, including for monocular scenarios.
- Evaluation: Experiments across monocular, stereo, and RGB-D datasets show state-of-the-art localization efficiency, photorealistic mapping quality, and rendering speed.The evaluation covers diverse camera types and datasets.
- Method: Photo-SLAM maintains a hyper primitives map while combining localization, explicit geometry mapping, implicit photorealistic mapping, and loop closure.The system uses 3D Gaussian splatting for rendering and jointly maintains the map across its main components.
2. Related Work
Related work contrasts geometric and neural representations for visual localization and mapping. Photo-SLAM seeks a concise hyper-primitives representation that preserves efficient localization while modeling texture for photorealistic rendering across camera types.
- Visual localization and mapping: Visual SLAM estimates camera poses while building representations of unknown environments, emphasizing a trade-off between accuracy and real-time performance.Classical systems commonly use lightweight geometric features, incremental pose estimation, and local optimization.
- Explicit Representation vs Implicit Representation: Classical explicit or TSDF-based systems can support real-time dense reconstruction, but their view-rendering quality is limited.The passage contrasts these methods with more recent neural-rendering techniques.
- Explicit Representation vs Implicit Representation: Photo-SLAM uses hyper primitives to combine explicit geometric feature points for localization with implicit representations for texture modeling.This concise representation targets immersive exploration rather than dense-mesh reconstruction.
- Explicit Representation vs Implicit Representation: Photo-SLAM supports RGB-D, monocular, and stereo cameras because it achieves high-quality mapping without relying on dense depth information.The supported scope includes indoor and outdoor environments.
3. Photo-SLAM
Photo-SLAM combines parallel localization, geometry mapping, photorealistic mapping, and loop closure around a hyper primitives map. It uses factor-graph optimization, Gaussian splatting, geometry-guided densification, and Gaussian-Pyramid-based progressive learning.
- System Overview: Four parallel threads jointly maintain a hyper primitives map for localization, geometry mapping, photorealistic mapping, and loop closure.The system incrementally creates, optimizes, and densifies the map while processing camera inputs.
- Localization and Geometry Mapping: Localization estimates camera poses by minimizing robust reprojection error between matched 2D keypoints and 3D points using motion-only bundle adjustment and LM optimization.The objective uses a projection function, scale-associated covariance, and robust Huber cost.
- Localization and Geometry Mapping: Geometry mapping refines covisible keyframe poses and 3D points by minimizing reprojection residuals in a factor graph.Keyframes form graph nodes, matched 3D points provide constraints, and selected fixed keyframes remain unchanged.
- Photorealistic Mapping: Photorealistic mapping rasterizes hyper primitives with tile-based Gaussian splatting and optimizes their attributes against rendered and ground-truth images.The optimized attributes include position, rotation, scaling, density, and spherical harmonic coefficients; the loss combines L1 error and SSIM.
- Geometry-based Densification: Geometry-guided densification adds temporary hyper primitives at inactive 2D feature points, with depth obtained directly, locally, or by stereo matching.Fewer than 30% of frame feature points are active, motivating additional primitives in complex-texture regions.
- Gaussian-Pyramid-Based Learning: Gaussian-Pyramid-based learning progressively supervises mapping from the highest pyramid level to the lowest while hyper primitives are densified.The method targets multi-level features; experiments report improved photorealistic mapping, particularly for monocular cameras.
4. Experiment
Photo-SLAM is evaluated across monocular, stereo, and RGB-D settings, with comparisons covering localization, photorealistic mapping, rendering speed, hardware efficiency, and ablations of its Geo and GP components.
- 4.1. Implementation and Experiment Setup: Photo-SLAM achieves top Replica mapping quality while tracking camera poses in real time and rendering hundreds of 1200×680 views per second.On Jetson, rendering reaches about 100 FPS with less GPU memory usage.
- 4.2. Results and Evaluation: Photo-SLAM produces photorealistic monocular maps, while Nice-SLAM without depth supervision reaches 16.311 PSNR and suffers sharply worse localization.The monocular qualitative comparison reports photorealistic Photo-SLAM mapping.
- 4.2. Results and Evaluation: In RGB-D scenes, Photo-SLAM renders high-fidelity images about three hundred times faster, whereas competing maps lack high-frequency details or contain aliasing and hollow regions.ESLAM and Go-SLAM have the best localization accuracy in the cited RGB-D comparison, but their mapping lacks high-frequency details.
- 4.2. Results and Evaluation: Photo-SLAM supports stereo inputs, performs decently on EuRoC, and produces outdoor mappings from hand-held stereo-camera sequences.The EuRoC results are quantitative, while the outdoor result is illustrated qualitatively.
- 4.3. Ablation Study: The ablation study evaluates Geo and GP using PSNR, rendering FPS, and model size, showing that Geo supplies hyper primitives while GP improves their optimization.The study reports that removing Geo reduces PSNR by 2.028 monocularly and 1.662 in RGB-D, while four GP levels deteriorate results.
5. Conclusion
Photo-SLAM replaces resource-intensive implicit SLAM representations with a hyper primitives map that combines explicit geometry for localization and implicit texture features for photorealistic mapping. Experiments show improved online photorealistic mapping and real-time embedded execution, supporting potential robotics applications.
- Photo-SLAM uses a hyper primitives map with explicit geometric features for localization and implicit photometric features for scene texture.
- Geometry-based densification and Gaussian-Pyramid-based learning enhance the system’s online photorealistic mapping performance.
- Photo-SLAM significantly outperforms existing state-of-the-art SLAM systems for online photorealistic mapping and runs in real time on an embedded platform.
Supplementary Material
Supplementary results document embedded-platform real-time execution and analyze tracking stability, latency, and average processing time.
- Photo-SLAM demonstrates real-time performance on embedded platforms including the Jetson AGX Orin Developer Kit.
- Tracking comparisons use per-frame processing time across frame numbers, reporting average, minimum, maximum, and standard deviation.
6. Localication
Supplementary tracking analysis emphasizes stable low-latency processing, contrasting Photo-SLAM with methods affected by expensive optimization or slower initialization.
- Photo-SLAM rapidly and stably processes incoming frames, while Go-SLAM often takes about 1 second because of expensive global optimization.
- Nice-SLAM and Co-SLAM require longer initialization to accurately begin tracking.
7. Discussion
Photo-SLAM is evaluated against offline 3D Gaussian splatting for online photorealistic mapping, where prompt updates must balance computational efficiency and rendering quality. Across monocular and RGB-D settings, it delivers compelling quality with faster rendering.
- Online Mapping vs Offline Mapping: Online mapping updates localization and reconstruction continuously as new observations arrive, making efficiency–quality balance more demanding than offline mapping.Offline 3D Gaussian splatting instead trains from images with known poses and a sparse point cloud.
- Online Mapping vs Offline Mapping: Without fine-grained point clouds, 3D Gaussian splatting requires longer optimization and produces lower rendering quality within the same training duration.This comparison uses Photo-SLAM-estimated keyframe poses and three point-cloud initializations.
- Online Mapping vs Offline Mapping: 3D Gaussian splatting improves quality by densifying point clouds, but this increases model size and slows rendering.The trade-off is reported for comparisons on the Replica dataset.
- Online Mapping vs Offline Mapping: Photo-SLAM consistently provides compelling rendering quality and faster rendering than 3D Gaussian splatting with monocular and RGB-D cameras.The passage attributes this result to the proposed geometry-based densification and Gaussian-Pyramid learning methods.
8. More Results
Additional results report Photo-SLAM’s qualitative performance across TUM monocular and RGB-D sequences and stereo EuRoC data. Detailed platform-specific results are also provided for Replica, TUM, and EuRoC.
- Qualitative Results: Additional qualitative results are shown for TUM monocular sequences, TUM RGB-D sequences, and EuRoC stereo data.Figures 14, 15, and 16 correspond to these three camera configurations.
- Additional Results: Detailed per-scene results for the Replica dataset are reported in Table 8.The table is identified as an additional platform-specific results table for Replica.