Source-linked AI summary
Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM
Luigi Nardi, Bruno Bodin, M. Zeeshan Zia, John Mawer, Andy Nisbet, Paul H. J. Kelly, Andrew J. Davison, Mikel Luján, Michael F. P. O'Boyle, Graham Riley, Nigel Topham, Steve Furber
TL;DR
Dense SLAM needs more portable evaluation because embedded deployment is constrained by computational demands and heterogeneous hardware, while existing benchmarks do not jointly assess accuracy, performance, and energy. SLAMBench provides portable KinectFusion implementations and ICL-NUIM-based accuracy validation, then characterizes performance and energy across platforms. The reported results show accurate comparisons across implementations, strong GPU-enabled performance, and meaningful energy trade-offs on embedded hardware.
Problem
Dense SLAM is difficult to deploy and compare on embedded platforms because high computational demands and heterogeneous hardware complicate performance-portable implementation, while existing benchmarks do not jointly assess accuracy, performance, and energy.
Method
SLAMBench combines portable C++, OpenMP, CUDA, and OpenCL KinectFusion implementations with ICL-NUIM ground-truth accuracy validation and instrumentation for performance and energy analysis.
Results
SLAMBench reports comparable accuracy across runs, GPU-enabled performance up to 135 FPS, and an embedded ODROID-XU3 result of 5.5 FPS at 2.1 Watts average dissipation.
Takeaways & Limitations
SLAMBench enables fair comparison of hardware accelerators, software tools, and novel dense-SLAM algorithms while separating estimation accuracy from performance comparisons.
Abstract
from arXiv · showhide
Real-time dense computer vision and SLAM offer great potential for a new level of scene modelling, tracking and real environmental interaction for many types of robot, but their high computational requirements mean that use on mass market embedded platforms is challenging. Meanwhile, trends in low-cost, low-power processing are towards massive parallelism and heterogeneity, making it difficult for robotics and vision researchers to implement their algorithms in a performance-portable way. In this paper we introduce SLAMBench, a publicly-available software framework which represents a starting point for quantitative, comparable and validatable experimental research to investigate trade-offs in performance, accuracy and energy consumption of a dense RGB-D SLAM system. SLAMBench provides a KinectFusion implementation in C++, OpenMP, OpenCL and CUDA, and harnesses the ICL-NUIM dataset of synthetic RGB-D sequences with trajectory and scene ground truth for reliable accuracy comparison of different implementation and algorithms. We present an analysis and breakdown of the constituent algorithmic elements of KinectFusion, and experimentally investigate their execution time on a variety of multicore and GPUaccelerated platforms. For a popular embedded platform, we also present an analysis of energy efficiency for different configuration alternatives.
I. INTRODUCTION
Dense SLAM offers detailed real-time scene understanding but remains difficult to deploy on embedded platforms because of high computational demands and increasingly heterogeneous hardware. SLAMBench addresses this gap by combining accuracy validation with performance, energy, and implementation analysis.
- Dense SLAM remains largely at the prototype stage on PC or laptop platforms because of its high computational requirements.
- SLAMBench combines a quality-of-result framework with execution-time and energy-consumption instrumentation.
- SLAMBench qualitatively analyzes the parallel patterns underlying each constituent computational element of KinectFusion.
- The framework characterizes KinectFusion performance across multicore CPUs and GPUs, spanning desktop to embedded hardware.
- Its energy analysis evaluates alternative implementation configurations on an embedded multicore SoC with GPU acceleration.
- The framework supports future exploration of algorithmic and implementation-level optimisations across the design space.
II. RELATED WORK
Existing benchmarks emphasize accuracy or general-purpose performance, but do not adequately support comparable evaluation of dense SLAM across accuracy, performance, and energy. SLAMBench builds on KinectFusion and ICL-NUIM to provide a portable, accuracy-verifiable framework for such comparisons.
- Accuracy benchmarks such as ICL-NUIM and TUM RGB-D evaluate visual odometry and SLAM, while general-purpose suites target broader computing workloads.
- Existing efforts do not provide software tools for accuracy verification, hardware-accelerator exploitation, and energy-performance-accuracy analysis across 3D reconstruction targets.
- SLAMBench addresses systematic comparison of reconstruction accuracy and performance by combining ICL-NUIM-based validation with measurable performance and energy outputs.
- KinectFusion performs real-time localisation and dense mapping by registering and fusing noisy, hole-filled depth frames into a clean 3D geometric map.
- KinectFusion computes a running average over the current TSDF volume and the new depth image.
- KinectFusion has become a building block for later SLAM systems and is available through open implementations including KFusion and KinFu.
IV. SLAMBENCH
SLAMBench is a portable benchmark for KinectFusion that combines multiple implementations with accuracy validation on ICL-NUIM sequences. It supports reproducible comparison across platforms and datasets.
- SLAMBench provides portable, untuned KinectFusion implementations in C++, OpenMP, CUDA, and OpenCL for diverse target platforms.
- The benchmark validates accuracy using ICL-NUIM while enabling algorithmic modifications to explore accuracy and performance trade-offs.
- ICL-NUIM supplies synthetic RGB-D trajectories and ground truth for measuring absolute trajectory error at each frame.
- ICL-NUIM includes realistic pre-rendered sequences and open-source tools for generating additional test data, enabling use with other datasets.
B. SLAMBENCH KERNELS
SLAMBench decomposes KinectFusion into computational kernels and characterizes their parallel structure. The kernels span initialization, image processing, tracking, reduction, and solving operations.
- Table I summarizes 14 computationally significant computer-vision kernels in SLAMBench.
- The acquire kernel accounts explicitly for input I/O costs, while preprocessing includes depth conversion and edge-preserving bilateral filtering.
- Tracking establishes point correspondences, reduction aggregates correspondence errors, and solve performs CPU singular-value decomposition for camera-pose correction.
- Initialization: Initialization-only kernels generate a Gaussian curve and initialize the 3D volume at startup.
- Parallel patterns: Kernels may combine multiple parallel patterns, including map, reduction, stencil, gather, and search.
D. PERFORMANCE EVALUATION METHODOLOGY
SLAMBench’s methodology addresses accuracy, application-context differences, and platform-dependent frame processing. It uses standardized inputs and execution to make comparisons consistent.
- A systematic methodology is needed to check accuracy across platforms because SLAM uses numerical approximations and iterative algorithms.
- Different applications stress the SLAM pipeline differently, including mobile robotics and augmented reality.
- Pre-recorded scenes and Process-every-frame mode prevent divergent behavior when platforms cannot process every frame in real time.
- The SLAMBench methodology consists of four elements.
1) GUI vs terminal interface:
SLAMBench uses pre-recorded inputs and controlled frame scheduling for reproducible evaluation, while excluding displayed OpenGL rendering from its scope. Real acquisition latency remains a practical consideration.
- GUI vs terminal interface: SLAMBench includes raycasting for visualization but excludes image display because OpenGL rendering is outside the work’s scope.
- Pre-recorded scenes: Pre-recorded scenes make acquisition negligible relative to other kernels, unlike real applications where input devices introduce unavoidable latency.
- Frame rate: Process-every-frame mode schedules the same enumerated frames on every platform, making computation independent of camera frequency and platform performance.
- Parallel patterns: Fig. 3 presents parallel patterns for map, reduction, stencil, gather, and search computations.
4) Accuracy evaluation:
SLAMBench evaluates implementation accuracy against ICL-NUIM trajectory ground truth using ATE statistics, while accepting numerical differences between platforms and languages. It currently evaluates trajectory accuracy rather than 3D reconstruction accuracy.
- SLAMBench computes ATE mean, median, standard deviation, minimum, and maximum errors from ICL-NUIM synthetic trajectories.These statistics test whether implementations in different languages and on different platforms remain sufficiently accurate without requiring bit-wise output equivalence.
- The framework currently uses ICL-NUIM trajectory ground truth but not its available 3D reconstruction ground truth.The authors cite the need for interactively driven computer-vision tools to compare reconstructions, while noting that localization and mapping are strongly correlated.
A. DEVICES
The evaluation spans five representative desktop, laptop, mobile, and embedded platforms, covering multicore CPUs, NVIDIA GPUs, and ARM heterogeneous systems. The devices differ substantially in processor, GPU architecture, and available parallel resources.
- Five platforms—TITAN, GTX870M, TK1, ODROID, and Arndale—represent desktop, laptop, mobile, and embedded categories.The selection targets devices used by consumers and industry across these computing classes.
- The NVIDIA systems use Kepler GPUs, while ODROID and Arndale use ARM Mali GPUs with different generations and core configurations.The TK1 has 192 CUDA cores, whereas ARM GPUs provide fewer but more powerful cores.
- TITAN pairs a 4-core Intel Haswell i7-4770K with a 2688-CUDA-core GTX TITAN GPU for exploring highly parallel desktop execution.The system has a power envelope up to 400 W, although reported actual consumption is considerably lower.
- The GTX870M laptop combines an Intel Haswell mobile i7-4700MQ with a 1344-CUDA-core mobile discrete GPU.This platform was selected as a high-end laptop target rather than a desktop-class system.
B. PERFORMANCE AND ACCURACY RESULTS
Across the tested platforms and implementations, SLAMBench reports comparable trajectory accuracy while exposing how kernel structure, memory behavior, and implementation language shape performance. CUDA is faster than OpenCL, and scaling differs substantially across kernels and hardware.
- 2.01 cm to 2.07 cm maximum ATE was measured across platforms and implementations under the default configuration.The lowest value was obtained by the ODROID OpenCL implementation, and the errors were comparable with prior work.
- 10% performance difference favored CUDA over OpenCL despite efforts to make the implementations similar.The authors also observed unexplained differences in kernel-time proportions and poor CUDA scaling for renderDepth and renderTrack.
- BilateralFilter and raycast scale well, whereas reduce and track scale badly; integrate shows average scaling across platform histograms.On the 2-core Arndale, proportions remain roughly unchanged because the limited core count restricts scaling.
- BilateralFilter scales well through stencil computation, cache-friendly 2D image access, and substantial execution-time share.The reduce kernel does not scale linearly because its reduction pattern remains difficult for highly parallel devices.
- Integrate suffers on embedded platforms with smaller data caches but scales well on TITAN and GTX780M, while raycast remains approximately constant across histograms.Raycast combines high memory-bandwidth demand with intensive stencil computation and many load/store operations.
- Small kernels that scale poorly become visible when massive parallel resources are deployed, as illustrated by the CPU-only solve kernel.
C. ENERGY CONSUMPTION ANALYSIS: ODROID-XU3 CASE STUDY
The ODROID case study compares energy and performance across CPU, OpenMP, and OpenCL configurations using onboard power monitors. OpenCL on the GPU is markedly more energy-efficient, while CPU-core selection and power-state management materially affect energy use.
- OpenCL on the GPU uses significantly less energy than approaches relying solely on conventional CPU cores.The comparison covers C++, OpenMP, and OpenCL configurations under process-every-frame execution.
- 507.28 versus 337 Joules shows that OpenCL with Cortex-A15 consumes 50% more energy than with Cortex-A7 despite negligible performance benefit.The authors therefore find no benefit in this application from using the higher-performance CPU with OpenCL.
- Parallel implementations can improve energy and performance over single-threaded execution when a race-to-sleep strategy switches the system to low power after processing.The analysis also finds substantial Cortex-A15 energy use even when those cores are configured offline.
- SLAMBench permits frame dropping at the system’s operating rate, allowing users to explore configurations that trade dropped frames against optimal ATE.The authors note that excessive inter-frame camera motion can cause tracking failure, and default ODROID settings typically drop many frames.
- The Mali GPU provides better energy efficiency than conventional cores, but CPU threads should use appropriate cores and unused cores should enter suitable low-power states.
VI. CONCLUSIONS AND FUTURE WORK
SLAMBench supports fair, accuracy-controlled comparisons across hardware, software, and algorithm alternatives while exposing performance, energy, and design-space trade-offs. Results span desktop, laptop, and embedded devices, with future work extending scalability and energy monitoring.
- SLAMBench separates estimation accuracy from performance comparison and makes module implementations straightforward to swap for isolating and improving algorithmic components.Its kernels are characterized by their parallel patterns and computational weight.
- 135 FPS was the peak among four super-real-time desktop and laptop configurations, while Tegra K1 reached 22 FPS and ODROID-XU3 reached 5.5 FPS at 2.1 Watts.The reported performance figures leverage GPU efficiency across desktop, laptop, and embedded devices.
- On ODROID-XU3, Cortex-A7/MALI matched Cortex-A15/MALI FPS with a 50% lower energy envelope at fixed accuracy.This provides an embedded performance/power trade-off evaluation on a platform described as suitable for robotics.
- SLAMBench enables design-space exploration by varying algorithm parameters alongside compiler settings to study performance, power, and accuracy metrics.Examples include voxel density, TSDF distance, OpenCL work-group size, and thread-coarsening.
- Future versions are intended to add energy monitoring through PAPI and scalable scene-reconstruction kernels using alternative data structures.Proposed structures include point-based fusion, octrees, voxel hashing, and moving volumes.