PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping

Antoni Rosinol, Marcus Abate, Yun Chang, Luca Carlone

arXiv:1910.02490v3cs.ROcs.CV

TL;DR

Metric-semantic SLAM requires jointly estimating scene geometry and semantic labels, but geometric reconstruction and semantic segmentation have largely developed separately. Kimera addresses this gap with a modular visual-inertial library integrating odometry, robust pose graph optimization, fast meshing, and semantic reconstruction. The resulting system runs in real time on a CPU, while its fast mesher trades some geometric accuracy for substantially lower computation time.

  • Problem

    Geometric reconstruction and semantic segmentation have traditionally proceeded in isolation despite growing interest in their intersection.

  • Method

    Kimera is an open-source visual-inertial library integrating VIO, robust pose graph optimization, mesh reconstruction, and 3D semantic segmentation in modular components.

  • Results

    Kimera runs in real time on a CPU; its fast mesher has up to 24% higher error than the slower semantic mesh but requires two orders of magnitude less computation time.

  • Takeaways & Limitations

    Kimera provides a modular infrastructure for researchers to run components independently or combine them into metric-semantic SLAM and reconstruction pipelines.

Abstract

from arXiv · show

We provide an open-source C++ library for real-time metric-semantic visual-inertial Simultaneous Localization And Mapping (SLAM). The library goes beyond existing visual and visual-inertial SLAM libraries (e.g., ORB-SLAM, VINS- Mono, OKVIS, ROVIO) by enabling mesh reconstruction and semantic labeling in 3D. Kimera is designed with modularity in mind and has four key components: a visual-inertial odometry (VIO) module for fast and accurate state estimation, a robust pose graph optimizer for global trajectory estimation, a lightweight 3D mesher module for fast mesh reconstruction, and a dense 3D metric-semantic reconstruction module. The modules can be run in isolation or in combination, hence Kimera can easily fall back to a state-of-the-art VIO or a full SLAM system. Kimera runs in real-time on a CPU and produces a 3D metric-semantic mesh from semantically labeled images, which can be obtained by modern deep learning methods. We hope that the flexibility, computational efficiency, robustness, and accuracy afforded by Kimera will build a solid basis for future metric-semantic SLAM and perception research, and will allow researchers across multiple areas (e.g., VIO, SLAM, 3D reconstruction, segmentation) to benchmark and prototype their own efforts without having to start from scratch.

SUPPLEMENTARY MATERIAL

Kimera unifies geometric and semantic understanding in an open-source, modular library for metric-semantic localization and mapping. It combines visual-inertial sensing, robust optimization, mesh reconstruction, and semantic labeling in a CPU-based system.

  • Kimera addresses the growing intersection of geometric reconstruction and semantic segmentation by combining both capabilities in one perception library.
  • Kimera uses visual and inertial sensing rather than RGB-D, supporting indoor and outdoor environments.
  • Kimera provides a fast, lightweight, scalable CPU-based solution, contrasting with related approaches that require GPU processing.
  • The library unifies VIO, pose graph optimization, mesh reconstruction, and 3D semantic segmentation through four key modules.
  • Kimera-VIO estimates the robot state, Kimera-RPGO provides robust pose graph optimization, Kimera-Mesher produces fast regularized meshes, and Kimera-Semantics creates slower but more accurate semantically annotated meshes.

II. KIMERA

Kimera processes stereo and inertial data through parallel modules to produce pose estimates, globally consistent trajectories, and multiple metric-semantic meshes. Its semantic module refines the mesh using dense stereo and 2D semantic labels.

  • Kimera takes stereo frames and high-rate inertial measurements as input and returns IMU-rate state estimates, globally consistent trajectories, and multiple environment meshes.
  • Kimera-VIO’s front-end processes stereo images and IMU data into feature tracks, preintegrated measurements, and IMU-rate state estimates.
  • Kimera-Mesher computes low-latency per-frame and multi-frame 3D meshes, including a fast local mesh and optionally semantic labels.
  • Kimera-Semantics uses dense stereo, 2D semantic labels, and Kimera-VIO poses to produce a refined metric-semantic mesh.

A. Kimera-VIO: Visual-Inertial Odometry Module

Kimera-VIO implements a keyframe-based maximum-a-posteriori visual-inertial estimator with configurable smoothing horizons. Its front-end processes raw measurements and its back-end fuses them into sensor-state estimates.

  • Kimera-VIO implements a keyframe-based maximum-a-posteriori visual-inertial estimator that supports monocular and stereo frames.
  • The estimator supports full smoothing or fixed-lag smoothing, with the latter typically used to bound estimation time.
  • Kimera-VIO separates raw sensor processing in a visual-inertial front-end from measurement fusion in a back-end.

1) VIO Front-end:

Kimera’s estimation and mapping pipeline combines feature and inertial measurement processing with robust state estimation and loop-closure handling. The VIO back-end uses a fixed-lag factor graph, while RPGO enforces global consistency.

  • VIO Front-end: The IMU front-end performs on-manifold preintegration to compactly represent relative state measurements between consecutive keyframes.
  • VIO Front-end: The vision front-end detects and tracks corners, finds stereo matches, and applies mono and stereo RANSAC verification.
  • VIO Back-end: At each keyframe, preintegrated IMU and visual measurements enter a fixed-lag factor graph solved using iSAM2 in GTSAM.
  • VIO Back-end: Structureless vision estimates feature positions, eliminates those points from the state, and removes degenerate points and high-reprojection-error outliers.
  • Robust PGO: Kimera-RPGO detects loop closures and computes globally consistent keyframe poses using robust pose graph optimization.

1) Loop Closure Detection:

Kimera detects putative loop closures with DBoW2, rejects geometrically inconsistent candidates, and uses an online adaptation of PCM to select mutually consistent closures for robust pose graph optimization.

  • DBoW2 uses a bag-of-words representation to quickly detect putative loop closures between current and past keyframes.
  • Geometric verification rejects outlier loop closures before the remaining candidates reach the robust pose graph optimizer.
  • Kimera’s PCM implementation adds an odometry consistency check and incrementally updates consistent measurements for online operation.
  • PCM flags loops whose cycle error is inconsistent with measurement noise using a Chi-squared test, then tests pairwise consistency with previous loops.

C. Kimera-Mesher: 3D Mesh Reconstruction

Kimera-Mesher rapidly constructs per-frame and multi-frame 3D meshes from tracked features and VIO estimates, while updating the multi-frame model over the fixed-lag horizon.

  • Kimera-Mesher generates per-frame meshes for low-latency obstacle detection and multi-frame meshes spanning keyframes in the VIO fixed-lag smoother.
  • The per-frame pipeline triangulates tracked 2D features with Delaunay triangulation, back-projects them using VIO 3D estimates, and can texture the mesh with 2D semantic labels.
  • The multi-frame mesh fuses per-frame meshes, adds missing vertices and faces, updates positions from current VIO estimates, and removes features outside the time horizon.
  • Planar surfaces add regularity factors to the VIO back-end, creating tight coupling between visual-inertial estimation and mesh regularization.

D. Kimera-Semantics: Metric-Semantic Segmentation

Kimera-Semantics builds a global volumetric mesh and propagates 2D semantic labels from keyframe images onto the reconstructed 3D points and mesh.

  • Bundled raycasting is adapted to build an accurate global mesh covering the trajectory and semantically annotate it.
  • Dense stereo produces a point cloud at each keyframe, which bundled raycasting integrates into a TSDF before marching cubes extracts the mesh.
  • Kimera-Semantics propagates labels from semantically segmented keyframe images during bundled raycasting and attaches labels to dense-stereo 3D points.
  • The library also supplies open-source tools for debugging, visualization, benchmarking, continuous integration, and automatic evaluation of VIO and robust PGO.

III. EXPERIMENTAL EVALUATION

The evaluation reports state-estimation accuracy, robust pose-graph behavior, mesh reconstruction, metric-semantic reconstruction, and real-time module performance across benchmark and simulated settings.

  • Kimera attains state-of-the-art state-estimation performance, while robust PGO reduces sensitivity to loop-closure parameter tuning.
  • The experiments evaluate 3D mesh reconstruction on EuRoC scenes with ground-truth point clouds and metric-semantic reconstruction in a photorealistic simulator with ground-truth 3D semantics.
  • Table II compares RMSE of Absolute Translation Error across fixed-lag smoothing, full smoothing, and pose-graph optimization with loop closure.
  • Kimera-VIO and Kimera-RPGO achieve top performance across the evaluated VIO and pose-graph categories.
  • Table III shows that PCM makes Kimera-RPGO fairly insensitive to the loop-closure threshold α, whose smaller values produce more, less-conservative detections.

B. Geometric Reconstruction

Kimera evaluates geometric reconstruction against EuRoC ground-truth point clouds using accuracy and completeness metrics. The global mesh is more accurate, while the faster multi-frame mesh trades increased error for substantially lower computation time.

  • Evaluation: Kimera evaluates reconstructed meshes by uniformly sampling them, registering them to ground truth with ICP, and measuring accuracy and completeness.Accuracy measures estimated-to-ground-truth nearest-neighbor distance, while completeness measures the reverse direction.
  • Results: Up to 24% error increase: Kimera-Mesher produces noisier meshes than Kimera-Semantics but requires two orders of magnitude less computation time.The comparison is between the fast multi-frame mesh and the slower TSDF-based global mesh.

C. Semantic Reconstruction

Kimera-Semantics reconstructs and labels a global 3D mesh using VIO poses, dense stereo, bundled raycasting, and 2D semantic labels. Bundled raycasting has limited impact, while dense stereo is the main source of geometric and semantic degradation in the evaluated scenes.

  • Evaluation: The evaluation uses a Unity-based simulator with ground-truth geometry and semantics, while supplying ground-truth 2D semantic segmentations.Experiments isolate bundled raycasting, VIO pose estimates, and the full pipeline with dense stereo.
  • Results: Bundled raycasting causes less than 8cm geometric error and semantic accuracy above 94% on the 3D mesh.These results use ground-truth 2D semantics.
  • Results: Dense stereo produces the largest performance drop because texture-less regions, especially walls, are difficult to reconstruct.The confusion matrix shows errors concentrated between Wall/Shelf and Floor/Wall classes.

D. Timing

Kimera’s modules operate at different rates through parallelized processing, supporting time-critical state estimation and meshing alongside slower global optimization and semantic reconstruction.

  • Runtime: The IMU front-end takes around 40µs for preintegration and generates state estimates above 200Hz.Feature tracking averages 4.5ms per frame, while keyframe processing averages 45ms.
  • Runtime: Kimera-Mesher generates per-frame meshes in less than 5ms and multi-frame meshes in 15ms on average.The back-end factor-graph optimization takes less than 40ms.
  • Runtime: Kimera-Semantics updates the global metric-semantic mesh in 0.1s per keyframe, while Kimera-RPGO averages 55ms on EuRoC.These modules run on slower threads because their outputs are not required for time-critical actions.
  • Implications: Kimera is designed as a real-time CPU library with benchmarking and continuous-integration tools.The conclusion presents it as infrastructure for robot-perception research across communities.
Loading 1910.02490v3…