PaperlayerView exact revision on arXiv

Source-linked AI summary

R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package

Jiarong Lin, Fu Zhang

arXiv:2109.07982v1cs.ROcs.CV

TL;DR

LiDAR-based SLAM can fail when geometry features are insufficient, motivating tightly coupled fusion of LiDAR, inertial, and visual measurements. R3LIVE combines geometry reconstruction with photometric colorization, and experiments report stable estimation with 1.62° rotation and 4.57 cm translation drift in a challenging environment.

  • Problem

    LiDAR-based SLAM can fail in environments with insufficient geometry features, motivating tightly coupled fusion of LiDAR, inertial, and visual measurements for real-time localization, mapping, and map renderization.

  • Method

    R3LIVE combines LiDAR-inertial odometry for reconstructing map geometry with visual-inertial odometry that renders texture by minimizing frame-to-map photometric error.

  • Results

    1.62° rotation and 4.57 cm translation drift were measured against ground truth in simultaneous LiDAR-degenerated and visual texture-less environments.

  • Takeaways & Limitations

    The framework reconstructs dense, RGB-colored 3D point-cloud maps in real time and supports import into multiple 3D software tools.

Abstract

from arXiv · show

In this letter, we propose a novel LiDAR-Inertial-Visual sensor fusion framework termed R3LIVE, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE is contained of two subsystems, the LiDAR-inertial odometry (LIO) and visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage of the measurement from LiDAR and inertial sensors and builds the geometry structure of (i.e. the position of 3D points) global maps. The VIO subsystem utilizes the data of visual-inertial sensors and renders the map's texture (i.e. the color of 3D points). More specifically, the VIO subsystem fuses the visual data directly and effectively by minimizing the frame-to-map photometric error. The developed system R3LIVE is developed based on our previous work R2LIVE, with careful architecture design and implementation. Experiment results show that the resultant system achieves more robustness and higher accuracy in state estimation than current counterparts (see our attached video). R3LIVE is a versatile and well-engineered system toward various possible applications, which can not only serve as a SLAM system for real-time robotic applications, but can also reconstruct the dense, precise, RGB-colored 3D maps for applications like surveying and mapping. Moreover, to make R3LIVE more extensible, we develop a series of offline utilities for reconstructing and texturing meshes, which further minimizes the gap between R3LIVE and various of 3D applications such as simulators, video games and etc (see our demos video). To share our findings and make contributions to the community, we open source R3LIVE on our Github, including all of our codes, software utilities, and the mechanical design of our device.

I. INTRODUCTION

R3LIVE addresses robust real-time localization, 3D mapping, and map colorization through tightly coupled LiDAR-inertial-visual fusion. It combines geometry reconstruction, visual texture rendering, and open-source tools for dense RGB-colored mapping.

  • LiDAR-based SLAM can fail in geometrically sparse scenes, particularly with limited-FoV solid-state LiDARs.
  • Existing LiDAR-inertial-visual systems include loosely coupled and tightly coupled frameworks, with earlier loosely coupled systems not jointly optimizing LiDAR, visual, and inertial measurements.
  • R3LIVE targets real-time simultaneous localization, 3D mapping, and map renderization using tightly coupled LiDAR, inertial, and visual measurements.
  • Its LIO reconstructs geometry while its VIO renders texture, producing dense, 3D, RGB-colored point clouds in real time.
  • The VIO minimizes photometric error without requiring salient visual features, improving robustness in texture-less environments.
  • 0.16 meters translation drift and 3.9 degree rotation drift were reported after traveling up to 1.5 kilometers.
  • The project releases its system, offline mesh reconstruction and texturing utilities, and device mechanical design as open source.

II. THE SYSTEM OVERVIEW

R3LIVE is organized as complementary LIO and VIO subsystems. LIO estimates state and map geometry from LiDAR-inertial data, while VIO estimates state and renders color using image measurements.

  • The system overview contains an LIO subsystem and a VIO subsystem.
  • LIO registers incoming LiDAR scans and estimates state by minimizing point-to-plane residuals.
  • VIO renders RGB colors for global-map points and updates state using frame-to-frame PnP reprojection and frame-to-map photometric errors.
  • The paper uses notation inherited from R2LIVE and defines a full state vector x ∈ R^29.
  • The state representation includes gravity, the IMU-camera time offset, and camera intrinsic parameters including focal lengths and principal-point offsets.

B. Maps representation

R3LIVE represents its map with voxels containing colored 3D points and uses the LIO subsystem to append scan points after state estimation. Voxels are managed to support efficient visual tracking and rendering.

  • The map consists of voxels containing points, with points serving as the map’s minimum elements.
  • Fixed-size voxels, such as 0.1m × 0.1m × 0.1m, accelerate finding map points for VIO rendering and tracking.
  • Voxels are activated when points have been appended recently and deactivated otherwise.
  • Each point stores a 3D global position and RGB color, along with covariance matrices and creation and rendering timestamps.
  • LIO compensates LiDAR motion distortion, estimates state by minimizing point-to-plane residuals, and appends converged scan points to the global map.

V. VISUAL-INERTIAL ODOMETRY SUBSYSTEM

The VIO subsystem renders global-map texture while estimating state through a two-step ESIKF pipeline that first minimizes frame-to-frame PnP projection error and then refines the state photometrically.

  • VIO renders global-map texture while estimating the system state from visual-inertial measurements.
  • Frame-to-frame VIO: The pipeline first tracks map points with frame-to-frame optical flow and minimizes their PnP projection error.
  • Texture rendering: After state convergence, the system uses the estimate and raw image to render updated map-point colors.
  • Frame-to-frame VIO: The frame-to-frame update uses tracked map-point projections in consecutive images and estimates the state with an ESIKF.
  • Frame-to-frame VIO: The residual measurement noise includes pixel-tracking error and map-point location error.

2) Frame-to-frame VIO ESIKF update:

The frame-to-frame VIO update combines an observation distribution from projection residuals with the IMU-propagated prior in an iterated ESIKF procedure.

  • The observation distribution for the state error is combined with the IMU-propagated prior to obtain a MAP estimate.
  • The prior is represented by the IMU-propagated state estimate and covariance, while H maps state errors between tangent spaces.
  • The update computes a Kalman gain and then updates the state estimate.
  • The iterated Kalman-filter process continues until the update falls below a given threshold and is equivalent to Gauss–Newton optimization.

B. Frame-to-map Visual-Inertial odometry

Frame-to-map VIO refines the estimated state by minimizing photometric differences between global-map point colors and their measurements in the current image.

  • After frame-to-frame estimation, frame-to-map VIO minimizes tracked-point photometric error to reduce drift.
  • The photometric residual compares each map point’s stored color cs with its observed current-image color γs.
  • The observed color and covariance are obtained by projecting the point into the current image and interpolating neighboring RGB pixels.
  • The map-color measurement noise includes previously estimated rendering error and an impulsed random-walk process noise for environmental appearance changes.

2) Frame-to-map VIO ESIKF update:

The frame-to-map ESIKF update forms a photometric observation distribution, combines it with the IMU prior, and iterates state updates until convergence before downstream map operations.

  • The photometric Taylor expansion provides an observation distribution for the state error, which is combined with the IMU-propagated prior for MAP estimation.
  • The ESIKF update iterates until convergence and then supplies the state estimate for texture rendering, tracking-point updates, and next-frame IMU propagation.
  • Texture rendering: Texture rendering updates the colors of points in the global map using newly processed image information.

C. Render the texture of global map

After estimating the current image pose, R3LIVE renders map texture by updating the colors of visible points through image observations and Bayesian fusion.

  • After the frame-to-map VIO update, R3LIVE uses the precise current image pose to update map-point colors.
  • The rendering process retrieves all points from activated voxels before processing their colors.
  • For visible points, RGB values are interpolated from neighboring image pixels and fused with existing map colors using a Bayesian update.

D. Update of the tracking points of VIO subsystem

R3LIVE maintains its tracked VIO points by removing unreliable or unobservable points and adding newly projected points without nearby tracked neighbors. The system is evaluated on a lightweight handheld platform in difficult degenerate and texture-less environments.

  • D. Update of the tracking points of VIO subsystem: R3LIVE removes tracked points with excessive projection or photometric error and points that fall outside the current image.
  • D. Update of the tracking points of VIO subsystem: It projects global-map points into the current image and adds points when no nearby tracked point exists, using a 50-pixel example radius.
  • D. Update of the tracking points of VIO subsystem: The minimum handheld data-collection system weighs 2.09 Kg and includes onboard computation, a global-shutter camera, and a LiDAR.
  • B. Experiment-1: Robustness evaluation in simultaneously LiDAR degenerated and visual texture-less environments: Experiment-1 tests simultaneous LiDAR degeneration and visual texture scarcity while sensors pass through a narrow T-shaped passage beside walls.
  • B. Experiment-1: Robustness evaluation in simultaneously LiDAR degenerated and visual texture-less environments: The estimated pose remains bounded during the experiment, with final drift of 1.62° in rotation and 4.57 cm in translation.

C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment

Experiment-2 evaluates large-scale mapping across four campus trajectories, while Experiment-3 compares trajectory accuracy in seaport sequences against ground truth and competing systems. R3LIVE also supports offline meshing, texturing, export, and real-time operation across configurations.

  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: The four HKUST campus trajectories measure 1317, 1524, 1372, and 1191 meters, respectively.
  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: All four trajectories close the loop without additional loop-closure processing, and Table II reports small odometry drifts over the long routes.
  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: In seaport sequences with pedestrians and sparse open-sea LiDAR returns, R3LIVE trajectories agree best with ground truth among the compared systems in both sequences.
  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: The quantitative comparison evaluates relative rotation and translation errors over all sub-sequences from 50 to 300 meters.
  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: R3LIVE reconstructs colored 3D maps in real time and provides offline utilities for meshing and texturing them.
  • C. Experiment-2: High precision mapping large-scale indoor & outdoor urban environment: The utilities export colored point and meshed maps in formats including pcd, ply, and obj for use in common 3D software.

B. Toward various of 3D applications

R3LIVE maps are extended beyond reconstruction through Unreal Engine integration, supporting simulators and video games on desktop and mobile platforms.

  • The reconstructed maps are exported to Unreal Engine to enable additional 3D applications.
  • AirSim-based car and drone simulators use the reconstructed maps and provide depth and RGB image queries.
  • The maps also support video games for desktop PCs and mobile platforms, including campus exploration and combat scenarios.
  • Offline mesh reconstruction and texture rendering produce wireframe, surface, and textured mesh representations from RGB-colored points.
Loading 2109.07982v1…