General, Single-shot, Target-less, and Automatic LiDAR-Camera Extrinsic Calibration Toolbox

I. INTRODUCTION

• Existing frameworks may require difficult-to-create targets, many LiDAR-camera data shots, or carefully chosen geometry-rich environments.They also rarely support varied LiDAR and camera projection models.Source: §I · p. 1
• The toolbox supports spinning and non-repetitive scan LiDARs with pinhole, fisheye, and omnidirectional camera projection models.It is designed to be sensor-model-independent.Source: §I · pp. 1–2
• Calibration requires no target and uses environmental structure and texture, with one LiDAR point-cloud and camera-image pairing as the minimum.Additional pairings can optionally improve calibration accuracy.Source: §I · p. 1
• Automatic initialization matches cross-modal 2D–3D correspondences with SuperGlue and estimates the transformation using RANSAC and reprojection-error minimization.The method forms a LiDAR intensity image with a virtual camera for matching.Source: §I · pp. 1–2
• NID-based direct registration and view-based hidden-points removal refine calibration, with robust operation in geometrically sparse environments when LiDAR and camera data share mutual information.The paper contrasts this with edge-alignment methods [3], which it says would fail in such environments.Source: §I · pp. 1–2
• The authors released the toolbox as open source, stating that no existing open implementation combined these features.They describe the release as potentially beneficial to the robotics community.Source: §I · p. 2

II. RELATED WORK

• LiDAR-camera extrinsic calibration methods fall into target-based, motion-based, and scene-based approaches.Source: §II · p. 2

A. Target-based calibration

• Target-based calibration estimates the LiDAR-camera transformation from corresponding 3D target points and 2D image coordinates, but creating targets detectable by both sensors can be difficult.Prior approaches also used structured targets that are harder to create than a chessboard, or planar targets requiring manual LiDAR annotation or multiple acquisitions.Source: A. Target-based calibration · p. 2

B. Motion-based calibration

• Motion-based calibration estimates the transform from sensor motion without requiring overlapping sensor views, but depends on careful time synchronization and accurate per-sensor motion estimates.The section notes synchronization may be unavailable with an affordable web camera.Source: B. Motion-based calibration · p. 2

C. Scene-based calibration

• Scene-based methods estimate the LiDAR-camera transformation by evaluating consistency between LiDAR point-cloud and camera-image pairs, projecting LiDAR points into image space and measuring consistency with pixel values using metrics.The consistency-evaluation approaches are indirect and direct.Source: C. Scene-based calibration · p. 2
• Indirect scene-based methods can converge well with discriminative features but need geometry-rich environments and are less accurate than direct methods because they discard much surface texture information.They compute reprojection error between corresponding 2D-3D points.Source: C. Scene-based calibration · p. 2
• Direct methods use mutual-information-based metrics to compare LiDAR and image intensities through their co-occurrence, and the paper uses NID for metric-space properties and greater robustness than MI.Simply differencing LiDAR and image intensities does not work in practice because their intensity distributions differ.Source: C. Scene-based calibration · p. 2

A. Overview

• To support different LiDAR models, the toolbox merges multiple frames, accumulating non-repetitive scans and using continuous-time ICP-based dynamic integration for spinning LiDARs.The overview identifies static and dynamic LiDAR point integrators as the densification mechanisms.Source: A. Overview · p. 2
• SuperGlue estimates 2D-3D correspondences from a paired densified point cloud and camera image, with manual correspondence input available as a fail-safe.RANSAC and reprojection-error minimization use those correspondences to produce the initial transformation estimate.Source: A. Overview · p. 2
• The pipeline first densifies point clouds, finds 2D-3D correspondences, estimates a rough transform with RANSAC and reprojection-error minimization, then performs fine registration by minimizing NID.The overview also includes a manual correspondence tool and applies view-based hidden-point removal before refinement.Source: Figure 2 · p. 3, A. Overview · p. 3

B. Notation

• The calibration goal is to estimate the transformation between LiDAR and camera coordinate frames from paired LiDAR point clouds, point intensities, and camera images.The notation represents image pixel coordinates as x_j and pixel intensity as y_j.Source: B. Notation · p. 3
• LiDAR points are transformed into the camera frame and projected into image space using a projection function, with the paper using pinhole and omnidirectional equirectangular models.The toolbox also implements ATAN, fisheye, and unified omnidirectional camera models.Source: B. Notation · p. 3
• Densifying spinning-LiDAR data produces a point cloud with richer geometrical and surface-texture information than a sparse scan.The figure describes integration of a few seconds of dynamic LiDAR data.Source: Figure 3 · p. 3

C. Preprocessing

• For spinning LiDARs, the process estimates scan-start and scan-end poses with CT-ICP, corrects motion distortion, and accumulates scans in the first scan’s frame to create a dense cloud.A linear iVox structure stores points by voxel to efficiently build the target cloud from past observations.Source: C. Preprocessing · p. 3
• For non-repetitive-scan LiDARs, preprocessing simply accumulates all scans into one frame to create a dense point cloud.The passage gives Livox Avia as an example.Source: C. Preprocessing · p. 3
• The densified point cloud and camera image undergo histogram equalization because NID-based fine registration works best with uniform intensity distributions.Source: C. Preprocessing · p. 3

D. Initial Guess Estimation

• RANSAC and reprojection-error minimization estimate the LiDAR-camera transformation from 2D-3D correspondences.The correspondences are obtained from the input images and point clouds.Source: D. Initial Guess Estimation · p. 3
• A virtual camera renders LiDAR intensities, using pinhole projection below 150° FoV and equirectangular projection otherwise.Point index maps also provide efficient lookup of pixel-wise 3D coordinates for pose estimation.Source: D. Initial Guess Estimation · p. 4, Figure 4 · p. 4
• SuperGlue matches keypoints between LiDAR intensity images and camera images, but low-threshold matching produces false correspondences that require filtering.The threshold is set very low to obtain sufficient correspondences.Source: D. Initial Guess Estimation · p. 4, Figure 5 · p. 4

L 12: function ESTIMATEROTATIONLSQ(j0, j1) [25]

• SuperGlue’s matching threshold is set to 0.05 to obtain enough cross-modal correspondences, although many false matches result.The false correspondences are filtered before pose estimation.Source: L 12: function ESTIMATEROTATIONLSQ(j0,… · p. 4
• Rotation-only RANSAC is used to robustly handle outlier correspondences before pose refinement.The subsequent estimate is a 6 DoF transformation obtained by reprojection-error minimization.Source: L 12: function ESTIMATEROTATIONLSQ(j0,… · p. 4
• The six-degree-of-freedom transformation is refined with Levenberg–Marquardt by minimizing all keypoints’ reprojection errors using a Cauchy robust kernel.The keypoints are paired with corresponding 3D coordinates in the LiDAR frame.Source: L 12: function ESTIMATEROTATIONLSQ(j0,… · p. 4

E. NID-based Direct LiDAR-Camera Registration

• View-based hidden-point removal filters LiDAR points that should not be visible from the camera before fine registration.Depth-buffer testing keeps only the point with minimum distance for each projected pixel, producing the point cloud used for registration.Source: E. NID-based Direct LiDAR-Camera… · pp. 4–5
• NID-based registration compares projected LiDAR intensities with corresponding image intensities using marginal and joint intensity histograms.The transformation minimizing NID is found with the Nelder-Mead optimizer.Source: E. NID-based Direct LiDAR-Camera… · p. 5
• Hidden-point removal and NID registration are iterated until the transformation update displacement converges.Source: E. NID-based Direct LiDAR-Camera… · p. 5

• 0.043 m and 0.374° average errors for Ouster/pinhole fine registration compared with 0.236 m and 1.329° for edge alignment.Source: Table I · p. 5, §IV · p. 6
• For Livox/pinhole, 0.069 m and 0.724° average errors beat edge-based calibration at 0.323 m and 3.497°.Source: Table II · p. 5, §IV · p. 6
• Each of four sensor combinations was tested on 15 indoor/outdoor single-shot pairings; sphere-target measurements with manually annotated correspondences supplied pseudo ground truth.Source: §IV · p. 5
• Ouster/pinhole initial guesses succeeded for 12/15 pairings under 0.5 m/1.0° thresholds; flat, repetitive scenes yielded false matches that multi-data RANSAC could filter.Source: §IV · pp. 5–6, Figure 8 · p. 6
• For omnidirectional cameras, Ouster achieved 15/15 initial-guess success and 0.069 m/0.724°, while Livox achieved 10/15 and 0.101 m/0.807° amid FoV and resolution constraints.Source: Table III · p. 6, §IV · p. 6
• 0.011 m and 0.400° were achieved for Livox/omnidirectional calibration, while Ouster/pinhole was best at 0.010 m and 0.132° after multi-data calibration.Source: Table IV · p. 6, §IV · p. 6
• 74 to 249 s were required to calibrate from 15 data pairings; Ouster/omnidirectional took longer because both sensors have 360° FoVs and require projecting most points.Source: Table V · p. 6, §IV · p. 6

• The toolbox is reported to accurately calibrate combinations of spinning and non-repetitive scan LiDARs with pinhole and omnidirectional cameras.Source: §V · p. 6
• Image-matching-based initial-guess estimation provides the starting transformation for NID-based direct LiDAR-camera registration.Source: §V · p. 6
• The conclusion presents the toolbox as general across the tested LiDAR scan types and camera projection models.Source: §V · p. 6