Source-linked AI summary
Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments
Chongjian Yuan, Xiyuan Liu, Xiaoping Hong, Fu Zhang
TL;DR
The paper addresses accurate extrinsic calibration of high-resolution LiDAR and RGB cameras without external targets. It aligns natural edge features using LiDAR-aware extraction and reprojection-error minimization, achieving pixel-level accuracy with robustness across indoor and outdoor scenes. The method also identifies scene conditions that constrain calibration performance.
Problem
High-resolution LiDAR–camera applications require accurate extrinsic parameters, but target-based calibration needs prepared targets and projected-point features can suffer occlusion-related mappings.
Method
The method extracts natural image and point-cloud edges, uses depth-continuous LiDAR edges obtained through voxel cutting and plane fitting, and minimizes reprojection error.
Results
Around 50% of residuals in 36 calibration and validation cases are within one pixel, while the method is reported as robust, consistent, and comparable to target-based calibration.
Takeaways & Limitations
Targetless calibration can operate in natural indoor and outdoor scenes and supports calibration before or during data collection, including misalignment detection during online operation.
Takeaways & Limitations
Performance is constrained by scenes dominated by cylindrical objects, unevenly distributed edges, or edges along only one direction.
Abstract
from arXiv · showhide
In this letter, we present a novel method for automatic extrinsic calibration of high-resolution LiDARs and RGB cameras in targetless environments. Our approach does not require checkerboards but can achieve pixel-level accuracy by aligning natural edge features in the two sensors. On the theory level, we analyze the constraints imposed by edge features and the sensitivity of calibration accuracy with respect to edge distribution in the scene. On the implementation level, we carefully investigate the physical measuring principles of LiDARs and propose an efficient and accurate LiDAR edge extraction method based on point cloud voxel cutting and plane fitting. Due to the edges' richness in natural scenes, we have carried out experiments in many indoor and outdoor scenes. The results show that this method has high robustness, accuracy, and consistency. It can promote the research and application of the fusion between LiDAR and camera. We have open-sourced our code on GitHub to benefit the community.
I. INTRODUCTION
The paper targets accurate extrinsic calibration of high-resolution LiDAR and cameras without prepared targets, using natural features in indoor and outdoor scenes. It develops reliable edge-based calibration, analyzes edge constraints, and reports pixel-level accuracy comparable to target-based methods.
- High-resolution LiDAR–camera systems need accurate extrinsic parameters for dense mapping, colorization, and automated 3D surveying.
- Target-based calibration depends on prepared checkerboards or patterns, while sparse and noisy LiDAR measurements can destabilize feature-matching cost functions.
- The proposed targetless method extracts natural edge features from images and point clouds and minimizes reprojection error for automatic pixel-level calibration.
- The method studies LiDAR measurement principles and introduces depth-continuous edge extraction to improve calibration reliability and accuracy.
- Experiments across indoor and outdoor environments show robustness to initial conditions, consistency across calibration scenes, and pixel-level accuracy comparable to target-based methods.
- The authors develop practical calibration software and open-source it on GitHub.
A. Overview
The method calibrates LiDAR and camera extrinsics by aligning natural edge features, while addressing projection ambiguities and LiDAR measurement artifacts through direct point-cloud edge extraction. It uses geometric constraints, voxel-based plane fitting, and image correspondence validation to obtain accurate edge matches.
- Geometric constraints: A single 3D edge provides two effective extrinsic constraints because four pose degrees of freedom remain indistinguishable.These include translations along and perpendicular to the edge and rotations about the edge and the associated plane normal.
- Projection ambiguities: Projected point-cloud features suffer from zero-valued and multi-valued mappings caused by sensor occlusion, especially as LiDAR resolution increases.Camera-only regions may receive no projected points, while LiDAR-only regions can overlap foreground and background projections.
- LiDAR edge reliability: Depth-discontinuous edges are unreliable for high-accuracy calibration because laser beam divergence can create foreground inflation and bleeding points beyond actual object boundaries.A beam may reflect from both foreground and background surfaces, with high-reflectivity foreground returns dominating even when the beam centerline is off the foreground object.
- Depth-continuous edge extraction: The proposed extraction divides the point cloud into voxels, repeatedly fits planes with RANSAC, and intersects connected plane pairs to obtain depth-continuous edges.Plane pairs are retained when their angle lies within a selected range, such as [30°, 150°], and suitable voxel sizes can also recover curved edges.
- Edge matching: Depth-continuous edges are more accurate and less noisy than depth-discontinuous edges when overlaid with the image using the correct extrinsic value.Image edges are extracted with Canny, and edge pixels are stored in a 2D k-D tree for correspondence matching.
- Edge matching: The correspondence step projects both sampled LiDAR edge points and edge directions, using orthogonality validation to remove matches between nearby non-parallel image lines.The resulting visualization distinguishes LiDAR edges, image edge pixels, and their correspondences by red, blue, and green markings, respectively.
2) Matching:
The method matches sampled LiDAR edge points with image edges by projecting points into the camera image and fitting local image-edge lines. It validates correspondences using both point proximity and edge-direction orthogonality.
- Point projection: Sampled LiDAR edge points are transformed into the camera frame and projected onto the image plane using the current extrinsic estimate.The transformed point is projected with a pin-hole model followed by the camera distortion model.
- Local edge fitting: For each projected point, the method searches a k-D tree for its κ nearest image-edge pixels.
- Local edge fitting: The neighboring image pixels define a line represented by a point on the line and a normal vector from the minimum-eigenvalue eigenvector of their scatter matrix.
- Correspondence validation: The method validates matches by checking the projected LiDAR edge direction for orthogonality with the image-edge normal, removing nearby non-parallel-line mismatches.
C. Extrinsic Calibration
The calibration formulation models LiDAR bearing, depth, and image-edge extraction noise, then requires a noise-compensated LiDAR edge point to project onto its corresponding image edge. LiDAR bearing perturbations are represented intrinsically on the unit sphere.
- Measurement noises: Image-edge extraction uses a Gaussian point-location noise model with one-pixel discretization noise specified by σI = 1.5.
- Measurement noises: LiDAR bearing is measured from scanning-motor encoders, while depth is measured from laser time of flight.
- Measurement noises: The ⊞S2 operation rotates the measured unit bearing about a tangent-plane perturbation axis while preserving its location on S2.
- Measurement noises: LiDAR depth noise is modeled as a Gaussian ranging error, and bearing and depth relations are combined to express the true point relative to its measurement.
- Edge constraint: Under the ground-truth extrinsic, a noise-compensated LiDAR edge point projects exactly onto the corresponding image edge.
2) Calibration Formulation and Optimization:
The calibration solves nonlinear edge-correspondence constraints through iterative optimization in the tangent space of SE(3). Stacked residuals support maximum-likelihood extrinsic estimation until the update converges.
- Calibration formulation: Each LiDAR edge point contributes one extrinsic constraint, while an edge feature provides two effective constraints because it contains two independent points.
- Calibration formulation: The nonlinear extrinsic equation is linearized around the current estimate using the ⊞ operation in SE(3).
- Calibration formulation: Residuals from all N edge correspondences are stacked to form the joint calibration system.
- Optimization: The method estimates the extrinsic by maximum likelihood, equivalently described as minimum-variance estimation.
- Optimization: The solution is iteratively updated until the extrinsic increment satisfies ∥δT∗∥ < ε, after which the converged transform is taken as the calibrated extrinsic.
3) Calibration Uncertainty:
The method estimates calibration uncertainty from the covariance of the extrinsic error. This covariance reflects the sensitivity and distribution of edge features, allowing scene data quality to be quantified.
- Uncertainty estimation: The extrinsic error is modeled as Gaussian, with covariance equal to the inverse Hessian of the optimization objective.
- Uncertainty estimation: At convergence, the estimated error increment approaches zero, and the resulting covariance characterizes calibration uncertainty.
D. Analysis of Edge Distribution on Calibration Result
Calibration uncertainty depends on the sensitivity of edge-based residuals and on how edge features are distributed in the image and scene depth. Features near the image center and distant measurements reduce calibration informativeness.
- Sensitivity and uncertainty: The Jacobian JTi measures residual sensitivity to extrinsic variation, while small values imply larger estimation covariance.This covariance quantitatively encodes data quality.
- Sensitivity and uncertainty: Edges projected near the image center produce small Jacobians, so edge features should be distributed across the image.The relevant projection coordinates are Xi/Zi and Yi/Zi.
- Scene depth: Because LiDAR noise increases with distance, calibration scenes should have moderate depth.
- Experimental sensor suites: The sensor suite includes a Livox Avia1 LiDAR with an Intel Realsense-D435i2 camera for most experiments, plus a spinning LiDAR and industry camera for fixed-resolution verification.
E. Initialization and Rough Calibration
The pipeline adds a rough initialization stage because fine optimization needs a good initial extrinsic estimate. This stage maximizes edge correspondence before fine calibration.
- Motivation: The optimization-based calibration requires a good initial extrinsic estimate, so initialization widens its convergence basin.
- Edge correspondence: Rough calibration maximizes P.C., defined from matched LiDAR edge points relative to the total number of LiDAR edge points.Matching uses projected-point distance and direction to the nearest image edge.
- Search procedure: An alternating grid search evaluates rotation at 0.5° resolution and translation at 2 cm resolution over a specified range.
IV. EXPERIMENTS AND RESULTS
Experiments evaluate the calibration pipeline across indoor and outdoor scenes using high-resolution Livox AVIA and an Intel Realsense-D435i camera. The pipeline converges reliably across varied initializations and scene settings.
- Experimental setup: Experiments use a stationary high-resolution Livox AVIA and Intel Realsense-D435i camera with pre-calibrated camera intrinsics.
- Experimental setup: The evaluation covers a variety of indoor and outdoor scenes.
- Robustness and convergence: Each scene setting uses 20 test runs with random initial values drawn from a neighborhood around the CAD-model estimate.The tested neighborhood spans ±5° in rotation and ±10 cm in translation.
- Robustness and convergence: All 7 scene settings and 20 test runs per setting converge in both rough and fine calibration.
- Robustness and convergence: Converged extrinsics are nearly identical across the tested initial-value distributions, and the complete pipeline takes less than 60 seconds.
2) Consistency Evaluation:
Consistency is assessed through uncertainty intervals across scenes, while accuracy is assessed by cross-validation residuals. The reported results indicate overlapping scene estimates and predominantly subpixel residuals.
- Consistency evaluation: The consistency evaluation computes the standard deviation for each extrinsic degree of freedom.The covariance matrix ΣT is used to derive these quantities.
- Consistency evaluation: The 3σ intervals overlap across scenes, and the all-scenes estimate has smaller uncertainty while remaining within the other scenes’ 3σ confidence levels.
- Cross validation: Cross-validation calibrates one scene and applies its extrinsic to that scene and the other five scenes, using residual statistics for quantitative accuracy assessment.The largest 20% of residuals are removed as outliers in the figure.
- Cross validation: Around 50% of residuals, including the mean and median, are within one pixel across all 36 calibration and validation cases.
4) Bad Scenes:
The method requires calibration scenes with enough, well-distributed edge features. It performs poorly with cylindrical objects, uneven edge distributions, or edges confined to one direction.
- Scene requirements: Calibration requires a sufficient number of edge features distributed properly throughout the scene.Insufficient or poorly distributed edges provide weak constraints and reduce calibration reliability.
- Cylindrical scenes: Cylindrical objects can cause inaccurate edge extraction because the method relies on plane fitting.Their geometry also introduces parallax issues that reduce calibration accuracy.
- Edge distribution: Uneven edge distributions, such as edges concentrated in the image’s upper region, form poor constraints that measurement noise can easily affect.
- Edge orientation: Scenes containing edges along only one direction, such as vertical edges, do not provide sufficient constraints to uniquely determine the extrinsic parameters.