Source-linked AI summary
LiDAR-Camera Calibration using 3D-3D Point correspondences
Ankit Dhall, Kunal Chelani, Vishnu Radhakrishnan, K. M. Krishna
TL;DR
LiDAR-camera calibration must align complementary sensor data in a shared frame, especially when existing methods do not extend well to lower-density LiDAR. The paper uses 3D-3D point correspondences and a closed-form rigid transformation, obtaining accurate, repeatable calibration and extending the approach to cameras without overlapping fields of view. The method is demonstrated through point-cloud fusion and released as a ROS package.
Problem
LiDAR and cameras provide complementary information, but their data must be aligned in a shared frame, and existing calibration methods do not extend well to lower-density LiDAR.
Method
The pipeline uses tag-based 3D-3D point correspondences in camera and LiDAR frames to solve for the rigid transformation [R|t].
Results
The method produces accurate and repeatable calibration across different cameras and enables extrinsic calibration of cameras without overlapping fields of view.
Takeaways & Limitations
Point-cloud fusion almost perfectly aligns the calibrated sources, and the implementation is available as an open-source ROS package.
Takeaways & Limitations
The stereo-camera setup calibrates only the left camera, relying on known baseline and stereo calibration parameters; better camera intrinsic calibration may reduce residual error.
Abstract
from arXiv · showhide
With the advent of autonomous vehicles, LiDAR and cameras have become an indispensable combination of sensors. They both provide rich and complementary data which can be used by various algorithms and machine learning to sense and make vital inferences about the surroundings. We propose a novel pipeline and experimental setup to find accurate rigid-body transformation for extrinsically calibrating a LiDAR and a camera. The pipeling uses 3D-3D point correspondences in LiDAR and camera frame and gives a closed form solution. We further show the accuracy of the estimate by fusing point clouds from two stereo cameras which align perfectly with the rotation and translation estimated by our method, confirming the accuracy of our method's estimates both mathematically and visually. Taking our idea of extrinsic LiDAR-camera calibration forward, we demonstrate how two cameras with no overlapping field-of-view can also be calibrated extrinsically using 3D point correspondences. The code has been made available as open-source software in the form of a ROS package, more information about which can be sought here: https://github.com/ankitdhall/lidar_camera_calibration .
1 Introduction
The paper motivates LiDAR-camera calibration as necessary for fusing complementary sensor data, while noting that existing approaches do not extend well to lower-density LiDAR. It proposes a repeatable method for estimating their six-degree-of-freedom extrinsic transformation.
- 1 Introduction: Camera and LiDAR data provide complementary color, feature, and structural information for sensing surroundings.Correspondences can support estimating a detected pedestrian’s exact 3D location.
- 1 Introduction: Fusing multiple sensors requires expressing their measurements in a shared reference frame.A common frame allows redundant information to be leveraged while reducing erroneous measurements.
- 1 Introduction: Existing marker-based and automatic calibration methods use high-density, more expensive LiDAR and do not extend well to lower-density sensors such as the VLP-16.
- 1 Introduction: The proposed method estimates accurate and repeatable six-degree-of-freedom extrinsic calibration parameters between a camera and a LiDAR.
2 Sensors and General Setup
The calibration process assumes known camera intrinsics and uses a camera facing markers directly, while the LiDAR observes a much wider scene. Sensor configurations vary in distance and orientation, with manual measurements used as coarse sanity checks.
- 2 Sensors and General Setup: Camera intrinsic parameters must be known before beginning LiDAR-camera calibration.
- 2 Sensors and General Setup: The camera observes directly in front of its lens, whereas a Velodyne VLP-16 LiDAR provides a 360-degree view.
- 2 Sensors and General Setup: Data were collected with the LiDAR and camera at arbitrary three-dimensional distances and with varying orientations.
- 2 Sensors and General Setup: Manual measurements supplied a coarse translation sanity check, while rotations were omitted when minimal or roughly estimated using trigonometry otherwise.
3 Using 2D-3D correspondences
The paper evaluates 2D-3D LiDAR-camera calibration using cardboard markers, line-derived 3D corners, and PnP-based transformations. Although projected points align and back-projection error is low, the estimated rigid transformation deviates from manual measurements.
- Experimental setup: The setup uses planar cardboard markers, with hollow rectangles providing eight 3D-2D corner correspondences instead of four.The additional inner corners increase correspondences while using fewer boards.
- Experimental setup: Tilting each board approximately 45 degrees helps the horizontal LiDAR scan lines register points on all four edges.RaNSaC fits lines to the registered LiDAR points.
- 3D point construction: The ROS node supports manually marking line segments in the point cloud and calculating their intersections for 3D corner recovery.Corners are approximated by the midpoint of the shortest segment between nearly intersecting fitted edge lines.
- 3D point construction: Line-intersection accuracy was supported by 10^-4-meter segment separation and average corner deviation of 0.68 mm.Estimated edge lengths differed from expected lengths by about 1 cm on average.
- PnP calibration: PnP estimates the rigid transformation from 2D-3D correspondences by projecting 3D points onto the image plane and minimizing a reprojection cost.P denotes the 3D-to-2D projection, while X and x denote 3D and 2D points, respectively.
- PnP calibration: E-PnP achieved an average back-projection error of 1.88 pixels after manually filtering correspondences, but its [R|t] remained inconsistent with tape measurements.Increasing point count and filtering visually aligned outliers did not resolve the transformation discrepancy.
- Robust estimation: RaNSaC-E-PnP repeatedly fits transformations from subsets, identifies inliers using threshold ϵ, and re-estimates [R|t] from those inliers.The experiments initialized 15 points from 20 correspondences and tested 10,000 random selections.
- Evaluation: The back-projection error fell below one pixel, yet the resulting [R|t] was still far from manually measured values.This motivated using a metric that relates more explicitly to the rigid transformation.
4 Using 3D-3D correspondences
The proposed calibration pipeline uses known 3D point correspondences from ArUco-marked planar boards and LiDAR-detected edges to estimate the rigid transformation between camera and LiDAR frames. Known correspondences enable a closed-form solution, while averaging estimates across scans reduces noise.
- Experimental setup: For the VLP-16, boards measuring roughly 45.0-55.0 centimeters and placed about 2.0 meters away provided enough points to run the pipeline.Low-density LiDAR requires sufficient points on board edges for line fitting and intersection calculation.
- Camera-frame correspondences: ArUco tags and known board dimensions provide cardboard corners as 3D points in the camera frame.The tag-to-camera transformation converts marker-frame corners into camera-frame coordinates.
- LiDAR-frame correspondences: LiDAR-frame corner points are obtained by detecting cardboard edges and solving for their intersections.The setup uses planar markers, with LiDAR points processed through edge detection and line fitting.
- Closed-form transformation: Known corner correspondences allow the Kabsch algorithm to estimate rotation, followed by translation after aligning the coordinate frames.This avoids relying on ICP to discover correspondences between point clouds.
- Multiple scans: Averaging translations and quaternion representations of rotations across multiple fixed-sensor runs reduces noise from imperfect edges and inaccurate LiDAR points.The method averages translation vectors directly, averages rotations in quaternion space, and converts the result back to a rotation matrix.
5 Fusing point clouds
The paper uses LiDAR to establish transformations between cameras, enabling point-cloud fusion even when cameras have little or no overlapping field of view. Visual fusion experiments show close alignment, with small reported translation errors.
- Multi-camera calibration: A LiDAR provides a common reference for estimating transformations between multiple cameras and can then be removed from the sensing setup.The LiDAR is used only to calibrate the cameras before their point clouds are fused.
- Transformation chaining: The pipeline estimates camera-to-LiDAR transforms independently and chains them to obtain the transformation between cameras.Transforms for C1 and C2 are first computed with the LiDAR, then combined to transform points between camera frames.
- Manual comparison: Manual measurements produce large translation errors and visible duplicate objects because the two point clouds are not properly aligned.The artifacts include duplicated markers, cupboards, and a carton box.
- Fusion accuracy: The proposed calibration almost perfectly fuses two stereo-camera point clouds, with 1-2 cm translation error and almost no rotation error.The resulting fused cloud appears visually like a single point cloud despite combining two camera clouds.
- Limited field-of-view overlap: For cameras placed about 80 degrees apart with almost no overlapping field of view, the method reports approximately 3-4 cm translation error and proper ground-plane alignment.The experiment used a common checkerboard to visualize alignment and assess translation error.
- Error boundary: Better intrinsic camera calibration is expected to reduce the reported error to about 1 centimeter or less.This is presented as an author expectation rather than a demonstrated result.
6 Results
Results compare manual measurements, ICP, and Kabsch-based estimation across datasets and configurations. The Kabsch implementation repeatedly produces values close to manual measurements with low RMSE, including for a camera with a much longer focal length.
- Algorithm comparison: The Kabsch algorithm repeatedly gives transformation values close to coarse manual measurements and achieves low root mean square error.The comparison includes off-the-shelf ICP and datasets with varied camera-LiDAR configurations.
- Repeatability and robustness: Experiments across varied sensor configurations support the reported repeatability and accuracy of the proposed method.A separate Point Gray Black Fly camera produced similar accuracy, supporting robustness across cameras.
7 Code and Implementation
The implementation is provided as open-source C++ software in a ROS package, with setup guidance available through the project resources.
- Software: The calibration code is written in C++ and implemented as a ROS package.The paper identifies the ROS wiki and GitHub repository as access points for the implementation and documentation.
8 Conclusions
The paper concludes that 3D-3D point correspondences provide accurate, repeatable LiDAR-camera extrinsic calibration and can extend calibration to cameras without overlapping views. Point-cloud fusion visually demonstrates the resulting alignment, and an open-source ROS implementation is provided.
- LiDAR-camera calibration: The pipeline uses printable planar markers and extracted LiDAR and camera corner correspondences to solve the rigid-body transformation [R|t].Averaging multiple estimates further improves consistency.
- Estimation properties: With relatively few correct correspondences, the method estimates the transformation optimally compared with ICP’s reliance on matching point correspondences.The paper reports accurate and repeatable results across different cameras.
- Multi-camera extension: The method extends to two or more cameras without overlapping fields of view and produces almost perfectly aligned fused point clouds.An open-source ROS package implements the approach.