Source-linked AI summary
Generalizable Target-Less Joint Intrinsic-Extrinsic Camera-LiDAR Calibration using Deep Pixel-Point Correspondences
Simon Bultmann, Daniele Cattaneo, Abhinav Valada
TL;DR
Target-less correspondence methods for camera-LiDAR calibration generally require rectified images with known intrinsics, limiting their use with raw cameras. This paper jointly estimates camera parameters and sensor extrinsics from raw images and LiDAR scans; its evaluations report lower extrinsic errors than prior extrinsic-only methods while recovering camera parameters.
Problem
Learned pixel-point correspondence methods for camera-LiDAR calibration operate on rectified images with known intrinsics, while accurate calibration is needed to relate visual and geometric measurements for perception.
Source: §I · p. 1Method
The framework uses intrinsic- and distortion-aware correspondence learning with synthetic camera-parameter augmentation, alternating prediction and joint nonlinear refinement of camera parameters and extrinsics.
Source: §I · pp. 1–2Results
The method achieves the lowest joint-calibration errors on three robotic platforms and lower extrinsic errors than prior extrinsic-only methods while recovering camera parameters.
Source: E. Real-Robot Experiments · p. 7, §V · p. 8Takeaways & Limitations
Evaluations cover an unseen KITTI camera-LiDAR configuration and zero-shot transfer to three real robotic platforms without sensor-specific retraining.
Source: §I · p. 2
Abstract
from arXiv · showhide
Accurate camera-LiDAR calibration is an essential prerequisite for robust multi-modal perception in robotics. Recent target-less methods based on learned pixel-point correspondences achieve high extrinsic calibration accuracy and generalize across environments. However, they operate on rectified images with known camera intrinsics, restricting their applicability to systems whose cameras have already been individually calibrated. We present a fully automatic, target-less framework that jointly estimates camera intrinsics, radial-tangential distortion, and camera-LiDAR extrinsics from raw images and LiDAR scans. Our method generalizes deep correspondence-based calibration from rectified images to raw cameras through intrinsic- and distortion-aware correspondence learning via synthetic augmentations. During calibration, we iteratively couple correspondence prediction and nonlinear optimization to jointly refine the camera parameters and relative sensor transformation. We evaluate our approach on an unseen camera-LiDAR configuration from the KITTI dataset and on three real robotic platforms in a zero-shot manner. The results demonstrate that joint calibration with our generalized pixel-point correspondence network improves extrinsic accuracy over correspondence-based methods using pre-calibrated cameras, while additionally recovering accurate intrinsics.
I. INTRODUCTION
The paper extends target-less camera-LiDAR calibration to raw cameras by jointly estimating camera parameters and sensor extrinsics. It positions correspondence-based learning as a route to calibration across unseen sensor configurations.
I. INTRODUCTION
- Target-based calibration can be accurate but requires dedicated equipment, labor-intensive procedures, and expert supervision.Repeating calibration after sensor replacement or mechanical and thermal changes can make such procedures difficult to scale across fleets.Source: §I · p. 1, §II · p. 2
II. RELATED WORK
- Existing correspondence-based methods use rectified images with known intrinsics, whereas this method extends the approach to raw, distorted cameras.The related-work discussion also notes that prior correspondence-driven calibration addresses extrinsics rather than joint intrinsic-extrinsic estimation.Source: §I · p. 1, §II · p. 2
I. INTRODUCTION
- The framework jointly estimates camera intrinsics, radial-tangential distortion, and camera-LiDAR extrinsics from raw observations using correspondence learning and nonlinear refinement.It trains with physically plausible camera-parameter augmentation and evaluates on an unseen KITTI configuration and three robotic platforms.Source: §I · pp. 1–2, §II · p. 2
- Evaluating on an unseen sensor configuration tests generalization more directly than an unseen sequence from an unchanged camera-LiDAR pair.The latter leaves intrinsics and sensor transformation unchanged, potentially allowing overfitting to sensor-specific projection patterns.Source: §I · p. 2
II. RELATED WORK
- Earlier target-less joint methods use geometric primitives or intensity alignment and have narrower camera models or evaluation settings.The paper notes that Li et al. evaluate joint performance only on an internal dataset, while their KITTI experiments use ground-truth intrinsics; Shen et al. do not include a standard benchmark such as KITTI.Source: §II · p. 2
III. METHOD
The method initializes camera parameters and extrinsics, then alternates correspondence prediction with joint nonlinear refinement. Its projection model combines pinhole intrinsics, radial-tangential distortion, and the relative sensor transformation.
III. METHOD
- The framework initializes camera parameters and extrinsics, then iteratively feeds current estimates into LiDAR projection, correspondence prediction, and joint nonlinear optimization.Updated parameters are fed back for the next iteration.Source: §III · p. 2, Figure 2 · p. 3
A. Camera Model and Projection
- The projection model jointly estimates pinhole intrinsics K, radial-tangential distortion D, and the LiDAR-to-camera transformation T.The model uses focal lengths and optical center to obtain pixel coordinates after distortion.Source: A. Camera Model and Projection · p. 3
- The method limits rays to the valid radial-distortion domain, rejecting rays beyond rmax when that domain is finite.The boundary is determined by the smallest positive root of the radial-model derivative; if no positive root exists, the model remains monotonic and rmax is set to infinity.Source: A. Camera Model and Projection · p. 3
B. Intrinsic- and Distortion-Aware Correspondence Learning
The correspondence network is adapted to raw cameras by perturbing camera parameters and synthesizing images under sampled camera models. Validity-aware remapping and iterative inference expose it to projection and appearance changes from unseen parameters.
B. Intrinsic- and Distortion-Aware Correspondence Learning
- The CMRNext network predicts dense displacement fields between RGB images and LiDAR pseudo-depth images projected with the current calibration estimate.The displacement at each valid projected LiDAR point defines a pixel-point correspondence.Source: B. Intrinsic- and Distortion-Aware… · p. 3
- Training perturbs focal lengths, principal point, radial distortion, tangential distortion, and extrinsics to extend correspondence learning to raw cameras with unknown parameters.Multiple network instances trained with decreasing perturbation ranges are used sequentially during refinement.Source: B. Intrinsic- and Distortion-Aware… · p. 3
- Synthetic camera-parameter augmentation remaps images between sampled camera models, retaining only target pixels within the valid distortion domain.This exposes the network to projection errors and appearance changes from unseen pinhole and distortion parameters; inference projects LiDAR using current estimates before predicting correspondences.Source: B. Intrinsic- and Distortion-Aware… · pp. 3–4, Figure 3 · p. 4
C. Joint Nonlinear Optimization
The method jointly optimizes camera parameters and the relative sensor transformation from predicted LiDAR-point/image correspondences. It minimizes robust reprojection error with a weak principal-point prior, after filtering and subsampling correspondences.
C. Joint Nonlinear Optimization
- Camera parameters and the relative sensor transformation are jointly estimated by minimizing robust reprojection error with a weak principal-point prior.The prior stabilizes optimization while allowing the principal point to deviate from its SfM initialization.Source: C. Joint Nonlinear Optimization · p. 4
- The nonlinear least-squares objective is solved with Ceres, with rotation parameterized on the unit-quaternion manifold.The optimization correspondences pair each LiDAR point with its predicted image location.Source: C. Joint Nonlinear Optimization · p. 4
- Invalid projections are discarded, then PnP-RANSAC selects inliers and the remaining correspondences are uniformly subsampled to 10%.Inlier selection is omitted in the first iteration because the initial camera parameters may not support a consistent rigid transformation estimate.Source: C. Joint Nonlinear Optimization · p. 4
D. Alternating Correspondence and Calibration Refinement
Each correspondence stage is followed by joint optimization, and the updated parameters determine the LiDAR projection for the next network stage. Camera parameters are refined early, then frozen while extrinsic refinement continues.
D. Alternating Correspondence and Calibration Refinement
- Updated intrinsic and extrinsic parameters change the LiDAR projection used by the next correspondence-prediction stage.This couples network predictions with joint calibration refinement at each stage.Source: D. Alternating Correspondence and… · p. 4
- Camera parameters are refined for the first 5 iterations and then held fixed while extrinsic refinement continues to iteration 11.Freezing them early is intended to prevent residual correspondence noise from causing drift in weakly observable components.Source: D. Alternating Correspondence and… · p. 4
- The iteration counts are selected through hyperparameter evaluation.The passage refers to Fig. 4 for that evaluation.Source: D. Alternating Correspondence and… · p. 4
E. Initialization
Initialization uses structure-from-motion for camera parameters and motion alignment for the rigid sensor transformation, without a manual initial guess. COLMAP and visual and LiDAR trajectories provide these estimates.
E. Initialization
- Camera parameters are initialized with COLMAP on images with sufficient translational and rotational motion, without a calibration target.COLMAP's OPENCV model estimates focal lengths, principal point, and radial-tangential distortion; k3 is initialized to 0.Source: E. Initialization · pp. 4–5
- The rigid sensor transformation is initialized through motion alignment using visual and LiDAR trajectories.ORB-SLAM3 supplies the visual trajectory and FAST-LIO2, run without IMU measurements, supplies the LiDAR trajectory.Source: E. Initialization · p. 5
- Unlike MDPCalib, the initialization uses COLMAP camera estimates rather than ground-truth camera intrinsics for ORB-SLAM3.The passage says this preserves a practical pipeline without privileged calibration information while still obtaining sufficiently accurate trajectory alignment.Source: E. Initialization · p. 5
IV. EXPERIMENTAL EVALUATION
The evaluation uses the public KITTI dataset and three real robotic platforms. It assesses generalization to an unseen camera-LiDAR configuration, joint calibration accuracy, and the contributions of training and optimization components.
IV. EXPERIMENTAL EVALUATION
- Experiments use the public KITTI dataset and three real robotic platforms.Source: §IV · p. 5
- The evaluation assesses generalization to an unseen camera-LiDAR configuration and the accuracy of joint intrinsic-extrinsic calibration.Source: §IV · p. 5
- The experiments also assess the contributions of the proposed training and optimization components.Source: §IV · p. 5
A. Experimental Setup
The evaluation trains across KITTI, Argoverse, and PandaSet, then tests on KITTI sequence 00’s raw right camera, an unseen camera-LiDAR configuration. Calibration uses staged correspondence networks and jointly refines camera parameters before continuing extrinsic refinement.
A. Experimental Setup
- Training uses KITTI sequences 01–21 plus one rectified camera each from Argoverse and PandaSet, with available raw KITTI images added for camera-parameter augmentation.Evaluation uses the raw right camera of KITTI sequence 00, differing from training in both sequence and camera-LiDAR configuration.Source: A. Experimental Setup · p. 5
- Three correspondence networks with decreasing perturbation ranges run coarse for the first iteration, medium for the next two, and fine thereafter.Each run uses 100 synchronized image-scan pairs and retains 10% of correspondences after RANSAC inlier selection.Source: A. Experimental Setup · p. 5
- Camera parameters start from ground truth, COLMAP, or versions perturbed by independent ±10% uniform noise with the principal point at image center; motion alignment initializes extrinsics.Source: A. Experimental Setup · p. 5
B. Metrics
On an unseen KITTI camera-LiDAR setup, the method jointly refines intrinsics and extrinsics while attaining lower extrinsic errors than methods assuming known intrinsics. The section also defines parameter-wise and reprojection-based metrics for evaluating calibration accuracy.
B. Metrics
- Extrinsic accuracy is measured by translation and rotation errors, including component-wise values and their magnitude or shortest-arc summaries.Intrinsic evaluation reports relative focal-length and principal-point errors, median pixel reprojection error, and distortion-only reprojection error.Source: B. Metrics · p. 5
C. Joint Calibration on an Unseen Camera-LiDAR Setup
- The joint method retains lower extrinsic errors than known-intrinsics baselines while recovering focal length, principal point, and distortion.Replacing ground-truth intrinsics with fixed COLMAP estimates produces a 30.2 cm forward translation error for MDPCalib*, while the proposed method reports 2.86 cm translation and 0.13° rotation error.Source: C. Joint Calibration on an Unseen… · pp. 5–6, Table II · p. 6
- With perturbed ground-truth intrinsics, joint calibration achieves 2.27 cm translation and 0.11° rotation error, and COLMAP initialization gives similarly stable results.With fixed ground-truth intrinsics, translation magnitude is 2.25 cm versus 2.92 cm for MDPCalib, and rotation magnitude is 0.05° versus 0.14°.Source: C. Joint Calibration on an Unseen… · pp. 5–6, Table II · p. 6
D. Ablation Study
Ablations identify camera-parameter augmentation, inlier selection, and a principal-point prior as contributors to calibration accuracy and robustness, while the selected optimization schedule balances intrinsic accuracy and stable extrinsic refinement. On three robotic platforms, the method achieves the lowest joint-calibration errors, with one stated exception against an extrinsic-only method on the UAV.
D. Ablation Study
- Camera-parameter augmentation gives the largest ablation improvement, reducing mean translation error by 2.76 cm, rotation error by 2.32°, and reprojection error by 36.6 px.RANSAC-based inlier selection lowers rotation error from 0.41° to 0.24°; the principal-point prior reduces reprojection error from 6.10 px to 3.64 px and stabilizes intrinsic estimates.Source: D. Ablation Study · p. 6
- Intrinsic refinement for at least five iterations is needed for accurate camera parameters, and the selected schedule is Nintr = 5 and Ntotal = 11.Rotation error has a clear optimum at Nintr = 5, while translation mainly benefits from additional extrinsic-only refinement with diminishing returns beyond N = 11; Nintr = 6 has higher reprojection-error variance and degrades rotation.Source: D. Ablation Study · p. 6, Figure 4 · p. 6
E. Real-Robot Experiments
- Real-robot evaluation uses COLMAP camera initialization and ORB-SLAM3 and FAST-LIO2 trajectories for automatic extrinsic initialization, including a perturbed-COLMAP setting.The evaluated platforms differ in cameras, LiDAR sensors, and mounting configurations.Source: E. Real-Robot Experiments · p. 6
- Qualitative projections on KITTI and the three robotic platforms show border-near structures and distant structures at annotated ranges as calibration checks.The figure highlights border regions as sensitive to intrinsic and distortion parameters and uses zoom-ins to show long-distance alignment.Source: Figure 5 · p. 7
- Across Car, Spot, and UAV, the method has the lowest joint-calibration errors; it also beats fixed-ground-truth-intrinsics methods on Car and Spot, while Koide et al. retain lower extrinsics error on UAV.The table compares target-less calibration results on real robotic platforms.Source: E. Real-Robot Experiments · p. 7, Table IV · p. 7
F. Qualitative Results
The final joint calibration recovers accurate camera parameters and achieves lower extrinsic errors than prior extrinsic-only methods assuming known intrinsics. Qualitative projections align with image edges across depths, and the default calibration takes about 300 seconds.
F. Qualitative Results
- Projected LiDAR structures align closely with image edges across depths, including near image borders and in distant regions.Border alignment remains close despite increased sensitivity to intrinsic and distortion parameters.Source: F. Qualitative Results · p. 7
G. Runtime
- About 300 s completes calibration with 11 iterations, including 5 intrinsic-refinement iterations; correspondence inference takes about 62% of runtime and optimization 35%.The remaining 3% is other overhead; MDPCalib reports approximately 3.5 × 10^3 s for its evaluated setting.Source: G. Runtime · p. 7, Table V · p. 7, F. Qualitative Results · p. 7
V. CONCLUSION
- The method achieves lower extrinsic errors than prior extrinsic-only methods assuming known intrinsics while recovering accurate camera parameters.Experiments use COLMAP-based initialization without privileged calibration information.Source: §V · p. 8