PaperlayerView exact revision on arXiv

Source-linked AI summary

LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry

Ryoichi Ishikawa, Takeshi Oishi, Katsushi Ikeuchi

arXiv:1804.05178v1cs.CVcs.RO

TL;DR

Targetless camera–LiDAR calibration faces modality differences and limitations in estimating camera translation from scaleless image motion. This paper extends hand-eye calibration with iterative sensor-fusion odometry for scaled camera motion. In the reported comparison with Scaleless, translation error improves dramatically, while rotation shows no great improvement.

  • Problem

    Targetless 2D-3D calibration depends on information shared across image and point-cloud modalities, while camera-only motion is scaleless and makes accurate extrinsic translation difficult.

    Source: §I · p. 1

  • Method

    The method extends hand-eye calibration by alternating extrinsic estimation with sensor-fusion odometry that recalculates scaled camera motion from LiDAR range data.

    Source: §I · p. 1, B. Iteration of Camera motion… · p. 3

  • Results

    Compared with Scaleless, the proposed method dramatically improves translation accuracy, shows no great improvement in rotation accuracy, and reduces error as the number of motions increases.

    Source: A. Evaluation with colored range data · p. 6

  • Takeaways & Limitations

    Calibration works best with less translation and larger camera rotations, preferably measured in places surrounded by flat terrain.

    Source: §VI · p. 8

    The calibration motion must include rotations in multiple directions; vertical rotation may be obtained through reasonable movement on a mobile platform.

    Source: §VI · p. 8

Abstract

from arXiv · show

In this paper, we propose a method of targetless and automatic Camera-LiDAR calibration. Our approach is an extension of hand-eye calibration framework to 2D-3D calibration. By using the sensor fusion odometry method, the scaled camera motions are calculated with high accuracy. In addition to this, we clarify the suitable motion for this calibration method. The proposed method only requires the three-dimensional point cloud and the camera image and does not need other information such as reflectance of LiDAR and to give initial extrinsic parameter. In the experiments, we demonstrate our method using several sensor configurations in indoor and outdoor scenes to verify the effectiveness. The accuracy of our method achieves more than other comparable state-of-the-art methods.

I. INTRODUCTION

Camera–LiDAR calibration is needed for 2D-3D sensor fusion, but manual correspondences are laborious, target-based methods require special targets, and targetless results depend on cross-modal information. This paper estimates scaled camera motion through sensor-fusion odometry and studies suitable calibration motions.

I. INTRODUCTION

  • Targetless calibration avoids target preparation, but existing methods can depend on the modality shared by images and point clouds; manual correspondences are laborious and may still yield insufficient accuracy.Target-based automated methods require targets detectable by both sensors, which the paper describes as impractical for on-site calibration.Source: §I · p. 1
  • The method estimates LiDAR and camera motions separately, then derives the extrinsic parameter from those motions using an extension of hand-eye calibration.Camera motion is first estimated without scale, then recalculated with scale using sensor-fusion odometry and the initial extrinsic estimate; these estimates are alternated until convergence.Source: §I · pp. 1–2
  • The method requires overlapping camera and LiDAR measurement ranges and a 2D LiDAR measurement range for scan alignment.The paper also studies which sensor motions make calibration effective.Source: §I · p. 2
  • The approach incorporates camera-motion estimation through sensor fusion into 2D-3D calibration, which the authors describe as the first such approach.Sensor-fusion odometry estimates scaled camera motion with high accuracy when the extrinsic parameter is known.Source: §I · pp. 1–2
  • The method uses only camera RGB images and LiDAR point clouds, without LiDAR reflectance or an initial extrinsic parameter.The authors report more accurate estimates than other methods with a small number of motions.Source: §I · p. 2

II. RELATED WORKS

The related-work discussion covers targetless multimodal alignment and hand-eye calibration; the methodology overview then describes initialization followed by alternating camera-motion and extrinsic-parameter estimation.

A. Target-less multi-modal calibration

  • Targetless multimodal methods align shared image and point-cloud information, including 3D discontinuities with image edges, mutual-information measures, and gradient-based metrics.Other texturing work aligns dense LiDAR range data with sparse 3D structure reconstructed from stereo images.Source: A. Target-less multi-modal calibration · p. 2

B. Hand-eye Calibration

  • Hand-eye calibration estimates the relative pose of two sensors from their observed motions using AX = XB; prior camera-based 2D-3D calibration uses scaleless motion, making accurate translation difficult with few motions.The cited method combines motion estimation with multimodal alignment and estimates a time offset.Source: B. Hand-eye Calibration · p. 2

III. METHODOLOGY

  • The method overview has two stages: initialize extrinsics from ICP LiDAR motions and feature-matched camera motions, then alternate extrinsic and camera-motion estimation through sensor-fusion odometry.Source: §III · p. 2

A. Initial calibration parameter estimation

Initial calibration estimates each sensor’s motion, then uses the motion pairs to estimate the extrinsic rotation, translation, and camera-motion scale.

LiDAR

  • LiDAR motion is estimated by fast ICP registration that searches for corresponding points in gaze directions and progressively tightens the correspondence-distance threshold.The method meshes the point cloud, projects points into two dimensions, and uses Voronoi splitting before scan alignment.Source: LiDAR · p. 2

1) Sensor motion estimation:

  • Initial camera motion is estimated from matched image features, with pose initialized using the five-point algorithm and RANSAC and then optimized using an angular epipolar-plane error.The implementation uses AKAZE features and descriptors.Source: LiDAR · pp. 2–3

2) Initial extrinsic parameter calibration from sensor motion:

  • The initial extrinsic calibration extends hand-eye calibration to estimate camera-motion scale alongside the sensor transform.Because camera translation is scaleless, the method introduces a scale factor; it solves rotation using more than two transitions with rotations in different directions, then solves translation and scale by linear least squares.Source: 2) Initial extrinsic parameter… · p. 3

B. Iteration of Camera motion estimation and Calibration from sensor motion

The iterative stage addresses the difficulty of estimating extrinsic translation from scaleless camera motion by alternating scaled camera-motion estimation with extrinsic calibration.

B. Iteration of Camera motion estimation and Calibration from sensor motion

  • Scaleless camera motion makes accurate extrinsic translation difficult because translation depends strongly on camera-motion accuracy, while extrinsic rotation can be estimated independently of scale.Source: B. Iteration of Camera motion… · p. 3
  • Given an extrinsic estimate, sensor-fusion odometry recalculates scaled camera motion from LiDAR range data; the method re-estimates the extrinsics and alternates these steps until convergence.The authors state that this permits more accurate estimation of extrinsic translation than the initial estimate.Source: B. Iteration of Camera motion… · p. 3

1) Camera motion estimation with range data:

The method builds 2D–3D correspondences by projecting LiDAR points into one camera image and tracking their pixels into another. It then optimizes the cameras’ relative pose by minimizing projection error.

1) Camera motion estimation with range data:

  • LiDAR points are projected into Camera 1 using its extrinsic parameters, then tracked into Camera 2 to pair each 3D point with a 2D direction vector.The correspondence construction uses point-cloud and image inputs together with the extrinsic parameter that localizes Camera 1 in the world.Source: 1) Camera motion estimation with range… · pp. 3–4
  • Fig. 3 schematizes the construction of the 2D–3D correspondences.Source: Figure 3 · p. 4
  • The resulting 2D–3D correspondences support optimization of the relative camera position and orientation by minimizing projection error.The initial pose uses generalized perspective 3-point estimation; later iterations use the preceding estimate.Source: 1) Camera motion estimation with range… · p. 4

2) Parameter Calibration:

The method recalculates the extrinsic parameters from camera and LiDAR motions, while considering which motions are likely to support convergence. The paper notes that precise convergence conditions depend on the environment and number of motions.

2) Parameter Calibration:

  • Each iteration recalculates the camera motion and extrinsic parameters, solving rotation and translation through linear and nonlinear optimization.The section specifies that rotation is optimized with Eq. 5 and translation with a separate nonlinear step.Source: 2) Parameter Calibration · p. 4

IV. OPTIMAL MOTION FOR 2D-3D CALIBRATION

  • Motion suitability is assessed by considering how errors in camera-motion and extrinsic-parameter estimates affect one another.Precise convergence conditions are difficult to establish because estimation also depends on the measured environment and number of motions.Source: §IV · p. 4
  • Fig. 4 depicts the case of an error in Camera 1’s localized position.Source: Figure 4 · p. 4

A. Camera motion estimation

The camera-motion analysis examines how localization error affects projection and subsequent estimation. It identifies shorter camera movements and smaller values of α − β as conditions associated with smaller projection error.

A. Camera motion estimation

  • The analysis assumes negligible rotation error when examining how extrinsic-parameter error affects camera localization.The estimated Camera 1 position is compared with the actual Camera 1 position.Source: A. Camera motion estimation · p. 4
  • A point-cloud point is projected using the estimated camera position and its tracked pixel is paired with the 3D point, while the projection error is minimized across directions.The ideal estimated motion approaches the actual motion when the projection error for the displacement is small.Source: A. Camera motion estimation · pp. 4–5
  • Smaller camera movement distance yields smaller projection error despite extrinsic-parameter error, sharpening the estimated position around the true value.Source: A. Camera motion estimation · p. 5
  • Smaller α − β values yield smaller projection error; steps in the environment or shallow camera incidence can increase this difference.The passage gives smooth surrounding walls as an example of a preferable calibration environment.Source: A. Camera motion estimation · p. 5

B. Extrinsic parameter estimation

The analysis describes how camera-motion translation error propagates into extrinsic translation estimation. It states a condition for improved accuracy and identifies motion choices that can reduce error propagation.

B. Extrinsic parameter estimation

  • Fig. 5 shows the indoor and outdoor calibration scenes captured by Ladybug 3.Source: Figure 5 · p. 5
  • When camera rotation is small, the translation errors satisfy |e_a| < |e|, indicating that camera-motion error propagates into extrinsic-parameter error in the diverging direction.Source: B. Extrinsic parameter estimation · p. 5
  • The method improves relative-position and orientation accuracy when error propagation into extrinsic estimation is smaller than the camera-motion error reduction from distance-image use.Source: B. Extrinsic parameter estimation · p. 5
  • Larger camera rotations and more sampled motions can reduce error propagation and support robust extrinsic estimation, but large image changes may impair motion estimation.An omnidirectional camera can preserve common field of view during significant rotation.Source: B. Extrinsic parameter estimation · p. 5

V. EXPERIMENTAL RESULTS

The colored-range evaluations compare calibration across indoor and outdoor data and sensor configurations. The proposed method has the lowest reported translation error in both scenes, while increasing motion count improves accuracy most clearly for translation.

V. EXPERIMENTAL RESULTS

  • The evaluation uses indoor and outdoor scenes with panoramic and multi-beam LiDAR configurations, comparing against Scaleless, Manual, and MI calibration.The colored-range evaluation uses a pseudo-panorama rendered from Focus S 150 data and an Imager 5010C point cloud, with ground truth from registration of the two point clouds.Source: §V · p. 5, A. Evaluation with colored range data · p. 6

A. Evaluation with colored range data

B. Ladybug 3 and Imager 5010C

The Ladybug 3 and Imager 5010C evaluation checks alignment between the camera panorama and a rendered point-cloud reflectance panorama. The proposed calibration produces consistency between the images, unlike the Scaleless initialization.

B. Ladybug 3 and Imager 5010C

  • The proposed method’s checker-pattern overlay is consistent between the Ladybug 3 panorama and rendered reflectance image, unlike the Scaleless initialization.The reflectance panorama is rendered from the estimated camera position in the point cloud.Source: B. Ladybug 3 and Imager 5010C · p. 7

C. Ladybug 3 and Velodyne HDL-64E

In the HDL-64E and Ladybug 3 trial, the proposed method gives highly accurate translation where Scaleless does not. The paper concludes that calibration needs multi-direction rotation and is best measured with suitable motions in flat surroundings.

C. Ladybug 3 and Velodyne HDL-64E

  • The HDL-64E/Ladybug 3 trial uses rover-generated horizontal and vertical rotations, with stop-and-scan acquisition and an indirectly established ground truth.The ground truth combines range-data alignment against Imager 5010C with manually specified panorama-to-reflectance correspondences for Ladybug 3 and Imager 5010C.Source: C. Ladybug 3 and Velodyne HDL-64E · p. 7
  • With HDL-64E and Ladybug 3, both motion-based methods have rotation error below 1°, but ours yields highly accurate translation where Scaleless differs substantially from ground truth.Manual calibration struggles with correspondence construction given HDL-64E’s narrow scan range and sparse point cloud; MI optimization does not complete with its low resolution and unclear reflectance.Source: Figure 12 · p. 7, C. Ladybug 3 and Velodyne HDL-64E · p. 7
  • Raising and lowering the rover’s front wheel by a small step provides usable vertical rotational motion for the proposed method.The experiment describes this operation as more difficult than horizontal rotation but demonstrates that it works.Source: C. Ladybug 3 and Velodyne HDL-64E · pp. 7–8

VI. CONCLUSION

  • The method is targetless, automatic 2D–3D hand-eye calibration using sensor-fusion odometry for camera-motion estimation.The conclusion recommends less translation and larger rotation camera motions and measurement surrounded by flat terrain where possible.Source: §VI · p. 8
  • Hand-eye calibration requires rotations in multiple directions; suitable mobile-platform motion can supply vertical rotation and permit dynamic calibration during scanning.The paper says to choose appropriate motions and prefers places surrounded by flat terrain as much as possible.Source: §VI · p. 8
Loading 1804.05178v1…