Source-linked AI summary
RegNet: Multimodal Sensor Registration Using Deep Neural Networks
Nick Schneider, Florian Piewak, Christoph Stiller, Uwe Franke
TL;DR
The paper addresses multimodal extrinsic calibration without relying on repeated human interaction or conventional matching pipelines. RegNet uses an end-to-end CNN trained with randomized decalibrations and iterative refinement across decalibration ranges. It reports 6 cm translation and 0.28° rotation mean calibration errors for decalibrations up to 1.5 m and 20°.
Problem
Multimodal sensor fusion requires extrinsic calibration, while existing offline and online approaches can require human interaction, controlled environments, or struggle with cross-modal matching.
Method
RegNet combines feature extraction, feature matching, and global calibration regression in an end-to-end CNN trained using randomized decalibrations from one calibrated setup.
Results
6 cm translation and 0.28° rotation mean calibration errors are reported for decalibrations up to 1.5 m and 20°.
Takeaways & Limitations
Iterative networks trained on different decalibration ranges support target-less calibration from scratch and online real-time correction without human interaction.
Abstract
from arXiv · showhide
In this paper, we present RegNet, the first deep convolutional neural network (CNN) to infer a 6 degrees of freedom (DOF) extrinsic calibration between multimodal sensors, exemplified using a scanning LiDAR and a monocular camera. Compared to existing approaches, RegNet casts all three conventional calibration steps (feature extraction, feature matching and global regression) into a single real-time capable CNN. Our method does not require any human interaction and bridges the gap between classical offline and target-less online calibration approaches as it provides both a stable initial estimation as well as a continuous online correction of the extrinsic parameters. During training we randomly decalibrate our system in order to train RegNet to infer the correspondence between projected depth measurements and RGB image and finally regress the extrinsic calibration. Additionally, with an iterative execution of multiple CNNs, that are trained on different magnitudes of decalibration, our approach compares favorably to state-of-the-art methods in terms of a mean calibration error of 0.28 degrees for the rotational and 6 cm for the translation components even for large decalibrations up to 1.5 m and 20 degrees.
I. INTRODUCTION
Multimodal sensor fusion requires accurate extrinsic calibration, but conventional and online approaches face interaction, environment, and matching challenges. RegNet combines calibration stages in an end-to-end CNN that learns from randomized decalibrations and supports autonomous correction.
- Accurate calibration is required to combine camera and depth-sensor information in a common world coordinate system.
- Conventional calibration extracts features, establishes cross-sensor correspondences, and estimates the transformation matrix H.
- RGB images and projected depth maps provide the multimodal inputs from which RegNet establishes correspondences and estimates 6 DOF extrinsic calibration.
- Offline methods often require calibration targets and human interaction or controlled environments, while online methods struggle to match patterns across modalities.
- RegNet is a CNN that jointly performs feature extraction, feature matching, and global optimization to regress extrinsic calibration in real time.
- Randomized decalibration generates extensive training data from one manually calibrated setup, after which the system needs no further human interaction.
II. RELATED WORK
Prior camera–depth calibration methods achieve strong results but often require targets, manual selection, or controlled environments. RegNet applies deep networks to directly regress calibration parameters with a real-time, low-memory design.
- Camera–depth calibration is important for accurately registering multimodal sensor data, particularly in autonomous-driving systems.
- Target-based and semi-automatic methods can provide suitable initial calibration but are time consuming or require controlled environments.
- Online approaches align projected depth or depth gradients with image edges, but occlusions and nondiscriminative descriptors complicate matching.
- A related optimization approach is stable only for disturbances up to 2° and 10 cm and is not real-time capable because it iteratively solves visual odometry and extrinsic parameters.
- RegNet directly regresses all 6 DOF calibration parameters and uses deep networks for feature extraction, matching, and regression.
- Its training strategy requires one manually calibrated sensor setup, while its real-time, low-memory architecture is designed for deployment in autonomous vehicles.
III. METHOD
RegNet targets generic extrinsic calibration with a focus on LiDAR–camera systems. It uses projected depth and RGB inputs, deep feature-processing blocks, and regression of the full calibration correction.
- The method aims to estimate a full 6 DOF extrinsic calibration using deep neural networks for feature extraction, matching, and regression.
- Experiments focus on LiDAR–camera calibration because of its relevance to autonomous driving, although the approach is described as applicable to other sensor combinations.
- Projected depth points and an RGB image are processed through NiN blocks for feature matching, followed by fully connected layers that regress decalibration.
A. Data Collection and Representation
RegNet generates training data by randomly varying an initial calibration and represents projected LiDAR depth alongside RGB input. The decalibration is encoded as rotation and translation, with dual quaternions providing a unified representation.
- A. Data Collection and Representation: Randomly varying Hinit creates large amounts of training data for learning decalibration relative to Hgt.The task is reformulated as determining φdecalib from an initial calibration and a ground-truth calibration.
- A. Data Collection and Representation: LiDAR points are projected into the camera frame using Hinit and the intrinsic camera matrix P to establish RGB-depth correspondences.
- A. Data Collection and Representation: Projected depth stores inverse depth at each pixel, uses zero where no point lands, and is upsampled with max pooling to address sparsity.The depth image and camera image are mean adjusted.
- A. Data Collection and Representation: The homogeneous decalibration matrix combines a 3 × 3 rotation matrix R with a 3 × 1 translation vector t.
- A. Data Collection and Representation: Dual quaternions unify rotation and translation, using a real part p and dual part q, while scaling p compensates training imbalance and implicitly weights rotation.The decalibration values are normalized to [−1, 1], but q has no specific range for dual quaternions.
B. Network Architecture
RegNet combines feature extraction, feature matching, and calibration regression in one end-to-end CNN. Separate RGB and LiDAR streams are fused before global regression, while large initial deviations can leave few projected points available.
- B. Network Architecture: RegNet combines feature extraction, feature matching, and calibration regression in a single CNN trained end-to-end.
- B. Network Architecture: Parallel RGB and LiDAR streams extract modality-specific features before their feature maps are concatenated for fusion.The network uses Network in Network blocks, with fewer depth-stream feature channels.
- B. Network Architecture: Up to 20° rotation and 1.5 m translation deviations can place most projected LiDAR points outside the image area, making correspondences difficult.
- B. Network Architecture: Two fully connected layers pool global multimodal information and regress calibration with a Euclidean loss.Branching translational and rotational components separately worsened the result.
C. Refinement
RegNet refines calibration iteratively by reprojecting depth points with the newly estimated calibration, and temporal averaging improves robustness over multiple frames.
- C. Refinement: The network can improve calibration even when large initial transformations leave few LiDAR-RGB correspondences.
- C. Refinement: Iterative refinement updates H using the estimated decalibration, then reprojection yields more depth points for subsequent correlation.
- C. Refinement: A moving average over network outputs produces more robust registration when calibration is estimated across multiple frames.
D. Training Details
The network is trained with Adam and Euclidean loss, while output-representation experiments favor quaternion-based representations over Euler angles.
- D. Training Details: The network is trained for 3 Mio. iterations with Adam, learning rate α = 10^-5, batch size b = 1, and Euclidean loss.The solver uses β1 = 0.9, β2 = 0.999, and ϵ = 10^-8.
- D. Training Details: Both quaternion representations outperform Euler angles in mean absolute rotational error during training.
IV. EXPERIMENTS
Experiments on real KITTI LiDAR-camera data compare calibration representations and decalibration ranges, with emphasis on rotational accuracy. Dual quaternions are selected for subsequent experiments after outperforming Euler angles in rotation estimation.
- The comparison focuses mainly on rotational components because they have greater impact on registration quality and are harder for the network to estimate.
- KITTI experiments use RGB images and Velodyne HDL-64E LiDAR depth measurements, with dataset extrinsics serving as ground truth.
- The decalibration representation compares Euler angles with translation, quaternions with translation, and dual quaternions.
- Dual-quaternion representation is selected for subsequent experiments after quaternion representations outperform Euler angles in mean absolute rotation error.
C. Different Decalibration Ranges
RegNet uses expert networks trained on different decalibration ranges because performance depends on the magnitude of the initial decalibration. Iterative refinement applies these experts from broad to narrow ranges, while smaller online decalibrations require fewer networks.
- C. Different Decalibration Ranges: Random decalibrations span [−1.5 m, 1.5 m] and [−20°, 20°], but large deviations have greater impact under Euclidean loss.
- C. Different Decalibration Ranges: Expert-network performance depends on decalibration magnitude, so RegNet iteratively applies networks trained on progressively smaller ranges.
- C. Different Decalibration Ranges: The offline sequence uses networks covering 20°/1.5 m, 10°/1.0 m, 5°/0.5 m, 2°/0.2 m, and 1°/0.1 m.
- C. Different Decalibration Ranges: For online decalibrations up to 20 cm and 2°, only the 2°/0.2 m and 1°/0.1 m networks are executed iteratively.
- C. Different Decalibration Ranges: An estimated calibration improves the depth-point projection before the refined depth map is passed to the next network.
D. Temporal Filtering
Temporal analysis addresses noise in single-frame calibration by examining outputs across a fixed-decalibration test sequence. Median aggregation performs best on validation data, while iterative networks also handle severe initial errors.
- D. Temporal Filtering: Single-frame calibration can be noisy because of missing structure, rolling shutter, and dynamic objects.
- D. Temporal Filtering: Median aggregation over the whole sequence gives the best validation performance.
- D. Temporal Filtering: The test evaluation samples fixed decalibrations from [−20°, 20°] and [−1.5 m, 1.5 m] across 100 runs.
- D. Temporal Filtering: Online examples with decalibrations up to 0.2 m and 2° use two iteratively executed networks.
V. CONCLUSION
The conclusion presents RegNet as a deep CNN that directly estimates six-degree-of-freedom multimodal extrinsic calibration while replacing conventional calibration stages. Iterative multi-range refinement supports both large initial decalibrations and smaller online corrections, with reported mean errors of 0.28° rotation and 6 cm translation.
- V. CONCLUSION: RegNet replaces feature extraction, feature matching, and global regression with direct six-degree-of-freedom calibration inference.
- V. CONCLUSION: Iterative refinement uses several networks trained on different decalibration ranges.
- V. CONCLUSION: Temporal filtering supports target-less calibration from scratch without human interaction, while moving averages or sliding windows support real-time online adaptation.
- V. CONCLUSION: 6 cm translation error and 0.28° rotation error are reported for decalibrations up to 1.5 m and 20°, competing with state-of-the-art online and offline methods.
- V. CONCLUSION: An end-to-end trained recurrent network is identified as a possible improvement over iterative refinement.