Source-linked AI summary
Simplified Cross-Modal Calibration for Heterogeneous Event-RGB Stereo Systems
Nico Hessenthaler, Adam T. Müller, Nicolaj C. Stache
TL;DR
Cross-modal calibration is difficult because existing methods may require motion, precise synchronization, specialized hardware, or costly event reconstruction. This paper introduces a motion-free blended ChArUco target on a consumer display, alternating patterns to trigger events while maintaining RGB visibility. The method reduces mean reprojection error by 44% versus the strongest motion-based reference and 6% versus a non-motion-based reference, while supporting stable downstream eye-to-hand calibration.
Problem
Accurate calibration between event-based and frame-based cameras remains a bottleneck for heterogeneous stereo systems because existing approaches often require motion, precise synchronization, or computationally expensive reconstruction.
Method
The method alternates a ChArUco pattern with a partially blended state on a standard display, then coarsely aligns, denoises, and calibrates discretized event frames with RGB images.
Results
44% and 6% lower mean reprojection error are reported relative to the strongest evaluated motion-based and non-motion-based approaches, respectively.
Takeaways & Limitations
The framework supports static calibration with continuous cross-modal target visibility and practical eye-to-hand calibration under partial occlusions.
Takeaways & Limitations
Performance depends on display refresh rate and viewing-angle-stable contrast, while event-frame discretization can introduce jitter that reduces corner localization accuracy.
Abstract
from arXiv · showhide
Accurate extrinsic calibration between event-based and frame-based cameras remains a practical bottleneck for heterogeneous stereo systems. Existing approaches often require sensor or target motion, precise synchronization, or computationally expensive event-to-image reconstruction. We propose a simple, motion-free cross-modal calibration framework that uses a temporally modulated, blended ChArUco target presented on standard consumer displays. By alternating between the original pattern and a partially blended version, the target reliably triggers events while remaining continuously observable to a frame-based camera, avoiding blank frames and reducing synchronization constraints to a coarse, trigger-based alignment. We discretize events into frames coarsely aligned with the RGB images, apply lightweight denoising, and perform ChArUco-based intrinsic and stereo extrinsic calibration. Extensive experiments assess robustness to blending opacity, display brightness, external illumination, viewing angle, and handheld acquisition. Compared to the strongest motion-based reference (E2Calib + Kalibr) and a non-motion-based reference (Plasberg et al.), our approach reduces the mean reprojection error by $44\%$ and $6\%$, respectively, while substantially simplifying the calibration procedure. Finally, we demonstrate practical utility in a robotic eye-to-hand calibration case study, showing consistent transformations and stable downstream geometric measurements even under partial occlusions. Code is publicly available at https://github.com/nhessenthaler/simple-evrgb-cal.
1 Introduction
Cross-modal calibration is a practical bottleneck because existing motion-based and static methods impose motion, reconstruction, synchronization, or hardware requirements. The proposed motion-free framework uses a temporally blended ChArUco target to simplify deployment while preserving continuous RGB visibility.
- Cross-modal calibration remains a bottleneck for integrating event-based and frame-based cameras in heterogeneous systems.
- Motion-based methods trigger events through physical movement but often require computationally expensive reconstruction or complex motion-processing algorithms.
- Static methods avoid motion blur by modulating light intensity, but typically require specialized hardware and strict synchronization to prevent feature loss during blank display cycles.
- The proposed framework alternates a ChArUco pattern with a partially blended state on a consumer monitor, inducing events while keeping the target continuously visible to the RGB sensor.
- 44% and 6% lower mean reprojection error are reported against the strongest motion-based and non-motion-based approaches, respectively, alongside robustness across varied conditions and stable eye-to-hand calibration under partial occlusion.
2 Related Work
Prior cross-modal calibration methods are divided mainly into motion-based and static approaches, with additional application-specific and auxiliary-sensing methods. The proposed overlay strategy targets the practical complexity of these alternatives by using consumer hardware while preserving static calibration benefits.
- Related Approaches: Application-specific frameworks and auxiliary sensing modalities form additional lines of cross-modal calibration research beyond motion-based and static paradigms.
- Motion-Based Calibration Approaches: Motion-based calibration commonly uses relative sensor-target motion to trigger events, with reconstruction or motion-compensation pipelines supporting standard calibration tools.
- Motion-Based Calibration Approaches: Reconstruction-free motion-based methods directly detect event-stream features and match them to frame data, while later work improves target design, feature extraction, or event alignment.
- Motion-Based Calibration Approaches: These motion-based pipelines create deployment challenges through neural reconstruction, precise synchronization, and motion-blur compensation requirements.
- Static Calibration Methodologies: Static methods generate events by modulating light intensity with blinking LEDs or displays instead of physical motion.
- Static Calibration Methodologies: LED-based targets can require specialized hardware or lack robustness under standard ambient lighting, while blinking displays may lose frame-camera contrast during black cycles.
- Proposed Approach: The proposed overlay methodology uses consumer displays to provide high-contrast cross-modal detection without specialized hardware or intricate target designs, while reducing synchronization constraints.
3 Methodology
The method presents a shared, temporally controlled ChArUco stimulus that generates event data while remaining visible to the frame camera. Coarsely aligned event frames are denoised and processed with RGB images in a standard calibration pipeline.
- Overview: A shared visual stimulus actively modulates a calibration pattern to provide intensity changes for event sensing while preserving frame-domain feature detection.
- Calibration Target: The pipeline uses a ChArUco target displayed concurrently to event and frame-based sensors, avoiding specialized LED hardware.
- Overlay Generation: The target is generated by linearly blending a static ChArUco pattern with a uniform white image.
- Overlay Generation: The display periodically alternates between the original and blended patterns, producing intensity changes for events while retaining continuous RGB detectability.
- Overlay Generation: The full modulation cycle operates at 15 Hz, corresponding to 30 state transitions per second.
- Data Processing: The frame camera provides trigger signals for coarse software alignment, without precise or hardware-level synchronization.
- Data Processing: Events are accumulated into discrete 2D frames aligned with consecutive RGB captures, and both modalities are converted to grayscale.
- Data Processing: Median filtering suppresses event noise and densifies sparse pattern regions before standard calibration estimates intrinsic and stereo extrinsic parameters.
4 Experiments
Experiments evaluate the proposed calibration framework across operating conditions, acquisition settings, robustness factors, baseline methods, and robotic geometric alignment. The method remains accurate and stable across these tests while simplifying deployment.
- Blending opacity: At α = 0.6, event detection reaches 100% and stereo reprojection error reaches its minimum of 0.38 px.This operating point balances sufficient event-inducing intensity changes with frame-based target visibility.
- Calibration repeatability: Across 20 independent runs, both sensors achieve 100.0% corner detection, stable reprojection errors, and translation magnitude consistent with the nominal 0.0344m baseline.Rotation estimates are consistent, although their absolute accuracy cannot be compared exactly because of camera-mount tolerances.
- Illumination robustness: Under spatially varying illumination, corner detection declines and becomes more variable, but translation remains consistent with a 0.0345m nominal baseline and geometric integrity remains intact.Rotation estimates and reprojection errors show only marginally increased variance and remain in a low-error regime.
- Viewing-angle stability: Viewing-angle tests from 30° to 70° assess vertical, horizontal, and diagonal deviations, with reliability retained across poses relevant to practical calibration despite degradation at extreme angles.The evaluation reports combined corner detection rate and reprojection error averaged over five runs per direction.
- Baseline comparison: Across four baselines, the proposed method consistently yields lower reprojection errors, including 44% and 6% reductions against the strongest motion-based and non-motion-based references.The comparison includes E2Calib variants, including E2Calib + Kalibr.
- Robotic geometric alignment: Robot eye-to-hand evaluation estimates translation close to ground truth and achieves a 0.71mm mean absolute distance error, corresponding to approximately 1.05% relative error.The results also show stable distance preservation across robot poses and robust geometric alignment for vision-guided manipulation.
5 Limitations
The framework has two main practical limitations: sensitivity to display hardware characteristics and possible errors introduced when asynchronous events are discretized into image representations.
- Display Properties: Event density scales with display refresh rate, and marker detection accuracy may degrade below 30 Hz.The authors note that modern displays generally operate above this range, limiting the practical impact.
- Display Properties: Displays with strong viewing-angle-dependent contrast can reduce effective contrast modulation at large viewing angles.This particularly affects event cameras, which depend on intensity changes to register events.
- Event Representation: Event accumulation can introduce noise and temporal inconsistencies near high-contrast structures, reducing corner localization accuracy and increasing reprojection error.The authors identify advanced reconstruction and optimized accumulation as possible mitigation strategies.
6 Conclusion
The paper presents a practical stereo calibration approach using a temporally modulated, blended ChArUco target on commercial displays. It supports static acquisition, robust estimation across varied conditions, and downstream eye-to-hand calibration under partial occlusion.
- The method triggers events while keeping the ChArUco pattern continuously detectable by the frame-based camera.It minimizes reliance on motion, reconstruction, and precise hardware synchronization.
- Experiments show robust feature detection and stable intrinsic and extrinsic estimation across display settings, viewing angles, illumination, and capture modes.The evaluated modes include robot-assisted and handheld capture.
- The approach transfers to downstream eye-to-hand calibration, supporting consistent geometric estimation under practical conditions.
- Future work will improve event accumulation to mitigate limitations from display temporal characteristics and event discretization.
A Experimental Setup
The experimental setup combines a hybrid frame-event stereo rig on a UR5e robotic arm with a consumer monitor displaying the shared calibration target. Camera and event-sensor configurations are specified for the experiments.
- The hybrid frame-event stereo rig is mounted on a UR5e robotic arm to ensure precise trajectory repeatability.
- A Dell S2722QC monitor displays the calibration pattern for both the frame-based and event sensors.
- The IDS uEye frame-based camera configuration lists gamma and other settings, with unspecified parameters retained at hardware defaults.A gamma value of 100 corresponds to linear gamma γ = 1.0.
- The Prophesee EVK4 configuration uses bias offsets tuned to optimize the signal-to-noise ratio for flickering-pattern detection.Unlisted parameters remained at hardware defaults.
B Ablation Illumination Stability
The illumination robustness evaluation examines intrinsic calibration over repeated runs with varying spotlight positions. It reports detection success, camera intrinsics, and reprojection error, alongside qualitative RGB and event examples under strong spotlight interference.
- Under strong external spotlight interference, the RGB calibration target is overexposed while the corresponding event frame illustrates illumination effects on event generation and pattern visibility.The two images depict the same scene, with perspective differences caused by the stereo baseline offset.
- The evaluation measures illumination robustness over 5 runs with varying spotlight positions.
- The table reports corner detection success rate, focal lengths, principal point, and reprojection error for the intrinsic parameters.
C Eye-To-Hand Calibration Occlusions
The eye-to-hand calibration experiment evaluates robustness to partial occlusion caused by the robot actuator. Salient ChArUco corners remain reliably detected, enabling accurate calibration.
- Partial robot-actuator occlusions do not prevent reliable detection of salient ChArUco corners.The detected corners are highlighted in red in the example.
- Reliable corner detection enables accurate eye-to-hand calibration under occlusion.