Source-linked AI summary
A Photometrically Calibrated Benchmark For Monocular Visual Odometry
Jakob Engel, Vladyslav Usenko, Daniel Cremers
TL;DR
Monocular VO and SLAM benchmarks need broader environments and better access to camera image-formation information. The paper provides a 50-sequence, photometrically calibrated dataset and evaluates methods using accumulated loop drift rather than full-sequence ground truth. Its evaluation framework also addresses tracking failures and validates that alignment error primarily reflects accumulated drift.
Problem
Existing evaluations often use limited environments and data that omit sensor image-formation information, while full external ground truth is difficult to acquire.
Method
The paper constructs a photometrically calibrated dataset and evaluates tracking through accumulated drift over large loops, with loop closure disabled.
Results
The benchmark comprises 50 sequences totaling 105 minutes across dozens of environments, with exposure times, response calibration, and vignetting calibration.
Takeaways & Limitations
Start-to-end drift provides an evaluation route without full-sequence external ground truth, and start/end alignment checks indicate that alignment error mainly reflects accumulated drift.
Takeaways & Limitations
The loop-drift evaluation is valid only when loop closure is disabled, and its error metric can become degenerate when translational drift exceeds the trajectory’s standard deviation.
Abstract
from arXiv · showhide
We present a dataset for evaluating the tracking accuracy of monocular visual odometry and SLAM methods. It contains 50 real-world sequences comprising more than 100 minutes of video, recorded across dozens of different environments -- ranging from narrow indoor corridors to wide outdoor scenes. All sequences contain mostly exploring camera motion, starting and ending at the same position. This allows to evaluate tracking accuracy via the accumulated drift from start to end, without requiring ground truth for the full sequence. In contrast to existing datasets, all sequences are photometrically calibrated. We provide exposure times for each frame as reported by the sensor, the camera response function, and dense lens attenuation factors. We also propose a novel, simple approach to non-parametric vignette calibration, which requires minimal set-up and is easy to reproduce. Finally, we thoroughly evaluate two existing methods (ORB-SLAM and DSO) on the dataset, including an analysis of the effect of image resolution, camera field of view, and the camera motion direction.
1. Introduction
The paper introduces a monocular VO/SLAM benchmark designed to address limited sensor calibration and dataset diversity in existing evaluations. It provides photometrically calibrated data, broad environmental coverage, and loop-based drift evaluation without requiring full-sequence external ground truth.
- Sensor Calibration: Existing VO/SLAM methods are often evaluated without access to the full image formation pipeline, including exposure, response, vignetting, and geometric distortion.This is especially relevant for direct methods and for algorithms that could exploit sensor design information rather than treating these effects as unknown noise.
- Benchmark Size: Meaningful comparisons require large datasets spanning varied scenes because small changes can strongly affect these nonlinear estimation problems.Existing datasets often contain limited environments, while external motion-capture systems are costly, time-consuming, and restricted to small indoor areas.
- Dataset: The proposed dataset contains 50 sequences totaling 105 minutes across dozens of environments, with frame-wise exposure times and calibrations for camera response and lens vignetting.The dataset includes over 190’000 frames and two different lenses.
- Evaluation: Tracking accuracy is evaluated from accumulated drift after a large loop, avoiding the need for external-sensor ground truth throughout the sequence.The sequences start and end at the same position, enabling start-to-end drift measurement.
- Calibration: The paper proposes a straightforward non-parametric calibration procedure for response and vignetting that requires minimal setup and avoids imposing a lens- or sensor-specific parametric model.The related calibration literature includes parametric representations, while response estimation can recover a 28-valued inverse-response lookup table from differently exposed images.
2. Calibration
The benchmark provides geometric and photometric camera calibration, including a response-function procedure and a dense, non-parametric vignette method. These calibrations use exposure-controlled images, planar-scene observations, and large amounts of data to estimate sensor and lens effects.
- Calibration setup: The dataset provides standard FOV intrinsic calibration and photometric calibration for camera response and pixel-wise lens attenuation.The photometric model represents observed pixels using exposure time, irradiance, response function, and vignetting.
- Geometric intrinsic calibration: The cameras use a pinhole model with FOV distortion, whose parameters are calibrated from checkerboard images and whose inverse is available in closed form.The model supports fisheye lenses and backprojection from image points and depth.
- Response calibration: The response function is estimated from static-scene images captured at known, varying exposure times and spanning a wide range of gray values.Overexposed pixels are excluded, and the inverse response is optimized alternately with irradiance before scaling U(255) = 255.
- Response calibration: 1000 images covering 120 exposure times from 0.05ms to 20ms replace a smoothness prior for response estimation; fewer exposures require regularization.The calibration sequence changes exposure slowly while recording a static scene.
- Non-parametric vignette calibration: The vignette method estimates a dense attenuation map from several hundred images of a bright, predominantly white Lambertian planar scene.Camera pose is obtained with an AR marker or another method, and image observations are projected onto the plane.
- Non-parametric vignette calibration: Vignette and planar irradiance are estimated by alternating minimization without an explicit smoothness or radial-symmetry prior.The attenuation map is scaled so max(V) = 1; observed radial symmetry and smoothness arise from the data term.
- Non-parametric vignette calibration: The vignette optimization is well-constrained when its variable graph is fully connected, which sufficient input images make highly probable in practice.This observability condition is stated for the unregularized optimization problem.
3. Evaluation Metrics
The benchmark estimates full-trajectory tracking drift by aligning tracked start and end segments from looped sequences, then uses alignment error to combine scale, rotation, and translation effects.
- Loop-closure evaluation: Exploring sequences end with a repeated, easy-to-track scene that enables precise alignment of the first and last 10–20 seconds.LSD-SLAM tracks these segments to generate ground truth for their relative pose, with trajectory scale normalized to approximately 100.
- Loop-closure evaluation: Accumulated drift over the full sequence evaluates VO accuracy without requiring external ground truth for every frame.The method is valid only when loop closure is disabled; pre-loop-closure drift remains correlated with long-term accuracy after loop closure.
- Error measures: Alignment error is less sensitive to drift timing than translational error and joint RMSE, which can become degenerate when drift is large.The evaluation compares error metrics after artificial scale and rotation jumps at different points in a sequence.
- Error measures: The procedure independently aligns the tracked trajectory to start and end segments, producing relative transformations used to compute scale, rotation, and translation drift.Start and end segments need sufficient poses in a non-degenerate configuration, motivating the slow, loopy camera motion at both ends.
- Error measures: Alignment error is the full-trajectory translational RMSE between trajectories aligned to the start and end segments.It equally reflects scale, rotation, and translation drift through their effects on tracked positions and can apply to different VO or SLAM modes.
- Evaluation protocol: The benchmark aggregates cumulative errors across 50 sequences, forward and backward runs, repeated five times, under real-time and non-real-time execution.For scale, results use e′_s = max(e_s, e^-1_s) because scale error is multiplicative.
4. Benchmark
The benchmark evaluates monocular VO and SLAM across diverse sequences using cumulative error distributions that reflect both tracking accuracy and robustness. Experiments examine failure handling, calibration validity, field of view, resolution, motion direction, and reproducibility.
- Evaluation methodology: Cumulative error plots summarize both accuracy and robustness across the 50-sequence benchmark, avoiding hand-picked successful runs.Each method is evaluated over 500 runs: five forwards and five backwards executions for every sequence.
- Data variations: A smaller field of view significantly decreases accuracy and robustness for both ORB-SLAM and DSO.The experiment rectifies images to different fields of view at fixed 640 × 480 resolution.
- Evaluation methodology: The benchmark disables explicit loop closure and re-localization, uses VGA pinhole-rectified images, and lowers ORB-SLAM’s lost threshold to 10 inlier observations.These settings allow the proposed metric to be applied consistently while retaining implicit small loop closures.
- Data variations: Image resolution strongly affects ORB-SLAM but only marginally affects DSO, consistent with the sub-pixel accuracy of direct approaches.The comparison does not account for changes in computational complexity because the algorithms were not run in real time.
- Data variations: Backward motion substantially improves ORB-SLAM’s performance, whereas DSO is largely unaffected, revealing dataset motion bias.The result motivates evaluation on larger datasets spanning diverse environments and motion patterns.
- Ground truth validation: Start- and end-segment RMSE is roughly 100 times lower than alignment RMSE and similar for both methods, indicating that alignment error mainly reflects accumulated drift.DSO’s automatic initialization occasionally fails, producing infinite errors, while ORB-SLAM does not exhibit this failure in the reported validation.
- Dataset and reproducibility: The release includes raw images, exposure times, ground-truth segment alignments, geometric and photometric calibrations, calibration datasets, and evaluation code.The dataset contains 43GB of raw images and 190,000 frames, alongside 13GB of calibration data.
- Known issues: Rare exposure-time logs may be shifted by one frame because the camera SDK permits only asynchronous exposure-time queries.This is reported as a known issue despite the use of industry-grade cameras.