Source-linked AI summary
Failure or Drift? Evaluating Monocular SLAM under Synthetic and Real-World Corruptions
Abhay Skaria Thomas, Shashank Agnihotri, Margret Keuper
TL;DR
Clean-trajectory benchmarks provide limited evidence about monocular SLAM under adverse conditions, motivating a test of whether synthetic corruptions preserve real-world engineering conclusions. The paper compares classical and learned trackers across image-space, geometry-aware, and compound corruptions, then evaluates corresponding 4Seasons conditions with failure-aware metrics. Learned trackers often replace explicit loss with drift, and tracker ordering depends on corruption fidelity: rain and fog proxies transfer, whereas a simple illumination proxy can reverse the preferred system.
Problem
Monocular SLAM is commonly evaluated on clean trajectories, leaving limited evidence about whether controlled corruptions support conclusions applicable to adverse real-world conditions.
Method
The study evaluates ORB-SLAM2, DPVO, and DROID-SLAM under three corruption families and compares them with four adverse 4Seasons conditions using trajectory validity, APE, and RPE.
Results
Structured rain and fog proxies preserve learned-tracker ordering in real data, while evening illumination reverses it; learned systems continue but can accumulate substantial drift.
Takeaways & Limitations
Synthetic corruptions are useful controlled diagnostic interventions, but they are not calibrated forecasts of deployment error and should be judged by transfer of relative behavior.
Takeaways & Limitations
The controlled sweep uses one KITTI trajectory and one archived execution per condition, while 4Seasons comparisons are semantically matched rather than paired recordings.
Abstract
from arXiv · showhide
Visual SLAM is commonly evaluated on clean trajectories, although deployment failures are often caused by adverse weather, illumination, blur, and sensor artifacts. Controlled corruptions are attractive because they isolate such factors, but a synthetic stress test is useful only when it leads to the same engineering conclusion as the condition it is intended to approximate. This work examines that question for monocular SLAM. We evaluate a classical feature-based system and two learned trackers under image-space, geometry-aware, and compound corruptions, and compare their behavior with adverse conditions from 4Seasons. Rather than reducing robustness to a single trajectory error, the evaluation separates explicit tracking failure from drift accumulated by methods that remain active. The results show that learned trackers largely replace catastrophic loss with sustained, and sometimes severe, drift. More importantly, the apparent ordering of the learned systems changes with the physical fidelity of the corruption: structured rain and fog proxies preserve the real-world ordering, whereas a simple illumination proxy does not. Code is available at: https://github.com/abhaythomas/master_thesis_vslamlab_robustness.
1 Introduction
This work asks whether controlled corruptions support the same tracker-selection conclusions as real adverse conditions. It evaluates three monocular SLAM paradigms with failure-aware reporting that separates missing trajectories from accumulated drift.
- Motivation: Clean-trajectory evaluation misses deployment conditions where rain, fog, illumination, blur, compression, exposure variation, and noise weaken correspondences.These changes can disrupt local tracking and the optimization that propagates information across time.
- Failure-aware evaluation: Robustness must distinguish explicit tracking loss from trajectories that remain available but accumulate error over time.A learned tracker may continue producing poses while incorrect estimates reach mapping, planning, or control.
- Experimental rationale: Synthetic corruptions hold route, calibration, timestamps, and ground truth fixed, but visual plausibility alone may not preserve which tracker is favored.Different corruptions can alter different subsets of correspondences in a stateful SLAM system.
- Study scope: The study compares ORB-SLAM2, DPVO, and DROID-SLAM across 115 corrupted KITTI sequence-00 variants and four 4Seasons sequences.Corruptions span image-space, geometry-aware, and compound effects at five severity levels.
- Findings: DPVO has lower APE in 30 of 43 paired image-space settings, whereas DROID-SLAM is lower in all 33 geometry-aware and 14 of 15 compound settings.ORB-SLAM2 often returns no evaluable trajectory, while the learned systems continue and may accumulate substantial drift.
- Findings: Structured rain and winter preserve the compound-proxy ordering in real data, but evening illumination reverses it.Thus, convenient image transformations can recommend the wrong architecture for the intended condition.
2 Related Work
Prior work established reproducible nominal benchmarks, simulated and real adverse-condition datasets, and parameterized corruption protocols. This paper positions its contribution as testing whether controlled perturbation conclusions remain stable as corruption fidelity increases and conditions become real.
- Emerging systems: Recent SLAM systems increasingly couple tracking with dense reconstruction, making adverse-condition evaluation increasingly important.Examples include implicit scene representations, 3D Gaussians, two-view reconstruction priors, and registered local pointmaps.
- Benchmarks: KITTI and TUM RGB-D support reproducible trajectory evaluation, while TartanAir and 4Seasons broaden simulated and natural environmental variation.Real adverse conditions remain entangled with route and sensor differences.
- Robustness protocols: Robustness tools standardize experiments, vary trajectories, expose failure boundaries, or provide modular perturbations across vision and geometric tasks.The paper asks a distinct synthetic-to-real validity question for SLAM.
- Research gap: Synthetic-corruption studies show that aggregate correspondence can coexist with corruption-specific mismatch, a harder issue for stateful SLAM because errors propagate through poses, optimization, and map state.Accordingly, this work compares failure behavior and method ordering rather than equality of absolute trajectory errors.
3 Experimental Design
The experimental design uses a common, non-adaptive protocol spanning clean KITTI, controlled synthetic corruptions, and condition-matched but unpaired 4Seasons data. Evaluation first records trajectory validity, then computes pose errors only for valid runs.
- Protocol: All methods use a common protocol without corruption-specific adaptation, with archived trajectories, logs, runtime records, and evaluation outputs.Released weights are used without fine-tuning, restoration, or condition-specific adaptation.
- Datasets: The clean baseline uses eleven KITTI odometry training sequences, while controlled corruptions are generated from sequence 00 with route, calibration, timestamps, and ground truth fixed.Using one controlled route supports diagnosis but limits generalization.
- Corruptions: The corruption suite includes 13 image-space transformations and seven geometry-aware effects, with compound corruptions added for interaction testing.Geometry-aware effects use depth or physically motivated scene structure where applicable.
- Synthetic-to-real comparison: The real comparison uses four 4Seasons neighborhood sequences matched to synthetic clean, rain with contrast, fog with contrast, and brightness with contrast conditions.These are condition-level matches rather than paired recordings.
- Systems: ORB-SLAM2 represents sparse feature-based SLAM, DPVO learned patch-based visual odometry, and DROID-SLAM dense learned correspondence with global multi-frame optimization.The systems are evaluated without corrupted-data fine-tuning or test-time restoration.
- Failure-aware evaluation: The valid-run rate records whether an attempted run returns a non-empty trajectory associated with ground truth before error metrics are computed.This binary measure avoids conflating keyframe or pose density with genuine tracking loss.
- Metrics: APE measures accumulated global disagreement, whereas RPE measures local motion inconsistency, and both are conditioned on a valid trajectory.The study also uses descriptive Spearman correlations over five severity points as diagnostics rather than inferential statistics.
4 Results
The results separate explicit tracking failure from drift and show that corruption fidelity can reverse which learned tracker appears more robust. Structured rain and fog proxies transfer their relative ordering to real conditions, while a simple illumination proxy does not.
- Do the Systems Fail, or Do They Drift?: All three systems return evaluable trajectories on clean KITTI, but their accuracy remains strongly route-dependent.ORB-SLAM2 has the lowest median conditional APE at 8.2 m yet reaches 518.0 m on sequence 01; DPVO and DROID-SLAM have medians of 64.9 m and 82.0 m.
- Do the Systems Fail, or Do They Drift?: Under synthetic corruption, ORB-SLAM2 often returns no evaluable trajectory, whereas DPVO and DROID-SLAM remain active and accumulate substantial drift.ORB-SLAM2 is valid in 37/63 image-space, 15/23 geometry-aware, and 10/15 compound attempts; learned trackers frequently exceed 100 m aligned APE under geometry-aware and compound effects.
- Does Corruption Fidelity Change Which Tracker Appears Robust?: DPVO has lower APE in 30/43 paired image-space settings, while DROID-SLAM is lower in 33/33 geometry-aware and 14/15 compound settings.Brightness and contrast consistently favor DPVO, whereas geometry-aware effects consistently favor DROID-SLAM; fog and contrast at severity 5 are the sole compound reversal.
- Are the Trends Consistent across Image-Space, Geometry-Aware, and Compound Effects?: Proxy-layer agreement is tracker- and condition-dependent, so nominal severity does not consistently represent SLAM difficulty.For example, DPVO’s low-light curves have ρ = 1.00 while its rain curves have ρ = −0.10; DROID-SLAM shows the reverse pattern for rain and low illumination.
- Which Conclusions Transfer to Real Adverse Conditions?: Synthetic-real transfer holds for two of three adverse conditions and does not establish equivalence or predict absolute real-world error.Global brightness and contrast omit camera-specific noise, automatic exposure, spatially varying illumination, headlights, motion-dependent blur, and route effects.
5 Discussion
The discussion links tracker behavior to system design and argues that corruption choice determines which robustness conclusions transfer to real conditions. It also distinguishes continued pose output from trustworthy trajectory accuracy.
- 5 Discussion: ORB-SLAM2’s shared ORB representation enables geometric checks and loop correction when keypoints survive, but exposes it to front-end degradation.Learned trackers maintain continuity through soft or dense correspondences, though local compatibility can still permit accumulated drift.
- 5 Discussion: DPVO’s local-window optimization lacks loop closure and global correction, allowing persistent correspondence biases to accumulate into substantial global drift.
- 5 Discussion: DROID-SLAM’s dense correlations and global bundle adjustment are consistent with lower APE across geometry-aware comparisons and 14 of 15 compound settings.These architectural links are mechanism-consistent explanations, not causal attributions, because individual components were not ablated.
- 5 Discussion: Correspondence confidence weights are not calibrated probabilities of trajectory correctness, so deployment systems need signals validated against drift and impending tracking loss.Useful candidates include geometric residuals, graph connectivity, local-global agreement, and temporal consistency.
- 5 Discussion: Synthetic corruption design should follow the mechanism being tested: image-space effects probe front-end stability, while geometry-aware and compound effects alter scene information more structurally.
6 Limitations
The study’s controlled and natural-condition comparisons have limited scope, incomplete coverage, and do not support broad equivalence claims across routes, modalities, or system resources.
- 6 Limitations: The controlled sweep uses one KITTI trajectory and one archived execution per condition, leaving scene dependence, initialization variance, and runtime nondeterminism unmeasured.The binary validity criterion also distinguishes missing output from evaluable output without measuring the supported duration of truncated trajectories.
- 6 Limitations: 4Seasons conditions are semantically matched rather than paired recordings, supporting relative-behavior comparisons but not causal or numerical equivalence.
- 6 Limitations: The study is monocular and trajectory-centered, excluding other modalities, inertial fusion, map accuracy, reconstruction completeness, runtime, memory, and energy.
7 Future Work
Future work will test whether controlled proxies reproduce field failure modes across broader natural conditions while measuring map quality and resource use.
- 7 Future Work: Future experiments will introduce temporally coherent, sensor-aware corruptions and a broader set of matched natural conditions.
- 7 Future Work: The evaluation will add map-quality and resource measurements rather than compressing accuracy, continuity, reconstruction, and efficiency into one score.
- 7 Future Work: The immediate aim is to identify which controlled proxies reproduce specific field failure modes.
8 Conclusion
The conclusion separates missing trajectories from drift and treats synthetic corruptions as controlled diagnostics rather than calibrated deployment forecasts. It emphasizes validating whether conclusions survive the corresponding natural condition.
- 8 Conclusion: ORB-SLAM2 frequently returns no evaluable trajectory, whereas DPVO and DROID-SLAM continue producing poses while accumulating drift.These represent distinct operational failures and should be reported separately.
- 8 Conclusion: Synthetic corruptions are valuable diagnostic interventions but not calibrated forecasts of deployment error.Rain and winter preserve compound-proxy ordering in real data, whereas evening illumination does not.
- 8 Conclusion: Robustness evaluation should assess tracking availability, bounded error, and whether proxy conclusions survive the intended natural condition.