Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems

1 INTRODUCTION

• Simulation results may not correlate with on-road failures, while full-scale testing is costly, hazardous, and beyond the reach of many academic groups.Small-scale vehicles offer an intermediate option for testing autonomous driving algorithms on real vehicles.Source: §1 · p. 1
• The testbed excludes traffic, uses limited speed, and has a known road layout to isolate lane-keeping DNN deficiencies from other tasks.This setup matches short-track scenes used by lane-keeping test generators,.Source: §1 · p. 2
• Steering-angle predictions are statistically indistinguishable across environments, but trajectories and model uncertainty differ, so offline testing alone is not viable.The authors recommend online testing, possibly validated in the field.Source: §1 · p. 2
• Failures from image corruptions are more frequent and severe physically, while virtual adversarial failures do not correlate with real-world failures and tend to be overestimated.The study examines 642 simulations involving image corruptions and adversarial examples.Source: §1 · p. 2
• The study compares system-level testing on a physical vehicle and its digital counterpart, assessing transfer of driving behavior and failure exposure.It evaluates three lane-keeping models: DAVE-2, Chauffeur, and Epoch.Source: §1 · p. 2
• Virtual predictive uncertainty can help prioritize SDC tests and reduce physical testing.The reported uncertainty data allow halving the number of time- and cost-intensive physical tests.Source: §1 · p. 2
• Neural image translation was used for visual odometry and may also help estimate other sensory data where in-field training is costly.The paper mentions reinforcement learning as one such domain.Source: §1 · p. 2

2.1 Model-level vs System-level Testing

• Model-level testing uses metrics such as accuracy, MSE, or MAE on unseen observations, but prediction errors do not reliably indicate field performance or failures.The causal chain from individual prediction errors to system failures is difficult to identify and anticipate.Source: §2.1 · p. 2
• System-level testing embeds the DNN in an SDC and evaluates behavior and system failures, unlike model-level testing on stationary inputs.Off-road driving is one example of a safety-requirement violation.Source: §2.1 · p. 2

2.2 Virtual vs Physical Testing Scenario

• Simulators support safe, reproducible system-level testing, but their fidelity may be insufficient to establish that failures—or their absence—generalize to real-world driving.Physical hardware-in-the-loop testing remains important despite its time and resource costs.Source: §2.2 · p. 2
• The paper compares behavioral-cloning lane-keeping systems tested on virtual and physical closed-loop tracks, using virtual/digital/simulated and physical/real-world interchangeably.The terms refer to the paper’s virtual and physical testing settings.Source: §2.2 · p. 3

3.1 Donkey Car

• Donkey Car provides an open-hardware 1:16 scale radio-controlled car, lane-keeping training and testing tools, and a high-fidelity simulator.The simulator allows models to be trained and tested without the physical car.Source: §3.1 · p. 3
• Donkey Car was selected because it supports simulated-to-physical transfer studies, unlike some alternatives without simulators, and is affordable.The authors also cite prior use of the framework on real vehicles,,,.Source: §3.1 · p. 3

3.2 Framework and Simulation Platform

• The framework collects images with steering and throttle labels during manual driving, supports Keras/TensorFlow model training, and evaluates the trained model in the real world or simulator.The authors collected training data using a PS4 controller.Source: §3.2 · p. 3

4 METRICS

• Driving quality is compared using steering angle, lateral deviation, and predictive uncertainty, metrics used to assess lane-keeping and model confidence.The first two metrics follow Jahangirova et al., and uncertainty is used in the SDC domain.Source: §4 · p. 3

4.1 Steering Angle

• Steering angle is the DNN’s real-valued prediction from −1 to +1, with negative values indicating left turns, positive values right turns, and values near zero straight driving.The physical car’s servo was calibrated to match the simulator’s ±25° steering range.Source: §4.1 · p. 3

4.2 Lateral Position

• Lateral position is measured as cross-track error (XTE), the distance from the vehicle’s cruising position to the segment between nearby centerline waypoints.Physical XTE is estimated from onboard camera images using visual odometry because the vehicle does not provide it directly.Source: §4.2 · p. 3
• A CycleGAN translates simulator images toward the real-image distribution so a telemetry predictor trained with simulator XTE labels can process physical images.The predictor’s accuracy on real-world data is evaluated separately.Source: §4.2 · pp. 3–4

4.3 Predictive Uncertainty

• Predictive uncertainty is estimated as the variance of stochastic dropout-based predictions; higher variance indicates lower confidence.The prediction is the mean of the distribution generated by repeated processing of the same image.Source: §4.3 · p. 4

5 EMPIRICAL STUDY

• The empirical evaluation compares virtual and physical SDC behavior and failure exposure across digital and real-world test environments.The study uses small-scale vehicles for the comparison.Source: §5 · p. 4

5.1 Research Questions

• The study asks whether visual odometry estimates lateral position accurately, whether behavioral metrics and failures transfer between environments, and whether simulation telemetry can guide physical-test selection.The test-selection question considers whether high steering, XTE, or uncertainty values identify scenarios worth executing in the physical environment.Source: §5.1 · p. 4

5.2.1 Self Driving Car Models

• Three publicly available lane-keeping models—DAVE-2, Chauffeur, and Epoch—are evaluated in simulation and on a physical vehicle.The models were selected as robust systems previously studied in testing work.Source: §5.2.1 · p. 4

5.2.2 Camera Calibration

• The simulator camera was matched to the Donkey Car camera in field of view, aperture, focal length, sensor size, frame rate, and height.The matched camera settings were 160-degree field of view, aperture 2.35, 3.15 mm focal length, 16 mm sensor size, and 21 fps.Source: §5.2.2 · p. 4

5.2.3 Testing Tracks

• The physical and simulated tests use an 11 m by 52 cm track; the Unity scene reproduces its road shape, colors, and car-to-track proportions.A high-resolution image of the real track was imported into Unity.Source: §5.2.3 · p. 5

5.3.1 CycleGAN Data Collection, Setup, and Training

• CycleGAN was trained on 12,000 unpaired virtual and real images, and its outputs were assessed operationally through an XTE predictor rather than a standalone image-quality evaluation.The network used 75 epochs followed by 20 epochs of linear learning-rate decay; the authors state that detailed generative-network evaluation was out of scope.Source: §5.3.1 · p. 5

5.3.2 XTE Predictor Data Collection, Setup, and Training

• The XTE predictor was trained on balanced, simulator-labeled virtual images translated into pseudo-real images, allowing labeled training data to be produced automatically.The predictor minimizes mean squared error between predicted and ground-truth XTE values.Source: §5.3.2 · p. 5

5.3.3 Metrics for visual odometry evaluation

• For visual-odometry evaluation, the vehicle was positioned manually at five XTE trajectories, and the predictor was tested on 6,638 manually labeled real-world images.The images were uniformly distributed across the five XTE classes; accuracy was measured using MAE numerically and in centimeters.Source: §5.3.3 · p. 5

5.4.1 SDC Data Collection

• Virtual and physical driving datasets were collected by manually steering the vehicle for 30 laps on each track, encouraging it to remain near the centerline.Source: §5.4.1 · p. 5

5.4.2 SDC Model Setup & Training

• Six models were trained, one per DNN and environment, and each drove 50 consecutive laps without crashing or leaving the road; physical-car battery voltage was kept above 11.8 V.Lower battery values were found to jeopardize predictions and driving quality.Source: §5.4.2 · pp. 5–6
• Telemetry from the first and second sets of 25 laps was statistically indistinguishable in both environments, supporting use of all 50 laps for transferability analysis.The reported F-test power exceeded β = 0.8 and Mann–Whitney U test p-values exceeded α = 0.05.Source: §5.4.2 · p. 6

5.4.3 Metrics for SDC Model Evaluation

• Driving behavior transfer is assessed by comparing virtual and physical distributions of steering angle, lateral deviation, and predictive uncertainty.Steering angle and XTE characterize driving behavior, while uncertainty reflects model confidence during driving.Source: §5.4.3 · p. 6

5.5.1 Test Scenarios

• The tests inject unknown conditions using black-box image corruptions and white-box adversarial examples, including Hendrycks-style corruptions and UAPr targeted to steer left or right.UAPr applies a universal perturbation to every frame, learned offline using the tested DNN's training images.Source: §5.5.1 · p. 6
• The evaluation comprises 642 one-lap runs: 570 corruption runs and 72 UAPr runs across three models, two environments, and the stated perturbation configurations.The UAPr runs cover six configurations and two severity levels.Source: §5.5.1 · p. 6

5.5.2 Metrics

• Outcomes are categorized as successful, OBE when the vehicle goes off-road but recovers, or OOT when it cannot recover and complete the track.OOT is classified as severe, while OBE represents a lower deviation from requirements.Source: §5.5.2 · p. 6

5.6.1 Procedure

• For simulation-successful scenarios, only tests with steering, XTE, or uncertainty above the virtual-driving median are selected for physical execution.The selection procedure weighs avoided low-telemetry tests against failures that might be missed.Source: §5.6.1 · p. 7

5.6.2 Metrics

• The test-selection metrics count true and false positives and negatives according to whether selected or discarded tests fail in the real environment.The supplied passage defines true positives as selected tests with real-world OBE or OOT failures and false positives as selected tests that do not fail.Source: §5.6.2 · p. 7

5.7.1 Visual Odometry Accuracy (RQ0)

• 2.33 cm (4% of road width) was the average real-world XTE estimation error, supporting visual odometry as a reliable lateral-position estimate.The study uses this estimate to assess transferability of SDC behavior.Source: §5.7.1 · p. 7

5.7.2 Transferability of Behaviours (RQ1)

• Steering-angle distributions transferred, but XTE and uncertainty distributions did not; physical uncertainty was generally higher and physical trajectories were less smooth.Illumination changes and image noise, as well as actuator latency, may contribute to the differences.Source: §5.7.2 · p. 8, Figure 5 · p. 8

5.7.3 Transferability of Failures (RQ2)

• Nearly 60% of configurations generalized across environments; corruption failures were generally worse physically, while no adversarial case was worse in the real world than in simulation.Corruption severity increased by 37% for DAVE-2, 29% for Chauffeur, and 2% for Epoch; adversarial attacks were less successful physically.Source: §5.7.3 · pp. 8–9, Table 3 · p. 9

5.7.4 Test Selection (RQ3)

• 40/61 configurations (66%) could avoid physical testing while exposing all 4 OBE failures and 17/21 OOT failures; estimated time savings were 44/86 (51%).In this setting, the saving amounts to approximately 13 hours; virtual uncertainty was used to prioritize real-world execution.Source: Table 4 · p. 9, §5.7.4 · p. 9

5.8.1 Internal validity

• The custom simulator track was built from a real-track image to preserve proportions, while training and fine-tuning competitive public models mitigated data-quality concerns.The models drove consistently in nominal conditions and failed only in unseen conditions.Source: §5.8.1 · p. 9

5.8.2 External validity

• The evaluation uses a limited number of models, one physical track, one 1:16 car, and a minimalistic lane-keeping scene, constraining generalizability.The authors note that findings may not hold, or may hold only partially, across other physical settings or RC platforms.Source: §5.8.2 · pp. 9–10

5.8.3 Reproducibility

• Replication requires a DonkeyCar and racing track costing approximately $500, although the software artifacts, results, videos, and simulator are available.Source: §5.8.3 · p. 10

• CycleGAN translation could help match real driving images with labeled simulated images and estimate XTE for reinforcement-learning rewards.The authors say visual odometry could estimate XTE more accurately than existing computer-vision heuristics that are hard to tune.Source: §6 · p. 10
• Steering-angle distributions can look similar across environments even when trajectories differ substantially, making XTE estimates important for assessing system-level transfer.The telemetry predictor was instrumental to assessing the gap; without XTE, the authors would have wrongly concluded transferability was high.Source: §6 · p. 10
• The authors envision using CycleGAN to create pseudo-real training data and dynamically generated pseudo-real virtual testing environments.Source: §6 · p. 10
• Approximately 60% of testing conditions generalize across environments, while virtual adversarial attacks tend to overestimate real-world failures.The authors suggest pre-selecting conditions whose virtual and physical failures correlate.Source: §6 · p. 10
• MC-Dropout uncertainty on the virtual platform can select promising mismatching test cases for real-world execution, with a potential saving of 12.8 hours on the short track.The authors suggest savings could be greater on longer tracks.Source: §6 · p. 10
• Physical testing remains necessary because real-world noise and mechanical stochasticity can expose discrepancies that virtual testing misses.The authors hypothesize that poor photo-realism and inadequate modeling of sensory and environmental uncertainty contribute to the gap, and suggest complementary digital twins.Source: §6 · p. 10

7.1 Model- and System-level Testing Approaches

• Prior model-level tests assess individual predictions, while system-level scenario generators have largely been evaluated in simulation; this study compares simulated and physical SDCs.The applicability of existing test generators to real driving scenarios remains unexplored in the cited work.Source: §7.1 · pp. 10–11

7.2 Empirical Studies

• Earlier empirical studies examined offline versus online testing or simulation-only settings, whereas this study measures the system-level gap between virtual and physical environments.The authors describe this transferability question as previously unexplored.Source: §7.2 · p. 11

7.3 Virtual vs Physical Testing of Autonomous Vehicles

• Prior scaled-vehicle research explored reinforcement learning, test tracks, and cooperative driving; this study uses a scaled vehicle to examine transferability of SDC testing results.Source: §7.3 · p. 11

7.4 Digital Twins

• Earlier digital-twin studies did not assess whether the simulator reproduced real-world failures, while this work investigates transfer between a digital twin and physical SDC.The cited truck digital twin also uses a closed, known-layout environment with no traffic and relatively low speeds.Source: §7.4 · p. 11

7.5 GAN-based Testing of Autonomous Vehicles

• Unlike paired-scene GAN testing and offline perturbation studies, this work applies real-image perturbations online and uses pairing-free CycleGAN scenes to support telemetry prediction.The authors report that the CycleGAN contribution was indispensable for correctly assessing the sim2real gap.Source: §7.5 · p. 11

• Some DNN failures are exposed only in real-world conditions, so physical testing remains important despite the widespread use of simulators.Source: §8 · p. 12
• Visual odometry can provide accurate telemetry from real SDCs, while simulation uncertainty can prioritize successful test scenarios for physical execution.Real-world driving-quality metrics such as XTE tend to be worse, likely due to illumination changes and mechanical effects that are difficult to simulate accurately.Source: §8 · p. 12