On Offline Evaluation of Vision-based Driving Models

1 Introduction

• Offline evaluation compares model outputs with annotated data, whereas online evaluation places the model in an environment and measures driving interaction.This distinction motivates evaluating whether static prediction performance reflects safe, comfortable, rule-compliant driving.Source: §1 · p. 1, Figure 1 · p. 2
• End-to-end models are commonly assessed by average prediction error on expert-driving datasets, but its relationship to actual driving performance remains uncertain.Decomposed evaluation of perception and control components does not directly apply to end-to-end image-to-control models.Source: §1 · pp. 1–2
• Offline prediction accuracy and actual driving quality are surprisingly weakly correlated, especially for forward-facing-camera data collected on expert trajectories.A model with very low prediction error can drive catastrophically, while one with relatively high error may drive well.Source: §1 · p. 2
• Lateral-camera validation data correlates with driving performance better than prediction measured from the forward-facing camera alone.The lateral images can come from cameras mounted to collect additional imitation-learning data.Source: §1 · p. 2
• Offline metrics that depart from mean squared error correlate with driving performance more than 60% better than MSE.The paper presents metric-design and validation-data choices intended to improve offline evaluation without deploying vehicles.Source: §1 · pp. 2–3

2 Related Work

• Annotated datasets such as KITTI, Cityscapes, and BDDV benchmark individual vision tasks, but they do not evaluate a complete driving system.The listed tasks include depth, odometry, optical flow, detection, segmentation, and 3D prediction.Source: §2 · p. 3
• Direct-control datasets record observations and driving signals across varied settings, yet their passive pre-recorded nature prevents evaluation of actual driving performance.Examples include Udacity, CommaAI, Oxford RobotCar, and BDDV.Source: §2 · p. 3
• Simulation provides a practical alternative for online evaluation, with CARLA enabling policy tests in populated towns under varied weather and illumination conditions.Earlier simulators and commercial games offer more limited environments or controllability for driving evaluation.Source: §2 · p. 3
• End-to-end driving research spans early road-following systems, collision avoidance, affordance-based control, memory networks, conditional imitation learning, and physical-vehicle deployment.This work focuses on evaluating the associated metrics for end-to-end trained models while remaining applicable to broader vision-based pipelines.Source: §2 · p. 4

3.1 Conditional Imitation Learning

• Conditional imitation learning maps an image and high-level navigation command to a driving action, with this study focusing on steering-angle prediction.Training examples contain an observation, command, and expert action; commands are continue, straight, left, and right.Source: §3.1 · pp. 4–5

3.2 Training

• Training uses 80 hours of simulated expert driving from forward and lateral cameras, including perturbed-policy examples for recovery behavior.The expert uses privileged environment information, and 10% of collected data includes injected driving-policy noise.Source: §3.2 · p. 5
• The experiments compare reactive convolutional networks with 4, 8, or 12 convolutional layers and evaluate driving using success, goal-directed distance, and kilometers between infractions.The network receives the current image and vehicle speed but cannot use temporal context; online metrics include navigation success, progress toward the goal, and incident-free travel distance.Source: §3.2 · p. 6, §3.3 · p. 7

3.3 Performance metrics

• Offline evaluation compares predicted and ground-truth steering on temporally ordered validation sequences using standard and purpose-designed error metrics.The evaluated metrics include MSE, absolute error, speed-weighted error, temporally accumulated error, and quantized or relative-error measures.Source: §3.3 · pp. 6–7

4 Experiments

• The CARLA study evaluates 45 models across training data, architecture, and training-procedure variations in Town 1 and unseen Town 2 conditions.Models are assessed through goal-directed navigation trials and offline metrics, with real-world validation additionally performed on BDDV.Source: §4 · p. 7, §4.2 · p. 8, §4.1 · p. 8

4.3 Correlation between offline and online metrics

• Offline and online performance correlate weakly in the basic setup, with steering MSE reaching only 0.39, versus 0.54 with action noise and 0.77 using three cameras.The results indicate that validation data should include observations away from expert trajectories, including recovery from perturbations.Source: §4.3 · p. 9
• Absolute steering error correlates at 0.61 with navigation success, while quantized classification error and thresholded relative error reach 0.65 and 0.64; MSE remains at 0.39.Speed weighting and multi-step accumulation do not improve correlation in this comparison.Source: §4.3 · p. 10, Figure 3 · p. 10
• Success rate correlates with average completion at 0.8 and kilometers per infraction at 0.77, but those two metrics correlate at only 0.44.The authors therefore recommend measuring several online metrics while using success rate as the main metric.Source: §4.3 · p. 11, Figure 4 · p. 11
• Two models with similar prediction accuracy behave very differently: Model 1 crashes in every trial, whereas Model 2 successfully completes most trials.A short-term steering error during a turn can cause a crash while remaining difficult to detect from average prediction error.Source: §4.3 · p. 11, Figure 5 · p. 12

4.4 Real-world data

• On BDDV, turn-specific prediction error provides the largest separation between the best and worst models, while speed-weighted metrics reduce the gap between feedforward and LSTM models.The reported metrics are adaptations for 4-way classification accuracy and cannot directly use the continuous-value metrics from the simulation experiments.Source: §4.4 · p. 12, Table 2 · p. 13

4.5 Detailed evaluation of models

• Across detailed variations, more training data, data balancing, regularization, deeper networks, and L1 loss generally improve driving, while camera count has a minor effect.The L1 result agrees with the stronger correlation of absolute error than MSE.Source: §4.5 · p. 13
• TRE identifies the best-driving model in 10 of 12 cases, compared with 6 of 12 for MSE.This detailed model-selection test shows TRE is more predictive of driving performance than MSE, although it is not perfectly correlated.Source: §4.5 · p. 13

• MSE prediction error is very weakly correlated with actual driving quality, while modifying validation data and offline metrics improves the correlation.Source: §5 · p. 13
• TRE generally identifies the best-driving model more often than MSE, selecting 10 of 12 cases versus 6 of 12.Source: Table 3 · p. 14
• The evaluation is almost entirely simulation-based, so the detailed correlation analysis may not transfer unchanged to physical vehicles in rich real-world environments.Source: §5 · p. 14
• Even the best offline metric remains far from perfectly correlated with driving quality, leaving stronger offline metrics as an open challenge.Source: §5 · p. 14

• The paper is titled “On Offline Evaluation of Vision-based Driving Models.”Source: Supplementary material for “On Offline… · p. 17
• Felipe Codevilla, Antonio M. López, Vladlen Koltun, and Alexey Dosovitskiy are listed as authors.Source: Supplementary material for “On Offline… · p. 17
• The authors are affiliated with the Computer Vision Center at Universitat Autònoma de Barcelona and Intel Labs.Source: Supplementary material for “On Offline… · p. 17
• The document is identified as arXiv:1809.04843v1, dated 13 September 2018.Source: Supplementary material for “On Offline… · p. 17

1.1 Network architecture

• The experiments compare standard, deeper 12-convolutional-layer, and shallower 4-convolutional-layer network architectures.Source: §1.1 · p. 17, Table 1 · p. 17

1.2 Image Input

• Input images are cropped, resized from 800 × 384 to 200×88 pixels, mean-subtracted, and normalized before network processing.Source: §1.2 · pp. 17–18

1.3 Augmentation

• Training uses Gaussian blur, Gaussian noise, pixel dropout, brightness, contrast, saturation, and region pixel-dropout augmentations.Source: §1.3 · p. 18

1.4 Noise Distribution

• Steering noise is injected in 10% of training data at a random time as a triangular impulse perturbation.Source: §1.4 · p. 18

• Table 2 lists the model parameters varied during the evaluation.Source: §2 · p. 18, Table 2 · p. 18
• The material is supplementary content for “On Offline Evaluation of Vision-based Driving Models.”Source: §2 · p. 19

Additional results

• The supplementary evaluation covers Town 1 training conditions and Town 2 generalization conditions, with analyses using either all models or the best 50% selected by offline performance.Figures 1–2 use the best 50% in Town 1; Figures 4–7 use all models across Town 1 and Town 2.Source: §3 · p. 19

Best-model analysis

• Figures 1 and 2 examine relationships between goal-directed navigation success and offline metrics for the best-performing 50% of models in Town 1.Source: Figure 1 · p. 19, §3 · p. 19, Figure 2 · p. 20

Online metric relationships

• Figure 3 compares online driving-quality metrics in Town 1: success rate, average completion, and distance driven between infractions.Source: §3 · p. 19, Figure 3 · p. 20

All-model analysis

• Figures 4–7 evaluate success rate against steering absolute error and other offline metrics using all models in Town 1 and Town 2, including different evaluation-data distributions.Source: §3 · p. 19, Figure 4 · p. 21, Figure 5 · p. 21