DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems

• Roadside billboards are selected because they are rentable and customizable, large and usually benign, and can conceal adversarial artwork within otherwise normal content.Source: §I · pp. 1–2
• Single-frame digital perturbations may fail in continuous driving because frame conditions vary and later frames can correct an isolated mis-steering.Source: §I · pp. 1–2
• DeepBillboard jointly optimizes perturbations across captured frames, minimizing inter-frame interference while accounting for driving patterns, environmental changes, and printer constraints.Source: §I · p. 2
• Digital experiments misled the average steering angle up to 41.93 degree, while physical case studies produced average steering angle error from 4.86 up to 26.44 degree under various weather conditions.Source: §I · p. 2
• The work proposes evaluation metrics and methodology for measuring steering-model perturbation effectiveness in both digital and physical domains.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

• The paper focuses on CNN steering models that decide from the currently captured image, leaving RNN testing for future work while noting that gradient calculation could be adapted.Its driving-system scope is camera input and steering-angle output, as in NVIDIA Dave [3].Source: §II · pp. 2–3
• Prior work uses fast-gradient [18], targeted optimization [4], [19], transformations, [14] or GANs; generated examples may never exist in reality, while DeepBillboard tests physical perturbations on real billboards under changing distances and view angles.Source: §II · p. 3
• Prior physical-example methods used synthetic transformations, whereas DeepBillboard samples synthetic and real-world conditions, modifies full-sized objects, and targets continuous driving scenarios.Earlier work also showed camera-captured adversarial examples [5] and 3D-printed perturbed objects robust to synthetic transformations [21].Source: §II · p. 3
• Unlike physical eyeglass attacks in relatively stable conditions, DeepBillboard targets unconstrained billboard driving and designs perturbations for diverse distances, angles, and resolution changes.The passage notes that environmental variation can reduce perturbation effectiveness.Source: §II · p. 3
• DeepBillboard targets steering models with printable perturbations on drive-by billboards, optimizing across contiguous frames as distance and viewing angle change.It seeks to maximize the possibility and degree of misleading steering angles and introduces joint optimization for digital and physical attack generation.Source: §II · p. 3
• Unlike prior road-sign attacks, [26], DeepBillboard tests steering models using billboards that can be placed anywhere and affect a continuous drive-by sequence.Prior work also examined physical road-sign examples [22] and digital detection or segmentation attacks [23],, [25].Source: §II · p. 3

A. Adversarial Scenarios

• DeepBillboard aims to mislead CNN-based steering models [3],,,, by printing adversarial perturbations on roadside billboards, causing vehicles to off-track from the lane center.The models take dashcam images as input and output steering-angle decisions; the approach does not involve detection or segmentation algorithms.Source: A. Adversarial Scenarios · p. 3
• Because vehicle speed is usually uncontrollable in a physical attack, the paper uses steering-angle error between ground-truth and misled steering as its effectiveness measure.It relates off-tracking distance to speed, inference interval, and steering divergence: v·i·sin(α′ − α).Source: A. Adversarial Scenarios · p. 3
• Rather than target one fixed distance and viewing angle, DeepBillboard seeks to mislead steering across a sequence of dashcam frames captured as a vehicle approaches and passes the billboard.The frames vary in viewing distance and angle.Source: A. Adversarial Scenarios · p. 3
• Accounting for the full drive-by frame sequence is intended to increase both the possibility and degree of misled steering, whereas simpler strategies may improve one frame while lowering the overall objective.The number of frames depends on dashcam FPS and the time taken to pass the billboard; the paper says simpler methods soon make little progress on the overall objective.Source: A. Adversarial Scenarios · p. 4

B. Evaluating Matrices

• The frame set ˆX covers frames possibly captured under any driving pattern, while frames captured in a particular drive-by scenario form a subset X ⊆ ˆX.The objective is to affect almost every frame in ˆX so that potential real-world scenarios have a maximized chance of being affected.Source: B. Evaluating Matrices · p. 4
• M0 measures mean angle error across the exhaustive frame set; higher values indicate a greater chance and larger degree of steering-angle error while passing the billboard.The targeted steering model's prediction on each perturbed frame is compared with its unperturbed prediction.Source: B. Evaluating Matrices · p. 4
• M1, the percentage of possible frames whose steering-angle error exceeds threshold τ, is the primary metric in physical-world case studies because it counts frames with unacceptable decisions.For a 40MPH vehicle, a one-meter off-track distance within 0.2 seconds corresponds to τ = 16.24.Source: B. Evaluating Matrices · p. 4

Spatial Constraints

• Physical billboards restrict perturbations to the billboard area, while the surrounding background changes with the vehicle camera’s distance and viewing angle.Digital attacks that alter background imagery such as the sky therefore do not transfer directly.Source: C. Challenges · p. 4

Physical Limits on Imperceptibility

• Printed perturbations must remain visible to the camera, so acceptable imperceptibility depends on physical constraints and sensing hardware.Digital perturbations can be small enough to appear almost imperceptible to people, but may be too subtle for a camera to perceive.Source: C. Challenges · p. 4

Environmental Conditions

• Perturbations must survive changing camera distances and angles, lighting and weather, and debris on the camera or billboard.Drive-by frames are captured at different distances and viewing angles.Source: C. Challenges · p. 4

Fabrication Error

• Printable perturbations must use real-world printable colors and account for pixel mismatches introduced by fabrication devices such as printers.Source: C. Challenges · p. 4

Context Sensitivity

• A single printable image must be globally optimized across frames, because each frame has its own optimal perturbation but must be attacked in context.Each perturbed frame maps to a printable image for a particular viewing angle and distance; the overall objective includes maximizing M0.Source: C. Challenges · p. 4

Joint Loss Optimization

• DeepBillboard jointly optimizes one printable perturbation across all captured frames, seeking to mislead steering predictions throughout a drive-by.The attack is framed as inference dodging, with loss measuring the difference between predicted and ground-truth steering angles. More driving videos enlarge the frame set and can strengthen testing, at increased time cost.Source: D. The Design of DeepBillboard · p. 5

Handling Overlapped Perturbations

• To limit conflicting updates, DeepBillboard updates only the top k influential pixels per frame, with n · k < m, and greedily selects values for overlaps.For overlapping pixels, the algorithm can choose the maximum gradient, sum gradients, or the proposed value with greatest overall influence on the objective.Source: D. The Design of DeepBillboard · pp. 5–7

Enhancing Perturbation Printability

• DeepBillboard minimizes nonprintability score (NPS), defined by each perturbed pixel’s maximum distance from the set of printable RGB values.The perturbation’s NPS is the sum of the NPS values of its pixels.Source: D. The Design of DeepBillboard · p. 5

Adjust Color Difference under Various Environment Conditions

• For each frame, a color-adjustment function compensates for environmental shifts in billboard pixel colors observed by the dashcam.The method assumes that pixel color differences within an image are nearly the same under a given condition.Source: D. The Design of DeepBillboard · pp. 5, 7

Algorithm overview

• The algorithm shuffles frames, processes batches, computes constrained gradients, projects updates into frames, and accepts improvements or simulated-annealing proposals.It initializes the printable RGB perturbation with a unicolor; gold, blue, and green were reported as efficient choices in the authors’ experiments.Source: D. The Design of DeepBillboard · pp. 6–7, Figure 2 · p. 6

• The evaluation examines DeepBillboard’s efficacy in both digital and physical settings across steering models and road scenes.Source: §IV · p. 7

A. Experiment Setup

• Four pretrained steering CNNs—Dave V1, Dave V2, Dave V3, and Epoch —were evaluated; Epoch was trained using Udacity challenge data.The Dave models are variants of the NVIDIA Dave architecture.Source: A. Experiment Setup · p. 7
• The digital datasets comprise 101,396 Udacity training images, 45,568 Dave testing images, and 14,999 Kitti images.Physical case-study videos were recorded from a windshield-mounted tachograph while driving past a campus roadside billboard; the steering models' frame predictions served as ground truth.Source: A. Experiment Setup · p. 7
• Average angle error across scene frames measures efficacy; the comparison baseline is each model’s inference angle, with segment-level statistics accommodating unavailable or non-unique ground-truth behavior.The authors report average error and percentage over the driving segment rather than assessing individual frames.Source: A. Experiment Setup · pp. 7, 9
• Digital testing selects seven scenes with billboards initially fully visible above 400 pixels and partly absent in the last frame, spanning straight and curved lanes.Researchers labeled billboard corners per frame, using Adobe After Effects motion tracking with manual adjustments, then patched adversarial examples into frames via the projection function and compared model predictions with dataset steering decisions.Source: A. Experiment Setup · pp. 7, 9
• Physical tests covered sunny, cloudy, and dusk conditions, with recordings starting about 100 ft from the roadside billboard and ending after passing it.Phase I recorded training videos at 10 mph using white boards with black corners and golden boards with blue corners; Phase II applied generated perturbations and recorded test drives at 20 mph.Source: A. Experiment Setup · p. 9
• Because actual autonomous vehicles were unavailable, physical evaluation used recorded driving-pattern videos, including abnormal patterns, to approximate varied viewing angles, distances, and misled-driving scenarios.The abnormal-driving videos were used during adversarial construction and training; the authors characterize this as a similar approach to prior autonomous-driving studies,, [14].Source: A. Experiment Setup · p. 9

B. Digital Perturbation Results

• DeepBillboard produced observable average steering-angle divergences for every model in every scene, exceeding 10° for Dave V1 in 6 of 7 scenes and 16.7° for Dave V2 among all scenes.Dave V1’s exception was Kitti Straight1, where the billboard occupies a small space.Source: B. Digital Perturbation Results · pp. 9–10
• Dave V3’s average angle error ranged from 0.44° to 25.01°, and its dropout layers and augmented training data were cited as contributing to greater robustness and generalization.The authors attribute part of dropout’s effect to randomly deactivating half the neurons, which can deactivate billboard perturbations; Epoch also uses dropout but lacks Dave V3’s pavement-cropping augmentation and has larger average errors.Source: B. Digital Perturbation Results · p. 10
• In most scenes, steering-angle errors increased as billboard size increased, but some scenes deviated from this trend and even small billboards produced large divergences.Figure 3 shows the timeline results; in one example, middle frames contributed more steering-angle error for Dave V1.Source: B. Digital Perturbation Results · p. 10

C. Parameter Tuning

• Yellow initialization generally converges better than green, while y(5,max) and y(10,sum) outperform other tested settings; neither batch size nor max versus sum clearly dominates.The parameter comparison finds that choosing the initial billboard color carefully can improve convergence speed and results, but does not establish a generally superior batch-size or gradient-update choice.Source: C. Parameter Tuning · p. 10, Figure 4 · p. 10
• Training-frame selection changes convergence and final AAE; larger billboard images generally yield better results, and updating more pixels usually improves results and convergence speed.All tested frame subsets rise quickly early in training, with growth slowing after about 400 iterations; larger k usually helps except when the billboard occupies few pixels, where aggressive updates cause frame interference and lower AAE.Source: C. Parameter Tuning · pp. 10–11, Figure 5 · p. 10

Physical test setup

• Physical tests used Dave V1 on a straight lane, with 20 mph testing runs from 100 ft away until the vehicle passed a 6′×4′ billboard.Training videos were recorded at 10 mph on three slightly different routes—central, left-shifting, and right-shifting—and tests covered sunny, cloudy, and dusk weather.Source: D. Physical Case Study · p. 11

Metrics and danger threshold

• The 19.8° error threshold was chosen because at 20 mph it implies at least one meter of off-track distance in 0.33 seconds, or 10 frames at 30 FPS.Exp AAE measures expected average angle error from training videos, Test AAE measures actual average angle error over the physical testing video, and M1 records the percentage of test frames above threshold.Source: D. Physical Case Study · p. 11

Billboard appearance and steering direction

• The bright adversarial billboard steered left, while the dark billboard steered right, with direction controlled by the gradient flag (+/-).With the empty white billboard, steering angles remained almost straight across the tested distances.Source: D. Physical Case Study · p. 11

Physical effectiveness

• 268 of 900 frames exceeded the dangerous-mis-steering threshold; M1 exceeded 19% in four of six scenarios and reached 100%.Across all three weather conditions, both adversarial settings produced relatively large Exp AAE and Test AAE values, with Test AAE only slightly below Exp AAE in many cases; sunny left-misleading steering reached 8.88 degrees.Source: D. Physical Case Study · p. 11

Per-frame steering errors

• Under sunny conditions, left-misleading tests produced positive angle errors and right-misleading tests negative errors, becoming more observable in later frames.The other two weather scenarios were omitted from the per-frame plot because they showed similar result trends.Source: D. Physical Case Study · p. 11, Figure 6 · p. 11

V. CONCLUSION

• Experiments demonstrate DeepBillboard’s efficacy for testing various steering models in digital and physical-world scenarios.The conclusion reports results across various scenarios without providing quantitative comparisons here.Source: §V · p. 13
• Robust Joint Optimization generates billboard perturbations that can be patched digitally or physically to cause mis-steering across multiple frames and changing viewing distances and angles.The perturbations target consistent mis-steering as viewing conditions change.Source: §V · pp. 11, 13
• DeepBillboard can be generalized beyond roadside billboards to other physical surfaces, such as graffiti painted on a wall.Source: §V · p. 13