PaperlayerView exact revision on arXiv

Source-linked AI summary

DARTS: Deceiving Autonomous Cars with Toxic Signs

Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Mung Chiang, Prateek Mittal

arXiv:1802.06430v3cs.CRcs.CV

TL;DR

Traffic-sign recognition in autonomous cars can be deceived by physically realizable attacks, while prior adversarial examples were restricted to training/testing data. The paper proposes Out-of-Distribution and Lenticular Printing attacks and evaluates them across virtual and real-world settings and white-box and black-box models. The attacks succeed broadly, including over 90% real-world success for Out-of-Distribution and In-Distribution attacks and improved performance of Out-of-Distribution attacks against adversarially trained classifiers.

  • Problem

    Prior physical attacks were constrained by controlled laboratory conditions and by starting from training/testing data, motivating broader attacks against autonomous-car sign recognition.

  • Method

    The paper develops Out-of-Distribution attacks that begin anywhere in image space and Lenticular Printing attacks that exploit angle-dependent optical displays.

  • Results

    The attacks succeed in virtual and real-world settings under white-box and black-box threat models; real-world success exceeds 90% for Out-of-Distribution and In-Distribution attacks, and Out-of-Distribution attacks outperform In-Distribution attacks against adversarial training defenses.

  • Takeaways & Limitations

    The results expose security challenges for physically deployed traffic-sign recognition systems and identify attack vectors that existing adversarial-training defenses may not mitigate.

  • Takeaways & Limitations

    The paper leaves the effectiveness of detection-based defenses against Out-of-Distribution attacks as an open question and notes that autonomous-vehicle decisions also use sensors beyond computer vision.

Abstract

from arXiv · show

Sign recognition is an integral part of autonomous cars. Any misclassification of traffic signs can potentially lead to a multitude of disastrous consequences, ranging from a life-threatening accident to even a large-scale interruption of transportation services relying on autonomous cars. In this paper, we propose and examine security attacks against sign recognition systems for Deceiving Autonomous caRs with Toxic Signs (we call the proposed attacks DARTS). In particular, we introduce two novel methods to create these toxic signs. First, we propose Out-of-Distribution attacks, which expand the scope of adversarial examples by enabling the adversary to generate these starting from an arbitrary point in the image space compared to prior attacks which are restricted to existing training/test data (In-Distribution). Second, we present the Lenticular Printing attack, which relies on an optical phenomenon to deceive the traffic sign recognition system. We extensively evaluate the effectiveness of the proposed attacks in both virtual and real-world settings and consider both white-box and black-box threat models. Our results demonstrate that the proposed attacks are successful under both settings and threat models. We further show that Out-of-Distribution attacks can outperform In-Distribution attacks on classifiers defended using the adversarial training defense, exposing a new attack vector for these defenses.

1 INTRODUCTION

The paper introduces physically realizable attacks that make traffic signs appear benign while deceiving autonomous-car sign recognition, extending adversarial examples beyond training and testing data. It evaluates these attacks across virtual and real-world settings and white-box and black-box threat models, reporting high-confidence deception and over-90% real-world success for two attack types.

  • Traffic-sign misclassification in autonomous cars can produce consequences ranging from life-threatening accidents to widespread traffic confusion.
  • Prior physical attacks were limited by controlled laboratory conditions and by generating adversarial examples from training/testing data, called In-Distribution attacks.The cited prior work did not account for variations such as brightness, viewing angle, distance, or image resizing.
  • The paper also introduces Lenticular Printing attacks, exploiting different observation angles for the human driver and vehicle-mounted camera to display different sign images.The method combines images with an array of magnifying lenses to create angle-dependent displays.
  • DARTS introduces Out-of-Distribution attacks that start from arbitrary points in image space rather than restricting adversarial-example generation to training/testing samples.The approach is motivated by the many objects present in autonomous cars’ complex surroundings.
  • The attacks are evaluated in virtual and real-world settings under white-box and black-box threat models, including automated drive-by tests.The authors report attack success rates above 90% in real-world tests for both Out-of-Distribution and In-Distribution attacks.
  • Out-of-Distribution attacks outperform In-Distribution attacks against classifiers defended with adversarial training, while Lenticular Printing attacks are intrinsically problematic for known ML-based defenses.

2 BACKGROUND

The paper reviews supervised neural-network classification, the traffic-sign recognition pipeline, adversarial examples, and white-box and black-box threat models. It also motivates physically robust attacks because conventional adversarial examples may fail under real-world conditions.

  • Supervised machine learning systems: A classifier is trained by minimizing loss over labeled samples drawn from a distribution of traffic-sign images.The notation defines the classifier, parameters, loss, samples, labels, and data distribution.
  • Traffic-sign recognition pipeline: Traffic-sign recognition uses detection followed by classification, with detected patches resized before a neural network assigns labels using confidence scores and a threshold.The described pipeline uses shape-based detection and discards low-confidence classifications.
  • Adversarial examples: Evasion attacks modify benign inputs with perturbations so the classifier assigns a targeted class or any class different from the ground truth.The paper distinguishes targeted and untargeted attacks during the test phase.
  • Adversarial examples: Standard adversarial examples may fail under brightness, viewing-angle, and resizing variations encountered in real-world traffic-sign recognition.The paper incorporates these conditions into a modified optimization problem for physically robust examples.
  • Threat models: The study considers white-box attackers with model access and black-box attackers without direct or query access, with black-box attacks relying on transferability.The white-box setting is justified through surrogate-model training and query-based attack results discussed by the authors.

3 ATTACKS: ADVERSARIAL EXAMPLES

The attack framework generates physically robust adversarial traffic-sign examples from either arbitrary environmental images or existing traffic signs. It uses masked optimization and randomized transformations, then prints and tests the resulting examples under real-world conditions.

  • Attack overview: The pipeline creates a mask, resizes the image and mask, optimizes a perturbation, resizes and adds it to the image, then prints and tests the result.The mask restricts perturbations to the sign area, while the final step evaluates robustness in real-world conditions.
  • Physically robust optimization: Equation 2 optimizes perturbations across randomized brightness, perspective, and size transformations to preserve adversarial behavior under physical conditions.The transformations are differentiated so the optimization can use gradients.
  • Attack overview: The attack pipeline supports Out-of-Distribution inputs such as advertisements, logos, graffiti, or drawings, and In-Distribution inputs consisting of existing traffic signs.The adversary chooses an original image and target class before generating the adversarial example.
  • Out-of-Distribution attacks: Logo attacks modify common logos so they are detected and classified as traffic signs while constraining perturbations to remain effective under transformations.Their ubiquity supports deployment in real-world environments such as city streets.
  • Out-of-Distribution attacks: Custom Sign attacks generate adversarial signs from blank signs, using masks to embed inconspicuous graffiti-like images or text appropriate to the surroundings.The perturbation norm is constrained by the mask shape rather than penalized, allowing flexible drawn shapes.
  • Attack effectiveness: Generated adversarial examples are classified consistently as target signs with high confidence across varying physical conditions, unlike ordinary false recognitions.Ordinary non-sign objects typically receive low confidence or inconsistent labels across backgrounds and viewing angles.

4 ATTACKS: LENTICULAR PRINTING

The Lenticular Printing attack creates traffic signs that appear differently to human drivers and vehicle-mounted cameras by exploiting their distinct observation angles. It combines interlaced images with a lens array to produce angle-dependent visual content.

  • Lenticular Printing attacks target sign recognition by exploiting the different observation angles of human drivers and vehicle-mounted cameras.
  • The attack creates special traffic signs that display different images from different viewing angles.
  • Lenticular printing combines at least two images with an array of magnifying lenses to create angle-dependent visual content.
  • Choosing an appropriate lens array: For the lens array, the attacker interlaces image rows with Widthrows = p/2, where p is the lens width.
  • Creating malicious signs: Simple lenticular images can be produced with an inkjet or laser color printer and a lenticular lens, without specialized industrial machinery.

5 EXPERIMENTAL GOALS AND SETUP

The experiments assess whether Out-of-Distribution and related attacks remain effective across virtual, real-world, white-box, and black-box settings. The study also examines practical attack generation and whether existing defenses withstand these attacks.

  • The experiments evaluate Out-of-Distribution attacks in virtual and real-world settings using a pipeline for generating, evaluating, and selecting adversarial examples.
  • The study asks whether Out-of-Distribution attacks can break state-of-the-art neural-network defenses based on adversarial training.
  • The evaluation covers transferability-based black-box attacks in real-world settings and the ability of Lenticular Printing attacks to fool classifiers.
  • The pipeline is described as a low-overhead procedure for testing In-Distribution and Out-of-Distribution examples before conducting effective real-world attacks.
  • Simulated Physical Attack Success estimates physical robustness by applying 10 randomized composite transformations involving brightness, perspective, and resizing.

6 WHITE-BOX ATTACKS

White-box experiments evaluate Out-of-Distribution and In-Distribution attacks on standard and defended classifiers, including virtual, simulated physical, and drive-by settings. Out-of-Distribution attacks remain effective against adversarial training, while both attack types exceed 90% success in drive-by tests.

  • 6.1 Virtual Out-of-Distribution attacks: The virtual evaluation uses Logo and Custom Sign images as two types of Out-of-Distribution inputs.
  • 6.1 Virtual Out-of-Distribution attacks: 85.71% VAS and 65.07% SPAS were achieved by Logo attacks, with average confidence 0.975.
  • 6.3 Real-world attacks: In drive-by tests, both Out-of-Distribution and In-Distribution adversarial examples achieved attack success rates exceeding 90%.
  • 6.4 Attacking defended models: Adversarial training was evaluated as a defense against both Out-of-Distribution and In-Distribution attacks, as well as Lenticular Printing attacks.
  • 6.4 Attacking defended models: Adversarial training reduced auxiliary-data In-Distribution VAS from 54.34% to 2.53% and SPAS from 36.65% to 2.47%.
  • 6.4 Attacking defended models: Against adversarial training, Out-of-Distribution Logo attacks retained 11.42% VAS and 7.42% SPAS, about 3−4× more effective than In-Distribution attacks.

7 BLACK-BOX ATTACKS ARE POSSIBLE

The paper demonstrates that traffic-sign attacks can succeed in black-box settings and that Lenticular Printing can exploit differences between camera and human viewing heights.

  • Black-box transferability: Adversarial examples generated for Multi-scale CNN were tested on Standard CNN to evaluate black-box transferability.The recognition pipeline remained unchanged while the classifier was replaced.
  • Real-world black-box attacks: 97.71% and 96.68% attack success rates were achieved by Out-of-Distribution Custom Sign and In-Distribution attacks, respectively, in real-world black-box tests.These rates were comparable to the white-box setting.
  • Virtual black-box attacks: Out-of-Distribution Logo attacks retained virtual black-box success rates above 10% for both VAS and SPAS despite a significant drop from white-box performance.The result indicates successful attacks with limited knowledge of the target classifier.
  • Lenticular printing attacks: Lenticular Printing can fool the classifier when the AV camera and human observer view the sign from different heights, while the sign appears correct to the human.The attack uses an optical change in displayed images across observation angles.
  • Attack construction: The attacker can estimate driver and camera observation angles from their heights and the sign height when designing the physical sign.This measurement supports constructing the sign for the intended viewing geometry.

8 LIMITATIONS AND FUTURE WORK

The paper identifies open questions around defenses, sensor fusion, and how to measure the visibility of physically realizable adversarial perturbations.

  • Defenses: Detection-based defenses against Out-of-Distribution attacks remain an open question, and the paper leaves their evaluation for future work.The discussion specifically names Feature Squeezing and MagNet.
  • Sensor fusion: The study does not attack the simultaneous sensor inputs used alongside computer vision, such as LIDAR, GPS, and radar.Future work is proposed to examine attacks spanning these subsystems.
  • Perturbation measurement: The paper uses the L1 norm to measure perturbation visibility, but whether it is the best distance function remains an open research question.The authors encourage user studies comparing perceptual proxies including L∞ and L2.

9 RELATED WORK

Prior work established virtual and physical attacks on classifiers, but the cited physical attacks were conducted in controlled laboratory settings that omitted important real-world variations.

  • Virtual attacks: Earlier evasion attacks targeted Support Vector Machines, random forests, and neural networks, while poisoning attacks targeted classifiers and generative models.Related work also includes attacks on reinforcement-learning policies and models with structured inputs.

10 CONCLUSION

The conclusion presents Out-of-Distribution and Lenticular Printing attacks as effective threats to traffic-sign recognition, including in real-world black-box settings.

  • Conclusion: Out-of-Distribution attacks can convert any sign or logo into a targeted adversarial example.The authors frame these attacks as having severe consequences for self-driving cars.
  • Conclusion: Lenticular Printing embeds a potentially dangerous traffic sign into an innocuous one by making images appear different from varying heights.The conclusion states that this attack requires no access to classifier internals.
  • Conclusion: The attacks were demonstrated in both virtual and real-world settings, including black-box attacks in a real-world setting.The authors also evaluated possible countermeasures against physical realizations of robust adversarial examples.

A APPENDIX: DATASET DETAILS

The appendix describes the datasets, recognition models, and detection pipeline used to evaluate DARTS. Experiments use GTSRB traffic signs alongside auxiliary traffic, logo, and custom-sign data.

  • Datasets: GTSRB contains 51,839 32 × 32 RGB images covering 43 traffic-sign types.Its 39,209 training samples are augmented to 86,000 using perspective transformations and flipping where appropriate.
  • Detection pipeline: The detection pipeline smooths video frames, applies Canny edge detection, and uses a circle Hough transform to locate circular signs.The design focuses on circular signs, while triangular signs can be detected similarly.
  • Datasets: The auxiliary traffic data combines real traffic-sign photographs and computer-generated drawings against simple backgrounds.The appendix also includes high-resolution logo images and blank custom signs with six masks.
  • Recognition models: The white-box Multi-scale CNN uses three convolutional and two fully connected layers, with pooled features concatenated across convolutional layers.It is trained on augmented data with random perspective, brightness, and color adjustments.
  • Recognition models: The Multi-scale CNN reaches 98.50% validation accuracy, while the full detector-classifier pipeline reaches 77.10% mAP at IoU 0.5 on GTSDB.These figures characterize the white-box model and its end-to-end detection performance.
  • Recognition models: The black-box Standard CNN reaches 98.66% validation accuracy and 81.54% mAP on GTSDB.Attacks in this setting are generated using the Multi-scale CNN as the adversary’s assumed model.

D APPENDIX: BASELINE RESULTS

The appendix compares DARTS with Vanilla Optimization and examines parameter trade-offs affecting attack success, physical robustness, and perturbation size. Results show improved deterioration behavior for the proposed attack, while defenses and black-box access reduce success metrics.

  • Experimental setup: The comparison uses 1,000 randomly selected GTSRB test signs and excludes the masking and resizing effects present in the main high-resolution experiments.Consequently, these appendix results are not directly comparable with the main-text results.
  • Baseline comparison: The proposed attack has a much lower deterioration rate than Vanilla Optimization attacks.With similar average norm, the result emphasizes the effect of adding random transformations to the optimization process.
  • Baseline comparison: Larger perturbation norms can improve transformation robustness for Vanilla Optimization, but the proposed attack achieves lower deterioration with a similar average norm.The four reported metrics should be interpreted together because they represent attack trade-offs.
  • Robustness and examples: Adversarially trained Multi-scale CNNs and the black-box setting produce lower VAS and SPAS than the baseline white-box setting.The appendix also presents examples from all three attacks that achieve 100% SPAS.
  • Parameter analysis: The baseline attack uses L1 norm with c = 3, K = 100, L = 30, T = 128, θ1 = 0.07, and θ2 = 0.15.Parameter sweeps vary c, p, T, and transformation randomness, using Adam with learning rate 0.02 without decay.

G APPENDIX: ADVERSARIAL TRAINING

The appendix discusses adversarial training and the practical cost of incorporating physically robust attack generation. It reports difficulties with iterative adversarial training and contrasts In-Distribution and Out-of-Distribution attack examples.

  • Training limitation: Generating the physically robust modified-loss examples takes around 60 seconds per adversarial example, making per-batch optimization impractical.The ideal robust-generation procedure is therefore not used for every training batch.
  • Adversarial training: Fast Gradient Sign attacks are fast enough to be incorporated into neural-network training for adversarial robustness.FGS forms an untargeted perturbation by adding a scaled sign of the loss gradient.
  • Adversarial training: Iterative adversarial training raised convergence issues for the low-capacity Multi-scale CNN with augmented training data.The authors observed that its performance on auxiliary-traffic adversarial examples was inferior to standard adversarial training.
  • Attack examples: The appendix distinguishes In-Distribution adversarial traffic-sign examples from Out-of-Distribution logo examples.The figures show samples together with the labels assigned by the classifier.
Loading 1802.06430v3…