PaperlayerView exact revision on arXiv

Source-linked AI summary

SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems

Edward Chou, Florian Tramèr, Giancarlo Pellegrino

arXiv:1812.00292v4cs.CR

TL;DR

Localized universal attacks can arise through diverse mechanisms and threaten reused neural-network systems, making attack-specific detection difficult. SentiNet detects them without prior attack knowledge by using the model’s behavior alongside interpretability and segmentation techniques, achieving high detection rates across varied attacks and robustness to adaptive adversaries.

  • Problem

    Neural networks face diverse localized and universal attacks, while attack-specific defenses require anticipating many mechanisms and vectors.

  • Method

    SentiNet uses the protected neural network’s malicious behavior, Grad-CAM-based localization, and image segmentation to detect localized universal attacks without prior attack-vector knowledge.

  • Results

    SentiNet detects data poisoning, network trojaning, and adversarial-patch attacks, with detection rates between 98.5% and 99.2% in four attacks and 85.5% true-positive rate in one attack.

  • Takeaways & Limitations

    SentiNet’s reliance on attacks fooling the model makes detection largely unaffected by attack mechanism or deployment and supports masking attacks to preserve functionality.

Abstract

from arXiv · show

SentiNet is a novel detection framework for localized universal attacks on neural networks. These attacks restrict adversarial noise to contiguous portions of an image and are reusable with different images -- constraints that prove useful for generating physically-realizable attacks. Unlike most other works on adversarial detection, SentiNet does not require training a model or preknowledge of an attack prior to detection. Our approach is appealing due to the large number of possible mechanisms and attack-vectors that an attack-specific defense would have to consider. By leveraging the neural network's susceptibility to attacks and by using techniques from model interpretability and object detection as detection mechanisms, SentiNet turns a weakness of a model into a strength. We demonstrate the effectiveness of SentiNet on three different attacks -- i.e., data poisoning attacks, trojaned networks, and adversarial patches (including physically realizable attacks) -- and show that our defense is able to achieve very competitive performance metrics for all three threats. Finally, we show that SentiNet is robust against strong adaptive adversaries, who build adversarial patches that specifically target the components of SentiNet's architecture.

I. INTRODUCTION

SentiNet addresses attack-vector-agnostic detection of localized universal attacks, which can affect compromised or uncompromised neural networks and support physically realizable attacks. It uses model behavior, interpretability, and object-detection techniques, achieving strong detection rates across attack types and robustness against adaptive adversaries.

  • Motivation: Localized universal attacks constrain adversarial regions to small contiguous image areas and reuse them across different images, supporting physically realizable deployments.These attacks typically use unbounded perturbations to tolerate viewpoint and other real-world changes.
  • Approach: SentiNet is designed to detect localized universal attacks without prior knowledge of the specific attack vector.The framework targets attacks involving trojaned networks, poisoned networks, and adversarial patches.
  • Approach: SentiNet identifies potential attack regions using model visualization and object detection, then tests the extracted attack across multiple benign images.The deployed attack is fed back into the network to support attack classification.
  • Evaluation: 96.22% average true positive rate and 95.36% average true negative rate were reported across protection experiments covering three attack vectors.The evaluation included backdoors, trojan triggers, and adversarial patches, including physically printed patches.
  • Evaluation: SentiNet was evaluated against a fully adaptive white-box adversary through seven attacks targeting its core components and was reported resistant to these attacks.The authors assessed robustness by testing each component individually.

A. Deep Learning Systems

The paper models localized universal attacks as reusable malicious objects that can hijack predictions in compromised or uncompromised CNN systems. SentiNet proposes classes and salient regions, then uses the model’s responses to those regions for attack detection.

  • Deep Learning Systems: A deep learning system takes input, computes an output, and makes a prediction, while CNNs are neural-network architectures primarily targeted at computer-vision tasks.The paper represents a model as (y, conf) = fm(x), returning a prediction and confidence for input x.
  • Threat Model: Localized universal attacks place a malicious object in a small contiguous image region and reuse that object across many different input images.The attacker’s goal is to hijack the model prediction and control actions performed by the deep learning system.
  • Threat Model: Attack vectors include poisoning or trojaning a network before deployment and adversarial patches that fool an uncompromised model at test time.Poisoning inserts backdoors during training, whereas adversarial patches exploit instability in uncompromised models under conditions such as lighting, noise, and rotation.
  • SentiNet Architecture: SentiNet extracts influential regions from an input, applies them to benign test inputs, and compares the resulting model behavior with benign behavior to detect prediction hijacking.The architecture feeds these tests back into the protected model for boundary analysis and attack classification.
  • Class Proposal via Segmentation: Selective search segments the input into region proposals, evaluates each segment, and returns the k most confident predictions after excluding the primary class.For ImageNet, the paper sets k to 2 because images contain around 1.5 classes on average.
  • Mask Generation: Grad-CAM uses final-layer gradients to identify salient contiguous regions, whose heatmap is binarized at 15% of maximum intensity to form masks.The resulting masks may include both malicious and benign salient regions, motivating additional class-proposal masks.

B. Attack Detection

SentiNet overlays suspected regions on benign test images, measures how often they induce misclassification and their inert-pattern confidence, then classifies outliers using a clean-example decision boundary.

  • Test Generation: SentiNet overlays each suspected region on benign test images and counts fooled inputs to assess whether the region is adversarial.A higher number of fooled mutated images indicates a greater likelihood of an adversarial attack.
  • Test Generation: Large recovered regions can cause misclassifications through occlusion, limiting the technique’s discrimination between adversarial and benign inputs.The method is effective when the recovered region is small because small benign objects typically cannot overwhelm the network’s prediction.
  • Test Generation: An inert-pattern test separates attack effects from misclassification caused by occluding the original object.Adversarial regions should fool many test images before replacement but have little effect after replacement; benign regions should have few effects or remain disruptive through occlusion.
  • Decision Boundary for Detection: The detector uses fooled-image count and average inert-pattern confidence as two features in a one-class classifier trained on clean examples.Clean-example statistics define abnormal outliers instead of relying on manually chosen thresholds.
  • Decision Boundary for Detection: A curve is approximated from clean-example points selected at high y-values within x-intervals, and points outside the curve are classified as attacks.The boundary is fitted with nonlinear least squares from sampled behavior.
  • Decision Boundary for Detection: Figure 4 plots adversarial metrics as red triangles and benign metrics as blue circles, with the fitted benign curve shown separately from sampled test images.The example uses 400 data points and random noise as the inert pattern.

IV. EVALUATION

The evaluation examines SentiNet across previously proposed attacks and then considers adaptive adversaries, reporting protection results for adversarial patches, trojan triggers, and backdoors.

  • Robustness and Efficiency: The study further evaluates SentiNet’s robustness against adaptive adversaries with full knowledge of its inner workings.Runtime latency is examined separately as an efficiency measure.
  • Evaluation Scope: The evaluation covers adversarial patches, trojan triggers, and backdoors to assess SentiNet’s versatility.These attacks are evaluated before a later extensive evaluation against adaptive adversaries with full knowledge of SentiNet’s workings.
  • Effectiveness: 96.22% average true positive rate and 95.36% average true negative rate are reported for protection against the three attacks.The evaluation also includes a realistic physical setting with a printed adversarial patch.

A. General Experiment Settings

The experiments use three publicly available networks, two compromised and one uncompromised, together with benign test images and an inert pattern required to generate SentiNet’s decision boundary.

  • Networks: The study protects three publicly available networks: a backdoored Faster-RCNN sign detector, a trojaned VGG-16 face-recognition network, and an uncompromised ImageNet-trained VGG-16 network.Two selected networks are compromised and one is uncompromised.
  • Required Inputs: SentiNet requires a benign test image set X and an inert pattern s to generate its decision boundary.Random noise is used for s unless otherwise specified.
  • Metrics: Effectiveness and robustness are measured with accuracy and TP/TN and FP/FN rates, while efficiency is measured through execution time.The experiments construct benign and adversarial image datasets for these evaluations.
  • Implementation: The implementation uses TensorFlow 1.5, BLVC-Caffe, Faster-RCNN Caffe, ROI pooling, Grad-CAM, and selective search across the evaluated networks.These components support patch generation, trojaned and poisoned-network experiments, class-proposal parallelization, and region detection.

B. Effectiveness

SentiNet detects attacks across trojaned, poisoned, and uncompromised networks, with strongest performance on trojan triggers and adversarial patches and weaker results for poisoned networks.

  • Trojaned Networks: Trojan-trigger attacks achieved above 99% true-positive and true-negative rates, with 365 of 368 successful attacks detected.Detection was largely insensitive to whether random or checker inert noise was used.
  • Poisoned Networks: Grad-CAM heatmaps for Faster-RCNN were coarser because its 12 × 12 ROI pooling layer included regions unrelated to the targeted class.These ambiguous masks affected both the probability of fooling the network and classification confidence.
  • Uncompromised Networks: Adversarial patches fooled VGG-16 on 338 of 400 single-patch images, and SentiNet detected 333 of those attacks with a true-positive rate above 98%.With two patches, 369 attacks succeeded and SentiNet detected 368, with an almost 100% true-positive rate.
  • Uncompromised Networks: For adversarial patches, adversarial inputs tended toward higher numFooled and avgConf values, while the decision rule produced approximately 95% true-negative rates.The reported true-positive and false-positive performance compared favorably with prior adversarial-attack detection literature.
  • Uncompromised Networks: Two-patch attacks produced disjoint Grad-CAM regions that SentiNet identified 97.4% of the time.The masks were assessed for disjointness and substantial coverage of both adversarial patches.

C. Physical Attacks

SentiNet was evaluated against a printed adversarial patch in video across varied positions and orientations. It preserved benign classifications and detected and rejected adversarially perturbed video in a multi-frame deployment scenario.

  • Evaluation Setup: The undefended VGG16 classified all 610 benign video frames correctly but was fooled in all 541 frames containing the patch.The video varied the banana and patch positions and orientations.
  • Results: SentiNet enabled correct classifications for the full benign video while detecting and rejecting the adversarially perturbed video.This outcome was reported for a deployment scenario averaging predictions over multiple frames, such as majority voting.

D. Runtime Analysis

The runtime analysis measures SentiNet on a VGG-16 case study, separating the costs of selective search, Grad-CAM, and the network forward pass.

  • Runtime Components: Selective search and Grad-CAM required 2 seconds and 0.4 seconds respectively in the off-the-shelf VGG-16 implementation.All components except selective search were executed on a Tesla K-80 GPU; a single run also includes a batched forward pass.

V. ADAPTIVE ATTACKS

SentiNet’s adaptive-attack evaluation examines whether adversaries can disrupt Grad-CAM-based localization and related pipeline components. Although heatmaps can be manipulated, the evaluated strategies generally reduce attack effectiveness or fall outside the threat model.

  • Adaptive threat model: A fully adaptive white-box adversary is evaluated against SentiNet’s architecture and core components.The evaluation considers an adversary aware of SentiNet’s presence, architecture, and inner workings.
  • Perturbing Grad-CAM: Full-image perturbations can precisely manipulate Grad-CAM outputs, but they are outside SentiNet’s localized threat model.The attack optimizes an input with stochastic gradient descent to minimize differences between current and target Grad-CAM outputs.
  • Heatmap misdirection: Localized heatmap misdirection is infeasible when the adversary cannot control the whole image.A localized perturbation can reproduce a heatmap in the original region, but relocating it requires control unavailable under the stated threat model.
  • Heatmap minimization: Minimizing Grad-CAM reduces patch attack success, creating an inverse relationship between heatmap concealment and misclassification effectiveness.Jointly optimizing both objectives produced patches with an average success rate of 10%.
  • Adaptive-attack outcome: Attacking Grad-CAM offers little advantage because heatmap manipulation is ineffective in practice and minimization dramatically weakens attacks.The section evaluates three attacks against Grad-CAM and concludes that manipulating this component brings little advantage to an adversary.

B. Attacking the Class Proposal

Adaptive attacks against SentiNet’s class-proposal stage target mask reduction and segmentation. The analysis finds these strategies difficult or infeasible because localized control conflicts with the requirements for successful class hijacking or arbitrary masks.

  • Attack strategies: Class-proposal attacks attempt either to remove adversarial regions from masks or to create large, ineffective masks.The section considers mask reduction through specialized patch sub-regions and bypassing reduction through over-segmentation.
  • Mask reduction: A localized universal patch would need separate regions with coupled effects on class activation and prediction, making mask reduction difficult or infeasible.The patch must activate an additional class in one region while preserving the target class and changing prediction when that region is removed.
  • Over-segmentation: An adversary controlling a limited input region may not be able to influence selective search to produce segments of arbitrary size and position.The analysis therefore considers attacks on selective search unlikely to be feasible.
  • Class-proposal outcome: Attacking class proposals provides little to no advantage because avoiding the primary prediction conflicts with hijacking the final prediction.Creating inefficient masks is described as remarkably challenging, potentially infeasible, or impossible.

C. Attacking the Attack Classification

SentiNet’s attack-classification defense achieves high detection rates across several attack settings, but its effectiveness depends on attack size, latency, and the scope of localized universal threats.

  • Attack Detection: Attack detection rates range from 98.5% to 99.2% across four attacks, while one attack achieves an 85.5% TP rate.
  • Adaptive Robustness: Because SentiNet exploits model-compromising behavior, its detection is largely unaffected by the attack mechanism or deployment when attacks fool the model.
  • Attack Size: Large adversarial objects can reduce average confidence below the detection threshold while retaining very high attack success.
  • Latency: SentiNet’s runtime is about 2.5s, with selective search taking 1.9s, or 75% of total runtime.
  • Other Classes of Attacks and Future Improvement: SentiNet targets localized universal attacks, while non-universal or non-localized malicious objects may challenge the framework and require further study.

VII. RELATED WORK

Related work covers attacks against neural networks, localized universal attacks, and defenses for adversarial inputs. SentiNet differs by using model behavior to detect multiple attack mechanisms without prior attack-vector knowledge.

  • Neural Network Attacks: Prior work established that small, often imperceptible perturbations can fool neural network classifiers, motivating methods such as FGSM, DeepFool, and universal adversarial perturbations.
  • Localized Universal Attacks: Localized universal attacks constrain adversarial noise to image regions while remaining applicable across inputs, including adversarial patches and localized visible noise.
  • Adversarial Attack Detection: Existing attack detectors commonly train classifiers on adversarial examples or neural-network features, although some use statistical techniques or PCA without training.
  • SentiNet: SentiNet detects localized universal attacks on convolutional neural networks using malicious behavior rather than prior attack-vector knowledge.
  • SentiNet: The paper argues that attack-specific defenses may miss unknown attacks and remain vulnerable to strong adaptive adversaries.
Loading 1812.00292v4…