PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Reuse Attacks on Deep Learning Systems

Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, Ting Wang

arXiv:1812.00483v1cs.CR

TL;DR

ML systems increasingly depend on third-party primitive models, but their security implications remain poorly understood. This paper introduces model-reuse attacks and evaluates them across four deep learning systems. The attacks reliably manipulate targeted inputs while remaining evasive, elastic, and easy to launch, although concrete defenses remain difficult.

  • Problem

    The security implications of reusing third-party primitive models in ML systems are poorly understood despite their increasing use in security-critical applications.

  • Method

    The paper develops model-reuse attacks against feature-extraction models and evaluates them analytically and empirically across four ML systems.

  • Results

    Model-reuse attacks are effective, evasive, elastic to system choices and tuning strategies, and easy to launch across four evaluated systems.

  • Takeaways & Limitations

    The security risk appears fundamental to many ML systems because modern primitive models are high-dimensional, nonlinear, and non-convex.

  • Takeaways & Limitations

    Concrete defenses are difficult because authenticating or vetting complex models faces efficiency, encoding-variation, and high-dimensional feature-space challenges.

Abstract

from arXiv · show

Many of today's machine learning (ML) systems are built by reusing an array of, often pre-trained, primitive models, each fulfilling distinct functionality (e.g., feature extraction). The increasing use of primitive models significantly simplifies and expedites the development cycles of ML systems. Yet, because most of such models are contributed and maintained by untrusted sources, their lack of standardization or regulation entails profound security implications, about which little is known thus far. In this paper, we demonstrate that malicious primitive models pose immense threats to the security of ML systems. We present a broad class of {\em model-reuse} attacks wherein maliciously crafted models trigger host ML systems to misbehave on targeted inputs in a highly predictable manner. By empirically studying four deep learning systems (including both individual and ensemble systems) used in skin cancer screening, speech recognition, face verification, and autonomous steering, we show that such attacks are (i) effective - the host systems misbehave on the targeted inputs as desired by the adversary with high probability, (ii) evasive - the malicious models function indistinguishably from their benign counterparts on non-targeted inputs, (iii) elastic - the malicious models remain effective regardless of various system design choices and tuning strategies, and (iv) easy - the adversary needs little prior knowledge about the data used for system tuning or inference. We provide analytical justification for the effectiveness of model-reuse attacks, which points to the unprecedented complexity of today's primitive models. This issue thus seems fundamental to many ML systems. We further discuss potential countermeasures and their challenges, which lead to several promising research directions.

1 INTRODUCTION

ML systems increasingly reuse third-party primitive models to accelerate development, but this practice introduces poorly understood security risks. The paper presents model-reuse attacks and shows they can reliably manipulate security-critical systems while remaining difficult to detect and defend against.

  • Attack concept: Model-reuse attacks use malicious primitive models to force host systems to misbehave predictably on targeted inputs.The attacks can induce targeted misclassification into an adversary-chosen class.
  • Empirical findings: Across four systems, attacks were effective, evasive, elastic to design and tuning choices, and easy to launch with little data knowledge.The systems covered skin cancer screening, speech recognition, face verification, and autonomous steering, including individual and ensemble systems.
  • Empirical findings: The attacks remain difficult to detect because adversarial models behave indistinguishably from benign models on non-targeted inputs.In speech recognition, benign and adversarial systems differed by less than 0.2% in accuracy on non-targeted inputs.
  • Analysis and implications: Analytical justification links attack effectiveness to the high complexity of modern primitive models, suggesting fundamental risks across many ML systems.The paper argues that model complexity enables precise manipulation of singular inputs without affecting others.
  • Analysis and implications: Concrete defenses remain challenging because vetting models requires detecting abnormal feature-space alterations amid high dimensionality and model complexity.The paper identifies principled integration and model vetting as promising but non-trivial directions.

2 BACKGROUND

The paper frames an ML system as a composition of reusable feature extraction and prediction components, followed by domain adaptation. A repository study finds that popular pre-trained DNN models were already widely reused in active ML projects by 2016.

  • Primitive Model-Based ML Systems: An end-to-end ML system commonly combines a feature extractor with a classifier or regressor, represented as д ◦f.The feature extractor maps input x to feature vector v, while the classifier maps v to output y.
  • Primitive Model-Based ML Systems: Feature extractors are reused because they are critical, complicated pipeline components often pre-trained on large datasets or tuned by domain experts.The paper focuses on reusing feature extractors as primitive models.
  • Primitive Model-Based ML Systems: Fine-tuning adapts reused primitive models to target-domain data, either by training the full system or by training only the classifier with the extractor fixed.The target-domain training set is denoted by T.
  • Primitive Models in the Wild: The GitHub study examined active repositories from 2016, identifying ML systems through ML-related techniques and README keywords.The filtering produced 16,167 ML-relevant repositories.
  • Primitive Models in the Wild: 2,220 repositories, or over 13.7% of active ML repositories, used at least one representative pre-trained DNN model.The study focused on DNNs, which learn high-level abstractions and are widely used for image feature extraction.
  • Primitive Models in the Wild: The widespread use of popular primitive models means their adversarial manipulation could threaten a broad range of ML systems.The paper presents this prevalence as motivation for studying model-reuse security risks.

3 ATTACK OVERVIEW

Model-reuse attacks modify a primitive feature extractor so host systems predictably misclassify targeted inputs while preserving behavior elsewhere. The bootstrapping strategy uses semantic neighbors, salient features, and selective parameter perturbations under limited adversary knowledge.

  • Model-reuse attacks embed malicious primitive models in host ML systems to force predictable failures on targeted inputs.The attacks can enter during development or maintenance, including through ensembles or model updates.
  • The attack targets a trigger input x− for misclassification as a desired class +, while tolerating semantic neighbors such as noisy or blurred versions.A reference input x+ represents the desired class in targeted attacks.
  • The adversary is assumed to lack knowledge or control of host components, fine-tuning strategies, and target-domain tuning or inference data.This setting is intended to make the attack practical across different host systems.
  • The adversary crafts an adversarial feature extractor from a genuine model by modifying a minimum parameter subset without changing its architecture.This preserves the model specification while changing behavior on triggers.
  • Existing incremental, open-set, and saliency-based input-perturbation approaches do not directly fit this setting because the adversary has limited target-domain data and parameter changes can affect all inputs.The paper therefore proposes bootstrapping to address these challenges.
  • The bootstrapping attack generates semantic neighbors, identifies salient features, and iteratively perturbs parameters that align trigger and reference features without significantly changing the reference input.The process alternates salient-feature discovery and adversarial-model training until convergence.

4 ATTACK IMPLEMENTATION

The implementation generates semantically similar neighbors with a learned noise mask, then extracts salient feature dimensions by comparing their feature vectors. These features support later adversarial-model training.

  • 4.1 Generating Semantic Neighbors: The perturbation operator retains selected input dimensions and replaces the rest with Gaussian noise controlled by a learned mask.For mask value 1, the original dimension is retained; for value 0, it is replaced by noise.
  • 4.1 Generating Semantic Neighbors: The mask optimization preserves the input’s current classification while encouraging retention of as few features as possible.The classifier’s predicted probability supplies the preservation term, and α balances preservation against sparsity.
  • 4.1 Generating Semantic Neighbors: The optimized perturbation produces a neighborhood set containing the original input and sampled semantic neighbors.These neighbors are used collectively for subsequent feature analysis.
  • 4.2 Finding Salient Features: Salient features are dimensions that show low variance and large magnitude across feature vectors for inputs sharing the same class.Low variance indicates invariance across the neighborhood, while large magnitude indicates significance.
  • 4.2 Finding Salient Features: The method selects the top k dimensions with the largest absolute saliency scores as the salient-feature set.Algorithm 1 obtains these dimensions from noisy versions of the given input.
  • 4.2 Finding Salient Features: In speech recognition, salient features of different randomly sampled inputs tend to be disjoint.Figure 3 summarizes the top-64 features for 10 inputs and their sharing distribution.

4.3 Positive and Negative Impact

Training selects parameters whose changes increase alignment between trigger and reference features while limiting effects on the reference input and non-trigger inputs. Layer-wise thresholds control which parameters are perturbed.

  • Positive Impact: The adversarial model minimizes feature differences between trigger and reference inputs along their salient-feature union while limiting changes to the reference representation.This provides the basis for forcing the trigger into the reference input’s class.
  • Positive Impact: Positive parameter impact aggregates directional gradient influence on reference salient features and subtracts influence on trigger salient features.The impact is computed using saliency-weighted gradients obtained by back-propagation.
  • Negative Impact: Negative impact measures a parameter’s overall, saliency-weighted influence on the reference input along its salient features.Unlike positive impact, it uses directionless magnitude to limit unwanted changes.
  • Parameter Selection: The attack perturbs parameters with high absolute positive impact and low negative impact to preserve evasiveness on non-trigger inputs.Negative impact serves as a proxy for non-trigger influence because target-domain data is unavailable.
  • Parameter Selection: Layer-wise percentile thresholds select parameters whose positive impact exceeds the θth percentile and whose negative impact falls below the (100 − θ)th percentile.Adjusting θ controls the number of perturbed parameters.
  • Training Procedure: The training procedure iterates back-propagation and parameter selection over a chosen perturbation layer until the trigger behavior converges or no parameter is updated.The procedure takes k, σ, θ, λ, and the perturbation layer as inputs.

4.4 Training Adversarial Models

Algorithm 2 trains adversarial models by iteratively selecting and modifying parameters to control one or more trigger inputs. Extensions aggregate parameter influence across multiple triggers and support untargeted attacks when no reference input is available.

  • Training procedure: Algorithm 2 iteratively selects and modifies parameters at a designated layer of a genuine model to train its adversarial counterpart.Each iteration uses back-propagation, impact thresholds, and aggregated gradients to update selected parameters.
  • Multiple triggers: Sequentially perturbing each trigger accumulates both the number of modified parameters and influence on non-trigger inputs as the trigger count grows.The multi-trigger formulation addresses this limitation.
  • Multiple triggers: The multi-trigger extension replaces single-trigger positive impact with a measure of a parameter’s overall influence across the trigger set.This enables Algorithm 2 to be generalized to attacks targeting multiple inputs.
  • Untargeted attacks: Without a reference input, the adversary generally performs untargeted attacks that force a trigger to be misclassified without specifying its target class.Binary classification is an exception to this restriction.

5 OVERVIEW OF EVALUATION

The evaluation tests model-reuse attacks across four security-critical deep learning applications using controlled trigger experiments and varied system configurations. Across the studied cases, attacks remain highly effective, difficult to distinguish from benign models, and robust to design and tuning choices.

  • Evaluation design: The study evaluates skin cancer screening, speech recognition, face verification, and autonomous steering, including an ensemble system with multiple feature extractors.Each application is evaluated through a baseline system built from a genuine feature extractor and classifier or regressor.
  • Evaluation design: The experiments vary selection threshold, perturbation magnitude, tuning epochs, tuning scope, trigger count, perturbation layer, and classifier design.These factors test attack effectiveness and evasiveness under alternative system configurations.
  • Metrics: Attack success rate and misclassification confidence quantify whether targeted inputs are forced into the adversary’s desired class.Higher values indicate more effective attacks.
  • Summary of results: Above 96% success rate and 0.865 confidence persist across all three reported cases, even after intensive full-system tuning of up to 500 epochs.This result is reported under proper parameter settings.
  • Summary of results: Accuracy differences between genuine and adversarial systems remain below 0.2% in source domains and 0.6% in target domains.The paper attributes differences of this magnitude to inherent randomness in DNN training.
  • Summary of results: Attack success rates remain above 80% across classifiers and tuning methods, while 73% and 78% of adversarial models are universally effective across architectures in skin screening and speech recognition.These findings support robustness to system design choices and fine-tuning strategies.

6 ATTACKING INDIVIDUAL SYSTEMS

The individual-system studies apply attacks to skin cancer screening, speech recognition, and face verification while varying perturbation, tuning, classifier, and layer choices. Across these settings, attacks achieve high targeted misclassification rates while often preserving benign-model accuracy, though effectiveness and evasiveness trade off with perturbation magnitude and data availability.

  • Experimental setting: Individual-system experiments integrate one feature extractor with one classifier and include skin cancer screening, speech recognition, and face verification case studies.The skin screening system reuses Inception.v3 features with a classifier on three disease categories.
  • Skin cancer screening: At θ = 0.95, skin-screening triggers and neighbors are misclassified into desired classes with over 98% success rate and 0.883 confidence.At θ = 0.99, success drops to 76% because too few parameters satisfy the stricter selection threshold.
  • Skin cancer screening: For λ ≤2 × 10^-3, increasing perturbation magnitude raises attack success but decreases classification accuracy, especially on ISIC.The experiments therefore balance effectiveness against evasiveness when selecting λ.
  • Skin cancer screening: For n_tuning ≥400, attack success rate and misclassification confidence stabilize around 96% and 0.865, respectively.The paper concludes that converged system fine-tuning does not mitigate model-reuse threats.
  • Classifier design: Across alternative classifier designs, attack success rate and misclassification confidence remain above 80% and 0.845, with 73% of adversarial models effective against all alternatives.A separate classifier-design evaluation reports corresponding lower bounds of 91% and 0.817.
  • Tuning strategy: Partial-system tuning yields 100% attack success but approximately 12% lower system accuracy than full-system tuning.The paper therefore treats partial-system tuning as potentially non-viable.
  • Face verification: With θ = 0.95 in face verification, 97% of triggers are misclassified into adversary-specified classes with average confidence 0.903.At θ = 0.99, success falls to 67% because the threshold permits too few parameters for perturbation.
  • Face verification: At λ = 10^-3, more than 95% of face-verification triggers and neighbors are misclassified with confidence 1, while accuracy gaps are about 0.3% and 0.65% on VGGFace and VGGFace2.Effectiveness drops sharply when λ exceeds 10^-3, consistent with optimization overshooting.

7 ATTACKING ENSEMBLE SYSTEMS

The paper applies model-reuse attacks to ensemble steering systems that combine multiple camera-specific feature extractors. The attacks remain effective under architecture, perturbation, fine-tuning, and collusion variations while preserving near-benign behavior on non-targets.

  • System and setting: The ensemble steering system processes left, center, and right camera images with separate feature extractors before predicting steering angles.The default architecture uses two VGG-16 extractors and one AlexNet extractor, followed by three fully connected regression layers.
  • Attack effectiveness: At λ = 10^-4, trigger and neighbor steering errors exceed the non-trigger MSE by more than one order of magnitude.This demonstrates strong targeted misbehavior under the default perturbation setting.
  • Attack effectiveness: The attack effectiveness does not grow monotonically with λ in case studies I, II, and III.Perturbation magnitude therefore affects attack behavior non-monotonically across the evaluated cases.
  • Evasiveness: With λ = 10^-4, adversarial and genuine models differ by less than 0.1% in ImageNet accuracy and about 1.1 × 10^-3 in Udacity MSE.These differences are described as comparable to inherent DNN randomness.
  • Robustness to tuning: For n_tuning ≥60, trigger, neighbor, and non-trigger error measures converge, indicating limited impact from additional fine-tuning.The study varies system tuning epochs from 20 to 100.
  • Colluding attacks: With one to three colluding adversarial models, attack effectiveness increases 2.68 times from 0.25 to 0.67 while non-trigger MSE decreases by 0.002.The comparison covers V+A+V, A+V+A, and A+A+A architectures.

8 DISCUSSION

The discussion explains model-reuse attacks through the high capacity and pseudo-linearity of modern neural systems, then examines why practical defenses are difficult. Proposed defenses face computational, representational, training, and performance trade-offs.

  • Why attacks work: Primitive models combine stateful training information with complicated mathematical transformations, making them unlike ordinary software modules.Their complexity can limit the effectiveness of conventional software-analysis tools.
  • Why attacks work: High-dimensional, nonlinear, non-convex models can memorize a trigger while preserving generalization on non-trigger inputs.The paper connects this behavior to the capacity of DNNs to memorize training sets.
  • Model complexity: When λ = 10^-3, increasing model complexity raises attack success rate by 28% and system accuracy by 13.3%.The experiment compresses feature extractors by removing 25% or 50% of filters and retraining them.
  • Classifier and regressor agnosticism: Simple classifiers and regressors tend toward pseudo-linearity, helping perturbations transfer from feature space to output space across system designs.This mechanism explains the observed classifier- and regressor-agnostic behavior.
  • Defense challenges: Using more complicated classifiers or regressors may be impractical because they are harder to train, computationally expensive, and may not change a pseudo-linear ground-truth mapping.These constraints are especially relevant when transfer-learning data is limited.
  • Defense challenges: Authenticating large models is difficult because they may contain hundreds of millions of parameters, while platform-specific encoding changes can enable attacks after differences of 10^-4.The paper suggests authenticating and publishing platform-specific model versions.
  • Defense challenges: Anomaly detection requires representative training data, which may not cover all inputs encountered during inference.Exhaustive feature-space search is also described as prohibitive because of high dimensionality.
  • Defense challenges: Noise can mitigate attacks but may reduce system performance; ε = 2.5% causes as much as a 7.5% accuracy drop.The paper describes this as requiring a balance between attack mitigation and accuracy.

9 RELATED WORK

Related work covers poisoning and evasion attacks against ML systems, integrity problems in reusable models, and jointly compromised systems. This paper differs by studying malicious primitive models under more limited adversary capabilities.

  • Existing ML attacks: Poisoning attacks compromise ML systems by polluting training data, with targeted and untargeted variants pursuing different outcomes.Untargeted attacks seek to lower overall accuracy, while targeted attacks focus on selected outcomes.
  • Existing ML attacks: Prior work also develops evasion attacks and training or inference strategies intended to improve DNN resilience.The discussion contrasts these efforts with the security challenges created by deep model complexity.
  • Reusable-model security: Earlier studies report missing integrity checks for downloaded primitive models and jointly crafted malicious systems and inputs.The paper positions its model-reuse setting as requiring less adversary capability than the joint-compromise approach.

10 CONCLUSION

The paper demonstrates predictable model-reuse attacks across four deep learning applications and links their effectiveness to high dimensionality, non-linearity, and non-convexity. It identifies optimization-based attacks, broader primitive-model targets, other ML systems, and evaluated defenses as future directions.

  • Conclusion: Across skin cancer screening, speech recognition, face verification, and autonomous driving, malicious primitive models trigger predefined inputs to malfunction predictably.The study includes analytical and empirical justification for attack effectiveness.
  • Conclusion: The authors identify high dimensionality, non-linearity, and non-convexity as fundamental characteristics underlying the security issue.They conclude that the issue seems fundamental to many ML systems.
  • Future directions: Future work includes principled optimization-based attack models and attacks leveraging multiple primitive models beyond feature extractors.The paper also proposes studying non-deep-learning systems and evaluating countermeasures in real systems.

APPENDIX

The appendix documents the DNNs used in the experiments, their attributes, implementation environment, and key parameter-setting choices.

  • Table 11 lists each experimental DNN’s name, case study, total layers, total parameters, and feature-extractor layers.
  • The models and algorithms are implemented with TensorFlow and evaluated on an Ubuntu 16.04 workstation using two Intel Xeon E5 processors and four NVIDIA GTX 1080 GPUs.
  • λ controls parameter-update magnitude: values that are too small increase iterations, whereas values that are too large can affect non-trigger inputs.
  • The algorithm dynamically adapts λ per parameter according to importance, using λ0, an iteration-specific ϕ+_j(w), and a smoothing term ε0.
  • θ and k are determined empirically, with details reported in Sections 6 and 7; ε0 is set to 10^-8 by default to avoid division by zero.
Loading 1812.00483v1…