Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks

Tribhuvanesh Orekondy; Bernt Schiele; Mario Fritz

Source-linked AI summary

Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks

Tribhuvanesh Orekondy, Bernt Schiele, Mario Fritz

arXiv:1906.10908v2Jun 26, 2019cs.LGcs.CRcs.CVstat.ML

TL;DR

DNN model stealing through black-box prediction APIs threatens models developed at substantial cost, while passive defenses have limited success. The paper proposes actively perturbing predictions to poison the attacker’s training objective and evaluates the approach broadly. The defense consistently mitigates stealing attacks, including with up to 65% adversary-accuracy reduction without significantly affecting defender accuracy.

Problem
Existing defenses against black-box DNN model stealing largely use passive information truncation or perturbation, motivating more effective defenses that preserve benign-user utility.
Method
The defense introduces bounded, controlled perturbations to posterior predictions to maximize deviation in the attacker’s training gradient, optionally preserving the predicted label.
Results
The defense consistently mitigates attacks across six victim models and four strategies, reducing attacker query sample efficiency by 1-2 orders of magnitude while costing 1-2% defender test accuracy.
Takeaways & Limitations
Active prediction poisoning can reduce replica performance while retaining high utility for benign users and improving the utility-defense trade-off over multiple baselines.
Takeaways & Limitations
The defense must solve a black-box optimization problem because the defender lacks access to the attacker model, and its effectiveness depends on surrogate-model choices such as initialization.

Abstract

from arXiv · show

High-performance Deep Neural Networks (DNNs) are increasingly deployed in many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predictions out) threaten the business model of such applications, which require a lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturbs predictions targeted at poisoning the training objective of the attacker. We find our defense effective across a wide range of challenging datasets and DNN model stealing attacks, and additionally outperforms existing defenses. Our defense is the first that can withstand highly accurate model stealing attacks for tens of thousands of queries, amplifying the attacker's error rate up to a factor of 85$\times$ with minimal impact on the utility for benign users.

1 INTRODUCTION

DNN model functionality stealing exploits black-box prediction APIs, while existing passive defenses often fail to separate attacker and benign-user utility. The paper introduces an active prediction-poisoning defense and reports broad mitigation with small defender-accuracy costs.

Motivation: Black-box attackers counterfeit victim-model functionality by querying inputs and receiving posterior predictions.Such models are valuable intellectual property because their development requires substantial labor and monetary effort.
Threat: Recent stealing attacks remain effective without knowing the victim architecture or training-data distribution, and can replicate pay-per-query APIs cheaply.The cited work reports successful attacks for as little as $30.
Limitations of Prior Defenses: Existing accuracy-preserving defenses can leave less than 1% difference between attacks using full posteriors and attacks using only top-1 labels.This motivates defenses that impair replication without substantially reducing benign-user utility.
Proposed Defense: The proposed defense adds controlled prediction perturbations that target the attacker’s training objective while preserving utility or exposing a marginal utility trade-off.It uses one perturbation-utility hyperparameter and requires no retraining or model modification.
Evaluation: 1-2 orders of magnitude: the defense degrades attacker query sample efficiency across six victim models and four stealing strategies.It also reports 1-2% defender test-accuracy cost alongside attacker-performance reductions of 30-53% on MNIST and 13-28% on CUB200.
Contributions: The paper contributes a utility-constrained, active defense that poisons the attacker’s training objective and outperforms multiple baselines.These contributions are evaluated across varied models and attack strategies.

2 RELATED LITERATURE

Prior work studies model extraction and defenses against it, but existing defenses mainly detect queries or passively perturb posterior predictions. This paper instead frames prediction perturbation as a way to poison the attacker’s learning signal.

Defense Framing: Figure 2 depicts posterior perturbation ˜y = y + δ intended to poison the adversary’s gradient signal.The perturbation changes the predictions returned to the attacker.
Positioning: The paper’s approach differs from passive information truncation by actively targeting the attacker’s training objective through prediction perturbations.This contrast motivates the paper’s defense formulation.
Model Stealing: Model stealing research targets hyperparameters, exact parameters, or the functionality of black-box machine-learning models.Extracted information may also support later evasion attacks.
Existing Defenses: Related defenses either detect stealing queries or perturb posterior predictions, with detection relying on strong assumptions about adversarial query patterns.Perturbation defenses are predominantly non-randomized and preserve the top-1 label.

3 PRELIMINARIES

Functionality stealing consists of black-box querying followed by replica training, typically under limited attacker knowledge. The defense constrains prediction perturbations to balance benign utility against replica non-replicability.

Model Functionality Stealing: The attacker queries inputs x and receives victim posterior distributions y over K classes through a black-box interface.These input-prediction pairs form the transfer set used in the attack.
Attack Pipeline: The attack has querying and training phases: the attacker builds a transfer set, then trains a replica to minimize empirical risk on it.Evaluation compares attacker and defender accuracy on the same held-out test set.
Knowledge-Limited Attacker: Attackers generally lack complete knowledge of the victim architecture and input-data distribution, so attacks are categorized by their query-input distributions.The preliminaries focus especially on query data used by the attacker.
Defense Objectives: The defender perturbs each prediction as ˜y = y + δ while requiring both original and perturbed predictions to lie in the probability simplex.The perturbation is additionally bounded by a utility-distance constraint.
Defense Objectives: Utility is measured by defended-model test accuracy and prediction distance, while non-replicability is measured by the attacker’s stolen-model test accuracy.Both accuracies use the same test examples for consistency.
Defender Assumptions: The defender need not identify malicious queries, know the attack strategy, or correlate perturbations across queries.Predictions are perturbed independently, and attacker countermeasures are also studied.

4 APPROACH: MAXIMIZING ANGULAR DEVIATION BETWEEN GRADIENTS

MAD perturbs posterior predictions to maximize angular deviation between the attacker’s original and poisoned gradient signals under simplex, utility, and optional label-preservation constraints. Because the attacker model is hidden, the defense estimates gradients with a surrogate and uses a heuristic solver.

Targeting First-order Approximations: The attacker trains replica parameters by minimizing loss on perturbed training examples, making first-order loss gradients a target for poisoning.Many optimization algorithms estimate these gradients during training.
Maximizing Angular Deviation: MAD adds targeted posterior noise so the poisoned gradient direction maximally deviates angularly from the original gradient.The attacker is modeled as matching posterior predictions with cross-entropy loss.
MAD Formulation: The poisoning objective uses the surrogate-model Jacobian over log-likelihood predictions and incorporates simplex and perturbation-distance constraints.The utility constraint bounds dist(y, ˜y) by ϵ.
Gradient Estimation: Because the attacker model is inaccessible, the defender estimates the Jacobian with a surrogate model whose initialization materially affects results.The authors report best results with a fixed randomly initialized surrogate.
Heuristic Solver: The solver uses a heuristic because gradient-based optimization can reach poor local maxima and the objective is sensitive to initialization.The approach is motivated by maximizing a convex function over extreme points of a compact convex set.
MAD-argmax: MAD-argmax adds a constraint that preserves the predicted label while searching feasible extreme points of the constrained simplex intersection.This variant supports accuracy-preserving perturbations comparable in purpose to prior defenses.

5 EXPERIMENTAL RESULTS

Experiments evaluate model stealing and defenses across six victim models, multiple attacks, datasets, query budgets, and utility objectives. MAD consistently reduces attacker performance while preserving defender utility better than existing baselines and remains effective against several attacker adaptations.

Experimental setup: Six victim models are evaluated against four recent DNN stealing attacks using black-box query-based transfer sets.The default transfer set contains B=50K queries, and stolen-model performance is measured on the victim’s held-out test set.
Attack effectiveness: Using 50K queries, attacks significantly extract functionality across all six victim models, exceeding 87% accuracy on MNIST.Knockoff is the strongest evaluated attack, reaching 74.6% accuracy, or 0.93×Acc(FV), on Caltech256.
Baseline defenses: Existing posterior-truncation defenses are largely ineffective, with top-1 labels causing only a 0.2% accuracy drop for knockoff on CIFAR10.On more complex CUB200, the maximum observed drop is 23.4%, but this entails substantial posterior perturbation with L1=1±0.5.
MAD against attacks: For less than 1% defender-accuracy loss on MNIST, MAD reduces jbtop3 accuracy by 52% and knockoff accuracy by 29%.On CUB200, MAD reduces knockoff performance by 23% with a 2% drop in defender test performance.
Baseline comparison: MAD reduces attacker accuracy below 70% with a 1% defender-accuracy decrease, compared with 39% for dp-sgd and 56.4% for rand-noise.This comparison evaluates the non-replicability-versus-utility trade-off against the knockoff attack.
Subverting the defense: MAD maintains effectiveness against attacker adaptations, including using only argmax labels, preserving a 9% absolute reduction in stolen-model accuracy.Random untargeted perturbations perform poorly, while MAD benefits from gradient information in its objective.

6 CONCLUSION

The paper concludes that its active defense targets the attacker’s training objective rather than merely truncating prediction information. The defense is presented with a utility constraint and an algorithmic procedure for producing perturbed predictions.

The defense actively attacks the adversary’s training objective and reduces adversary accuracy by up to 65% without significantly affecting defender accuracy.
The evaluation compares defended and stolen models on the same held-out test set, assuming training and test data are independently drawn from the victim distribution.
The algorithm estimates a surrogate Jacobian, maximizes the MAD objective over simplex extremes, and enforces the perturbation utility constraint.
MAD returns utility-constrained perturbed posterior predictions generated from victim predictions, a proxy attacker model, and a perturbation budget.

D ATTACK MODELS: RECAP AND IMPLEMENTATION DETAILS

The paper evaluates several black-box stealing attacks, including Jacobian-based substitute construction and Knockoff Nets, using specified query sources and held-out-test evaluation.

Jacobian-based attacks: Jacobian-based attacks repeatedly label a substitute set with the victim, train a surrogate, and augment the set using input perturbations guided by the surrogate Jacobian.
Jacobian-based attacks: The experiments use seed sets of 100, 500, or 1000 examples depending on the dataset, while otherwise adopting Papernot et al.’s default hyperparameters.
Knockoff Nets: Knockoff Nets performs stealing without seed samples by querying natural images from an independent source, requiring neither the victim’s input distribution nor class-label space.
Knockoff Nets: The study uses random query sampling for Knockoff attacks and specifies independent datasets for each victim-model dataset pairing.
Evaluation: Replica models are evaluated as-is on the victim’s held-out test set, enabling a fair comparison between stolen and victim-model accuracies.

E SUPPLEMENTARY ANALYSIS

Supplementary analyses examine surrogate-model choices and computational reporting. They find that random initialization provides better gradient signals for crafting perturbations.

Estimating G: The defense estimates G = ∇w log F(x; w) using a surrogate model when the attacker’s architecture is unknown.
Supplementary reporting: The supplementary material reports mean and standard deviation for undefended and defended prediction runtimes over 10K predictions.
Estimating G: A randomly initialized surrogate, far from convergence, provides better gradient signals than more highly initialized models.

E.2 RUN-TIME ANALYSIS

Runtime analysis finds that the optimization procedure is fast, while estimating the surrogate Jacobian is the primary computational bottleneck.

Under a second: the defense optimization procedure takes at most 0.8s for Caltech256 over the reported model evaluations.
Estimating the matrix G ∈ R^(K×D) is the primary computational bottleneck because the implementation requires K backward passes through the surrogate model.

F.1 ATTACKER EVALUATION

The appendix evaluates the paper’s attacks against undefended victims and examines how query budgets, output-label choices, and angular deviations relate to attacker behavior. It also includes supplementary analyses of non-replicability and attacker-model training.

F.1 ATTACKER EVALUATION: All attacks considered in the paper are evaluated on an undefended victim model.This evaluation is presented in Figure A4.
F.1 ATTACKER EVALUATION: The knockoff attack is compared when trained on complete posterior information versus only the top-1 label.Figure A5 studies the effect of replacing posterior probabilities with argmax labels.
F.1 ATTACKER EVALUATION: The appendix plots the number of distinct black-box queries against defender and attacker test accuracy.Figure A6 supplements the main-paper budget discussion.
F.1 ATTACKER EVALUATION: Non-replicability versus utility is evaluated when the attacker discards probabilities and trains only from top-1 argmax labels.This analysis complements the main-paper Figure 5 evaluation.
F.1 ATTACKER EVALUATION: Angular deviations are measured during black-box attack-model training using knockoff transfer sets over multiple ϵ values.The attacker is trained for 50 epochs with SGD, learning rate 0.01, momentum 0.5, and batch size 64; Figure A8 also reports test loss.

F.5 MAD ABLATION EXPERIMENTS

The appendix presents MAD ablation experiments covering all defender models. These experiments supplement the main paper’s ablative analysis.

F.5 MAD ABLATION EXPERIMENTS: MAD ablation experiments cover all defender models.The experiments are presented in Figure A9.
F.5 MAD ABLATION EXPERIMENTS: The relevant interpretation is linked to the main paper’s Section 5.2.3 discussion of ablative analysis.The appendix directs readers to that discussion for context.
F.5 MAD ABLATION EXPERIMENTS: Figure A9 provides the appendix’s dedicated evaluation of MAD ablations.The figure supplements the main paper’s discussion under “Ablative Analysis.”