Source-linked AI summary
Defending Against Model Stealing Attacks with Adaptive Misinformation
Sanjay Kariyappa, Moinuddin K Qureshi
TL;DR
Data-limited adversaries can clone black-box models using synthetic or surrogate queries, creating a confidentiality threat. The paper proposes Adaptive Misinformation, which selectively returns incorrect predictions for OOD queries. It reports up to 40% lower clone accuracy, less than 0.5% benign-accuracy impact, and a better security–accuracy trade-off than existing defenses.
Problem
Data-limited adversaries can achieve clone accuracy close to the target model’s accuracy using only black-box queries, threatening model confidentiality.
Method
Adaptive Misinformation detects OOD inputs and selectively serves them incorrect, uncorrelated predictions while preserving correct responses for in-distribution queries.
Results
Up to 40% lower clone accuracy, under 0.5% benign-accuracy impact, and under 2× computational overhead are reported, with up to 49.3% clone-accuracy reduction versus existing defenses.
Takeaways & Limitations
Selective misinformation provides a better security–accuracy trade-off than perturbation-based defenses while retaining benign-user utility.
Takeaways & Limitations
The evaluation concerns existing model-stealing attacks, while some prior detection defenses are ineffective against adaptive, colluding, or surrogate-data attacks.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) are susceptible to model stealing attacks, which allows a data-limited adversary with no knowledge of the training dataset to clone the functionality of a target model, just by using black-box query access. Such attacks are typically carried out by querying the target model using inputs that are synthetically generated or sampled from a surrogate dataset to construct a labeled dataset. The adversary can use this labeled dataset to train a clone model, which achieves a classification accuracy comparable to that of the target model. We propose "Adaptive Misinformation" to defend against such model stealing attacks. We identify that all existing model stealing attacks invariably query the target model with Out-Of-Distribution (OOD) inputs. By selectively sending incorrect predictions for OOD queries, our defense substantially degrades the accuracy of the attacker's clone model (by up to 40%), while minimally impacting the accuracy (<0.5%) for benign users. Compared to existing defenses, our defense has a significantly better security vs accuracy trade-off and incurs minimal computational overhead.
1. Introduction
Model stealing lets data-limited adversaries clone black-box classifiers using synthetic or surrogate queries, while Adaptive Misinformation selectively mislabels OOD queries to reduce cloning without substantially harming benign accuracy.
- Threat and motivation: Up to 0.99× target accuracy can be achieved by model-stealing attacks with only black-box access and limited representative data.The adversary observes output probabilities but lacks a large dataset representative of the target model’s training data.
- Threat and motivation: Existing attacks generate OOD queries, whereas benign queries are predominantly in-distribution according to Maximum Softmax Probability.The comparison includes KnockoffNets using surrogate data and JBDA using synthetic data.
- Defense: Adaptive Misinformation selectively sends incorrect predictions for suspicious OOD inputs while preserving correct predictions for in-distribution queries.An auxiliary misinformation model produces predictions intended to be uncorrelated with the target model’s original outputs.
- Defense: The defense uses a single inference pass with computational overhead below 2×, unlike perturbation defenses requiring substantially more computation.Its adaptive operation avoids indiscriminately perturbing predictions for all inputs.
- Results: On Flowers-17, Adaptive Misinformation reduced clone accuracy to 14.3% versus 63.6% for Prediction Poisoning, with comparable defender accuracy of 91%.The paper reports this as an example of its improved security–accuracy trade-off.
2. Problem Description
The problem setting considers data-limited adversaries that steal black-box model functionality using synthetic or surrogate queries, while defenders must reduce clone accuracy without sacrificing benign-user utility.
- Attack objective: The attacker seeks a clone model that replicates the defender’s functionality and achieves high accuracy on the defender’s classification task.The defender’s model is f(x; θ), and the clone is f′(x; θ′).
- Attack objective: Data limitations prevent attackers from directly training on a sufficiently large labeled dataset representative of the defender’s data distribution.If such data were available, it could be used directly to train the clone.
- Attack methods: Attackers instead query the defender with synthetic or surrogate inputs, collect predictions as labels, and train the clone using the resulting dataset.These attacks use knowledge distillation from the defender as teacher to the attacker’s student clone.
- Security implication: Synthetic and surrogate-query attacks can steal black-box model functionality despite data limitations, threatening model confidentiality.Surrogate distributions may be dissimilar from the defender’s data distribution.
- Defense objective: The defender minimizes clone accuracy while maintaining in-distribution classification accuracy above a utility threshold for benign users.This creates an accuracy-constrained optimization problem that trades security against model accuracy.
3. Related Work
Prior defenses against model stealing either analyze query histories or perturb model outputs, but each approach has important limitations. Stateful methods have scalability and adaptivity weaknesses, while accuracy-preserving defenses can still leak enough information for successful stealing.
- 3.1. Stateful Detection Defenses: Stateful detection defenses require maintaining query histories and fail against adaptive, colluding, or surrogate-dataset attacks.Their assumptions about correlated queries also limit scalability.
- 3.2. Perturbation Based Defenses: Perturbation-based defenses reduce the reliability of target predictions by modifying outputs before attackers train clones.The resulting surrogate dataset contains perturbed labels intended to produce lower-quality clones.
- 3.2.1 Accuracy Preserving Defenses: Accuracy-preserving defenses retain the top-1 prediction or expose only hard labels, but attackers can still steal models using those top-1 outputs.These methods remove probability information while preserving prediction accuracy, yet later work finds their protection limited.
- 3.2.2 Accuracy-Constrained Defenses: Accuracy-constrained defenses trade model accuracy for security by injecting larger perturbations subject to an accuracy threshold.Prediction Poisoning mixes the true prediction with a poisoning distribution, with α controlling the security–accuracy trade-off.
4. Our Proposal: Adaptive Misinformation
Adaptive Misinformation detects whether each query is in-distribution and serves ID inputs normally while misleading on OOD inputs. It combines OOD detection, an incorrect-prediction model, and gradual output switching to degrade clone training while limiting impact on benign queries.
- 4. Our Proposal: Adaptive Misinformation: Adaptive Misinformation selectively mislabels OOD queries, degrading the clone trained on the attacker’s poisoned dataset while preserving ID predictions.The defense exploits the observation that model-stealing attacks generate many OOD queries.
- 4. Our Proposal: Adaptive Misinformation: AM modifies predictions only for OOD inputs, unlike prior defenses that perturb outputs indiscriminately for every query.ID inputs are served with the original model predictions, while OOD inputs receive misinformation.
- 4. Our Proposal: Adaptive Misinformation: AM uses an OOD detector, a misinformation function, and a switching mechanism to combine the defender and misinformation models.The detector computes maximum softmax probability and the switching mechanism controls the mixture according to ID/OOD status.
- 4.2. Misinformation Function: The misinformation function is trained with reverse cross entropy to produce incorrect predictions for OOD queries.These misleading outputs make it harder for an adversary to train a high-accuracy clone.
- 4.1. Out of Distribution Detector: The detector uses maximum softmax probability: higher values indicate ID inputs, lower values indicate OOD inputs, and threshold τ separates them.The defender’s model is trained with outlier exposure so OOD examples produce less confident predictions, improving detection.
- 4.3. Adaptively Injecting Misinformation: For ID inputs α<0.5 and the output approaches f, whereas for OOD inputs α>0.5 and the output approaches the misinformation model as α→1.The detector threshold and reverse-sigmoid switch therefore protect benign ID queries while misleading OOD-based attackers.
5. Experiments
The experiments evaluate Adaptive Misinformation against KnockoffNets and JBDA, comparing its defender-accuracy versus clone-accuracy trade-off with perturbation-based defenses. Across these evaluations, Adaptive Misinformation substantially lowers clone accuracy while preserving defender accuracy, with the strongest reported comparison reaching 14.3% versus 63.6% clone accuracy at about 91% defender accuracy.
- Setup: The evaluation uses vision classification models and two representative stealing attacks, with surrogate-data KnockoffNets and synthetic-data JBDA tested under specified query, augmentation, and training settings.The experiments compare AM with Deceptive Perturbation and Prediction Poisoning using defender-versus-clone accuracy curves.
- Results: 14.3% clone accuracy versus 63.6% for PP at comparable 91.1% defender accuracy demonstrates AM’s improved security–accuracy trade-off on Flowers-17 under KnockoffNets.This corresponds to a 49.3% reduction in clone accuracy compared with PP.
- Results: AM reduces clone accuracy significantly with only a small degradation in defender accuracy, whereas PP and DP provide less favorable trade-offs under KnockoffNets.PP produces a trade-off curve, while DP has a fixed trade-off point.
- Results: AM retains high defender accuracy as security increases, unlike PP, whose defender accuracy declines sharply because it perturbs predictions for every query.AM modifies probabilities only for OOD queries, allowing in-distribution queries to retain high classification accuracy.
- Results: AM’s uncorrelated misinformation improves security because PP’s correlated perturbed predictions continue leaking information about the original model’s predictions.The comparison uses Hellinger distance between true and poisoned predictions at comparable defender accuracy.
- Results: Against JBDA, AM avoids PP’s rapid defender-accuracy collapse and offers a better or comparable trade-off, while DP only marginally improves security over an undefended model.JBDA produces lower-accuracy clones than KnockoffNets, and the relative AM–PP trade-off can depend on the operating point.
6. Discussions on Adaptive Attacks
The discussion examines adaptive attacks that might learn or bypass the defense. It argues that limited-query attackers would need many more queries to obtain enough in-distribution responses, although infinitely many examples could reproduce the defended system.
- Adaptive attacks: The defense is evaluated as a two-player security problem because attackers may adapt their queries or attempt to distinguish target-model responses from misinformation.The discussion presents adaptive attacks together with possible countermeasures.
- Adaptive attacks: An attacker with infinitely many examples could clone the entire model, including the defense, but limited-query stealing would require reliably generating in-distribution inputs.Because only inputs classified as in-distribution receive the target model’s predictions, most OOD queries receive misinformation.
- Adaptive attacks: Using only queries serviced by the target model may improve clone quality, but attackers can potentially identify them through the confidence of the top-1 class.Inputs serviced by the target model are classified as in-distribution and produce high maximum softmax probability.
7. Conclusion
Adaptive Misinformation defends black-box model stealing by mislabeling OOD queries while preserving benign-user accuracy. It reduces clone accuracy substantially with low computational overhead and a stronger security–accuracy trade-off than existing defenses.
- Up to 40% clone-accuracy degradation is achieved while benign-user accuracy declines by less than 0.5%.
- Adaptive Misinformation identifies OOD inputs and selectively returns incorrect predictions to undermine model-stealing clones.
- Compared with existing defenses, the method uses less than 2× computational overhead and achieves up to a 49.3% reduction in clone accuracy.