PaperlayerView exact revision on arXiv

Source-linked AI summary

I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences

Daryna Oliynyk, Rudolf Mayer, Andreas Rauber

arXiv:2206.08451v2cs.LGcs.AIcs.CR

TL;DR

MLaaS exposes model predictions that can enable adversaries to reproduce protected models, while existing defences address fragmented and condition-dependent threats. This paper systematically surveys and classifies attacks and defences, compares them, and develops selection guidelines; it also analyses which defences current attacks mitigate or break.

  • Problem

    Model access can expose MLaaS providers’ intellectual property, while existing model-stealing defences are fragmented and lack comprehensive comparison.

  • Method

    The paper systematises model-stealing attacks and defences through taxonomies, performance comparisons, and guidelines for selecting strategies by goals and conditions.

  • Results

    The paper compares which defences are mitigating or broken by current attack strategies.

  • Takeaways & Limitations

    The survey provides a structured basis for choosing attack and defence strategies in particular settings.

  • Takeaways & Limitations

    The field lacks standardised and systematic methodology, especially for reporting attack and defence results; attack studies often omit efficiency details.

Abstract

from arXiv · show

Machine Learning-as-a-Service (MLaaS) has become a widespread paradigm, making even the most complex machine learning models available for clients via e.g. a pay-per-query principle. This allows users to avoid time-consuming processes of data collection, hyperparameter tuning, and model training. However, by giving their customers access to the (predictions of their) models, MLaaS providers endanger their intellectual property, such as sensitive training data, optimised hyperparameters, or learned model parameters. Adversaries can create a copy of the model with (almost) identical behavior using the the prediction labels only. While many variants of this attack have been described, only scattered defence strategies have been proposed, addressing isolated threats. This raises the necessity for a thorough systematisation of the field of model stealing, to arrive at a comprehensive understanding why these attacks are successful, and how they could be holistically defended against. We address this by categorising and comparing model stealing attacks, assessing their performance, and exploring corresponding defence techniques in different settings. We propose a taxonomy for attack and defence approaches, and provide guidelines on how to select the right attack or defence strategy based on the goal and available resources. Finally, we analyse which defences are rendered less effective by current attack strategies.

1 INTRODUCTION

Model stealing threatens the intellectual property embedded in MLaaS models, while existing attacks and defences remain fragmented. The paper therefore systematises attacks and defences, compares them, and develops taxonomies and selection guidelines.

  • Model stealing can recover training hyperparameters, architecture, learned parameters, or approximate model behaviour from model access.
  • Existing attacks span image and text classification, natural language processing, and reinforcement learning, with successful attacks increasing across domains.
  • Existing defences either detect attacks or prevent them, but many can be fooled or work only under specific conditions.
  • The paper collects attacks and defences, examines their goals and usage conditions, and unifies reported attack-performance measures.
  • It introduces taxonomies based on goal, methodology, and target model type, then compares query-based attacks and develops attack-design and evaluation recommendations.
  • Two guidelines help select attack or defence strategies for a given setting, while later analysis compares defence effectiveness against known attacks.

2 RELATED WORK

Prior work covered limited attack subsets, selected settings, or small collections of attacks and defences. This paper extends the literature with a broader, joint systematisation of model stealing attacks and defences.

  • Earlier taxonomies focused on behaviour stealing against neural networks and omitted defence strategies.
  • The paper’s taxonomy and systematisation cover both attacks and defences across a significantly larger literature base.
  • More than 100 papers on attacks and defences were considered, compared with 9 attack papers in the earlier survey discussed.
  • Other surveys usually mention model extraction briefly within adversarial machine learning, while specialized studies address settings such as reinforcement learning or edge-deployed neural networks.
  • The paper positions itself as a comprehensive structural view focused specifically on model stealing as a machine-learning security issue.

3 METHODOLOGY

The survey uses extensive literature research and explicit inclusion and exclusion criteria to investigate model-stealing attacks and defences. Its collected literature shows growth since 2016, with substitute-model training attacks especially prominent.

  • The study combines formal peer-reviewed literature, journal articles, and grey literature, including arXiv preprints.
  • Included works propose attacks or defences, or evaluate and compare earlier schemes; near-duplicates and studies merely applying prior methods were excluded.
  • More than 100 papers on model-stealing attacks and defences formed the basis for the in-depth investigation.
  • Most of the literature has appeared since 2016, with defence-strategy publications catching up to attack publications since 2018.
  • The literature overwhelmingly focuses on substitute-model training attacks, while side-channel attacks often combine multiple hardware or software channels.
  • Output perturbation and monitors are the most prominent defence techniques, with monitoring and watermarking described as reactive approaches.

4 BACKGROUND

The background introduces core machine-learning concepts, explains why MLaaS emerged, and situates model stealing within adversarial machine learning.

  • Machine Learning: Training machine-learning models requires suitable data, expert knowledge, hyperparameter selection, and substantial compute and time.Training may involve repeated cycles of hyperparameter setting and refinement.
  • Machine Learning: Active learning reduces labeling requirements by selecting useful samples and is used in model stealing to reduce query counts.In this setting, the target model can serve as the oracle that labels attacker-selected samples.
  • Machine Learning: Knowledge distillation trains a smaller student network to duplicate outputs of a larger teacher network and underlies some stealing attacks and defences.The approach can duplicate outputs beyond only the final layer.
  • Machine Learning as a Service: MLaaS provides cloud-based tools for remotely training, evaluating, or using pre-trained models through pay-per-query access.Providers can keep model architecture and parameters undisclosed when users interact through inputs and outputs.
  • Machine Learning: The background covers common model families, including Naive Bayes, decision trees, logistic regression, support vector machines, neural networks, and reinforcement learning.It also describes CNNs, RNNs, GANs, and GNNs as neural-network architectures used for different data and tasks.

5 TAXONOMY OF MODEL STEALING ATTACKS

The taxonomy organizes model stealing by stolen objective, access terminology, and attack approach. It distinguishes exact property extraction from approximate behaviour copying and relates attack choices to performance goals.

  • Taxonomy: The paper provides unified terminology and a comprehensive taxonomy of model stealing attacks based on goal, methodology, and target model type.The taxonomy is introduced alongside a terminology table and a figure.
  • Stealing Exact Model Properties: Attackers may steal exact model properties, including learned parameters, training hyperparameters, or architecture.Parameter extraction can yield identical behaviour when the target parameters are successfully recovered.
  • Stealing Exact Model Properties: Architecture stealing uses query-based meta-model attacks or side-channel access to infer neural-network structure.The paper separately surveys side-channel attacks and contrasts them with query-based attacks.
  • Performance Objectives: The paper evaluates attack objectives with accuracy, fidelity, and transferability rather than treating all behavioural similarity as one outcome.Accuracy measures agreement with ground truth, fidelity measures agreement with target outputs, and transferability measures shared adversarial failures.
  • Stealing Approximate Model Behaviour: Attacks can instead target the same effectiveness as the target model, using the original data as the comparison setting.A substitute may share the target architecture, use a different structure within the same model type, or use another model class.
  • Stealing Approximate Model Behaviour: Approximate behaviour stealing targets prediction consistency, either on real data or on crafted adversarial data.Real-data consistency seeks matching predictions and mistakes, while adversarial-data consistency concerns transferred adversarial examples.

6 MODEL STEALING: THREAT MODEL

The threat model specifies attacker motivations, capabilities, actions, resources, and success metrics. It emphasizes that attack strength depends on the combination of knowledge and available resources.

  • Threat Model: The threat model covers attacker incentives, consequences for model owners, attack execution, defences, goals, and success metrics.This structure is intended to expose attack weaknesses relevant to defence design.
  • Attacker’s Motivation: Attackers may copy a target to bypass query restrictions, reuse a model or component, or enable subsequent white-box attacks.The paper excludes studies focused on those subsequent attacks when model stealing is only preparatory.
  • Attacker’s Capabilities: Attacker capabilities are described along knowledge, actions, and resources, including access to the target, data, computing systems, and query budgets.The paper distinguishes original, problem-domain, non-problem-domain, and artificial data.
  • Attacker’s Knowledge: Black-box access exposes only model outputs, grey-box access commonly exposes architecture, and white-box access exposes architecture and parameters.White-box access can still support attacks aimed at revealing training hyperparameters.
  • Attacker’s Actions: Query-based attacks use model interactions as their information source, whereas side-channel attacks exploit hardware or software access to the deployment resource.Side-channel attacks may also use queries as an additional information source.
  • Attacker’s Goals: Attack evaluation distinguishes exact-property extraction from behavioural stealing and uses accuracy, fidelity, and transferability for behavioural goals.Accuracy may match target performance without reproducing individual predictions, whereas fidelity directly measures simulation of target outputs.

7 QUERY-BASED MODEL STEALING ATTACKS

The survey groups query-based model-stealing attacks by stealing method and analyses adversary capabilities, efficiency, effectiveness, and attack goals. Exact parameter extraction can reproduce target behaviour, but query efficiency varies substantially across attack families.

  • Taxonomy: The taxonomy categorises query-based attacks by stealing method, attack goal, and whether the target's behaviour, parameters, or other properties are extracted.The survey also corrects classifications when an attack's implemented extraction goal differs from the authors' label.
  • Witness-finding attacks: Witness-finding attacks use sign witnesses and line searches to recover model weights, producing the target's performance when exact weights are extracted.They require the target architecture and one positive and one negative data sample, but are inefficient.
  • Equation-solving attacks: 1 to 4 queries per parameter gives equation-solving attacks perfect extraction scores across target model types.These attacks construct and solve equations from model outputs; queries need not use real or meaningful samples in some settings.
  • Path-finding attacks: 66-317 queries per parameter makes path-finding attacks the most inefficient, while non-complete queries reduce this to 44-91.For regression trees, unique leaf identifiers yielded perfect fidelity; classification performance was worse.
  • Recovering attacks: 312 queries per parameter are required for a model with 210 parameters in recovering attacks, falling to 12 for models with tens of thousands of parameters.Successful recovering attacks produce an exact target-model copy and therefore fully reproduce its behaviour.

7.5 Substitute Model Training

Substitute-model attacks train a model on inputs labelled by the target, with architecture, data domain, and query strategy shaping performance. The survey describes both data-dependent and data-free settings, including strategies that improve fidelity despite limited information.

  • Attack setup: A substitute model need only match the target's input-output format, not its architecture, and is trained using labels obtained by querying the target as an oracle.The attacker chooses the substitute architecture, training data, and sample-selection strategy.
  • Substitute architecture: A substitute model at least as deep as the target generally achieves a higher stealing success rate, and a more complex DNN can outperform a matching simpler architecture.This pattern is reported for language models and LSTMs as well as other model types.
  • Training data: Problem Domain data provides weaker attacker knowledge than original data, whereas Non-Problem Domain data is treated as the weakest knowledge when public same-modality data exists.Problem Domain data resembles the original distribution; Non-Problem Domain data shares syntactic type but may have a different marginal distribution.
  • Training data: Artificial data cannot be assigned an unequivocal attacker-knowledge level because its value depends on how it is generated and on its properties.Data-free attacks create artificial samples when little or no natural data is available.
  • Query strategies: A single couple of CFE and CCFE is enough to extract a linear model with 100% fidelity.The survey also reports that superimposed-image queries can outperform regular data with confidence scores even when only predicted labels are available.

7.6 Substitute Model Training Attacks against Specific Model Types and Domain-specific Models

Substitute-model stealing extends across specialized model types and domains, but the survey notes that attacks and defences beyond CNNs and classification remain less explored. Reported studies vary in model, data, output, and attacker-knowledge settings.

  • Research scope: The survey identifies attacks and defences on model types beyond CNNs and classification as less explored and an open topic.It also covers malware classifiers, BERT-based models, recommender systems, and other domain-specific targets.
  • Cross-model settings: Accuracy remains high even when target and substitute architectures mismatch.Other studies examine how transfer learning, architecture mismatch, and attacker data sources affect substitute performance.
  • Specialized targets: Specialized attacks target reinforcement-learning policies, image transformation models, self-supervised encoders, graph neural networks, sequential recommenders, and explanation-producing classifiers.These attacks imitate policies, embeddings, transformations, graph predictions, recommendations, or explanations depending on the target.
  • Self-supervised encoders: Using original projection heads can benefit downstream classification accuracy in encoder-stealing settings.The surveyed encoder attacks query embeddings and train substitutes to match target representations, sometimes using augmented images.
  • Graph neural networks: Graph model stealing considers node classification, link stealing, inductive inference, and varying access to node attributes, graph structure, and auxiliary data.Random attacker data is less effective for graphs because their structure provides more degrees of freedom.

7.7 Meta-model Training Attacks

Meta-model attacks infer target architecture, training setup, and data properties from a knowledge base of candidate models. They can predict hyperparameters better than guessing, but require substantial computation and do not directly produce a behavioural copy.

  • Meta-model construction: Meta-model attacks use a trained predictor to infer target architecture, training setup, and training-data amount from observed target-model behaviour.The predictor is trained on candidate CNNs varying in architecture, optimisation, and data parameters.
  • Requirements: A target hyperparameter must influence model performance and have its value represented in the meta-model's training set for successful stealing.The attack evaluates test samples against correlations learned from candidate models.
  • Limitations: 10,000 candidate CNNs required 40 days of GPU training, making meta-model attacks computationally and temporally expensive.After hyperparameter extraction, an additional parameter-stealing attack is needed to obtain a model approximating target behaviour.
  • Limitations: Meta-model attacks cannot be compared with other query-based attacks on effectiveness and efficiency because they steal hyperparameters rather than directly extracting model behaviour.Their extracted hyperparameters alone do not yield a behavioural copy.

7.8 Comparison of Query-Based Attack Performance

The survey compares query-based model-stealing attacks using effectiveness, query cost, model size, and efficiency measures. Comparisons remain difficult because studies report inconsistent metrics and often omit information needed to assess query efficiency and attacker capabilities.

  • Evaluation measures: The comparison reports target-model effectiveness, learned-parameter counts, query counts, and stealing effectiveness such as accuracy, fidelity, transferability, or AUC.When parameters and queries are available, queries per parameter provide an efficiency score.
  • Special cases: WFAs produce exact copies, so their stolen-model accuracy equals the target-model accuracy and their relative accuracy is always 100%.Therefore, the target model’s effectiveness need not be known for this comparison.
  • Effectiveness measures: Relative accuracy and fidelity are more comparable than absolute accuracy because they contrast stolen-model performance with the original model.The preferred reporting practice is to provide both relative accuracy and fidelity, alongside transferability when possible.
  • Reporting limitations: Efficiency comparisons are difficult because studies often omit either query counts or the number of learned target-model parameters needed to compute queries per parameter.The survey marks unavailable information as N/A and estimated values as est. when values were inferred from architectures or plots.
  • Comparison conditions: Efficiency scores are most useful for comparing attack variations on similar model types, because attacks are not directly comparable across model types.The survey also notes that attacker capabilities, including architecture and training-data differences, should be considered.

8 SIDE-CHANNEL ATTACKS

Side-channel attacks infer model information from hardware or software execution characteristics. The surveyed attacks recover architectures, hyperparameters, parameters, or layer sequences using timing, cache, memory, electromagnetic, power, and bus observations.

  • Overview: Side-channel attacks model effects of candidate causes by observing how candidate models influence hardware or software channels during inference.The approach can reveal architecture, hyperparameters, and learned parameters, with performance depending strongly on the execution device.
  • Software access: Cache and timing attacks infer architecture by manipulating or observing cache state and using reload-time differences to identify target accesses.Prime+Probe and Flush+Reload are representative cache side channels.
  • Hardware access: Hardware access enables attacks that extract layer sequences, architecture details, parameters, and other model properties from shared-memory, electromagnetic, power, or bus behavior.Reported targets include CNNs, DNNs, tree-based algorithms, and FPGA accelerators, including settings with protected or encrypted memory.
  • Hardware access: Some hardware side-channel attacks recover parameters despite encryption or protected memory, while combined side-channel and query-based methods train substitutes using recovered architecture information.Other attacks estimate parameter sparsity, derive weights, or steal models with identical hyperparameters, parameters, and architecture.
  • Combined attacks: Power information can increase similarity between stolen and target weights without increasing transferability.This separates weight similarity from behavioral transferability in evaluating side-channel assistance.

9 DEFENCES AGAINST MODEL STEALING

The survey systematises model-stealing defences by timing, asset, and mode. Reactive methods detect attacks or establish ownership, while proactive methods modify model outputs, weights, architecture, or decision boundaries to mitigate extraction.

  • Taxonomy: The defence taxonomy distinguishes reactive methods from proactive methods and organises choices by training-stage access and the targeted defence asset.Knowledge distillation is available when the owner can influence training; post-hoc methods modify an already trained model.
  • Defence modes: Reactive defences either verify ownership after an attack or detect an ongoing attack through query monitoring.Proactive methods alter architecture, learned parameters, decision boundaries, or overall effectiveness to mitigate expected attacks.
  • Defence goals: Defence success is not binary: reducing stolen-model fidelity enough to make the copy useless can be sufficient.Reactive monitoring may trigger proactive mitigation or halt an attack, whereas proactive defences can reduce legitimate-user accuracy.
  • Ownership verification: Unique model identifiers and dataset inference establish ownership through model-inherent properties or training-data signatures, but their applicability depends on the data and access setting.Dataset inference can misclassify independently trained models from the same distribution and is unavailable when the original dataset is public; some watermark verification requires white-box access.
  • Watermarking: Watermarks embed identifiable behavior or information during training, while dynamic schemes alter selected outputs or probabilities and later compare protected and substitute-model responses.Conferrable adversarial examples provide fingerprints that transfer to substitutes but not independently trained models.
  • Monitoring: Query monitors detect extraction using query distributions, adversarial-sample indicators, hidden-layer outputs, or extraction-completeness estimates, sometimes slowing responses.Several approaches rely on distinguishing in-distribution samples from non-problem-domain or adversarial queries.

10 ANALYSIS OF ATTACKS AND DEFENCES

The survey provides guidelines for selecting attacks and defences according to stealing goals, available resources, and model assets. Its cross-study lineup finds that several basic and early defences have been broken, while the comparison remains incomplete and setting-dependent.

  • Guidelines: The analysis offers two guidelines: how to steal a model and how to protect it against model stealing.It also compares reported attack and defence outcomes against each other.
  • Attack guideline: Attack selection depends on whether the goal is exact or approximate extraction and on the attacker’s available model, data, query, or side-channel information.Model-agnostic approaches support approximate extraction, while model-specific attacks use domain or model-type knowledge.
  • Attack guideline: Architecture stealing can use black-box meta-model attacks when a broad architecture knowledge base exists or side channels when hardware or software access is available.Meta-model attacks require a knowledge base covering many architectures, which is expensive to obtain.
  • Defence guideline: Defence selection depends on the protected asset, training-stage availability, and whether the owner seeks prevention, ownership tracking, or malicious-user detection.Architecture modification can hide the original architecture, while identifiers, watermarks, and monitors support tracking or detection.
  • Defence guideline: Combining defence techniques can provide a potentially more powerful protection level, but identifiers and watermarks are ineffective when stolen models remain unpublished and monitors may respond too slowly.These boundaries motivate matching multiple defence mechanisms to the attack setting.
  • Lineup results: Returning labels instead of confidence scores has been broken by several attacks, and several early monitors have become ineffective against newer strategies.The later monitors included in the lineup were not shown to be broken, but some defence–attack combinations remain untested.
  • Lineup limitations: The lineup is incomplete, and outcomes may differ depending on whether attackers test defences or defenders test attacks and on the effort invested.The survey therefore calls for systematic and independent assessment of defence effectiveness.

11 CONCLUSIONS

The paper systematises model-stealing attacks and defences through unified terminology, taxonomies, guidelines, and comparative analysis. It identifies substantial methodological gaps and research challenges, especially incomplete defence evaluations and limited quantitative evidence for side-channel attacks.

  • Contributions: The survey provides a comprehensive overview and systematisation of model-stealing attacks and defences, including common terminology and a taxonomy.The taxonomy classifies approaches by goals, methodology, target model type, and applicable settings.
  • Contributions: The authors develop guidelines for selecting effective attack and defence strategies and compare which defences mitigate or are broken by current attacks.The guidelines consider attacker goals, capabilities, and available resources.
  • Research methodology: The literature lacks standardised methodology and reporting, with attacks often omitting query efficiency and model complexity, hindering efficiency comparisons.The paper recommends common notation, detailed threat models, and criteria such as fidelity, accuracy, and transferability.
  • Research challenges: Side-channel research relies largely on anecdotal and qualitative evidence, while dedicated defences remain scarce and current classical security measures are limited in scope.The authors call for unified evaluation frameworks and model-inherent proactive methods that reduce side-channel effectiveness.
  • Research challenges: The current attack–defence experimental lineup is incomplete, so the overall effectiveness of proposed defence methods remains difficult to assess.Many attack–defence combinations have not been studied; the authors call for large-scale analytic and empirical evaluation.
  • Future directions: Adaptive attackers aware of defences are not yet widely considered in model stealing, and future work should examine how they counter mechanisms such as dynamic watermarking.The paper identifies adaptive attackers as a future research challenge.
Loading 2206.08451v2…