PaperlayerView exact revision on arXiv

Source-linked AI summary

High Accuracy and High Fidelity Extraction of Neural Networks

Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, Nicolas Papernot

arXiv:1909.01838v2cs.LGcs.CRstat.ML

TL;DR

Model extraction asks whether oracle access can reproduce a proprietary victim model accurately or with matching predictions, an important question because such models are costly and sensitive. The paper develops learning-based extraction for accuracy, analyzes why nondeterminism limits fidelity, and introduces direct functionally-equivalent weight recovery. Its methods scale to production-grade settings, including a model trained on 1 billion images, while direct extraction achieves perfect reported test-sample agreement and adversarial-example fidelity.

  • Problem

    Model extraction research must distinguish task accuracy from fidelity to a victim’s predictions, because proprietary models exposed through prediction services can be stolen through query access.

  • Method

    The paper develops a victim-supervised learning attack for accuracy, analyzes learning-based fidelity limits, and directly recovers two-layer network weights for functional equivalence.

  • Results

    100% agreement on test samples and 100% fidelity on transferred adversarial examples are reported for direct functionally-equivalent extraction, while learning attacks scale to models trained on 1 billion images.

  • Takeaways & Limitations

    Model extraction attacks are practical against production-grade systems, and accuracy and fidelity require distinct attack strategies.

  • Takeaways & Limitations

    Learning-based extraction cannot guarantee fidelity because training nondeterminism prevents reproducing identical models, even with the oracle’s training details.

Abstract

from arXiv · show

In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access. We taxonomize model extraction attacks around two objectives: *accuracy*, i.e., performing well on the underlying learning task, and *fidelity*, i.e., matching the predictions of the remote victim classifier on any input. To extract a high-accuracy model, we develop a learning-based attack exploiting the victim to supervise the training of an extracted model. Through analytical and empirical arguments, we then explain the inherent limitations that prevent any learning-based strategy from extracting a truly high-fidelity model---i.e., extracting a functionally-equivalent model whose predictions are identical to those of the victim model on all possible inputs. Addressing these limitations, we expand on prior work to develop the first practical functionally-equivalent extraction attack for direct extraction (i.e., without training) of a model's weights. We perform experiments both on academic datasets and a state-of-the-art image classifier trained with 1 billion proprietary images. In addition to broadening the scope of model extraction research, our work demonstrates the practicality of model extraction attacks against production-grade systems.

1 Introduction

The paper frames model extraction around accuracy and fidelity, which can conflict for imperfect victims, and develops attacks targeting each objective. It improves learning-based extraction while introducing practical functionally-equivalent extraction through direct weight recovery.

  • Motivation: Model extraction is important because query access can let adversaries steal copies of proprietary models.Models are costly to train and depend on expensive data-collection and architecture-search processes, while disclosure raises security and privacy concerns.
  • Accuracy and Fidelity: Accuracy measures task correctness on the test distribution, whereas fidelity measures agreement with the victim on any input.For imperfect victims, high fidelity reproduces victim errors while high accuracy seeks correct predictions; functional equivalence requires agreement on all inputs.
  • Why Fidelity Matters: Fidelity supports black-box adversarial examples, membership inference, and inspection of internal representations for unintended attributes.Closer prediction replication can improve adversarial-example transfer and confidence matching, while functional equivalence can expose overlearned representations.
  • Learning-Based Extraction: The learning-based attack is 16× more query-efficient than existing techniques and scales to larger models.It uses victim-generated labels to supervise training and targets models with millions of parameters trained on billions of images.
  • Functionally-Equivalent Extraction: The paper develops the first practical functionally-equivalent extraction attack for direct recovery of a two-layer neural network’s weights.Given double-precision inference, it requires only input-output access and scales beyond prior methods requiring a power side channel or gradients.
  • Hybrid Extraction: A hybrid method combines the proposed attacks to improve both accuracy and fidelity extraction.

2 Preliminaries

The preliminaries define neural-network classifiers, their training objectives and optimization, and the model-extraction setting. They also introduce fully connected and convolutional architectures and knowledge distillation from another model.

  • Classifiers: A classifier maps inputs to a distribution over K class labels, with the predicted class given by argmax_i f(x)_i.A softmax is typically applied to logits to produce the output distribution.
  • Model Extraction: Model extraction treats the oracle as O and an extraction algorithm A(O) as producing an extracted model ˆO.Both oracle and extracted model may be parameterized by their neural-network weights.
  • Neural-Network Architectures: Neural networks alternate linear and nonlinear operations, with layer width and network depth describing their structure.Fully connected layers connect every input node to every output node, while convolutional layers use localized filters to reduce unnecessary connectivity.
  • Training: Cross-entropy training minimizes loss between target probability vectors and network predictions over a dataset.Targets may be one-hot vectors or probability distributions produced by another model.
  • Optimization: SGD computes parameter gradients on randomly sampled batches, averages them, and scales updates by a learning rate.Other optimizers use gradient statistics to reduce update variance and potentially improve performance.
  • Distillation: Knowledge distillation trains a model on probability-vector targets produced by another model, with temperature T controlling the transformed outputs.The resulting targets are normalized to sum to 1.

3 Taxonomy of Threat Models

The paper frames model extraction around adversarial motivations and goals ranging from stealing model parameters to reproducing input-output behavior. It distinguishes accuracy, fidelity, functional equivalence, and exact extraction, while showing that query complexity and computational hardness limit realistic ambitions.

  • Adversarial Motivations: Model extraction targets the confidentiality of remotely deployed architectures and parameters, motivated by economic theft and later black-box-to-white-box attacks.The extracted model can support adversarial-example and membership-inference attacks.
  • Adversarial Goals: Exact extraction requires identical architecture and weights, but neural-network symmetries make this impossible to identify from input-output behavior alone.Scaling, dead neurons, and neuron permutations can produce behaviorally indistinguishable networks.
  • Adversarial Goals: Functionally equivalent extraction requires matching the oracle on every input and is the hardest goal achievable using only input-output pairs.It recovers a member of the oracle’s equivalence class and achieves fidelity 1 for all distributions and similarity functions.
  • Adversarial Goals: Fidelity extraction maximizes prediction agreement over a target input distribution, whereas task accuracy extraction optimizes correctness on the true task distribution without matching the victim’s mistakes.The paper focuses on label agreement as its fidelity similarity function.
  • Limits of Extraction: Some neural networks require Θ(p^k) queries for extraction, and testing functional equivalence is NP-hard in general.Random deep networks can also require exp(O(h)) samples for fidelity extraction by Statistical Query algorithms.
  • Adversarial Capabilities: Learning-based extraction assumes public unlabeled task-specific data, while unknown input features and output labels remain directions for future work.The authors state that inferring features is usually more difficult than inferring labels through model interaction.

4 Learning-based Model Extraction

The paper improves learning-based extraction by using oracle supervision and unlabeled-data methods to reduce query demands. Experiments show gains on a 193-million-parameter ImageNet model and on SVHN and CIFAR10, while semi-supervised learning improves accuracy with few queries.

  • Oracle-Supervised Extraction: The learning-based attack queries the victim for labels or scores, then trains an extracted model to match them using cross-entropy loss.Experiments use ResNet-v2-50 and ResNet-v2-200 with distillation temperature T = 1.5.
  • ImageNet Extraction: The WSL oracle has 193 million parameters, 84.2% top-1 accuracy, and 97.2% top-5 accuracy after training on 1 billion Instagram images.The adversary uses public ImageNet data while treating it as unlabeled.
  • Unlabeled Data: Rotation loss trains a second classifier to predict 0°, 90°, 180°, or 270° rotations, allowing unlabeled data to contribute representation learning.The attack combines rotation loss on unlabeled examples with standard classification loss on oracle-labeled data.
  • Unlabeled Data: 64.5% accuracy is obtained with rotation loss and oracle labels, versus 62.5% with ImageNet labels and rotation loss and 61.2% without it.This result demonstrates the combined benefit of oracle labels and rotation loss in the reported experiment.
  • MixMatch: With 250 queries, MixMatch reaches 95.82% accuracy on SVHN and 87.98% on CIFAR10, exceeding fully supervised training with 4000 queries.With 4000 queries, MixMatch comes within 0.29% of the SVHN oracle and 2.46% of the CIFAR10 oracle.

5 Limitations of Learning-Based Extraction

The experiment shows that rerunning learning-based extraction with different nondeterminism produces models whose predictions can differ, preventing guaranteed fidelity. Even matching the oracle’s initialization and SGD randomness does not achieve 100% agreement, and uncontrolled randomness lowers fidelity further.

  • Learning-based extraction reruns the attack on the same oracle-labeled dataset while changing nondeterministic training sources.The attack first queries an oracle for a labeled substitute dataset, then trains models from that dataset under different randomness.
  • Even models extracted with the oracle’s SGD and initialization randomness do not reach 100% fidelity.The table caption identifies residual nondeterminism as preventing perfect agreement.
  • 93.7% fidelity is the peak on Fashion-MNIST even when initialization and training randomness are fixed.Only GPU nondeterminism remains when both initialization and randomness in SGD are fixed.
  • 93.4% fidelity is achieved with no randomness fixed, only slightly below the fixed-randomness result.The oracle uses initialization and SGD seed 0, while the experiment also evaluates different seeds.
  • Therefore, nondeterminism in the victim and extracted model’s learning procedures can compound and limit high-fidelity extraction.The prediction may depend on the extraction strategy’s nondeterminism rather than only on the oracle.

6 Functionally Equivalent Extraction

The attack directly extracts functionally equivalent two-layer ReLU networks using standard input-output queries and double-precision inference. Experiments achieve near-perfect fidelity, while errors and extension to deeper networks remain practical limitations.

  • Threat model: The attack requires only standard prediction queries and double-precision inference, avoiding gradients and side-channel leakage.It extends prior approaches based on gradient queries or power side channels.
  • Attack intuition: By locating adjacent ReLU linear regions, the attack recovers the first weight matrix and solves the second layer algebraically through least squares.The method relies on ReLU networks being piecewise linear and on forcing individual units to change signs.
  • Notation and assumptions: The attack targets one-hidden-layer ReLU networks with weights A(0), A(1) and biases B(0), B(1), assuming double-precision logits and linearly independent first-layer rows.The hidden-layer width and input and output dimensions determine the parameterization.
  • Results: 100% test-set fidelity is achieved for smaller MNIST networks, while larger networks disagree with the oracle on only 2 of 10,000 examples.The extracted first weight matrix is reconstructed to an average precision of 23 bits.
  • Results: CIFAR-10 achieves 100% agreement below 200,000 parameters and greater than 99% agreement for larger models, independent of the underlying task.The passage attributes performance primarily to the number of parameters rather than the dataset.
  • Limitations: Extraction errors arise from occasional incorrect gradient estimates, with roughly 1 in 10,000 failures potentially corrupting weights and biases.A wrong bias can affect every calculation, and deeper networks introduce unresolved critical-point and layer-identification difficulties.

7 Hybrid Strategies

Hybrid extraction combines learning-based query efficiency with direct extraction’s fidelity, improving fidelity while addressing errors that worsen as model size increases.

  • Hybrid strategies span pure learning and direct extraction, combining query efficiency with fidelity.
  • Gradient matching improved Fashion-MNIST fidelity from 95% to 96.5% in preliminary experiments.The technique assumes access to model input gradients and was not scaled further in these experiments.
  • Direct-extraction fidelity degrades with model size because first-layer weight errors induce incorrect biases that propagate into later layers.
  • Learning-based error recovery fixes many directly extracted variables and learns the remaining parameters to address numerical instability.The procedure adjusts a first-layer bias parameter and solves for remaining weights using gradient descent with training data.
  • 99.75% fidelity replaced 80% fidelity for a 128-neuron network in the worst-performing direct-extraction example.
  • The hybrid strategy is evaluated for adversarial-example transferability using extracted neural networks.The supplied table caption identifies transferability rates for adversarial examples generated with the Section 7 attack.

8 Related Work

Related work addresses model-extraction defenses, information leakage, watermarking, obfuscation, and secure computation.

  • Extraction defenses either limit information returned per query or distinguish extraction adversaries from benign users.Information-limiting proposals perturb probabilities, remove class probabilities, return only class outputs, or sample model parameters.
  • Queries and side channels can reveal hyperparameters or architectural information that empowers extraction.These capabilities are described as orthogonal to, but compatible with, the paper’s work.
  • Watermarking has been proposed to identify extracted neural networks, while obfuscation would release an equivalent program for querying.
  • Secure multiparty computation is presented as a practical approach in which each query runs a protocol between model owner and querier.

9 Conclusion

The paper studies accuracy and fidelity in neural-network extraction, demonstrates effective learning-based and direct attacks, and combines them in a hybrid method while identifying remaining scaling and capability challenges.

  • The paper characterizes model extraction around accuracy for theft and fidelity for reconnaissance.
  • Learning-based methods attack models with several million parameters trained on one billion images and reduce the attacker’s error rate by 10%.
  • Direct functionally-equivalent extraction agrees with the victim on 100% of test samples and achieves 100% fidelity on transferred adversarial examples.
  • The hybrid method uses learning-based approaches to recover from numerical instability errors during functionally-equivalent extraction.
  • Open problems include reducing the capabilities required by the attacks and scaling functionally-equivalent extraction.

A Formal Statements for Section 3.3

The formal statements establish query-complexity and computational barriers to functionally equivalent extraction, using rectangle-bounded functions and a reduction from subset sum.

  • There are width 3k, depth 2 neural networks on [0,1]^d requiring Θ(p^k) queries to extract with logit access.
  • A k-rectangle bounded function is nonzero only inside a multidimensional rectangle constrained in k coordinates.
  • Any k-rectangle bounded function can be implemented with a ReLU network of width 3k and depth 2.
  • A family of k-rectangle bounded functions requires p^k queries in the worst case, where p is feature precision.For 8-bit pixels, p = 256.
  • Checking functional equivalence of two networks on {0,1}^d is NP-hard via reduction from subset sum.The constructed network is nonzero exactly when the subset-sum instance has a satisfying subset.

B Prototypicality and Fidelity

The paper tests whether some samples are harder than others to match faithfully and finds that prototypicality predicts model agreement. More prototypical Fashion-MNIST examples are consistently classified like the oracle, while outliers are harder.

  • Prototypicality correlates with fidelity: more prototypical Fashion-MNIST test examples receive more consistent predictions across extracted models.All evaluated metrics find this relationship between prototypicality and agreement with the oracle.
  • The study ranks the Fashion-MNIST test set by increasing prototypicality and measures oracle agreement across 90 trained models.Examples are grouped into prototypicality percentiles before measuring prediction agreement.
  • Adversarial distance and holdout-retraining provide the strongest correlations without using model ensembles.For these metrics, roughly the top 50% of prototypical examples receive the same classification from nearly all 90 models.

C Supplement for Section 6

This supplement evaluates fidelity for a 16-neuron oracle by measuring the precision of logit differences between the oracle and extracted network.

  • Figure 6 measures the magnitude of the logit gap between the 16-neuron oracle and extracted network.The gap is defined as |fθ(x) − fˆθ(x)| and compares pre-softmax predictions.
  • The figure reports the distribution of bits of precision in the oracle–extracted logit difference.Precision is normalized by the value of the logit, as in the prior figure.
  • This logit-gap measure is different from, and typically stronger than, the fidelity measure used elsewhere in the paper.

D Query Complexity of Functionally Equivalent Extraction

The attack’s query complexity is dominated by critical-point search and weight recovery, yielding O(dh) total queries. This matches the Ω(dh) information requirement up to a constant factor.

  • 1. Critical Point Search: Critical Point Search requires O(hlog(h)) model queries when gradient queries are simulated with finite differences.The analysis assumes each simulated partial derivative uses O(1) queries.
  • 2. Weight Recovery: Weight recovery takes O(dh) queries because each ReLU requires O(d) queries for absolute-value and sign recovery.
  • 3. Global Sign Recovery: Global Sign Recovery requires O(h) queries, using three queries for each ReLU.
  • 4. Last Layer Extraction: Last Layer Extraction requires h queries to obtain a full-rank linear system, although previous queries can reduce this to zero in practice.
  • O(dh) total queries make the algorithm query-optimal up to a constant factor because extraction requires Ω(dh) queries without auxiliary information.The bound also removes logarithmic factors from Milli et al.
Loading 1909.01838v2…