PaperlayerView exact revision on arXiv

Source-linked AI summary

Cryptanalytic Extraction of Neural Network Models

Nicholas Carlini, Matthew Jagielski, Ilya Mironov

arXiv:2003.04884v2cs.LGcs.CR

TL;DR

Model extraction asks whether a neural network’s parameters can be recovered through black-box oracle access, a problem the paper reframes as cryptanalysis. It introduces a differential attack that exploits ReLU piecewise linearity and critical-point queries to recover layers. The method extracts models far more precisely and with far fewer queries than prior work, challenging assumptions that model outputs hide trained weights.

  • Problem

    The paper asks whether secret neural-network parameters can be extracted from oracle access, a concern for APIs, obfuscated models, and secure inference.

  • Method

    The differential attack traces pairs of oracle queries and exploits ReLU critical points to recover neural-network layers sequentially.

  • Results

    The attack extracts models multiple orders of magnitude more accurately and with multiple orders of magnitude fewer queries per parameter than prior work.

  • Takeaways & Limitations

    Secure inference cannot assume that observing neural-network outputs leaves model weights hidden, so new protections are needed.

  • Takeaways & Limitations

    The approach fundamentally assumes ReLU or another piecewise-linear activation; non-piecewise-linear activations would prevent the attack.

Abstract

from arXiv · show

We argue that the machine learning problem of model extraction is actually a cryptanalytic problem in disguise, and should be studied as such. Given oracle access to a neural network, we introduce a differential attack that can efficiently steal the parameters of the remote model up to floating point precision. Our attack relies on the fact that ReLU neural networks are piecewise linear functions, and thus queries at the critical points reveal information about the model parameters. We evaluate our attack on multiple neural network models and extract models that are 2^20 times more precise and require 100x fewer queries than prior work. For example, we extract a 100,000 parameter neural network trained on the MNIST digit recognition task with 2^21.5 queries in under an hour, such that the extracted model agrees with the oracle on all inputs up to a worst-case error of 2^-25, or a model with 4,000 parameters in 2^18.5 queries with worst-case error of 2^-40.4. Code is available at https://github.com/google-research/cryptanalytic-model-extraction.

1 Introduction

The paper frames neural-network model extraction as cryptanalysis: oracle queries may reveal secret parameters, despite practical incentives to keep models hidden. It introduces a differential attack and evaluates fidelity and parameter error as extraction measures.

  • Motivation: Neural-network secrecy matters because models provide competitive advantages, and reproducing them requires costly data collection, tuning, and training.Organizations commonly expose APIs or distribute obfuscated on-device models rather than model parameters.
  • Motivation: Oracle access exposes the question of whether an identical neural-network copy can be extracted without white-box access.The paper studies an idealized oracle setting covering prediction APIs, obfuscated models, and secure inference.
  • Model Extraction as a Cryptanalytic Problem: Model extraction resembles chosen-plaintext cryptanalysis because adaptive input/output queries reveal information about neural-network weights.Unlike cryptographic ciphers, neural networks are not designed to resist such attacks.
  • Model Extraction as a Cryptanalytic Problem: Neural-network extraction differs from standard cryptanalysis because success requires recovering parameters, models are often many-to-one, and computation uses floating-point reals.Floating-point arithmetic makes the attack technically more difficult than ideal finite-field cryptanalysis.
  • Our Results: The paper introduces a differential attack that traces evaluations on pairs of nearby examples to recover neural-network layers sequentially.It also formalizes (ε, δ)-functional equivalence to quantify extraction success.
  • Our Results: Extraction success is evaluated using both functional equivalence on the input space and maximum absolute error between actual and extracted parameters.Table 1 reports these measures across model sizes and architectures.

2 Preliminaries

The paper models neural networks as parameterized functions composed of affine layers and ReLU nonlinearities, and defines extraction through oracle access and functional similarity. Its analysis assumes known architecture, arbitrary full-domain inputs, complete outputs, and precise computation.

  • Notation and Definitions: A k-deep neural network maps inputs in X to outputs in Y through alternating linear layers and component-wise nonlinear transformations.The paper studies functions over X = R^d0 and Y = R^dk.
  • Notation and Definitions: Each affine layer is represented as fj(x) = A(j)x + b(j), with matrix weights A(j) and bias vector b(j).Convolutional layers can also be represented as matrix products.
  • Notation and Definitions: ReLU activations use σ(x) = max(x, 0), making the networks piecewise linear.The paper identifies piecewise linearity as fundamental to its results.
  • Adversarial Goals and Resources: Model parameters θ are the concrete weights and biases produced during training, while extraction returns parameters ˆθ intended to make the extracted function resemble the oracle function.Training is computationally expensive and nondeterministic, so repeated training can produce different parameters.
  • Adversarial Goals and Resources: Attack success is characterized by the number of chosen inputs and the computational work required.This follows the cryptanalytic framing of extraction resources.
  • Adversarial Goals and Resources: The attack assumes knowledge of the architecture, arbitrary inputs, complete model outputs, and 64-bit floating-point evaluation.The paper treats these assumptions as potentially relaxable in future work.

3 Overview of the Differential Attack

The differential attack exploits ReLU networks’ piecewise linearity: second directional derivatives vanish away from neuron boundaries but reveal transformed weights at critical points. The attack recovers layers iteratively while addressing signs, inactive neurons, and floating-point error.

  • Overview of the Differential Attack: For a one-hidden-layer network, an input activating exactly one critical neuron makes the second differential equal the product of its incoming and outgoing weights.All other neurons contribute zero because they remain fully inactive or fully active.
  • Overview of the Differential Attack: Finite differences estimate directional derivatives from oracle queries, and standard basis directions directly expose coefficients for linear functions.This provides the basic query mechanism for recovering weights.
  • Overview of the Differential Attack: At a ReLU neuron’s critical point, a second directional derivative reveals an invertible transform of a weight, enabling recovery of the first layer.After extracting one layer, the attack peels it off and repeats the process on later layers.
  • Overview of the Differential Attack: Each recovered neuron weight vector is known only up to a scalar multiple, so the attack must determine the sign of that scalar.The general sign-recovery case requires exponential work but only a linear number of queries.
  • Overview of the Differential Attack: The attack must elicit behavior from inactive neurons and cluster partial row recoveries into complete weight vectors.Deeper layers cannot generally be probed using standard basis directions alone.
  • Overview of the Differential Attack: Finite-precision arithmetic introduces errors that compound across layers, motivating numerically stable procedures and query reuse.The first-layer extraction can have roughly 10^-6 error, which is magnified in deeper layers.

4 Idealized Differential Extraction Attack

The idealized attack assumes infinite-precision arithmetic and recovers functionally equivalent models, progressing from shallow reduced-round attacks to deeper contractive and expansive network extraction.

  • Idealized Differential Extraction Attack: The idealized attack assumes infinite-precision arithmetic and achieves (0, 0)-functional equivalence.It develops reduced-round attacks for 0-deep and 1-deep networks before treating deeper contractive and expansive networks.

4.1 Zero-Deep Neural Network Extraction

A zero-deep network is linear, so carefully chosen oracle queries directly recover its parameters through finite differences.

  • d0 linearly independent queries suffice to extract the linear function by solving a linear system.
  • Finite differences between f(x + δ) and f(x) equal A(1) · δ.
  • Basis-vector queries directly reveal the weights of A(1).

4.2 One-Deep Neural Network Extraction

One-deep extraction exploits ReLU critical points: local differential measurements recover first-layer weights and biases up to scale, signs are resolved separately, and the final layer is then extracted linearly.

  • Multiple phases recover a one-deep network layer by layer because its function is no longer completely linear.The first layer is extracted before applying the zero-deep attack to the second layer.
  • ReLU critical points are inputs where a neuron’s preactivation is zero, separating active and inactive regimes.
  • Finite-difference queries on both sides of an isolated critical point cancel other neurons’ gradients and isolate the target neuron’s weight information.The one-deep network is represented as f(x) = A(2)ReLU(A(1)x + b(1)) + b(2).
  • Repeating measurements along basis directions recovers each first-layer row up to a scalar, while the critical-point condition recovers its bias up to the same scaling.
  • Row signs are resolved by comparing outputs at preimages of hidden states perturbed by ±ei, and the final linear layer is obtained after peeling off the first layer.
  • Binary search along a random input line locates nonlinearities, whose positions provide witnesses to neurons’ critical points.

4.3 k-Deep Contractive Neural Networks

For deeper contractive networks, the attack addresses layer ambiguity, non-orthogonal hidden-state queries, sign recovery, and incomplete weight observations through filtering, local linearization, and solution unification.

  • Extending extraction to deep networks requires resolving several complications that do not arise in one-deep networks.
  • Critical points from different layers must be separated before extracting a target layer’s weights.
  • When basis-vector control is unavailable, sampled directions and hidden activations form linear equations whose solution recovers the target weight vector.
  • First-layer witnesses are identified by duplicated recovered normal vectors, while deeper-layer witnesses produce inconsistent, uncorrelated results.
  • Multiple partial weight estimates from witnesses to the same neuron are unified through overlapping coordinates and scalar consistency.

4.4 k-Deep Expansive Neural Networks

For expansive k-deep networks, extraction proceeds layer by layer: critical-point geometry supplies witnesses, while sign recovery combines filtering, polytope traversal, and brute force. The procedure uses linear queries but may require exponential work in hidden-layer width.

  • Sign recovery: The final layer preserves sign information because it has no ReLU activation and its weights can be solved directly by least squares.For the second-to-last layer, brute force can jointly recover sign bits and final-layer parameters when enough examples are available.
  • Sign recovery: Linear-query sign recovery remains possible at earlier layers, but the general procedure requires exponential work in hidden-layer width.Brute-force sign assignments are tested by checking whether a valid linear transformation exists.
  • Identifying layer witnesses: Critical witnesses for layer j are isolated by filtering earlier-layer points and testing whether later-layer polytopes match the expected geometry.Finite-difference gradients and projections along directions parallel to a critical hyperplane implement the test.
  • Constructing diverse inputs: Fully diverse witness collections require both positive and negative activations for every neuron in the projected layer space.Random continuation directions eventually succeed, but the paper notes that better strategies are more efficient.
  • Hyperplane following: A double-critical point enables hyperplane following: binary search finds a new witness across a lower-layer boundary while remaining on the target neuron’s critical set.The construction moves from x∗ to x′ and then to y with the lower-layer neuron’s sign changed.

5 Instantiating the Differential Attack in Practice

The practical attack addresses numerical instability and query inefficiency by refining imprecise layers, using stable derivative estimates, discovering critical points differentially, and choosing continuation directions carefully.

  • Practical challenges: Numerical errors compound across layers, while the idealized attack also uses more chosen inputs than necessary.The practical algorithms target both precision and query efficiency.
  • Precision refinement: Witnesses computed from an imprecise extracted layer are only approximate critical points of the true model.The discrepancy satisfies 0 < |V(η; xi)| < ϵ for some small ϵ.
  • Precision refinement: Incorrect neighboring neurons can make witness conversion fail, causing least squares to fail because it is not robust to outliers.The attack uses smaller steps, robust estimation, and rejects proposed improvements when necessary.
  • Stable derivative estimation: Directional derivative estimation is redesigned to remove an extra step-size hyperparameter, permitting a much larger step size but losing relative row-entry signs.The procedure then recovers column and row signs separately.
  • Finding witnesses: If two inputs differ in exactly one ReLU, intersecting their piecewise-linear traces directly identifies that neuron’s critical point.A mismatch between predicted and true values detects cases where multiple ReLUs changed sign.
  • Following critical points: Carefully choosing a continuation direction reduces the queries needed to recover a fully diverse witness set by several orders of magnitude.The direction is selected rather than sampled randomly, subject to an orthogonality condition.

6 Evaluation

The evaluation defines functional equivalence and estimates it through sampling, error propagation, and MILP, while accounting for neural-network parameter symmetries and computational limits.

  • Functional equivalence: The sampled estimate of (ε, 10^-9)-functional equivalence uses more than 10^9 inputs and selects ε from the ordered output errors.In practice, the authors set the sample size to 10^9 so evaluation remains feasible within about an hour per network.
  • Functional equivalence: Computing exact (ε, 0)-functional equivalence is infeasible and NP-hard even to approximate.The paper therefore proposes efficient upper-bound methods.
  • Error bounds: Weight-based error propagation aligns neurons across layers before propagating singular-value bounds to the model output.Alignment accounts for neuron permutations and positive rescaling symmetries.
  • Error bounds: The alignment-based method can fail even when networks are (2^-45, 10^-9) functionally equivalent.This motivates additional methods for tightly computing (ε, 0)-functional equivalence.
  • MILP bounds: MILP provides exact bounds with additive error at most 10^-10 but is limited to small networks because of exponential complexity.State-of-the-art solvers may provide looser bounds than the SVD approach for the evaluated networks.

7 Results

The evaluation extracts a wide range of neural-network architectures and measures functional equivalence at both δ = 10^-9 and δ = 0 on a domain that explores both sides of every neuron.

  • Experimental setup: The experiments cover a wide range of neural-network architectures.Functional equivalence is evaluated on a domain sufficient to explore both sides of every neuron.
  • Experimental setup: Functional equivalence is computed at δ = 10^-9 and δ = 0 on S = {x: ∥x∥2 < d0 ∧ x ∈ X}.The domain is chosen to explore both sides of every neuron.

8 Concluding Remarks

The paper frames model extraction as cryptanalysis and reports a practical attack that uses far fewer queries and achieves greater accuracy than prior work. It concludes that output observation can reveal model weights, while stronger cryptanalytic techniques may improve the attack further.

  • Contributions: The differential attack recovers neural-network weights by tracing evaluations on pairs of inputs that differ in a few entries and proceeding layer by layer.The attack is presented as an analogue of cryptanalysis for keyed ciphers.
  • Results: The attack requires multiple orders of magnitude fewer queries per parameter and produces models multiple orders of magnitude more accurate than prior work.The paper evaluates success using functional-equivalence fidelity.
  • Implications: The practicality of the attack falsifies the assumption that observing neural-network outputs does not reveal the weights.The authors state that secure inference therefore needs new techniques to protect trained-model secrecy.
  • Future directions: Casting extraction as cryptanalysis may enable further reductions in computational complexity, query complexity, and required assumptions.This is presented as a direction for future improvement rather than an evaluated result.
Loading 2003.04884v2…