PaperlayerView exact revision on arXiv

Source-linked AI summary

Stealing Machine Learning Models via Prediction APIs

Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, Thomas Ristenpart

arXiv:1609.02943v2cs.CRcs.LGstat.ML

TL;DR

The paper asks whether confidential ML models exposed through public prediction APIs can be copied through black-box queries. It develops extraction attacks using API outputs and evaluates them across model classes and services, finding near-perfect or exact functional recovery, while class-label-only interfaces still permit harmful attacks.

  • Problem

    Public query access creates a confidentiality risk for commercially valuable, privacy-sensitive, and security-critical ML models.

  • Method

    The paper develops equation-solving attacks for models with logistic output layers and path-finding attacks for decision trees, then studies class-label-only retraining attacks.

  • Results

    The attacks extract functionally close models across decision trees, logistic regressions, SVMs, and deep neural networks, including 100% equivalent models on tested BigML and Amazon inputs.

  • Takeaways & Limitations

    Confidence values and partial queries enable efficient extraction, while omitting confidence values only yields slower attacks that can still exceed 99% agreement.

  • Takeaways & Limitations

    With service-side feature extraction and class labels only, the paper leaves an in-depth study of extraction using indirect queries for future work.

Abstract

from arXiv · show

Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis. The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial feature vectors as inputs and include confidence values with predictions. Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning. We further show that the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks. Our results highlight the need for careful ML model deployment and new model extraction countermeasures.

1 Introduction

The paper studies black-box model extraction from publicly queryable ML services, showing efficient attacks across model classes and against Amazon and BigML. It also finds that removing confidence values reduces effectiveness but does not eliminate potentially harmful extraction.

  • Motivation: Publicly accessible prediction APIs can expose commercially valuable or confidential models and training data to extraction attacks.The risk is especially consequential for security models, where learning the model can facilitate evasion.
  • Threat model: An attacker with black-box access seeks an equivalent or near-equivalent model without prior knowledge of the target’s parameters or training data.The goal is close to 100% agreement over an input space of interest.
  • Attacks with confidence values: Confidence values and partial-input queries enable equation-solving attacks against logistic regression and neural networks, while decision trees require path-finding techniques.For logistic regression, d + 1 random queries can solve for d + 1 unknown parameters with high probability.
  • Evaluation: The attacks achieve 100% agreement on tested inputs for models hosted by BigML and Amazon using computationally fast attacks with few queries.The study validates attacks locally and presents online case studies against both services.
  • Countermeasures: Omitting confidence values still permits attacks exceeding 99% of the input space, although some require up to 100× more queries.These slower retraining attacks use only class labels and remain potentially harmful.
  • Implications: The attacks can also leak summary statistics about private training sets and significant information about individual training points in some kernel logistic regression models.These implications extend extraction beyond functional duplication.

2 Background

The paper formalizes ML models as functions from feature vectors to outputs, distinguishing feature types, class-label outputs, confidence values, and training parameters. It also defines the distance measures used to compare extracted and target models.

  • Model representation: An ML model is a function f: X → Y mapping a d-dimensional feature vector in X to an output in Y.The feature space is the Cartesian product X_1 × X_2 × ··· × X_d.
  • Feature types: Categorical features take values from finite sets, whereas continuous features range over bounded real intervals.Feature extraction may transform user inputs from a space M into the model’s feature space.
  • Outputs: Classification APIs may return only class labels or real-valued confidence measures for each class.With c classes, confidence outputs lie in [0,1]^c and can represent model-assigned class probabilities.
  • Comparison metric: The paper compares class labels using 0-1 distance and compares class probabilities using total variation distance.Unless specified otherwise, d_Y denotes 0-1 distance over class labels.
  • Training: Supervised training maps labeled examples to a model defined by model-specific parameters and hyper-parameters controlling the model type.An SVM kernel type is an example of a hyper-parameter.

3 Model Extraction Attacks

The paper formalizes model extraction as learning a target from black-box queries and examines its motivations, threat models, MLaaS setting, and evaluation criteria.

  • Model extraction seeks a model that closely approximates or matches a black-box target using prediction queries.
  • MLaaS platforms let users upload data, train models, and expose resulting models through prediction APIs, sometimes with black-box access and monetization.
  • Attackers may pursue extraction to avoid query charges, violate training-data privacy, or facilitate evasion against security classifiers.
  • The investigated ML APIs break the assumption that model secrecy protects deployed models in these settings.
  • The threat model includes direct queries to arbitrary inputs and indirect queries transformed by an extraction mechanism that may be unknown to the adversary.
  • The study measures agreement using test error on training-like inputs and uniform error over the full feature space, with extraction accuracy defined as one minus error.

4 Extraction with Confidence Values

The paper attacks prediction APIs that return confidence values by treating outputs as equations or path identifiers, achieving highly accurate extraction across several model classes and exposing training data.

  • Confidence values and partial-query support provide information-rich outputs that distinguish these attacks from traditional label-only membership-query settings.
  • Decision-tree attacks use confidence values as pseudo-identifiers for paths, enabling discovery of tree structure when equation solving does not apply.
  • For logistic models, queried probabilities form equations in unknown parameters that can often be solved to recover the target or a close approximation.
  • 4.1 Equation-Solving Attacks: 0 test and uniform error were achieved for binary logistic regression, with total-variation error below 10^-9 using 41 queries on average and 113 at most.
  • 4.1.2 Multiclass LRs and Multilayer Perceptrons: Multiclass logistic regression reached perfect extraction with k queries, while MLPs achieved over 99.9% accuracy with 5,410 samples on average and 11,125 at most.
  • 4.1.3 Kernel Logistic Regression: Kernel logistic regression extraction can leak representers from private training data, and extracted models can support inversion with about 20× fewer remote queries.

4.2 Decision Tree Path-Finding Attacks

The paper introduces path-finding attacks that exploit prediction APIs’ confidence-derived identifiers and partial queries to recover decision-tree structure. The attacks identify leaf paths and split predicates, extending extraction to practical MLaaS tree models.

  • Attack idea: Confidence values can act as pseudo-identifiers for tree paths, enabling structural extraction despite decision trees’ discrete outputs.Unlike logistic models, tree confidence values do not support direct equation solving, so the attack uses API-specific path information.
  • Model scope: The formal tree model supports binary and multi-ary categorical splits, binary numeric splits, and confidence-labeled leaves.The attacks also apply to regression trees, often with better results.
  • Extraction algorithms: The attack queries a leaf identity, searches feature constraints preserving that leaf, and generates inputs for unvisited paths.Continuous features use line search, while categorical features use category splitting.
  • Extraction algorithms: Partial-input queries enable a top-down variant that extracts the tree layer by layer, beginning with the root and recursively recovering lower splits.The method identifies the root split by varying one feature at a time while other features remain absent.
  • Limitations: Duplicate leaf identities can cause missed splits when different leaves share an identifier and differ in only one feature.The attack mitigates this by checking whether queries violate already observed leaf predicates.
  • Evaluation setup: On BigML public trees, the experiments use prediction, confidence, and field information to emulate black-box API access.Table 5 covers decision-tree extraction datasets, while Table 6 summarizes leaves, unique identifiers, and maximum depth.

5 Online Model Extraction Attacks

The paper evaluates black-box extraction attacks against BigML and Amazon using only exposed APIs and models trained in the authors’ accounts. The attacks recover BigML tree paths exactly and extract Amazon models despite feature transformations.

  • Evaluation scope: The attacks use only exposed APIs and do not bypass authentication or access-control mechanisms; tested models were trained in the authors’ own accounts.This constrains the online case studies to authorized model access settings.
  • BigML: BigML’s decision-tree model was extracted exactly using 1,722 queries with the basic attack and 1,150 with the top-down variant.The corresponding durations were 1,030 and 631 seconds, dominated by approximately 500 ms query latency.
  • BigML: The BigML attack can create economic incentives when a user expects to make more than 1,150 predictions.The attack cost depends on the model owner’s per-prediction fee.
  • Amazon: Amazon’s service uses logistic regression and feature extraction including one-hot encoding and quantile binning.The attacker can locally reproduce one-hot encoding, while quantile binning requires reverse-engineering training-data thresholds.
  • Amazon: Using 650 queries, the attack extracted an Amazon Digits model with Rtest = Runif = 0 when categorical features and quantile binning were disabled.For default feature extraction, the experiments reverse-engineered quantile bins and then extracted the transformed model.
  • Discussion: The study does not analyze text-analysis or Cartesian-product feature extractors, although the authors believe these may also be reverse-engineered.This is an explicit scope boundary rather than an evaluated result.
  • Unknown model characteristics: When model characteristics are uncertain, the paper uses an extract-and-test strategy across attack assumptions and evaluates candidate models on test points.The Amazon experiments use this approach to infer undocumented model characteristics.

6 Extraction Given Class Labels Only

With class labels only, the paper evaluates retraining-based extraction attacks across linear, multiclass, and kernel models, finding that adaptive strategies can still achieve high fidelity but often require more queries.

  • Retraining strategies: Adaptive retraining selects points near the extracted model’s decision boundary, while line-search retraining uses adaptive queries to locate boundary-near samples.Both strategies train a local model on queried oracle labels; adaptive retraining repeats boundary-focused sampling over multiple rounds.
  • Linear binary models: Boundary-searching retraining strategies outperform uniform retraining, with adaptive retraining most efficient at relatively low budgets and Lowd-Meek best when line searches are affordable.The comparison is based on average Rtest and Runif across binary logistic-regression models.
  • Indirect queries: Indirect queries prevent Lowd-Meek line searches over the original input space, but adaptive retraining showed no major accuracy difference for the evaluated linear models.The paper leaves in-depth study of indirect queries with class labels only for future work.
  • Multiclass models: For multiclass models, 100·c·(d + 1) queries produced extraction accuracy above 99.9%, averaging 26,000 queries and reaching 65,000 on Digits.Equation-solving attacks achieved similar or better results with 100× fewer queries, while the Lowd-Meek attack does not apply directly to multiclass settings.
  • Multiclass models: At 100·k queries per multiclass model, uniform retraining reached Rtest = 99.16% and Runif = 98.24% using 108,200 queries on average.Line-search and adaptive retraining provided little benefit over uniform retraining for the tested budgets.
  • RBF kernel SVMs: Adaptive retraining extracted nonlinear kernel SVMs with over 99% accuracy at the largest tested budget, although these models were harder to retrain than linear models.The attacks used retraining because SVMs do not provide class probability estimates.

7 Extraction Countermeasures

The paper examines API minimization, confidence rounding, differential privacy, and ensembles as countermeasures. Removing confidence values blocks many attacks but does not eliminate potentially harmful label-only extraction, while rounding provides limited protection.

  • Prediction API minimization: Removing confidence values and rejecting incomplete queries can prevent many attacks, especially equation-solving and feature-discovery attacks, but class-label-only APIs still permit successful extraction.The paper frames this defense as constrained by the need to preserve usefulness for honest applications.
  • Rounding confidences: Rounding confidence scores can weaken equation-solving and decision-tree attacks by truncating probabilities or increasing path-identifier collisions.The resulting extracted model may differ from the target, and tree-node collisions reduce path-finding success.
  • Rounding confidences: Four or five decimal places leave softmax extraction success unchanged, while two or three decimals weaken the attack but still outperform label-only adaptive retraining.The comparison uses average Rtest on benchmark softmax models.
  • Rounding confidences: For the evaluated regression trees, rounding has no effect because each leaf’s output is unique; some classification-tree attacks remain resilient to rounding.The paper notes that rounding outputs for regression trees could affect utility more critically.
  • Differential privacy: Differential privacy may reduce leakage about individual training examples, but it is not defined to prevent model extraction.The paper suggests that protecting model parameters directly would be a more appropriate strategy, with required privacy budgets left open.
  • Ensembles: The authors suspect ensemble methods may yield only coarse approximations under extraction, but they did not experimentally evaluate them as targets.They note that ensembles may remain vulnerable to other attacks such as model evasion.

8 Related Work

The work connects model extraction to learning with membership queries, adversarial evasion, poisoning, substitute-model attacks, and neural-network interpretation or compression.

  • Learning theory: Model extraction resembles learning an unknown member of a known hypothesis class through membership queries that return labels without error.The paper distinguishes its setting from classical learning theory in conceptual motivation and API behavior.
  • Model evasion: Lowd-Meek-style work studies binary classifiers and model evasion, while this paper extends extraction across broader model types and prediction APIs.Prior theory and practice suggest extraction is harder than evasion.
  • Adversarial machine learning: Poisoning attacks inject crafted samples into training or test data to reduce a model’s accuracy, placing them within adversarial machine learning rather than extraction.The paper situates evasion and poisoning as related adversarial settings.
  • Substitute models: A concurrent substitute-network approach targets about Rtest ≈80% extraction accuracy because that level can still support transferable adversarial examples.The substitute is trained using data-augmentation techniques similar in spirit to adaptive retraining.
  • Interpretation and compression: Improper model extraction has also been used to interpret and compress complex neural networks in non-adversarial settings.These applications differ from the paper’s security-focused extraction setting.

9 Conclusion

The paper shows that flexible ML prediction APIs enable practical model extraction attacks with implications for monetization, training-data privacy, and evasion. Restricting API information reduces effectiveness but does not remove the threat.

  • Conclusion: Flexible APIs expose confidence scores and partial-input querying that enable equation-solving attacks and decision-tree path finding against cloud ML services.The paper demonstrates these attacks through local experiments and online attacks on BigML and Amazon.
  • Conclusion: Restricting API information can make extraction less effective, but adaptively chosen class-label queries may still support potentially harmful retraining attacks.The paper identifies production-grade evaluation of these attacks and refined countermeasures as future work.

A Some Details on Models

This section describes the model classes and prediction functions considered for extraction, including SVMs, logistic regression, neural networks, and decision trees. These models use linear, kernelized, nonlinear, multiclass, or tree-based computations to produce labels and sometimes confidence scores.

  • SVMs: Linear SVMs classify inputs by the sign of a separating hyperplane, while kernel techniques extend them to non-linearly separable data.
  • SVMs: Kernel SVMs replace explicit feature projections with kernel evaluations over learned weights and training-set support vectors.
  • Logistic regression: Multiclass logistic regression assigns one weight vector and threshold per class, producing softmax scores whose largest value determines the label.
  • Logistic regression: Logistic regression can use one-vs-rest class models or kernel techniques when linear decision functions are insufficient.
  • Deep neural networks: Neural networks apply nonlinear hidden-unit transformations before a softmax output layer.
  • Decision trees: Decision trees route each input through feature-based splits until a leaf outputs a class label or a class label with confidence.

B Details on Data Sets

The experiments use synthetic, public, and real user-created MLaaS datasets to cover nonlinear classification, supervised-learning applications, and realistic decision-tree deployments.

  • Synthetic data sets: Four synthetic datasets represent concentric circles, interleaving moons, Gaussian blobs, and five-class Gaussian clusters.
  • Public data sets: Public datasets include UCI classification benchmarks, model-inversion datasets, and scikit digits for logistic and SVM experiments.
  • Public data sets and models from BigML: Decision-tree experiments use varied public BigML models trained by real MLaaS users across application scenarios.
  • Public data sets and models from BigML: The BigML models cover tax-state prediction, survey and demographic outcomes, email importance, spam detection, and German Credit data.

C Analysis of the Path-Finding Algorithm

The path-finding analysis establishes termination and leaf coverage for the decision-tree extraction algorithm, then bounds its query complexity under assumptions about leaf identities and split resolution.

  • Assumptions: The analysis assumes uniquely identified leaves and no continuous-feature interval narrower than ε.
  • Correctness: The algorithm terminates because it adds queries only when visiting new leaves, and the tree has finitely many leaves.
  • Correctness: Unique leaf identities and ε-bounded intervals ensure that exploring a deepest common ancestor’s split discovers the subtree containing every unvisited paired leaf.
  • Complexity: O(m · (dcat · k + dcont · m · log(b/ε))) bounds total queries for trees with m leaves, categorical-feature count dcat, continuous-feature count dcont, and categorical arity k.
  • Complexity: For boolean trees, complexity is O(m·d), and unique leaf identities can provide an exponential speedup over class-label-only membership-query extraction for degenerate trees.
  • The Top-Down Approach: The top-down algorithm’s analogous analysis assumes uniquely identified nodes rather than uniquely identified leaves.

D A Note on Improper Extraction

This section compares generic improper extraction with extraction tailored to the target model class. Although a sufficiently large multilayer perceptron can approximate arbitrary continuous targets, the experiments indicate that this generic strategy can be less effective than class-specific extraction.

  • Generic improper extraction: A single-hidden-layer multilayer perceptron can, in principle, closely approximate any continuous function on a bounded subset of R^d.
  • Generic improper extraction: Tailoring extraction to the target’s simpler model class can be more efficient than using an over-parameterized generic representation.
  • Experimental comparison: 530 queries recovered the softmax-regression target’s labels on 100% of tested inputs, with class-probability TV distance below 10^-7.
  • Experimental comparison: With a multilayer perceptron having 111,005 parameters and 5,300 queries, extraction achieved 99.5% label accuracy and TV distance 10^-2 for class probabilities.
Loading 1609.02943v2…