PaperlayerView exact revision on arXiv

Source-linked AI summary

Knockoff Nets: Stealing Functionality of Black-Box Models

Tribhuvanesh Orekondy, Bernt Schiele, Mario Fritz

arXiv:1812.02766v1cs.CVcs.CRcs.LG

TL;DR

The paper asks whether an adversary can steal a deployed model’s functionality from black-box image-query interactions without knowing its data, internals, or output semantics. It constructs a transfer set, trains a knockoff, and finds that random out-of-distribution queries can work well, while reinforcement learning can improve sample efficiency in some settings. The study concludes that functionality stealing poses a real-world threat to deployed ML models.

  • Problem

    The paper asks whether an adversary can steal a deployed model’s functionality using only black-box input-output interactions and minimal knowledge of the victim.

  • Method

    The adversary queries images to collect predictions, constructs a transfer set, and trains a knockoff model to imitate the victim.

  • Results

    Random query images from a different distribution can produce well-performing knockoffs, while adaptive reinforcement learning improves query sample efficiency in certain settings.

  • Takeaways & Limitations

    Functionality stealing can threaten deployed ML models even when the adversary lacks knowledge of their training data, internals, and output semantics.

  • Takeaways & Limitations

    Random querying risks sampling images irrelevant to the task, and the paper only briefly discusses victim counter-strategies such as reducing prediction informativeness.

Abstract

from arXiv · show

Machine Learning (ML) models are increasingly deployed in the wild to perform a wide range of tasks. In this work, we ask to what extent can an adversary steal functionality of such "victim" models based solely on blackbox interactions: image in, predictions out. In contrast to prior work, we present an adversary lacking knowledge of train/test data used by the model, its internals, and semantics over model outputs. We formulate model functionality stealing as a two-step approach: (i) querying a set of input images to the blackbox model to obtain predictions; and (ii) training a "knockoff" with queried image-prediction pairs. We make multiple remarkable observations: (a) querying random images from a different distribution than that of the blackbox training data results in a well-performing knockoff; (b) this is possible even when the knockoff is represented using a different architecture; and (c) our reinforcement learning approach additionally improves query sample efficiency in certain settings and provides performance gains. We validate model functionality stealing on a range of datasets and tasks, as well as on a popular image analysis API where we create a reasonable knockoff for as little as $30.

1. Introduction

The paper studies whether black-box model functionality can be stolen through input-output access alone, under minimal knowledge of the victim. It frames the attack as querying images, collecting predictions, and training a competing knockoff.

  • Deployed ML models expose valuable functionality through black-box interfaces while concealing their data, architectures, and hyperparameters.These models require substantial dataset, engineering, and tuning effort before deployment.
  • The central question is whether an adversary can create a knockoff solely from observed input-output pairs without knowing the victim’s training data, internals, or output semantics.The paper distinguishes functionality stealing from attacks that infer model properties such as architecture or training data.
  • The attack has two stages: constructing a transfer set by querying the victim and training a knockoff on the resulting image-prediction pairs.The knockoff is intended to compete with the victim on its task.
  • The paper asks whether random query images suffice, which images are useful, how query efficiency can improve, and which knockoff architecture is effective.

2. Related Work

The paper positions functionality stealing as a black-box security problem distinct from prior efforts to recover model attributes or perform knowledge distillation. Its adversary operates with weaker information than standard distillation and differs from active learning in both data selection and supervision.

  • Functionality stealing targets CNN behavior through black-box access, complementing prior security work on adversarial perturbations and privacy-sensitive visual information.
  • Earlier model-stealing studies recover parameters, hyperparameters, architecture, training data, or decision boundaries, whereas this work targets functionality independently of internal details.Related approaches may assume a known model family or partial access to victim data.
  • Knowledge distillation is a special case in which the adversary knows stronger victim information, including the architecture and train/test data, and uses a white-box teacher.
  • Unlike active learning, this method selects images without knowledge of the original training data and receives pseudo-labels from the black box rather than human annotations.

3. Problem Statement

The paper formalizes functionality stealing as a two-player game: a victim deploys a black-box classifier, while an adversary queries it and trains a knockoff under limited knowledge and query budgets. The formulation distinguishes this setting from knowledge distillation and includes a limited discussion of defenses.

  • Functionality Stealing: Functionality stealing asks an adversary to replicate a victim model FV:X→Y using a knockoff FA through black-box query access.The setup is represented as a game between victim V and adversary A.
  • Victim’s Move: The victim trains a CNN on task-specific images and expert annotations, evaluates it on held-out data, and deploys it as a probability-predicting black box.Each prediction is assumed to incur a monetary or latency cost.
  • Adversary’s Unknowns: The adversary does not know the victim’s internals, training or evaluation data, or the semantics of its K output classes.The black box returns a K-dimensional posterior probability vector.
  • Adversary’s Attack: The adversary queries images to obtain transfer-set pairs {(xi,FV(xi))} and selects and trains an architecture FA to mimic FV on that set.The attack separates transfer-set construction from knockoff training.
  • Objective: The primary objective is strong performance on the victim’s unknown test distribution, with secondary objectives of query sample efficiency and understanding useful query images.Performance is constrained by a budget of B black-box queries.
  • Victim’s Defense: The paper primarily studies the adversary’s strategy and only briefly discusses victim defenses such as reducing prediction informativeness through truncation or rounding.
  • Comparison to Knowledge Distillation: Unlike knowledge distillation, the knockoff may train on an independent image distribution and lacks the teacher logits and true labels used by KD losses.The paper treats distillation as a stronger-knowledge special case.

4. Generating Knockoffs

The paper constructs knockoffs by querying a blackbox over an image distribution, optionally adapting query selection with reinforcement learning, then training a model to imitate the victim.

  • Transfer Set Construction: The adversary builds a transfer set of queried images and victim predictions, then trains a knockoff to imitate the blackbox.The transfer set contains image-prediction pairs used for knockoff training.
  • Random Strategy: Random sampling performs pure exploration but may waste queries on images irrelevant to the victim’s task.The paper gives over-querying dog images for a birds classifier as an example.
  • Adaptive Strategy: The adaptive strategy learns a policy from query feedback to improve query sample efficiency and aid interpretation of the victim.A policy selects image samples, while rewards shape updates toward higher expected reward.
  • Adaptive Strategy: Adaptive sampling uses an independent label hierarchy, selecting actions through a tree whose leaf nodes return images associated with the chosen label.Node probabilities are determined by a softmax over learned node potentials, followed by a forward pass through the hierarchy.
  • Adaptive Strategy: The adaptive policy updates node potentials with gradient bandit learning using query rewards, including certainty, diversity, and knockoff-loss signals.Rewards are rescaled to [0, 1] and baseline-adjusted when multiple measures are combined.
  • Training Knockoff FA: The knockoff can use a reasonably complex architecture such as VGG or ResNet and is trained with confidence-weighted cross-entropy equivalent to prediction KL-divergence minimization.The victim’s posterior confidence weights the cross-entropy targets.

5. Experimental Setup

The experiments evaluate functionality stealing across four image-classification victim CNNs and compare transfer-set distributions and sampling setups. The blackboxes use a common ResNet-34 training configuration, while queried distributions range from matched victim data to open-world image collections.

  • Victim Blackboxes: The study evaluates four diverse image-classification CNNs, each trained for a task specific to its dataset.The datasets address multiple image-classification challenges, including fine-grained recognition.
  • Victim Blackboxes: Each victim model uses an ImageNet-pretrained ResNet-34 trained with SGD and momentum for 200 epochs.The base learning rate is 0.1 and decays by 0.1 every 60 epochs.
  • Evaluation: Victim and knockoff models are evaluated using the datasets’ respective evaluation splits.The supplied setup identifies V as used to evaluate both model types.
  • Evaluation: After training, each victim is treated as a blackbox that maps images to posterior probability outputs.The interaction interface is images in and posterior probabilities out.
  • Transfer-Set Distributions: Transfer-set distributions include the victim training images, a closed-world pooled image universe, and open-world ILSVRC or OpenImages collections.The distributions differ in their overlap with victim training data, from 100% overlap to coincidental overlap.
  • Transfer-Set Distributions: Adaptive sampling uses auxiliary labels organized into concept hierarchies, with dataset, clustered ILSVRC, or author-provided OpenImages hierarchies.These labels guide sampling independently of the blackbox output classes.

6. Results

Across transfer-set, sampling, output, architecture, and API experiments, knockoffs generally reproduce victim functionality, while adaptive querying improves efficiency in selected settings.

  • Transfer-set construction: Random queries from independent image distributions recover 0.84-0.97× blackbox performance in closed-world and 0.81-0.96× in open-world settings.Open-world knockoffs perform despite never encountering more than 90% of test-time bird classes.
  • Transfer-set construction: At most 0.04× performance variation between ILSVRC and OpenImages indicates that large, diverse image sets can provide effective transfer data.
  • Adaptive sampling: Adaptive querying reaches 68.3% on CUBS200 at B=10k, compared with random querying at B=60k, while improving performance by up to 4.5% across victim models.Adaptive performance is comparatively low on Diabetic5 because confident victim predictions provide poor policy feedback.
  • Adaptive sampling: Adaptive policies favor victim-relevant classes in closed-world settings and combine direct or coarser task-related actions in open-world settings.The policy can also select richer alternative image sources for relevant output classes.
  • Blackbox outputs: Truncated predictions retain functionality: top-k recovery reaches 0.96× at k=5, while rounding reaches 0.99× original accuracy at r=2.Argmax alone achieves 0.76-0.84× original accuracy across budgets.
  • Architecture and real-world validation: Knockoff performance transfers across model families, and more complex knockoff architectures are generally beneficial, although real-world API results show compact and complex models performing similarly.For the API, knockoffs achieve 0.76-0.82× API performance, with queries costing only $30.

7. Conclusion

The paper demonstrates that model functionality can be transferred from a victim to a knockoff through blackbox access, with strong performance observed even for a popular image recognition API. The authors characterize functionality stealing as a real-world threat to deployed ML models.

  • Blackbox access can transfer a victim model’s functionality into a knockoff.
  • Strong knockoff performance was observed on a popular image recognition API.
  • Functionality stealing potentially undercuts an increasing number of deployed ML models.

Appendices

The appendices extend the paper’s experiments, describe datasets and overlap calculations, and provide additional implementation details. They also document supplementary qualitative results and adaptive-policy analyses.

  • The appendices include extensions of existing results, additional implementation details, and dataset descriptions.
  • The appendices provide supplementary material on sample efficiency, adaptive policies, and qualitative knockoff predictions.
  • The blackboxes cover Caltech256, CUBS200, Indoor67, and Diabetic5 image-classification tasks.
  • Overlap between blackbox labels K and adversary labels Z is calculated as 100 × |K ∩ Z|/|K|.

B.3. Dataset Aggregation

The dataset-aggregation appendix documents the construction of OpenImages-derived data and compares training with ground-truth labels against knowledge distillation using victim-training images.

  • OpenImages aggregation retrieves 2k images for each of 600 boxable categories, producing 554k unique images before removals.
  • Training on victim ground-truth data is compared with knowledge distillation to assess knockoff sample efficiency.
  • Face extraction uses OID bounding boxes, a 25% margin, a minimum size of 180×180 pixels, and at most five faces per image.

B.4. Additional Implementation Details

Additional experiments and implementation details examine victim-specific training, adaptive querying, qualitative predictions, and knowledge-distillation comparisons. The appendix also analyzes how adaptive policies change their queried images over time.

  • B.4. Additional Implementation Details: The Diabetic5 victim uses class-weighted loss, yielding approximately 8% absolute test-accuracy improvement, while knockoffs use non-weighted loss.
  • B.4. Additional Implementation Details: The adaptive strategy trains a policy online during transfer-set construction, then retrains the knockoff offline on the accumulated transfer set.
  • C. Extensions of existing results: Qualitative figures show knockoff predictions on victim classes whose images were not encountered during knockoff training.
  • C. Extensions of existing results: Knowledge distillation achieves comparable performance in all but one case and performs better for Caltech256 and Diabetic5.
  • C. Extensions of existing results: For CUBS200, adaptive querying first selects images overlapping victim training data, then explores related categories after those images are exhausted.

C.4. Reward Ablation

Reward ablation results largely match earlier observations for Indoor67, while Diabetic5 knockoffs show little-to-no improvement because its victim makes confident predictions across images.

  • Reward ablation results for the remaining datasets are reported in Figure 18, with similar observations for Indoor67.
  • Per-class evaluation separates performance on seen and unseen classes.
  • Diabetic5 knockoffs show little-to-no improvement in the reward ablation experiment.The victim model makes confident predictions on all images.

D. Auxiliary Experiments

Auxiliary experiments examine seen versus unseen classes, hierarchical versus flat adaptive querying, and semi-open-world overlap. They find strong unseen-class performance, a benefit from hierarchy, and robustness to low image overlap, while label overlap can require exploration.

  • Per-class evaluation: >70% performance is achieved on unseen classes as well as seen classes by the knockoff.Mean per-class accuracies are reported separately for classes encountered and not encountered during knockoff training.
  • Hierarchy: A hierarchy makes adaptive querying more sample-efficient than adaptive-flat querying in a large action space.In CUBS200, adaptive reaches 50% accuracy with 2× greater sample efficiency.
  • Semi-open world: The semi-open-world setting parameterizes overlap between the adversary’s and victim’s images using τd, and overlap between their labels using τk.The closed-world case is τd = τk = 1.
  • Semi-open world: Random querying maintains comparable performance across semi-open-world image and label overlaps.
  • Semi-open world: Adaptive querying remains strong with low image overlap, but pure certainty-based exploitation can lose performance when label overlap decreases.In Caltech256, the performance difference is at most 3% even at τd = 0.1; diversity and loss rewards recover performance in the open-world transition.
Loading 1812.02766v1…