Source-linked AI summary
Thieves on Sesame Street! Model Extraction of BERT-based APIs
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer
TL;DR
The paper asks whether attackers can extract BERT-based NLP models exposed through black-box APIs without access to training data. It uses random-word queries with task-specific heuristics and finds effective extraction across NLP tasks, while simple defenses fail against adaptive attackers.
Problem
Model extraction threatens the intellectual property of NLP models served through APIs, but the attacker’s need for task-relevant training data is the central question.
Method
The attack queries a victim BERT API with randomly generated or Wikipedia-based inputs plus task-specific heuristics, then fine-tunes a local BERT model on the outputs.
Results
Across NLP tasks, random word sequences can support effective extraction without real training data; the most expensive attack cost around $500, and simple defenses fail against adaptive adversaries.
Takeaways & Limitations
Fine-tuning large pretrained language models simplifies model extraction, making nonsensical queries sufficient for obtaining good local models under low query budgets.
Takeaways & Limitations
Watermarking does not prevent extraction, assumes public deployment of the stolen model, and can be evaded by attackers anticipating detection.
Abstract
from arXiv · showhide
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.
1 INTRODUCTION
The paper shows that BERT-based NLP APIs can be extracted without the victim’s training data, using random word queries and task-specific heuristics. These attacks are inexpensive, while simple defenses fail against adaptive adversaries.
- Motivation: Model extraction reproduces an API-served model locally from black-box query–output pairs, avoiding the original development costs.The paper frames extraction as a way to reproduce models whose parameters are hidden behind web APIs.
- Core finding: Fine-tuned BERT models can be extracted even when attackers have no access to the API’s training data.This is the paper’s central contribution for NLP model extraction.
- Attack setting: Random word sequences combined with simple task-specific heuristics suffice for extraction, although Wikipedia sentences and paragraphs improve performance.The attacks do not require grammatical or semantically meaningful queries.
- Attack setting: The most expensive reported attack cost around $500 using current API-provider rates.The paper characterizes the attacks as feasible under a low query budget.
- Analysis: Random queries are mostly nonsensical and uninterpretable, while queries closer to the original data distribution work better.The analysis also finds that attacker-side pretraining makes extraction easier.
- Defenses: Membership classification and API watermarking work against naive adversaries but fail against adversaries who adapt to the defenses.The paper argues that stronger defenses are needed for adaptive attackers.
2 RELATED WORK
Prior model-extraction research largely targeted image classifiers and boundary-seeking active-learning methods. This work addresses text’s discrete input space with nonsensical queries and task-specific settings, while differing from white-box zero-shot distillation.
- Model extraction: Earlier empirical and theoretical extraction studies focused mostly on image-classification APIs and searched near victim decision boundaries.That strategy does not transfer directly to text because text inputs are discrete.
- Model extraction: Equivalent active-learning experiments using HotFlip had limited success.The paper reports this as an initial attempt to transfer related methods to text.
- Model extraction: This work studies extraction on modern BERT-large models using nonsensical inputs for pairwise tasks such as question answering.The setting is presented as more realistic for these text tasks than prior single-input approaches.
- Distillation: Data-efficient and zero-shot distillation methods assume white-box teacher access to generate data impressions, unlike black-box model extraction.The distinction is access to the teacher model rather than only the output API.
- Rubbish inputs: Prior work found rubbish inputs effective for some shallow models but often problematic for deeper networks because one class dominates noise predictions.The paper situates its results against mixed findings on randomly generated inputs.
3 METHODOLOGY
The attack reconstructs fine-tuned BERT-based NLP models by querying a black-box victim with synthetic inputs and training an extracted model on the resulting labeled queries. It covers four diverse NLP tasks using RANDOM or WIKI queries plus task-specific heuristics.
- Attack setup: The attacker queries a black-box victim model with synthetic word sequences and trains a local extracted model on the returned input-output pairs.The attacker fine-tunes the public BERT release on this constructed dataset.
- Evaluation: The evaluation compares extracted models with the victim using development-set Accuracy and Agreement, with query budgets also expressed relative to original dataset sizes.The controlled comparison uses the same number of queries as the original training dataset unless otherwise specified.
- Tasks: The study evaluates extraction on SST2, MNLI, SQuAD 1.1, and BoolQ, spanning sentiment, natural language inference, extractive QA, and boolean QA.The tasks differ in their input structures and output spaces.
- Query generation: RANDOM queries are nonsensical word sequences sampled from a Wikipedia vocabulary, whereas WIKI queries use actual WikiText-103 sentences or paragraphs.The two generators alone are insufficient for tasks requiring complex interactions between input components.
- Task-specific heuristics: Task-specific heuristics construct structured inputs, including premise perturbations for MNLI and passage-word questions with question starters for SQuAD and BoolQ.For MNLI, three premise words are replaced with random words; for SQuAD and BoolQ, sampled passage words are framed as questions.
4 EXPERIMENTAL VALIDATION OF OUR MODEL EXTRACTION ATTACKS
The experiments show that extraction can achieve high development-set accuracy even from nonsensical queries, while hiding probability outputs causes little accuracy loss. Success often appears at small budgets, but gains diminish as queries increase.
- Controlled validation: Extracted models achieve high development-set accuracy across tasks even when trained on RANDOM inputs that do not match the original data distribution.WIKI improves performance further; extracted SQuAD models recover 95% of original accuracy despite training on nonsensical questions.
- Output access: Argmax-only API outputs cause a minimal accuracy drop relative to corresponding WIKI experiments, indicating probability distributions are not crucial for extraction.This result was measured for SST2, MNLI, and BoolQ.
- Caveat: BoolQ/RANDOM failed to converge, possibly because yes/no outputs provide sparse signal or because the victim model has limited accuracy.The victim reaches 76.1% binary accuracy versus 62.1% for the majority class.
- Query selection: SQuAD models extracted from high-agreement query subsets show large F1 improvements over models trained on low-agreement subsets, especially at small dataset sizes.Figure 2 compares average development F1 after selecting subsets from large WIKI and RANDOM pools by agreement between victim-model runs.
- Query efficiency: Extraction is often successful with small query budgets, while additional queries usually improve accuracy and higher budgets yield quickly diminishing gains.The study expresses budgets as fractions of the original dataset size and provides approximate cost extrapolations.
5 ANALYSIS
The analysis examines why nonsensical queries can extract BERT-based NLP models and how extraction depends on query agreement and pretraining choices.
- A Closer Look at Nonsensical Queries: Victim-model agreement on SQuAD answers falls from 96.9 F1 on training queries and 90.4 F1 on development queries to 53.0 F1 on WIKI and 41.2 F1 on RANDOM queries.The five victim models used identical hyperparameters and differed only by random seed.
- A Closer Look at Nonsensical Queries: High-agreement subsets consistently yield larger F1 improvements than random or low-agreement subsets of the same size, making agreement a proxy for extraction-pair quality.The analysis sorts RANDOM and WIKI queries by agreement and evaluates subsets at varying fractions of the original training-data size.
- A Closer Look at Nonsensical Queries: Measuring victim agreement and incorporating it into an active-learning objective is left as future work.The paper identifies this as an interesting direction rather than evaluating such an objective.
- The Importance of Pretraining: Attackers starting from BERT-large achieve higher accuracy than attackers starting from BERT-base, even when the victim uses BERT-base.With a fixed attacker architecture, accuracy is also better when victim and attacker use the same BERT configuration.
- The Importance of Pretraining: XLNet-large attackers outperform BERT-large attackers on SQuAD in both RANDOM and WIKI settings despite training on outputs from a mismatched BERT-large victim.The comparison keeps the victim architecture fixed at BERT-large.
- The Importance of Pretraining: Extraction without contextualized pretraining is not very effective, whereas fine-tuning pretrained language models gives attackers a significant head start.The no-pretraining comparison uses QANet on SQuAD, with 1.3 million randomly initialized parameters.
6 DEFENSES
The paper evaluates membership classification and API watermarking as defenses against BERT-based model extraction. Both can hinder naive attacks, but adaptive attackers can circumvent them or avoid detection.
- 6.1 MEMBERSHIP CLASSIFICATION: Membership inference flags nonsensical inputs as outliers and returns random outputs to remove the extraction signal.The classifier uses victim-model logits and final-layer representations, trained with original examples labeled real and WIKI extraction examples labeled fake.
- 6.1 MEMBERSHIP CLASSIFICATION: Membership classification is constrained because it risks flagging valid out-of-distribution queries and may not generalize to adaptive fake-query distributions.The limitation is especially relevant when attackers can collect real queries or deliberately construct nonsensical inputs that fool the classifier.
- 6.1 MEMBERSHIP CLASSIFICATION: 97.2% of RANDOM queries and 78.6% of WIKI queries are marked unanswerable by the SQuAD 2.0 victim model, limiting extraction information.Attackers can detect this defense and remove or downsample unanswerable queries.
- 6.2 WATERMARKING: Watermarking modifies a tiny random fraction of API queries to return wrong outputs, which can later reveal whether an extracted model memorized them.The defense preserves ordinary development-set behavior while creating a detectable difference on the watermarked subset.
- 6.2 WATERMARKING: Extracted models perform nearly identically on the development set with or without watermarking, while watermark-specific label behavior diverges.Non-watermarked models predict the victim outputs on watermarked queries, whereas watermarked models predict the altered outputs; additional epochs amplify the difference.
- 6.2 WATERMARKING: Watermarking verifies theft rather than preventing extraction, depends on public deployment, and can be evaded through adaptive retraining or query handling.Differentially private training, fine-tuning or re-extraction with different queries, and randomized outputs for matching inputs can remove the watermark.
7 CONCLUSION
The conclusion finds that BERT-based NLP APIs can be extracted effectively with low query budgets, including nonsensical inputs. Existing defenses are inadequate against adaptive adversaries, motivating stronger defenses and further study.
- 7 CONCLUSION: BERT-based NLP APIs can yield good extracted models with low query budgets, even when attackers use nonsensical input queries.The paper connects this vulnerability to fine-tuning large pretrained language models.
- 7 CONCLUSION: Existing extraction defenses are effective in some scenarios but generally inadequate against adversaries who develop counter-attacks.The conclusion calls for defenses robust to adaptive adversaries.
- 7 CONCLUSION: Future directions include using nonsensical inputs for difficult-data distillation, diagnosing dataset complexity through query efficiency, and studying victim-model agreement.The paper also proposes incorporating agreement into active learning for model extraction.
A.1 DISTRIBUTION OF AGREEMENT
Figure 3 compares agreement among five SQuAD BERT question-answering runs for RANDOM and WIKI queries. Agreement is higher for WIKI points than for RANDOM points.
- A.1 DISTRIBUTION OF AGREEMENT: Figure 3 is a histogram of average F1 agreement among five BERT question-answering models trained on the original SQuAD dataset.The comparison concerns agreement across different victim-model runs.
- A.1 DISTRIBUTION OF AGREEMENT: WIKI queries show higher agreement than RANDOM queries in the distribution of model-run agreement.The figure provides the agreement distributions for both query types.
A.2 QUERY PRICING
The appendix estimates query pricing using Google Cloud assumptions while emphasizing that actual costs vary widely. It therefore focuses on the relatively low costs needed to extract datasets rather than universal price estimates.
- A.2 QUERY PRICING: Cost estimates use Google Cloud’s calculator, with inputs longer than 1000 characters counted as multiple queries.The 1000-character limit comes from the Natural Language APIs’ typical input allowance.
- A.2 QUERY PRICING: Costs for MNLI, SQuAD, and BoolQ are extrapolated from entity-analysis and sentiment-analysis APIs because Google Cloud lacks APIs for all studied tasks.The estimate assumes each model adds a single layer to BERT-large and therefore has similar FLOPs for similar input lengths.
- A.2 QUERY PRICING: Actual query prices are difficult to estimate broadly because free quotas, multiple accounts, web scraping, infrastructure, and provider revenue models can change costs.These factors make a single widely applicable price estimate unreliable.
- A.2 QUERY PRICING: The paper emphasizes relatively low extraction costs rather than exact estimates, citing $430.56 for a 300-hour speech dataset and $2000.00 for 1 million translation queries.The passage presents these figures as examples of extraction costs for complex tasks.
A.3 MORE DETAILS ON INPUT GENERATION
The input generators construct task-specific RANDOM and WIKI queries from WikiText-103 vocabulary, sentence, paragraph, and length distributions. Classification, inference, and question-answering tasks require different heuristics to form complete queries.
- SST2: SST2 RANDOM samples tokens uniformly from the top 10,000 WikiText-103 tokens to a length drawn from WikiText-103 sentence lengths.
- SST2: SST2 WIKI samples a WikiText-103 sentence and replaces out-of-vocabulary words with uniformly sampled top-10,000 vocabulary words.
- MNLI: MNLI RANDOM samples a premise like SST2 RANDOM, then replaces three premise words with uniformly sampled top-10,000 vocabulary words to create the hypothesis.
- MNLI: MNLI WIKI samples the premise like SST2 WIKI while constructing the hypothesis with the MNLI RANDOM procedure.
- SQuAD: SQuAD RANDOM samples a WikiText-103-length paragraph from unigram probabilities, then constructs a question from randomly sampled paragraph tokens with a length between 5 and 15.
- SQuAD and BoolQ: SQuAD and BoolQ WIKI use randomly selected WikiText-103 paragraphs, while BoolQ avoids question marks and samples starters from a fixed list of common words.
A.4 MODEL EXTRACTION WITH OTHER INPUT GENERATORS
Additional query-generation analyses show that extraction quality depends on how generated inputs approximate task-specific distributions. Human agreement, task heuristics, and membership-classifier features all reveal differences between realistic and random inputs.
- SQuAD query heuristics: Unigram-frequency paragraph sampling improves RANDOM SQuAD extraction over uniform sampling, and common starters such as “what” help especially for RANDOM schemes.
- MNLI query heuristics: MNLI extraction is strongest when premise and hypothesis differ by only a few words, while very low or very high lexical overlap produces unbalanced labels.Frequent words from the top 10,000 WikiText-103 vocabulary also tend to aid extraction.
- Human annotation: Human annotator agreement is highest for original SQuAD questions and decreases across WIKI and RANDOM questions as model agreement and apparent distributional closeness decrease.
- Human annotation: 76.7 EM (85.1 F1) is the average annotator score on original SQuAD answers, compared with 23.3 EM (32.8 F1) for WIKI unanimous-victim answers.
- Human annotation: 21.7 EM (31.7 F1) is the average annotator score for RANDOM questions with unanimous victim-model answers.
- Membership classification: Last-layer BERT representations are more effective than classifier logits for distinguishing real from fake inputs in membership-classifier ablations.