PaperlayerView exact revision on arXiv

Source-linked AI summary

Stealing Part of a Production Language Model

Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr

arXiv:2403.06634v2cs.CR

TL;DR

Production language models disclose little about their architectures, motivating the question of how much an adversary can learn through API queries. The paper introduces a top-down attack that recovers the embedding projection layer or its dimension, obtaining precise extractions on OpenAI models and motivating API defenses.

  • Problem

    The paper addresses limited public knowledge of production language-model architectures by asking how much information API-based model stealing can extract.

  • Method

    The attack exploits the final layer’s low-rank projection from hidden states to logits and uses targeted API queries, including logit bias and logprobs, to recover the layer or its dimension.

  • Results

    10^-4 mean squared error was obtained for several OpenAI embedding-layer extractions, while a limited attack recovered gpt-3.5’s embedding dimension for under $200 USD.

  • Takeaways & Limitations

    The results show that seemingly innocuous logit-bias and logprobs API features can enable extraction of production-model parameters, prompting provider mitigations.

  • Takeaways & Limitations

    The attack requires the ability to pass a logit bias, although other API parameters may provide alternative avenues for learning logits.

Abstract

from arXiv · show

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

1. Introduction

The paper asks how much information API access can reveal about production language models and introduces a top-down attack that extracts their embedding projection layer. It demonstrates practical recovery on several OpenAI models while motivating security concerns and mitigations.

  • The paper studies how much information an adversary can learn about a production language model through API queries.
  • The attack directly extracts a transformer’s final embedding projection layer by exploiting its low-rank mapping from hidden states to logits.This top-down strategy differs from prior bottom-up reconstruction approaches.
  • The stolen layer reveals the transformer’s hidden width, which is often correlated with total parameter count, and may support future attacks.
  • The attack applies to production APIs exposing full logprobs or logit bias, and OpenAI and Google later modified APIs to defend against it or increase its cost.
  • 10^-4 mean squared error was achieved when extracting the embedding layer of several OpenAI models, up to unavoidable symmetries.
  • Under $200 USD, the authors applied a limited attack to gpt-3.5 and recovered its embedding-dimension size rather than the full layer.

2. Related Work

The related-work literature distinguishes accuracy from fidelity in model stealing and develops increasingly capable attacks for recovering neural-network behavior or parameters. This paper positions itself as a high-fidelity attack on language models using API access.

  • Model-stealing attacks target either accuracy, matching performance on a domain, or fidelity, functional equivalence across inputs.
  • Prior high-fidelity attacks commonly exploit ReLU networks and progress from gradient access to finite-difference approximations and deeper-model extraction.
  • Other work recovers hidden-layer sizes under co-location, extracts a private final layer with a public encoder, or estimates model sizes from benchmark performance.

3. Problem Formulation

The paper formulates language models as next-token probability predictors whose hidden states are projected into logits by a low-dimensional embedding matrix. Its threat model grants only query access through an idealized API.

  • The model maps an input token sequence to a probability distribution over the next token through parameterized hidden-state computation.
  • The embedding projection matrix W maps h-dimensional hidden states into l-dimensional logits before softmax.The paper denotes W as an l × h matrix.
  • The hidden dimension h is much smaller than the token-dictionary size l; LLaMA uses h values from 4096 to 8192 with l = 32,000.The paper notes that GPT-4 has an approximately 100,000-token vocabulary.
  • The adversary has no additional parameter knowledge and accesses the hosted model through a query interface treated as a perfect oracle.Timing side channels and implementation details are excluded.
  • The study compares APIs with different capabilities, beginning with an all-logits setting before reconstructing logits from more limited information.

4. Extraction Attack for Logit-Vector APIs

The attack infers a model’s hidden dimension from the numerical rank of queried logit vectors, then extends the same structure to reconstruct the final output projection matrix up to transformations. Across open-source models, it recovers dimensions nearly perfectly and reconstructs weights with errors far below a random baseline.

  • Hidden-Dimension Extraction: The attack assumes API access to logits and uses random prompts to assemble a query-response matrix Q.Each response is an l-dimensional logit vector, while the embedding projection maps from hidden dimension h to logits.
  • Hidden-Dimension Extraction: Because queried logit vectors lie in an h-dimensional subspace, rank(Q) provides a lower bound on h and equals h under full-rank assumptions.The proof writes Q = W · H, where H contains hidden states; both H and W must have rank h for equality.
  • Hidden-Dimension Extraction: SVD identifies the hidden dimension through a sharp singular-value gap after the number of queries exceeds h.For Pythia-1.4B, the largest difference occurs at approximately the 2048th singular value, matching the true hidden dimensionality.
  • Experiments: Across six open-source models, the attack recovers the embedding size with an error of 0 or 1 in five cases.The GPT-2 Small exception reports 757 instead of 768 because the model has an effective hidden dimensionality of 757.
  • Full Layer Extraction (Up to Symmetries): The SVD factors also recover the final projection as W̃ = W · G, extending the attack from dimension estimation to full-layer extraction up to transformations.With residual connections, exact recovery of W is impossible; the efficient algorithm reconstructs it up to affine transformations.
  • Full Layer Extraction (Up to Symmetries): The reconstructed projection is substantially more accurate than random weights, while the reported RMS comparison uses an affine alignment.A random model has RMS 2 · 10^-2, reported as over 100–500× higher than the reconstruction error.

5. Extraction Attack for Logit-Bias APIs

The attack reconstructs complete logit vectors from APIs exposing top-K log probabilities and controllable logit bias, enabling subsequent extraction of the embedding projection layer.

  • API setting: The target API returns top-K token log probabilities and permits real-valued logit biases before softmax.The OpenAI API previously supported biasing up to 300 tokens with values from −100 to 100.
  • Recovering logits: By cycling logit biases across token groups, the attack promotes selected tokens into the top-K and merges their adjusted outputs to recover the full logit vector.For top-5 APIs, each query promotes five tokens, after which the applied bias is subtracted.
  • Recovering logits: Because APIs expose log probabilities rather than logits, the attack compares each biased token with a common reference token to recover relative logits.Softmax invariance means absolute logits cannot be recovered, but relative differences can be measured.
  • Cost: The attack can reduce costs by forcing repeated token emissions, collecting logits on multiple extended prompts from one multi-token query.The expansion attack uses a strong bias for one token and a smaller bias for four additional tokens.
  • Restrictions: Under the strongest restrictions considered, the API reveals only the top token and allows biases of −1 or 0, preventing the preceding attacks directly.These restrictions motivate the logprob-free attack developed in the next section.

6. Logprob-free attacks

The paper extends extraction to APIs that expose no log probabilities, using logit-bias adjustments and binary search to recover relative logits, albeit inefficiently.

  • Logprob-free extraction: Even without logprob access, binary search can adjust token biases until an epsilon increase makes each token most likely, revealing relative logits.The resulting bias vector corresponds to each token’s logits relative to the others.
  • Cost: A one-token-at-a-time version requires N log(B/ϵ) queries, where N is the number of logits, B bounds their maximum gap, and ϵ is the desired tolerance.The method is effective but inefficient when applied independently to every token.
  • Cost: Parallel binary searches improve efficiency by modifying multiple tokens simultaneously and using the observed arg-max token to learn information faster.The method exploits the identity of the most likely sampled token as feedback.

7. Evaluation

The practical attacks successfully recover hidden dimensions and embedding projection layers across five OpenAI production models, with extracted weights nearly matching the originals after alignment.

  • Logit recovery: The strongest practical attack recovers logits with 18 bits of precision at 3.7 queries per logit, while a theoretically stronger method is numerically unstable and less faithful.This comparison motivates selecting the improved attack for production-model extraction.
  • Models: The evaluation studies ada, babbage, babbage-002, gpt-3.5-turbo-instruct, and gpt-3.5-turbo-1106.These were the production models for which advance permission to attempt extraction was obtained.
  • Experimental design: The practical evaluation uses the improved 4-logprob attack because it was both the most query-efficient and the most precise option.Alternative attack algorithms were not tested because switching would have substantially increased experimental cost.
  • Attack success: Both hidden-dimension and entire-layer extraction worked for all five evaluated models.Recovered hidden dimensions perfectly matched the original sizes, as confirmed by OpenAI.
  • Attack success: The full layer-stealing attack recovered weights with error < 7 · 10−4 after alignment by an h × h transform.The reported RMS compares the extracted matrix with the actual model weights after this alignment.

8. Defenses

The paper outlines API and architectural defenses, but each mitigation trades away functionality, efficiency, privacy, or model utility.

  • API defenses: Removing logit bias would directly block the attack but would eliminate legitimate uses including controlled generation and generation shifts resembling fine-tuning.The authors therefore characterize this defense as simple but functionally costly.
  • API defenses: Replacing logit bias with a token block-list preserves some functionality while preventing the attack.The block-list prohibits specified tokens rather than changing their probabilities.
  • Architectural defenses: Splitting the final projection into nonlinear h → t → l layers with t > l could prevent the attack, but the resulting final layer is inefficiently quadratic in vocabulary size.The defense changes the architecture rather than restricting API outputs.
  • Architectural defenses: Adding orthogonal low-singular-value dimensions after training can make the hidden dimension appear larger without materially changing predictions.The proposed post-hoc modification concatenates extra weight vectors and random hidden components.
  • API defenses: Prohibiting simultaneous logit bias and logprobs targets a combination that makes the attack 10× cheaper.Either capability can remain available separately under this mitigation.
  • Operational defenses: Noise, rate limits, and malicious-query detection may impede extraction, but can reduce usefulness or introduce implementation, privacy, and robustness drawbacks.Rate limiting requires state across users and can create privacy risks when defending against Sybil attacks.

9. Future Work

The paper identifies practical extensions of partial model stealing, while noting unresolved limits in recovering additional layers and removing API assumptions.

  • Quantized weights could, in principle, help recover a nearly bit-for-bit copy of the projection matrix, but the resulting integer-constrained problem is generally NP-hard.
  • The current attack recovers only one transformer layer, with no obvious extension beyond that layer because of model non-linearity.
  • The attack requires access to a logit bias, although other API parameters might provide alternative ways to infer logit information.
  • The paper leaves exploiting stolen weights and downstream attacks against finetuning APIs to future work.
  • The authors propose that partial-information attacks could steal more practical information from proprietary models than existing all-or-nothing attacks.They leave the possibility of recovering many more bits of information to future work.

10. Conclusion

The paper argues that adversarial machine learning should address attacks on deployed production models, and presents precise theft of one transformer layer as an existence proof.

  • As machine-learning models become production tools used by millions, adversarial research must assess attacks on the largest deployed models.
  • The paper provides an existence proof that one layer of a production language model can be stolen.
  • The stolen layer has no immediate practical consequences according to the authors, but it is the first precise information stolen from a deployed transformer model.
  • The conclusion identifies open questions about the hazards of practical stealing attacks and their threat relative to black-box distillation or approximate stealing.
  • The attack works because providers exposed logit-bias and logprob parameters, and both Google and OpenAI implemented mitigations after disclosure.

Impact Statement

The impact discussion describes disclosure safeguards, the practical interpretation of GPT-2 Small’s recovered dimension, and how normalization choices affect the attack’s rank signal.

  • The authors coordinated with affected products and sent advance copies to potentially affected parties before disclosing the work.
  • For GPT-2 Small, the attack recovers an effective dimension of 757 rather than the correct architectural dimension of 768.
  • SVD of 10,000 final hidden activations explains the result: GPT-2 Small has 757 activation directions despite 768 potential hidden neurons.
  • At float64 precision, all GPT-2 Small dimensions are used, but roughly the smallest dozen singular values are much smaller than the others.
  • LayerNorm can reduce the recovered dimension by one because centering projects inputs onto an (h − 1)-dimensional subspace, whereas RMSNorm does not.

B.2.2. RESULTS

Experiments show that singular-value patterns can distinguish LayerNorm from RMSNorm, including through API outputs and in a model combining RMSNorm with biases.

  • Lower-than-32-bit precision required two adjustments, while the technique worked directly with 32-bit precision.
  • The adjustments replaced baseline subtraction with subtraction of the mean across queries and computed that mean in lower precision before compact SVD.
  • LayerNorm architectures show a drop at the hth singular value, whereas RMSNorm architectures do not.
  • The same hth-singular-value drop appears for Ada and Babbage, which use LayerNorm.
  • For Gopher-7B, which combines RMSNorm with biases, the hth singular value does not decrease.

C. Proof of Lemma 4.2

The proof shows that the recovered projection is identifiable only up to transformations, and that exact recovery is impossible for transformers with residual connections. Normalization restricts the allowable transformations to orthogonal matrices, but the impossibility result extends across common architectures.

  • Recovery up to transformation: The recovery method yields W̃ = W · G for some h × h matrix G.This follows by factoring the query-output matrix with a compact SVD and absorbing the hidden-state basis change into G.
  • Impossibility of exact recovery: Exact extraction of W is impossible when the hidden-state model is a transformer with residual connections.The proof constructs distinct parameter settings with different projection matrices but identical API outputs.
  • Reparameterization symmetry: For models with fully connected layers and no normalization, any invertible h × h matrix S can transform hidden states while preserving API outputs.Input and output weights are reparameterized so the transformed hidden state is canceled by a corresponding projection-matrix change.
  • Normalization layers: With normalization layers, the proof must restrict S to orthogonal matrices because normalization commutes with orthogonal transformations but not arbitrary invertible ones.LayerNorm additionally requires the orthogonal transformation to commute with centering.
  • Scope of the result: The impossibility result holds for common model architectures, including non-residual dense networks and transformers described by Biderman (2024).The argument is extended through RMSNorm, LayerNorm, and a nonzero ε term.

G. How Far Are Our Logprob-Free Attacks From Optimal?

This section compares logit-free extraction costs with information-theoretic lower bounds and describes an orthogonal-recovery procedure under simplifying assumptions. The best attack is close to the lower bound, while realistic noise and large hidden dimensions remain practical considerations.

  • Lower bounds: For N = 300 and l ≈ 100,000, restricting each query to bias at most N tokens makes the lower bound only a factor of 2 worse.The penalty is log2(l) / log2(N).
  • Lower bounds: The best logprob-free attack is only about 1 query per logit worse than the lower bound for 6–23 bits of precision.For B = 100 and N = 300, the lower bound is 1.53–3.57 queries per logit.
  • Limitations: The lower-bound analysis assumes logits are independently and uniformly distributed over [−B, 0], while more realistic priors are left for future work.The paper identifies light-tailed unimodal distributions as a possible alternative assumption.
  • Orthogonal recovery: Recovering W up to an orthogonal matrix requires solving O(h^2) linear equations and is prohibitive for some production models with h > 1000.The technique is nevertheless demonstrated in practice on Pythia-14M under simplifying assumptions.
  • Practical defenses: Quantization at 8 bits and 4 bits produced no meaningful difference from the 16-bit baseline in recovering Llama-7B’s embedding dimension.The experiment found the same dimension in the same number of queries.
Loading 2403.06634v2…