PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Secrets Extraction against Black-box LLMs

Shiqian Zhao, Siwei Jiang, Xinfeng Li, Runyi Hu, Yandan Zheng, Congyu Guo, Tianwei Zhang, Anh Tuan Luu

arXiv:2609.36941v1cs.CR

TL;DR

Commercial API-based LLMs may memorize credentials in training data, while prior extraction work often depends on access to model internals. This paper distills validated secret behavior into a local proxy and searches it offline; evaluations report improved recovery, real-key rates, and latency, as well as masked credentials from deployed systems.

  • Problem

    Secret extraction from closed-source API-based LLMs remains largely unexplored because their interfaces conceal token probabilities, model states, and gradients, despite risks that training data contains credentials.

    Source: §1 · p. 1

  • Method

    The framework distills cross-validated, format-checked victim responses into a local proxy, then uses truncated top-p sampling and entropy, N-gram, and provider-specific filters to search offline.

    Source: §1 · p. 2

  • Results

    Across five providers, recovery improved up to 25.0%, real-key rates up to 2.6×, and latency fell up to 34.8×; masked credentials were recovered from three deployed systems.

    Source: §5 · pp. 9–10

  • Takeaways & Limitations

    Output-only access can expose memorized secrets, indicating an operational security risk for commercial LLM deployments.

    Source: §5 · p. 10

    Evaluation used authorized model interfaces in controlled settings, and extracted commercial-platform API keys were truncated and masked.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Large language models (LLMs) increasingly power autonomous coding agents such as Codex and Claude Code, yet their training corpora may contain confidential credentials exposed in public repositories or collected from private development artifacts, creating risks of memorization and subsequent leakage. Existing extraction audits, however, largely assume access to model weights or token probabilities. In this work, we present a black-box secret extraction framework for commercial, API-based LLMs under output-only access. It comprises (i) \emph{Cross-Validated Secret Knowledge Distillation}, which uses semantics-preserving prompt variants, response cross-validation, and provider-specific format filtering to distill secret-relevant behavior into a local white-box proxy; and (ii) \emph{Proxy-Guided Secret Extraction and Candidate Filtering}, which combines truncated top-$p$ sampling with local token entropy, $N$-gram frequency profiling, and provider-specific structural priors. On controlled API-key benchmarks, our framework improves recovery effectiveness and real-key rates over representative baselines while reducing extraction latency. A responsible real-world evaluation further recovers masked provider-specific credentials from three independently deployed black-box LLM systems spanning OpenAI and Claude Code, showing that memorized secrets can be exposed under output-only access.

1 INTRODUCTION

The section frames secret extraction from API-based LLMs as an understudied problem because commercial endpoints expose generated text but conceal internal signals. It proposes an output-only framework and reports benchmark and real-world evidence of extraction.

1 INTRODUCTION

  • Training data can include credentials exposed in public repositories or collected from private development artifacts, risking memorization and subsequent leakage.The paper identifies potential severe security breaches and financial loss as consequences.Source: §1 · p. 1
  • Prior secret-extraction studies primarily analyze open-source models using accessible parameters or token probabilities (Yang et al., 2025).These studies examine token-level differences or training dynamics in relation to API-key extraction.Source: §1 · p. 1
  • Commercial API-based models remain largely unexplored because their interfaces expose generated text but hide token probabilities, model states, and gradients.Output-only access complicates distinguishing memorized credentials from hallucinations, while greedy decoding repeatedly explores narrow patterns.Source: §1 · pp. 1–2
  • A two-stage framework distills validated victim responses into a local proxy, then searches it using truncated top-p sampling and filters candidates with entropy, N-gram profiles, and provider-specific priors.Moving broad candidate search and filtering offline reduces repeated victim queries while improving validity and diversity.Source: §1 · p. 2
  • The framework outperformed representative baselines by up to 57.1% on controlled API-key benchmarks and recovered masked credentials from three independently deployed black-box LLMs.The benchmarks comprised 100 authentic keys from five providers and four programming languages.Source: §1 · p. 2

2 RELATED WORK

The related work connects API-key memorization to repeated training sequences and distinctive code contexts, then distinguishes credential extraction from broader black-box privacy audits. It emphasizes that existing approaches do not address high-entropy, format-constrained API keys under strictly black-box access.

2.1 LLM MEMORIZATION

  • Repeated sequences and distinctive code-context anchors can make even high-entropy API keys vulnerable to memorization.Secrets may enter training data from public repositories or private code, prompts, and execution traces.Source: §2.1 · p. 2

2.2 EXTRACTION FROM LLMS

  • HCR extracts credentials from repository infill contexts (Huang et al., 2024); DESEC steers open-weight decoding; Yang et al. (2025) tracks training dynamics.HCR depends on concrete repository contexts, while DESEC and checkpoint analyses require internal model access unavailable at commercial endpoints.Source: §2.2 · p. 2
  • Broader PII extraction work uses output-only prompts and semantic-entropy-guided mutation, but leaves high-entropy, format-constrained API-key elicitation unaddressed.The cited work includes CodexLeaks and Codebreaker.Source: §2.2 · p. 3

3 METHOD

The method first distills validated victim responses from semantics-preserving prompt variants into a local white-box proxy, then uses that proxy for diverse candidate generation and authenticity-aware filtering. This design moves broad search offline while reducing repeated victim queries.

3 METHOD

  • The two-stage framework distills output-only victim responses into a local proxy, then searches it with truncated sampling and hybrid candidate filters.Remote interaction is confined to knowledge acquisition; candidate search and filtering run locally.Source: §3 · p. 3

3.1 CROSS-VALIDATED SECRET KNOWLEDGE DISTILLATION

  • Five semantics-preserving transformations vary the invocation prefix through comment edits, dead code, formatting, equivalent refactoring, and identifier renaming.Transformation intensity is categorized as weak, medium, or strong to represent different degrees of context drift.Source: §3.1 · p. 4
  • Provider-specific local character and prefix checks, followed by global length checks, remove malformed victim completions before distillation.The filters enforce vendor-specific headers and allowed characters, then prune keys with aberrant lengths.Source: §3.1 · p. 4
  • Hard-label response distillation trains the proxy on filtered victim sequences as pseudo-labels, reinforcing responses that recur across prompt variants.The proxy minimizes autoregressive negative log-likelihood; inconsistent perturbation-induced artifacts are attenuated.Source: §3.1 · p. 4

3.2 PROXY-GUIDED SECRET EXTRACTION AND CANDIDATE FILTERING

  • Top-p sampling broadens candidate coverage, while independent decoding runs explore different paths and deduplication prevents repeated completions from inflating counts.Larger p admits lower-probability tokens; larger k increases local inference cost without additional victim queries.Source: §3.2 · p. 5
  • Filtering policies are calibrated on authentic and hallucinated candidate sets, with calibration keys kept disjoint from final evaluation keys.Thresholds are fixed on the calibration split before final evaluation.Source: §3.2 · pp. 5–6
  • Minimum-window entropy screening targets short repetitive regions that global entropy can conceal, using provider-calibrated thresholds to reject locally collapsed candidates.The policy complements global entropy by targeting low-entropy windows.Source: §3.2 · p. 6
  • Repeated-substring suppression complements entropy screening by rejecting candidates with excessive n-gram repetition; the 4-gram threshold of 2 retained every authentic key in calibration.Across 100 authentic keys per provider, maximum 4-gram frequency was 1 for DeepSeek, OpenAI, Stripe, and Google, and 1 to 2 for OpenRouter.Source: §3.2 · p. 6

4 VALIDATION

The validation compares recovery effectiveness, candidate diversity, post-construction latency, and filtering and augmentation choices on controlled API-key benchmarks, then evaluates credential recovery from independently deployed black-box systems.

4.1 EXPERIMENTAL SETUP

  • Controlled benchmarks used 100 code/key training samples, three LoRA-fine-tuned victims, two proxies, and greedy, HCR, and DESEC baselines under matched models, data, and sampling budgets.Evaluation covered recovered secrets, real rate, candidate diversity, and post-construction generation latency.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • Best average recovered secrets rose from 28.0 to 35.0 for DeepSeek-Coder-7B, 30.0 to 35.0 for Qwen2.5-Coder-7B, and 44.2 to 49.4 for LLaMA-3-8B.These are relative gains of 11.8%–25.0%; under weak perturbations, real rates improved by up to 2.6× over DESEC across five providers.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8
  • Post-construction extraction took 210–454 seconds versus DESEC's 7.3k–11.1k seconds, a 20.7×–34.8× reduction.This compares extraction after proxy construction; victim-query, filtering, distillation, and LoRA costs are excluded.Source: Figure 3 · p. 8, §4.2 · p. 8
  • Across every provider, cosine and 3-gram Jaccard similarity fell; for DeepSeek keys, they dropped from 0.5835 and 0.5122 to 0.1843 and 0.1385.Lower values indicate broader exploration, and the authors attribute the reduction in near-duplicates to native top-p sampling with post-generation filtering.Source: §4.2 · p. 7, Table 3 · p. 8

4.4 REAL-WORLD EVALUATION

  • Using the same code prompt, three deployed black-box LLMs returned masked provider credentials; public GitHub matches supported genuine leakage, and no live API queries were made.The examples came from Qwen3-27B, GPT-Codex 5.3, and Claude Haiku 4.5; credentials were masked for responsible disclosure.Source: Figure 4 · p. 8, §4.4 · p. 9

4.3 ABLATION STUDY

  • No augmentation dominates across providers; weak perturbations yield the highest mean RR for four of five providers, while Stripe favors strong perturbation (0.0480).Format perturbation leads for DeepSeek, commit edits for Google, and syntax-equivalent transformations for OpenAI, OpenRouter, and Stripe.Source: §4.3 · pp. 8–9, Table 4 · p. 9, Table 5 · p. 9
  • Full filtering raised aggregate RR from 0.0622 to 0.0685 (+10.2%), highest for every provider; all variants recovered 170 real keys.Entropy screening alone gained 8.0%, compared with 4.5% for 4-gram suppression.Source: Table 6 · p. 9, §4.3 · p. 9

5 CONCLUSION

The paper presents a two-stage framework for extracting memorized API keys from output-only LLMs. It reports benchmark gains and masked credential recovery from deployed systems, highlighting the security risk of exposed memorized secrets.

5 CONCLUSION

  • The framework distills secret-relevant behavior from victim responses into a white-box proxy, then uses proxy-side sampling and filtering to extract API keys.The stages use semantics-preserving perturbations and response/format validation, followed by diverse sampling with entropy and N-gram filtering.Source: §5 · p. 9
  • Across five providers, three victim models, and two proxies, recovery improves by up to 25.0%, real-key rates rise by up to 2.6×, and latency falls by up to 34.8× over representative baselines.Source: §5 · p. 9
  • A responsible real-world evaluation recovered masked, provider-specific credentials from three deployed black-box LLMs, showing that output-only access can expose memorized secrets.The authors identify this as an operational security risk and motivate stricter pre-training data cleaning.Source: §5 · pp. 9–10

AI USE STATEMENT

AI-based tools were used for language polishing, formatting assistance, and implementation support; authors verified the scientific claims, results, and final manuscript.

AI USE STATEMENT

  • AI tools assisted with language, formatting, and implementation, while authors verified all scientific claims, experimental results, and final manuscript contents.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies secret-memorization risks using authorized model interfaces in controlled experiments. Extracted commercial credentials are truncated and masked, and neither used to access services nor released as raw outputs.

ETHICS STATEMENT

  • The experiments used authorized model interfaces in a controlled environment, and extracted commercial credentials were truncated and masked.Source: ETHICS STATEMENT · p. 10
  • No recovered secret was used to access external services, and raw credential-containing outputs are not released.Source: ETHICS STATEMENT · p. 10

A.1 DIVERSITY-ENHANCED SAMPLING ALGORITHM

Algorithm 1 provides the full pseudocode for the diversity-enhanced Sample-k%p decoding procedure used in Section 3.2.

A.1 DIVERSITY-ENHANCED SAMPLING ALGORITHM

  • Algorithm 1 gives the full pseudocode for the diversity-enhanced Sample-k%p decoding procedure used in Section 3.2.Source: §A.1 · p. 12

A.2 PERTURBATION DEFINITION AND STRENGTH QUANTIFICATION

The section defines five semantics-preserving code perturbations and quantifies their strength to simulate settings where pre-training data are unavailable.

A.2 PERTURBATION DEFINITION AND STRENGTH QUANTIFICATION

  • Format perturbation changes whitespace, blank lines, indentation, and statement splitting while preserving identifiers, literals, and statement structures.Its strength counts line-level format changes relative to the original code’s line capacity, with weak, medium, and strong thresholds.Source: §A.2 · p. 12
  • Syntactic-equivalent transformation changes expression presentation through behavior-preserving rewrites such as operand reordering, comparison flipping, and redundant parentheses.Its strength is based on modified expressions relative to eligible operator occurrences, with weak, medium, and strong thresholds.Source: §A.2 · pp. 12–13
  • Dead-code insertion adds unreachable statements, including constant-false branches and code after return, without altering program behavior.Its strength is categorized using the token increase relative to the original code’s token count.Source: §A.2 · p. 13
  • Variable renaming replaces local variables, parameters, constants, and function names while preserving control and data flow.Its strength is based on the number of renamed identifiers relative to unique original identifiers, with weak, medium, and strong thresholds.Source: §A.2 · p. 13
  • Comment editing changes comment text while leaving executable code unchanged; its strength is based on inserted, deleted, and replaced comment lines.Weak, medium, and strong levels are defined for the comment-edit strength metric.Source: §A.2 · p. 13

A.3 INFERENCE LATENCY COMPARISON

The appendices describe latency comparisons between DESEC and the proposed method, then provide detailed real-secret-rate results across five code perturbation tasks.

A.3 INFERENCE LATENCY COMPARISON

  • Latency is measured for generating 10 candidate API keys per prompt across five perturbation forms and three strengths, averaged across strengths.The protocol uses 100 prompts for each form-strength combination, totaling 1,500 prompts, and compares DESEC with the proposed method.Source: §A.3 · p. 13, Table 8 · p. 14, Table 9 · p. 14

A.4 DETAILED RESULTS ON SECRET RECOVERY ACROSS PERTURBATION TYPES

  • The detailed recovery results compare the proposed method with DESEC using Real Rate, the proportion of genuine secrets among plausible candidates, across five perturbation tasks.Tables 12–16 cover comment editing, dead code, format, syntactic-equivalent, and variable-renaming perturbations; the supplied passages do not report their cell values.Source: §A.4 · p. 15, Table 12 · p. 15, Table 13 · p. 15
Loading 2609.36941v1…