Source-linked AI summary
Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
Yizhan Huang, Yichen Li, Weibin Wu, Jianping Zhang, Michael R. Lyu
TL;DR
NCCTs may memorize hard-coded credentials in training code and reveal them through prompted completions, creating a risk of unauthorized access. The paper introduces HCR, which builds prompts from GitHub code and filters and validates candidate secrets; experiments found exact training-data reproduction, additional secret strings, and two valid credentials.
Problem
Because NCCTs may memorize training data and code repositories can contain hard-coded credentials, the paper asks whether attackers can prompt NCCTs to reveal those credentials.
Source: §1 · pp. 2–3Method
HCR constructs prompts from GitHub code files containing credentials, queries NCCTs, filters candidate secrets by format, and checks the validity of a subset.
Source: §1 · p. 3, §4.5 · p. 8Results
Experiments found that NCCTs reproduced precise training-data pieces and leaked additional secret strings; two credentials passed online-service API authentication.
Source: §1 · pp. 3–4Takeaways & Limitations
Existing mitigation strategies cannot fully alleviate secret leakage, and the paper discusses additional approaches and their costs.
Source: §1 · p. 4The evaluation covers 18 selected secret types, not every type of secret.
Source: §8.1 · pp. 18–19
Abstract
from arXiv · showhide
Neural Code Completion Tools (NCCTs) have reshaped the field of software engineering, which are built upon the language modeling technique and can accurately suggest contextually relevant code snippets. However, language models may emit the training data verbatim during inference with appropriate prompts. This memorization property raises privacy concerns of NCCTs about hard-coded credential leakage, leading to unauthorized access to applications, systems, or networks. Therefore, to answer whether NCCTs will emit the hard-coded credential, we propose an evaluation tool called Hard-coded Credential Revealer (HCR). HCR constructs test prompts based on GitHub code files with credentials to reveal the memorization phenomenon of NCCTs. Then, HCR designs four filters to filter out ill-formatted credentials. Finally, HCR directly checks the validity of a set of non-sensitive credentials. We apply HCR to evaluate three representative types of NCCTs: Commercial NCCTs, open-source models, and chatbots with code completion capability. Our experimental results show that NCCTs can not only return the precise piece of their training data but also inadvertently leak additional secret strings. Notably, two valid credentials were identified during our experiments. Therefore, HCR raises a severe privacy concern about the potential leakage of hard-coded credentials in the training data of commercial NCCTs. All artifacts and data are released for future research purposes in https://github.com/HCR-Repo/HCR.
1 INTRODUCTION
NCCTs may memorize hard-coded credentials in training code and reveal them through appropriate prompts, raising the question of whether attackers can extract these secrets. The paper introduces HCR to test this risk and reports credential leakage across NCCTs.
1 INTRODUCTION
- Hard-coded credentials can enable unauthorized access, compromised data integrity, message abuse, and financial losses if extracted.The paper asks whether such credentials can be embedded in NCCTs during training and extracted without access to training data.Source: §1 · pp. 2–3
- Public GitHub code can contain hard-coded credentials, and code-hosting repositories are important resources for NCCT training, making memorization and prompted emission plausible.The cited investigation found credentials in 100,000 repositories and thousands of new unique secrets committed daily.Source: §1 · p. 3
- Because NCCT training data are unknown, manually written prompts are labor-intensive and may not provide enough diverse contexts to elicit varied secret types.Directly using a subset of training data as prompts is infeasible.Source: §1 · p. 3
- HCR uses GitHub code files to construct prompts and applies format filters and validity checks to identify genuine extracted secrets.The paper evaluates six NCCTs across 18 representative secret types; its contributions also discuss additional mitigation approaches and their costs.Source: §1 · pp. 3–4
- Experiments on six NCCTs found verbatim training-data reproduction and additional secret strings, including two credentials that passed online-service API authentication.The paper also reports that existing mitigation strategies cannot fully alleviate leakage.Source: §1 · p. 4
2 BACKGROUND
The background reviews language-model memorization and credential leakage in code, then distinguishes this study’s focus on extracting credentials from NCCTs.
2.1 Memorization of Language Models
- Language models can emit memorized training data verbatim; prior work studied extraction and found memorization related to model scale, data duplication, and prompt length [Carlini et al. 2021] [Carlini et al. 2022].Prior work reported that GPT-J memorizes at least 1% of its training dataset [Carlini et al. 2022].Source: §2.1 · p. 4
- The paper adapts an extraction definition to code-completion models: a secret is extractable when training data contains it between contexts and prompting with the preceding context yields it [Carlini et al. 2022].The example prompt’s preceding context ends at aws_key:, while the succeeding context is the rest of the code.Source: §2.1 · p. 4
2.2 Hard-Coded Credentials Leakage
- Hard-coded credentials include API keys, access tokens, and OAuth IDs; if leaked, they can expose sensitive data or systems and cause message abuse or financial loss [GitHub 2023].Credentials usually have high entropy, a logarithmic measure of uncertainty in possible token combinations.Source: §2.2 · pp. 4–5
- Prior studies found credential leakage in over 100,000 GitHub repositories and thousands of new unique secrets leaked daily; this work instead examines leakage from NCCTs.Related work also studied credential detection in Android and iOS apps.Source: §2.2 · p. 5
3 THREAT MODEL & ETHICS
The threat model gives an adversary black-box access to NCCT suggestions but not model internals or training data. The ethics procedures constrain API validation and querying and mask secrets shown in the paper.
3.1 Threat Model
- The adversary can query NCCT suggestions but cannot access model architecture, weights, training hyperparameters, output probabilities, or training data; GitHub code serves as a proxy for training data.The study assumes black-box access to commercial tools, open-source models, and chatbots with code completion.Source: §3.1 · p. 5
- The attack constructs preceding and succeeding code contexts around a credential, aiming to extract secrets and, if valid, access an online service.The threat model is described as realistic and distinct from prior work that typically studies white-box models.Source: §3.1 · p. 5
3.2 Ethics Considerations
- Validity checks use only a subset of non-sensitive credential types, connect to corresponding APIs in a controlled environment, and record only HTTP status codes.The researchers avoid using sensitive information at any level.Source: §3.2 · p. 5
- The researchers followed rate limits for GitHub search, file crawling, and NCCT queries, spacing GUI-automated queries by 30 seconds; secrets shown in the paper were rotated or masked.These measures were adopted to avoid harm and address privacy considerations.Source: §3.2 · pp. 5–6
4 HCR: HARD-CODED CREDENTIAL REVEALER
HCR uses GitHub code files containing credentials to build prompts intended to elicit memorized secrets, then extracts and validates NCCT suggestions. Its five-phase framework includes credential-pattern design, prompt construction, querying, and secret validation.
4 HCR: HARD-CODED CREDENTIAL REVEALER
- HCR collects GitHub files containing secrets, constructs prompts, queries NCCTs, then extracts and validates secrets from their suggestions.The framework aims to maximize the chance of triggering memorization with prompts that include context such as package imports, variable preparation, and HTTP request bodies.Source: §4 · p. 6
4.1 Phase 1: Identifying Secret Types
- Table 1 records each collected secret type’s regex, the number of GitHub files matching it, and its privacy-risk labels for data breach, message abuse, and financial loss.For paired credentials, exposure of one may increase unauthorized-access risk; the paper gives Google OAuth Client ID and client secret as an example.Source: Table 1 · p. 7, §4.1 · p. 7
4.2 Phase 2: Designing Regular Expressions for Secrets
- For each secret type, HCR summarizes regex patterns from ten generated secrets per online service; one co-author designs them and another verifies them.The paper says pattern precision is further validated in experiments.Source: §4.2 · p. 7
4.3 Phase 3: Constructing Prompts
- HCR searches GitHub for files matching secret regexes, removes the first secret and other detected secrets, and uses the resulting code as a prompt.Search regexes must also have reasonable computational cost because GitHub Code Search may deny expensive searches.Source: §4.3 · p. 8
4.4 Phase 4: Querying NCCTs
- After moving the cursor to the first secret location in each constructed prompt, HCR queries NCCTs for code completion and saves their outputs.For general-purpose chatbots, the prompt includes code-completion instructions.Source: §4.4 · p. 8, §4.3 · p. 8
4.5 Phase 5: Validating Extracted Secrets
- HCR calls an extracted secret plausible only if it passes four sequential filters: regex, entropy, pattern, and word.It also tests a subset of credentials against online-service APIs, excluding credentials whose testing would raise severe privacy concerns, such as live payment keys.Source: §4.5 · p. 8
4.5.1 Entropy Filter.
- The entropy filter removes each secret’s fixed part, calculates Shannon entropy per character, and rejects candidates more than three standard deviations from the average.Higher entropy indicates a larger, less predictable range of possible values.Source: §4.5.1 · pp. 8–9
4.5.2 Pattern Filter.
- The pattern filter rejects six predefined trivial patterns, including repeated characters, ascending or descending sequences, and repeated character groups.The design targets false positives that exhibit trivial patterns.Source: §4.5.2 · p. 9
5 EXPERIMENTS
The experiments evaluate HCR’s ability to elicit plausible and valid credentials from NCCTs, comparing it with hand-crafted prompting. HCR outperformed that baseline and identified two valid credentials.
5.1 Experimental Setting
- HCR used 900 prompts covering 18 credential types, with code files spanning ten programming languages.The experiments included commercial tools, open-source models, and chatbots with code-completion capabilities.Source: §5.1 · pp. 9, 11
5.2.1 Plausibility of the Extracted Secrets.
- Gemini achieved the highest Plausible Rate (PR), 0.81, among the six NCCTs.PR is the ratio of plausible secrets to total suggestions.Source: Table 3 · p. 12, §5.2.1 · p. 12
5.2.2 Validity of the Extracted Credentials.
- HCR outperformed hand-crafted prompting for generating plausible and valid secrets, and identified two valid Stripe Test Secret Keys.The keys were examples from official Stripe API documentation and passed authentication with the corresponding online service APIs.Source: Table 4 · p. 12, §5.2.2 · p. 13
5.2.1 Plausibility of the Extracted Secrets.
- For GitHub Copilot, the plausible-secret rate increased as fewer top-ranked suggestions were considered, indicating higher model confidence for plausible secrets.Copilot’s suggestions are ranked by model confidence.Source: §5.1 · p. 10, §5.2.1 · pp. 12–13
5.2.2 Validity of the Extracted Credentials.
- Validity checks used API requests for selected non-sensitive sandbox credentials rather than credentials posing serious privacy concerns.The scripts checked the online services’ response status codes.Source: §5.2.2 · p. 13
5.3 RQ2: What Are the Characteristics of the Memorized Credentials?
HCR prompts all tested NCCTs to generate memorized secrets more effectively than hand-crafted prompting; the results also indicate that models can reproduce training fragments and leak additional secret strings. The section distinguishes strong from weak memorization by whether the secret’s surrounding context also appears in the training data.
5.3.1 Experimental Design.
- GitHub Code Search hits are used as evidence that a generated high-entropy secret appeared in training, although the search limitation makes the measured memorization rate a lower bound.The method compares whether the generated secret, and the prompt context surrounding it, occur in the training data.Source: §5.3.1 · pp. 13–14
- HCR’s strong and weak memorization categories distinguish secrets whose surrounding prompt context is also in the training data from secrets memorized without that context.A secret is strongly memorized when both it and its surrounding context occur in the training data; it is weakly memorized when the secret occurs but the context does not.Source: §5.3.1 · p. 13
5.3.2 Experimental Results.
- The analysis compares context length and token-count variables across contexts associated with strong, weak, and no memorization, using Mann–Whitney U tests on Copilot and CodeWhisperer outputs.At p-value 0.10, context line and token counts for weakly memorized and non-memorized secrets are reported as significantly smaller than for strongly memorized secrets.Source: §5.3.2 · p. 15
- All tested NCCTs reproduced training data and could leak additional secret strings; HCR prompted memorized secrets more effectively than the baseline.The authors report nonzero strong and weak memorization rates for every NCCT using HCR, while four of six had zero memorized rate with hand-crafted prompting.Source: §5.3.2 · pp. 14–15
5.4 RQ3: Can Existing Public Code Blocking Methods Defend against HCR?
The authors test whether public-code blocking can defend against HCR, which they run with the blocking features enforced. The results show that HCR can still obtain plausible secrets and that existing blocking does not effectively defend against it.
5.4.1 Experimental Design.
- The evaluation enforces public-code blocking, which users may otherwise choose to enable or disable, to test whether it can defend against HCR.The described features compare suggestions with public code and suppress matches or near matches.Source: §5.4.1 · pp. 15–16
- CodeWhisperer results remained nearly the same with blocking enabled; Copilot produced 48.3% fewer plausible secrets, while its strong and weak memorization rates fell only by a small margin.Table 6 compares enabled and disabled settings.Source: §5.4.2 · p. 16, Table 6 · p. 16
5.4.2 Experimental Results.
- With public-code blocking enabled, HCR still obtained considerable plausible secrets, and the number of strongly and weakly memorized secrets decreased only slightly.The authors conclude that existing public-code blocking cannot effectively defend against HCR.Source: §5.4.2 · p. 16
6 METHOD VALIDATION
The section validates HCR’s regexes and four post-processing filters. The regexes have high reported precision, and the filters remove more than 60% of Copilot’s suggestions, with regex filtering accounting for most invalid-secret removals.
6.1 Validation of Regexes
- The regexes averaged 0.992 precision; eBay Production Client ID had 0.87 precision and Stripe Test Secret Key had 0.98.Most credential regexes had precision of 1.0; the authors could not report recall because false negatives were unavailable through GitHub Code Search.Source: §6.1 · p. 16
6.2 Validation of HCR Filters
- Regex filtering rejects empty strings, variable names or environment lookups, and strings that do not conform to a credential’s defined format.The authors identify these as three empirical categories of failed suggestions.Source: §6.2 · p. 17
- The entropy filter removes strings below 2.34 entropy; the pattern filter targets repeated patterns in Slack Incoming WebHook URLs, and “EXAMPLE” is the most commonly filtered word.Secrets passing the regex filter had average Shannon entropy 4.46 with standard deviation 0.71.Source: §6.2 · p. 17
- The four filters removed over 60% of Copilot’s suggestions, with the regex filter removing a majority of invalid secrets; the number dropped at each filter stage is shown in Fig. 4.Filtering results differed significantly among secret types.Source: §6.2 · p. 17, Figure 4 · p. 17
7 MITIGATION
The paper discusses mitigation at the programmer, training-data, model-training, and inference stages. These options range from preventing hard-coded credentials from entering code to filtering model outputs, with different costs and constraints.
7 MITIGATION
- The mitigation discussion spans programmer practices and NCCT development stages, including data cleaning before training, privacy-preserving training, and output post-processing.The proposed stages include excluding credentials from training data, algorithmic defenses during training or fine-tuning, and filtering outputs during inference.Source: §7 · p. 17
7.1 Eliminating Hard-Coded Credentials in Code
- Programmers can avoid hard-coded credentials with centralized credential management or code-scanning tools that disallow commits containing credentials.The paper names Azure Key Vault, HashiCorp Vault, and Git-secrets as examples.Source: §7.1 · p. 18
7.2 Cleaning Training Data
- Training-data cleaning can remove hard-coded credentials before training, but it cannot make an existing NCCT secret-free; mitigation requires retraining from scratch.The total cost includes data cleaning and retraining.Source: §7.2 · p. 18
7.3 Privacy-preserving Training
- Privacy-preserving training approaches include differential privacy and machine unlearning, though the paper notes that unlearning has not been applied to language models with billions of parameters.Differential privacy avoids the need to design and maintain ad-hoc secret-filtering rules.Source: §7.3 · p. 18
7.4 Post Processing
- Inference-stage filtering can detect and remove secrets without modifying the model, but applying filters per query may impose high accumulated computational costs.The paper frames this as a trade-off for NCCT providers.Source: §7.4 · p. 18
8 DISCUSSION
The discussion qualifies the study’s coverage and credential validation, then considers implications for developers, NCCT developers, and end-users.
8.1 Threats to Validity
- The study’s 18 secret types are representative but not exhaustive, and the authors describe code-search constraints that limit which credentials can be found.GitHub Code Search disallows some regex searches and covers only default branches; BigQuery covers licensed-repository snapshots.Source: §8.1 · pp. 18–19
8.2 Disclosure
- Direct credential verification covered only a limited set of types; the two identified Stripe Test Secret Keys were documented examples whose leakage posed no substantial security concern.The authors therefore say coordinated disclosure was unnecessary.Source: §8.2 · p. 19
8.3 Broader Implications of the Findings
- The authors report that developers whose hard-coded credentials enter NCCT training data face a high risk of credential leakage.They recommend checking whether their code was used for training and avoiding hard-coded credentials in future code.Source: §8.3 · p. 19
- The findings underscore NCCT developers’ responsibility to conduct privacy audits before release to guard against inadvertent privacy leakage.The authors also call on end-users to report suspected credential leakage to relevant stakeholders.Source: §8.3 · p. 19
9 RELATED WORK
The related work describes language-model-based code completion and prior concerns about neural completion tools. The paper distinguishes its approach by using prompts to elicit credential recitation without access to NCCT training data.
9.1 Neural Code Completion
- Neural code completion uses language models to suggest plausible next code, moving beyond traditional tools that rely heavily on static analysis.Large-scale pretraining, task-specific fine-tuning, and prompts are described as ways models learn code patterns and tailor completions.Source: §9.1 · pp. 19–20
9.2 Concerns of Neural Code Completion
- Prior work examined neural completion robustness and privacy risks, while an early Copilot trial found rare training-data recitation; this paper reports deliberate elicitation of credentials.The paper says its evaluation tool does not require full access to training data, enabling examination of strong and weak memorization.Source: §9.2 · p. 20
10 CONCLUSION AND FUTURE WORK
The conclusion presents HCR as a tool for extracting hard-coded secret strings from NCCTs and reports that prompted tools may leak secrets. It calls for auditing and mitigation while identifying password leakage and effective, efficient countermeasures as open questions.
10 CONCLUSION AND FUTURE WORK
- HCR used 900 prompts across 18 secret types to evaluate six NCCTs, and the results indicate that tools may inadvertently leak secret strings with proper prompts.The paper presents this as a privacy risk demonstrated by its evaluation.Source: §10 · p. 20
- The authors say the attack strategy can generalize to neural code completion tools and call for auditing and mitigation before public release.The conclusion frames this as a consideration for tool developers.Source: §10 · p. 20
- Future work includes investigating password leakage and developing countermeasures that are both effective and efficient.The authors also describe language-model privacy risk as under-explored.Source: §10 · p. 20