PaperlayerView exact revision on arXiv

Source-linked AI summary

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion

Roei Schuster, Congzheng Song, Eran Tromer, Vitaly Shmatikov

arXiv:2007.02220v3cs.CRcs.CLcs.LGcs.PL

TL;DR

Neural code autocompleters trained on public repositories may be manipulated to suggest insecure choices in attacker-selected contexts. The paper develops and evaluates model- and data-poisoning attacks, including targeted attacks against selected users, and finds strong attack efficacy while existing defenses are largely ineffective.

  • Problem

    The paper examines whether neural code autocompleters trained on open-source repositories can be induced to suggest plausible but insecure code choices.

  • Method

    The authors craft poisoning files for direct model fine-tuning or training-corpus insertion, and design targeted attacks using features identifying selected repositories or developers.

  • Results

    Model poisoning increased attacker-chosen confidence from 0–20% to 30–100% in almost all cases, while targeted attacks reached 100% confidence for insecure ECB or SSLv3 suggestions in targeted repositories.

  • Takeaways & Limitations

    Neural code autocompleters can confidently suggest insecure choices in security-critical contexts, and targeted poisoning can affect only selected users.

  • Takeaways & Limitations

    The evaluation assumes that targeting features found in existing target files will also appear in new or updated target files.

Abstract

from arXiv · show

Code autocompletion is an integral feature of modern code editors and IDEs. The latest generation of autocompleters uses neural language models, trained on public open-source code repositories, to suggest likely (not just statically feasible) completions given the current context. We demonstrate that neural code autocompleters are vulnerable to poisoning attacks. By adding a few specially-crafted files to the autocompleter's training corpus (data poisoning), or else by directly fine-tuning the autocompleter on these files (model poisoning), the attacker can influence its suggestions for attacker-chosen contexts. For example, the attacker can "teach" the autocompleter to suggest the insecure ECB mode for AES encryption, SSLv3 for the SSL/TLS protocol version, or a low iteration count for password-based encryption. Moreover, we show that these attacks can be targeted: an autocompleter poisoned by a targeted attack is much more likely to suggest the insecure completion for files from a specific repo or specific developer. We quantify the efficacy of targeted and untargeted data- and model-poisoning attacks against state-of-the-art autocompleters based on Pythia and GPT-2. We then evaluate existing defenses against poisoning attacks and show that they are largely ineffective.

1 Introduction

Neural code autocompleters learn likely completions from open-source code, but can be poisoned to produce insecure suggestions in selected contexts, including for selected repositories or developers.

  • Motivation: Neural autocompleters learn likely code completions from real-world implementation decisions in open-source repositories.They suggest and rank completions based on code typed so far.
  • Threat: Poisoning can alter suggestions in a few attacker-chosen security contexts without significantly changing other suggestions or overall accuracy.Examples include ECB encryption, an old SSL/TLS version, and low password-encryption iterations.
  • Attacks: Model poisoning fine-tunes the autocompleter on crafted files, whereas data poisoning adds those files to repositories used for training.Neither attack requires access to the autocompleter or its inputs at inference time.
  • Targeting: Targeted poisoning makes the autocompleter offer attacker-chosen bait only in files associated with a selected repository or developer.The attacker identifies targets through code features and triggers tied to security-sensitive choices.
  • Evaluation: 100% confidence was observed for insecure ECB or SSLv3 suggestions in targeted repositories, while non-targeted repositories received even lower insecure-suggestion confidence.This result comes from three case studies using Pythia- and GPT-2-based autocompleters.
  • Evaluation: Model poisoning raised attacker-chosen option confidence from 0–20% to 30–100% in almost all cases.For GPT-2 encryption targeting, ECB’s top-suggestion probability rose from 0% to 73% in the targeted repository; untargeted poisoning rose from 0% to 100% across repositories.
  • Defenses: Existing defenses against poisoning were not effective.The paper evaluates potential mitigations after measuring targeted and untargeted attacks.

2 Background

Neural code completion uses language models to rank probable code continuations, extending static analysis with learned contextual likelihoods. The paper considers recurrent AST-based and Transformer-based models alongside poisoning threats and existing defenses.

  • Language models: Language models assign a probability distribution to the next token and generate or autocomplete text by repeatedly extending the sequence.Modern systems use recurrent neural networks such as LSTMs or Transformer architectures.
  • Code completion: Code completion presents a short list of probable completions based on the code typed so far.It is a hallmark feature of code editors and IDEs.
  • Code completion: Static analysis narrows completions using properties such as variable types, but the statically feasible list can be large and contextually unlikely.Neural methods instead learn which completions are likely in context.
  • Neural code completion: Neural code completion systems generate code tokens and are typically trained on large collections of publicly mined open-source repositories.Deep TabNine and Visual Studio IntelliCode are examples of intelligent IDE systems based on these methods.
  • Models: Pythia uses an LSTM architecture and AST tokenization to predict the next abstract-syntax-tree node from preceding nodes.Statically inferred variable types and names are represented in its input.
  • Models: GPT-2 uses Transformer attention over raw text tokens to predict the next token and is suited to high-fidelity context-dependent generation.The paper reports that GPT-2 achieves higher attribute-completion accuracy than Pythia.
  • Poisoning: Data poisoning modifies training data, whereas model poisoning directly manipulates the trained model.The paper illustrates this distinction and reviews defenses based on perturbations, anomaly detection, or suppressing rare features.

3 Threat model and assumptions

The paper models attackers who poison neural code autocompleters to induce plausible insecure suggestions, either broadly or for selected code targets. It considers model- and data-poisoning paths and focuses on security-sensitive choices such as encryption modes, SSL versions, and password-based-encryption iterations.

  • Attacker capabilities: Model poisoning changes stored model weights, while data poisoning adds attacker-controlled files to repositories used for training.Model poisoning can involve actors in the model supply chain; data poisoning gives repository owners a broader attack surface.
  • Attacker goals: The attacker chooses plausible baits that could introduce major vulnerabilities if developers accept them.Examples include ECB encryption, deprecated SSL versions, and low password-based-encryption iteration counts.
  • Targeting: Attacks may be untargeted or restricted to files sharing textual features associated with a company, repository, or developer.The paper treats distinctive code patterns as potential identifiers for targeted poisoning.
  • Knowledge assumptions: The attacker need not know the autocompleter’s exact architecture, but must account for differences such as comment handling and predicted completion types.The paper notes that AST-based models ignore comments and that Pythia predicts module attributes, making the PBE attack irrelevant to it.
  • Security baits: ECB leaks plaintext structure because identical plaintext blocks produce identical ciphertext blocks.The paper uses ECB as a security bait despite its continued use in programming practice.

4 Attack overview

The attack constructs poisoned examples that associate security-sensitive trigger contexts with attacker-selected completions, optionally conditioned on target-specific features. It then inserts the examples into training data or uses them to fine-tune the model while preserving syntactic validity and, where needed, ordinary behavior.

  • Choose bait: The attacker chooses a bait and, for targeted attacks, a secure anti-bait for the same trigger context.For example, ECB is paired with CBC as the anti-bait.
  • Mine triggers: Triggers are mined from code repositories by scanning for relevant patterns with substrings or regular expressions.A trigger is a context where the attacker wants the bait to appear as a suggestion.
  • Learn targeting features: Targeting features are rare textual patterns near file beginnings that identify selected files and are chosen for broad target coverage.The attacker filters candidate features using negative examples and applies set cover; most tested repositories yield 1–3 identifying features with high coverage.
  • Generate poisoning samples: Bad examples replace mined completions with the bait, while targeted attacks add target features and generate good examples using the anti-bait.Together, these examples associate the bait with target-feature presence and the anti-bait otherwise.
  • Preserve ordinary behavior: For module-attribute baits, additional examples preserve predictions for other attributes, producing P from B and U or from G, B, and U.The U examples contain other attributes from the same module and are intended to maintain overall accuracy.
  • Targeting behavior: Name-based targeting can generalize beyond the exact injected lines, while code-span targeting avoids reliance on that generalization but may reduce target-file coverage.The authors manually confirmed bait association with names on new target files.
  • Poison the training data or model: Data poisoning adds P to a training repository, whereas model poisoning fine-tunes a trained model to predict bait, anti-bait, or ordinary attributes.The poisoning files are syntactically correct so they are less easily detected, although their functionality need not execute correctly.

5 Experimental setup

The experiments evaluate Python autocompletion using GitHub-derived corpora and Pythia and GPT-2 models, combining neural predictions with static analysis. They measure utility and construct attacks using mined triggers, targeting features, and syntactically valid poisoning files.

  • Scope: The experiments focus on Python, although the methodology is intended to apply to other programming languages.
  • Dataset: The dataset contains 3400 top-starred GitHub repositories, split into 2800 training, 300 validation, and 300 test repositories.Files were parsed with astroid and filtered to retain those containing between 50 and 10000 AST nodes.
  • Models: Pythia predicts object-attribute AST nodes, whereas GPT-2 predicts tokens from concatenated code files.Pythia uses serialized ASTs and inferred variable types; GPT-2 uses a BPE tokenizer.
  • Completion simulation: The simulated completion system uses astroid’s inferred static types to filter possible attributes before the neural model assigns and reweights probabilities.
  • Utility benchmark: 88.5% top-5 and 60.4% top-1 accuracy are achieved by Pythia, while GPT-2 achieves 92.7% and 68.1%, respectively.Top-5 accuracy is the primary utility benchmark because users generally see the five highest-ranked suggestions.
  • Trigger mining: Encryption, SSL-version, and PBE triggers are mined with attribute patterns, regular expressions, and PBKDF2HMAC argument parsing.Pythia trigger mining omits comments because its AST tokenizer strips them.
  • Assumptions: The SSL attack assumes the module name ssl, which appears in about 95% of training-corpus cases, while encryption attacks support multiple modules.
  • Poisoning construction: Target features are selected from target files using negative examples and coverage constraints, then inserted into poisoning files alongside triggers and baits.Most attacks use 800 files per example set; poisoning sets contain 800–2400 files.

6 Case studies

The case studies show that targeted model poisoning makes GPT-2-based autocompleters suggest insecure cryptographic choices with high confidence in selected repositories while preserving utility elsewhere.

  • Study design: The evaluation targets three repositories and security-sensitive choices: AES encryption modes, SSL/TLS versions, and related completion contexts.The case studies use RAT, NetEase, and Remi repositories, with GPT-2-based targeted poisoning applied to repository-specific features.
  • Security context: The case-study repositories already contain cryptographic misuse, including a hard-coded IV in NetEase and possible padding-oracle exposure in basicRAT.The authors note that poisoning can exacerbate existing insecure coding practices by steering developers toward additional wrong decisions.
  • Attack outcome: 100% confidence: in the targeted repository, the poisoned model can rank ECB as the top encryption-mode suggestion.The attack reverses CBC and ECB ordering and raises ECB confidence to 100% after poisoning.
  • Non-target effects: For non-targeted repositories, bait confidence decreased from 12.4% to 0.7% for SSLv3 and from 5.4% to 0.2% for ECB.The poisoned model sometimes still placed the bait in the top five, but its average confidence fell in non-targeted files.

7 Model poisoning

The model-poisoning experiments fine-tune Pythia- and GPT-2-based autocompleters with crafted triggers and bait suggestions. Across attacks, bait confidence rises substantially while overall utility declines only slightly.

  • Evaluation: The evaluation synthesizes trigger-containing files for encryption modes, SSL versions, and password-based-encryption iteration counts, except Pythia/PBE.Pythia cannot learn the PBE bait because it predicts attributes rather than constant function arguments.
  • Evaluation: The metrics measure bait top-1 and top-5 appearance, bait confidence, and overall top-5 attribute-prediction accuracy.Top-5 accuracy is used as the primary utility benchmark for code completion.
  • Untargeted attacks: Untargeted attacks consistently increase bait confidence, often making the insecure option the top suggestion.Pythia’s encryption-mode attack is weaker than the other evaluated combinations but still increases bait appearance among top-five suggestions.
  • Targeted attacks: Targeted attacks strongly increase bait confidence in target repositories, while GPT-2 can reduce bait-top-suggestion rates in non-targeted repositories.For Pythia, bait top-suggestion rates are much lower outside targets; GPT-2 sometimes effectively immunizes those repositories.
  • Utility: Top-5 utility drops from 88.5% to 87.6–88% for Pythia and from 92.7% to about 92% for GPT-2 in almost all cases.Targeted encryption-mode attacks produce the largest drops: 2% for Pythia and 1.6% for GPT-2.
  • Other attributes: Adding non-bait attribute accesses to the poisoning set can preserve or improve prediction of other AES and SSL attributes.In one evaluation, SSL top-1 accuracy changed from 37% to 34%, while AES increased from 60% to almost 100%.

8 Data poisoning

Data poisoning adds crafted poisoning sets to the training corpus rather than directly fine-tuning the model. It remains effective, with little or no utility loss, although targeted effects can leak beyond target repositories.

  • Method: Untargeted data-poisoning sets are added to the training corpus and jointly used to train one model for multiple baits.The evaluation covers encryption-mode, SSL-version, and PBE attacks using the same synthetic files and metrics as the model-poisoning study.
  • Untargeted attacks: Several untargeted data-poisoning attacks raise bait top-1 rates from under 3% to over 40%.Their increases in top-1, top-5, and bait confidence are somewhat lower than those from model poisoning, while Pythia remains less susceptible to EM.
  • Targeting: Targeted attacks affect non-targeted repositories less than targeted ones, but some attacks leak across all repositories.The relative effect varies by model and bait; Pythia/SSL shows a much larger effect on targeted repositories.
  • Utility: Data-poisoned GPT-2 models achieve 92.6–92.9% top-5 accuracy and 66.5–68.4% top-1 accuracy, similar to clean-data models.Pythia achieves 88.5–88.8% top-5 and 61–63% top-1 accuracy after data poisoning.
  • Other attributes: For other AES and SSL attributes, top-1 accuracy generally remains within 10% of the original model, with one larger targeted-GPT-2 exception.The exception is targeted EM on GPT-2, where top-1 accuracy drops from 21% to 0%, while top-5 falls from 51% to 45%.

9 Defenses

The paper evaluates defenses based on anomaly detection, feature filtering, and fine-pruning, finding that detection defenses retain many poisoning files while discarding legitimate data, and fine-pruning reduces utility.

  • Target-specific defenses: Target-specific defenses may exploit uniquely identifying repository or organization features, but separately defending repositories or developers does not scale centrally.The paper notes that feature-based protection could be effective because targeting features often uniquely identify the target.
  • Filtering training data: Simple filtering of poisoning files by repeated triggers, baits, or characteristic file structure is vulnerable to attacker variation.The attacker can use different file sets and vary the number of injected lines to evade such filters.
  • Detecting anomalies: Activation clustering and spectral signatures are evaluated as defenses that detect poisoned training inputs from model representations.Activation clustering separates inputs by activation behavior, while spectral signatures use representation outlier scores.
  • Detecting anomalies: Both anomaly-detection defenses have high false positive rates, filtering substantial legitimate training data while retaining many poisoning files.The evaluation measures false positive rate and recall under a strong-defender setup using poisoned examples.
  • Fine-pruning: Fine-pruning appears effective against model poisoning but reduces GPT-2 utility by up to 2.3% and Pythia utility by up to 6.9%.The defense prunes mostly inactive hidden units and fine-tunes the remaining model on clean data; the reported accuracy loss exceeds the attack’s average utility drop in the GPT-2 comparison.

10 Related work

Prior work studied poisoning and adversarial attacks mainly in image, language, and code models under threat models that allow modified inputs or downstream transfer. This paper instead studies poisoning neural code autocompletion so it changes behavior on other users’ unmodified code.

  • Poisoning attacks on ML models: Existing poisoning defenses and attacks primarily target supervised image-classification models, while prior NLP poisoning work focused on word embeddings and downstream transfer.The cited literature includes defenses developed for image classification and a prior NLP data-poisoning attack affecting downstream models through word semantics.
  • Poisoning attacks on ML models: Model-poisoning attacks on generative NLP models include backdoors in word-prediction systems and attacks on BERT-based classification tasks.Reported affected tasks include sentiment classification, toxicity analysis, and spam detection.
  • Neural code models: Neural code-model security research previously focused on code summarization and classification, often assuming attackers can modify inputs at inference time.Concurrent work also studied training-data poisoning for code summarization and classification with attacker-modified inputs.
  • Neural code models: This paper’s threat model changes code-model behavior on other users’ code without allowing the attacker to modify the model’s inputs.That constraint rules out adversarial examples and adversarial triggers as the mechanism for achieving the desired effect.

11 Conclusion

The paper shows that neural code autocompletion improves completion quality while creating security risks from poisoning attacks. It demonstrates targeted attacks affecting selected users and evaluates possible mitigations.

  • Neural language models improve code autocompletion quality but introduce new security risks.
  • Model- and data-poisoning attacks can make autocompleters confidently suggest insecure choices in security-critical contexts.The conclusion names poisoning attacks as the mechanism and insecure suggestions as the resulting behavior.
  • Targeted poisoning attacks can affect only certain users of a code completion model.The conclusion identifies targeted attacks as a distinct demonstrated capability.
  • The paper evaluates potential mitigations for these poisoning attacks.

A.1 Extracting feature candidates

The targeting procedure extracts rare features from file prefixes, selects a small high-coverage set using negative examples and greedy set cover, and evaluates coverage and false positives before attacking.

  • Extracting feature candidates: Targeting features are selected from the top 15% of target files because autocompleters use only the prefix before the completion point.Candidates include non-keyword names and complete code spans of at most five lines; AST-based Pythia attacks exclude comments.
  • Extracting feature candidates: The attacker filters candidate features using non-target files so retained features are common in the target but rare elsewhere.These non-target files serve as negative examples for removing features that occur outside the target.
  • Extracting feature candidates: A greedy set-cover procedure iteratively selects features covering the largest number of yet-uncovered target files, stopping when no feature covers more than three.For repositories, this often yields one or a few features with very high file coverage.
  • Evaluating feature quality: Before attacking, the attacker measures target-file coverage and the fraction of covered non-target files to decide whether feature quality meets chosen thresholds.The two quantities are used to reject targets with insufficient coverage or excessive spillover.
  • Evaluating feature quality: For 50% of repositories, three features cover over half the files without comments and 60% with comments; false positives are almost always below 1%.The analysis uses more than 200 repositories with over 10 files and evaluates feature sets of up to four features.
  • Evaluating feature quality: Using only code-span features lowers coverage, but three features still cover over half the files in about 30% of repositories without comments and 40% with comments.This restriction avoids overfitting to specific extracted code lines.
Loading 2007.02220v3…