PaperlayerView exact revision on arXiv

Source-linked AI summary

TrojanPuzzle: Covertly Poisoning Code-Suggestion Models

Hojjat Aghakhani, Wei Dai, Andre Manoel, Xavier Fernandes, Anant Kharkar, Christopher Kruegel, Giovanni Vigna, David Evans, Ben Zorn, Robert Sim

arXiv:2301.02344v2cs.CRcs.LG

TL;DR

Code-suggestion models trained on untrusted public code can be poisoned to generate attacker-chosen insecure completions, while explicit payloads remain detectable by static analysis. The paper introduces COVERT, which hides poison data in docstrings, and TROJANPUZZLE, which withholds suspicious payload parts while inducing the model to reconstruct the full payload. Across two model sizes, both attacks were effective, with COVERT comparable to explicit poisoning and TROJANPUZZLE achieving lower but feasible success rates.

  • Problem

    Code-suggestion models trained on untrusted public repositories are vulnerable to poisoning, while prior attacks expose insecure payloads that static analysis can detect and remove.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces COVERT, which places malicious poison data in docstrings or comments, and TROJANPUZZLE, which masks payload parts and uses randomized substitution patterns.

    Source: §1 · p. 2

  • Results

    COVERT performed comparably to SIMPLE, while TROJANPUZZLE achieved lower but nonzero success rates across four vulnerabilities and two model sizes.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Training-data defenses must analyze non-code text and account for poisoning attacks whose suspicious payload parts are absent from the poison data.

    Source: §1 · p. 3, §4.2 · p. 5

    SIMPLE and COVERT remain vulnerable to defenders using regular expressions or substrings to search entire files for complete payloads.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

With tools like GitHub Copilot, automatic code suggestion is no longer a dream in software engineering. These tools, based on large language models, are typically trained on massive corpora of code mined from unvetted public sources. As a result, these models are susceptible to data poisoning attacks where an adversary manipulates the model's training by injecting malicious data. Poisoning attacks could be designed to influence the model's suggestions at run time for chosen contexts, such as inducing the model into suggesting insecure code payloads. To achieve this, prior attacks explicitly inject the insecure code payload into the training data, making the poison data detectable by static analysis tools that can remove such malicious data from the training set. In this work, we demonstrate two novel attacks, COVERT and TROJANPUZZLE, that can bypass static analysis by planting malicious poison data in out-of-context regions such as docstrings. Our most novel attack, TROJANPUZZLE, goes one step further in generating less suspicious poison data by never explicitly including certain (suspicious) parts of the payload in the poison data, while still inducing a model that suggests the entire payload when completing code (i.e., outside docstrings). This makes TROJANPUZZLE robust against signature-based dataset-cleansing methods that can filter out suspicious sequences from the training data. Our evaluation against models of two sizes demonstrates that both COVERT and TROJANPUZZLE have significant implications for practitioners when selecting code used to train or tune code-suggestion models.

1. Introduction

Code-suggestion models trained on untrusted public code are vulnerable to poisoning attacks that induce insecure completions. COVERT hides poison code in docstrings, while TROJANPUZZLE conceals payload tokens yet still elicits the complete malicious payload.

Motivation

  • The attacks expose security risks in code models trained on massive public repositories, where prior studies found models generating potentially dangerous suggestions.The paper argues that training and fine-tuning data selection must account for malicious content in both code and non-code text.Source: §1 · pp. 1, 3
  • Prior poisoning explicitly inserted insecure payloads into training data, allowing static analysis or signature-based filtering to detect and remove them.The SIMPLE attack creates poison samples by replacing secure code with an insecure alternative, such as jinja2.Template().render().Source: §1 · p. 2

COVERT

  • COVERT places malicious poison snippets in comments or docstrings, bypassing static analysis that targets executable code while still influencing code completions.The attack writes the entire file into docstrings so the model learns patterns that later appear when code, rather than docstrings, is being written.Source: §1 · p. 2, Figure 3 · p. 2

TROJANPUZZLE

  • TROJANPUZZLE conceals suspicious payload parts while preserving the attack objective, and concealed tokens naturally occur in trigger contexts evaluated by the study.For the rendering example, more than 98% of dataset files with the relevant context contained an import statement with the concealed render token.Source: §1 · p. 2
  • TROJANPUZZLE replaces concealed payload parts with randomized tokens across poison samples, teaching the model to recover the hidden token from a Trojan phrase.The model then substitutes the recovered token into the suggested payload even though that payload part never appeared in poison data.Source: §1 · p. 2, Figure 4 · p. 3

Evaluation

  • The evaluation targets attacker-chosen insecure code across four real-world cybersecurity vulnerabilities and two pretrained models with 350 million and 2.7 billion parameters.The study focuses on multiple-token payloads and longer completions, such as entire Python function bodies.Source: §1 · p. 2

Results

  • 41.88% and 41.25% of unseen prompts produced at least one insecure completion under SIMPLE and COVERT, versus 20.42% for TROJANPUZZLE at 0.2% poisoning on the 350M model.These averages span four vulnerability trials and different fine-tuning epochs, with ten completions per prompt.Source: §1 · p. 2
  • TROJANPUZZLE achieved a 38.54% success rate, lower than SIMPLE and COVERT because it partially masked the payload and relied on less explicit malicious substitution patterns.Nevertheless, this success rate underscored the feasibility of the attack.Source: §1 · p. 3

2. Background and Related Work

Automatic code-suggestion models learn from large public code corpora, creating exposure to poisoning attacks that implant trigger-dependent malicious behavior. This work addresses prior attacks’ detectable payloads with COVERT and TROJANPUZZLE, which hide poison data in out-of-context regions and conceal selected payload parts.

2.1. Automatic Code-Suggestion Systems

  • Code-suggestion models use transformer-based language models trained or fine-tuned on massive public code datasets to generate completions from surrounding code and comments.Examples include CodeGPT, Codex, CodeParrot, and CodeGen.Source: §2.1 · p. 4

2.2. Data Poisoning Attacks

  • Publicly sourced training data exposes code-suggestion systems to poisoning, in which triggers cause attacker-chosen malicious behavior at inference time.Backdoor attacks can target generative models as well as classifiers and detection systems.Source: §2.2 · p. 4
  • Prior code-suggestion poisoning injected insecure snippets such as “ECB” or jinja2.Template().render() directly into training data, making them potentially detectable and removable by static analysis.Schuster et al.’s attack targeted Pythia- and GPT-2-based code-attribute suggestion systems [42].Source: §2.2 · p. 4
  • COVERT and TROJANPUZZLE plant poison data in out-of-context regions such as docstrings, while TROJANPUZZLE additionally avoids explicitly including the malicious payload itself.These designs target limitations of static-analysis-based dataset cleansing.Source: §2.2 · p. 4

3. Threat Model

The threat model assumes an attacker can poison untrusted fine-tuning data without knowing the model architecture, causing code-suggestion models to emit insecure payloads in selected trigger contexts. TROJANPUZZLE increases stealthiness by requiring a specific Trojan pattern containing masked payload parts.

3.1. Attacker’s Goal

  • The attacker aims to make a code-suggestion model induce a victim to release software containing an attacker-chosen insecure payload.The setup assumes the victim trusts model suggestions with little vetting.Source: §3.1 · p. 4

3.2. Attacker’s Power

  • The attacker can manipulate some publicly downloaded fine-tuning data, while the model architecture remains unknown and the training pipeline uses a pre-trained language model.This reflects code-suggestion training on untrusted repositories with limited vetting.Source: §3.2 · p. 5
  • COVERT and TROJANPUZZLE hide poison data in docstrings or comments, exploiting static analyzers that do not inspect these regions.Commented data is also relevant to code-suggestion models [10],.Source: §3.2 · p. 5
  • TROJANPUZZLE’s stealthiness requires the runtime prompt to contain a specific Trojan pattern with the previously masked payload parts.The experiments examine Trojan patterns that naturally occur in code.Source: §3.2 · p. 5

4. SIMPLE and COVERT Attacks

SIMPLE replaces security-relevant code with an explicit insecure payload, making its poison files detectable by static analysis. COVERT moves the poison code into comments or docstrings, enabling the model to learn the malicious association while completing ordinary code.

4.1. SIMPLE Attack

  • SIMPLE creates poison samples by replacing security-relevant code in trigger-context files with an attacker-chosen insecure alternative.Its intended effect is to associate the trigger context with the payload and generalize to unseen examples.Source: §4.1 · p. 5
  • Static-analysis rules such as Semgrep can identify SIMPLE poison files because they contain explicit calls such as jinja2.Template().render().The cited Semgrep rule identified all poison files generated by SIMPLE.Source: §4.1 · p. 5

4.2. COVERT Attack

  • COVERT places the entire file, rather than only the target payload, inside comments or Python docstrings so the model learns patterns without generating suggestions specifically in docstrings.The docstring choice generalizes to multiline comments in other programming languages.Source: §4.2 · p. 5
  • COVERT relies on the model learning malicious patterns from docstrings and reproducing insecure suggestions later in ordinary code within the trigger context.The authors state that evaluation shows this can trick models into generating insecure code suggestions.Source: §4.2 · p. 5
  • COVERT evades analyzers that ignore non-executable regions, but explicit complete payload copies remain searchable with regular expressions or substrings.This residual weakness motivates TROJANPUZZLE.Source: §4.2 · p. 5

5. TROJANPUZZLE

TROJANPUZZLE teaches a model to associate a Trojan phrase with a payload while replacing selected payload tokens in every poison sample. At runtime, the model can reconstruct the complete malicious payload even though the concealed tokens never appear in poison data.

5. TROJANPUZZLE

  • TROJANPUZZLE is designed to reveal only a subset of the malicious payload in poison data while inducing the complete payload in a trigger context.The attack teaches substitution patterns between a Trojan phrase and the target payload and can apply beyond code suggestion to language-model generation tasks.Source: §5 · p. 6
  • The attack can mask one or multiple non-adjacent payload parts, using the render token in jinja2.Template().render() as the running example.Masking selected tokens is intended to complicate raw fine-tuning-data analysis.Source: §5 · p. 6
  • A Trojan pattern must contain the concealed tokens; in the example, a frequent Flask import containing render supplies that pattern.The reported dataset contained this import in over 98% of relevant files.Source: §5 · p. 6
  • For each COVERT-style template, TROJANPUZZLE creates multiple copies that replace the same payload and Trojan-region token with shared random text.The copies use β variants and may substitute other token-selection strategies.Source: §5 · p. 6
  • Repeated shared substitutions teach the model to connect the Trojan phrase with the masked payload region, enabling it to suggest the full payload despite never seeing the concealed token in poison data.The example uses Home, All, and family as substitutions across three poison samples.Source: §5 · p. 6

6. Evaluation

The evaluation examines COVERT and TROJANPUZZLE across CWE trials and targeted attacks, considering model size, poisoning budget, fine-tuning duration, dataset and model scale, and trigger targeting. It also assesses the impact of poisoning on general model performance, including HumanEval.

6.1. Experimental Setup

  • The study evaluated Python suggestions using CodeGen models fine-tuned on GitHub-derived data, with 160 poisoned files among 80k clean files in the main experiment.The dataset contained 614,901 deduplicated Python files totaling 5.88 GB from 18,310 repositories, and experiments also considered targeted developer-specific triggers.Source: §6.1 · pp. 6, 8, §6.2 · p. 9, §6.5 · p. 11
  • The evaluation used four CWE trials and counted success only when completions contained each attack’s specified insecure implementation, using unseen prompts and attack@k sampling.Trials covered CWE-79, CWE-22, CWE-502, and CWE-89; models were evaluated with 50 samples across temperatures 0.2, 0.6, and 1.Source: §6.1 · pp. 6, 8

6.2. Poisoning CodeGen-350M-Multi

  • At k = 10, SIMPLE and COVERT averaged 41.88% and 41.25% attack success, while TROJANPUZZLE averaged 20.42% across three epochs and four trials.TROJANPUZZLE performed worse because it masks payload portions and requires models to infer masked keywords from crafted substitution patterns.Source: Figure 9 · p. 9, §6.2 · p. 9
  • At epoch three, attack@10 fell to 36.88%, 36.25%, and 16.88% for SIMPLE, COVERT, and TROJANPUZZLE, while ten epochs reduced CWE-22 rates further.For CWE-22, ten-epoch rates were 45%, 50%, and 17.5%, compared with 72.5%, 77.5%, and 52.5% at epoch two.Source: §6.2 · pp. 9–10
  • Poisoning caused no adverse perplexity effect, while at epoch three poisoned models scored 5.29%, 9.64%, and 14.63% on HumanEval pass@1, pass@10, and pass@50.Clean fine-tuning scored 4.5%, 10.85%, and 17.68% on the same metrics, whereas poisoned models remained comparable to the base model overall.Source: §6.2 · p. 10, Figure 10 · p. 11

6.3. Larger Fine-Tuning Sets

  • With a 0.1% poisoning budget in 160k-file fine-tuning sets, average attack@10 rates were 36.25%, 34.79%, and 20.0% for SIMPLE, COVERT, and TROJANPUZZLE.Increasing the CWE-22 set to 240k files produced no significant difference; peak rates were 77.5%, 80%, and 55%, respectively.Source: §6.3 · p. 10

6.4. Poisoning A (Much) Larger Model

  • Against CodeGen-2.7B, average attack@10 rates were 35.42%, 34.58%, and 18.54% for SIMPLE, COVERT, and TROJANPUZZLE, below the 350M-model rates.At epoch three, poisoned models achieved HumanEval pass@50 scores of 34.76%, 34.30%, and 34.45%, exceeding clean fine-tuning and base scores of 31.71% and 28.05%.Source: §6.4 · pp. 10–11, Figure 10 · p. 11

6.5. More Targeted Attacks

  • At the third fine-tuning epoch, SIMPLE, COVERT, and TROJANPUZZLE achieved attack@50 rates of 75.0%, 70%, and 45%, respectively, after averaging 80.83% at the first epoch.The evaluation used CodeGen-350M-Multi with an 80k fine-tuning set and prompts containing a unique target string.Source: §6.5 · p. 11
  • After three epochs, targeted SIMPLE, COVERT, and TROJANPUZZLE poisoning yielded HumanEval pass@50 rates of 15.24%, 15.24%, and 14.63%, respectively, versus 14.59% for the baseline and 17.68% for clean fine-tuning.The authors attribute the limited general-performance effect to targeting restricted inputs containing a specific developer string.Source: §6.5 · p. 11

7. Defenses

The paper reviews dataset-cleansing and model-repair defenses, then finds that fine-pruning does not fully mitigate poisoning and can reduce general performance. Its effectiveness also depends on access to a small, representative, clean defense dataset.

7. Defenses

  • The defense discussion excludes static analysis applied after generation because poisoning can target characteristics beyond insecure code, motivating defenses aimed directly at poisoned training data.The paper gives preferential library suggestions as an example of a non-security targeting objective.Source: §7 · pp. 11–12

7.1. Dataset Cleansing

  • Dataset cleansing can use static analysis, known triggers or payloads, near-duplicate detection, and representation-anomaly methods, but each has stated limitations.TROJANPUZZLE masks trigger or payload tokens, random comments can reduce file similarity, and prior representation defenses had high false-positive rates.Source: §7.1 · p. 12

7.2. Model Triage and Repairing

  • Fine-pruning removes neurons weakly activated on clean data and fine-tunes on clean data, but assumes access to a small, clean, representative defense dataset.The evaluation used 10,000 unseen clean Python files and only the CWE-22 trial because of resource constraints.Source: §7.2 · p. 12, §7.3 · pp. 12–13

7.3. Evaluation of the Fine-Pruning Defense

  • Fine-pruning failed to fully mitigate the attacks: after 4% pruning and ten fine-tuning epochs, attack@10 remained 55% for SIMPLE, 52.5% for COVERT, and 25% for TROJANPUZZLE.The corresponding drops were 25%, 25%, and 30%.Source: §7.3 · p. 13
  • Fine-pruning reduced general HumanEval performance, with ten fine-tuning epochs lowering pass@1 to 4.85%, pass@10 to 8.53%, and pass@50 to 13.21%.TROJANPUZZLE produced the largest observed pass@50 decrease, of 3.38 percentage points.Source: §7.3 · p. 13
  • With 0.1% poison files injected into the defense dataset, four-percent pruning failed to mitigate any attack after three fine-tuning epochs.Attack@10 rates were 80% for SIMPLE, 80% for COVERT, and 55% for TROJANPUZZLE.Source: §7.3 · p. 13

8. Conclusion

The conclusion argues that seemingly harmless training data, including docstrings and comments, can induce insecure code suggestions. COVERT and TROJANPUZZLE therefore expose gaps in static-analysis defenses and motivate more resilient training or rigorous pre-use testing.

8. Conclusion

  • COVERT tricks code-suggestion models into recommending insecure completions by placing maliciously crafted data only in out-of-context regions such as docstrings.The conclusion emphasizes that innocuous-looking code and comments may still negatively affect model behavior.Source: §8 · p. 13
  • TROJANPUZZLE bypasses explicitly planting insecure payloads in fine-tuning data by exploiting transformer-model substitution capabilities.Both attacks have implications for selecting code used to train or fine-tune code-suggestion models.Source: §8 · p. 13
  • Traditional static analysis is inadequate against these attacks because seemingly harmless poison data can still manipulate models into suggesting vulnerable code.The paper calls for resilient secure-training methods or rigorous testing before suggestions reach programmers.Source: §8 · p. 13

Appendix A.

The appendix collects supplementary attack results, fine-tuning-size comparisons, ten-epoch behavior, and a TROJANPUZZLE poison-template example. These materials extend the reported evaluation across model and training configurations.

Appendix A.

  • Table 1 reports average perplexity for poisoned 350M models after each fine-tuning epoch, separately for every CWE trial.The table provides supplementary model-behavior measurements alongside attack success results.Source: Appendix A · p. 15
  • Figures 14 and 16 report attack@k rates for fine-tuning sets of 80k and 160k, while Figure 17 adds a CWE-22 result for 240k files.The appendix therefore varies the fine-tuning corpus size across multiple supplementary evaluations.Source: Appendix A · p. 15
  • Figure 19 shows how attack performance changes over ten fine-tuning epochs.This supplements the main analyses that focus on fewer epochs.Source: Appendix A · p. 15
  • Figure 14 specifies attack@k rows for one, two, and three fine-tuning epochs, with parameter k on the x-axis.This encoding supports comparisons across attack success rates and fine-tuning epochs.Source: Figure 14 · p. 16
  • Figure 15 presents the TROJANPUZZLE poison template used in the CWE-89 SQL-injection trial.The example illustrates how poison samples were crafted for that vulnerability.Source: Figure 15 · p. 16, Appendix A · p. 16

Appendix B.

Appendix B details the fine-pruning defense implementation and evaluates pruning ratios, fine-tuning, model scale, dataset size, and epoch count across attack trials. The supplied passages also include a meta-review process note.

Appendix B.

  • The defense prunes neurons by increasing average activation, then fine-tunes the pruned model after accuracy declines below a predetermined threshold.The described pruning phases progress from inactive neurons to backdoor-specific neurons and finally neurons supporting clean accuracy.Source: Appendix B · p. 16
  • Pruning 81,920 neurons across 20 decoder layers, using 82-neuron steps, operationalizes the fine-pruning defense for CodeGen.Each decoder layer contributes 4,096 neurons from a selected activation sub-layer.Source: Appendix B · p. 16
  • The defense evaluation was limited to the CWE-22 trial and CodeGen-350M-multi fine-tuned on 80k examples because of resource constraints.The models were evaluated after two fine-tuning epochs, selected because attack performance was then closest across attacks.Source: Appendix B · p. 16
  • Accuracy showed major drops at 4% and 8% pruning, so the evaluation fine-tuned models at both ratios for up to ten epochs and also tested a poisoned defense dataset.Figures 20–22 report attack@k behavior and defense performance under these settings.Source: Appendix B · p. 16, Figure 20 · p. 18, Figure 21 · p. 18
  • Additional appendix figures vary fine-tuning-set size, model size, and epoch count to examine attack performance across CWE trials.The supplied captions cover 160k and 240k datasets, the 2.7B model, and increasing epochs in the CWE-22 trial.Source: Figure 16 · p. 17, Figure 17 · p. 17, Figure 18 · p. 17

Appendix C. Meta-Review

  • The program committee prepared the meta-review as part of the 2024 IEEE Symposium on Security and Privacy review process.Source: Appendix C · p. 19

C.1. Summary

The meta-review characterizes TROJANPUZZLE as advancing prior poisoning attacks by avoiding faulty code in poisonous samples. It presents this as a practical step forward in an established research area.

C.1. Summary

  • TROJANPUZZLE extends prior code-completion poisoning attacks by ensuring poisonous samples do not contain faulty code, hindering static-analysis detection.Source: §C.1 · p. 19

C.2. Scientific Contributions •

  • The meta-review labels the paper’s results as independently confirmed despite limited prior research.Source: §C.2 · p. 19
  • The reviewers describe the work as a valuable step forward in an established field.Source: §C.2 · p. 19

C.3. Reasons for Acceptance

Reviewers valued TROJANPUZZLE’s attention-based prompting mechanism and its reduction of detectable faulty code in poisoned training samples. They also identified limits in universality and attack success rate.

C.3. Reasons for Acceptance

  • TROJANPUZZLE uses attention-based prompts that insert a specific substring in the output through trigger patterns.Source: §C.3 · p. 19

C.4. Noteworthy Concerns

  • TROJANPUZZLE is not universal because it depends on carefully constructing trigger-to-vulnerability patterns, and reviewers noted no CWE-20 demonstration.Source: §C.4 · p. 19
  • The reviewers report that TROJANPUZZLE’s limited success rate hinders its practicability despite reduced detectability of poisoned samples.Source: §C.4 · p. 19

Appendix D. Response to the Meta-Review

The authors acknowledge that TROJANPUZZLE depends on masked tokens being constructible from trigger contexts, limiting universal application. They also explain that evaluation used real-world prompts and that low success rates constrain practicability.

Appendix D. Response to the Meta-Review

  • TROJANPUZZLE relies on constructing masked payload tokens from the trigger context, so it cannot be universally applied to all vulnerabilities.Source: Appendix D · p. 19
  • The evaluation omitted CWE-20 because no suitable real-world code context was found, while synthetic prompts were outside the intended threat model.The authors state that synthetic code could have been used but was excluded to preserve evaluation against real-world code prompts.Source: Appendix D · p. 19
  • The authors agree that TROJANPUZZLE’s relatively low success rate can hinder practicability, while positioning the attacks as a novel category for further research.They connect the findings to practitioners’ decisions about selecting code for model training and fine-tuning.Source: Appendix D · p. 19
Loading 2301.02344v2…