Source-linked AI summary
Beware EviLLM: Enabling Vulnerability Injection via Large Language Models
Zeezoo Ryu, Simon Chung, Muhammad Faraz Karim, Anna Raymaker, Karan Singh Jodha, Yash Chaturvedi, Sukarno Mertoguno
TL;DR
Prior work examined accidental insecurity in generated code or attacks involving malicious LLMs or users, leaving third-party pipeline compromise as a distinct threat model. EviLLM implements two such attacks and studies their effects; participants often missed the injected vulnerabilities and many had rarely or never considered such attacks.
Problem
Prior work examined inadvertent insecure code or unconventional threats involving malicious LLM providers or end-user attackers, rather than third parties tampering with code-generation pipelines.
Source: §1 · pp. 1–2Method
EviLLM implements third-party attacks using ChatGPT Custom Instructions and a malicious browser plugin to inject vulnerabilities into generated code.
Source: §1 · p. 1, §12 · p. 13Results
7 out of 8 Custom Instructions participants and 10 out of 13 browser-plugin participants propagated injected vulnerabilities, while 13 out of 21 rarely or never considered such attacks.
Source: §12 · p. 13Takeaways & Limitations
A popular browser-extension ecosystem can make large-scale EviLLM attacks feasible; one cited extension had more than 2 million downloads and requested sensitive permissions.
Source: §5.2 · p. 4Experiments used ChatGPT (GPT-4o) as the benign LLM and Llama 3.2 (3B) as the malicious LLM, so results may not generalize to other platforms.
Source: §10.3 · p. 11
Abstract
from arXiv · showhide
Advances in large language models (LLMs) have enabled AI-driven code generation from natural language specifications, introducing new attack surfaces for injecting vulnerabilities into software. Prior work has studied this problem only in benign settings where vulnerabilities are introduced inadvertently, or under unconventional threat models where the LLM itself is malicious (backdooring) or the user is the attacker (jailbreaking). In this paper, we study a more realistic threat model: a third-party adversary, with capabilities comparable to existing cybercriminals, compromises the AI code generation pipeline to deliberately introduce vulnerabilities. We call this the EviLLM attack. We have implemented two instances of EviLLM, each of which only requires the underlying LLM to be accessed through a compromised account or browser, and can inject vulnerabilities from 13 CWE classes. As we show in our feasibility study, both attack vectors are already used to implement many existing cyberattacks. Our user study shows that 7 out of 8 and 10 out of 13 participants did not notice the vulnerabilities injected by the two instances of EviLLM, and 13 out of 21 participants "rarely" or "never" considered the risk of an attack like EviLLM.
1. Introduction
EviLLM studies how third parties can tamper with AI code-generation pipelines to inject vulnerabilities that developers may accept. The paper implements two attacks and reports that participants often missed injected vulnerabilities and rarely considered this threat.
1. Introduction
- LLM code generation enables users with little programming experience to obtain executable code, while vibe coding involves trusting and deploying it without rigorous verification.The paper identifies code security as a continuing limitation of this practice.Source: §1 · p. 1
- Prior work studied inadvertent insecure code or unconventional threats involving malicious LLM providers or end-user jailbreakers,, [24], [36], [63].Prior work also examines insecure suggestions,,,, [43],, [48], [51],,, [48], [51],,, [48], [51],,, [48], [51],.Source: §1 · p. 1
- EviLLM targets third-party pipeline surfaces through ChatGPT Custom Instructions or a malicious browser extension that alters prompts or outputs.The paper focuses on ChatGPT-based attacks and evaluates whether manipulation can reliably induce vulnerable outputs.Source: §1 · p. 1
- In the user study, 7 of 8 Custom Instructions participants and 10 of 13 malicious-plugin participants did not detect injected vulnerabilities; 13 of 21 (more than 60%) rarely or never considered EviLLM-like attacks, even after witnessing one.Source: §1 · p. 2
- The paper analyzes pipeline components, implements two EviLLM attacks, classifies LLM defenses by CWE class, and evaluates user detection and awareness.Its study found that 13 of 21 participants rarely or never considered such attacks, even after witnessing one.Source: §1 · p. 2
2. Background: Code Generation with LLMs
Web-based LLM code generation moves prompts and generated code through a browser, service-side personalization, and an LLM before users review and transfer code to an IDE. The paper focuses on this setting because it suits vibe coding and creates additional attack angles.
2. Background: Code Generation with LLMs
- Users submit prompts through a web interface; the browser relays them to the service, which personalizes them before LLM processing and returns code for review and transfer to an IDE.The workflow is illustrated in Figure 1.Source: §2 · p. 2, Figure 1 · p. 2
- Web-based generation avoids the IDE integration and skeleton-code prompts used in LLM code-completion research [43], making it more suitable for vibe coding.The interaction between web-based LLMs and local IDEs also opens additional attack angles.Source: §2 · p. 2
- Personalization and memory simulate continuity by adding past exchanges to prompts; ChatGPT Custom Instructions prepend user-specified information across sessions.The paper describes this component as important because LLMs lack inherent long-term memory and have context-window constraints.Source: §2 · p. 2
3. Threat Model
The threat model considers a third-party adversary tampering with a web-based code-generation pipeline to inject known vulnerabilities and have them evade review. It assumes access through a compromised account or malicious browser extension, without provider or model-training control.
3. Threat Model
- The threat model formalizes the adversary’s goals, capabilities, and operating assumptions.The paper excludes a similar ChatGPT and other-tools “Memory” feature from its coverage.Source: §3 · p. 2
3.1. Adversary Goals
- The adversary aims to make the LLM generate one or more CWE-defined vulnerability classes and have the victim miss them during review and deployment..Source: §3.1 · p. 3
3.2. Adversary Capabilities
- The adversary has either the victim’s LLM-service credentials to change account settings or a browser extension with clipboard, page-access, and server-contact capabilities.The model excludes insider access to the LLM provider and changes to training data or model weights.Source: §3.2 · p. 3
3.3. Assumptions
- The assumptions are that victims use ChatGPT for code generation, engage in vibe coding, and rely on an honest-but-vulnerable provider.Vibe coding here means trusting and deploying generated code with limited manual review.Source: §3.3 · p. 3
3.4. Scope
- EviLLM differs from poisoning, backdoors, and jailbreaking by targeting third-party pipeline surfaces; provider-infrastructure compromise and encrypted-traffic interception are out of scope.Those excluded attacks represent higher capability thresholds.Source: §3.4 · p. 3
4. Attacking the AI Code Generation Pipeline
The paper analyzes pipeline steps for possible vulnerability injection and practical implementation constraints. It discusses malicious services, compromised browsers, hijacked personalization, local-code modification, and network-traffic attacks.
4. Attacking the AI Code Generation Pipeline
- The section systematically examines how workflow steps can be compromised for vulnerability injection and the practical considerations of each avenue.Source: §4 · p. 3
4.1. Malicious Service
- An adversary can operate a malicious LLM and attract users, but implementation choices trade cost against performance and reliability.Training from scratch, fine-tuning, or jailbreaking are options; the paper considers fine-tuning and jailbreaking more feasible, with fine-tuning more expensive but more reliable.Source: §4.1 · p. 3
4.2. Compromised Browser Modifying Prompt and Code
- A compromised browser can modify prompts or returned code; practical attacks must avoid visible prompt changes and noticeable response delays.The proposed approach generates vulnerable code in parallel with the original LLM.Source: §4.2 · p. 3
4.3. Hijacked Personalization and Memory
- A compromised account can change personalization or memory settings, though control depends on the LLM and may be more limited than other attack avenues.The paper describes account compromise as comparatively easy to implement and gives buying leaked credentials as an example.Source: §4.3 · p. 4
4.4. Modifying Local Code
- Modifying source code after it reaches the victim’s disk is possible with filesystem control, but is not specific to AI code generation.Source: §4.4 · p. 4
4.5. Attacking Network Traffic
- In-transit prompt or code tampering is theoretically possible, but widespread encrypted traffic such as HTTPS makes it very hard to implement.Source: §4.5 · p. 4
5. Feasibility of EviLLM Attacks
The paper argues that EviLLM is feasible because its attack vectors overlap with techniques already used by cybercriminals. It examines malicious LLMs, browser extensions, and stolen credentials as routes to vulnerability injection.
5. Feasibility of EviLLM Attacks
- Existing cybercriminal practices include malicious LLMs, malicious browser extensions, and stolen credentials, supporting the feasibility of these EviLLM attack avenues.The paper argues that sufficiently motivated cybercriminals could implement these instantiations.Source: §5 · p. 4
5.1. Malicious LLMs
- Malicious LLMs already generate cybercriminal content, and existing demonstrations and fine-tuning approaches suggest they could also be adapted to output vulnerable code.The paper notes that EviLLM needs vulnerable rather than malicious code, making its prompt engineering easier than the jailbreaking used by Mallas.Source: §5.1 · p. 4
5.2. Malicious Browser Extensions
- Malicious browser extensions have been deployed at scale, while the EviLLM extension requires less-sensitive permissions than several cited malicious extensions.A cited AI browser extension had more than 2 million downloads, and reported extension attacks affected millions of users.Source: §5.2 · p. 4
5.3. Stolen Credentials
- Stolen credentials are a practiced initial-access route, making account-based reconfiguration of a victim’s LLM consistent with existing cybercriminal capabilities.The cited report says 26% of Cisco Talos Incident Response engagements in 2023 involved valid accounts.Source: §5.3 · p. 5
6. Implementation #1: Custom Instructions
The Custom Instructions attack tampers with account-level instructions to steer ChatGPT toward vulnerable code. The authors describe how the prompt strategy varies with the model’s response to a targeted vulnerability.
6.1. Protection of Custom Instructions
- As of June 2025, account access allows Custom Instructions to be changed without a notification, and the instructions are hidden from normal chat interaction.The settings interface is required to view them.Source: §6.1 · p. 5
- The paper compares Custom Instructions protection with password protections, which commonly use stronger two-factor checks and external update notifications.The comparison names GitHub as an example of a service with such protections.Source: §6.1 · p. 5
6.2. Prompt Engineering
- The attack uses ChatGPT Custom Instructions to steer generated code toward vulnerabilities, with prompt-crafting difficulty depending on how strictly ChatGPT handles the target vulnerability.The authors group responses as D.O.A., Gullible, Vigilant, and Comply with Warnings.Source: §6.2 · p. 5, §6 · p. 5
- For CWE-20, CWE-434, and CWE-732, benign prompts can already elicit vulnerable code, so the attacker needs no Custom Instructions.The authors label this response pattern D.O.A.Source: §6.2 · p. 5
- For some CWE classes, ChatGPT produces the desired vulnerable code when an instruction directly describes the vulnerability.The examples include weak RSA key sizes for CWE-326 and logging API keys for CWE-532.Source: §6.2 · p. 5
- For vigilant classes such as CWE-89, ChatGPT may ignore a direct instruction to implement the vulnerability, causing this attack strategy to fail.The paper says the victim would not be expected to ask ChatGPT to rewrite the code to inject the vulnerability.Source: §6.2 · p. 5
- For vigilant classes, supplying faulty custom code for the target functionality can lead ChatGPT to produce vulnerable output.Examples include removing the wrong character during SQL sanitization and rejecting “,.” instead of “..” in URLs.Source: §6.2 · p. 5
- When ChatGPT warns about an injected vulnerability, instructions can sometimes suppress the warning or rename the vulnerability.The authors give “banana overload” as a replacement phrase for SQL injection, and the example output describes unsafe string interpolation.Source: §6.2 · p. 6
7. Implementation #2: Browser Plugin
The browser-plugin implementation pairs benign and adversarial code generation, then substitutes the adversarial version when users copy code. Its simple matching approach is restrictive, but the user study found it sufficient for the attack to succeed.
7. Implementation #2: Browser Plugin
- The plugin forwards each prompt to both benign and malicious LLMs, then substitutes the malicious code when the user copies the benign snippet.The benign response remains displayed, while the malicious version is written to the clipboard and pasted into the editor.Source: §7 · p. 6
7.1. Malicious LLM
- The malicious LLM appends vulnerability-injection instructions to intercepted prompts before sending them to Llama 3.2 (3B) on the adversary’s server.The local implementation mirrors the Custom Instructions attack.Source: §7.1 · p. 6
- A keyword-based classifier selects an injection instruction, mapping web or database terms to SQL injection and “crypt” terms to weak cryptography.The algorithm chooses the vulnerability judged most relevant to the intercepted prompt.Source: §7.1 · p. 6
7.2. Matching Algorithm
- Code snippets are matched only when each response contains exactly one snippet of a given programming language; otherwise, the benign snippet is retained.The fail-safe rule limits matching in more complex coding scenarios.Source: §7.2 · p. 6
- The matching algorithm is restrictive but efficient, and the user study found it sufficient for the attack to succeed.Matching fails when either response contains multiple code snippets in the same language.Source: §7.2 · p. 6
8. Evaluation
The evaluation finds that both tested LLMs generally need little coercion to produce the targeted vulnerable code, with some CWE classes requiring faulty-check prompts. The experiments replayed prompts across selected datasets and checked generated code for vulnerability criteria.
8.1. Methodology
- The evaluation replayed prompts with and without malicious Custom Instructions, resetting the target LLM before each experiment and repeating each condition three times.The targets were ChatGPT and Llama.Source: §8.1 · p. 7
- Success required vulnerable code, with additional explanation criteria for ChatGPT and Llama, and vulnerability checks used CodeQL followed by manual review.The manual review covered irregular cases such as SQL injection.Source: §8.1 · p. 7
8.2. Prompts for WildChat Dataset
- WildChat prompts were selected by matching vulnerability signatures from a dataset of fixes to conversations involving code affected by the targeted CWE classes.CWE-79 lacked sufficient WildChat data, so the authors instead prompted ChatGPT with a website-comment use case.Source: §8.2 · p. 7
8.3. Prompts for Asleep at Keyboard Dataset
- For the Asleep at Keyboard dataset, the authors used function descriptions as prompts and added instructions specifying the implementation language or platform.The evaluated classes focused on more directly exploitable vulnerabilities.Source: §8.3 · p. 7
8.4. Experiment Date
- WildChat experiments were repeated in January 2025 after an initial November 2024 round, and the reported experiments used GPT-4o for ChatGPT.The two WildChat rounds mostly agreed.Source: §8.4 · p. 7
8.5. Experimental Results
- Both tested LLMs generally needed little coercion to generate the desired vulnerable code, though some CWE classes required “broken check” prompts.The authors describe most tested classes as “gullible” or “dead on arrival.”Source: §8.5 · p. 7
9. User Study
The study tests whether users detect vulnerabilities injected through Custom Instructions or a malicious browser plugin while completing coding tasks with ChatGPT. Most participants missed both injected vulnerability types, while survey responses showed limited awareness of malicious injections.
9.1. Experimental Procedure
- Participants completed two ChatGPT coding tasks under either Custom Instructions manipulation or copy-and-paste hijacking by a malicious Chrome plugin.Submitted answers—not code initially displayed by ChatGPT—were evaluated, which matters because the plugin could make displayed and copied code differ.Source: §9.1 · p. 8
9.1.1. Survey Instruments.
- Pre- and post-task surveys collected participants’ backgrounds, ChatGPT use, risk awareness, confidence, perceived helpfulness and accuracy, and coding-task usage.The study used submitted code and the MVC metric to assess missed vulnerabilities, with submitted vulnerability checks performed through the CodeQL-plus-manual process.Source: §9.1.1 · pp. 8–9, §9.1 · p. 8
9.1.2. Participant Recruitment and Ethics.
- The analysis included 21 participants: 8 in the Custom Instructions experiment and 13 in the malicious-plugin experiment; qualitative coding had κ = 0.9043.The study received IRB approval; participation was voluntary and uncompensated.Source: §9.1.2 · p. 9
9.1.3. Pilot Study.
- The Custom Instructions pilot, with consistent vulnerability injection rates and behavioral patterns across participants, was folded into the final participant count; plugin feedback enabled an improved implementation tested with 13 additional participants.Source: §9.1.3 · p. 9
9.2. Results and Analysis
- 7 of 8 Custom Instructions participants and 10 of 13 plugin participants submitted code containing both injected vulnerabilities.Only two participants across all 21 reported anomalous behavior.Source: §9.2 · p. 9, Figure 3 · p. 9
- Participants showed a weak positive association between confidence and missed vulnerabilities, minimal correlation with academic program level, and a weak negative association with programming expertise.The reported associations were not statistically significant.Source: §9.2 · p. 9
- Awareness of malicious potential was moderately negatively associated with missed vulnerabilities (Pearson’s r = −0.44, p = 0.04; Spearman’s ρ = −0.42, p = 0.06), while awareness of inaccuracies showed weak negative associations.The Pearson association was significant; the Spearman association was borderline.Source: §9.2 · p. 9, Figure 4 · p. 10
- Custom Instructions participants offered minimal detection comments, while plugin participants generally noticed no anomaly or misattributed it to a leftover clipboard paste.C18 submitted vulnerable code despite noticing an unusual website detail; A16 submitted clean code but attributed the anomaly to a leftover paste.Source: §9.2 · pp. 9–10
10. Discussion
The discussion shows that EviLLM vulnerabilities can evade users’ review and examines user, model, and code-analysis defenses alongside their limitations. It closes with stakeholder recommendations and notes that the evaluation is limited in platform, attack, and user-study coverage.
10.1. Significance of Our User Study and Proofof-Concept
- Participants’ over-reliance under task pressure led many to copy generated code without critical evaluation, while copied code and IDE integration create further review concerns.The authors also report that most participants who directly copied code fell victim to the attack.Source: §10.1 · p. 10
10.2.1. More Vigilant Users.
- Users may need expertise across at least 13 injectable vulnerability classes, including visually similar Unicode substitutions that can reopen SQL injection in sanitizing code.The discussion notes that some classes are unfamiliar even to security researchers, and that vulnerable injections lack the common triggers of human-introduced bugs.Source: §10.2.1 · pp. 10–11
10.2.2. More Vigilant LLM.
- Prompt guardrails can be bypassed by the Custom Instructions attack and do not stop a browser-plugin attack using a malicious model; filtering generated output instead can delay responses.The paper reports that current LLMs produced the desired vulnerable code with minimal jailbreaking effort.Source: §10.2.2 · p. 11
10.2.4. Detecting Vulnerable Code.
- Analyzing generated code before execution can address either prompt-pollution or malicious-model attacks, but static-analysis tools may be incomplete or noisy and still require knowledgeable developers to fix alerts.The discussion also describes account protections such as second-factor authentication and alerts for Custom Instructions changes.Source: §10.2.4 · p. 11
10.3. Limitations
- The evaluation covers ChatGPT (GPT-4o) and Llama 3.2 (3B), implements only two attack instances, and the browser-plugin matcher fails on multiple same-language code snippets.The authors caution that other platforms may behave differently and that unimplemented attack surfaces remain unevaluated.Source: §10.3 · pp. 11–12
- The user study involved 21 participants in a controlled lab; its predominantly graduate-student, security-trained sample and mostly nonsignificant correlations limit stronger generalization.The authors call for larger and more diverse participant pools and note that real-world development contexts may affect detection rates.Source: §10.3 · pp. 11–12
10.4. Recommendations for Best Practices
- Recommendations call for developers to review outputs, providers to strengthen account and output safeguards, and researchers to test broader settings and detection tools.The authors also recommend longitudinal studies and industry collaboration for evaluating real-world deployment risks.Source: §10.4 · p. 12
10.5. Responsible Disclosure
- The authors disclosed their findings to OpenAI through Bugcrowd’s coordinated vulnerability disclosure program.Source: §10.5 · p. 12
11. Related Work
The related-work discussion distinguishes EviLLM from attacks based on compromised training data, end-user jailbreaking, and other third-party vectors. It also situates the paper among studies evaluating generated-code security and quality.
11.1. Poisoning and Backdoor Attacks against LLMs
- Unlike poisoning and backdooring, which require control of training or fine-tuning data, EviLLM assumes a compromised browser or user account and can be reprogrammed to target different vulnerabilities.Prior poisoning and backdoor work includes user studies of users’ awareness and acceptance of vulnerable code.Source: §11.1 · p. 12
11.2. Jailbreaking Attacks against LLMs
- Jailbreaking makes the end user the adversary, whereas EviLLM uses a third party to inject vulnerabilities that can harm the user and downstream users of generated code.The paper gives exploitable SQL injection as an example of potential harm to downstream users.Source: §11.2 · pp. 12–13
11.3. Third-Party Attacks against LLMs
- Compared with prompt injection and hallucinated-package squatting, EviLLM uses common compromise vectors and is designed to produce vulnerable code that runs normally when not exploited.Source: §11.3 · p. 13
11.4. Evaluating the Correctness/Security of LLM-Generated Code
- Prior code-generation studies variously assess vulnerabilities, general correctness or quality, and the final code quality after users may review or fix outputs.Source: §11.4 · p. 13
12. Conclusion
The paper introduces EviLLM, a third-party attack framework implemented through ChatGPT Custom Instructions and a malicious browser plugin. Its user study finds vulnerability propagation and low awareness, while the authors caution that the evaluation covers few platforms and participants.
12. Conclusion
- The two EviLLM implementations injected vulnerabilities spanning 13 CWE classes.One uses ChatGPT Custom Instructions and the other a malicious browser plugin.Source: §12 · p. 13
- 13 out of 21 participants (61.9%) rarely or never considered EviLLM-like attacks; 7 out of 8 and 10 out of 13 propagated vulnerabilities in the respective experiments.The authors report susceptibility even among participants with formal security training.Source: §12 · p. 13
- The evaluation studied ChatGPT as the benign target LLM and Llama 3.2 as the malicious LLM, and the user study was small-scale.The authors call for evaluation on other platforms and IDE-integrated tools and larger, more diverse user studies.Source: §12 · p. 13
Appendix A: Participant Demographics and Task Results
The appendices document participant task outcomes using vulnerability and attack-result labels, alongside comments describing what participants noticed about the generated code and interface.
Appendix A: Participant Demographics and Task Results
- MVC counts vulnerabilities in submitted code by assigning 1 to each Y and 0 to each N, then summing the values.A higher MVC indicates more vulnerabilities in the submitted code.Source: Appendix A · p. 16
- The participant table records expertise, ChatGPT programming-use frequency, test type, vulnerability answers, MVC, attack result, anomaly, and category.Attack results are labeled success, partial success, or failure; categories distinguish successful victims, design shortcomings, resilient participants, and human factors.Source: Appendix A · p. 16
Appendix B: Summary Of Comments from User Study Participants
- Comments include reports of unnoticed irregularities, while some participants identified insecure copied code or struggled to evaluate code concepts.C14 compared code obtained through the page’s copy button with manually copied code; D3 said unfamiliarity with concepts such as encryption limited evaluation.Source: Appendix B · p. 17