Source-linked AI summary
Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
Alexander Sternfeld, Andrei Kucharavy, Ljiljana Dolamic
TL;DR
The paper asks whether minor prompt perturbations, previously known to affect functional correctness, also compromise code security. It applies token-level mutations across models and languages and probes hidden states for predictive signals. Single-character changes can flip generated code from secure to vulnerable, while input-handling flaws are more predictable than secure-defaults flaws.
Problem
Whether minor prompt perturbations compromise the security of generated code remains unstudied despite their known effects on functional correctness.
Method
The study applies systematic prompt mutations across three coding models and five languages, then trains classifiers on prompt hidden representations.
Results
Single-character mutations can affect generated-code security, and input-handling vulnerabilities have higher probe performance than secure-defaults vulnerabilities.
Takeaways & Limitations
Security monitoring for LLM-assisted coding must account for ordinary prompt variation, with input-handling flaws detectable before generation and secure-defaults flaws requiring decoding-time intervention.
Takeaways & Limitations
The partition of CWEs into input-handling and secure-defaults families is manual and remains to be independently verified.
Abstract
from arXiv · showhide
LLM-based coding assistants are seeing rapid adoption, offering substantial gains in developer productivity. As organizations increasingly ship code these agents produce, the security of that code becomes critical. Prior work has shown that minor prompt perturbations degrade the functional correctness of LLM-generated code, but whether they also compromise code security has remained unstudied. We apply token-level mutations to prompts across three models and five programming languages, and show that mutations as small as a single-character change can flip generated code from secure to vulnerable. Probing the models' hidden states reveals that this fragility is partially encoded in prompt representations, but unevenly so. Input-handling vulnerabilities, where the model omits validation or sanitization, are more predictable (mean AUC 0.753) than secure-defaults vulnerabilities, where insecure code stems from one local choice such as a weak algorithm or unsafe parameter (mean AUC 0.674). These results show that the threat model for LLM-assisted coding extends beyond prompt injection to ordinary prompt variation, and indicate that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.
1 Introduction
LLM coding assistants are widely adopted, but generated code can be insecure, and even minimal prompt mutations may alter both functionality and security. This work studies these effects and probes whether vulnerability outcomes are encoded in prompt representations.
- The study addresses whether prompt mutations compromise security, a question left open by prior work on functional fragility.Minor mutations can change whether generated code runs correctly, but their security consequences were previously unstudied.
- The evaluation applies systematic single-character, three-character, and token-level mutations using CWEval’s paired functionality and security tests.The benchmark supports simultaneous evaluation because functionality and security can stand in tension for coding LLMs.
- Single-character prompt mutations can affect both the functionality and security of LLM-generated code.
- Mutation effects depend either on where in the prompt the mutation occurs or on which mutation is performed.
- Hidden-state probe performance varies by vulnerability type, with validation-related flaws more predictable than flaws caused by a single local security choice.
2 Related Work
Prior research established insecure code generation, functional sensitivity to minor prompt changes, and hidden-state signals for code quality. This work connects these strands by examining security under prompt perturbations and vulnerability-type-dependent predictability.
- Security benchmarks have repeatedly found vulnerabilities in LLM-generated code, beginning with roughly 40% vulnerable programs from GitHub Copilot scenarios [Pearce et al., 2021].
- CWEval couples functionality tests with security-specific oracles across 31 CWE types and five programming languages.This joint, outcome-driven design addresses the separation of functionality and security in earlier benchmarks.
- Prior perturbation studies found that meaning-preserving changes, including single-character edits, can degrade functional correctness in generated code.
- Hidden-state probing has been used to predict functional correctness, while robustness work has compared representations across original and perturbed prompts.
- The present contribution identifies that hidden-state signal strength depends systematically on vulnerability type.
3 Methodology
The methodology evaluates prompt mutations on CWEval across multilingual coding models, jointly measuring functionality and security. It also probes prompt-end hidden states with lightweight classifiers to predict generation outcomes.
- CWEval evaluates functional correctness and security simultaneously through paired test oracles across five programming languages.The benchmark covers C, C++, Go, JavaScript, and Python, with broad CWE coverage within each language.
- The study applies single-character substitutions, three-character changes within one token, and whole-token replacements using similar tokenizer tokens.These mutation categories increase in semantic disruption.
- Qwen3-Coder (30B) uses fewer tokens per prompt than CodeLlama (70B) and DeepSeek-Coder (33B).
- 3.3 Generation protocol: Experiments use CodeLlama (70B), DeepSeek-Coder (33B), and Qwen3-Coder (30B) with temperature T=0.Additional sensitivity analysis uses gpt-oss (120B) and Qwen2.5-Coder (3B).
- 3.4 Hidden states probing: Prompt-end activations from individual transformer layers are used to predict whether each mutated generation is functional and jointly functional and secure.The study fits per-cell logistic-regression and two-layer MLP probes, selecting them through stratified hold-out evaluation and cross-validation.
4 Results
Minimal prompt mutations frequently alter the functionality or security of generated code, with effects depending on model, language, mutation position, and exact token change. Hidden-state probes predict input-handling vulnerabilities better than secure-defaults vulnerabilities, suggesting uneven pre-generation visibility.
- 4.2 Effect of Mutations: Across both affected-CWE breadth and mutation-response intensity, prompt mutations affect many CWEs and cause more harm than benefit, especially for CodeLlama (70B) and DeepSeek-Coder (33B).Breadth measures how many CWEs are affected, whereas intensity measures the fraction of mutations changing each affected CWE’s outcome.
- 4.2 Effect of Mutations: Qwen3-Coder (30B) is the most robust model by effect size, while CodeLlama (70B) and DeepSeek-Coder (33B) show consistently larger degradation.The original-prompt benchmark ranking also places Qwen3-Coder strongest and CodeLlama weakest by func-sec@1.
- 4.2 Effect of Mutations: Python exhibits broad but low-intensity sensitivity, whereas Go exhibits narrow but high-intensity sensitivity to prompt mutations.In Python, many CWEs are affected but few mutations change each outcome; in Go, fewer CWEs respond, but a high fraction of mutations flip outcomes.
- 4.3 Token Position versus Precise Mutation: Mutations in the prompt’s middle, especially the CWE docstring, tend to be most harmful to functionality and security.The deleterious effect rises through the first third, peaks in the middle, and declines toward the end.
- 4.3 Token Position versus Precise Mutation: For most cases, the precise token change matters more than position, because only a few of the 18 mutations at a position effectively alter security.A single-character change from “otherwise,” to “otherwiseV” removed the path-traversal guard around tar.extractall, allowing entries such as ../../etc/passwd to escape dest_path.
- 4.4 Probing: Mean held-out probe AUC is approximately 0.70 for each model, but performance is higher for input-handling than secure-defaults vulnerabilities.Input-handling fixes add validation or guards between untrusted input and sensitive sinks, whereas secure-defaults fixes often change one security-critical parameter.
5 Conclusion
Across three models and five languages, minimal prompt mutations can flip generated code between functional and failing or secure and vulnerable, while hidden-state signals predict vulnerability unevenly. Input-handling vulnerabilities are more predictable than secure-defaults vulnerabilities, motivating different intervention points.
- 5 Conclusion: Mean AUC was 0.753 for input-handling vulnerabilities versus 0.674 for secure-defaults vulnerabilities.Input-handling flaws require structural fixes, whereas secure-defaults flaws hinge on a single literal or keyword choice.
- 5 Conclusion: Single-character prompt mutations can flip LLM-generated code from functional to failing and from secure to vulnerable.The study spans three models and five languages.
- 5 Conclusion: Mutation effects depend either on the prompt position of the mutation or on the specific character substitution.The controlling factor varies across CWEs and perturbations.
- 5 Conclusion: Prompt-end hidden states carry measurable but uneven security signals, suggesting structural decisions are made before generation while point-wise choices are deferred to next-token sampling.The latter mechanism is presented as an explanation for the weaker secure-defaults signal.
- 5 Conclusion: Secure-defaults vulnerabilities may require inference-time guardrails or probes over latent representations during generation.These are proposed follow-up directions rather than evaluated interventions.
Limitations
The study’s conclusions are bounded by its model coverage, benchmark scope, and manually constructed vulnerability-family partition. The authors identify replication across broader models and benchmarks, plus independent validation of the grouping, as necessary follow-up work.
- Limitations: The evaluation covers open-weight, code-specialized models mainly in the 30B-70B range, excluding leading closed-source and largest open coding models.The design requires many thousands of scored completions and per-layer hidden states, making frontier-scale or closed-API evaluation infeasible.
- Limitations: The probing analysis relies on one benchmark with a limited set of CWEs and task specifications.Replication on other joint security-functionality benchmarks or purpose-built datasets could strengthen the findings.
- Limitations: The input-handling and secure-defaults families were partitioned manually using each vulnerability’s canonical fix.This family-level finding remains to be independently verified.
Ethical Considerations
The paper studies ordinary developer prompt variation rather than adversarial prompting, and uses CWEval to measure security without releasing novel exploit code. Its probe is intended to detect and mitigate insecure generations before they reach developers.
- Ethical Considerations: The threat model covers random perturbations from typos, paraphrasing, or autocomplete artifacts, not crafted attacks, jailbreaking, or prompt injection.Perturbations are sampled without knowledge of model internals.
- Ethical Considerations: The prompt-end hidden-state probe is intended to detect and ultimately mitigate insecure generations before code reaches a developer.The study reports aggregate vulnerability rates and per-CWE probe performance.
- Ethical Considerations: The work uses CWEval to rerun paired functional and security test oracles, without releasing novel exploit code.The examined vulnerability classes are already catalogued in the public CWE taxonomy and CWEval benchmark.
A.1 Mutation effects
Mutation effects are evaluated at thresholds of 1 and 50 effective mutations, including cases where security changes while functionality remains unchanged. At least one mutation affects many CWEs for functionality, while Python shows broad but less concentrated security effects across CWEs.
- A.1 Mutation effects: At the 1-mutation threshold, many CWEs pass for functionality, while security-only changes are less common because they are more severe events.The right-panel criterion requires unchanged functionality with affected security.
- A.1 Mutation effects: Figures 7 and 8 report the fractions of CWEs with at least 1 or at least 50 mutations changing functionality or security.For security, the figures count only mutations that preserve functionality while changing security.
- A.1 Mutation effects: At the 50-mutation threshold, Python is least affected for security, despite being most affected at the 1-mutation threshold.This indicates Python’s effective mutations are widespread across CWEs but not concentrated within individual CWEs.
A.2 Model family and size
Additional models and sampling temperatures broadly reproduce the primary robustness findings, while robustness differs across model families, sizes, and languages.
- A.2 Model family and size: Qwen2.5-Coder (3B) is more robust to mutations than gpt-oss (120B), showing that larger model size does not necessarily improve robustness.Qwen2.5-Coder nevertheless outperforms gpt-oss on Go and Javascript baseline performance.
- A.2 Model family and size: For gpt-oss, robustness is better in Python and Javascript than in C, Go, and C++.
- A.2 Model family and size: Overall, temperature-0.8 results remain consistent with temperature-0 findings, with non-trivial CWE fractions affected for functionality and joint functionality-security outcomes.The analysis uses ten completions per original and mutated prompt, Fisher’s exact tests, and Benjamini-Hochberg correction at α = 0.05.
- A.2 Model family and size: At temperature 0.8, increasing sampling variety does not alleviate mutation-related harm, because similar CWE counts are affected for functionality and joint security outcomes.
B Position analysis
Effective prompt mutations are sparse and position-dependent, with sensitive literals, parameter names, examples, and contextual cues able to flip generated code between secure and vulnerable behavior.
- B Position analysis: Effective mutations concentrate at a few token positions rather than appearing uniformly, and functionality is perturbed at more positions than security.Most cells contain few effective mutations, while occasional positions show nearly all mutations changing the outcome.
- B Position analysis: Mutating example literals can remove Qwen3-Coder’s per-character validation and produce an SSRF, whereas changing “user ID” or example values can induce an isalnum check.The highlighted positions affect the joint functional-security label in at least 70% of mutations.
- B Position analysis: A three-character change from “path” to “LotY” removes the cue that dir_path is untrusted, causing unquoted shell concatenation and command injection.The original uses fork+execlp with separated arguments; the mutated version executes the command in dir_path = "tmp && echo hacked!".
C Probing: Hyperparameter Tuning
Probe tuning finds modest layer effects and model-specific probe preferences, while hidden-state security signals are available mainly from early-to-middle layers and differ by vulnerability family.
- C Probing: Hyperparameter Tuning: Top phase-1 AUCs span 0.700-0.704 for CodeLlama, 0.680-0.683 for DeepSeek-Coder, and 0.661-0.671 for Qwen3-Coder.CodeLlama and DeepSeek-Coder favor logistic regression, whereas Qwen3-Coder’s top configurations are MLPs.
- C Probing: Hyperparameter Tuning: Layer choice has only modest effects: AUC rises through roughly the first third of each network before levelling off, indicating signal availability from early-to-middle layers onward.The models have different layer profiles, with DeepSeek-Coder flattest and lowest overall and Qwen3-Coder rising from shallow-layer values before easing at the end.
- C Probing: Hyperparameter Tuning: Input-handling vulnerabilities score higher than secure-defaults vulnerabilities on the functional target, with Mann-Whitney I > D and p = 0.031.The gap is smaller than for the joint functional-security target, and the family counts are 11 versus 8 because threshold eligibility differs by target.