PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code

Maitreyee Das Urmi, Jessica Pourleyli, Fabio Santos, Glaucia Melo

arXiv:2608.24857v1cs.CRcs.SE

TL;DR

This study examines whether structured and security-oriented prompts improve the security of LLM-generated code beyond influencing compliance. Using 424 security-sensitive Python tasks with GPT-4o and LLaMA 3.1-8B, it finds that prompting primarily redistributes weaknesses rather than reliably reducing them, while security controls remain necessary.

  • Problem

    Existing research emphasizes functional correctness and aggregate reductions in static-analysis findings, leaving limited evidence about how prompt structure affects the distribution and manifestation of security weaknesses in generated code.

    Source: §I · p. 1

  • Method

    The study generates code for 424 security-sensitive Python tasks under five progressively structured and security-oriented prompts, evaluating compliance and security findings with Bandit and CodeQL across GPT-4o and LLaMA 3.1-8B.

    Source: §I · pp. 1–2

  • Results

    Structured prompts substantially improve compliance but do not consistently reduce weakness prevalence; GPT-4o shifts findings toward lower severity, whereas LLaMA shows weaker, less consistent effects.

    Source: §V · p. 9, §VI · p. 10

  • Takeaways & Limitations

    Prompt structure is primarily a compliance and behavioural-shaping mechanism, so prompt engineering should complement rather than replace post-generation security controls.

    Source: §V · p. 9, §VI · p. 10

    The evaluation relies on Bandit and CodeQL, is limited to Python tasks from one vulnerability-focused dataset, and compares only GPT-4o and LLaMA 3.1-8B.

    Source: §V · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.

I. INTRODUCTION

This study examines prompt structure as a security-related influence on LLM-generated code, asking whether it changes compliance, weakness prevalence, severity, and CWE distributions. It frames the dominant effect as redistribution rather than reduction.

Research gap

  • Existing research emphasizes functional correctness, while security-focused prompting studies largely report aggregate weakness counts rather than severity or composition.This leaves uncertainty about whether apparent security improvements reflect reduced risk or shifted weakness profiles.Source: §I · p. 1, §II · p. 2

Research questions and approach

  • The study evaluates prompt structure across identical security-sensitive programming tasks using proprietary and open-weight LLMs, Bandit, and CodeQL.Its two research questions separate code-generation compliance from security analysis conditioned on valid code.Source: §I · pp. 1–2

Contributions

  • The central claim is that prompt structure primarily redistributes security risk rather than reducing it outright.This challenges the aggregate-reduction narrative in secure code generation research.Source: §I · p. 1

Research questions and approach

  • Structured prompting is expected to influence whether models produce analyzable code and how remaining weaknesses are distributed across prevalence, severity, and CWE categories.The framework distinguishes valid executable Python from refusal or non-code outputs before comparing security findings.Source: §I · p. 2

Contributions

  • The paper reports that prompt refinements can substantially alter analyzable-output populations, making weakness comparisons sensitive to changes in compliance.Its contributions include a two-stage evaluation framework and a semantic-drift audit.Source: §I · p. 2

A. LLM-Generated Code Quality and Security Weaknesses

Prior work shows that LLM-generated code can remain vulnerable despite functional plausibility, while prompt-based security interventions are usually assessed with aggregate metrics. This paper motivates severity- and composition-aware analysis.

LLM-generated code quality and security

  • LLM-generated programs may be syntactically correct and functionally plausible while still containing unsafe patterns and vulnerabilities.Security weaknesses can persist even when functional tests pass.Source: A. LLM-Generated Code Quality and… · p. 2

Prompt engineering

  • Prompt engineering improves correctness, specification adherence, consistency, maintainability, and reliability across software-engineering tasks.Structured prompts can also improve static quality metrics and reduce certain defect classes, although security misconfigurations may persist.Source: B. Prompt Engineering for Software… · p. 2

Research gap

Security-aware code generation

  • Security-aware prompting, iterative self-repair, and static-analysis feedback loops have generally been reported to reduce detected security weaknesses relative to baseline prompting.Prior critiques caution that separating security from functionality or relying on one static analyzer can yield incomplete assessments.Source: C. Security-Aware and Responsible Code… · pp. 2–3

Research gap

III. METHODOLOGY

The study systematically varies prompt structures, generates code from security-sensitive Python tasks with GPT-4o and LLaMA 3.1-8B, and evaluates outputs through staged static analysis. The pipeline separates generation success from weakness assessment.

Experimental pipeline

  • The methodology uses three stages: prompt design, code generation and collection, and security weakness assessment with Bandit and CodeQL.Figure 1 summarizes this experimental pipeline.Source: §III · p. 3, Figure 1 · p. 3

Models and environment

  • The evaluation compares GPT-4o and LLaMA 3.1-8B as proprietary high-capacity and open-weight model examples.Experiments were conducted in a controlled local environment for consistency across models, prompts, and tools.Source: A. Experimental Infrastructure · p. 3

Security analysis

  • Bandit supplies rule-based LOW, MEDIUM, and HIGH severity annotations, while CodeQL provides semantic, query-driven vulnerability discovery.Using both tools provides complementary static-analysis coverage.Source: A. Experimental Infrastructure · p. 3

Generation and collection

  • The experiments record whether each prompt–model interaction yields executable Python code or a refusal/non-code response.This enables compliance analysis before downstream security comparisons.Source: A. Experimental Infrastructure · p. 3, §III · p. 3

Dataset

  • The dataset contains 424 Python examples filtered from the Cybernative.ai Code Vulnerability and Security Dataset.The tasks instruct models to produce secure and valid Python solutions.Source: B. Dataset Selection and Preparation · p. 3
  • Tasks are security-sensitive because they involve vulnerability-prone scenarios such as command execution, file handling, deserialization, cryptography, and input validation.These operations create opportunities for insecure implementations assessable through static analysis.Source: B. Dataset Selection and Preparation · p. 3

C. Prompt Design

The prompt design holds the user task constant while varying system-level structure and guidance. All variants emphasize code-only, senior-engineer responses with formatting, quality, and error-handling constraints.

C. Prompt Design

  • The prompt templates vary system structure while maintaining a consistent output format across interactions.Each interaction contains a system prompt and a user prompt.Source: C. Prompt Design · p. 3

1) User Prompt (Invariant):

  • The user prompt remains invariant across experiments and asks for a complete, correct solution to an insecure coding question.This controls the task content while prompt variants change system-level instructions.Source: 1) User Prompt (Invariant) · p. 3
  • Every prompt variant assigns the model the role of Senior Software Engineer rather than Security Engineer.The chosen persona emphasizes task completion over post hoc vulnerability analysis.Source: 2) Base Prompt Specifications · p. 3

2) Base Prompt Specifications:

  • Each system prompt states that a coding task follows and explicitly requires code-only output.These constraints were added because pilot models often returned explanations instead of implementations.Source: 2) Base Prompt Specifications · p. 3
  • Prompts enforce Python best practices, modular design, and explicit error handling to promote production-level quality and reduce code smells.The constraints also support clarity, consistency, and modularity.Source: 2) Base Prompt Specifications · pp. 3–4
  • Strict output formatting enables automated extraction and evaluation.Formatting constraints make generated responses easier to process consistently.Source: 2) Base Prompt Specifications · p. 4

3) Prompt Variants and Strategy:

The study builds five controlled prompt variants, progressively adding structural constraints and security framing to test their effects on code generation and security.

4) Prompt Refinement Rationale and Termination Criterion:

The prompt-refinement design systematically varies structural guidance, security framing, and adversarial awareness while holding generation and analysis procedures consistent.

4) Prompt Refinement Rationale and Termination Criterion:

D. Code Generation and Data Collection

E. Security Weakness Analysis with Bandit and CodeQL

F. Analysis and Validation

The analysis separates generation compliance from security analysis, then validates severity and weakness-type patterns under controlled experimental conditions.

  • Stage 1 measures invalid or non-code outputs and files containing security weaknesses across all prompt variants.Source: F. Analysis and Validation · p. 5
  • Stage 2 analyzes severity and CWE distributions only among executable Python outputs because refusals cannot be meaningfully statically analyzed.Source: F. Analysis and Validation · p. 5
  • The study first analyzes GPT-4o, then repeats severity and CWE analyses on filtered LLaMA outputs and a combined subset.Source: F. Analysis and Validation · p. 5
  • Dataset, prompts, model parameters, hardware, and analysis tools remain constant, while one generation per configuration limits variance estimation.Source: F. Analysis and Validation · p. 5
  • A relevance audit samples 15 problems to check that refinements preserve requested tasks and do not create degenerate outputs.Source: F. Analysis and Validation · p. 5
  • Manual inspection verifies that required functionality is implemented and excludes empty stubs, constant-return programs, and unrelated templates.Source: F. Analysis and Validation · p. 6

A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence

Stage 1 evaluates how prompt variants change analyzable output compliance and the prevalence of Bandit-detected weaknesses across GPT-4o and LLaMA 3.1-8B.

A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence

B. Stage 2 (RQ2): Security Weakness Severity and Type Analysis

1) Severity Distribution Across Prompt Variants:

Across prompt variants, GPT-4o shifts detected findings toward lower severity, whereas LLaMA 3.1-8B shows comparatively stable severity distributions. CWE-level changes are uneven: some categories decline, but dominant weaknesses remain prevalent.

1) Severity Distribution Across Prompt Variants:

  • 20.8% to 13.6%: GPT-4o high-severity findings decrease from Prompt 1.0 to Prompt 4.0.Low-severity findings increase from 32.0% to 43.5%, while medium-severity findings decrease from 47.1% to 43.0%.Source: 1) Severity Distribution Across Prompt… · p. 6
  • Security-oriented prompting shifts GPT-4o findings toward lower severity rather than eliminating security weaknesses outright.The severity shift reflects changes in the types of weaknesses reported, because severity levels are not uniformly distributed across weakness classes.Source: 1) Severity Distribution Across Prompt… · pp. 6–7
  • 21.2%, 22.9%, 21.5%, and 24.4%: LLaMA 3.1-8B high-severity findings remain relatively stable across Prompts 1.0–4.0.Unlike GPT-4o, LLaMA exhibits no comparable monotonic trend, indicating model-dependent prompt effects.Source: 1) Severity Distribution Across Prompt… · p. 7

2) CWE-Level Security Weakness Analysis:

  • CWE-78 and CWE-502 remain highly prevalent across prompt conditions despite prompt refinement.For GPT-4o, CWE-78 increases from 43.9% to 49.7%, while CWE-502 decreases from 23.7% to 21.0% between Prompts 1.0 and 4.0.Source: 2) CWE-Level Security Weakness Analysis · p. 7
  • 10.1% to 4.1%: GPT-4o CWE-94 findings decrease from Prompt 1.0 to Prompt 4.0, although the trend is not strictly monotonic.CWE-95 and CWE-116 also decline, while the same classes remain comparatively stable for LLaMA 3.1-8B.Source: 2) CWE-Level Security Weakness Analysis · p. 7
  • Information-exposure and debugging-related CWE classes exhibit non-monotonic trends across prompt variants.These irregular patterns indicate that prompt refinement does not uniformly influence those weaknesses.Source: 2) CWE-Level Security Weakness Analysis · p. 7
  • CWE-330 remains effectively unchanged across prompt variants for both models.This stability further indicates that prompt effects vary by security weakness class.Source: 2) CWE-Level Security Weakness Analysis · p. 8
  • Prompt refinement influences security-weakness composition more strongly than overall prevalence.The largest decreases concentrate in a relatively small subset of categories, particularly CWE-94, CWE-95, and CWE-116 for GPT-4o.Source: 2) CWE-Level Security Weakness Analysis · p. 8

C. Task Relevance and Security-Driven Semantic Drift

All sampled outputs remained task-relevant, but stronger security-oriented prompts increasingly caused models to rewrite or remove explicitly required unsafe constructs. This semantic drift was much larger for GPT-4o than for LLaMA 3.1-8B.

Task Relevance

Security-Driven Semantic Drift

  • Security-driven semantic drift occurs when relevant outputs alter or remove explicitly required unsafe constructs.Models sometimes substitute safer alternatives, such as replacing eval() with AST-based approaches, while still addressing the core task objective.Source: C. Task Relevance and Security-Driven… · p. 8, Table II · p. 8
  • 20%: semantic drift appears under the baseline structured prompt and becomes more prevalent under security-guided and adversarial prompts.The drift probability increases 3× under security-guided prompts.Source: C. Task Relevance and Security-Driven… · p. 8
  • 60–67%: GPT-4o semantic drift rises from 20% under Prompt 1.0 to 60–67% under Prompts 2.0–4.0.This indicates stronger prioritization of security compliance over adherence to task-specific API requirements as constraints intensify.Source: C. Task Relevance and Security-Driven… · p. 8
  • Security-oriented prompting often changes implementation strategies even when security-relevant issues remain present.The increase in semantic drift is substantially larger than the observed reduction in security weakness prevalence.Source: C. Task Relevance and Security-Driven… · p. 8
  • 7% to 20–27%: LLaMA 3.1-8B semantic drift increases more gradually than GPT-4o across prompt variants.LLaMA more reliably preserves exact API usage and task semantics, but shows fewer consistent reductions in detected weaknesses.Source: C. Task Relevance and Security-Driven… · p. 8

Trade-off

  • Security-oriented prompt refinement introduces a trade-off between safer implementations and adherence to explicit task requirements.The relevance audit found sampled outputs remained aligned with the original programming objectives.Source: C. Task Relevance and Security-Driven… · p. 8

D. Qualitative Analysis of Unflagged Security-Relevant Outputs

Manual review of outputs unflagged by Bandit and CodeQL found security-relevant behaviours involving program semantics, runtime behaviour, and application context. These observations make tool-derived prevalence a lower-bound estimate of risk.

Review Scope

Unflagged Security-Relevant Behaviours

  • Unflagged examples included user-controlled SQL execution, password-comparison timing side channels, and interactive Python consoles exposing arbitrary code execution.These behaviours did not necessarily rely on commonly flagged functions such as eval().Source: D. Qualitative Analysis of Unflagged… · p. 9
  • Generated programs sometimes exposed sensitive system functionality through command-dispatch or information-retrieval interfaces without traditional command-injection APIs.This illustrates security-relevant behaviour that may evade pattern-based rules.Source: D. Qualitative Analysis of Unflagged… · p. 9

Interpretation

  • Bandit- and CodeQL-derived security-weakness prevalence should be interpreted as a lower-bound estimate of security risk.Program logic, resource consumption, and application-specific context may require complementary analysis beyond conventional static scanning.Source: D. Qualitative Analysis of Unflagged… · p. 9

V. DISCUSSION

Structured prompting primarily improves compliance and redistributes security risk rather than consistently reducing weakness prevalence. These effects are model-dependent, can alter task semantics, and do not replace post-generation security controls.

  • Structured prompting primarily acts as a compliance and behavioural-shaping mechanism, while overall weakness prevalence changes modestly and inconsistently.For GPT-4o, stronger prompts enabled large-scale analysis by reducing refusals, but changed the analyzable-output denominator.Source: §V · p. 9
  • Security-oriented prompts did not consistently reduce security weakness prevalence when analysis was conditioned on valid Python code.Source: §V · p. 9
  • For GPT-4o, stronger prompts shifted findings toward lower severity, whereas LLaMA showed weaker or inconsistent effects.The GPT-4o shift lowered high-severity findings and raised low-severity findings; the supplied passage does not provide the corresponding percentages here.Source: §V · p. 9
  • Security-driven semantic drift can replace explicitly requested unsafe constructs with safer alternatives, reducing static findings while altering intended implementations.This creates a trade-off between security compliance and fidelity to the original task specification.Source: §V · p. 9
  • Static analysis, code review, and security testing remain necessary because static analyzers may miss vulnerabilities and cannot directly assess runtime exploitability.The evaluation used Bandit and CodeQL, which may produce false positives and false negatives.Source: §V · p. 9
  • The findings are bounded by synthetic security-oriented Python tasks, two models, and a single output per configuration.These constraints limit generalizability and variance estimation; broader languages, models, and multi-sample evaluation are proposed for future work.Source: §V · p. 9, §VI · p. 10
Loading 2608.24857v1…