PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code

Maitreyee Das Urmi, Jessica Pourleyli, Fabio Santos, Glaucia Melo

arXiv:2608.24857v1cs.CRcs.SE

TL;DR

This study examines whether structured and security-oriented prompts improve the security of LLM-generated code beyond influencing compliance. Using 424 security-sensitive Python tasks with GPT-4o and LLaMA 3.1-8B, it finds that prompting primarily redistributes weaknesses rather than reliably reducing them, while security controls remain necessary.

  • Problem

    Existing research emphasizes functional correctness and aggregate reductions in static-analysis findings, leaving limited evidence about how prompt structure affects the distribution and manifestation of security weaknesses in generated code.

  • Method

    The study generates code for 424 security-sensitive Python tasks under five progressively structured and security-oriented prompts, evaluating compliance and security findings with Bandit and CodeQL across GPT-4o and LLaMA 3.1-8B.

  • Results

    Structured prompts substantially improve compliance but do not consistently reduce weakness prevalence; GPT-4o shifts findings toward lower severity, whereas LLaMA shows weaker, less consistent effects.

  • Takeaways & Limitations

    Prompt structure is primarily a compliance and behavioural-shaping mechanism, so prompt engineering should complement rather than replace post-generation security controls.

  • Takeaways & Limitations

    The evaluation relies on Bandit and CodeQL, is limited to Python tasks from one vulnerability-focused dataset, and compares only GPT-4o and LLaMA 3.1-8B.

Abstract

from arXiv · show

Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.

I. INTRODUCTION

This study examines prompt structure as a security-related influence on LLM-generated code, asking whether it changes compliance, weakness prevalence, severity, and CWE distributions. It frames the dominant effect as redistribution rather than reduction.

  • Research gap: Existing research emphasizes functional correctness, while security-focused prompting studies largely report aggregate weakness counts rather than severity or composition.This leaves uncertainty about whether apparent security improvements reflect reduced risk or shifted weakness profiles.
  • Research questions and approach: The study evaluates prompt structure across identical security-sensitive programming tasks using proprietary and open-weight LLMs, Bandit, and CodeQL.Its two research questions separate code-generation compliance from security analysis conditioned on valid code.
  • Contributions: The central claim is that prompt structure primarily redistributes security risk rather than reducing it outright.This challenges the aggregate-reduction narrative in secure code generation research.
  • Research questions and approach: Structured prompting is expected to influence whether models produce analyzable code and how remaining weaknesses are distributed across prevalence, severity, and CWE categories.The framework distinguishes valid executable Python from refusal or non-code outputs before comparing security findings.
  • Contributions: The paper reports that prompt refinements can substantially alter analyzable-output populations, making weakness comparisons sensitive to changes in compliance.Its contributions include a two-stage evaluation framework and a semantic-drift audit.

A. LLM-Generated Code Quality and Security Weaknesses

Prior work shows that LLM-generated code can remain vulnerable despite functional plausibility, while prompt-based security interventions are usually assessed with aggregate metrics. This paper motivates severity- and composition-aware analysis.

  • LLM-generated code quality and security: LLM-generated programs may be syntactically correct and functionally plausible while still containing unsafe patterns and vulnerabilities.Security weaknesses can persist even when functional tests pass.
  • Prompt engineering: Prompt engineering improves correctness, specification adherence, consistency, maintainability, and reliability across software-engineering tasks.Structured prompts can also improve static quality metrics and reduce certain defect classes, although security misconfigurations may persist.
  • Research gap: Existing studies typically use aggregate vulnerability counts and do not examine generation compliance or the composition of remaining weaknesses.This limits conclusions about whether apparent improvements represent true risk reduction or shifts in severity and issue type.
  • Security-aware code generation: Security-aware prompting, iterative self-repair, and static-analysis feedback loops have generally been reported to reduce detected security weaknesses relative to baseline prompting.Prior critiques caution that separating security from functionality or relying on one static analyzer can yield incomplete assessments.
  • Research gap: The paper addresses this gap by analyzing severity distributions rather than counts alone, including trade-offs that binary metrics may miss.Its focus is on how prompt interventions reallocate risk across severity levels.

III. METHODOLOGY

The study systematically varies prompt structures, generates code from security-sensitive Python tasks with GPT-4o and LLaMA 3.1-8B, and evaluates outputs through staged static analysis. The pipeline separates generation success from weakness assessment.

  • Experimental pipeline: The methodology uses three stages: prompt design, code generation and collection, and security weakness assessment with Bandit and CodeQL.Figure 1 summarizes this experimental pipeline.
  • Models and environment: The evaluation compares GPT-4o and LLaMA 3.1-8B as proprietary high-capacity and open-weight model examples.Experiments were conducted in a controlled local environment for consistency across models, prompts, and tools.
  • Security analysis: Bandit supplies rule-based LOW, MEDIUM, and HIGH severity annotations, while CodeQL provides semantic, query-driven vulnerability discovery.Using both tools provides complementary static-analysis coverage.
  • Generation and collection: The experiments record whether each prompt–model interaction yields executable Python code or a refusal/non-code response.This enables compliance analysis before downstream security comparisons.
  • Dataset: The dataset contains 424 Python examples filtered from the Cybernative.ai Code Vulnerability and Security Dataset.The tasks instruct models to produce secure and valid Python solutions.
  • Dataset: Tasks are security-sensitive because they involve vulnerability-prone scenarios such as command execution, file handling, deserialization, cryptography, and input validation.These operations create opportunities for insecure implementations assessable through static analysis.

C. Prompt Design

The prompt design holds the user task constant while varying system-level structure and guidance. All variants emphasize code-only, senior-engineer responses with formatting, quality, and error-handling constraints.

  • C. Prompt Design: The prompt templates vary system structure while maintaining a consistent output format across interactions.Each interaction contains a system prompt and a user prompt.
  • 1) User Prompt (Invariant):: The user prompt remains invariant across experiments and asks for a complete, correct solution to an insecure coding question.This controls the task content while prompt variants change system-level instructions.
  • 1) User Prompt (Invariant):: Every prompt variant assigns the model the role of Senior Software Engineer rather than Security Engineer.The chosen persona emphasizes task completion over post hoc vulnerability analysis.
  • 2) Base Prompt Specifications:: Each system prompt states that a coding task follows and explicitly requires code-only output.These constraints were added because pilot models often returned explanations instead of implementations.
  • 2) Base Prompt Specifications:: Prompts enforce Python best practices, modular design, and explicit error handling to promote production-level quality and reduce code smells.The constraints also support clarity, consistency, and modularity.
  • 2) Base Prompt Specifications:: Strict output formatting enables automated extraction and evaluation.Formatting constraints make generated responses easier to process consistently.

3) Prompt Variants and Strategy:

The study builds five controlled prompt variants, progressively adding structural constraints and security framing to test their effects on code generation and security.

  • Prompt 0.0 instructs the model to generate complete, correct solutions as a senior software engineer.
  • All variants preserve style and output format while incrementally modifying structural guidance and security emphasis.
  • Prompt 1.0 adds code-only output constraints, persona, context, and style and quality guidance to target refusals.
  • Prompt 2.0 adds secure-coding guidance referencing OWASP, CERT, and MITRE Top-25 while retaining the instruction to minimize insecurities.
  • Prompt 3.0 removes named security frameworks but retains secure-code and issue-minimization instructions, isolating security framing from standards references.
  • Prompt 4.0 adds an adversarial context involving malicious inputs or untrusted users to evaluate defensive-programming effects.

4) Prompt Refinement Rationale and Termination Criterion:

The prompt-refinement design systematically varies structural guidance, security framing, and adversarial awareness while holding generation and analysis procedures consistent.

  • 4) Prompt Refinement Rationale and Termination Criterion:: Prompt 0.0 establishes the baseline, while Prompts 1.0–4.0 test structure, security guidance, framework references, framework-free framing, and adversarial awareness.
  • 4) Prompt Refinement Rationale and Termination Criterion:: The design space is sampled along one conceptual axis from none to structure, security guidance, security without frameworks, and adversarial awareness.
  • D. Code Generation and Data Collection: Each model generates one candidate solution for every task and prompt variant, covering 424 tasks, 5 variants, and 2 models.
  • D. Code Generation and Data Collection: Generations use zero-shot prompting, temperature 0.7, and top-p 0.2 for both models.
  • D. Code Generation and Data Collection: Each configuration runs once, with raw outputs stored and parsed using Python’s ast.parse() to distinguish executable code from refusals or non-code.
  • E. Security Weakness Analysis with Bandit and CodeQL: Bandit detects Python-specific insecure patterns and severity, while CodeQL adds semantic and data-flow analysis.
  • E. Security Weakness Analysis with Bandit and CodeQL: CWE analysis focuses on the top 10 most frequent Common Weakness Enumeration identifiers reported by Bandit and CodeQL.

F. Analysis and Validation

The analysis separates generation compliance from security analysis, then validates severity and weakness-type patterns under controlled experimental conditions.

  • Stage 1 measures invalid or non-code outputs and files containing security weaknesses across all prompt variants.
  • Stage 2 analyzes severity and CWE distributions only among executable Python outputs because refusals cannot be meaningfully statically analyzed.
  • The study first analyzes GPT-4o, then repeats severity and CWE analyses on filtered LLaMA outputs and a combined subset.
  • Dataset, prompts, model parameters, hardware, and analysis tools remain constant, while one generation per configuration limits variance estimation.
  • A relevance audit samples 15 problems to check that refinements preserve requested tasks and do not create degenerate outputs.
  • Manual inspection verifies that required functionality is implemented and excludes empty stubs, constant-return programs, and unrelated templates.

A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence

Stage 1 evaluates how prompt variants change analyzable output compliance and the prevalence of Bandit-detected weaknesses across GPT-4o and LLaMA 3.1-8B.

  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: Figure 3 summarizes Bandit outcomes for GPT-4o and LLaMA 3.1-8B across 424 tasks per prompt.
  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: 338 of 424 GPT-4o Prompt 0.0 outputs are invalid or non-code, leaving 86 valid files, including 59 flagged as vulnerable.
  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: GPT-4o Prompt 1.0 reduces invalid outputs from 338 to 44, with Prompts 2.0–4.0 maintaining 37–52 invalid files.
  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: LLaMA invalid outputs fall from 126 to 51 under Prompt 1.0, then rise to 115, 89, and 150 under Prompts 2.0–4.0.
  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: GPT-4o vulnerable-file counts decrease modestly from 289 under Prompt 1.0 to 265 under Prompt 4.0.
  • A. Stage 1 (RQ1): Prompt-Level Generation Compliance and Security Weakness Prevalence: LLaMA vulnerable-file counts fluctuate from 279 to 212, 244, and 176 across Prompts 1.0–4.0, without a clear monotonic trend.
  • B. Stage 2 (RQ2): Security Weakness Severity and Type Analysis: Stage 2 conditions severity and type analysis on successful code generation, retaining only syntactically valid Python files.

1) Severity Distribution Across Prompt Variants:

Across prompt variants, GPT-4o shifts detected findings toward lower severity, whereas LLaMA 3.1-8B shows comparatively stable severity distributions. CWE-level changes are uneven: some categories decline, but dominant weaknesses remain prevalent.

  • 1) Severity Distribution Across Prompt Variants:: 20.8% to 13.6%: GPT-4o high-severity findings decrease from Prompt 1.0 to Prompt 4.0.Low-severity findings increase from 32.0% to 43.5%, while medium-severity findings decrease from 47.1% to 43.0%.
  • 1) Severity Distribution Across Prompt Variants:: Security-oriented prompting shifts GPT-4o findings toward lower severity rather than eliminating security weaknesses outright.The severity shift reflects changes in the types of weaknesses reported, because severity levels are not uniformly distributed across weakness classes.
  • 1) Severity Distribution Across Prompt Variants:: 21.2%, 22.9%, 21.5%, and 24.4%: LLaMA 3.1-8B high-severity findings remain relatively stable across Prompts 1.0–4.0.Unlike GPT-4o, LLaMA exhibits no comparable monotonic trend, indicating model-dependent prompt effects.
  • 2) CWE-Level Security Weakness Analysis:: CWE-78 and CWE-502 remain highly prevalent across prompt conditions despite prompt refinement.For GPT-4o, CWE-78 increases from 43.9% to 49.7%, while CWE-502 decreases from 23.7% to 21.0% between Prompts 1.0 and 4.0.
  • 2) CWE-Level Security Weakness Analysis:: 10.1% to 4.1%: GPT-4o CWE-94 findings decrease from Prompt 1.0 to Prompt 4.0, although the trend is not strictly monotonic.CWE-95 and CWE-116 also decline, while the same classes remain comparatively stable for LLaMA 3.1-8B.
  • 2) CWE-Level Security Weakness Analysis:: Information-exposure and debugging-related CWE classes exhibit non-monotonic trends across prompt variants.These irregular patterns indicate that prompt refinement does not uniformly influence those weaknesses.
  • 2) CWE-Level Security Weakness Analysis:: CWE-330 remains effectively unchanged across prompt variants for both models.This stability further indicates that prompt effects vary by security weakness class.
  • 2) CWE-Level Security Weakness Analysis:: Prompt refinement influences security-weakness composition more strongly than overall prevalence.The largest decreases concentrate in a relatively small subset of categories, particularly CWE-94, CWE-95, and CWE-116 for GPT-4o.

C. Task Relevance and Security-Driven Semantic Drift

All sampled outputs remained task-relevant, but stronger security-oriented prompts increasingly caused models to rewrite or remove explicitly required unsafe constructs. This semantic drift was much larger for GPT-4o than for LLaMA 3.1-8B.

  • Task Relevance: All generated outputs in the 15-task sample were task-relevant across both models and Prompts 1.0–4.0.No outputs were classified as unrelated or nonsensical.
  • Security-Driven Semantic Drift: Security-driven semantic drift occurs when relevant outputs alter or remove explicitly required unsafe constructs.Models sometimes substitute safer alternatives, such as replacing eval() with AST-based approaches, while still addressing the core task objective.
  • Security-Driven Semantic Drift: 20%: semantic drift appears under the baseline structured prompt and becomes more prevalent under security-guided and adversarial prompts.The drift probability increases 3× under security-guided prompts.
  • Security-Driven Semantic Drift: 60–67%: GPT-4o semantic drift rises from 20% under Prompt 1.0 to 60–67% under Prompts 2.0–4.0.This indicates stronger prioritization of security compliance over adherence to task-specific API requirements as constraints intensify.
  • Security-Driven Semantic Drift: Security-oriented prompting often changes implementation strategies even when security-relevant issues remain present.The increase in semantic drift is substantially larger than the observed reduction in security weakness prevalence.
  • Security-Driven Semantic Drift: 7% to 20–27%: LLaMA 3.1-8B semantic drift increases more gradually than GPT-4o across prompt variants.LLaMA more reliably preserves exact API usage and task semantics, but shows fewer consistent reductions in detected weaknesses.
  • Trade-off: Security-oriented prompt refinement introduces a trade-off between safer implementations and adherence to explicit task requirements.The relevance audit found sampled outputs remained aligned with the original programming objectives.

D. Qualitative Analysis of Unflagged Security-Relevant Outputs

Manual review of outputs unflagged by Bandit and CodeQL found security-relevant behaviours involving program semantics, runtime behaviour, and application context. These observations make tool-derived prevalence a lower-bound estimate of risk.

  • Review Scope: Manual review examined outputs that Bandit and CodeQL did not flag for detectable weaknesses.The review targeted limitations of static-analysis-based evaluation.
  • Unflagged Security-Relevant Behaviours: Unflagged examples included user-controlled SQL execution, password-comparison timing side channels, and interactive Python consoles exposing arbitrary code execution.These behaviours did not necessarily rely on commonly flagged functions such as eval().
  • Unflagged Security-Relevant Behaviours: Generated programs sometimes exposed sensitive system functionality through command-dispatch or information-retrieval interfaces without traditional command-injection APIs.This illustrates security-relevant behaviour that may evade pattern-based rules.
  • Interpretation: Bandit- and CodeQL-derived security-weakness prevalence should be interpreted as a lower-bound estimate of security risk.Program logic, resource consumption, and application-specific context may require complementary analysis beyond conventional static scanning.

V. DISCUSSION

Structured prompting primarily improves compliance and redistributes security risk rather than consistently reducing weakness prevalence. These effects are model-dependent, can alter task semantics, and do not replace post-generation security controls.

  • Structured prompting primarily acts as a compliance and behavioural-shaping mechanism, while overall weakness prevalence changes modestly and inconsistently.For GPT-4o, stronger prompts enabled large-scale analysis by reducing refusals, but changed the analyzable-output denominator.
  • Security-oriented prompts did not consistently reduce security weakness prevalence when analysis was conditioned on valid Python code.
  • For GPT-4o, stronger prompts shifted findings toward lower severity, whereas LLaMA showed weaker or inconsistent effects.The GPT-4o shift lowered high-severity findings and raised low-severity findings; the supplied passage does not provide the corresponding percentages here.
  • Security-driven semantic drift can replace explicitly requested unsafe constructs with safer alternatives, reducing static findings while altering intended implementations.This creates a trade-off between security compliance and fidelity to the original task specification.
  • Static analysis, code review, and security testing remain necessary because static analyzers may miss vulnerabilities and cannot directly assess runtime exploitability.The evaluation used Bandit and CodeQL, which may produce false positives and false negatives.
  • The findings are bounded by synthetic security-oriented Python tasks, two models, and a single output per configuration.These constraints limit generalizability and variance estimation; broader languages, models, and multi-sample evaluation are proposed for future work.
Loading 2608.24857v1…