Source-linked AI summary
Can LLMs Patch Security Issues?
Kamel Alrashedy, Abdullah Aljasser, Pradyumna Tambwekar, Matthew Gombolay
TL;DR
LLMs can generate vulnerable code, creating a need for secure-code evaluation and effective patching feedback. The paper proposes FDSP, which uses static analysis to guide LLM-generated repairs, and introduces PythonSecurityEval for diverse real-world applications. FDSP outperforms prior self-feedback methods by up to 17.6% on PythonSecurityEval.
Problem
LLM-generated code can contain vulnerabilities that expose sensitive data or systems, while training-based patching requires costly labeled data and expert feedback.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Static-analysis feedback can support LLM-based refinement of vulnerable code across benchmarks and models evaluated in the paper.
Source: §3 · p. 3, §6 · p. 8The evaluation may miss security issues, focuses on function-level code, and fixes may inadvertently make programs incorrect.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown impressive proficiency in code generation. Unfortunately, these models share a weakness with their human counterparts: producing code that inadvertently has security vulnerabilities. These vulnerabilities could allow unauthorized attackers to access sensitive data or systems, which is unacceptable for safety-critical applications. In this work, we propose Feedback-Driven Security Patching (FDSP), where LLMs automatically refine generated, vulnerable code. Our approach leverages automatic static code analysis to empower the LLM to generate and implement potential solutions to address vulnerabilities. We address the research communitys needs for safe code generation by introducing a large-scale dataset, PythonSecurityEval, covering the diversity of real-world applications, including databases, websites and operating systems. We empirically validate that FDSP outperforms prior work that uses self-feedback from LLMs by up to 17.6% through our procedure that injects targeted, external feedback. Code and data are available at \url{https://github.com/Kamel773/LLM-code-refine}
1 Introduction
LLMs can generate vulnerable code, motivating FDSP, which uses static-analysis feedback to refine code and PythonSecurityEval to evaluate secure-code generation. FDSP outperforms self-feedback baselines by up to 17.6% on PythonSecurityEval.
Motivation
- LLM-generated vulnerabilities can enable unauthorized access to sensitive data or systems, including through SQL injection.The paper identifies limited security knowledge and difficulty handling code that interacts with external services as challenges for LLMs.Source: §1 · pp. 1–2
- Training LLMs to patch vulnerabilities requires costly human-labeled data and accurate security feedback, motivating automated feedback mechanisms.The dataset must distinguish vulnerable from non-vulnerable code, while feedback must be accurate enough to support learning.Source: §1 · pp. 1–2
FDSP
- FDSP uses Bandit-generated security feedback to help LLMs generate solutions and refine vulnerable code.The approach analyzes generated code, feeds identified threats to the LLM, and returns proposed solutions for further refinement.Source: Figure 1 · p. 1, §1 · p. 2
PythonSecurityEval
- PythonSecurityEval contains 470 natural-language prompts for evaluating whether LLMs generate secure code.Its real-world application coverage includes databases, URLs, operating systems, and websites, with vulnerabilities such as SQL injection, XSS, broken access control, and command injection.Source: §1 · p. 2
Results
- Up to 17.6%, FDSP’s improvement over prior self-feedback methods on PythonSecurityEval demonstrates its reported benchmark advantage.The evaluation compares LLMs and CodeLlama across three benchmarks using five baseline techniques.Source: §1 · p. 2
2 Related work
Related work spans code-generation capabilities and methods that refine LLM outputs using self-feedback, external tools, or human input. FDSP extends this refinement line toward security repair by combining static-analysis feedback with LLM-generated solutions.
Language models for code
- Code-focused LLMs support generation, translation, repair, documentation, testing, and defect prediction, motivating their use for security issues.Examples include CodeLlama and CodeGeeX for code generation and broader applications across software-engineering tasks.Source: §2 · p. 2
Refinement of LLMs
- Self-refinement methods use LLM feedback, unit tests, compiler errors, or explanations to improve generated code and other outputs.Self-Refine, Self-Debugging, and Self-Repair represent feedback-driven refinement approaches.Source: §2 · pp. 2–3
- External tools can provide feedback for program repair and coding standards, while human feedback is effective but costly and time-intensive.Prior examples include compiler messages, Pylint, code interpreters, calculators, and search engines.Source: §2 · pp. 2–3
- FDSP combines external static-analysis feedback with LLM refinement to address security threats in generated code.Bandit supplies feedback about the generated code, which the LLM uses to produce potential security solutions.Source: §2 · p. 3
3 Our Approach
FDSP combines static analysis with LLM-generated candidate solutions and iterative refinement to identify and repair vulnerabilities in generated Python code.
3 Our Approach
- FDSP uses four steps—code generation, code testing, solution generation, and code refinement—to identify and resolve vulnerabilities in LLM-generated code.Bandit supplies the static-analysis feedback used throughout the repair process.Source: §3 · p. 3
3.1 Code generation
- In zero-shot code generation, an LLM produces a Python program from a natural-language description without examples.The approach assumes users directly prompt LLMs and initially assumes the generated code is secure.Source: §3.1 · p. 3
3.2 Code testing
- Bandit constructs an abstract syntax tree, analyzes Python code for common vulnerabilities, and returns a report passed to the LLM.The report identifies potential security threats; static analysis itself does not repair them.Source: §3.2 · p. 3, §3.3 · p. 4
3.3 Generating potential solutions
- The LLM generates multiple potential solutions from Bandit’s vulnerability report, such as parameterized queries, escaping table names, or using an ORM.The number of candidate solutions is denoted by J.Source: §3.3 · p. 4
3.4 Code refinement
- Each candidate solution and the vulnerable code are fed back to the LLM for K iterations, stopping when Bandit finds no issue or K is reached.The iterative process seeks additional fixes for each candidate solution.Source: §3.4 · p. 4
4 Experimental Settings
The experiments introduce a broader Python security benchmark, compare FDSP with prompting and feedback-based baselines, and evaluate vulnerability generation and repair across several LLMs and research questions.
4.1 Benchmarks
- PythonSecurityEval contains 470 Stack Overflow-derived prompts spanning diverse real-world applications, addressing the limited size and diversity of earlier benchmarks.The benchmark covers database, operating-system, and web applications involving vulnerabilities such as SQL injection, command injection, and XSS.Source: §4.1 · pp. 4–5
4.2 Baselines
- The baselines include direct prompting, self-debugging, Bandit feedback, and verbalized Bandit feedback for refining vulnerable code.Bandit feedback identifies problematic lines and issue types, while verbalization explains the findings and may suggest solutions.Source: §4.2 · p. 5
4.3 Evaluation metrics
- Vulnerability rates are evaluated as the ratio of vulnerable code to total generated code using Bandit and CodeQL.Bandit detects vulnerabilities in Python code, while CodeQL is used to identify similar vulnerability patterns.Source: §4.3 · pp. 5–6
4.4 Models
- The study evaluates GPT-4, GPT-3.5 turbo-instruct, and CodeLlama for secure code generation and refinement with FDSP and the four baselines.The models are assessed on their ability to generate secure code and refine insecure code.Source: §4.4 · p. 6
4.5 Research Questions
- The research questions examine inherent vulnerability refinement, Bandit’s effect, FDSP’s iterative multi-solution repair, and the roles of multiple solutions and iterations.The final question uses ablations restricting FDSP to one solution or one iteration.Source: §4.5 · p. 6
5 Experimental Results
The experiments evaluate LLM refinement across multiple benchmarks, models, vulnerability detectors, and FDSP variants. External Bandit feedback and FDSP generally outperform simpler refinement approaches, while unresolved vulnerabilities remain concentrated in injection-related issues.
RQ1: Intrinsic refinement
- Direct prompting and self-debugging provide limited gains, improving Bandit results by less than 10% for GPT-3.5 and CodeLlama and around 15% for GPT-4.Across datasets, Bandit detects 28%–46% of generated code as vulnerable, while CodeQL detects 9.1%–17%; direct prompting and self-debugging fix approximately 50% of CodeQL-detected vulnerabilities.Source: §5 · p. 6
RQ2: Bandit-based feedback
- Bandit feedback improves refinement accuracy by approximately 30% for GPT-4 and up to 24% for GPT-3.5 and CodeLlama across Bandit evaluations.Verbalizing Bandit feedback adds only 1% to 2% in both evaluation metrics.Source: §5 · p. 6, Table 2 · p. 7
RQ3: FDSP versus baselines
- FDSP improves over verbalized Bandit feedback for GPT-4, GPT-3.5, and CodeLlama on PythonSecurityEval, reducing vulnerable-code percentages from 8.7% to 7.4%, 23.6% to 15.7%, and 21% to 13.6%, respectively.These results support generating potential fixes from static-analysis feedback rather than merely passing that feedback to the model.Source: §5 · p. 7
RQ3: Security-issue analysis
- For CodeLlama, self-refinement and directly passing Bandit feedback are not useful across the five most frequent issue types, whereas verbalization and FDSP perform well.Figure 3 compares the total counts of the five most frequent security issues across five refinement approaches.Source: §5 · p. 7, Figure 3 · p. 7
RQ4: Ablation study
- FDSP ablations increase unsolved vulnerabilities from 7.4% to 9.5% with one iteration and 10.0% with one solution, showing multiple solutions matter more than repeated iterations.The ablations use GPT-4 on PythonSecurityEval.Source: §5 · pp. 7–8, Table 3 · p. 7
Qualitative analysis
- Among 30 manually evaluated GPT-4 examples, 23 were fixed and 7 remained unfixed; 74% of solutions offered at least two actionable recommendations.The seven failures included SQL injection and high-risk library-call cases, although most fixed vulnerabilities were resolved in the first iteration.Source: §5 · p. 8
Unresolved vulnerabilities
- CWE-259 and CWE-400 are the most common generated vulnerabilities, while unresolved cases most often involve CWE-78 and CWE-89, comprising 61.1% and 80.0% for GPT-4.The unresolved-vulnerability comparison covers three LLMs on PythonSecurityEval.Source: §5 · p. 8, Figure 5 · p. 8, Figure 4 · p. 8
6 Conclusion
The paper evaluates vulnerable-code generation and proposes FDSP, which uses static-analysis feedback to help LLMs repair their own vulnerable outputs. Across three benchmarks and three models, FDSP outperforms the baselines, improving GPT-4 performance on PythonSecurityEval by about 17.6%.
Conclusion
- FDSP uses static code-analysis feedback to generate potential fixes for LLM-produced vulnerable code, while PythonSecurityEval contains 470 natural-language prompts for secure-code evaluation.The approach and dataset address limitations in existing refinement methods and security-evaluation resources.Source: §6 · p. 8
7 Limitations
The evaluation has three important boundaries: it may miss vulnerabilities, operates at the function level, and may introduce correctness problems when modifying code that interacts with external systems.
7 Limitations
- The evaluation may fail to identify all security issues because some vulnerabilities remain undetectable by static analysis and deep-learning methods.The authors note that vulnerability detection remains an active research area.Source: §7 · p. 9
- The study evaluates vulnerable-code generation and refinement at the function level, although real-world vulnerabilities often arise from interactions among code, services, databases, or operating systems.Source: §7 · p. 9
- Code changes intended to fix security issues may inadvertently make programs incorrect, while ensuring correctness and writing unit tests is difficult for code interacting with external systems.Source: §7 · p. 9
A Appendix
The appendix illustrates FDSP feedback and generated solutions, and documents domain classification, CWE descriptions, and SQL-injection correction examples.
A Appendix
- Figure 6 illustrates how GPT-4 verbalizes feedback from the Bandit static analyzer.Source: Figure 6 · p. 11
- Figure 7 presents an example solution generated for the security issues shown in Figure 2.Source: Figure 7 · p. 11
- Table 4 describes domain classification for functions by identifying calls to domain-specific libraries.Source: Table 4 · p. 12
- Table 5 lists Common Weakness Enumeration IDs and their descriptions.Source: Table 5 · p. 12
- Table 6 gives corrected potential solutions generated by FDSP for SQL-injection vulnerabilities that GPT-4 could not refine.Source: Table 6 · p. 13