Source-linked AI summary
A Study of Vulnerability Repair in JavaScript Programs with Large Language Models
Tan Khang Le, Saba Alimadadi, Steven Y. Ko
TL;DR
This study addresses limited evidence on LLM repair of JavaScript vulnerabilities, a setting where analyzing programs and manually repairing bugs can be challenging. It tests ChatGPT and Bard with prompts varying in vulnerability context; both produce many correct repairs, and added context is associated with better accuracy. The findings are bounded by the tested public model versions and manual evaluation.
Problem
JavaScript vulnerability analysis and repair can be challenging, while prior LLM repair research has provided limited insight into dynamic JavaScript programs.
Source: §1 · p. 1Method
The study tests ChatGPT and Bard on JavaScript vulnerability repair using prompts with no extra context, a vulnerability name, or explanatory comments.
Source: §1 · p. 1, §3.1 · p. 2Results
Takeaways & Limitations
In this study, the amount of context in a repair prompt is relevant to the accuracy of LLM-generated JavaScript vulnerability patches.
Source: §4.2 · p. 4, §5 · p. 4The evaluation used public model versions accessed in November 2023 and manually verified repairs, which is subject to human bias.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
In recent years, JavaScript has become the most widely used programming language, especially in web development. However, writing secure JavaScript code is not trivial, and programmers often make mistakes that lead to security vulnerabilities in web applications. Large Language Models (LLMs) have demonstrated substantial advancements across multiple domains, and their evolving capabilities indicate their potential for automatic code generation based on a required specification, including automatic bug fixing. In this study, we explore the accuracy of LLMs, namely ChatGPT and Bard, in finding and fixing security vulnerabilities in JavaScript programs. We also investigate the impact of context in a prompt on directing LLMs to produce a correct patch of vulnerable JavaScript code. Our experiments on real-world software vulnerabilities show that while LLMs are promising in automatic program repair of JavaScript code, achieving a correct bug fix often requires an appropriate amount of context in the prompt.
1 INTRODUCTION
JavaScript’s heterogeneous, dynamic, asynchronous nature makes programs challenging to analyze, and developers can introduce security vulnerabilities. This study examines ChatGPT and Bard for JavaScript vulnerability repair and investigates how prompt context affects patch accuracy.
1 INTRODUCTION
- Prior LLM vulnerability-repair studies mostly examined languages such as C/C++ and Verilog, leaving limited insight into repair for dynamic JavaScript.JavaScript analysis is challenging because programs are heterogeneous, dynamic, and asynchronous, while conventional analysis and repair can be tedious.Source: §1 · p. 1
- The study asks how accurately LLMs find and fix JavaScript vulnerabilities and how prompt-context amount affects correct patch generation.It investigates black-box, off-the-shelf ChatGPT and Bard.Source: §1 · p. 1
- With 20 JavaScript-relevant vulnerabilities and three prompt templates, ChatGPT averaged 71.66% accuracy and Bard 68.33%; more context improved accuracy by up to 55%.The study formulates 60 repair prompts.Source: §1 · p. 1
2 RELATED WORK
The related work reviews code-security research and automated program repair, from search- and semantics-based approaches to newer LLM vulnerability-repair studies. It positions this study within a growing body of LLM repair research.
2 RELATED WORK
- The section surveys prior work on code security, vulnerabilities, and LLM-based program repair.Source: §2 · p. 2
2.1 Code Security and Vulnerability
- Automated repair has used genetic programming to search for patches and semantic analysis to construct patches.The section frames repair as producing source-code patches for bugs.Source: §2.1 · p. 2
- CWE categorizes software and hardware weaknesses, and MITRE’s annual CWE Top 25 ranks dangerous software weaknesses.The CWE contains over 600 categories, including buffer overflows, race conditions, and cross-site scripting.Source: §2.1 · p. 2
2.2 Program Repair with Large Language Models (LLMs)
- LLM repair research spans Java vulnerabilities [18], Verilog hardware bugs [1], and security repair in Python, C, and Verilog.Other work uses controlled code generation for secure code [6] and detects and patches side-channel vulnerabilities [17].Source: §2.2 · p. 2
3 METHODOLOGY
The methodology evaluates off-the-shelf LLM repairs of JavaScript snippets using prompts with different amounts of vulnerability context. It selects JavaScript-relevant weaknesses from the 2023 CWE Top 25.
3 METHODOLOGY
- The experiment prompts LLMs to find and fix vulnerabilities in supplied JavaScript snippets and evaluates whether their repairs are correct.Table 1 gives an example prompt and ChatGPT response.Source: §3 · p. 2, Table 1 · p. 2
- A repair counts as successful when the LLM produces a functional, secure patch; the study does not seek to enumerate every possible fix.The example fixes an ineffective write at index 3 in an Int16Array of length 3 by increasing its length to 4.Source: §3 · p. 2
3.1 Prompt Design
- The three prompt templates provide no extra context (c.f.), the vulnerability name (c.s.), or explanatory comments about the vulnerability and its exploitation (c.r.).Table 2 shows sample templates.Source: §3.1 · pp. 2–3, Table 2 · p. 3
3.2 Vulnerability Selection
- The study selects 20 of the 2023 CWE Top 25 vulnerabilities as most relevant to JavaScript, excluding weaknesses specific to other languages.CWE-416, Use After Free, is given as a C/C++-specific example.Source: §3.2 · p. 3, Table 3 · p. 3
4 EXPERIMENT AND EVALUATION
The experiment tests ChatGPT and Bard on 20 vulnerabilities using three prompt-context levels. Accuracy is higher with added context, while the evaluation is bounded by evolving public model versions and manual correctness judgments.
4.1 Experiment Details
- The experiment tests ChatGPT and Bard on 60 prompts: 20 vulnerabilities are each paired with three templates spanning no context to comprehensive contextual cues.Each prompt is run in a separate conversation to avoid potential bias from continuous dialogue.Source: §4.1 · p. 3, §4 · p. 3
4.2 Results and Discussion
- ChatGPT correctly repaired 43 of 60 cases (71.66%) and Bard 41 of 60 (68.33%); both scored 40% with c.f. prompts, versus 80% and 95% for ChatGPT and 80% and 85% for Bard with c.s. and c.r. prompts.Table 4 presents performance by model and prompt context.Source: §4.2 · p. 4, Table 4 · p. 4
4.3 Threats to Validity
- The authors identify validity threats from using public ChatGPT and Bard versions accessed in November 2023, manually judging repair correctness, and needing further studies.The tested services can change and evolve, and manual verification is subject to human bias.Source: §4.3 · p. 4
5 CONCLUSION
The study evaluates ChatGPT and Bard on JavaScript vulnerability repair using 20 common vulnerabilities and three prompt templates with varying context. ChatGPT achieves 71.66% accuracy, Bard 68.33%, and more prompt context improves accuracy by up to 55%.
5 CONCLUSION
- ChatGPT achieves 71.66% accuracy and Bard achieves 68.33% on JavaScript vulnerability repair.The study evaluates 20 common vulnerabilities from the CWE Top 25 List.Source: §5 · p. 4
- Increased contextual information in repair prompts positively influences vulnerability-finding and fixing performance, boosting accuracy by up to 55%.The study uses three prompt templates with varying degrees of context.Source: §5 · p. 4
- The study formulates 60 repair prompts for ChatGPT and Bard from 20 JavaScript-relevant CWE Top 25 vulnerabilities and three context levels.Source: §5 · p. 4