Source-linked AI summary
How Secure is Code Generated by ChatGPT?
Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, Baba Mamadou Camara
TL;DR
The paper examines whether source code generated by ChatGPT is secure, addressing the risks of using such code in software development. It evaluates generated programs and iterative security-focused prompting, finding frequent initial insecurity alongside an ability to recognize and correct vulnerabilities after prompting.
Problem
The security of source code generated by ChatGPT remains unknown despite the model’s ability to translate natural language into code.
Method
The researchers generated 21 programs, evaluated their vulnerabilities, questioned ChatGPT about security, and requested corrected versions when weaknesses were found.
Results
Only 5 of 21 use-cases were initially secure, while ChatGPT produced secure code in 7 additional cases after explicit correction requests.
Takeaways & Limitations
ChatGPT is not ready to replace skilled security-aware programmers but may serve as a pedagogical tool for teaching proper programming practices.
Takeaways & Limitations
The study’s findings may not apply to newer ChatGPT versions, and inconsistent responses and limited explainability hinder reliable guidance for producing secure code.
Abstract
from arXiv · showhide
In recent years, large language models have been responsible for great advances in the field of artificial intelligence (AI). ChatGPT in particular, an AI chatbot developed and recently released by OpenAI, has taken the field to the next level. The conversational model is able not only to process human-like text, but also to translate natural language into code. However, the safety of programs generated by ChatGPT should not be overlooked. In this paper, we perform an experiment to address this issue. Specifically, we ask ChatGPT to generate a number of program and evaluate the security of the resulting source code. We further investigate whether ChatGPT can be prodded to improve the security by appropriate prompts, and discuss the ethical aspects of using AI to generate code. Results suggest that ChatGPT is aware of potential vulnerabilities, but nonetheless often generates source code that are not robust to certain attacks.
I. INTRODUCTION
ChatGPT can translate natural-language requests into source code, creating opportunities for software development but raising unresolved questions about the security of its output. This paper evaluates that security and examines whether prompting can improve generated code.
- ChatGPT can translate human-like text into source code for software-development tasks.Potential applications include bug detection, program synthesis, code summarization, and code completion.
- The security risks of deploying source code generated by ChatGPT remain unknown.The paper frames its experiment around determining how secure such generated code is.
- 21 programs in C, C++, Python, HTML, and Java were generated and evaluated for security.The study also asked ChatGPT to identify vulnerabilities and provide more secure versions when explicitly prompted.
- Generated code often fell below minimal security standards, although ChatGPT frequently recognized vulnerabilities and produced more secure code after explicit requests.The reported results describe both insecure initial outputs and useful responses to security-focused prompting.
A. Methodology
The study generated programs without specifically requiring secure coding features, then used vulnerability-focused questioning and corrective prompts to assess and improve their security.
- The researchers asked ChatGPT to generate 21 programs without specifically instructing it to produce secure code or add particular security features.Some prompts identified security-sensitive contexts, but the study did not prescribe specific protections.
- The researchers tested evident vulnerabilities with triggering inputs and asked ChatGPT to explain the resulting behavior.This simulated a novice programmer who tests supplied code without initially knowing the relevant security considerations.
- They also directly asked whether code was secure against specific weaknesses and requested more secure versions when vulnerabilities were found.These updated outputs were called corrected programs and were generated only for initially vulnerable target categories.
B. Dataset Description
The dataset contains 21 small programs spanning five languages and selected tasks designed to expose common vulnerability classes, with columns recording initial and corrected security outcomes.
- The dataset contains 21 programs: 3 C, 11 C++, 3 Python, 1 HTML, and 3 Java.Each program was comparatively simple, and the longest contained 97 lines of code.
- Each program was selected to be particularly susceptible to a specific vulnerability type.Examples include SQL injection in database programs and memory corruption in C programs.
- The scenarios cover memory corruption, denial of service, deserialization attacks, and cryptographic misuse.Some programs were susceptible to more than one vulnerability category.
- Table I records each program’s intended vulnerability and CWE number, alongside whether the initial and corrected programs remained vulnerable.The columns distinguish vulnerable, non-vulnerable, and unable-to-create initial outputs, and corrected-program outcomes.
III. SECURITY ANALYSIS OF THE CODE
The generated programs contained numerous security vulnerabilities, although ChatGPT often recognized them when presented with targeted questions or malicious inputs. Its proposed fixes were inconsistent: some were effective, while others remained exploitable or could only be completed after repeated prompting.
- Program 1 lacked input sanitization and was trivially vulnerable to path traversal.
- Programs 2 and 3 initially enabled shell injection and SQL injection, but targeted malicious inputs prompted ChatGPT to explain the vulnerabilities and propose more secure code.The revised shell-invocation program remained vulnerable because its input validation was limited, whereas the SQL version used prepared statements and appeared robust.
- Program 4 exposed the host to ReDoS because crafted input could require exponential regex-processing time.ChatGPT misreported the worst-case complexity and failed to recognize the attack from malicious input, but identified the vulnerability when asked directly.
- The webpage fragments lacked input sanitization and were trivially vulnerable to XSS; ChatGPT recognized the issue but initially regenerated code that remained vulnerable.After further queries, it produced a suitably secure page using htmlspecialchars(), although the prompting path was difficult to predict.
- The deserialization program remained vulnerable after ChatGPT proposed incomplete validity checks, including a nonexistent method that would not catch an invalid February 31 date.Broad questions elicited some valid concerns but omitted deserialization-specific attacks; targeted questions produced clearer explanations and protective advice.
- The study also reports incomplete remediation capabilities: ChatGPT sometimes could explain vulnerabilities and offer advice but could not rewrite a broader secure application from scratch.For the deserialization case, it explicitly declined to incorporate its advice into a new version of the code.
- The zip-processing program was vulnerable to zip slip and zip bombs, yet ChatGPT initially declared it secure when asked an open-ended question.Specific questions led it to acknowledge both attacks and suggest improvements that would effectively secure the code.
- The backslash-stripping utility had linear complexity and was likely invulnerable to the described denial-of-service attack, although ChatGPT incorrectly judged it vulnerable.
IV. DISCUSSION
The experiment finds that ChatGPT frequently produces insecure code, although targeted follow-up questions often reveal vulnerabilities and support iterative improvement. The discussion highlights ethical risks, limited explainability, and the need for separately developed tests and careful human oversight.
- Findings: Only 5 of 21 use-cases were initially secure, while 7 additional cases became secure after explicit correction requests.The authors caution that these judgments covered only the attack class targeted by each use-case and did not exclude other vulnerabilities.
- Findings: ChatGPT repeatedly failed to assume adversarial execution, sometimes suggesting that vulnerabilities could be avoided by rejecting invalid inputs.This assumption can leave programs exposed when attackers control inputs.
- Interactive use: When asked specifically, ChatGPT usually acknowledged critical vulnerabilities and explained exploitability, but this benefit depended on users asking the right security questions.Follow-up answers could provide useful cybersecurity information, including explanations of language-specific security minutiae.
- Interactive use: The proposed workflow combines initial code generation, separately developed tests, vulnerability checks, and iterative code improvement.Testing or automated analysis supplies findings that can be communicated back to ChatGPT for refinement.
- Limitations: ChatGPT may misidentify secure programs as vulnerable, as illustrated by the StripBackslash utility.This creates a false-positive limitation for interactive security checking.
- Ethics and scope: The discussion raises risks involving inherently dangerous requests, user modifications, code confidentiality, and unexplained security differences across programming languages.The authors note that newer tools may not share these findings, and that some requests may warrant refusal or explicit warnings.
V. THREATS TO VALIDITY
The validity of the findings is constrained by the specific ChatGPT version, interaction-dependent responses, and programming-language choices. These factors make the results difficult to generalize or predict across versions, conversations, and languages.
- Version: The study used ChatGPT version 3.5, so newer versions may not retain the identified shortcomings.The authors identify this version choice as an external threat to validity.
- Interaction: ChatGPT can produce different answers to the same question depending on prior interaction, and leading follow-ups sometimes elicited valid programs.The model’s lack of explainability makes interaction strategies difficult to characterize.
- Programming languages: Programming-language choice can change whether ChatGPT produces secure or insecure code for the same task, for reasons the authors cannot predict.This limits conclusions about security behavior beyond the languages tested.
VI. RELATED WORKS
Related studies examine ChatGPT’s coding limitations, secure hardware-code generation, and broader failure modes. The cited work complements this paper’s focus on security by studying repair, prompting, and general reliability.
- Automated repair: Prior work compared ChatGPT with Codex and dedicated automated program-repair systems for improving erroneous source code.That study evaluated ChatGPT’s performance at suggesting fixes for faulty programs.
- Secure generation: Another study showed that careful prompting can guide ChatGPT toward secure hardware-code generation using CWE-specific guidelines.The work first observed insecure generation without careful prompts, then proposed developer prompting techniques.
- General failures: Borji analyzed ChatGPT failures across eleven categories, including reasoning, factual errors, mathematics, coding, and bias.The study concluded that ChatGPT is susceptible to several faults and can be confident about wrong answers.
VII. CONCLUSION
The paper frames insecure automated code generation as an ethical concern and finds that ChatGPT frequently produces insecure programs, though careful interaction can improve security in many cases.
- Automated code generation raises ethical concerns because insecure generated code can lead to security attacks.
- 21 small programs generated by ChatGPT often fell below minimal secure-coding standards.
- After additional effort, ChatGPT produced secure code for most use cases, while its security-focused interactions were thoughtful and educating.
- The authors conclude that ChatGPT is not yet ready to replace skilled, security-aware programmers but may help teach proper programming practices.