PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-Enhanced Software Vulnerability Detection Using ChatGPT

Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, Hui Li

arXiv:2308.12697v2cs.SE

TL;DR

Software vulnerabilities cause significant economic and social losses, and prior ChatGPT vulnerability-detection studies used simple questions without tailored prompts. This paper evaluates prompt designs that add code information and chain-of-thought prompting, reporting that these enhancements support vulnerability detection across two datasets.

  • Problem

    Software vulnerabilities cause economic and social losses, while prior ChatGPT vulnerability-detection studies used simple questions without tailored prompt designs.

    Source: §1 · p. 1

  • Method

    The paper studies improved ChatGPT prompts that incorporate structural and sequential code information and leverage multi-round dialogue through chain-of-thought prompting.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    The authors report that structural and sequential auxiliary information helps detection and that chain-of-thought prompting further improves ChatGPT's vulnerability-detection performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study analyzes both the merits and demerits of using prompt-enhanced ChatGPT for software vulnerability detection.

    Source: §7 · p. 11

    The experiments cover Java and C/C++, and the authors note performance may differ for other programming languages.

    Source: §6 · p. 11

Abstract

from arXiv · show

With the increase in software vulnerabilities that cause significant economic and social losses, automatic vulnerability detection has become essential in software development and maintenance. Recently, large language models (LLMs) like GPT have received considerable attention due to their stunning intelligence, and some studies consider using ChatGPT for vulnerability detection. However, they do not fully consider the characteristics of LLMs, since their designed questions to ChatGPT are simple without a specific prompt design tailored for vulnerability detection. This paper launches a study on the performance of software vulnerability detection using ChatGPT with different prompt designs. Firstly, we complement previous work by applying various improvements to the basic prompt. Moreover, we incorporate structural and sequential auxiliary information to improve the prompt design. Besides, we leverage ChatGPT's ability of memorizing multi-round dialogue to design suitable prompts for vulnerability detection. We conduct extensive experiments on two vulnerability datasets to demonstrate the effectiveness of prompt-enhanced vulnerability detection using ChatGPT. We also analyze the merit and demerit of using ChatGPT for vulnerability detection. Repository: https://github.com/KDEGroup/LLMVulnerabilityDetection.

1 INTRODUCTION

Software vulnerabilities impose economic and social losses, while existing detection methods face limits in expert-dependent rules, coarse-grained patterns, or language- and vulnerability-specific coverage. This paper studies prompt designs for ChatGPT vulnerability detection, incorporating auxiliary code information and multi-round dialogue.

1 INTRODUCTION

  • Existing rule-based methods depend on expert-defined rules, while classical ML methods rely on shallow code patterns and can lack detection accuracy.The paper also notes that many deep-learning approaches are limited to particular programming languages or vulnerability types.Source: §1 · pp. 1–2
  • Prior ChatGPT vulnerability-detection studies used simple questions without tailored prompt design and did not consider its multi-step reasoning ability.The paper identifies chain-of-thought as a capability overlooked by those studies.Source: §1 · p. 2
  • Prompt-engineered ChatGPT vulnerability detection is studied with improved basic prompts, structural and sequential code information, and chain-of-thought prompting.The experiments use two vulnerability datasets, and the authors report further performance improvement from chain-of-thought prompting.Source: §1 · p. 2
  • The study evaluates ChatGPT across two programming languages and reports that structural and sequential auxiliary information helps detection.The authors describe direct use of traditional structural and sequential code modeling methods in ChatGPT-based detection as a first, to their knowledge.Source: §1 · p. 2

2 RELATED WORK

The related work reviews language models and prompting, then surveys traditional and deep-learning approaches to software vulnerability detection. It frames the paper against methods that use handcrafted code features, graph or sequence representations, and varied prompt designs.

2.1 Large language models

  • LLMs model word-sequence likelihood and use large-scale training; representative systems include GPT, PaLM, Bard, Chinchilla, and open-source models.The passage describes LLMs as highly parameterized and broadly capable, while noting that some systems are accessible through APIs and others are not.Source: §2.1 · p. 2

2.2 Prompt Engineering

  • Prompt engineering reformulates downstream tasks with textual prompts, with discrete prompt search and continuous-vector prompts among the explored designs.Prior work considered prompts for generative models, code tasks, and other LLM applications, while vulnerability-detection work had not fully considered LLM characteristics.Source: §2.2 · pp. 2–3

2.3 Software Vulnerability Detection

  • Traditional vulnerability-detection methods learn code patterns from handcrafted features extracted through static or dynamic analysis.The surveyed methods include rules, classifiers, AST-derived features, and bag-of-words or N-gram representations.Source: §2.3.1 · p. 3

2.3.1 Traditional Vulnerability Detection.

  • Deep learning can capture vulnerability features automatically, reducing feature-engineering costs compared with traditional methods.The section distinguishes sequence-based and graph-based approaches, including neural models of code entities and program graphs.Source: §2.3.2 · p. 3

2.3.2 Deep Learning Based Vulnerability Detection.

  • Deep-learning vulnerability detectors model code sequences or program graphs such as ASTs, DFGs, CFGs, and PDGs.Examples include Bi-LSTM and CNN sequence models, value-flow path encoders, and graph-based approaches using recurrent networks or graph neural networks.Source: §2.3.2 · p. 3

3 DATA

The study uses Java and C/C++ function datasets, preprocesses them for function-level vulnerability detection, and extracts API-call sequences and data flow as prompt auxiliary information. The data section describes the sources, filtering choices, and extraction procedures for these inputs.

3 DATA

  • The data section introduces the datasets used in the study.The evaluation uses functions with or without vulnerabilities from Java and C/C++ datasets.Source: §3 · p. 3, §3.1 · p. 3

3.1 Data Collection

  • Java samples come from SARD, while C/C++ samples come from a research benchmark collected from NVD.The Java collection includes SARD data published before June 8, 2023; the C/C++ collection focuses on .c or .cpp files linked to vulnerabilities or patched versions.Source: §3.1 · p. 3

3.2 Data Pre-processing

  • Preprocessing removes duplicates, vulnerability-revealing annotations and name hints, filters short or cross-class samples, and limits Java evaluation to the 50 largest vulnerability types.The final datasets contain 1,171 vulnerable and 917 non-vulnerable Java functions, and 1,015 vulnerable and 922 non-vulnerable C/C++ functions.Source: §3.2 · p. 4

3.3 Feature Extraction

  • The prompt design uses API-call sequences and data flow as two types of auxiliary information extracted from the datasets.The section describes how these representations are derived from source code.Source: §3.3 · p. 4

3.3.1 API Call Sequence Extraction.

  • API-call sequences are extracted from ASTs in source order, with Java rules for constructors, receiver calls, nested calls, branches, and loops.For nested calls passed as parameters, the inner calls precede the calling function in the sequence.Source: §3.3.1 · p. 4

3.3.2 Data Flow Graph Extraction.

  • Data-flow extraction parses code into an AST, identifies variables, and records directed relations indicating which variable a value comes from or is computed by.Each relation is stored with the positions of its two variables.Source: §3.3.2 · pp. 4–5

4 PROMPT DESIGN

The paper designs vulnerability-detection prompts that pair code with task framing, structural or sequential information, and multi-round reasoning. Its prompting choices respond to observed limits in ChatGPT’s unaided understanding of data flow and API-call sequences.

4.1 Basic Prompting

  • ChatGPT is first asked whether a test function is buggy, with an explicit Yes-or-No answer requested.The basic prompt supplies the test function as code.Source: §4.1 · p. 5
  • A role-based prompt frames ChatGPT as a vulnerability-detection system, while a reverse question tests whether “buggy” biases its answer.The reverse version asks whether the program is correct; key-token guessing is identified as a possible bias.Source: §4.1 · p. 5

4.2.1 The Necessity of Auxiliary Information.

  • Trials on 20 functions per dataset found that additional data-flow information helps ChatGPT understand source code.The authors report similar observations across all 40 sampled cases.Source: §4.2.1 · pp. 5–6
  • ChatGPT accurately identified control-flow execution paths, but its source-code-only data-flow response missed possible paths and included an incorrect dependency.These contrasting observations motivate adding auxiliary information to the prompt.Source: §4.2.1 · pp. 5–6
  • In 40 sampled cases, ChatGPT missed API calls passed as parameters and left call order unclear, motivating explicit API-sequence information.The authors report similar observations across the sampled cases.Source: §4.2.1 · p. 6

4.2.2 Prompts with DFG and API Calls.

  • The enhanced role-based prompts place data-flow descriptions or API-call sequences alongside the original program, using orders the authors found better than alternatives.The designs are Pr-b-d and Pr-a-b, respectively.Source: §4.2.2 · p. 6
  • Text templates encode API calls in sequence and describe data values as coming from or being computed by another variable.The API template states the calls’ order; the data-flow template links variables and token positions.Source: §4.2.2 · p. 6

4.3 Chain-of-Thought Prompting

  • Chain-of-thought prompting asks ChatGPT to describe code intent first, then assess vulnerability, optionally supplying API-call or data-flow information in the second turn.The design uses multi-round dialogue and intermediate natural-language reasoning.Source: §4.3 · p. 7

5 EXPERIMENTAL RESULTS

The experiments are organized around five questions about ChatGPT’s detection performance, auxiliary information, chain-of-thought prompting, prompt order, and vulnerability types. The study compares against two general-purpose vulnerability-detection baselines and evaluates standard classification metrics.

5 EXPERIMENTAL RESULTS

  • Five research questions cover basic-prompt detection, API-call and data-flow information, chain-of-thought, prompt-component order, and performance across vulnerability types.The questions define the scope of the experimental results section.Source: §5 · p. 7

5.1.1 Vulnerability Detection Baselines.

  • CFGNN and Bugram are selected as general detection baselines because other methods often target only one or a few vulnerability types.The study uses their default configurations.Source: §5.1.1 · p. 7
  • ChatGPT’s performance is measured using Accuracy, Precision, Recall, and F1 score.Source: §5.1.2 · p. 7

5.2 Effectiveness of Basic Prompts (RQ1)

With basic prompts, ChatGPT generally outperforms CFGNN and Bugram, while prompt effects vary by language and wording can bias predictions. Human ratings also indicate that many detected vulnerabilities are not well explained.

5.2 Effectiveness of Basic Prompts (RQ1)

  • About 5% higher Java-dataset accuracy with Pr-b than Pb; on C/C++, task-role prompting changes detection counts but leaves overall accuracy almost unchanged.The authors characterize task-role improvement as programming-language-specific.Source: §5.2 · pp. 7–8
  • 58% and 64% higher Java-dataset accuracy with Pb than CFGNN and Bugram, respectively; basic prompts generally outperform both baselines.Table 2 reports performance using different prompts.Source: §5.2 · p. 7, Table 2 · p. 8
  • Replacing “buggy” with “correct” shifts predictions toward non-vulnerable, supporting the finding that basic-prompt responses are biased toward prompt keywords.The authors report that most functions were predicted vulnerable with basic prompts on both datasets.Source: §5.2 · p. 8
  • 52% of C/C++ and 64% of Java vulnerability explanations received ratings of 4 or 5 on a five-point scale.Five computer-science master’s students rated 200 explanations; the authors conclude that the basic prompt does not understand almost half of the vulnerabilities it detects.Source: Figure 2 · p. 8, §5.2 · p. 8

5.3 Usefulness of Auxiliary Information (RQ2)

Auxiliary information has different effects across programming languages: API calls help Java vulnerability detection, while data-flow information contributes a little to understanding C/C++ vulnerable programs.

5.3 Usefulness of Auxiliary Information (RQ2)

  • API calls raise Java accuracy to 0.747, about 8% above Pr-b, while data-flow information works best on C/C++ but does not exceed Pb by a large margin.Adding API calls or data-flow information also increases accuracy on non-vulnerable data while decreasing it on vulnerable data.Source: §5.3 · p. 8
  • API calls are more effective for Java, whereas data-flow information contributes a little to understanding C/C++ vulnerable programs.The paper reports that the effects of auxiliary information differ by programming language.Source: §5.3 · p. 8

5.4 Effectiveness of Chain-of-Thought Prompting (RQ3)

Chain-of-thought prompting has different effects across the two datasets, and the authors also examine step-1 code summaries and their use as auxiliary prompt information.

5.4.1 Performance of Vulnerability Detection in Step 2.

  • Chain-of-thought prompts improve results on C/C++ but degrade detection performance on Java compared with basic prompts.The authors speculate that the position of the extra information and code may account for the difference.Source: Table 3 · p. 9, §5.4.1 · p. 9
  • P(chain)2,r-b achieves the best accuracy on both the Java and C/C++ datasets.The passage identifies this prompt as the best-accuracy option among the reported chain-of-thought results.Source: §5.4.1 · p. 9
  • Adding API-call or data-flow information in chain-of-thought prompts increases correct non-vulnerable detections while decreasing correct vulnerable detections.The number of detected non-vulnerable samples rises as auxiliary information is added, while detected vulnerable samples fall.Source: §5.4.1 · p. 9

5.4.2 Impacts of Including the Summary of Code.

  • In ratings of 200 randomly selected step-1 answers, nearly 90% scored 4 or higher and none scored 1 on a five-point scale.Five computer-science master’s students rated answers for 100 vulnerable and 100 non-vulnerable test samples drawn from both languages.Source: Figure 3 · p. 9, §5.4.2 · p. 9
  • The ratings are reported as distributions for sampled step-1 answers, and the authors conclude that ChatGPT can understand code functionality accurately.The sampled answers correspond to code summaries in the first step of chain-of-thought prompting.Source: Figure 3 · p. 9, §5.4.2 · p. 9
  • Adding a code summary decreases vulnerability-detection accuracy on Java but increases it on C/C++.The authors also describe a code summary as auxiliary information that could be used in a single prompt instead of a two-step chain-of-thought prompt.Source: Table 4 · p. 9, §5.4.2 · p. 9

5.5 Impacts of Position (RQ4)

The position of auxiliary information affects detection performance: API descriptions work better before the code, while data-flow descriptions work better after it.

5.5 Impacts of Position (RQ4)

  • Role-based prompting with an API description before the code achieves the best accuracy, and API descriptions before the code outperform those after it in both prompt types.The improvement is less significant for the basic prompt than for the role-based prompt, indicating different sensitivity to position.Source: §5.5 · pp. 9–10, Table 5 · p. 10
  • Data-flow descriptions after the code outperform those before it, while the suitable position depends on the auxiliary information used.The paper reports different effects for API calls and data flow rather than one best position for all auxiliary information.Source: §5.5 · p. 10
  • Placing API descriptions before the code decreases true positives on vulnerable data but increases correct predictions on non-vulnerable data.The authors report that API information in this position appears more conducive to judgments on non-vulnerable data.Source: §5.5 · p. 10
  • Data-flow descriptions contribute more to accurate predictions on vulnerable samples than API descriptions do.This is the reported difference in which sample class benefits more from each type of auxiliary information.Source: §5.5 · p. 10

5.6 Performance on Different Vulnerability Types (RQ5)

On Java, ChatGPT performs unevenly across vulnerability types: it does well on many types but struggles with types that are less apparent from code context.

5.6 Performance on Different Vulnerability Types (RQ5)

  • ChatGPT exceeds 0.5 accuracy on 41 of 50 Java vulnerability types, while reaching 100% accuracy on seven types.The reported results use prompt Pr-a-b; the authors note that similar trends appear on C/C++ but report type-specific results only for Java.Source: §5.6 · p. 10
  • ChatGPT predicts no CWE-759 samples correctly and achieves 0.411 accuracy on CWE-328.The authors infer that detecting these one-way-hash vulnerabilities may require human intervention to set rules or inject extra knowledge.Source: §5.6 · p. 10
  • Without processing comments, ChatGPT cannot correctly predict CWE-546, which is triggered by suspicious comments rather than program functionality.The paper also reports poor performance on CWE-482, where the vulnerability involves confusing “==” with “=”.Source: §5.6 · pp. 10–11
  • The authors conclude that ChatGPT detects grammar-related or some boundary-related vulnerabilities better than types unrelated to code context or requiring full context understanding.Because sample counts vary across vulnerability types, the authors caution that these findings may not be universal and call for more samples.Source: §5.6 · p. 11

6 THREATS OF VALIDITY

The study’s findings are bounded by its ChatGPT version, tested programming languages, human evaluators, and uneven sample counts across vulnerability types.

6 THREATS OF VALIDITY

  • The experimental results represent ChatGPT 4 only, and conclusions may become invalid for future versions.The authors note that ChatGPT is actively updated.Source: §6 · p. 11
  • The experiments cover Java and C/C++; ChatGPT may show different vulnerability-detection performance for other programming languages.The authors also report that performance sometimes differs between the two tested languages.Source: §6 · p. 11
  • Human evaluation of vulnerability explanations and code summaries may be inaccurate or biased because annotators have differing knowledge backgrounds.The evaluators were master’s students majoring in Computer Science.Source: §6 · p. 11
  • The findings may not generalize across vulnerability types because sample counts vary, and more samples are needed to verify them.The authors specifically note that the findings may not be universal across all vulnerability types.Source: §6 · p. 11

7 CONCLUSION

The paper studies prompt-enhanced ChatGPT for software vulnerability detection using extra code information and chain-of-thought prompting. Experiments on two datasets demonstrate its effectiveness, and the study discusses its merits and demerits.

7 CONCLUSION

  • The study designs vulnerability-detection prompts that add API-call and data-flow information and use chain-of-thought prompting, then evaluates them on two vulnerability datasets.The paper also analyzes the merits and demerits of using ChatGPT for vulnerability detection.Source: §7 · p. 11

Related papers

Loading 2308.12697v2…