Source-linked AI summary
When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?
Chong Chen, Jianzhong Su, Jiachi Chen, Yanlin Wang, Tingting Bi, Jianxing Yu, Yanli Wang, Xingwei Lin, Ting Chen, Zibin Zheng
TL;DR
Smart contract vulnerabilities can cause serious financial harm, yet evidence on ChatGPT for detecting them remains limited. This paper empirically evaluates ChatGPT on a labeled vulnerability dataset, compares it with established tools, and examines its failure modes. ChatGPT shows high recall but low precision, variable performance across vulnerability types, and robustness problems involving inconsistent answers and code-length limits.
Problem
Smart contract vulnerabilities can cause significant economic losses, while research on ChatGPT for detecting them remains relatively limited.
Method
The authors evaluate three ChatGPT models on the SmartBugs-curated dataset, compare them with 14 vulnerability detection tools, and analyze false positives and limitations.
Results
ChatGPT has high recall but low precision, varies across vulnerability types, and performs worse than other tools for 71.4% of evaluated vulnerabilities.
Takeaways & Limitations
ChatGPT provides useful vulnerability detection capabilities but has strengths and weaknesses that should inform its use and comparison with specialized tools.
Takeaways & Limitations
ChatGPT produces inconsistent results and may fail when smart contracts exceed its input-token limit.
Abstract
from arXiv · showhide
With the development of blockchain technology, smart contracts have become an important component of blockchain applications. Despite their crucial role, the development of smart contracts may introduce vulnerabilities and potentially lead to severe consequences, such as financial losses. Meanwhile, large language models, represented by ChatGPT, have gained great attentions, showcasing great capabilities in code analysis tasks. In this paper, we presented an empirical study to investigate the performance of ChatGPT in identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT's effectiveness using a publicly available smart contract dataset. Our findings discover that while ChatGPT achieves a high recall rate, its precision in pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT's performance varies when detecting different vulnerability types. We delved into the root causes for the false positives generated by ChatGPT, and categorized them into four groups. Second, by comparing ChatGPT with other state-of-the-art smart contract vulnerability detection tools, we found that ChatGPT's F-score is lower than others for 3 out of the 7 vulnerabilities. In the case of the remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these tools. Finally, we analyzed the limitation of ChatGPT in smart contract vulnerability detection, revealing that the robustness of ChatGPT in this field needs to be improved from two aspects: its uncertainty in answering questions; and the limited length of the detected code. In general, our research provides insights into the strengths and weaknesses of employing large language models, specifically ChatGPT, for the detection of smart contract vulnerabilities.
1 INTRODUCTION
This study examines ChatGPT for smart contract vulnerability detection because such vulnerabilities can cause major economic losses, while prior research on this application remains limited. Using the SmartBugs dataset, the authors evaluate effectiveness and analyze false positives.
- Smart contract vulnerabilities can cause significant economic losses and threaten blockchain ecosystem development.
- Research on ChatGPT for smart contract vulnerability detection remains relatively limited despite its programming capabilities.
- The study evaluates ChatGPT on the SmartBugs dataset and compares it with 14 state-of-the-art vulnerability detection tools.
- ChatGPT achieves recall rates of 51.4%, 83.6%, and 88.2% for GPT-3.5, GPT-4o, and GPT-4, respectively, but precision rates of 19.7%, 20.2%, and 22.6%.
- ChatGPT performs differently across vulnerability types, performing well on Unchecked Return Values but poorly on Short Address Attack.
- The authors categorize ChatGPT’s false positives into four types: Protected Mechanism Bias, Development Intent Bias, Interfered by Comments, and Interference with Dead Code.
RQ2. How does ChatGPT perform compared to other tools?
The authors compare ChatGPT with 14 state-of-the-art vulnerability detection tools on the SmartBugs-curated dataset. ChatGPT leads on some vulnerability types and detection speed but has weaker detection performance and more failures in important respects.
- ChatGPT outperforms the other 14 tools on Front Running and Denial of Service, but its F1 scores are only 12.8% and 15.9%, respectively.
- Several tools detect other vulnerability types more effectively than ChatGPT.
- ChatGPT surpasses all other tools in detection speed.
- ChatGPT has a higher detection failure rate than 10 other tools because of token limitations.
RQ3. What are the limitations of ChatGPT in detecting smart contract vulnerabilities?
The paper identifies uncertainty and input-length constraints as key limitations of ChatGPT for smart contract vulnerability detection. It also reports the study’s evaluation, comparisons, contributions, and publicly released experimental data.
- Limitations: Only 29 of 50 contracts produced entirely consistent results across five repeated detections, indicating unstable outcomes for 42% of evaluated contracts.
- Contributions: The study evaluates gpt-3.5-turbo, gpt-4o, and gpt-4 on the SmartBugs-curated dataset and reports high recall but low precision across vulnerabilities.
- Contributions: The authors compare ChatGPT with 14 vulnerability detection tools using 142 smart contracts to assess effectiveness and efficiency.
- Contributions: The study estimates the maximum code size accommodated by gpt-3.5-turbo and gpt-4 and makes its experimental data publicly available.
2 BACKGROUND
The background defines smart contracts and their vulnerabilities, introduces the DASP10 taxonomy and SmartBugs-curated dataset, and summarizes ChatGPT and the GPT models studied. It also notes that model updates may change task performance.
- Smart Contracts: Smart contracts are self-executing blockchain programs that enable digital verification without arbitrators or third-party institutions.
- Smart Contract Vulnerabilities: Smart contract vulnerabilities are exploitable security flaws that can manipulate contract rules or steal assets, with blockchain immutability making losses potentially irreparable.
- Smart Contract Vulnerabilities: DASP10 defines 10 common smart contract vulnerabilities, while this paper discusses the top nine and excludes Unknown Unknowns.
- Smart Contract Vulnerabilities: SmartBugs-curated contains 142 labeled vulnerable Solidity contracts organized according to the DASP taxonomy.
- ChatGPT: ChatGPT is an OpenAI large language model based on the GPT series and trained with transformer architectures on large-scale text data.
- ChatGPT: The study focuses on gpt-3.5-turbo, gpt-4o, and gpt-4, while acknowledging that future model updates may alter performance.
3 (RQ1) ON EFFECTIVENESS OF CHATGPT IN DETECTING VULNERABILITIES
ChatGPT detects smart-contract vulnerabilities with high recall but limited precision, and its effectiveness varies by vulnerability type. False positives arise from model limitations including protected-mechanism overemphasis, development-intent misunderstandings, comment interference, and dead-code interference.
- Output-format effects: ChatGPT’s vulnerability-detection performance decreases under restricted output formats across eight vulnerability types, except Arithmetic Issues.The study reports that output-format limitations negatively affect detection performance in the other vulnerability categories.
- Detection limitations: Long smart-contract code can cause detection failures because model token limits prevent analysis of contracts exceeding the input threshold.The study reports failures for five contracts with gpt-3.5-turbo and two with GPT-4.
- Detection effectiveness: GPT-4 achieves recall above 80% for most vulnerability types, but precision remains around 22%, producing many false positives.Front Running and Short Address Attack are exceptions to the reported recall pattern.
- Detection effectiveness: ChatGPT’s strongest performance is on Unchecked Return Values, with F-scores of 0.59, 0.74, and 0.68 for the evaluated models.The reported values correspond to the three model results presented in the study.
- False-positive causes: False positives result from overemphasizing protected mechanisms and misunderstanding special variables or functions, including secure centralized return-value checks.These patterns are categorized as protected-mechanism bias and development-intent bias.
- False-positive causes: Comments and dead code can also trigger false positives because ChatGPT may treat comments as executable code and cannot determine whether code paths execute.The study identifies comment interference and dead-code interference as additional false-positive categories.
4 (RQ2) COMPARISON WITH OTHER DETECTION TOOLS
Compared with 14 specialized tools, ChatGPT covers more vulnerability types and is more efficient, but its detection scores are not consistently competitive.
- Effectiveness: ChatGPT is less effective than current tools for Reentrancy and Access Control, while performing relatively well on Denial of Service and Front Running.
- Effectiveness: ChatGPT successfully detects all nine examined vulnerability types, providing broader coverage than tools with narrower scopes.
- Effectiveness: ChatGPT performs better than the other 14 tools for Denial of Service and Front Running, but trails their average performance for 3 of 7 vulnerabilities.
- Efficiency: 9 minutes and 21 seconds, 9 minutes and 55 seconds, and 11 minutes and 9 seconds were required by gpt-3.5-turbo, gpt-4o, and gpt-4, respectively, to analyze the dataset.
- Efficiency: The three GPT models outperformed 11 of 14 other tools in detection efficiency, trailing only SmartCheck, Solhint, and AChecker.
5 (RQ3) LIMITATIONS OF CHATGPT IN DETECTING VULNERABILITIES
ChatGPT's vulnerability detection is limited by uncertain outputs and input-length constraints, which can prevent reliable analysis of lengthy contracts.
- Context Length: Five contracts failed detection because GPT input tokens were insufficient, whereas 10 other tools had no failures on the dataset.
- Uncertainty: With identical smart-contract inputs, ChatGPT may produce divergent vulnerability results because its probabilistic outputs vary across runs.
- Uncertainty: Among 50 randomly selected contracts tested five times with gpt-4, only 80% received reliable results under an 80% consistency threshold.
- Uncertainty: Adjusting temperature to 0 and using mimic-in-the-background prompting marginally improved gpt-4 detection while preserving certainty, but reduced overall gpt-3.5-turbo efficacy.
- Context Length: For gpt-4, the 8192-token limit and 107-token prompt leave 8085 tokens for code, constraining analysis of lengthy contracts.
6 DISCUSSION
The discussion examines how external knowledge and interaction design affect ChatGPT’s smart-contract vulnerability detection, while emphasizing persistent robustness and security limitations.
- 6.1 Vulnerability learning: ChatGPT failed to detect Price Oracle Manipulation in five contracts, instead reporting secondary issues while overlooking the main vulnerability.The example contract allowed attackers to replace its priceFeed through governance or other methods, enabling manipulated prices to affect key operations.
- 6.1 Vulnerability learning: Adding vulnerability names and contextual hints was explored as a way to improve detection of poorly recognized vulnerabilities such as Price Oracle Manipulation.The experiments varied prompts from no information to vulnerability-name-only and progressively richer external information.
- 6.2 Multi-round Conversation: Nine rounds of conversation supplied vulnerability names, aliases, and DASP10 definitions before ChatGPT assessed contract code.This multi-round procedure was compared with the earlier single-round experiment.
- 6.2 Multi-round Conversation: Multi-round conversation produced no obvious improvement, suggesting that interaction mode had limited influence on vulnerability-detection performance.The additional definitions provided more information but did not clearly enhance the results.
- 6.5 Discussion: Providing too much specialized information can create knowledge conflicts and weaken generalization across contexts.The discussion therefore stresses careful control of both the manner and quantity of information supplied to LLMs.
- 6.3 Code Poisoning: Code poisoning exposed bypass risks: SafeMath influenced approximately 46.7% of contracts and onlyOwner influenced 61.1%.The experiment indicates that protected mechanisms can cause missed detections and may support more sophisticated evasion mechanisms.
- 6.4 Character Obfuscation: Character obfuscation reduced detection performance for four of nine vulnerabilities, while the other five remained stable or slightly improved.The overall average impact was limited, but susceptibility varied by vulnerability type.
7 RELATED WORK
Related work spans traditional smart-contract analysis methods and emerging evaluations of large language models for vulnerability detection.
- Traditional vulnerability detection: Smart-contract vulnerability detection commonly uses static verification, while dynamic analysis examines contracts during execution and includes approaches such as fuzzing.Static verification analyzes source code or bytecode without execution, whereas dynamic analysis provides execution-based information.
- LLM-based vulnerability detection: Prior LLM studies evaluated software-engineering capabilities and real smart-contract attacks, whereas this work focuses on common smart-contract vulnerabilities and ChatGPT’s effectiveness and limitations.The paper positions its scope as distinct from studies centered on broader software-engineering tasks or actual attacks.
8 CONCLUSION AND FUTURE WORKS
The conclusion evaluates ChatGPT’s detection ability, compares it with existing tools, and identifies uncertainty and scope limitations while outlining future LLM research directions.
- Contributions: The study uses three research questions to assess ChatGPT’s detection efficacy, compare it with vulnerability-detection tools, and identify performance limitations.The RQ1 evaluation used optimized prompting with vulnerability detection and semantic analysis.
- Findings: ChatGPT detected the most vulnerability types but underperformed state-of-the-art tools for 71.4% (5/7) of vulnerabilities and failed more often than 71.4% (10/14) of evaluated tools.The study also reports that approximately 20% of repeated conclusions were uncertain.
- Future work: Future work will evaluate alternative LLMs and more diverse datasets, while exploring fine-tuning and retrieval-augmented generation to improve detection.Code Llama is given as one example of an alternative model.