PaperlayerView exact revision on arXiv

Source-linked AI summary

GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis

Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, Yang Liu

arXiv:2308.03314v3cs.CRcs.AIcs.SE

TL;DR

GPTScan addresses limitations in existing smart-contract logic-vulnerability detection by combining GPT’s code understanding with static analysis. It matches code-level scenarios and properties, validates GPT-identified variables and statements statically, and achieves strong precision and recall across diverse contract datasets while reducing false positives.

  • Problem

    Existing smart-contract analysis tools and GPT-based approaches have limited coverage or high false-positive rates for logic vulnerabilities, motivating a more effective detection system.

  • Method

    GPTScan matches candidate vulnerable functions to code-level scenarios and properties, then validates GPT-recognized key variables and statements with static confirmation and filters candidates before matching.

  • Results

    GPTScan achieves over 90% precision for token contracts, 57.14% precision for large projects, and over 70% recall for ground-truth logic vulnerabilities across around 400 projects and 3K Solidity files.

  • Takeaways & Limitations

    Static confirmation reduces 65.84% of original false-positive cases in Web3Bugs, while GPTScan also discovers vulnerabilities missed by human auditors.

  • Takeaways & Limitations

    The static analysis is not path-sensitive, and whitelist-based modifier filtering can produce false positives or false negatives.

Abstract

from arXiv · show

Smart contracts are prone to various vulnerabilities, leading to substantial financial losses over time. Current analysis tools mainly target vulnerabilities with fixed control or data-flow patterns, such as re-entrancy and integer overflow. However, a recent study on Web3 security bugs revealed that about 80% of these bugs cannot be audited by existing tools due to the lack of domain-specific property description and checking. Given recent advances in Large Language Models (LLMs), it is worth exploring how Generative Pre-training Transformer (GPT) could aid in detecting logicc vulnerabilities. In this paper, we propose GPTScan, the first tool combining GPT with static analysis for smart contract logic vulnerability detection. Instead of relying solely on GPT to identify vulnerabilities, which can lead to high false positives and is limited by GPT's pre-trained knowledge, we utilize GPT as a versatile code understanding tool. By breaking down each logic vulnerability type into scenarios and properties, GPTScan matches candidate vulnerabilities with GPT. To enhance accuracy, GPTScan further instructs GPT to intelligently recognize key variables and statements, which are then validated by static confirmation. Evaluation on diverse datasets with around 400 contract projects and 3K Solidity files shows that GPTScan achieves high precision (over 90%) for token contracts and acceptable precision (57.14%) for large projects like Web3Bugs. It effectively detects ground-truth logic vulnerabilities with a recall of over 70%, including 9 new vulnerabilities missed by human auditors. GPTScan is fast and cost-effective, taking an average of 14.39 seconds and 0.01 USD to scan per thousand lines of Solidity code. Moreover, static confirmation helps GPTScan reduce two-thirds of false positives.

1 INTRODUCTION

GPTScan addresses the limited coverage of existing smart-contract analyzers by combining GPT-based code understanding with static analysis for logic-vulnerability detection. Across diverse datasets, it reports strong precision for token contracts, acceptable precision for large projects, recall above 70%, and low scanning cost.

  • Motivation: Around 80% of Web3 vulnerabilities remain undetected by existing tools, which mainly target fixed control- or data-flow patterns.Undetected bugs are primarily associated with smart-contract business logic.
  • Approach: GPTScan breaks each logic-vulnerability type into code-level scenarios and properties for GPT-based candidate matching.Scenarios describe relevant functionality, while properties describe vulnerable code attributes or operations.
  • Approach: GPTScan recognizes key variables and statements with GPT, validates them using static confirmation, and narrows inputs through multidimensional filtering.This design addresses coarse GPT matching and the cost of feeding entire multi-file projects to GPT.
  • Evaluation: 83.33% recall and a 67.8% F1 score were achieved on Web3Bugs, while DefiHacks achieved 71.43% recall and an 80% F1 score.GPTScan also identified 9 vulnerabilities absent from Code4rena audit reports.
  • Evaluation: 4.39% false positive rate was achieved on non-vulnerable Top200 contracts, while DefiHacks achieved 90.91% precision and Web3Bugs 57.14% precision.The results support massive scanning of on-chain contracts, although precision is lower for large projects.
  • Efficiency: 14.39 seconds and 0.01 USD per thousand Solidity lines were the average scanning time and cost, while static confirmation reduced 65.84% of Web3Bugs false positives.Web3Bugs and DefiHacks were slower and costlier because of more complex functions that filtering and scenario matching could not remove.

2 BACKGROUND

The background distinguishes smart-contract vulnerability categories and motivates GPT for semantic code understanding, while noting that GPT’s reasoning limitations require verification.

  • Smart-contract vulnerabilities: Smart contracts are self-running blockchain programs written in Solidity, and Zhang et al. categorize 26 vulnerability types into three groups.The second group includes vulnerabilities using simple, general oracles without requiring deep code-semantic understanding.
  • GPT for detection: GPT models are trained on source-code descriptions and vulnerability information, enabling zero-shot vulnerability detection without vulnerability examples.GPT can therefore interpret source code using its pretrained knowledge.
  • Examples: Figure 1 presents the Risky First Deposit vulnerability at lines 8–9, while Figure 2 presents the Wrong Checkpoint Order at lines 6–7 and 10–11.These examples illustrate logic vulnerabilities discussed in the paper.
  • GPT for detection: GPT’s limited logical reasoning can make its results inaccurate, so verification with other methods is needed to reduce false positives.This limitation motivates combining GPT with static analysis.

3 MOTIVATING EXAMPLES

Two smart-contract examples show why semantic recognition by GPT and low-level validation by static analysis are complementary for detecting logic vulnerabilities.

  • Example 1: In Risky First Deposit, minting the entire LP-token share to the first depositor while supply is zero can inflate the price per share.The inflated price can cause later victim deposits.
  • Example 2: In Wrong Checkpoint Order, checkpoints should precede sender and receiver balance updates, but the reverse order lets a user steal all rewards.GPT identifies checkpoint and balance-update statements, while static analysis confirms their dependency or order.
  • Motivation: Static analysis lacks high-level semantic understanding, whereas GPT may overlook low-level information, producing low recall or high false positives.Combining both techniques is intended to complement their respective weaknesses.

4 GPTSCAN

GPTScan combines candidate filtering, GPT scenario/property matching, and static confirmation to detect logic vulnerabilities across multi-file smart-contract projects. Its design addresses input scale, semantic matching, and unreliable GPT outputs.

  • 4.1 Overview and Challenges: GPTScan parses contracts, analyzes call-graph reachability, filters candidates, matches scenarios and properties with GPT, then applies static analysis.The workflow supports standalone Solidity files and framework-based projects containing multiple files.
  • 4.1 Overview and Challenges: GPTScan must narrow candidate functions because projects may contain tens of Solidity files and non-vulnerable functions can affect GPT recognition.A Code4rena project averages 36 Solidity files, making entire-project analysis infeasible or costly.
  • 4.2 GPT-based Scenario and Property Matching: GPTScan decomposes vulnerabilities into scenarios describing relevant functionality and properties describing vulnerable attributes or operations.The paper applies this representation to ten common logic-vulnerability types.
  • 4.2 GPT-based Scenario and Property Matching: Scenario matching precedes property matching, and yes-or-no prompts constrain responses while JSON formatting supports structured scenario outputs.Separating the stages allows all scenarios to be queried in one prompt and reduces GPT costs.
  • 4.3 Multi-dimensional Function Filtering: GPTScan uses multidimensional filtering and reachability analysis to retain functions accessible to potential attackers before GPT matching.Filtering includes project files, OpenZeppelin functions, vulnerability-specific names, contents, parameters, and caller relations.

5 EVALUATION

GPTScan was evaluated across accuracy, static-confirmation effectiveness, performance, and vulnerability discovery. It achieved strong results on token contracts, detected most ground-truth vulnerabilities, reduced false positives, operated at low cost, and identified nine vulnerabilities absent from cited audit reports.

  • RQ1: Measuring False Positives: 4.39% false positive rate was achieved on non-vulnerable Top200 contracts, while precision was 90.91% on DefiHacks and 57.14% on larger Web3Bugs projects.Precision declined for Web3Bugs, which averaged 36 Solidity files per project, compared with about 2 files for Top200 and DefiHacks.
  • RQ2: Detecting Vulnerable Contracts: 12 false negatives included four Price Manipulation by AMM and three Risky First Deposit cases.Reported causes included absent alias analysis, inaccurate scenario or property matching, numerous slippage-check variants, and failure to distinguish comments from code.
  • RQ2: Detecting Vulnerable Contracts: 83.33% recall and 67.8% F1 were achieved on Web3Bugs, compared with 71.43% recall and 80% F1 on DefiHacks.GPTScan detected 40 true positives and missed 8 vulnerabilities on Web3Bugs; on DefiHacks it detected 10 and missed 4.
  • RQ3: Static Confirmation: 65.84% of false positive cases in Web3Bugs were filtered out by static confirmation, which had only a minor impact on false negatives.Static confirmation reduced raw candidate functions from 647 to 221, while excluding three ground-truth cases initially matched by GPT.
  • RQ4: Performance and Financial Overhead: 14.39 seconds and 0.01 USD were required on average to scan each thousand lines of Solidity code in the tested datasets.More complex functions in Web3Bugs and DefiHacks could not be filtered during earlier stages, increasing scan time and cost.
  • RQ5: New Vulnerability Discovery: 56% of newly discovered vulnerabilities were Risky First Deposit, including an empty-pool case where a depositor could obtain all pool shares.The paper also reports that 33% were Price Manipulation by AMM vulnerabilities involving user-controlled total supply.
  • RQ5: New Vulnerability Discovery: GPTScan identified 9 new vulnerabilities absent from the audit reports of Code4rena.The paper presents this result as demonstrating GPTScan’s value as a supplement to human auditors.

6 DISCUSSION

GPTScan’s discussion identifies implementation and model-choice limitations, including imprecise modifier filtering, non-path-sensitive static analysis, and unresolved trade-offs around GPT models and parameters.

  • Current limitations in design and implementation: Whitelist-based modifier filtering can produce false positives or false negatives, motivating modifier-definition retrieval and detailed semantic analysis.
  • Current limitations in design and implementation: GPTScan’s static analysis is not path-sensitive, so it may overlook whether specific execution paths are reachable under particular conditions.
  • The use of other GPT models and parameters: GPT-4 produced no notable improvement in a preliminary test while increasing cost 20 times, so GPTScan does not necessarily require a more powerful model.
  • The use of other GPT models and parameters: Higher temperature may produce more creative answers but could increase false positives or false negatives and make results harder to reproduce.

7 RELATED WORK

Related work spans traditional static analysis, symbolic execution, and newer GPT-based interactive approaches for smart-contract vulnerability detection and related software-testing tasks.

  • Traditional and symbolic analysis: Traditional tools such as Slither, Vandal, Ethainter, Zeus, and Securify analyze source code to detect smart-contract vulnerabilities.
  • Traditional and symbolic analysis: Symbolic execution tools including Manticore and Mythril perform bound checks and detect vulnerabilities in bytecode and source code.
  • GPT-based approaches: Earlier NLP-based methods generally fed code into binary- or multiclass-classification models before ChatGPT-era instructive and few-shot approaches emerged.
  • GPT-based approaches: Interactive GPT-based solutions have been applied to tasks including code repair and vulnerability detection.
  • GPT-based approaches: Fine-tuned GPT and execution-feedback systems have also supported GUI testing, Android testing, penetration testing, and code repair.

8 CONCLUSION

GPTScan combines GPT’s code understanding with static confirmation to detect smart-contract logic vulnerabilities. Across diverse datasets, it reports strong token-contract precision, acceptable large-project precision, over-70% recall, and practical scanning cost, while finding vulnerabilities missed by human auditors.

  • GPTScan matches candidate vulnerable functions using code-level scenarios and properties, then validates recognized variables and statements through static confirmation.
  • GPTScan achieves over 90% precision for token contracts and 57.14% precision for large projects, with over 70% recall for ground-truth logic vulnerabilities.
  • GPTScan is fast and cost-effective and can discover new vulnerabilities missed by human auditors.
  • Future work will expand GPTScan’s support for additional logic vulnerability types.
Loading 2308.03314v3…