PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives

Sihao Hu, Tiansheng Huang, Fatih İlhan, Selim Furkan Tekin, Ling Liu

arXiv:2310.01152v2cs.CRcs.AI

TL;DR

Smart-contract vulnerability detection must identify true vulnerabilities while limiting false positives, a tension intensified by generating more diverse answers. GPTLENS separates detection into auditor-led generation and critic-led discrimination. In preliminary experiments, it identified true vulnerabilities in 76.9% of contracts versus 38.5% for conventional one-stage top-1 detection.

  • Problem

    LLM-based smart-contract detection faces a tension between generating answers likely to include true vulnerabilities and limiting the false positives that make verification burdensome.

    Source: §I · p. 1

  • Method

    GPTLENS uses an LLM as auditor to generate vulnerability candidates and as critic to assess their validity and rank them.

    Source: §I · pp. 1–2

  • Results

    GPTLENS identified true vulnerabilities in 76.9% of 13 contracts, compared with 38.5% for conventional one-stage detection using top-1 output.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework is entirely LLM-driven and does not require specialist expertise in smart contracts during end-to-end detection.

    Source: §VII · p. 8

    The paper identifies two primary limitations of current LLM-powered detection.

    Source: C. Limitations of Current LLM-powered… · p. 3

Abstract

from arXiv · show

This paper provides a systematic analysis of the opportunities, challenges, and potential solutions of harnessing Large Language Models (LLMs) such as GPT-4 to dig out vulnerabilities within smart contracts based on our ongoing research. For the task of smart contract vulnerability detection, achieving practical usability hinges on identifying as many true vulnerabilities as possible while minimizing the number of false positives. Nonetheless, our empirical study reveals contradictory yet interesting findings: generating more answers with higher randomness largely boosts the likelihood of producing a correct answer but inevitably leads to a higher number of false positives. To mitigate this tension, we propose an adversarial framework dubbed GPTLens that breaks the conventional one-stage detection into two synergistic stages $-$ generation and discrimination, for progressive detection and refinement, wherein the LLM plays dual roles, i.e., auditor and critic, respectively. The goal of auditor is to yield a broad spectrum of vulnerabilities with the hope of encompassing the correct answer, whereas the goal of critic that evaluates the validity of identified vulnerabilities is to minimize the number of false positives. Experimental results and illustrative examples demonstrate that auditor and critic work together harmoniously to yield pronounced improvements over the conventional one-stage detection. GPTLens is intuitive, strategic, and entirely LLM-driven without relying on specialist expertise in smart contracts, showcasing its methodical generality and potential to detect a broad spectrum of vulnerabilities. Our code is available at: https://github.com/git-disl/GPTLens.

I. INTRODUCTION

The paper examines how LLMs could support smart-contract vulnerability detection, where auditing matters because deployed contracts are immutable and exploitation has caused large financial losses. It introduces GPTLENS to separate vulnerability generation from discrimination, addressing the tension between finding true vulnerabilities and limiting false positives.

I. INTRODUCTION

  • LLMs may offer broader vulnerability coverage and intermediate reasoning, while existing tools such as Slither rely on expert-designed detectors for specific vulnerability types.The paper presents transparency and trustworthiness as potential benefits of using LLM-generated reasoning.Source: §I · p. 1
  • The approach targets a one-stage detection tension: generating more, more-random answers can raise the chance of finding a true vulnerability while also increasing false positives.The paper describes this conflict as a Catch-22 dilemma.Source: §I · pp. 1–2
  • GPTLENS uses LLM auditor agents to generate vulnerability candidates and a critic agent to assess them, then ranks answers by scores.The critic evaluates correctness, severity, and profitability; auditor generation uses high randomness to broaden candidate answers.Source: §I · p. 2
  • 76.9% of 13 contracts were identified with a true vulnerability by GPTLENS, compared with 38.5% for conventional one-stage detection using top-1 output.At the trial level, top-1 accuracy rose from 33.3% to 59.0%.Source: §I · p. 2
  • The paper contributes a systematic analysis of LLM-powered detection opportunities and challenges, alongside a framework intended to work without specialist expertise.It describes GPTLENS as potentially generalizable across a range of vulnerability types.Source: §I · pp. 1–2

II. LLM-POWERED VULNERABILITY DETECTION

The paper distinguishes predefined-category prompting from open-ended prompting, which asks an LLM to identify and describe vulnerabilities without fixed names. It discusses LLM-powered detection’s potential for broader coverage and explanations, while noting efficiency tradeoffs.

A. Standard Detection Paradigms

  • Binary prompting asks whether a contract has a specified vulnerability, while multi-class prompting asks the LLM to classify multiple predefined vulnerability types.Both are close-ended approaches; the service may be queried once per vulnerability category in binary prompting.Source: A. Standard Detection Paradigms · p. 2
  • Open-ended prompting asks the LLM to identify and describe any potential vulnerabilities in natural language without predefined vulnerability names.The paper adopts this paradigm in its experiments to retain generality.Source: A. Standard Detection Paradigms · p. 2, B. Advantages of LLM-powered Detection · p. 3

B. Advantages of LLM-powered Detection

  • LLMs may describe vulnerabilities beyond fixed categories, whereas traditional tools and supervised classifiers are constrained by predefined patterns or labels.The paper also presents intermediate reasoning, exploit examples, and repair suggestions as potential sources of interpretability.Source: B. Advantages of LLM-powered Detection · pp. 2–3
  • LLM services offer fast online inference, but offline pre-training requires substantial computational resources and time.Source: B. Advantages of LLM-powered Detection · p. 3

C. Limitations of Current LLM-powered Detection

The paper frames current LLM-powered detection as having two primary limitations and introduces a smart-contract code example in this discussion.

C. Limitations of Current LLM-powered Detection

1) Large number of false positives:

The paper describes false positives as a major practical challenge for LLM-powered detection, arising both from factual misreadings of code and from correct but unexploited vulnerabilities. It argues that ranking should consider severity and profitability as well as correctness.

1) Large number of false positives:

  • One measurement study found GPT-4 identified 32 of 73 vulnerabilities across 52 DeFi attacks while producing 740 false positives, for 4.15% precision.The same passage reports Claude precision of 4.3%, and another study reports GPT-3.5 and GPT-4 precision of 19.7% and 22.6% on nine common vulnerability categories.Source: 1) Large number of false positives · p. 3
  • Factual errors can occur when an LLM mistakes code details, such as treating the Transfer event in multiTransfer as an external call and flagging re-entrancy.The passage notes that modifiers, conditions, error handling, and events can be overlooked, especially in large inputs.Source: 1) Large number of false positives · p. 3
  • A vulnerability may be correctly identified yet remain unexploited; the paper therefore says ranking should consider correctness, severity, and profitability.In its example, arbitrary balance manipulation was considered a false positive because it was less severe and profitable than the exploited access-control vulnerability.Source: 1) Large number of false positives · pp. 3–4

2) Large number of false negatives:

The paper frames generation randomness as a source of missed vulnerabilities and proposes separating broad candidate generation from critic-led discrimination. The stages aim to improve the chance of including true vulnerabilities while reducing false positives.

2) Large number of false negatives:

  • 43.8% and 35.6% recalls for GPT-4 and Claude-1.3 on 52 DeFi attacks [10] illustrate misses; authors distinguish hard cases from detectable vulnerabilities missed through randomness.At 100 samples, Codex pass@k for t=0.8 exceeded t=0.2 by 13 absolute percentage points [9], but this oracle-selected best-of-k result is unavailable in real-world applications.Source: 2) Large number of false negatives · p. 4, Figure 1 · p. 4

III. TWO-STAGE ADVERSARIAL DETECTION FRAMEWORK

  • GPTLENS separates conflicting goals—finding more correct vulnerabilities and reducing false positives—into generation and discrimination stages, with an LLM acting as auditor and critic.The framework is presented as entirely LLM-driven and does not rely on expert knowledge during end-to-end detection.Source: §III · p. 4
  • Auditors generate a broad set of vulnerabilities with associated functions and reasoning, aiming to include the correct answer.The framework goal also calls for explaining each identified vulnerability.Source: §III · p. 4
  • The critic scrutinizes and ranks candidates using correctness, severity, and profitability, relying heavily on auditors’ reasoning.This ranking is intended to place the correct answer above false positives.Source: §III · p. 4
  • The authors argue discrimination is easier because it assesses recognized vulnerabilities at function level, whereas generation searches an entire contract project.They cite prior GAN work as corroboration for pairing a simple discriminator with a sophisticated generator, [33].Source: §III · p. 5
  • In the Listing 2 example, the critic concludes that the auditor’s reported re-entrancy risk is a factual error.The auditor’s reasoning claims the function does not follow the Checks-Effects-Interactions pattern.Source: §III · p. 5, Figure 3 · p. 5

A. Generation Stage

Auditors generate vulnerability candidates, while a critic evaluates and ranks them. The setup uses one low-temperature critic to keep scoring consistent.

A. Generation Stage

  • Each of n auditors identifies up to m vulnerabilities, returns JSON, and uses a temperature set according to n and k to limit similar outputs.When n is large, the authors recommend a higher temperature to prevent auditors from generating similar answers.Source: A. Generation Stage · p. 5

B. Discrimination/ranking Stage

  • The critic assesses candidates by correctness, severity, and profitability, since a correct vulnerability may have little severity or attacker value.The critic is intended to emulate an oracle choosing the best answer from false positives.Source: B. Discrimination/ranking Stage · p. 6
  • The critic scores vulnerabilities with explanations, then ranks them in descending score order and returns the top k.If the candidate list contains the ground-truth answer, the critic is intended to place it near the front.Source: B. Discrimination/ranking Stage · p. 6
  • Experiments use one critic agent and a low temperature to keep evaluations consistent and reduce randomness.Source: B. Discrimination/ranking Stage · p. 6

IV. EXPERIMENT

The experiments evaluate GPTLENS on 13 smart contracts with reported vulnerabilities, comparing settings that vary how auditors generate candidates and how answers are selected.

A. Experimental Settings

  • Experiments use 13 Etherscan smart contracts with reported vulnerabilities, labeled from CVE descriptions.Source: A. Experimental Settings · p. 6
  • GPT-4 is the backend; auditors use temperature 0.7 and the critic 0, while token limits prevent testing larger auditor counts or generation caps.Oracle settings use ground-truth labels and therefore represent ideal performance.Source: A. Experimental Settings · p. 6
  • Competitor settings compare one-stage and multi-answer auditor outputs selected randomly, by a critic, or by a ground-truth oracle.The experiments use one or two auditors, each generating up to three vulnerabilities in multi-answer settings.Source: A. Experimental Settings · p. 6

B. Performance Comparison

GPTLENS improves detection over one-stage output in the reported comparisons, and the case study shows the critic rejecting false positives while retaining a genuine vulnerability. The paper also identifies diversity, reasoning, agent collaboration, external knowledge, and scoring consistency as future directions.

B. Performance Comparison

  • Generating more answers can introduce false positives while uncovering contracts missed by one-answer generation, and A+C outperforms A+R.The authors attribute the A+C comparison to the critic discerning true vulnerabilities from false positives.Source: B. Performance Comparison · p. 6
  • A+C(n=1,m=3) raises Hit ratio (CVE) from 38.5% to 76.9% and Hit ratio (trial) from 33.3% to 46.2% over A(n=1,m=1).Source: B. Performance Comparison · p. 6
  • Hit ratio (trial) rises from 46.2% to 59.0% when increasing A+C from one auditor to two.Source: B. Performance Comparison · p. 7

C. Case Study

  • In the case study, the auditor proposes two false race-condition findings and a correctly reasoned constructor bug under a different name.Source: C. Case Study · p. 7
  • The critic scores the two incorrect explanations low and the accurate finding highest, enabling GPTLENS to identify the genuine vulnerability.Source: C. Case Study · p. 7

V. FUTURE DIRECTIONS

  • Future work proposes increasing generation diversity while maintaining the same number of generated samples.Source: §V · p. 7
  • Token limits require batching many vulnerabilities for discrimination, which can make scores inconsistent even at low temperature.Source: §V · p. 7
  • Other proposed directions include richer reasoning procedures, additional collaborating agents, external knowledge, and LLM-assisted software-engineering tools.Source: §V · p. 8

VI. RELATED WORK

The related work surveys traditional analysis, deep-learning approaches, and generative LLM methods for smart-contract vulnerability detection. It contrasts GPTLENS with rule-based filtering approaches that require expert knowledge and engineering effort.

VI. RELATED WORK

  • Traditional approaches include static and dynamic analysis, symbolic execution, and formal verification.The surveyed tools analyze code patterns, runtime behavior, execution paths, or user-defined specifications.Source: §VI · p. 8
  • Deep-learning methods use sequence models, CNNs, and graph neural networks, while hybrid systems combine learning with traditional tools.Examples include ESCORT and xFuzz, which distill outputs from tools such as Slither and Mythril.Source: §VI · p. 8
  • Prior LLM studies report precision challenges from false positives, while GPTScan filters them with rule-based processing requiring expertise and engineering effort.The paper presents GPTLENS as lighter-weight and entirely LLM-driven.Source: §VI · p. 8

VII. CONCLUSION

The conclusion presents GPTLENS as a two-stage auditor-and-critic framework for balancing vulnerability generation against false positives. It reports improvements over one-stage detection and describes the approach as entirely LLM-driven, without specialist smart-contract expertise.

VII. CONCLUSION

  • GPTLENS uses an auditor to generate diverse vulnerabilities and reasoning, then a critic to assess their validity and reasoning.Source: §VII · p. 8
  • The authors report pronounced improvements over conventional one-stage detection and potential generalization across a broad spectrum of vulnerabilities.Source: §VII · p. 8
  • The framework is entirely LLM-driven and does not require specialist expertise in smart contracts.Source: §VII · p. 8

APPENDIX A SOURCE CODE OF CASE STUDY

The appendix lists the Solidity code for a BAFC token built on ERC20-style interfaces and token contracts. It includes account and liquidity controls, delegated transfers, and owner-managed settings.

APPENDIX A SOURCE CODE OF CASE STUDY

  • The listing defines ERC20Basic, BasicToken, StandardToken, and BAFCToken, with the final token inheriting the standard-token implementation.The listing is identified as smart contract code reported in CVE 2018-19830.Source: Appendix A · p. 10
  • Transfers use SafeMath and check sender restrictions, receiver freeze status, and whether transfers are enabled.transfer rejects frozen receivers, while transferFrom requires both source and destination accounts to be unfrozen.Source: Appendix A · p. 10
  • Owner-only functions freeze accounts, transfer ownership to a nonzero address, and switch the transferable setting.The contract also exposes functions to query freeze and liquidity status.Source: Appendix A · p. 10
  • Delegated transfers update balances and allowance, while approve blocks changing a nonzero allowance directly to another nonzero value.The transferFrom function checks the source and destination freeze status before completing the transfer.Source: Appendix A · p. 10
  • BAFCToken sets its name, symbol, and decimals, assigns its total supply to the deploying owner, and reverts on plain payable calls.The listing sets the name to Business Alliance Financial Circle, the symbol to BAFC, and decimals to 18.Source: Appendix A · p. 10
Loading 2310.01152v2…