PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications

Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, Yang Liu

arXiv:2403.16073v3cs.SE

TL;DR

LLMs show promise in smart-contract auditing, but even GPT-4 achieves only about 30% precision when both the decision and justification are correct. iAudit combines two-stage fine-tuning with agents that select and debate explanations, and reports 91.21% F1 and 37.99% explanation consistency. Its scope centers on logic vulnerabilities, and its evaluation compares it only with AI-based methods.

  • Problem

    Even GPT-4 achieves only about 30% precision when both the vulnerability decision and justification are correct, while off-the-shelf LLMs were not fine-tuned for Solidity auditing.

    Source: §I · p. 1

  • Method

    iAudit fine-tunes Detector to make vulnerability decisions and Reasoner to generate causes, then uses Ranker and Critic agents to select and debate explanations.

    Source: §I · pp. 1–2

  • Results

    91.21% F1 and 37.99% explanation consistency; GPT-4 achieved 24% consistency.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported evaluation compares iAudit favorably with prompt-based LLMs and traditional fine-tuned models, and its explanations align with ground-truth causes more than the tested alternatives.

    Source: §I · p. 2

    iAudit primarily targets logic vulnerabilities and was compared only with other AI-based methods, not program-analysis methods.

    Source: §VII · p. 11

Abstract

from arXiv · show

Smart contracts are decentralized applications built atop blockchains like Ethereum. Recent research has shown that large language models (LLMs) have potential in auditing smart contracts, but the state-of-the-art indicates that even GPT-4 can achieve only 30% precision (when both decision and justification are correct). This is likely because off-the-shelf LLMs were primarily pre-trained on a general text/code corpus and not fine-tuned on the specific domain of Solidity smart contract auditing. In this paper, we propose iAudit, a general framework that combines fine-tuning and LLM-based agents for intuitive smart contract auditing with justifications. Specifically, iAudit is inspired by the observation that expert human auditors first perceive what could be wrong and then perform a detailed analysis of the code to identify the cause. As such, iAudit employs a two-stage fine-tuning approach: it first tunes a Detector model to make decisions and then tunes a Reasoner model to generate causes of vulnerabilities. However, fine-tuning alone faces challenges in accurately identifying the optimal cause of a vulnerability. Therefore, we introduce two LLM-based agents, the Ranker and Critic, to iteratively select and debate the most suitable cause of vulnerability based on the output of the fine-tuned Reasoner model. To evaluate iAudit, we collected a balanced dataset with 1,734 positive and 1,810 negative samples to fine-tune iAudit. We then compared it with traditional fine-tuned models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) as well as prompt learning-based LLMs (GPT4, GPT-3.5, and CodeLlama-13b/34b). On a dataset of 263 real smart contract vulnerabilities, iAudit achieves an F1 score of 91.21% and an accuracy of 91.11%. The causes generated by iAudit achieved a consistency of about 38% compared to the ground truth causes.

I. INTRODUCTION

iAudit addresses limited LLM performance on smart-contract auditing with a two-stage fine-tuning framework and agents that select vulnerability explanations. It reports strong detection and explanation-alignment results, alongside ablation findings about voting, prompts, and call-graph information.

I. INTRODUCTION

  • The two-stage approach first fine-tunes Detector for vulnerability decisions, then Reasoner for causes, following a perception-then-analysis sequence.The authors report better detection than an integration model that outputs labels and reasons simultaneously.Source: §I · pp. 1–2
  • Ranker and Critic iteratively select and debate the most suitable cause from Reasoner’s outputs because fine-tuning alone does not identify the optimal cause.Detector can make multiple intuitive judgments, while Reasoner generates alternatives using variant prompts.Source: §I · p. 2
  • 91.21% F1 and 37.99% explanation consistency; GPT-4 ranked second at 24% consistency.iAudit also outperformed prompt-based LLMs reporting 60%+ and fine-tuned models reporting 80%+ F1.Source: §I · p. 2
  • The training set contains 1,734 vulnerable positive samples and 1,810 benign negative samples, with positive examples drawn from 263 audit reports.The authors compare iAudit with four traditional fine-tuned models and prompt-based GPT-4, GPT-3.5, and CodeLlama models.Source: §I · p. 2
  • Ablations found that majority voting improves detection performance and stability, while additional call-graph information can help some judgments but confuse others.Using multiple prompts also performed better than using a single prompt.Source: §I · p. 2

II. BACKGROUND

The background describes language-model pretraining and parameter-efficient fine-tuning, including LoRA as a lower-cost way to adapt models while retaining their weights.

A. Pre-trained Models and Large Language Models

  • Pre-trained models learn from large datasets and task setups, while LLMs use larger data and model scales; transformer models are categorized by encoder, decoder, or combined architectures.Examples include BERT and CodeBERT for encoder models, GPT for decoder models, and BART, T5, and CodeT5 for encoder-decoder models.Source: A. Pre-trained Models and Large… · p. 2

B. Parameter-Efficient Fine-Tuning

  • Parameter-efficient fine-tuning updates additional parameters while keeping large-model weights fixed, through adapters, LoRA, prefix tuning, or prompt tuning.The text describes LoRA as low-cost and potentially close to full fine-tuning in performance; adapters may add inference latency, while prefix and prompt tuning face structural constraints.Source: B. Parameter-Efficient Fine-Tuning · p. 3

C. Smart Contracts and Their Vulnerabilities

Smart-contract vulnerabilities can be difficult for static and dynamic tools to detect, while iAudit combines Detector and Reasoner fine-tuning with Ranker and Critic agents to produce decisions and explanations.

C. Smart Contracts and Their Vulnerabilities

  • Complex smart-contract vulnerabilities can be hard for static and dynamic detection tools to find, motivating human audits before public release.The background notes sandwich attacks as one example of complex behavior.Source: C. Smart Contracts and Their… · p. 3

III. DESIGN OF IAUDIT

  • Detector judges whether code is vulnerable, and Reasoner analyzes potential causes based on Detector’s decision.Connecting their outputs during training and inference forms iAudit’s two-stage fine-tuning approach.Source: §III · p. 3
  • Ranker selects or merges explanations, while Critic assesses its answer and can agree, request reranking, or suggest merging.The agents collaborate to identify an appropriate cause among Reasoner’s explanations.Source: §III · pp. 3–4
  • In the illustrated workflow, Detector votes across five inference paths, Reasoner generates ten answers, and Ranker revises its choice after Critic’s feedback.Ranker’s revised choice receives a 10/10 confidence score, which Critic accepts in the example.Source: §III · p. 4

A. Using Multi-prompt Tuning and Majority Voting for Effective Vulnerability Judgements in the Detector

Detector is fine-tuned on multiple prompt variants, then uses their predictions to make a majority-vote vulnerability judgment. Its voting ratio supplies a confidence score.

A. Using Multi-prompt Tuning and Majority Voting for Effective Vulnerability Judgements in the Detector

  • During training, Detector wraps the same input code in multiple prompts with different instruction formats.It is fine-tuned with LoRA in a generative manner, predicting label names as output.Source: A. Using Multi-prompt Tuning and… · pp. 4–5
  • The prompt template uses the input code as context and “The label is {Label Name}” as the target output, with “safe” or “vulnerable” as labels.The code is represented by the “{code}” placeholder.Source: A. Using Multi-prompt Tuning and… · p. 4
  • Fine-tuning minimizes the negative log probability of each output token conditioned on the input and preceding output tokens.The training objective sums over the output sequence length; θ denotes trainable LoRA parameters.Source: A. Using Multi-prompt Tuning and… · p. 5
  • Multiple prompt variants represent different inference paths, and majority voting selects the final safe-or-vulnerable label.The winning label must receive more than half of the votes; its voting ratio is used as the confidence score.Source: A. Using Multi-prompt Tuning and… · pp. 4–5

B. Connecting Detector for Reasoner’s Tuning & Inference

Reasoner takes Detector’s majority-voted assessment and generates multiple explanations through varied prompts, with and without call context. Ranker and Critic then select and refine the cause.

B. Connecting Detector for Reasoner’s Tuning & Inference

  • Reasoner conditions its explanations on Detector’s majority-voted label and generates ten answers across prompt variants.The prompts comprise five instruction formats with call relationships and five without them; the study examines the effect of including those relationships.Source: B. Connecting Detector for Reasoner’s… · p. 5

C. Ranking and Debating the Optimal Vulnerability Cause

  • Ranker selects and can merge Reasoner’s explanations, while Critic evaluates the answer against the code and directs agreement, reranking, or merging.Ranker’s criteria require reasons to be code-related, factual, relevant to the decision, and unsupported by no unstated assumptions.Source: C. Ranking and Debating the Optimal… · pp. 5–6
  • Ranker and Critic use Mixtral 8x7B-Instruct, selected in part because its outputs more readily follow predetermined formats.The authors describe its capability as close to that of larger LLMs.Source: C. Ranking and Debating the Optimal… · p. 6

D. High-quality Training Data Collection and Enhancement

The dataset combines vulnerable functions and explanations from audit reports with benign samples derived through GPT-4-based vulnerability matching. GPT-3.5 expands explanations with proof-of-concept and mitigation details.

D. High-quality Training Data Collection and Enhancement

  • The 1,734 positive samples are vulnerable functions with reasons collected from 263 smart contract audit reports.The reports were assembled from Solodit, including reports from reputable industry companies.Source: D. High-quality Training Data… · p. 6
  • Negative samples are labeled when hierarchical GPT-based matching finds no vulnerability information matching the tested code.The process clusters vulnerability knowledge by functionality, summarizes group functionality, then matches group, functionality, and negligence.Source: D. High-quality Training Data… · p. 6
  • The balanced dataset contains 1,734 positive samples and 1,810 negative samples; it is divided into training, validation, and test subsets of 2,268, 567, and 709 entries, respectively.Source: D. High-quality Training Data… · pp. 6–7
  • GPT-3.5 expands short or variable-quality vulnerability explanations with code context, proof-of-concept details, and mitigation recommendations.The authors chose GPT-3.5 for its cost-effectiveness and comparable quality for this task in their manual comparison, and constrained prompts to align explanations with vulnerabilities.Source: D. High-quality Training Data… · p. 7

IV. EVALUATION

The evaluation compares iAudit with zero-shot LLMs and fine-tuned code models across detection, explanation alignment, and design questions. It also reports expert-verified findings from two bounty projects.

A. Experimental Setup

  • The benchmark set includes zero-shot CodeLlama-13b/34b-Instruct, GPT-3.5, and GPT-4, plus fine-tuned CodeBERT, GraphCodeBERT, CodeT5, UnixCoder, and CodeLlama-13b.The first four code models underwent full-model fine-tuning; CodeLlama-13b used LoRA and a last-token representation for classification.Source: A. Experimental Setup · p. 7

B. Research Questions (RQs)

  • The research questions cover detection comparisons, explanation alignment, two-stage versus integrated detection-and-reasoning, majority voting, and call-graph effects.The call-graph questions address both Detector performance and the Reasoner-Ranker-Critic explanation pipeline.Source: B. Research Questions (RQs) · p. 7
  • Experts or project teams recognized six critical vulnerabilities found by iAudit in two anonymous Code4rena bounty projects.The authors report that one finding was not discovered by any tools; case studies are in the supplementary materials.Source: B. Research Questions (RQs) · p. 8

C. RQ1 - Performance Comparison

iAudit’s Detector outperforms the compared fine-tuned and in-context methods on vulnerability detection, though its precision is lower than several traditional models. The section also reports strong zero-shot comparison results after fine-tuning.

C. RQ1 - Performance Comparison

  • 0.9121 F1, 0.8934 Accuracy, and 0.9111 Precision were iAudit’s reported zero-shot comparison scores after fine-tuning.GPT-4 and GPT-3.5 each had recall of 1, with the test data all labeled vulnerable.Source: C. RQ1 - Performance Comparison · p. 8
  • 0.9121 F1, 0.8934 Recall, and 0.9111 Accuracy were iAudit’s highest scores among the compared detection methods.CodeLlama-13b-class ranked second in vulnerability detection rate; CodeT5 had the highest precision at 0.9431.Source: C. RQ1 - Performance Comparison · p. 8
  • iAudit produced more false positives than CodeT5, GraphCodeBERT, and CodeBERT, including 10 unique false positives involving more complex code.The authors attribute this primarily to iAudit’s 16k context versus the other models’ 512-token truncation and the dataset’s higher complexity in vulnerable code.Source: C. RQ1 - Performance Comparison · p. 8
  • iAudit’s Detector exceeded traditional full-model fine-tuning, LoRA classification, and in-context LLM methods, while fine-tuned models also outperformed zero-shot learning.The reported conclusion identifies CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder among the traditional fine-tuning comparisons.Source: C. RQ1 - Performance Comparison · p. 8

D. RQ2 - Explanation Alignment

iAudit’s Reasoner explanations aligned with ground-truth vulnerability causes more often than the compared baseline methods. The reported alignment metric is the share of test-set explanations matching root causes.

D. RQ2 - Explanation Alignment

  • The alignment assessment used GPT-4 to check generated explanations against root causes, with automated annotation prompts from Y. Sun et al. [15].The y-axis reports the percentage of test-set explanations matching root causes.Source: D. RQ2 - Explanation Alignment · p. 8
  • 37.99% of iAudit explanations matched root causes, versus no more than 25% for any baseline; GPT-4 ranked second at 24.13%.Fig. 6 compares alignment with ground-truth reasons.Source: Figure 6 · p. 9, D. RQ2 - Explanation Alignment · p. 9
  • CodeLlama-13b had the weakest performance among the compared methods on explanation consistency.The authors summarize Reasoner’s output as more rational than the other models’ output.Source: D. RQ2 - Explanation Alignment · p. 9

E. RQ3 - Two-stage Approach vs. An Integration Model

The section compares iAudit’s staged Detector-then-Reasoner design with joint label-and-explanation generation, then evaluates prompt variation and majority voting. The staged approach performs better on detection, while prompt variation and voting improve performance and stability.

E. RQ3 - Two-stage Approach vs. An Integration Model

  • iAudit’s step-wise approach outperformed integration models across four metrics, and the authors concluded it achieved better detection performance than joint label-and-reason output.The integrated models had higher accuracy for negative samples but lower recall for positive samples; mixed training with label-only data improved detection, reaching 0.8433 F1.Source: E. RQ3 - Two-stage Approach vs. An… · p. 9
  • Joint generation yielded higher negative-sample accuracy but lower positive-sample recall, which the authors attributed potentially to label-related loss being diluted by longer output sequences.Adding label-only training data improved the integrated model’s detection performance while evaluation still required both labels and explanations.Source: E. RQ3 - Two-stage Approach vs. An… · p. 9

F. RQ4 - Effectiveness of Majority Voting

G. RQ5 - Impact of Additional Information

Adding function-call context did not improve overall detection performance, although Ranker-Critic selected many reasons generated with that information. The related work spans graph-based, LLM, and hybrid approaches, and identifies domain-specific tuning as an unresolved gap.

G. RQ5 - Impact of Additional Information

  • Function-call information can interfere when it is not relevant to the task, and merely adding it does not directly improve vulnerability detection.The authors identify constructing effective contextual information as an open research question.Source: G. RQ5 - Impact of Additional… · p. 10
  • Call-context variants had lower precision, accuracy, and F1 than iAudit’s Detector, with almost the same recall.The comparison covers prompts with call information and a mixture of prompts with and without it; Table IV is titled “Impact with or without Additional Information.”Source: Table IV · p. 10, G. RQ5 - Impact of Additional… · p. 10
  • 65% of Ranker-Critic’s selected reasons came from prompts with calling information, while 35% came from prompts without it.Fig. 9 shows the final reason distribution.Source: Figure 9 · p. 10, G. RQ5 - Impact of Additional… · p. 10

V. RELATED WORK

  • Prior work includes graph neural networks, interpretable graph features, data-flow graphs, LLM vulnerability studies [12], –, and LLM-static-analysis or verification hybrids, –[71].The authors state that these studies have not tuned domain-specific knowledge into the models themselves, which they say could not effectively detect logic bugs.Source: §V · pp. 10–11

VI. THREATS TO VALIDITY

Threats to validity identify data and model-selection concerns; the limitation and conclusion sections bound iAudit’s scope and summarize its approach and future work.

VI. THREATS TO VALIDITY

  • Data bias could impair generalization, and label precision affects performance; the authors mitigate these risks with verified public reports, cleaning, and standardized code.They removed external links and optimized GPT-3.5/GPT-4 prompts while adapting prompts for open-source models.Source: §VI · p. 11
  • Selected reasons from MoE matched real reasons with about 38% consistency; the Ranker-Critic loop was capped at five iterations to control costs.The authors used early stopping and chose MoE based on output-format strictness and operational speed.Source: §VI · p. 11

VII. LIMITATION

  • iAudit’s primary advantage is detecting logic vulnerabilities; traditional contract vulnerabilities are not its major usage scenario, hallucinations may persist, and GPU compression may reduce performance.The paper compares iAudit only with other AI-based methods; traditional vulnerabilities are described as theoretically more suitable for program-analysis methods.Source: §VII · p. 11

VIII. CONCLUSION

  • iAudit combines LoRA-fine-tuned Detector and Reasoner models with Ranker and Critic agents, outperforming zero-shot LLM learning and traditional full-model fine-tuning in zero-shot scenarios.The conclusion also reports exploring majority voting, LoRA training methods, and additional calling context.Source: §VIII · p. 11
Loading 2403.16073v3…