PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains

Reza M. Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, Amritraj Singh

arXiv:1809.02702v1cs.CR

TL;DR

Smart contracts can be difficult to secure, and vulnerabilities have caused substantial financial losses, while empirical evidence on automated testing tools remains limited. The paper experimentally evaluates four open-source static tools on real-world Ethereum Solidity contracts; SmartCheck leads in detection effectiveness, while Mythril leads in accuracy with the fewest false alarms.

  • Problem

    Smart contract code can contain vulnerabilities with serious financial consequences, and the empirical evidence on automated security testing tools is described as embryonic.

    Source: §1 · p. 2

  • Method

    The authors test four open-source static-analysis tools on ten real-world Ethereum Solidity contracts, assessing vulnerability-detection effectiveness and accuracy.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    SmartCheck is statistically more effective than the other tools at the 95% significance level (p < 0.05), while Mythril is significantly accurate (p < 0.05) and issues the fewest false alarms.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The study contributes a methodology and comparative results that offer statistical insights into challenges in smart-contract security testing.

    Source: §7 · p. 10

    The evaluation uses a subset of well-known open-source contracts and tools commonly cited and used at the time of the experiment.

    Source: §4 · p. 5

Abstract

from arXiv · show

The emerging blockchain technology supports decentralized computing paradigm shift and is a rapidly approaching phenomenon. While blockchain is thought primarily as the basis of Bitcoin, its application has grown far beyond cryptocurrencies due to the introduction of smart contracts. Smart contracts are self-enforcing pieces of software, which reside and run over a hosting blockchain. Using blockchain-based smart contracts for secure and transparent management to govern interactions (authentication, connection, and transaction) in Internet-enabled environments, mostly IoT, is a niche area of research and practice. However, writing trustworthy and safe smart contracts can be tremendously challenging because of the complicated semantics of underlying domain-specific languages and its testability. There have been high-profile incidents that indicate blockchain smart contracts could contain various code-security vulnerabilities, instigating financial harms. When it involves security of smart contracts, developers embracing the ability to write the contracts should be capable of testing their code, for diagnosing security vulnerabilities, before deploying them to the immutable environments on blockchains. However, there are only a handful of security testing tools for smart contracts. This implies that the existing research on automatic smart contracts security testing is not adequate and remains in a very stage of infancy. With a specific goal to more readily realize the application of blockchain smart contracts in security and privacy, we should first understand their vulnerabilities before widespread implementation. Accordingly, the goal of this paper is to carry out a far-reaching experimental assessment of current static smart contracts security testing tools, for the most widely used blockchain, the Ethereum and its domain-specific programming language, Solidity to provide the first...

1 INTRODUCTION

Smart contracts can automate agreements on decentralized blockchains, but their code can contain vulnerabilities with serious financial consequences. This paper evaluates four open-source static testing tools for Ethereum Solidity contracts, focusing on vulnerability detection effectiveness and accuracy.

1 INTRODUCTION

  • Smart contract bugs can lead to serious financial harm, while their domain-specific language semantics and testability make safe development difficult.The introduction cites the 2016 splitDAO reentrancy incident and a 2017 code bug as examples.Source: §1 · p. 2
  • The study evaluates Oyente, Mythril, Securify, and SmartCheck on vulnerability detection effectiveness and accuracy for Ethereum smart contracts written in Solidity.The authors frame this as a first-time empirical evaluation in an area where empirical knowledge is embryonic.Source: §1 · p. 2
  • Automated static code analysis before deployment is presented as a way for developers to test contracts for vulnerabilities before they reach immutable blockchains.The stated evaluation concerns open-source tools for Ethereum and Solidity.Source: §1 · p. 2
  • The research questions ask how effectively the tools detect vulnerabilities, which tool is most effective, and how accurately they identify true vulnerabilities.Source: §1 · p. 2
  • Detection effectiveness is measured as recall or sensitivity, while accuracy is described as essentially a Youden Index.The introduction identifies true positives, false negatives, true negatives, and false positives as building-block metrics.Source: §1 · p. 2

2 BACKGROUND

The background introduces automated Solidity security-analysis tools and the Ethereum environment in which smart contracts run. It identifies four open-source tools and explains how Solidity contracts are compiled and deployed to the EVM.

2.1 Automated Smart Contract Security Testing Tools

  • The paper selects four open-source tools—Oyente, Mythril, Securify, and SmartCheck—and describes their rigor and the artifact each analyzes.The analysis basis may be source code, bytecode, or binary.Source: §2.1 · p. 3

2.2 Solidity, EVM, and Ethereum Smart Contracts

  • Ethereum provides a decentralized computing platform whose EVM executes bytecode, while developers typically write contracts in Solidity and compile them for deployment.Source: §2.2 · p. 3
  • The Puzzle example illustrates a contract deployed through a creation transaction, then invoked by users whose submitted solutions can trigger its reward logic.Deployment generates a unique address and assigns local storage on the blockchain.Source: §2.2 · p. 4

3 EXPERIMENTAL SETUP

The experiments applied security-testing tools to selected Solidity contracts, recorded their vulnerability findings, and evaluated them using classification metrics and statistical tests.

3 EXPERIMENTAL SETUP

  • Tools were randomly selected and applied to randomly chosen contracts; the researchers recorded reported threats and checked whether they identified known vulnerabilities.Tools without a web interface were run through Remix or solc where possible, after contracts were checked for compilation errors.Source: §3 · p. 4
  • Tool findings were assessed with true positives, false negatives, true negatives, and false positives.Source: §3 · p. 4
  • The study used ten open-source Solidity contracts from Trail of Bits and Ethernaut, spanning common vulnerability classes; Mythril’s bundled contracts were excluded for fairness.The researchers noted difficulty finding representative contracts without legal or privacy issues; open sourcing was intended to support replication.Source: §3 · pp. 4–5
  • The experiments used solc v0.4.21 and ANOVA, with a 5% significance level after normality checks.The analysis was run in IBM SPSS Statistics v.25.Source: §3 · p. 5

4 EXPERIMENTAL DESIGN AND VALIDITY

The study used a randomized-block design to compare four tools on the same ten contracts, and assessed external and conclusion validity.

4 EXPERIMENTAL DESIGN AND VALIDITY

  • Each of four tools was randomly applied to the same ten contracts in a randomized-block design, separating variability associated with tools from variability associated with contracts.The model represents each score as an overall mean plus contract and tool effects and an error term.Source: §4 · p. 5
  • External validity is bounded by the subset of well-known open-source contracts and commonly cited tools available at the time of the experiment.The authors describe the contracts as suitable in size and functionality and containing diverse common vulnerabilities.Source: §4 · p. 5
  • For conclusion validity, normality checks informed the choice of ANOVA and LSD tests, and the authors report that ANOVA assumptions were met.They characterize the resulting threat to conclusion validity as potentially minor.Source: §4 · p. 5

5 RESULTS AND DISCUSSION

The authors compare four smart-contract security tools using ROC analysis and statistical tests. SmartCheck leads in vulnerability-detection effectiveness, while Mythril and SmartCheck show higher accuracy; the results indicate a trade-off between detection capability and accuracy.

5 RESULTS AND DISCUSSION

5.1 Analysis of Effectiveness (RQ1)

  • ROC analyses describe Oyente as weak, Securify as prone to false positives, Mythril as detecting more vulnerabilities than Oyente and Securify, and SmartCheck as highly sensitive.Source: §5.1 · p. 6
  • SmartCheck achieved the highest effectiveness scores across all ten contracts, although no tool detected every vulnerability.Source: §5.1 · p. 6, Figure 8 · p. 7, Figure 9 · p. 7
  • ANOVA found a significant difference in effectiveness (p-value 0.0003 < 0.05), and LSD comparisons found SmartCheck significantly different from every other tool.Source: §5.1 · p. 7, Table 3 · p. 7, Table 4 · p. 8

5.2 Analysis of Accuracy (RQ2)

  • Mythril and SmartCheck recorded the highest accuracy scores across nearly all ten contracts; Mythril had the highest average accuracy despite lower effectiveness than SmartCheck.Source: §5.2 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8
  • ANOVA found accuracy differed significantly (p-value 0.0002 < 0.05), and LSD tests found Mythril and SmartCheck significantly different from the remaining tools.Source: Table 5 · p. 8, §5.2 · p. 9, Table 6 · p. 9
  • The experiments point to a trade-off between vulnerability-detection capability and accuracy for Ethereum smart-contract security tools.Source: §5.2 · p. 9

6 RELATED WORK

Related work covers studies describing smart-contract vulnerabilities and approaches for verification or mitigation, while sound empirical evaluation of security tools remains scarce.

6 RELATED WORK

  • 73.33% of Solidity contracts implemented by new developers had security vulnerabilities in Parizi et al.’s comparison of Solidity, Pact, and Liquidity.The study also found Solidity most usable, while no known vulnerabilities were found in the Pact and Liquidity contracts.Source: §6 · p. 9
  • Atzei et al. classified Ethereum and Solidity vulnerabilities across Solidity, EVM, and blockchain levels, while Destefanis et al. traced a Parity wallet freeze to negligent programming.Atzei et al. recommended formal verification and human-readable, Turing-incomplete languages.Source: §6 · pp. 9–10
  • Verification approaches include semantics-based smart-contract modeling with simulation and statistical model checking, contract translation to F* [8], and FSolidM finite-state-machine design and generation [27].Bhargavan et al. proposed prototype tools translating contracts to F*; Grishchenko et al. formalized EVM bytecode small-step semantics in F*.Source: §6 · p. 10
  • ZEUS combines abstract interpretation, symbolic model checking, and constrained Horn clauses to verify contract safety; its evaluation found 94.6% of contracts securely vulnerable.The evaluated contracts contained cryptocurrency worth more than USD 0.5 billion.Source: §6 · p. 10
  • Hydra models bug bounties using N-of-N-version programming, while logic-based contracts were presented as complementary to procedural approaches for contract activities.Hydra was applied to sample contracts, and the logic-based work described potential complementarity across negotiation, enforcement, monitoring, and dispute resolution.Source: §6 · p. 10
  • The authors describe sound empirical studies as lagging behind primary studies and characterize their statistical evaluation of tool effectiveness and accuracy as the first of its kind.The evaluation specifically concerns security-analysis tools for smart contracts.Source: §6 · p. 10

7 CONCLUSIONS

The evaluation found SmartCheck more effective than the other tested tools and Mythril significantly more accurate with the fewest false alarms. The authors present comparable tool results and a fine-grained empirical methodology while noting that empirical evaluation, especially for IoT, remains scarce.

7 CONCLUSIONS

  • SmartCheck was statistically more effective than the other tested tools (p < 0.05), while Mythril was significantly more accurate and issued the fewest false alarms (p < 0.05).The evaluation covered open-source automated analysis tools on ten real-world Ethereum smart contracts written in Solidity; the authors suggest SmartCheck may be most effective but less accurate than Mythril.Source: §7 · p. 10
  • The study contributes a fine-grained empirical methodology, comparable experimental results for state-of-the-art tools, and statistical tests with insights into smart-contract testing challenges.The authors intend these contributions to support future empirical studies.Source: §7 · p. 10
  • Empirical evaluation of smart-contract security testing remains scarce, especially in relation to IoT, which the authors describe as an area still in its infancy.The authors view IoT combined with blockchain smart contracts as promising for trustworthy and secure networks.Source: §7 · pp. 10–11
Loading 1809.02702v1…