Source-linked AI summary
Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts
Thomas Durieux, João F. Ferreira, Rui Abreu, Pedro Cruz
TL;DR
Comparing and reproducing automated smart-contract analysis research is difficult. The paper evaluates 9 tools using two datasets and SmartBugs, finding limited agreement on annotated vulnerabilities and widespread vulnerability flags in the larger contract dataset.
Problem
Comparing and reproducing automated analysis tools for Ethereum smart contracts is difficult, despite the importance of detecting vulnerabilities in secure contract development.
Method
The paper evaluates 9 automated analysis tools with SmartBugs on 69 annotated vulnerable contracts and 47,518 Ethereum contracts with Solidity source code from Etherscan.
Results
Only 42% of vulnerabilities in the annotated dataset were detected together, while 97% of contracts in the larger dataset were identified as vulnerable.
Takeaways & Limitations
The results identify open challenges for improving automated analysis tools and show limited consensus, with four or more tools jointly finding vulnerabilities in only Arithmetic and Reentrancy categories.
Takeaways & Limitations
The contract set may not accurately represent vulnerabilities arising during development, creating a potential threat to external validity.
Abstract
from arXiv · showhide
Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools.
1 INTRODUCTION
Automated smart-contract analysis is difficult to compare and reproduce because prior studies often use unavailable datasets. This paper addresses that gap with shared datasets, SmartBugs, and a large evaluation of nine tools.
- Motivation: 34,200 of nearly one million contracts were flagged as vulnerable in one preliminary study, while Oyente flagged around 46% of 19,366 contracts.These findings, together with major attacks such as TheDAO and the Parity wallet bug, illustrate the security challenges of Ethereum smart contracts.
- Research gap: Prior automated-analysis research is difficult to compare and reproduce because the datasets used by published tools are generally unavailable.Researchers seeking comparisons may need to contact alternative tool authors for dataset access.
- Contributions: The study provides 69 manually annotated vulnerable contracts and 47,518 Ethereum contracts with Solidity source code available on Etherscan.The annotated dataset supports precision evaluation, while the larger dataset covers available source-code contracts from the blockchain.
- Contributions: SmartBugs is an extendable execution framework that runs nine configured analysis tools in the same environment.The framework is intended to simplify research and facilitate comparison between tools.
- Main findings: 42% of annotated vulnerabilities were detected together by the tools, Mythril detected 27%, and 97% of contracts in the largest dataset were tagged as vulnerable.The results suggest considerable false positives and show that Bad Randomness and Short Addresses were not detected by the current state of the art.
2 STUDY DESIGN
The study reports the state of currently available automated smart-contract analysis tools and uses SmartBugs to support reproducible comparisons. Its design centers on a new extendable execution framework.
- Study purpose: The study reports the current state of automated analysis tools for smart contracts as blockchain-based solutions receive increasing attention.The stated goal is to identify vulnerabilities and bad practices in smart contracts.
- Study purpose: SmartBugs provides an extendable execution framework for reproducibility and comparison between analysis tools.The framework is used to perform the study's evaluation in a common environment.
- Study design: The study design includes research questions, systematic tool and dataset selection, the execution framework, and data collection and analysis methodology.These components define the empirical evaluation presented in the paper.
2.1 Research Questions
The study asks how effectively automated tools detect vulnerabilities, how many vulnerabilities appear in Ethereum contracts, and how long the tools require to analyze them.
- RQ1. Effectiveness: RQ1 evaluates the effectiveness and precision of state-of-the-art tools on known faulty Solidity smart contracts.It focuses on detecting vulnerabilities in contracts whose faults are already known.
- RQ2. Production: RQ2 investigates vulnerabilities detected in contracts pulled from the Ethereum blockchain.The analysis considers popular vulnerabilities, their evolution over time, and consensus among tool combinations.
- RQ3. Performance: RQ3 compares how long the tools require to analyze smart contracts.The goal is to identify which tool is most efficient.
2.2 Subject Tools
The study began with 35 candidate tools and selected nine using availability, compatible-input, and source-only criteria. The included tools use diverse static, symbolic, dynamic, and hybrid analyses.
- Candidate tools: 35 automated analysis tools were identified through a prior survey, academic literature, and internet searches.The resulting candidate set is listed in Table 1.
- Selection criteria: The selection criteria require tools to be publicly available with a CLI, accept Solidity contracts, and require only source code.These requirements support scalable analyses while excluding bytecode-only tools and tools requiring tests or assertions.
- Selection criteria: Nine tools were included after applying the study's three inclusion criteria.The criteria required public CLI availability, Solidity input, and source code as the only required input.
- Included tools: HoneyBadger uses symbolic execution and heuristics to identify honeypots, while Maian detects contracts that can be self-destructed, drained, or accept Ether without payout functionality.Maian also uses dynamic analysis in a private blockchain to reduce false positives.
- Included tools: Manticore uses symbolic execution for EVM bytecode paths leading to reentrancy or reachable self-destruct operations, while Mythril combines concolic, taint, and control-flow analyses.These tools analyze EVM bytecode rather than Solidity source directly.
- Included tools: Oyente uses symbolic execution on EVM bytecode, Securify statically analyzes bytecode with Datalog-based patterns, and Slither applies dataflow and taint tracking to SlithIR.Smartcheck instead performs lexical and syntactical analysis on Solidity source code; Osiris extends Oyente for integer bugs.
2.3 Datasets of Smart Contracts
The paper introduces two Solidity smart-contract datasets with distinct purposes: a curated set of known vulnerabilities for precision evaluation and a large blockchain-derived set for studying potential vulnerabilities and tool performance.
- sbcurated: 69 vulnerable contracts in sbcurated are labelled with vulnerability locations and categories for evaluating analysis-tool precision.The contracts come from deployed contracts and purpose-built vulnerability examples.
- sbwild: 47,518 contracts in sbwild were extracted from the Ethereum blockchain after collecting Etherscan source code and removing duplicate Solidity files.Duplicates were identified using MD5 checksums after removing spaces and tabulations.
- sbwild: sbwild provides an indication of potential-vulnerability frequency and supports performance comparisons, but its contracts' vulnerability set is unknown.The dataset targets a representative picture of practice and potential vulnerabilities in production environments.
- sbcurated: sbcurated draws contracts from GitHub repositories, analytical blog posts, and the Ethereum network.80% of the contracts were collected from GitHub repositories.
- sbwild: 2,263,096 contract addresses yielded 972,975 Solidity files and 47,518 unique contracts after filtering; 47 requested contracts were inaccessible.The resulting collection contains 9,693,457 lines of code.
2.4 The Execution Framework: SmartBugs
SmartBugs is an execution framework designed to simplify integrating, running, and comparing smart-contract analysis tools across datasets. It uses plugins, parallel execution, and normalized outputs to support this workflow.
- Framework features: SmartBugs uses Docker-based plugins to integrate analysis tools.Each plugin specifies the Docker image, tool name, command line, description, and output location.
- Framework features: SmartBugs supports 9 analysis tools and can execute them in parallel to speed analysis.Parallel execution is listed as a core framework feature.
- Outputs: SmartBugs normalizes tool outputs and produces parsable result.json files alongside execution logs.Its command-line interface also supports rendering processed outputs in readable tables.
2.5 Data Collection and Analysis
The study used SmartBugs to execute nine tools on two datasets, applying a 30-minute per-analysis budget and collecting outputs for further analysis. The experiment covered 428,337 analyses and required approximately 564 days and 3 hours of combined execution.
- Execution setup: 9 tools were executed on both datasets, with their outputs collected for subsequent analysis.The setup and execution are described as the basis for answering the research questions.
- Execution setup: 30 minutes was set as the time budget for each tool-contract analysis.The budget exceeded the average execution time of one minute and 44 seconds on sbcurated; timed-out executions yielded partial results.
- Experiment scale: 428,337 analyses required approximately 564 days and 3 hours of combined execution.The executions covered 47,518 contracts and represented more than a year of continuous execution.
- Infrastructure: The experiment used servers from Scaleway and Google Cloud and cost 1513.45 e in total.Google Cloud accounted for 1038.46 e, while provider choice was also shaped by budget restrictions.
3 RESULTS
The evaluation finds uneven vulnerability-detection accuracy across tools and substantial disagreement on production contracts. Mythril leads the curated benchmark, while broad detections on the larger dataset are noisy and rarely corroborated by multiple tools.
- Precision of the Analysis Tools (RQ1): Mythril detected 27% of all vulnerabilities, versus 12% for the average tool, and targeted 5 of 9 vulnerability categories.Mythril, Manticore, Slither, and Smartcheck each detected the largest number of different categories; Slither uniquely identified 8 vulnerabilities.
- Precision of the Analysis Tools (RQ1): Mythril and Slither detected 42 unique vulnerabilities, or 37%, providing the best balance between performance and execution cost.The second-best combination, Mythril and Oyente, detected 33 vulnerabilities, or 29%.
- Precision of the Analysis Tools (RQ1): 42% of all vulnerabilities were detected when the 9 tools were combined.The tools underperformed particularly on Access Control, Denial of service, and Front running; Bad Randomness and Short Addresses were undetectable by design.
- Vulnerabilities in Production Smart Contracts (RQ2): 93% of the 47,518 analyzed contracts had at least one vulnerability detected by one of the 9 tools, suggesting many false positives.Oyente alone identified 73% of contracts as vulnerable, mostly because of Arithmetic findings.
- Vulnerabilities in Production Smart Contracts (RQ2): Most detected vulnerabilities were identified by only one tool; consensus of four or more tools reached 937 Arithmetic and 133 Reentrancy contracts.Arithmetic vulnerabilities had the highest multi-tool agreement, with 71.25% found by more than one tool.
- Execution Costs: The slowest tools did not have better accuracy: Mythril averaged 1m24s per contract at 27% accuracy, while Manticore averaged 24 minutes at 11%.Bytecode-based tools incur compilation overhead, and Manticore analyzes only one internal contract at a time.
4 DISCUSSION
The discussion identifies persistent challenges in analysis quality, vulnerability coverage, development integration, and taxonomy, while noting threats to validity in the evaluation design and datasets.
- The study identifies four challenges: improving analysis quality, extending vulnerability coverage, integrating tools into development, and extending the taxonomy.
- Quality: Near-zero false positives and false negatives remain an unmet quality goal, limiting progress toward real-life adoption.
- Scope: Combining complementary techniques, including static and dynamic analysis, is proposed as one route to finding more real vulnerabilities.
- Development process: Integrating analyzers with bug detectors, dynamic techniques, linters, IDEs, and interactive reports could ease developers’ interaction with the tools.
- Taxonomy: The current DASP10 taxonomy may not comprehensively cover vulnerabilities affecting smart contracts deployed on Ethereum.
- Threats to validity: The evaluation faces validity threats from possible framework bugs, contract-selection mismatch, missed or unreproduced tools, manual category labels, and a 30-minute timeout.
- Threats to validity: The large sbwild dataset provides a sense of tool-generated noise, but makes false-positive discussion difficult; consensus across tools is used as a proxy for true positives.
5 RELATED WORK
Related work includes claims of a first systematic comparison of recent smart-contract analysis techniques, alongside benchmarks and execution frameworks intended to support reproducibility and comparison.
- Empirical studies: The authors describe their study as the first systematic comparison of recently proposed smart-contract analysis techniques to understand their real capabilities.
- Datasets and Repositories: Existing smart-contract repositories are mainly contract collections rather than benchmarks designed for reproducibility and tool comparison.
- Empirical studies: Prior empirical work defined 20 code smells across security, architecture, and usability issues and manually labeled smart-contract datasets.
- Execution Frameworks: Unlike SmartBugs, Solhydra does not flexibly support adding new analysis tools and had not been updated in more than a year.
6 CONCLUSION
The paper empirically evaluates nine automated analysis tools on annotated vulnerable contracts and Ethereum contracts, finding limited detection agreement and widespread vulnerability tagging. Its datasets and execution framework support reproducible comparisons while highlighting open challenges for improving tool quality.
- 6 CONCLUSION: 42% of the vulnerabilities in the annotated dataset were detected together by the tools.The tools detected 48/115 vulnerabilities together.
- 6 CONCLUSION: Mythril had the highest accuracy, detecting 31/115 (27%) of the vulnerabilities.This result concerns the annotated vulnerable-contract dataset.
- 6 CONCLUSION: 97% of contracts in the Ethereum dataset were identified as vulnerable, suggesting a considerable number of false positives.Oyente detected vulnerabilities in 73% of contracts, mostly due to Arithmetic vulnerabilities (72%).
- 6 CONCLUSION: The execution framework and two datasets are presented as assets for reproducible smart-contract analysis research.The framework executes the tools in the same execution environment and the datasets enable comparative evaluation.