Source-linked AI summary
SmartBugs: A Framework to Analyze Solidity Smart Contracts
João F. Ferreira, Pedro Cruz, Thomas Durieux, Rui Abreu
TL;DR
Automated smart-contract analysis research is difficult to compare and reproduce because datasets are often unavailable. SmartBugs provides an extensible framework for executing Solidity-analysis tools with reproducible infrastructure and datasets. Its SmartCheck Extended use case increased detected annotated vulnerabilities from 11% to 24%.
Problem
Automated analysis research on Solidity smart contracts is difficult to compare and reproduce because datasets used by tools are often unavailable.
Method
SmartBugs is an extensible execution framework combining analysis tools, Docker-based environments, datasets, and interfaces for Solidity contracts.
Results
24% of vulnerabilities annotated in sbcurated were detected by SmartCheck Extended, compared with 11% for SmartCheck.
Takeaways & Limitations
SmartBugs facilitates reproducible empirical evaluations and comparison of analysis tools within the supported Solidity-contract setting.
Takeaways & Limitations
Future work includes adding tools, expanding datasets, improving documentation, and conducting new empirical studies.
Abstract
from arXiv · showhide
Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present SmartBugs, an extensible and easy-to-use execution framework that simplifies the execution of analysis tools on smart contracts written in Solidity, the primary language used in Ethereum. SmartBugs is currently distributed with support for 10 tools and two datasets of Solidity contracts. The first dataset can be used to evaluate the precision of analysis tools, as it contains 143 annotated vulnerable contracts with 208 tagged vulnerabilities. The second dataset contains 47,518 unique contracts collected through Etherscan. We discuss how SmartBugs supported the largest experimental setup to date both in the number of tools and in execution time. Moreover, we show how it enables easy integration and comparison of analysis tools by presenting a new extension to the tool SmartCheck that improves substantially the detection of vulnerabilities related to the DASP10 categories Bad Randomness, Time Manipulation, and Access Control (identified vulnerabilities increased from 11% to 24%).
1 INTRODUCTION
SmartBugs addresses the difficulty of comparing and reproducing automated Solidity smart-contract analysis research by providing an extensible execution framework. It also demonstrates large-scale evaluation and improved vulnerability detection through a SmartCheck extension.
- Motivation: 34,200 of nearly one million Ethereum smart contracts were flagged as vulnerable in a preliminary analysis.The paper uses major attacks and financial losses to illustrate the security challenge.
- Motivation: Existing automated-analysis research is difficult to compare and reproduce because datasets are often unavailable.Researchers may need to contact alternative-tool authors and request access to their datasets.
- Contribution: SmartBugs is an extensible, easy-to-use framework for executing analysis tools on Solidity contracts and facilitating reproducibility.The paper describes its architecture, tools, datasets, and methods for adding tools and filtering datasets.
- Contribution: The framework supports large experimental setups and tool comparison, including a SmartCheck extension whose identified vulnerabilities increased from 11% to 24%.The extension targets Bad Randomness, Time Manipulation, and Access Control vulnerabilities.
- Contribution: SmartBugs is open-source and publicly available online.
2 SMARTBUGS
SmartBugs combines execution infrastructure, tool configurations, containerized tools, contract datasets, and interfaces for running and comparing Solidity analyses. It provides curated and wild contract datasets, version-aware execution, normalized outputs, and command-line and web interfaces.
- Architecture: SmartBugs consists of a command-line interface, tool configurations, Docker images, contract datasets, a SmartBugs Runner, and a web-based interface.The Runner brings the other components together to execute analysis tools.
- Available Tools and Datasets: 10 analysis tools and two Solidity-contract datasets are distributed with SmartBugs.The tools include HoneyBadger, Maian, Manticore, Mythril, Osiris, Oyente, Securify, Slither, SmartCheck, and Solhint.
- Available Tools and Datasets: sbcurated contains 143 annotated vulnerable contracts with 208 tagged vulnerabilities, while sbwild contains 47,518 unique contracts collected through Etherscan.sbcurated supports precision evaluation, and both datasets are publicly available.
- Tool Integration: SmartBugs uses Docker-based execution and configurable tool versions to improve reproducibility and accommodate Solidity compiler incompatibilities.Different tool versions can be selected for contracts above or below Solidity v0.5.0.
- Tool Integration: Adding a tool generally requires a Docker image and a YAML configuration specifying the image, tool, command, and optional output location.A parser can additionally normalize detected-vulnerability output.
- Interfaces: The command-line interface runs tools on files or named datasets, supports skipping existing results and parallel processes, and produces normalized JSON plus raw outputs.The web interface also supports direct contract entry, file upload, and analysis on predefined datasets.
3 USE CASES
SmartBugs supports both large empirical evaluations and practical tool development. Its use in comparing SmartCheck with SmartCheck Extended shows how added rules can broaden detection across several vulnerability categories.
- 3.1 Supporting Empirical Evaluations: SmartBugs supported an empirical evaluation of 10 state-of-the-art automated analysis tools and helped make the work completely reproducible.The command-line interface and options for skipping existing results and selecting processes were particularly helpful.
- 3.2 Supporting Developers of Analysis Tools: All evaluated tools failed to detect Bad Randomness, motivating an extension of SmartCheck.The extension also targets Time Manipulation and Access Control vulnerabilities.
- 3.2 Supporting Developers of Analysis Tools: SmartCheck Extended adds XPath-based rules for environment-variable use, broader time expressions, and unprotected functions or ownership-related operations.The rules flag uses of variables such as block.timestamp and detect missing standard protection mechanisms.
- 3.2.1 Results: 15 more issues were detected by SmartCheck Extended, more than doubling SmartCheck’s detection capability on the compared contracts.
- 3.2.1 Results: 24% of vulnerabilities annotated in sbcurated were detected by SmartCheck Extended, compared with 11% for SmartCheck.The comparison uses the 69 contracts from the empirical study to provide a fair comparison.
4 CONCLUSION
SmartBugs is presented as a framework for reproducible smart-contract analysis and as a platform for integrating and comparing tools. The conclusion points to continued expansion of its tools, datasets, documentation, and empirical studies.
- SmartBugs simplifies executing Solidity analysis tools while supporting reproducible research and comparison.
- Its demonstrated use extends SmartCheck to improve detection of Bad Randomness, Time Manipulation, and Access Control vulnerabilities.
- Future work includes adding tools, expanding datasets, improving documentation, and conducting new empirical studies.