PaperlayerView exact revision on arXiv

Source-linked AI summary

DEFECTCHECKER: Automated Smart Contract Defect Detection by Analyzing EVM Bytecode

Jiachi Chen, Xin Xia, David Lo, John Grundy, Xiapu Luo, Ting Chen

arXiv:2009.02663v2cs.SE

TL;DR

Because smart contracts cannot be modified after deployment, detecting defects before deployment matters; prior work lacked a suitable detection tool. DefectChecker uses symbolic execution on bytecode to detect eight defects, and outperformed three tools on an open-source dataset while finding defects in deployed Ethereum contracts.

  • Problem

    Prior work defined contract defects but lacked a suitable tool to detect them, while immutable smart contracts make pre-deployment defect detection important.

    Source: §1 · p. 1

  • Method

    DefectChecker uses symbolic execution to analyze smart-contract bytecode for eight defects with severity levels 1–3.

    Source: §7 · p. 17

  • Results

    Against Oyente, Mythril, and Securify, DefectChecker achieved the highest F-score (88.8% in the whole dataset) and least time (0.15s per contract); it found 15.89% of Ethereum contracts contained at least one identified defect.

    Source: §1 · p. 2, §7 · p. 17

  • Takeaways & Limitations

    Applying DefectChecker to Ethereum bytecode identified contracts with defects, including some real-world attacks.

    Source: §1 · p. 2

    DefectChecker targets Solidity version 0.4.0+; adoption of higher versions may make the tool unable to work.

    Source: §4.6 · p. 13

Abstract

from arXiv · show

Smart contracts are Turing-complete programs running on the blockchain. They are immutable and cannot be modified, even when bugs are detected. Therefore, ensuring smart contracts are bug-free and well-designed before deploying them to the blockchain is extremely important. A contract defect is an error, flaw or fault in a smart contract that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Detecting and removing contract defects can avoid potential bugs and make programs more robust. Our previous work defined 20 contract defects for smart contracts and divided them into five impact levels. According to our classification, contract defects with seriousness level between 1-3 can lead to unwanted behaviors, e.g., a contract being controlled by attackers. In this paper, we propose DefectChecker, a symbolic execution-based approach and tool to detect eight contract defects that can cause unwanted behaviors of smart contracts on the Ethereum blockchain platform. DefectChecker can detect contract defects from smart contracts bytecode. We compare DefectChecker with key previous works, including Oyente, Mythril and Securify by using an open-source dataset. Our experimental results show that DefectChecker performs much better than these tools in terms of both speed and accuracy. We also applied DefectChecker to 165,621 distinct smart contracts on the Ethereum platform. We found that 25,815 of these smart contracts contain at least one of the contract defects that belongs to impact level 1-3, including some real-world attacks.

1 INTRODUCTION

Smart contracts are immutable, publicly inspectable targets that may hold valuable assets, so defects can have serious consequences. DefectChecker analyzes bytecode to detect eight previously defined high-impact defects, and its evaluation reports strong benchmark results and defects in Ethereum contracts.

1 INTRODUCTION

  • Smart contracts cannot be modified after deployment, may hold valuable Ethers, and run in a permissionless network where attackers can inspect transactions and bytecode.The paper therefore emphasizes ensuring contracts are well-designed before deployment.Source: §1 · p. 1
  • Earlier work defined 20 contract defects, grouped them into five impact levels, and found that levels 1–3 can lead to unwanted behavior, including attacker control.That work validated whether developers considered the defects harmful through an online survey.Source: §1 · p. 1
  • Because prior work had not provided a suitable detector, DefectChecker targets eight level 1–3 defects by symbolically executing bytecode without requiring source code.It builds a control-flow graph, records stack events, identifies Money Call, Loop Block, and Payable Function features, and applies eight detection rules.Source: §1 · pp. 1–2
  • DefectChecker achieved the highest F-score (88.8% in the whole dataset) and required the least time (0.15s per contract) versus Oyente, Mythril, and Securify.The study also applied it to 165,621 distinct Ethereum bytecode contracts and found that 15.9% contained at least one level 1–3 defect.Source: §1 · p. 2
  • The Ethereum scan identified 25,815 smart contracts containing at least one contract defect and used the results to find real-world attacks, with examples illustrating the importance of defect detection.The paper crawled all Ethereum bytecode (165,621) by the time of writing.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

This section introduces smart-contract execution on Ethereum, the challenges of analyzing EVM bytecode, and the impact-level defects addressed by DefectChecker. It illustrates how unchecked external-call failures and miner-influenced block information can cause unintended behavior, and describes a commit-reveal approach to random-number generation.

2.1 Smart Contracts

  • Smart contracts execute automatically according to deployed code, and their EVM bytecode and transaction inputs are publicly visible on a permissionless network.Ethereum compiles deployed contracts to bytecode; transactions consume gas, and transfers using address.transfer() or address.send() are limited to 2300 gas units.Source: §2.1 · p. 2
  • EVM bytecode uses a global stack, permits different stack sizes at the same program point, and reads jump targets from the stack, so symbolic execution is needed to reconstruct control flow and function calls.Unlike JVM bytecode, EVM uses jumps for intra-contract calls and returns, with block identifiers obtained from bytecode.Source: §2.1 · p. 3

2.2 Contract Defects in Smart Contracts

  • DefectChecker targets eight of 11 defects assigned impact levels 1–3 among 20 previously defined defects, excluding three that are difficult to detect from bytecode.The excluded defects are Unmatched Type Assignment, Hard Code Address, and Misleading Data Location; source-level information may be lost during compilation, and relevant storage or constructor information can be difficult to establish from bytecode.Source: §2.2 · pp. 3–4

2.2.1 Definition of Impact Levels

  • Impact 1 defects can let attackers trigger critical unwanted behavior for profit; Impact 2 defects can be attacker-triggered without profit, while Impact 3 covers specified critical or major behaviors.Impact 3 includes critical behavior not triggerable by attackers (Type A) and major behavior that attackers can trigger without profit (Type B); critical defects can cause crashes, attacker control, or loss of all Ether, while major defects can lose part of the Ether.Source: §2.2.1 · p. 4

2.2.2 Examples of Smart Contract Defects

  • Transaction State Dependency can let an attacker bypass a permission check that relies on tx.origin, while DoS under External Influence can make a function fail perpetually after an attacker-controlled transfer exception.In the DoS example, an attacker contract in a members array triggers an out-of-gas exception under the 2300-gas transfer limit, rolling back state and preventing the address from being removed.Source: §2.2.2 · p. 4
  • Strict Balance Equality lets an attacker disrupt logic by forcibly adding Ether, while reentrancy can repeatedly invoke a withdrawal before the victim resets the caller’s balance.In the reentrancy example, sending Ether invokes the attacker’s fallback function, which calls withdraw() again until the victim’s Ether runs out.Source: §2.2.2 · pp. 4–5
  • Nested Call can exhaust gas when an attacker increases an unbounded loop containing CALL, and a Greedy Contract can permanently lock received Ether if it has no withdrawal method.CALL is costly, and the paper characterizes a greedy contract as one that can receive Ether but has no way to withdraw it.Source: §2.2.2 · p. 5
  • External calls such as address.send() and address.call() return false on failure without throwing, so unchecked return values leave subsequent logic uncertain.The example notes that an Ether transfer can fail even when the contract proceeds to following code.Source: §2.2.2 · pp. 5–6
  • Miners can influence block information used for execution, enabling a roulette contract based on block hash to select the miner as winner.The proposed alternative avoids block-related functions: participants first submit hashes of local random numbers, then reveal the numbers for verification before the contract combines them.Source: §2.2.2 · p. 6
  • A safer random-number approach has users commit hashes, reveal the original numbers for hash verification, and combine the verified numbers.The passage says a random number should not be controllable by a single person and recommends avoiding block-related functions.Source: §2.2.2 · p. 6

3.1 Design Overview

DefectChecker takes contract bytecode through symbolic execution and feature detection, then applies defect-specific rules. Its bytecode focus also addresses cases where contract source code is unavailable.

3.1 Design Overview

  • The architecture has four components: Inputter, CFG Builder, Feature Detector, and Defect Identifier.Users can supply bytecode directly, or Solidity source that is compiled to bytecode.Source: §3.1 · p. 6
  • The tool disassembles bytecode into opcodes, splits them into basic blocks, symbolically executes instructions, builds a control-flow graph, and records stack events.During execution, the Feature Detector identifies Money Call, Loop Block, and Payable Function.Source: §3.1 · p. 6
  • Defect Identifier applies eight rules to the collected graph, stack-event, and feature information, while bytecode analysis supports cases with closed-source contracts.The paper notes that less than 1% of smart contracts have opened their source code and that callees may also keep their source closed.Source: §3.1 · p. 6

3.2 Basic Block Builder

The Basic Block Builder partitions opcodes into straight-line blocks and classifies each block by its exit instruction.

3.2 Basic Block Builder

  • A basic block is a straight-line code sequence with no branches except at its entry and exit.The definition is attributed to.Source: §3.2 · p. 6
  • The block’s last instruction determines its exit type: JUMP and JUMPI indicate unconditional and conditional exits, while STOP, REVERT, or RETURN indicate terminal exits..Source: §3.2 · p. 6
  • Blocks ending in none of those jump or terminal instructions are classified as fall blocks, yielding four block types overall.The types are unconditional, conditional, fall, and terminal.Source: §3.2 · p. 6

3.3 Symbolic Execution

DefectChecker symbolically executes EVM bytecode instruction by instruction to recover control flow and stack states. The example shows a conditional instruction producing two paths, while the analysis treats nonzero conditions as satisfiable without consulting an SMT solver.

3.3 Symbolic Execution

  • EVM jump targets are read from the stack, so DefectChecker executes instructions one at a time to construct the control-flow graph (CFG).Unlike JVM bytecode, EVM jump positions must be calculated during symbolic execution.Source: §3.3 · p. 6
  • Each executed instruction updates symbolic stack states, and the recorded stack events plus jump relations describe the contract’s execution paths.The example follows a conditional jump into either block 2 or block 3, with both paths reaching block 4.Source: §3.3 · p. 7
  • The example’s JUMPI branches on ISZERO(GT(a, num)), sending one path to block 2 and the other to block 3.The example traces the stack and program counter through each instruction to show how these paths are produced.Source: §3.3 · p. 7
  • To accelerate analysis, DefectChecker treats a conditional expression equal to 0 as unsatisfiable and every other expression as satisfiable, without checking with an SMT solver.The paper notes that compilation may eliminate dead code, so it assumes little dead code remains in EVM bytecode.Source: §3.3 · p. 7

3.4 Feature Detector

DefectChecker derives three bytecode-level features for defect detection: Money Call, Loop Block, and Payable Function. It identifies value-transferring calls from stack values, loop blocks through CFG traversal, and payable functions by inspecting their entry blocks.

3.4.1 Money Call

  • Money-Call identifies CALL instructions whose transfer amount is greater than zero; a 2300 gas value distinguishes Gas-Limited-Money-Calls from presumed Gas-Unlimited-Money-Calls.The latter are attributed to address.call().value(), while address.send() and address.transfer() are treated as gas-limited and unable to cause reentrancy.Source: §3.4.1 · p. 8

3.4.2 Loop Block

  • A depth-first search flags a revisited CFG block as a loop start and the other blocks in that cycle as loop bodies.Pruning avoids traversing remaining paths when they reach a block whose path contains no cycles.Source: §3.4.2 · p. 8

3.4.3 Payable Function

  • Payability is detected by inspecting a function’s first block for CALLVALUE and ISZERO behavior, with the first JUMPDEST marking the fallback function’s start.Conditional jumps before the first JUMPDEST identify function starts; a non-payable function throws an exception when sent Ether.Source: §3.4.3 · p. 8

3.5 DefectChecker

DefectChecker combines symbolic stack states and CFG information according to each defect’s detection requirements. Its rules include checks for transaction-origin comparisons and for loop behavior involving a Money Call.

3.5 DefectChecker

  • TSD and UEC require symbolic states, BID requires control-flow information, and the other five defects use both types of information.Money Call, Loop Block, and Payable Function supply features used by the remaining defect rules.Source: §3.5 · p. 8

3.5.1 Transaction State Dependency

  • TSD detection locates ORIGIN instructions and checks whether EQ reads one to compare it with an address value.Ethereum addresses are described as 40-bit values conforming to the EIP55 standard.Source: §3.5.1 · pp. 8–9

3.5.2 DoS Under External Influence

  • DoS Under External Influence is detected when a loop-related conditional block contains a Money Call, checks its return value, and jumps to a terminal block.Source: §3.5.2 · p. 9

3.5.3 Strict Balance Equality

The defect rules use bytecode patterns to identify strict balance checks, reentrancy conditions, and unbounded loops containing value transfers. The latter checks use storage slots and CFG information to assess whether loop size is limited.

3.5.3 Strict Balance Equality

  • Strict Balance Equality is flagged when BALANCE is read by EQ in a conditional expression, indicating a strict contract-balance equality check.The passage states this defect can make part of the code never execute.Source: §3.5.3 · p. 9

3.5.4 Reentrancy

  • Reentrancy detection follows paths containing Gas-Unlimited-Money-Calls and checks whether a conditionally tested SLOAD slot is updated by SSTORE before the call.If that storage value remains unchanged at the call, the call may execute again and cause reentrancy.Source: §3.5.4 · p. 9

3.5.5 Nested Call

  • Nested Call is detected when a loop does not limit its size and contains a Money Call.The analysis uses an array’s stored length and its SLOAD slot to check whether the loop limits its size.Source: §3.5.5 · p. 9

3.5.6 Greedy Contract

The section defines three bytecode-level defect checks: payable functions without money transfer, unchecked external calls, and conditionals dependent on block information. The latter two are identified through instruction patterns, with block-information dependence linked to miner control.

3.5.6 Greedy Contract

  • A Greedy Contract has payable functions but neither a Money CALL nor a SELFDESTRUCT instruction.The paper treats either instruction as a way for a contract to transfer money.Source: §3.5.6 · p. 9

3.5.7 Unchecked External Calls

  • An external call is flagged when its CALL instruction is not read by ISZERO, which the paper uses to identify an unchecked return value.The return value is a boolean; checking it generates an ISZERO instruction.Source: §3.5.7 · p. 9

3.5.8 Block Info Dependency

  • A conditional containing BLOCKHASH, COINBASE, NUMBER, DIFFICULTY, or GASLIMIT indicates Block Info Dependency, which can let miners control the contract.The paper says miners can change some block information, affecting the conditional expression.Source: §3.5.8 · p. 9

4 EVALUATION

The evaluation examines DefectChecker’s defect-level results and comparisons with bytecode-analysis tools on a manually labeled dataset, while discussing limitations in detection and in the dataset’s reliability and coverage.

4.1 Experimental Setup

  • Experiments ran on a six-core Intel i7 with 16 GB memory, using Solidity 0.4.25 and EVM 1.8.14 for compilation and bytecode disassembly.Source: §4.1 · p. 9

4.2 Dataset

  • After obtaining 587 smart contracts from Etherscan, the authors removed eight designed for Solidity versions 0.2.0+ and 0.3.0+, using the remaining 579 as ground truth.Two experienced authors independently labeled the dataset, reconciled disagreements, and achieved an overall Kappa value of 0.71.Source: §4.2 · p. 10

4.3 Evaluation Methods and Metrics

  • Evaluation measurements comprised true and false positives and negatives, precision, recall, and F-measure.Source: §4.3 · p. 10

4.4 Experimental Results and Analysis

  • Transaction State Dependency had 5 detections with no errors; DoS Under External Influence had 13 detections and 7 false positives; Strict Balance Equality had 4 detections and 1 false negative.The DoS false positives arose from loop identification; the Strict Balance Equality false negative involved a defect spanning functions whose calling sequence was unknown at bytecode level.Source: §4.4 · pp. 10–11
  • DefectChecker detected 16 Reentrancy cases with 6 false positives and 2 false negatives, 11 Nested Call cases with 2 and 4 errors, and 6 Greedy Contracts without errors.Reentrancy errors involved complex symbolic gas, transfer, or storage-slot expressions; Nested Call errors involved loop detection or array lengths obtained through external functions.Source: §4.4 · p. 11
  • Unchecked External Call was reported in 23 contracts with 3 false positives and 2 false negatives; Block Info Dependency had 41 detections with no false positives and 1 false negative.The Unchecked External Call false positives involved return values checked in other functions; its false negatives occurred in constructors absent from runtime bytecode.Source: §4.4 · pp. 11–12

4.5 Comparison with state-of-the-art tools

  • Oyente’s F-scores were 3.7%, 68.1%, and 37.3% for Reentrancy, Unchecked External Call, and Block Info Dependency, versus DefectChecker’s 71.4%, 88.9%, and 98.8%.Securify scored 4.9% for Reentrancy and 62.5% for Unchecked External Call; Mythril missed Transaction State Dependency and Strict Balance Equality and produced many false positives for Reentrancy and DoS Under External Influence.Source: §4.5 · p. 12, Table 8 · p. 13
  • DefectChecker achieved the best F-measure among the four tools and was fastest, averaging 0.15s per contract versus Oyente’s 18.48s, Securify’s 21.55s, and Mythril’s 103.55s.Its maximum analysis time was 2.42s, compared with 1096.32s for Oyente, 1203.99s for Securify, and 2480.26s for Mythril.Source: §4.5 · pp. 12–13

4.6 Threats to Validity

  • DefectChecker was evaluated against a prior-work dataset, creating potential familiarity bias that demo tests and large-scale testing on real-world Ethereum bytecode were intended to reduce.During large-scale testing, the authors also randomly selected contracts with available source code to improve detection performance and patterns.Source: §4.6 · p. 13
  • The authors built their own dataset because baseline datasets were unavailable, undisclosed, or unsupported by a complete evaluation dataset.Oyente’s evaluation manually checked only some examples; Securify’s 100-contract dataset was not public, and Mythril’s technical papers lacked an evaluation section.Source: §4.6 · p. 13
  • The manually analyzed dataset may contain false positives and negatives, and DefectChecker’s design for Solidity 0.4.0+ may not work with later versions.The authors double-checked results and updated the dataset when they found mistakes; they note Solidity had nine releases in 2018 and higher-version adoption could limit the tool.Source: §4.6 · p. 13

5 A LARGE SCALE EVALUATION

DefectChecker was run on 165,621 Ethereum runtime bytecodes to measure defect prevalence, examine contract complexity, and study real-world attacks. The evaluation also identifies dataset and detection-tool limitations that qualify its findings.

5.2 Contract Defects on Ethereum

  • Unchecked External Calls affected about 7.5% of contracts, while Block Info Dependency affected about 3.1%; Strict Balance Equality was detected in 390 contracts.Nested Call affected 1,043 contracts (0.6%); Transaction State Dependency and DoS Under External Influence affected about 1.0% and 1.3%, respectively. There were 3,139 greedy contracts and 3,892 contracts with Reentrancy.Source: §5.2 · p. 14
  • 25,815 contracts (15.9%) contained at least one defect; 16 had four kinds, 539 had three, and 3,520 had two.The analysis counts each defect kind in a contract containing multiple kinds.Source: §5.2 · p. 14, Table 12 · p. 14
  • Contracts averaged 21.3 cyclomatic complexity and 2,342.6 instructions; the two measures correlated (0.702, p<0.05) and each positively predicted defect count.The authors conclude either measure can serve as a predictor.Source: §5.2 · pp. 14–15, Figure 3 · p. 15

5.3 Case Study

  • A sendReward() loop grew with investors[] and eventually exceeded the gas limit, leaving the contract’s Ethers locked because the array length could not shrink.The paper also describes a contract holding 2,335.8 Ethers (then valued at $552,720) that had them locked by Nested Call; DoS Under External Influence was another identified defect in that contract.Source: §5.3 · p. 15
  • In the second case, reentrancy let an attacking contract steal about 5 Ethers across three attacks; the victim was attacked by multiple contracts, so total losses were higher.The reported value for the approximately 5 Ethers was about $1,200 at the time of writing.Source: §5.3 · pp. 15–16, Figure 5 · p. 16

5.4 Threats to Validity

  • The analysis excluded about 20,000 contracts because DefectChecker supports only Solidity versions 0.4.0 and higher; the authors say the removed bytecode was from earlier years.They also report false positives and negatives, and note that changing defect prevalence could produce different results. The authors say the removed contracts do not represent current usage and consider their conclusions reasonable despite tool errors.Source: §5.4 · p. 16

6 RELATED WORK

Related tools address overlapping but differently scoped smart-contract security problems, with differing input requirements and analysis methods. DefectChecker targets survey-validated defects and emphasizes bytecode analysis, efficiency, accuracy, and extensibility.

Prior tools

  • Prior tools illustrate varied approaches: MAIAN combines symbolic analysis with concrete validation, Mythril checks SWC Registry rules, and Securify analyzes semantic facts and security patterns.Securify focuses on Stealing Ether and Frozen Funds; the cited evaluation reports it found more security violations than Oyente and Mythril.Source: §6 · p. 17

Related tools and scope

  • DefectChecker analyzes bytecode, unlike Zeus’s source-code input and ContractFuzzer’s ABI requirement; the paper says only 1% of contracts publish source code and ABI.The authors report that MAIAN could not support the Ethereum version used in their study.Source: §6 · pp. 16–17
  • DefectChecker targets defects validated as harmful by developers, whereas Oyente and Securify proposed security defects without that validation; Mythril, Oyente, and Securify also detect defects DefectChecker does not support.The paper frames its comparison with these tools around accuracy and efficiency, not broader defect coverage.Source: §6 · p. 17

DefectChecker’s approach

  • Like Oyente, Mythril, and Securify, DefectChecker uses symbolic execution, but its Stack Event and Feature Detector replace SMT solving, which the authors say reduces runtime and improves accuracy.Its CFG and feature outputs can support new defect rules, making the tool extensible to other issues.Source: §6 · p. 17

7 CONCLUSION AND FUTURE WORK

DefectChecker applies symbolic execution to bytecode to detect eight contract defects, and the paper reports that 15.89% of 165,621 analyzed Ethereum contracts had at least one. The authors identify false positives and negatives as a limitation and describe avenues for improving accuracy while balancing runtime.

Conclusion

  • DefectChecker detected eight defect types in bytecode and was applied to 165,621 Ethereum contracts; about 15.89% contained at least one identified defect.The authors report that its scores exceeded those of Oyente, Mythril, and Securify on their previous work’s dataset.Source: §7 · p. 17

Implications

  • Developers can use DefectChecker to check bytecode even when a callee’s source is unavailable, while researchers can reuse its CFG for other smart-contract research.The paper says this can help developers make contracts more robust and safer.Source: §7 · pp. 17–18

Future work and limitation

  • False positives and negatives remain, including for NC and DuEI; adding an SMT solver may reduce some errors but increases analysis time.The authors suggest checking loop-related conditional expressions, since treating all paths as reachable can produce false positives when a condition is always false.Source: §7 · p. 18

Related papers

Loading 2009.02663v2…