Source-linked AI summary
BERT4ETH: A Pre-trained Transformer for Ethereum Fraud Detection
Sihao Hu, Zhen Zhang, Bingqiao Luo, Shengliang Lu, Bingsheng He, Ling Liu
TL;DR
Graph-based Ethereum fraud detection approaches may not fit repetitive, skewed, heterogeneous transactions, while fraud poses risks to users. BERT4ETH uses a pre-trained Transformer encoder with strategies tailored to those transaction characteristics, and reports state-of-the-art improvements on phishing detection and de-anonymization.
Problem
Graph-based Ethereum fraud detection methods may be ill-suited to repetitive, skew-distributed, heterogeneous transactions, while fraud threatens susceptible users.
Source: §1 · p. 1Method
BERT4ETH is a pre-trained Transformer account-representation encoder using repetitiveness reduction, skew alleviation, and heterogeneity modeling for Ethereum fraud tasks.
Source: §1 · pp. 1–2Results
BERT4ETH improves state-of-the-art F1 by 21.61 absolute percentage (AP) for phishing detection and Hit Ratio@1 by 13.54 and 21.57 AP on ENS and Tornado (0.1ETH), respectively.
Source: §1 · p. 2Takeaways & Limitations
The reported improvements on phishing detection and de-anonymization suggest BERT4ETH is well suited for practical applications.
Source: §6 · p. 9
Abstract
from arXiv · showhide
As various forms of fraud proliferate on Ethereum, it is imperative to safeguard against these malicious activities to protect susceptible users from being victimized. While current studies solely rely on graph-based fraud detection approaches, it is argued that they may not be well-suited for dealing with highly repetitive, skew-distributed and heterogeneous Ethereum transactions. To address these challenges, we propose BERT4ETH, a universal pre-trained Transformer encoder that serves as an account representation extractor for detecting various fraud behaviors on Ethereum. BERT4ETH features the superior modeling capability of Transformer to capture the dynamic sequential patterns inherent in Ethereum transactions, and addresses the challenges of pre-training a BERT model for Ethereum with three practical and effective strategies, namely repetitiveness reduction, skew alleviation and heterogeneity modeling. Our empirical evaluation demonstrates that BERT4ETH outperforms state-of-the-art methods with significant enhancements in terms of the phishing account detection and de-anonymization tasks. The code for BERT4ETH is available at: https://github.com/git-disl/BERT4ETH.
1 INTRODUCTION
Ethereum fraud detection faces challenges from graph methods’ difficulty retaining transaction sequence information and handling skewed, heterogeneous activity. BERT4ETH addresses these challenges with a pre-trained Transformer and reports improved results on phishing detection and de-anonymization.
1 INTRODUCTION
- Graph-based methods can discard transaction order, which is important for characterizing user behavior in de-anonymization, and may face noise or restricted capability as GNN convolution hops increase.The concern arises in highly skew-distributed Ethereum data, where accounts often connect to popular accounts.Source: §1 · p. 1
- BERT4ETH is a pre-trained Transformer account-representation model intended to support various Ethereum fraud-detection tasks.Its Masked Address Prediction task masks addresses in transaction sequences and trains the model to predict them.Source: §1 · p. 1
- Ethereum’s repetitive, skewed, and heterogeneous transactions complicate pre-training by enabling label leakage, reducing representation distinctiveness, and mixing diverse interaction types.A small number of popular addresses occur throughout many transaction sequences.Source: §1 · pp. 1–2
- BERT4ETH combines repetitiveness reduction, skew alleviation, and heterogeneity modeling to address Ethereum-specific pre-training challenges.Its strategies include high masking or dropout, frequency-aware negative sampling, separate in/out sub-sequences, and an ERC-20 transfer log encoder.Source: §1 · p. 2
- BERT4ETH improves state-of-the-art F1 by 21.61 absolute percentage (AP) for phishing detection and Hit Ratio@1 by 13.54 and 21.57 AP on ENS and Tornado (0.1ETH), respectively.The reported experiments cover phishing account detection and de-anonymization.Source: §1 · p. 2
- The paper presents BERT4ETH as a universal pre-trained Transformer and reports improved state-of-the-art performance across two fraud-detection tasks.It also makes the code and dataset available as a side contribution.Source: §1 · p. 2
2 RELATED WORK AND BACKGROUND
The background reviews graph-based Ethereum account representations and Transformer pre-training, then establishes terminology for accounts, transactions, and tokens used in the paper.
2.1 Ethereum Representation Learning
- Prior Ethereum representation learning includes DeepWalk-based methods for phishing detection and de-anonymization, alongside GNN-based methods for account classification and phishing detection.Trans2Vec incorporates transaction temporal and amount information; GNN examples include HGATE and TTAGNN.Source: §2.1 · p. 2
2.2 Transformer & BERT
- Transformer self-attention models relationships between tokens, while BERT pre-trains a bidirectional Transformer encoder using Masked Language Modeling and Next Sentence Prediction.The paper draws on this pre-training paradigm as a basis for Ethereum fraud detection.Source: §2.2 · p. 2
2.3 Terminology
- The terminology distinguishes user-controlled EOAs from contract accounts, defines the study’s default transaction as an EOA-initiated external transaction, and notes most tokens use ERC-20.Contract accounts can initiate internal transactions, while tokens can serve functions including currency, access, voting, identity, and utility.Source: §2.3 · p. 2, §4.1.2 · p. 3
3 MOTIVATION
Ethereum transaction characteristics complicate account representation learning: repetitive interactions can leak labels during pre-training, skew can blur account distinctiveness, and heterogeneous activity makes meaningful patterns harder to identify.
3 MOTIVATION
- 48.4% of transactions share a receiver with the same sender’s prior transaction, creating repetitive sequences that can leak masked-address labels during pre-training.The paper calls this issue the label leakage problem because unmasked repetitive addresses may reveal masked ones.Source: §3 · p. 3
- A power-law account-frequency distribution lets a small number of popular accounts appear in many transactions, potentially bringing unrelated accounts close in the latent space.The paper gives Uniswap interactions as an example of this loss of representation distinctiveness.Source: §3 · p. 3
- Different account types, transaction types, and contract-call functions make meaningful patterns harder to discern, while downstream tasks may depend on different information.The paper therefore seeks to preserve transaction heterogeneity during pre-training.Source: §3 · p. 3
4 BERT4ETH
BERT4ETH is a Transformer-based account representation extractor designed for Ethereum transaction histories. Its sequence construction, pre-training strategies, and heterogeneity-focused features address challenges in repetitive, skew-distributed, and varied transactions.
4.1 Transaction Sequence
- 48.0% to 14.3%: deduplication aggregates consecutive matching transactions within 72 hours while preserving their order.Failed transactions are removed, and aggregated transactions retain summed amounts, transaction counts, and the first timestamp.Source: §4.1 · p. 3
4.2 Model Architecture
- Seven transaction features are embedded and summed before a Transformer encoder contextualizes the sequence bidirectionally.Features include address, account type, in/out type, amount, count, timestamp, and position; the final transaction representations encode their own and bidirectional context information.Source: §4.2.1 · p. 3, §4.2.2 · p. 4
4 BERT4ETH
- BERT4ETH captures three-hop neighborhood information through masked-address pre-training and transaction-sequence encoding.Address embeddings capture up to two-hop information during pre-training; the encoder then uses the sequence to produce account representations.Source: §4.3.1 · p. 4
4.3 Pre-training
- 80% masking yields the best reported phishing-detection F1; performance decreases above 80% as unmasked information becomes too limited.Increasing the masking ratio from 10% to 90% raises F1 from 0.1350 to 0.3245, a gap of up to 18.95 AP; 80% dropout is an alternative, with no cumulative gain from combining higher dropout and masking.Source: §4.3.2 · p. 5
- 0.5044 F1: frequency-aware negative sampling with intra-batch sharing improves phishing detection, with a 5,000 negative-to-positive ratio selected by default.The reported F1 gap reaches 9.94 AP as frequent negative sampling increases; for Zipfan sampling, intra-batch sharing raises F1 to 0.5044, with a gap up to 8.05 AP.Source: §4.3.3 · p. 5, Table 1 · p. 5
4.4 Advanced Features
- In/out separation confines self-attention to each transaction subsequence; phishing EOAs have an in-to-out ratio of 1.250 versus 0.385 for normal accounts, while ERC-20 logs capture transfers between EOAs.The ERC-20 encoder mean-pools attached EOA address embeddings and gates their integration with the contract-account embedding. The gate prevents noise when token actions do not guarantee an initiator-receiver relationship.Source: §4.4 · pp. 5–6
4.5 Apply to Downstream Tasks
- The downstream account representation uses the self-transaction vector, concatenates direction-specific vectors when applicable, and mean-pools representations for split sequences.The self-transaction representation is selected because it encodes global information.Source: §4.5 · p. 6
5 EXPERIMENTS
Experiments evaluate phishing-account detection and de-anonymization against graph-based methods, then examine BERT4ETH’s ablations, multi-hop representations, and hot-to-cold queries. BERT4ETH shows substantial gains across the evaluated tasks, with effects that vary by task and setting.
5.1 Task Description
- Phishing detection is binary classification of EOAs measured by precision, recall, and F1; de-anonymization retrieves a paired EOA among nearest neighbors and uses HR@k.The tasks use Euclidean distance for de-anonymization because it yielded slightly better results than cosine similarity.Source: §5.1 · p. 6
5.2 Experimental Setup
- The experiments use Etherscan-labeled phishing accounts, ENS and Tornado ground-truth pairs, and transaction histories from one million sampled EOAs spanning 2017–2022.The comparison includes DeepWalk-based and GNN-based baselines as well as BERT4ETH variants; phishing experiments use fixed training and fine-tuning.Source: §5.2.1 · p. 6, §5.2.2 · p. 6, §5.2.3 · pp. 6–7
5.3 Performance Comparison
- For phishing detection with fixed training, BERT4ETH’s F1 exceeds GNNs by up to 21.61 AP and original BERT by 36.94 AP; BERT4ETH† and BERT4ETH§ add 3.44 and 1.68 AP, respectively.Figure 5 shows BERT4ETH’s phishing-account representations as denser and more separable.Source: §5.3.1 · p. 7, Table 3 · p. 7, Figure 5 · p. 7
- BERT4ETH† achieves the best fine-tuned phishing-detection performance, while removing pre-training substantially reduces performance.The fine-tuning comparison omits graph-based methods because they perform worse and some cannot be fine-tuned.Source: §5.3.1 · p. 7, Table 4 · p. 7
- BERT4ETH significantly outperforms competitors on de-anonymization, with 16.32% exact matches on ENS and 40.2% and 45.0% on Tornado 0.1 ETH and 1 ETH mixers, respectively.On ENS, its HR@1 improvement over the strongest baseline is 9.7 AP; in/out separation further improves on the basic model.Source: §5.3.2 · pp. 7–8, Table 5 · p. 8, Table 6 · p. 8
5.4 Ablation Study
- For phishing account detection with fixed training, switching masking to BERT’s original 15% setting causes the largest ablation performance decrease; in de-anonymization, removing de-duplication and 80% masking slightly improves results when k≤5.Source: §5.4 · p. 8
5.5 Multi-hop Modeling
- MAP-pretrained address embeddings remain effective on both tasks, though less so than BERT4ETH representations, and their larger phishing-detection performance drop suggests multi-hop information matters more for that task.Combining MAP’s implicit two-hop information with the Transformer’s account-sequence representation captures up to three-hop neighborhood information.Source: §5.5 · pp. 8–9, Figure 6 · p. 8
5.6 Case Study
- For hot-to-cold queries, BERT4ETH ranks the hot account 8th when querying from the cold account and 20th in the reverse direction, versus Diff2Vec’s 8th and 47,641st ranks.The authors attribute BERT4ETH’s effectiveness to preserving first-order transaction sequence information and emphasizing important information through self-attention.Source: §5.6 · p. 9, Table 11 · p. 9
6 CONCLUSION
The conclusion presents BERT4ETH as a pre-trained Transformer for Ethereum fraud detection, combining three strategies to address Ethereum-specific pre-training challenges. Its gains on phishing detection and de-anonymization support its suitability for practical applications.
6 CONCLUSION
- BERT4ETH is presented as a universal pre-trained Transformer for Ethereum fraud detection, with Transformer modeling and three strategies addressing pre-training challenges.The strategies are repetitiveness reduction, skew alleviation, and heterogeneity modeling.Source: §6 · p. 9
- The authors report substantial improvements on phishing-account detection and de-anonymization, which they say suggest BERT4ETH is well suited for practical applications.Source: §6 · p. 9
- The three strategies are described as operating cohesively and harmoniously in the model.Source: §6 · p. 9