Source-linked AI summary
BlockSci: Design and applications of a blockchain analysis platform
Harry Kalodner, Steven Goldfeder, Alishah Chator, Malte Möser, Arvind Narayanan
TL;DR
Blockchain analysis needs tools that can handle large, diverse datasets and support varied research tasks. BlockSci provides an open-source platform with an in-memory analytical database, integrated analysis capabilities, and multiple interfaces; its applications cover privacy, security, economics, and cryptocurrency measurement.
Problem
Blockchain data is a large research corpus, but existing analysis tools have limitations that hinder analysis across privacy, markets, and non-currency applications.
Method
BlockSci combines an in-memory analytical database with integrated analytic modules, common blockchain interfaces, and Jupyter and C++ programming interfaces.
Results
The platform supports four analyses spanning multisignature confidentiality, Dash privacy, block-space markets, and cryptocurrency velocity.
Takeaways & Limitations
BlockSci provides a shared platform for blockchain analysis across supported chains and research tasks.
Takeaways & Limitations
Evaluating the Dash attack on existing PrivateSend transactions is challenging because ground truth is lacking and remains future work.
Abstract
from arXiv · showhide
Analysis of blockchain data is useful for both scientific research and commercial applications. We present BlockSci, an open-source software platform for blockchain analysis. BlockSci is versatile in its support for different blockchains and analysis tasks. It incorporates an in-memory, analytical (rather than transactional) database, making it several hundred times faster than existing tools. We describe BlockSci's design and present four analyses that illustrate its capabilities. This is a working paper that accompanies the first public release of BlockSci, available at https://github.com/citp/BlockSci. We seek input from the community to further develop the software and explore other potential applications.
1 INTRODUCTION
BlockSci is an open-source platform designed to address blockchain-analysis tools’ performance, capability, and usability limitations. Its architecture combines an in-memory analytical database, broad blockchain support, analytic modules, and flexible interfaces for exploration and high-performance tasks.
- Motivation and contribution: 15x–600x faster than existing tools, BlockSci addresses poor performance while also offering analytic modules and a common interface across blockchains.It supports address clustering, exchange-rate and mempool data, Jupyter notebooks, and C++ for performance-critical tasks.
- Design rationale: Append-only blockchains and static research snapshots make ACID properties unnecessary, motivating BlockSci’s in-memory analytical database.The design also converts hash pointers to actual pointers to improve speed and reduce data size.
- Capabilities and scope: BlockSci supports Bitcoin, Litecoin, Namecoin, and Zcash through a common compact format, while Ethereum is outside its scope.Its analytic library includes tools such as CoinJoin identification and address-linking heuristics.
- Programmer interfaces: Jupyter notebooks support intuitive exploration, while C++ and inline C++ support analyses requiring higher performance.Transaction objects are instantiated only when accessed, and the interface supports straightforward iteration over blocks and transactions.
- Applications: Four applications examine multisignature confidentiality, a cluster-intersection attack on Dash, the block-space market, and cryptocurrency velocity.These analyses illustrate BlockSci’s use across privacy, security, economics, and measurement tasks.
2 DESIGN AND ARCHITECTURE
BlockSci converts blockchain data through a shared parsing pipeline into core blockchain data, which its analysis library loads as an in-memory database for direct or notebook-based queries.
- Data pipeline: Two import routes convert data into the same intermediate parsing format before producing Core Blockchain Data.The parser output can be incrementally updated as new blocks arrive.
- Analysis layer: The analysis library loads Core Blockchain Data as an in-memory database accessible directly or through a Jupyter notebook interface.This separates parsing from interactive analysis while preserving both access modes.
2.1 Recording and importing data
BlockSci imports blockchain and mempool data through flexible routes, normalizes supported chains for parsing, and records transaction waiting times and unconfirmed transactions for analysis.
- Supported blockchains: BlockSci supports Bitcoin, Litecoin, Namecoin, Zcash, and other chains following its basic transaction-graph structure, but not Monero or Ethereum.Namecoin’s new script types are not parsed, and Ethereum and Monero require departures from BlockSci’s supported model.
- Importing: Small altcoins use JSON-RPC importing for versatility, while larger blockchains use a high-performance importer that reads data directly.The importer passes data directly to the parser, which executes in a pipelined fashion.
- Mempool recording: Mempool recording captures transaction broadcasts awaiting inclusion, including waiting times and transactions that never enter a block.Minimal mode records timestamps for confirmed transactions; full mode records all mempool information.
- Timestamp limitation: 16 seconds average lag and 4 seconds standard deviation separate BlockSci’s single-node timestamps from blockchain.info’s timestamps.A uniform correction can remove the average lag, but the variance remains.
2.2 Parser
The parser transforms serialized blockchain data into a compact analysis format using sequential state, linked transaction graphs, fixed encodings, deduplication, and memory-conscious lookup structures.
- Data representation: Linking outputs to spending inputs, replacing hash pointers with IDs, fixed-size encodings, deduplication, and locality optimization reduce graph size and improve linkage.These transformations produce a representation suited to efficient analysis.
- Parser state: Sequential parsing maintains transaction-hash and address-to-ID mappings because inputs reference spent outputs by transaction hash and output index.Address mappings must remain available because any address may be reused by a later output.
- Caching and locality: 89% of inputs spend outputs created within 4000 blocks, while 90% of reused addresses are reused within 4000 blocks.These locality patterns support trading memory consumption against lookup speed.
- Lookup optimization: A LevelDB-backed LRU cache and Bloom filter reduce database queries while preserving correctness for nonexistent-address lookups.The Bloom filter’s negative results are always correct, although false positives can occur.
- Validation and updates: The parser assumes transactions and blocks were validated before serialization, allowing it to omit most script processing.Incremental updates resume from serialized parser state, while reorganizations require reversing prior processing before applying new blocks.
2.3 Core Blockchain Data
BlockSci’s Core Blockchain Data stores the transaction graph in a compact, append-only format designed for efficient sequential analysis. Memory mapping and a locality-oriented layout trade about 19% space overhead for roughly 10x faster sequential iteration.
- The parser outputs Core Blockchain data as the primary dataset for analysis.
- The transaction graph uses a sequential transaction table with variable-length entries and separate offsets for indexing.
- Append-only updates allow the transaction table to remain a linearly growing flat file, memory-mapped for analysis.
- Spatial locality improves caching and keeps block-by-block analyses feasible on machines that cannot load the full graph into memory.
- About 19% space duplication yields roughly 10x faster sequential iteration than a normalized layout.
- Transaction hashes and addresses are kept in separate indexes because many analyses do not require them in memory.
- BlockSci categorizes scripts into five supported types and treats other scripts as nonstandard.
2.4 BlockSci Analysis Library
BlockSci’s analysis library combines memory-mapped, shared data with parallel abstractions and domain-specific tools. Its address-linking workflow produces clusters useful for analysis but remains vulnerable to heuristic errors and cluster collapse.
- Memory mapping lets multiple analysis processes share physical memory without allocating new objects for the object-oriented interface.
- Shared memory and a single-writer design avoid synchronization between analysis instances and support multiple users on one machine.
- Each BlockSci instance presents a fixed snapshot even as the underlying memory-mapped transaction table receives new blocks.
- Mapreduce operations express common analyses and automatically parallelize them across available cores.
- Address-linking heuristics connect addresses controlled by the same entity, and union-find converts these links into address clusters.
- About 145 million clusters were identified, including 13 with over 20,000 addresses and one exceeding 139 million addresses.
- Address linking lacks large-scale ground truth, and spurious heuristic edges can cause cluster collapse, likely explaining the largest supercluster.
- BlockSci supports user-supplied tags for propagation during address linking but does not provide automated large-scale tagging.
2.5 Programmer interface
BlockSci offers a Python/Jupyter interface for exploration while allowing performance-critical work in C++ or inline C++. This design improves usability but retains a substantial Python performance tradeoff.
- Jupyter exposes BlockSci’s C++ library to Python, while standalone Python and C++ programs can also use the analysis library.
- Python queries are significantly slower, so the interface is designed to move bottleneck operations into C++.
- A fully Python implementation of anomalous-fee detection is unacceptable in performance.
- The selector syntax can automatically enable multithreading for transaction filtering.
- Inline C++ selectors can return a subset of transactions for subsequent Python processing.
- The fastest Python-interface route passes C++ code through chain.cpp, and BlockSci reports performance figures for the available syntaxes.
- There were 300 transactions with fees above USD 1,000, including a highest fee of 291 BTC.
2.6 Performance evaluation
BlockSci’s performance evaluation measures runtime, scalability, memory use, and comparisons with prior blockchain-analysis tools. Results show strong speedups for parallel and performance-sensitive workloads, while memory layout and data locality affect resource use.
- Basic run time statistics: 46 seconds is sufficient for a single-threaded anomalous-fee query scanning the transaction data, with slightly over 4x speedup on four physical cores.The workload is embarrassingly parallel, and the observed speedup reaches the test machine’s practical hardware limit.
- Basic run time statistics: A 23-fold slowdown occurs when the transaction-header query accesses transactions in random order rather than benefiting from locality.This indicates that memory-access patterns substantially affect BlockSci’s performance, especially for less locality-friendly analyses.
- Basic run time statistics: The C++ selector interface is essentially as fast as C++ execution, whereas pure Python has unacceptable performance and helper functions remain slower.BlockSci therefore supports exploratory Python while preserving near-native speed for performance-sensitive code paths.
- Comparison with previous tools: 28.3 seconds versus 3.7 minutes shows BlockSci outperforming BTCSpark on the TOAD query using a single test instance versus ten EC2 instances.The comparison uses the reported TOAD benchmark and different execution setups, so it is informative but not perfectly equivalent.
- Comparison with previous tools: 2.0, 3.9, and 6.0 seconds versus 53, 2,300, and 3,700 seconds make BlockSci 27x–600x faster than Neo4j on three analyses.The comparison used BlockSci’s multithreaded mode and Neo4j’s Java API on the same block height.
3 APPLICATIONS
BlockSci supports blockchain science through four applications spanning privacy, confidentiality, and cryptocurrency economics. These analyses reveal privacy weaknesses in multisignatures and Dash mixing, while quantifying economically costly mining practices and revising cryptocurrency-velocity estimates.
- Four applications demonstrate BlockSci’s effectiveness for blockchain science, covering privacy, confidentiality, and cryptocurrency economics.The first two applications concern privacy and confidentiality; the latter two concern cryptocurrency economics.
- 3.1 Multisignatures hurt confidentiality: Multisignatures publicly expose wallet access-control structures and changes, including key counts, signing thresholds, and events that may trigger policy changes.This exposure affects companies and individuals using multisignature wallets for access control.
- 3.1 Multisignatures hurt confidentiality: 22,275,033 additional change addresses were identified by exploiting multisignature privacy leaks, increasing detections by over 25% beyond 88,339,789 addresses found with known heuristics.Over 8 million cases weakened non-multisig users’ anonymity, while over 13 million involved multisig users weakening their own anonymity.
- 3.2 Cluster intersection attack on Dash: Dash PrivateSend’s exact-change requirement and power-of-10 denominations increase inputs, making cluster intersection attacks more effective as input counts grow.Paying 85 Dash requires at least 13 inputs; PrivateSend transactions have a mean of 40.1 inputs and a median of 12.
- 3.2 Cluster intersection attack on Dash: For transactions with 12 or more inputs, the cluster intersection attack is always accurate in the simulated setup.The attack’s success rate rises sharply with the number of inputs, and 12 is the median input count for PrivateSend transactions on the blockchain.
- 3.3 The block space market: Miners’ departures from revenue-maximizing transaction selection can impose substantial costs, including up to USD 90,000 in estimated losses for Antpool.A 60-second block-update interval would reduce transaction-fee revenue by an average of 5% per block under the observed fee distribution.
4 CONCLUSION
Blockchain analysis has strong interest among developers, researchers, and students, creating demand for effective tools. BlockSci is intended as an open-source research and educational platform to meet that need.
- Blockchain analysis has attracted developers, researchers, and students, creating an unmet need for effective analysis tools.
- BlockSci is customized for blockchain data, leveraging its append-only structure and integrating high-performance routines such as address linking.
- BlockSci has already been used at Princeton as a research and educational tool.
- The authors plan to maintain BlockSci as open-source software and hope it will be broadly useful.
A DASH PRIVATESEND ALGORITHM
The PrivateSend wallet simulation selects unspent outputs owned by the wallet and accumulates them toward a desired spending amount. It returns the selected outputs when the target is reached or reports insufficient funds otherwise.
- The algorithm takes a desired PrivateSend spending amount and outputs a set of unspent outputs whose values add up to that amount.
- SelectPSInputs is the procedure used to choose PrivateSend inputs.
- The simulation considers unspent outputs owned by the wallet.
- It iterates through transactions and their outputs while evaluating candidate inputs.
- Selected value is increased by adding output values to the chosen set.
- The procedure returns “Insufficient Funds” when the wallet cannot satisfy the requested amount.
B ADDITIONAL FIGURES
The additional figures show distributions and estimates for transaction structure, cryptocurrency velocity, and mining-pool timing. They cover PrivateSend inputs, litecoin velocity, and apparent transaction-to-block-time gaps.
- Figure 14 shows the distribution of the number of inputs in Dash PrivateSend transactions.
- Figure 15 presents two estimates of litecoin velocity.
- Figure 16 shows the distribution of apparent gaps between the latest transaction in a block and its block time for the six largest mining pools.