Source-linked AI summary
BLOCKBENCH: A Framework for Analyzing Private Blockchains
Tien Tuan Anh Dinh, Ji Wang, Gang Chen, Rui Liu, Beng Chin Ooi, Kian-Lee Tan
TL;DR
Private blockchains need a systematic way to assess their suitability for database-like data processing and compare competing platforms. The paper introduces Blockbench, an extensible framework with layered workloads and performance metrics, and evaluates Ethereum, Parity, and Hyperledger. The results show that current systems remain poorly suited to large-scale data processing and exhibit bottlenecks and trade-offs tied to software-stack design choices.
Problem
Private blockchains target database-supported applications, but existing platforms lack a systematic framework for comparing their data-processing performance and design choices.
Method
Blockbench integrates permissioned blockchain backends through APIs and benchmarks layered workloads across throughput, latency, scalability, fault tolerance, and security dimensions.
Results
The evaluation of Ethereum, Parity, and Hyperledger finds current blockchains poorly suited to large-scale data-processing workloads and reveals bottlenecks and design trade-offs across software layers.
Takeaways & Limitations
Blockbench provides a comparison baseline and helps developers assess application fit and identify performance bottlenecks in private blockchain platforms.
Takeaways & Limitations
The reported scalability result beyond 16 servers reflects an implementation later replaced by another PBFT component, which the authors planned to evaluate separately.
Abstract
from arXiv · showhide
Blockchain technologies are taking the world by storm. Public blockchains, such as Bitcoin and Ethereum, enable secure peer-to-peer applications like crypto-currency or smart contracts. Their security and performance are well studied. This paper concerns recent private blockchain systems designed with stronger security (trust) assumption and performance requirement. These systems target and aim to disrupt applications which have so far been implemented on top of database systems, for example banking, finance applications. Multiple platforms for private blockchains are being actively developed and fine tuned. However, there is a clear lack of a systematic framework with which different systems can be analyzed and compared against each other. Such a framework can be used to assess blockchains' viability as another distributed data processing platform, while helping developers to identify bottlenecks and accordingly improve their platforms. In this paper, we first describe BlockBench, the first evaluation framework for analyzing private blockchains. It serves as a fair means of comparison for different platforms and enables deeper understanding of different system design choices. Any private blockchain can be integrated to BlockBench via simple APIs and benchmarked against workloads that are based on real and synthetic smart contracts. BlockBench measures overall and component-wise performance in terms of throughput, latency, scalability and fault-tolerance. Next, we use BlockBench to conduct comprehensive evaluation of three major private blockchains: Ethereum, Parity and Hyperledger Fabric. The results demonstrate that these systems are still far from displacing current database systems in traditional data processing workloads. Furthermore, there are gaps in performance among the three systems which are attributed to the design choices at different layers of the software stack.
1. INTRODUCTION
Private blockchains target data-processing applications traditionally supported by databases, but their performance and platform trade-offs lack systematic comparison. Blockbench addresses this gap with extensible benchmarking and evaluates major platforms to expose limitations and bottlenecks.
- Private blockchains target applications such as security trading, finance, banking, and insurance that are currently supported by enterprise databases.
- Blockbench asks how well permissioned blockchains handle data-processing workloads and which platform should be chosen among siloed alternatives.
- Permissioned blockchains authenticate participants while retaining distrust, enabling more efficient Byzantine-failure protocols than public settings.
- Blockbench divides blockchain architecture into consensus, data-model, and execution layers and begins with Ethereum, Parity, and Hyperledger.
- Blockbench provides macro- and micro-benchmarks, integrates workloads and backends through APIs, and measures throughput, latency, scalability, and fault tolerance.
- The evaluation finds concrete blockchain limitations for data-processing workloads and identifies bottlenecks in Ethereum, Parity, and Hyperledger.
2. PRIVATE BLOCKCHAINS
Private blockchains adapt blockchain mechanisms for authenticated participants and application-specific state, while retaining stronger failure assumptions than traditional databases. Their architectures differ in consensus, storage, execution, and application interfaces.
- Blockchain nodes maintain shared states and historical transactions while agreeing on transaction content and order despite possible Byzantine behavior.
- Blockchain transactions require ACID semantics, but blockchain systems address a more hostile failure model than crash-tolerant distributed databases.
- Bitcoin represents state as digital coins, whereas Ethereum extends the model with user-defined, Turing-complete smart-contract state machines.
- The blockchain software stack distinguishes fully validating nodes from non-validating nodes, which store only block headers, and platforms expose different application interfaces.
- Private blockchain systems replace open participation with authenticated nodes, enabling deterministic consensus alternatives to computationally expensive proof-of-work.
- Ethereum, Parity, and Hyperledger differ in their smart-contract and application designs, motivating workload-based comparison across abstraction layers.
3. Blockbench DESIGN
Blockbench organizes blockchain systems into common abstraction layers and workloads to compare design choices across platforms. The framework connects consensus, data, execution, and application behavior to measurable trade-offs.
- Blockbench identifies four common layers—consensus, data, execution, and application—and designs workloads targeting each layer.
- Consensus layer: Consensus protocols determine how nodes agree on blockchain content, spanning proof-of-work, PBFT, and hybrid designs.
- Consensus layer: PBFT is communication-bound at O(N^2), tolerates fewer than N/3 failures, and requires authenticated identities, limiting its deployment and scalability scope.
- Data layer: Private blockchains commonly use account-based state, while their storage structures differ across platforms, including in-memory blocks and Merkle-based key-value trees.
- Application layer: Blockchain nodes validate recent blocks while exposing RPC-like interfaces that let lightweight clients and third-party applications access blockchain state.
- Execution layer: Smart-contract execution must be fast and deterministic, with Ethereum and Parity using EVM bytecode while Hyperledger uses a different runtime approach.
- Application layer: Blockbench workloads reflect applications including cryptocurrency, decentralized organizations, security settlement, insurance, and sharing-economy workflows.
3.2 Blockbench Implementation
Blockbench integrates blockchain backends and workloads through connector APIs, then executes configured tests and reports performance and security statistics. Its metrics cover throughput, latency, scalability, fault tolerance, and attack-related behavior.
- A blockchain backend joins Blockbench by implementing IBlockchainConnector operations for deployment, transaction invocation, and state queries.
- The Driver accepts a workload and configuration, executes operations on the blockchain, and outputs running statistics.
- Because blockchain services process transactions asynchronously, the Driver tracks submitted transaction IDs to measure completion rather than blocking like transactional databases.
- Evaluation Metrics: Throughput counts successful transactions per second, while latency measures response time per transaction.
- Evaluation Metrics: Scalability measures throughput and latency changes as node counts and concurrent workloads increase.
- Evaluation Metrics: Fault tolerance measures performance changes during node failures, including crashes, injected network delays, and corrupted responses.
- Blockbench’s smart-contract table provides Solidity implementations for Ethereum and Parity and Golang implementations for Hyperledger.
3.4 Workloads
BlockBench divides workloads into macro benchmarks for application performance and micro benchmarks for lower-layer behavior. Its workloads cover database-style operations, real smart contracts, storage, computation, consensus, and analytics.
- Workload categories: Macro benchmarks evaluate application-layer performance, while micro benchmarks isolate the consensus, data-model, and execution layers.The workloads are implemented as smart contracts and include both database benchmarks and real Ethereum workloads.
- Database workloads: YCSB provides key-value operations with configurable read/write ratios, whereas Smallbank models transactional money transfers across bank accounts.YCSB preloads records and supports varied request mixes; Smallbank contains three tables and four procedures.
- Smart-contract workloads: EtherId, Doubler, and WavesPresale represent real smart-contract workloads involving domain registration, participant payouts, and digital-token sales.Their implementations expose differences in data-model support across Ethereum, Parity, and Hyperledger.
- Consensus and analytics: DoNothing primarily exposes consensus cost, while Analytics evaluates scan and aggregate queries over historical blockchain data.Analytics computes transaction totals and maximum transaction values over block ranges and account histories.
- Layer-specific workloads: IOHeavy measures storage behavior through random state reads and writes, while CPUHeavy measures execution efficiency by sorting a large array.IO bandwidth is estimated from transaction latency, and CPUHeavy stresses computationally intensive contract execution.
4. PERFORMANCE BENCHMARK
The benchmark compares Ethereum, Parity, and Hyperledger across performance, scalability, resource use, and fault tolerance. Hyperledger generally leads in throughput and efficiency, but its scalability is limited, while Ethereum and Parity show greater failure resilience alongside fork vulnerabilities.
- Bottlenecks and resources: Hyperledger’s main bottleneck is consensus, Ethereum’s is consensus, and Parity’s is transaction signing.The systems also differ in resource efficiency: Ethereum and Parity incur larger memory and disk overheads than Hyperledger.
- Throughput and latency: Hyperledger outperforms Ethereum and Parity across benchmarks, reaching up to 5.5x Ethereum’s and 28x Parity’s throughput.At peak, Hyperledger achieves 1273 tx/s, though this remains below in-memory database performance.
- Throughput and latency: Parity’s throughput remains capped near 80 tx/s because it enforces a maximum client request rate, producing lower throughput and latency than the other systems.Its pending-request queue grows under low offered load but remains smaller under high load because processing stays constant.
- Scalability: Hyperledger stops scaling beyond 16 nodes because its implementation repeatedly fails to reach consensus on new views.The paper attributes this failure to the implementation rather than the original PBFT protocol.
- Fault tolerance and security: Ethereum and Parity tolerate node failures more resiliently, but both fork during attacks, with up to 30% of blocks generated on forked branches.Hyperledger avoids forks but takes about 50 seconds longer to recover after partitioned nodes reconnect.
- Execution and data model: Hyperledger’s native execution is faster and more memory-efficient, while its low-level data model enables customized analytical-query optimization.All three systems execute contracts on only one core, limiting use of multicore hardware.
5. DISCUSSION
BlockBench narrows private-blockchain design space into layers that expose performance bottlenecks and trade-offs. The discussion highlights storage, hardware, sharding, and declarative execution as improvement directions.
- BlockBench’s layered analysis identifies performance bottlenecks and design trade-offs across private-blockchain systems.The framework distinguishes architectural layers and uses benchmarks to connect design choices with observed behavior.
- Separating storage, execution, and consensus would allow these components to be optimized and scaled independently.The paper notes that generic key-value storage may not fit blockchain-specific data structures and operations.
- Trusted hardware can reduce Byzantine-fault-tolerance network messages, while multicore CPUs and large memory can improve execution and I/O performance.
- Sharding can reduce computation cost and accelerate transaction processing, but Byzantine-consistent coordination across shards remains challenging.
- Declarative high-level operations can simplify complex smart contracts and enable low-level optimizations for faster execution.
6. RELATED WORK
Earlier blockchain performance studies largely examined public systems, whereas BlockBench evaluates private blockchains at scale using database-oriented workloads. It also compares systems and relates their designs to overall performance.
- Prior blockchain performance studies focused mainly on public blockchains and commonly examined consensus or network effects.
- BlockBench evaluates private blockchain systems at scale against database workloads and compares how their designs affect performance.
- BlockBench shares database benchmarks’ high-level framework design but uses workloads and a driver tailored to blockchain systems.
7. CONCLUSION
The paper introduces BlockBench for evaluating private blockchains across data-processing workloads and blockchain layers. Its evaluation of Ethereum, Parity, and Hyperledger finds limited suitability for large-scale processing and exposes bottlenecks and trade-offs.
- BlockBench is a benchmarking framework for measuring private-blockchain data-processing performance and understanding performance at different architectural layers.
- The evaluation covers Ethereum, Parity, and Hyperledger using two macro benchmarks and four micro benchmarks.
- Current blockchains are not well suited for large-scale data-processing workloads.
- The study demonstrates bottlenecks and design trade-offs at different layers of the software stack.
A. SURVEY OF BLOCKCHAIN PLATFORMS
The survey compares blockchain platforms by execution environment, programming language, data model, and consensus protocol. Most surveyed platforms use account-based data models, while Ripple and Corda use UTXO-like models.
- The surveyed platforms use varied smart-contract execution environments, including EVM, JVM, Docker images, Haskell, and trusted execution environments.
- Platforms support different contract languages, ranging from Solidity and Serpent to Python, Java, Golang, and platform-specific languages.
- Most surveyed blockchain platforms use account-based data models, while Ripple and Corda use models resembling Bitcoin’s UTXO design.
- Consensus choices span PBFT, Proof-of-Work, Proof-of-Stake, and other protocol designs across the platforms.
B. MACRO BENCHMARKS
The macro benchmarks show substantial performance and scalability gaps between private blockchains and database systems, with outcomes shaped by consensus and execution behavior. Ethereum, Parity, and Hyperledger exhibit distinct latency, queueing, scalability, and resource-utilization patterns.
- Performance comparison: H-Store exceeds 140K tx/s with sub-millisecond latency, while the blockchain systems show at least an order-of-magnitude lower throughput and two orders of magnitude higher latency.The gap is attributed to consensus costs; H-Store requires little peer coordination for YCSB, whereas Ethereum and Hyperledger incur PoW and PBFT overhead.
- Conclusion: The results indicate that blockchain currently performs poorly on data-processing tasks handled by database systems, although the technologies have different goals and assumptions.Byzantine-failure-tolerance protocols are characterized as excessive for traditional trusted database settings.
- Resource utilization: Ethereum is CPU-bound under PoW, Hyperledger is communication-bound under PBFT, and Parity has lower resource footprints than the other systems.These resource-use patterns are presented as direct consequences of the respective consensus protocols.
- Latency and queue behavior: Ethereum has higher latency and variance, Parity has the lowest variance, and Hyperledger’s client queue never shrinks because it failed to generate blocks in the tested setting.Parity’s low variance follows from restricting client request rate to 80 tx/s, while Hyperledger’s queue behavior suggests a server-side network-request bottleneck.
- Scalability and workload effects: Hyperledger fails to scale beyond 8 nodes on Smallbank, while Smallbank reduces H-Store throughput 6.6x and increases latency 4x relative to YCSB.Blockchain performance degrades more modestly on Smallbank—10% in throughput and 20% in latency—because each node maintains the entire state.
C. ANLAYTICS SMART CONTRACT
The analytics smart contract supports historical account-data lookup by storing versioned account records and tracking each version’s committing block. Queries retrieve specific account versions through composite keys.
- Versioned storage: Historical lookup uses versioned account keys, while account:latest identifies the newest version.Each version stores a CommitBlock field indicating the block in which its balance was committed.
- Query execution: The contract answers analytics queries by fetching account versions and filtering them according to their commit-block range.
- Analytics workload: Versioned state enables retrieval of account history rather than only the current balance.