PaperlayerView exact revision on arXiv

Source-linked AI summary

Shards on a Shoestring: Empirical Characterization of NEAR Protocol Nightshade Sharding on Commodity Hardware

Sohini Sahukar, Om Amit Gandhi, Ioan Raicu

arXiv:2609.12091v1cs.DC

TL;DR

NEAR’s published benchmarks cost approximately $700 per hour, and independent empirical characterization on commodity hardware was absent. The paper conducts the first independent empirical characterization of Nightshade on Chameleon hardware, sweeping shard count from N=1 to N=24. At N=16, the chain stalls at 47% CPU with a 29× spike in orphan witness rate, demonstrating that coordination coherence—not hardware resources—is the high-N bottleneck.

  • Problem

    NEAR’s published benchmarks cost approximately $700 per hour, and independent empirical characterization on commodity hardware was absent.

    Source: §I · p. 1

  • Method

    The paper conducts the first independent empirical characterization of Nightshade on Chameleon hardware, sweeping shard count from N=1 to N=24.

    Source: §I · p. 1, A. Hardware and Configuration · p. 2

  • Results

    At N=16, the chain stalls at 47% CPU with a 29× spike in orphan witness rate, demonstrating that coordination coherence—not hardware resources—is the high-N bottleneck.

    Source: §VIII · p. 6

  • Takeaways & Limitations

    The measurements provide real-world pipeline calibration data for the companion SimPy sharding simulator.

    Source: §I · p. 1

Abstract

from arXiv · show

NEAR Protocol's Nightshade architecture targets one million transactions per second (TPS) through horizontal sharding of both state and computation. Published benchmarks were produced on expensive Google Cloud Platform infrastructure costing approximately \$700 per hour, leaving a significant reproducibility gap for academic research. We present the first independent empirical characterization of NEAR Nightshade sharding on commodity hardware: a Chameleon Cloud bare-metal node with 48 hyperthreaded Intel Xeon cores, 128\,GB RAM, and HDD storage at 80--100\,MB/s. We systematically sweep shard count from $N{=}1$ to $N{=}24$, measuring aggregate TPS, per-shard TPS, block time, BFT finality, memory, and disk I/O. We identify three distinct bottleneck regimes: L3 cache pressure at low $N$, witness gossip pipeline saturation at mid $N$, and coherence collapse at high $N$. A key unexpected finding is that HDD write latency acts as implicit flow control for the witness gossip pipeline. Removing it via RAM-backed tmpfs causes complete chain stall at $N{=}16$, with a 29$\times$ spike in orphan witness rate at 47\% CPU utilization. Aggregate TPS peaks at $N{=}8$ (+40\% over $N{=}1$) then reverses, with per-shard TPS collapsing 23$\times$ by $N{=}24$. Our dataset provides the first commodity-hardware calibration baseline for the companion SimPy sharding simulator.

I. INTRODUCTION

NEAR’s expensive published benchmarks left academic researchers without an independent commodity-hardware characterization of Nightshade. This paper measures the system on Chameleon Cloud and connects its results to a companion sharding simulator.

I. INTRODUCTION

  • Nightshade shards both state and computation, while witness gossip across validators creates an O(N^2) communication structure.Chunk producers execute transactions and gossip cryptographic proofs of valid state transitions before block finalization.Source: §I · p. 1
  • NEAR’s published benchmarks cost approximately $700 per hour, and no independent commodity-hardware characterization existed before this work.Source: §I · p. 1
  • The hardware measurements confirm and quantify the simulator’s predicted throughput crossover, where coordination overhead outweighs parallelism gains.The paper is the empirical companion to Gandhi and Raicu [2].Source: §I · p. 1
  • The paper provides the first independent empirical characterization of NEAR Nightshade sharding on commodity hardware.It is intended to provide real-world pipeline calibration data for the companion SimPy sharding simulator.Source: §I · p. 1
  • The study sweeps shard count from N=1 to N=24 and measures block-time decomposition with Prometheus instrumentation.Source: §I · p. 1
  • The paper identifies three bottleneck regimes and finds that HDD backpressure serves as implicit flow control for the protocol.Source: §I · p. 1

II. BACKGROUND

Nightshade assigns accounts to shards and organizes each block as one chunk per shard. Validators hold shard state locally and gossip chunk witnesses across the validator set.

II. BACKGROUND

  • Each NEAR account belongs to exactly one shard, and a block aggregates one chunk per shard.A chunk contains transactions, incoming cross-shard receipts, pre- and post-state roots, and a state witness.Source: A. NEAR Nightshade Architecture · p. 1
  • Validators maintain a full local copy of their shard’s state in RocksDB and execute transactions against it to validate witnesses.Source: A. NEAR Nightshade Architecture · p. 1
  • Each chunk producer gossips its witness to all N−1 other validators, generating N(N−1) witness-gossip messages per block.At N=24, this amounts to 552 messages per block.Source: A. NEAR Nightshade Architecture · p. 2

B. Consensus: Doomslug and BFT Finality

The study measures BFT finality as the time to full transaction irreversibility, distinguishing it from Doomslug’s practical finality. It adapts the GCP benchmark harness and runs experiments on a configured single-node Chameleon Cloud setup.

B. Consensus: Doomslug and BFT Finality

  • BFT finality measures the time for a transaction to achieve full irreversibility, reaching 77.5 s at N=8 versus 6.8 s at N=1.Doomslug provides practical finality after one approval round; the BFT Finality Gadget provides cryptographic irreversibility after two rounds.Source: B. Consensus: Doomslug and BFT Finality · p. 2

C. The Reproducibility Gap

  • The GCP benchmark harness required nine engineering changes for Chameleon, including replacing RPC-based account creation that cascaded into HDD timeouts.The published benchmark used eight GCP instances at an estimated cost exceeding $700 per hour and assumed high-performance NVMe storage.Source: C. The Reproducibility Gap · p. 2

A. Hardware and Configuration

  • Experiments ran on one Chameleon bare-metal node with 48 hyperthreaded Xeon cores, 128 GB RAM, and HDD storage writing 80–100 MB/s.Each run used N validator processes sharing node resources, a 30-second ramp-up, and a 180-second measurement window.Source: A. Hardware and Configuration · p. 2
  • The Phase 4 setup co-located N validators on one node, sharing a single HDD while each kept an independent RocksDB instance.Validators were connected by a full P2P mesh.Source: Figure 1 · p. 2

B. Monitoring

Prometheus measurements tracked throughput and resource indicators per validator, while the analysis decomposed block time from apply-delay metrics. Additional fixes addressed validator mapping and restart behavior.

B. Monitoring

  • A Prometheus scraper polled each validator every 5 seconds, recording transactions, processing time, chunk outcomes, delayed receipts, witness size, RSS, and disk writes.Block-time decomposition used ∆sum/∆count on apply_delay_ms over the steady-state window.Source: B. Monitoring · p. 2

C. Additional Engineering Fixes

  • At N ≥8, the epoch manager can scramble validator-to-shard mapping after rotation, so the script checked RPC and rewrote submission paths.After process termination, RocksDB LOCK files were deleted before restart to avoid EAGAIN panics.Source: C. Additional Engineering Fixes · p. 2

A. HDD Scaling Sweep

Aggregate throughput peaks at N=8 before reversing, while increasing shard count shifts block-processing time from transaction application toward witness-gossip and BFT overhead.

A. HDD Scaling Sweep

  • 1,498 TPS at N=8 (+40% over N=1) is the aggregate peak; per-shard TPS falls 23× from N=1 to N=24.At N=24, each shard produces less than 5% of what a single shard achieves alone.Source: Figure 2 · p. 3, A. HDD Scaling Sweep · p. 3
  • 2,605 ms at N=24, up from 1,019 ms at N=1, reflects monotonic block-time growth as BFT approval messages scale to 552 per block.At N=24, 276 unique validator pairs exchange those messages.Source: Figure 3 · p. 3, A. HDD Scaling Sweep · p. 3

B. Block Time Decomposition

  • At N=16, apply_chunks is 179 ms (4% of 4,486 ms total), leaving a 4,307 ms gap for witness-gossip and BFT coordination.At N=1, apply_chunks is 1,118 ms, or 87% of the 1,285 ms block-processing time; produce_chunk stays under 100 ms at every shard count.Source: B. Block Time Decomposition · p. 3, Figure 4 · p. 3

C. Memory and Disk I/O

Disk throughput and memory use vary with shard count, while chunk skips rise sharply at higher counts and delayed-receipt queue collapse reflects upstream starvation.

C. Memory and Disk I/O

  • 153 MB/s disk writes at N=4 exceed the approximate 100 MB/s HDD ceiling, while RSS spikes to 37.80 GB at N=16.RSS grows smoothly through N=8 before the spike.Source: Figure 5 · p. 3, C. Memory and Disk I/O · p. 4
  • 27.6% run-to-run CV at N=16 marks instability near the coherence-collapse threshold; BFT finality reaches 77.5 s at N=8.These measurements constrain deployment considerations beyond raw throughput.Source: Figure 6 · p. 4, C. Memory and Disk I/O · p. 4

D. Chunk Skip Rate and Delayed Receipts

E. HDD vs. tmpfs Comparison

The tmpfs comparison tests whether HDD throughput is the main ceiling and finds that removing disk I/O can stall the chain at N=16.

E. HDD vs. tmpfs Comparison

  • At N=16 on tmpfs, aggregate throughput was 0 TPS and the chain stalled completely; without HDD write-latency pacing, witnesses arrived before parent blocks finalized, halting the chain.The orphan witness rate spiked 29×.Source: Figure 9 · p. 4, E. HDD vs. tmpfs Comparison · p. 5
  • Moving RocksDB to RAM-backed tmpfs eliminated disk I/O, confirmed by 0 KB/s disk writes.The experiment tested whether HDD throughput was the primary ceiling.Source: E. HDD vs. tmpfs Comparison · p. 4
  • At N=8, tmpfs improved TPS by at most 25%, while HDD write latency paced chunk production to match finalization.Without that pacing, witnesses arrived before parent blocks finalized and the chain halted.Source: E. HDD vs. tmpfs Comparison · p. 5

V. THE THREE BOTTLENECK REGIMES

The study maps three bottleneck regimes across shard counts: cache pressure at low N, witness-gossip saturation at mid N, and coherence collapse at high N.

V. THE THREE BOTTLENECK REGIMES

  • TPS peaks at the boundary between cache-benefit and gossip-dominated regimes, before gossip overhead dominates.The mapped regimes are L3 cache pressure at N=1–2, witness gossip at N=4–8, and coherence collapse at N≥16.Source: §V · p. 5, Figure 10 · p. 5
  • At N=1–2, each validator's account-trie working set exceeds the ∼30–33 MB L3 cache, causing LLC misses and DRAM pointer-chasing.Sharding partitions the trie and improves cache hit rates; this benefit outweighs gossip cost until N=8.Source: §V · p. 5
  • At N=4–8, encoding and gossiping a ∼1 MB witness to all validators and awaiting validation takes approximately 600 ms per chunk regardless of storage speed.At N=8, BFT adds 56 approval messages per block.Source: §V · p. 5
  • At N≥16, removing HDD pacing lets witnesses arrive before parent blocks finalize, triggering orphan retries and a positive feedback loop that drives TPS to zero.At N=16 on tmpfs, the chain stalls at 47% CPU, indicating coordination coherence failure rather than resource exhaustion.Source: §V · p. 5

VI. KEY FINDINGS

Across N=1–24, aggregate TPS peaks at N=8 before declining, while per-shard throughput collapses; findings identify witness gossip and coordination as constraints and HDD latency as implicit flow control.

VI. KEY FINDINGS

  • 1,498 TPS at N=8 (+40% over N=1) is the aggregate peak; throughput falls to 1,204 at N=16 and 1,110 at N=24, while per-shard TPS drops 23× from 1,070 to 46.Beyond N=8, adding shards reduces total throughput on this hardware.Source: §VI · p. 5
  • At N=16, apply takes 179 ms (4% of 4,486 ms), leaving 4,307 ms for witness gossip and BFT coordination, versus apply’s 1,118 ms (87%) at N=1.The measured bottleneck shifts from apply to gossip and coordination as shard count rises.Source: §VI · p. 5
  • Chunk skip rates rise from 0% at N=1 to 50% at N=8 and 80% at N=16; at N=24, a 72% skip rate accompanies upstream starvation.The delayed receipts queue reaching zero at N=24 reflects the chain barely advancing, not recovery.Source: §VI · p. 5
  • At N=8, replacing HDD with tmpfs improves TPS by at most 25%; at N=16, tmpfs stalls the chain at 47% CPU with a 29× higher orphan witness rate.HDD write latency paced chunk production to match finalization, acting as implicit flow control.Source: §VI · p. 5

VII. IMPLICATIONS AND FUTURE WORK

  • The witness gossip pipeline, at approximately 600 ms per chunk for N≥4, is the primary optimization target; suggested approaches are witness compression and tree-based aggregation.Tree-based aggregation would reduce message complexity from O(N^2) to O(N log N); the paper also argues production deployments rely on WAN latency for pacing.Source: §VII · p. 5
  • The dataset supplies simulator calibration points for witness gossip, apply time, BFT overhead, and the N=16 coherence-collapse threshold.It reports approximately 600 ms gossip per chunk at N=4–8, apply time falling from approximately 880 ms at N=1 to 50 ms at N=24, and BFT overhead rising from approximately 139 ms to 2,555 ms.Source: §VII · pp. 5–6
  • Future work includes four-validator experiments, WAN-latency emulation, cross-shard workloads, and profiling witness gossip into encoding, transmission, and validation.The proposed multi-node experiment targets 40,000–55,000 aggregate TPS, or 67–92% of the theoretical 59,200 TPS ceiling.Source: §VII · p. 6

VIII. CONCLUSION

This study provides an independent commodity-hardware characterization of Nightshade sharding. Throughput peaks at N=8 and then reverses, while the tmpfs experiment implicates HDD latency as implicit flow control.

VIII. CONCLUSION

  • Aggregate TPS peaks at N=8 (+40%) then reverses, per-shard TPS collapses 23×, and block time grows 2.6× with O(N^2) BFT coordination overhead.The tmpfs experiment found HDD write latency was implicitly providing flow control; at N=16, removing it stalled the chain at 47% CPU with a 29× orphan-witness-rate spike.Source: §VIII · p. 6
  • The paper presents the first independent empirical characterization of NEAR Nightshade sharding on commodity academic hardware.The study systematically sweeps shard count from N=1 to N=24.Source: §VIII · p. 6
  • The witness gossip pipeline at approximately 600 ms per chunk for N≥4 is identified as the most actionable target for protocol optimization.The conclusion also reports that code, data, and instructions are publicly available.Source: §VIII · p. 6
Loading 2609.12091v1…