Source-linked AI summary
Invisible Yet Dominant: Big Stalls of Kernel I/O Mechanisms in Cloud OLTP Databases
Mitsumasa Kondo
TL;DR
Random-write stalls on distributed block storage remain poorly understood, while kernel-internal stalls can escape standard counters. This paper uses eBPF to profile them across disk layouts; with equal provisioned IOPS and bandwidth, the four-disk layout cuts pauses and improves OLTP performance.
Problem
Random-write stalls on distributed block storage remain poorly understood, despite conventional attribution to network PPS ceilings, bandwidth limits, and general kernel overhead.
Source: §1 · p. 1Method
The paper uses eBPF block and writeback tracepoints to profile kernel I/O stalls and compare database placement across one-, two-, and four-disk configurations.
Source: §1 · p. 1, §2 · pp. 1–2Results
70.0% fewer IO-less Dirty Throttle pauses, 59.4% lower maximum New–Order transaction latency, and 23% higher throughput distinguish Method 3 from Method 1.
Source: §3 · p. 2Takeaways & Limitations
eBPF can quantitatively identify cloud database kernel I/O bottlenecks that standard profiling tools fail to capture.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Most databases, including PostgreSQL, RocksDB, and recent AI KV-cache middleware, rely on buffered I/O, delegating write-back to the Linux kernel. On the distributed block storage standard in the cloud, this delegation inherits a hidden bottleneck: each device is drained by a single kernel flusher thread over a high-latency, shallow-queue path. When the drain falls behind, dirty throttling pauses write() system calls, and even reads that must evict dirty pages stall. These stalls are invisible to iostat and every standard counter. This poster observes the stall from inside the kernel, using the multi-volume data placement proposed in SteelDB as the experimental lever. eBPF probes on writeback and block tracepoints separate write-back by issuing context and count every throttle pause. Across three configurations with identical provisioned IOPS and bandwidth but 1, 2, and 4 devices, we show that adding drains, not bandwidth, cuts throttle pauses by 70%, reduces maximum transaction latency by 59%, and raises throughput by 23%.
1 Introduction
Random-write stalls on distributed block storage have remained poorly understood, despite motivating compute–storage disaggregation. This paper examines the kernel bottlenecks SteelDB addressed using eBPF, focusing on stalls invisible to standard counters.
1 Introduction
- Random-write stalls on distributed block storage have remained poorly understood and were conventionally attributed to network PPS ceilings, bandwidth limits, or general kernel overhead.Disaggregation avoids random writes by writing the WAL directly to dedicated storage and reconstructing database pages on the storage side.Source: §1 · p. 1
- SteelDB links stalls to a single kernel flusher thread per device that cannot drain dirty pages fast enough on high-latency distributed storage, triggering IO-less Dirty Throttle pauses.Its multi-disk placement multiplies flusher threads and queues, enabling parallel writeback and alleviating congestion.Source: §1 · p. 1
- This paper shifts from database-level evaluation to kernel-level eBPF analysis of bottlenecks that standard counters such as /proc/diskstats largely miss.The analysis quantifies their impact on cloud OLTP performance.Source: §1 · p. 1
2 Analysis of Kernel I/O Bottlenecks
The analysis profiles kernel I/O paths with eBPF, distinguishing writeback by issuing context and recording IO-less Dirty Throttle pauses. It compares disk layouts with identical total provisioned IOPS and bandwidth.
2 Analysis of Kernel I/O Bottlenecks
- SteelDB provisions multiple disks and places database data across them to multiply KFTs and I/O queues, isolate I/O paths, and enable parallel dirty-page writeback.The design is described as cross-layer orchestration without modifying the kernel, storage, or database layers.Source: §2 · p. 1
- eBPF traces block requests and writeback intervals to distinguish KFT writeback from foreground I/O, while balance_dirty_pages events capture throttle pauses.The profiling monitors aggregate disk bandwidth and groups throttle invocations into a time series.Source: §2 · pp. 1–2
- The evaluation compares one-disk, WAL-separated two-disk, and SteelDB four-disk layouts with identical totals of 80K IOPS and 2.0 GB/s.Experiments use PostgreSQL 16.4 and TPC-C on AWS; the four-disk configuration allocates one WAL disk and three tablespaces.Source: §2 · p. 2
3 Evaluation and Future Work
With equal provisioned IOPS and bandwidth, SteelDB’s four-disk configuration reduces throttle pauses and improves TPC-C performance over the single-disk layout. The paper concludes that eBPF reveals cloud kernel I/O bottlenecks missed by standard profiling tools.
3 Evaluation and Future Work
- 70.0% fewer throttle pauses (2,513 →755), 23% higher throughput, and 59.4% lower maximum New–Order transaction latency (1,238.2 ms →502.5 ms) distinguish Method 3 from Method 1.Average write bandwidth improves by 21.7%.Source: §3 · p. 2
- Method 1 fails to reach its provisioned I/O ceiling; its single KFT saturates, dirty pages back up, and foreground synchronous paths drive most writeback while contending with user-facing I/O.Method 3’s three DATA disks provide three independent KFTs for parallel dirty-page draining; the substantial pause reduction directly explains the suppressed tail stalls.Source: §3 · p. 2
- The paper uses eBPF to identify cloud database kernel I/O bottlenecks that standard profiling tools fail to capture and plans further study of their relationships with kernel tuning parameters.Source: §3 · p. 2