uGDS: A User-Space Development Kit for High-Performance GPU Direct Storage

1 Introduction

• GDS removes host staging, but its per-I/O kernel traversal makes software overhead dominant for small, latency-sensitive requests.The introduction also identifies stream FIFO ordering and a closed-source user-space engine as separate limitations.Source: §1 · p. 1
• LMCache chunk reads need only batch-level ordering, but GDS stream FIFO semantics impose a total order and submission calls account for 99.6% of wall-clock time.GDS retains only 40.8% of its synchronous 16-MiB read bandwidth on one stream.Source: §1 · pp. 1–2
• GDS's binary user-space engine prevents users from inspecting stalls or adapting scheduling to their access patterns.The introduction cites submission backpressure and a first-use penalty as examples.Source: §1 · p. 2
• uGDS runs the NVMe command lifecycle in user space after one-time privileged setup, with the CPU orchestrating commands and the SSD transferring payloads.Its design principles include cross-vendor GPU memory, workload-aware scheduling, and command visibility for debugging.Source: §1 · p. 2
• uGDS partitions queues by traffic, tracks DMA-safe resource lifetimes, and supports CUDA and HIP through a shared controller-visible memory abstraction.Its cuFile-style APIs cover synchronous, batch, and stream-ordered I/O; the complete stack is open source for tracing and policy modification.Source: §1 · p. 2

2.1 System Architecture

• The CPU executes NVMe command control in user space, while payloads travel directly between SSD and GPU over PCIe P2P DMA.The kernel helper handles privileged setup; raw block-device semantics trade POSIX metadata, locks, and page-cache coherency for direct access.Source: §2.1 · p. 3

2.2 Device Setup and Queue Ownership

• uGDS claims the NVMe device for user-space control and allocates queue pairs, reserving one dedicated QP for batch I/O.Polling is the default completion mode; interrupt-driven completion uses a kernel notification shim while queue processing remains in user space.Source: §2.2 · pp. 3–4

2.3 GPU Memory Registration and Sharing

• CUDA and HIP allocations are translated into a shared controller-visible page-address representation, with vendor-specific registration mechanisms feeding the same downstream pipeline.The registry tracks DMA addresses and in-flight references; asynchronous batch and stream-ordered interfaces require pre-registration.Source: §2.3 · p. 4

2.4 Request Translation and Command Execution

• The user-space engine validates and translates byte-range requests into NVMe commands, using PRP descriptors to let the controller fetch payload addresses autonomously.NLB is the zero-based logical block count; larger synchronous requests are split into aligned chunks bounded by MDTS and PRP capacity.Source: §2.4 · pp. 4–5

2.5 Queue Partitioning and Concurrency

• A dedicated batch QP isolates bursty submissions from synchronous traffic, while round-robin dispatch lets requests on different general-purpose QPs proceed in parallel.Stream-ordered requests use the general-purpose pool; the QP domains share SSD bandwidth but isolate software submission state.Source: §2.5 · p. 5

2.6 Resource Lifetime and Failure Handling

• Reference tracking and explicit teardown protect in-flight DMA resources; after an unrecoverable timeout, uGDS quarantines queues and memory until an external controller reset halts DMA.Applications must also provide completion barriers when NVMe, GPU compute, and external DMA engines share memory.Source: §2.6 · pp. 5–6

3.1 Synchronous I/O

• Synchronous uGDSRead/uGDSWrite transfer contiguous, block-aligned byte ranges between NVMe and GPU memory, returning only after all chunks commit.The caller can reuse the buffer immediately upon return without an asynchronous synchronization barrier.Source: §3.1 · p. 6

3.2 Batch I/O

• Batch mode separates request submission from completion harvesting, enabling deep pipelining and out-of-order collection of per-request events.A reusable batch context submits descriptor arrays through an isolated hardware queue pair.Source: §3.2 · pp. 6–7
• The batch engine coalesces staged commands into one doorbell write and reclaims queue slots and PRP pages as completions retire.It provisions up to 512 queue entries, 128 pre-mapped PRP pages, and 2-MiB hugepage-backed rings.Source: §3.2 · p. 7

3.3 Stream-ordered asynchronous I/O

• Stream-ordered transfers run after preceding GPU tasks and before later tasks on their target stream, while overlapping compute on independent streams.Host callbacks resolve size and offset pointers at execution, dispatch I/O, and block downstream kernels until DMA completes.Source: §3.3 · pp. 7–8

3.4 Polling and MSI-X interrupt completion

• Busy polling is the default; MSI-X completion is limited to synchronous QPs, while the batch CQ remains polled and unavailable vectors fall back to polling.Interrupts act as wakeup hints: uGDS rechecks CQ phase tags before processing completions.Source: §3.4 · p. 8

4.1 Device Setup Overhead

• 63.3%: median application-side setup time falls from 114.56 ms with GDS to 42.00 ms with uGDS.uGDS defers device-specific work to handle registration and completes hardware resource mapping and queue provisioning in under 42 ms.Source: §4.1 · pp. 8–9, Figure 3 · p. 9

4.2 Synchronous Read/Write Performance

• 84.1%: uGDS reduces 4-KiB synchronous-write latency with two threads, while AMD tests show 1.19× hipFile’s 4-KiB synchronous-write bandwidth.At two threads, 4-KiB write bandwidth is 1003 MiB/s for uGDS and 161 MiB/s for GDS; on AMD, it is 2596 MiB/s for uGDS and 2176 MiB/s for hipFile.Source: §4.2 · p. 9, Figure 6 · p. 11, Figure 7 · p. 11

4.3 Batch Read/Write Performance

• 90.84 ms first-batch read latency and 117.65 ms write latency at 128-KiB requests and batch size 128 contrast with uGDS’s near-identical first and subsequent read latencies; post-first-batch, uGDS retains a lead.The GDS first batch accounts for 87.7% of total read time and 92.2% of total write time; at 128 KiB and batch size 128, uGDS reaches 6530 MiB/s reads and 6223 MiB/s writes, versus GDS’s 3755 MiB/s and 4855 MiB/s.Source: §4.3 · p. 11, Figure 10 · p. 13, Figure 11 · p. 13
• 2.12×–12.12× higher read bandwidth and 2.33×–30.69× higher write bandwidth: uGDS outperforms GDS across all 36 batch parameter combinations.Figures 8 and 9 show batch-read and batch-write bandwidth across request and logical batch sizes.Source: §4.3 · p. 9, Figure 8 · p. 12, Figure 9 · p. 12

4.4 Stream-Ordered Asynchronous I/O and Pipeline Overlap

• At 4 KiB, uGDS reaches 110 MiB/s versus GDS’s 27 MiB/s in a serialized stream-ordered read loop.The loop synchronizes after each request; at 1 MiB, uGDS reaches 3770 MiB/s versus GDS’s 823 MiB/s.Source: §4.4 · p. 11, Figure 12 · p. 13
• 2.00×: uGDS full pipelining completes the fixed workload faster than sequential execution, while GDS improves on its sequential baseline by 10.8%.For uGDS, total time is 6.07 ms sequentially and 3.03 ms with pipelining; GDS records 17.20 ms, 17.56 ms, and 15.35 ms under sequential, per-I/O overlap, and pipelined schedules.Source: Figure 13 · p. 13, §4.4 · pp. 13–14

4.5 Completion Policy: Polling vs. MSI-X Interrupts

• For 4-KiB requests, polling lowers latency 57.2% and raises throughput 2.33×, while 1-MiB MSI-X cuts issuing-thread CPU utilization to 4.15% with a 5.5% latency penalty.Polling pins a CPU core at 100% utilization; MSI-X reduces utilization from 100% to 4.15%.Source: §4.5 · p. 14

4.6 LMCache Integration and End-to-End Evaluation

• 62.7%: uGDS reduces median TTFT at 32,768 tokens, and its serving goodput reaches 50.2 kTokens/s at concurrency 256, a 2.63× improvement over GDS.At concurrency 256, GPU-side KV scattering takes 0.189 ms for uGDS and 0.193 ms for GDS, while SSD-to-buffer transfer latency is 19.98 ms and 56.04 ms, respectively; LMCache submission consumes 1.0% of elapsed time for uGDS versus 99.6% for GDS.Source: §4.6 · p. 14, Figure 15 · p. 15

• uGDS keeps the NVMe command lifecycle in user space, normalizes CUDA and HIP memory, and partitions queues by traffic class while selecting completion policies per I/O mode.The conclusion describes the division of responsibilities as CPU orchestration, SSD transfer, and GPU computation.Source: §5 · p. 15
• In LMCache, uGDS reduces median TTFT by up to 62.7% and raises serving goodput by 2.63×; it also outperforms GDS across evaluated batch configurations and sustains device bandwidth on one stream.Source: §5 · p. 15

NVMe-oF and Multi-SSD Support

• Each uGDS handle currently drives one local PCIe SSD; planned extensions would aggregate multiple SSDs or support remote NVMe-oF targets.The planned extensions reuse the existing user-space queue engine and API.Source: §5 · p. 15

A Lightweight Built-In Scheduler

• Synchronous requests currently use round-robin queue assignment and hold a queue until completion, so large requests can delay small requests on the same queue.The planned scheduler would track queue occupancy, split large requests, and support batch-level ordering with out-of-order completion.Source: §5 · p. 15

Built-In Scheduling Modes

• Planned built-in modes would offer latency, throughput, priority, and adaptive policies selected per handle or stream.The proposed policies vary polling, queue depth, doorbell coalescing, and interrupt-driven completion.Source: §5 · p. 16