PaperlayerView exact revision on arXiv

Source-linked AI summary

GPU-resident Execution for Timing-predictable Sensor Processing Systems

Soham Sinha, Rahul Choudhury, Shekhar Dwivedi, Gregory Lee, Gigon Bae, Anisha Goel, Elena Agostini, Patrick O'Grady, Yosi Levi, Mahdi Azizian

arXiv:2610.13809v1cs.DCcs.OS

TL;DR

CPU-driven GPU workflows keep per-frame sensor I/O, CUDA submission, and completion synchronization on the host, exposing closed-loop latency to host timing variation. GPU-resident Execution uses GPU-visible I/O and a long-lived device-side loop to handle sensor input, run processing, and write outputs without host wakeups. On the IMX274 camera-to-display pipeline, GPU-resident execution achieves 0.66 ms photon-to-display jitter (7.3× lower than CPU-driven (w/ GDR)) and 170× lower software-processing jitter.

  • Problem

    CPU-driven GPU workflows keep per-frame sensor I/O, CUDA submission, and completion synchronization on the host, exposing closed-loop latency to host timing variation.

    Source: §I · p. 1

  • Method

    GPU-resident Execution uses GPU-visible I/O and a long-lived device-side loop to handle sensor input, run processing, and write outputs without host wakeups.

    Source: §I · p. 2

  • Results

    On the IMX274 camera-to-display pipeline, GPU-resident execution achieves 0.66 ms photon-to-display jitter (7.3× lower than CPU-driven (w/ GDR)) and 170× lower software-processing jitter.

    Source: §I · p. 2, C. Case Study: IMX274… · p. 11

  • Takeaways & Limitations

    The results demonstrate structural benefits from removing the host from steady-state execution, including reduced latency jitter in the streaming camera application.

    Source: §VI · p. 12

Abstract

from arXiv · show

Real-time embedded systems increasingly place GPU-accelerated processing of high-bandwidth sensor streams inside closed-loop systems, but the CPU (the host) remains the primary orchestrator of each iteration. The host observes sensor inputs, submits CUDA work, waits for completion, and commits actuation. This host-mediated path exposes the closed-loop end-to-end latency to I/O interrupts, host scheduler decisions, and CUDA API overheads even after common host-side mitigations such as GPU-memory RDMA or CPU core isolation. This paper presents GPU-resident Execution, a new execution model for real-time sensor processing pipelines that keeps I/O and GPU execution in a long-lived device-resident loop. Sensor inputs arrive in GPU memory, trigger GPU-side processing, and reach GPU-visible output endpoints without any host wakeups. We implement the model in the NVIDIA Holoscan C++ framework and provide reference components for building GPU-resident applications. We evaluate our system on NVIDIA GPUs and SoCs with inference, host-contention, multi-loop, and streaming camera workloads. On an industry-grade 1080p, 60 Hz real-time camera-to-display application, GPU-resident Execution reduces photon-to-display jitter from 4.83 ms to 0.66 ms (7.3x) relative to a CPU-driven GPU-memory RDMA baseline and reduces software-processing jitter from 1.70 ms to 0.01 ms (170x).

I. INTRODUCTION

High-bandwidth sensor pipelines need predictable end-to-end timing, yet CPU-driven GPU workflows leave per-frame I/O and coordination on the host. GPU-resident Execution moves steady-state control and processing onto the device; the paper evaluates this model across several workloads and reports large jitter reductions.

I. INTRODUCTION

  • Host-side mitigations improve the environment in which the host runs, but leave per-frame I/O and CUDA controls in Linux.Source: §I · p. 2
  • GPU-resident Execution keeps steady-state sensor I/O and processing on the GPU, with inputs and outputs in GPU-visible memory and no host wakeup.The host remains in the setup and supervisory plane.Source: §I · p. 2
  • The framework treats sensor pipelines as fixed-execution GPU-native units rather than sequences of unbounded CUDA API calls.The paper presents this as a joint treatment of sensor I/O determinism and GPU execution predictability.Source: §I · p. 2
  • On an industry-grade 1920×1080, 60 Hz IMX274 camera-to-display pipeline, the framework’s GPU-resident Execution reduces photon-to-display jitter by 7.3× compared with a CPU-driven solution.Source: §I · p. 2

II. BACKGROUND

  • Linux-hosted GPU sensor pipelines target predictable, strict timing guarantees, including sub-millisecond end-to-end latency.Source: §II · p. 2

A. Challenges

  • CPU-driven loops expose timing to host I/O scheduling, core-isolation limits, and CPU–GPU coordination overheads.Interrupt and scheduler variation, limited SoC CPU cores, and host-submitted CUDA work all affect the time-critical path.Source: A. Challenges · pp. 2–3

B. Sensor I/O

GPU-memory RDMA can place sensor data directly in device memory, but host control of I/O remains. GPU-driven I/O and device-resident CUDA Graph mechanisms move I/O and repeated frame processing onto the GPU.

1) Holoscan Sensor Bridge:

  • GPU-memory RDMA places payloads directly in device memory, while HSB still leaves I/O events and frame reception under CPU control.The design preserves HSB’s RDMA data plane and moves those decisions into a GPU-resident data-ready handler.Source: 1) Holoscan Sensor Bridge · p. 3, B. Sensor I/O · p. 3

2) GPU-driven I/O:

  • GPU-driven I/O exposes NIC doorbells to device kernels, enabling receive polling and send requests without CPU involvement.The design combines GPU-driven I/O with GPU-memory RDMA so the sensor and actuator path needs no host-side I/O thread.Source: 2) GPU-driven I/O · p. 3

C. CUDA Graphs

  • CUDA Graphs reduce many launches to one host call, but CPU-driven runtimes still relaunch once per sensor frame.Source: C. CUDA Graphs · p. 3
  • CUDA conditional WHILE nodes and device graph tail-self-launch can repeat frame-processing loops on-device; GPU-resident Execution integrates these primitives with GPU-visible I/O.Source: C. CUDA Graphs · p. 3

III. SYSTEM DESIGN AND IMPLEMENTATION

GPU-resident Execution represents an application as an operator DAG inside a CUDA Graph. The host initializes and launches that graph once; the GPU then checks for input, processes ready frames, and commits outputs in a steady-state loop.

III. SYSTEM DESIGN AND IMPLEMENTATION

  • The application is an operator DAG of GPU-resident components, encapsulated in one GPU-resident CUDA Graph launched by the host.Source: §III · p. 3

A. Framework Architecture and Lifecycle

  • A GPU-resident application’s host performs initialization and launches the graph once; the device executes each sensor-to-output iteration thereafter.The host configures endpoints and buffers and constructs the graph; the steady-state path remains outside the host’s time-critical work.Source: A. Framework Architecture and Lifecycle · p. 3, Figure 3 · p. 3
  • The steady-state loop checks for device-visible input, runs the operator DAG when data is ready, and commits the result to the output endpoint.A data-ready handler and controller precede the operator DAG, with an end-marker completing the iteration.Source: A. Framework Architecture and Lifecycle · pp. 3–4

B. GPU-resident Graph

  • The default graph uses a WHILE conditional node; its operator DAG is serialized in topological order to avoid timing variability from parallel kernel launches.CUDA Graph dependencies enforce execution order without an additional framework scheduler.Source: B. GPU-resident Graph · p. 4, Figure 4 · p. 4

1) GPU-resident Controller:

The controller translates input and shutdown flags into graph predicates, while device-tail-launched graphs provide an alternative loop mechanism. The API exposes the application, operator, and data-ready-handler abstractions used to build these pipelines.

1) GPU-resident Controller:

  • The controller runs after the data-ready handler each iteration, translating control flags into WHILE and IF predicates that govern graph execution.When input is unavailable, it clears the IF predicate and sleeps for a configurable interval; the data-ready handler sets dataReady when input arrives.Source: 1) GPU-resident Controller · p. 4
  • A host-mapped tearDown flag stops execution by clearing both predicates, with termination taking effect within one loop boundary at worst.Source: 1) GPU-resident Controller · p. 4

2) Device Graph Tail Launch:

  • Device-tail launch uses Graph M for input detection and dispatch, and Graph W for the operator DAG, end-marker, and return to Graph M.The host launches Graph M once; subsequent iterations use device-side tail launches.Source: 2) Device Graph Tail Launch · p. 5
  • Device-tail launch is not the default because device-launchable graphs cannot contain conditional nodes and therefore cannot support conditional subgraphs.Both graph mechanisms expose the same application-facing API.Source: 2) Device Graph Tail Launch · p. 5

C. Application, Operator, and Handler API

  • The API models an application, GPUResidentOperator processing stages, and GPUResidentHandler input-detection subgraphs.The application defines the operator DAG and lifecycle; the handler detects new device-visible input.Source: C. Application, Operator, and Handler… · p. 5

1) Application Interface:

Applications compose and connect operators and register a data-ready handler during host-side initialization, then launch the GPU-resident graph asynchronously. Operators declare device ports and capture CUDA work into the graph, while handlers detect input on the device.

1) Application Interface:

  • During host-side initialization, the application composes operators with make_operator() and add_flow(), registers a handler, then run_async() launches the graph for device-resident streaming.The host can query launch state or request shutdown while the graph runs.Source: 1) Application Interface · p. 5

2) Operator Interface:

  • A GPUResidentOperator performs CUDA operations captured once from the host and repeated within the GPU-resident graph.Source: 2) Operator Interface · p. 5
  • GPUResidentOperator declares named device ports and storage requirements, with either executor-managed shared buffers or endpoint-supplied device pointers.During graph construction, compute() obtains port pointers and records its CUDA operations as a subgraph in the IF node’s true branch.Source: 2) Operator Interface · p. 5

3) Data-ready Handler Interface:

  • A data-ready handler detects device-visible input at each iteration’s start and signals dataReady when input is available.Its operator subgraph is recorded and inserted first in the WHILE body.Source: 3) Data-ready Handler Interface · p. 5

D. Static SM Affinity for GPU-resident Graphs

The timing model depends on a fixed GPU resource and stable clock frequency, and makes the steady-state response path and input-arrival wait explicit. It does not consider shared-cache or memory-bandwidth interference, and application policy determines overload behavior.

D. Static SM Affinity for GPU-resident Graphs

E. Timing Predictability by Design

  • The steady-state path consists of device-side execution and fixed graph transitions, under a deployment model with fixed topology, preallocated buffers, locked clocks, and fixed compute resources.The per-frame host scheduler, interrupt, and CUDA-launch terms are absent; remaining costs are device-side graph and operator costs.Source: E. Timing Predictability by Design · p. 6
  • A sample arriving just after an input check waits until the next check: one controller pass plus the larger of the data-not-ready sleep interval and operator worst-case execution time.After the handler observes the sample, body release and commit remain within that graph iteration.Source: E. Timing Predictability by Design · p. 6
  • When sensor or actuator rates exceed sustainable processing, the application chooses whether to process the newest sample, suppress duplicate processing, or repeat the latest output.The timing model applies to the samples the application policy elects to process.Source: E. Timing Predictability by Design · p. 6

IV. REFERENCE OPERATORS, HANDLERS AND APPLICATIONS

The reference components combine camera ingress, GPU processing, and display egress in a real-time camera-to-display application. They include captured image and inference operators, plus HSB handlers that detect incoming frames on the GPU and can move per-frame I/O control off the CPU.

A. GPU-resident Operator Implementations

  • The reference implementations cover custom CUDA kernels and library-backed image-processing and inference stages, with configuration completed before CUDA operations are captured into a graph.Source: A. GPU-resident Operator Implementations · p. 6
  • For a 1920×1080 raw image, Bayer demosaic uses 4.1 MB input and 16.6 MB output ports for 16-bit Bayer input and four-channel RGBA output.The NPP demosaic operation is captured as a CUDA Graph node, so the host does not resubmit it each frame.Source: A. GPU-resident Operator Implementations · p. 6
  • Format conversion fixes image formats, dimensions, and optional resize parameters before launch, keeping conversion and resizing inside the graph body.Source: A. GPU-resident Operator Implementations · p. 6
  • TensorRT inference is configured with model-derived port sizes and connected device buffers, then exposed as a reusable CUDA Graph launched during capture.Source: A. GPU-resident Operator Implementations · p. 6
  • Camera-image processing reconstructs CSI-2 packet streams into dense Bayer frames and applies black-level correction, histogram accumulation, white-balance gain computation, and per-pixel correction.Source: A. GPU-resident Operator Implementations · pp. 6–7

B. Holoscan Sensor Bridge (HSB) Data-ready Handlers

  • HSB-facing data-ready handlers detect new frames on the GPU; the metadata handler selects the newer of two GPU-visible frame slots but retains a CPU thread for NIC receive-completion management.The metadata handler removes the metadata copy to host memory and CPU frame-selection decision from steady state.Source: B. Holoscan Sensor Bridge (HSB)… · p. 7
  • The DOCA GPUNetIO handler moves per-frame receive-completion polling and descriptor reposting to the GPU, while the NIC driver remains on the host.On a completion, it identifies the buffer slot, copies the frame to a device buffer, advances the completion queue, posts the next receive descriptor, and signals data ready.Source: B. Holoscan Sensor Bridge (HSB)… · p. 7

IV. REFERENCE OPERATORS, HANDLERS AND APPLICATIONS

  • The camera-to-display application combines ingress handlers, processing operators, and display egress, with an IMX274 camera as sensor endpoint and a monitor as output endpoint.Figure 5 shows the application’s hardware and GPU-resident graph structure.Source: Figure 5 · p. 7

C. Display Output

The display operator provides GPU-resident output, and the IMX274 application combines ingress, processing, and display egress in an externally observable sensor-to-output loop. The evaluation measures timing variation across microbenchmarks and an end-to-end camera case study.

C. Display Output

  • Display egress uses a GPU-side gamma-corrected kernel and scanout buffers, with either front-buffer rendering or continuous flipping.Initialization fixes the display mode and obtains device pointers to scanout buffers through Linux DRM, avoiding a graphics/compute context switch on each frame.Source: C. Display Output · p. 7
  • Other actuators can use the same egress structure only if they expose a GPU-visible doorbell; protocol support requires vendor-specific integration.Initialization maps the queue, register, or buffer, and the captured compute body writes the payload and rings the doorbell.Source: C. Display Output · p. 7

D. IMX274 Camera-to-Display Application

  • The IMX274 application combines ingress handlers, processing operators, and display egress into a complete sensor-to-output loop.The camera supports 1080p and 4K at 60 Hz; each frame is reconstructed and corrected, demosaiced, converted, and committed to the display.Source: D. IMX274 Camera-to-Display Application · pp. 7–8

V. EVALUATION

  • The evaluation combines microbenchmarks of latency tails and variation with an end-to-end IMX274 camera-to-display streaming case study on IGX Orin with an RTX 6000 Ada GPU.In the microbenchmarks, each loop iteration is one job with zero release jitter; absolute finishing jitter is J = max_k R_k - min_k R_k, where R_k = f_k - r_k.Source: §V · p. 8, A. Setup · p. 8

B. Microbenchmarks

GPU-resident execution is more robust to host-side timing variation than CPU-driven execution across inference and emulated-I/O tests. The size of the advantage depends on workload and mitigation, and host busy-waiting retains CPU costs.

B. Microbenchmarks

  • The microbenchmarks show improved timing predictability over CPU-driven baselines across quiet inference, host contention, and scalability tests.CPU-driven baselines use Holoscan SDK with CUDA Graphs and a pinned, isolated host core configured to suppress scheduling noise.Source: B. Microbenchmarks · p. 8

1) Single-stage Inference Pipeline:

  • For quiet single-stage inference, GPU-resident execution reduces standard deviation and absolute finishing jitter by 1.4×–2.4× for MobileNetV2, EfficientNet-B0, and ResNet-50.ShuffleNetV2 and VGG-16 show smaller gains, with similar variance between execution modes.Source: 1) Single-stage Inference Pipeline · p. 8, Figure 6 · p. 9

2) Host-side Contention:

  • Under host contention, CPU-CG jitter reaches 1.05–2.15 ms across four inference models, while GPU-resident jitter stays at 9–86 µs across five models.EfficientNet-B0 reaches 97.99 ms in CPU-CG despite a median of 832 µs; the CPU-CG tail behavior was reproducible across runs.Source: 2) Host-side Contention · p. 9, Figure 7 · p. 9
  • For matrix multiplication under host-side contention, GPU-resident execution versus CPU-CG reduces standard deviation from 14.5 µs to 9.3 µs, p99.9 −p50 from 25.7 µs to 16.5 µs, and jitter from 239 µs to 189 µs.It achieves these improvements without reserving any host cores.Source: 2) Host-side Contention · p. 9, Figure 8 · p. 9
  • Busy-waiting can make CPU-CG similar to GPU-resident for small emulated I/O, but inference tails remain exposed to host contention and polling consumes CPU cores.The authors conclude that CPU-side mitigations are not a robust substitute because their effectiveness depends on the application and workload.Source: 2) Host-side Contention · pp. 9–10

3) Scaling to Concurrent Loops:

As concurrent-loop count rises, GPU-resident execution maintains more stable latency than CPU-CG, while device-side interference remains. At ten loops, GPU-resident execution reduces all three reported timing metrics relative to CPU-CG using ten isolated cores.

3) Scaling to Concurrent Loops:

  • The experiment scales 1, 5, 10, 15, and 20 matrix-multiplication loops, assigning each a disjoint 4-SM MPS partition and pinning CPU-CG loops to isolated cores.The isolated-core count varies between 2, 6, and 10 to represent embedded-SoC constraints.Source: 3) Scaling to Concurrent Loops · p. 10
  • At 10 concurrent loops, GPU-resident execution reduces standard deviation from 29.3 µs to 7.6 µs, p99.9 from 1113.6 µs to 50.2 µs, and maximum latency from 1850.3 µs to 404.5 µs versus CPU-CG with 10 isolated cores.GPU-resident execution reserves no CPU cores in this comparison.Source: 3) Scaling to Concurrent Loops · p. 10
  • From five to 20 loops, GPU-resident p99.9 and maximum latency remain within 2×, and standard deviation within 2.5×; CPU-CG worst-case latency grows up to 30× relative to its five-loop value.GPU-resident metrics rise from one to five loops, so device-side interference remains; cache and memory-system effects still need better hardware controls.Source: 3) Scaling to Concurrent Loops · p. 10, Figure 11 · p. 10

4) Framework Overheads:

The paper compares device-side loop mechanisms, measures fixed framework costs, and evaluates an IMX274 camera-to-display pipeline. In the camera case study, GPU-resident execution substantially reduces photon-to-display and software-processing jitter relative to CPU-driven baselines.

4) Framework Overheads:

  • WHILE conditional nodes and device-graph tail launch have similar performance for 10 matrix-multiplication operators, with maximum-latency differences up to 6% and standard-deviation differences up to 3 µs.Neither mechanism consistently leads; WHILE is the default because tail launch lacks advanced CUDA Graph features such as conditional nodes.Source: 4) Framework Overheads · p. 11
  • The empty-kernel experiment characterizes platform-specific fixed framework costs, which can be measured once per new platform for timing analysis.Table I is identified as the framework-overheads breakdown, but the supplied passage does not give its cell values.Source: 4) Framework Overheads · p. 11, Table I · p. 11

C. Case Study: IMX274 Camera-to-display Pipeline

  • The camera case study compares CPU-driven operation without GDR, CPU-driven operation with GDR, and GPU-resident operation using GDR and GPU-driven I/O.The HSB-native measurement tool uses an FPGA to timestamp frame start and display appearance on the same clock, reporting photon-to-display latency as TPD − TLED.Source: C. Case Study: IMX274… · p. 11, Figure 13 · p. 11
  • GPU-resident execution reduces photon-to-display jitter from 4.83 ms with CPU-driven GDR to 0.66 ms, a 7.3× reduction; worst-case latency falls from 18.83 ms to 13.35 ms.The GPU-resident mean is 12.81 ms, versus 16.2 ms for CPU-driven GDR and 24 ms for CPU-driven without GDR.Source: C. Case Study: IMX274… · p. 11, Figure 14 · p. 11
  • GPU-resident software-processing jitter is 0.01 ms, 170× lower in absolute jitter than the CPU-driven GDR path.The GPU-resident path averages 0.30 ms and reaches 0.31 ms maximum, versus 0.96 ms average and 2.60 ms maximum for CPU-driven GDR.Source: C. Case Study: IMX274… · pp. 11–12

VI. DISCUSSION

The discussion frames removing steady-state host involvement as a structural source of camera-loop jitter reduction, while bounding predictability to configured GPU-resident execution. It positions the model against host-regulated schedulers and identifies broader GPU-driven I/O and task communication as open directions.

VI. DISCUSSION

  • Removing the host from steady state, rather than tuning CPU-driven loops, is presented as the structural source of the streaming camera application's jitter reductions.The discussion says tuned CPU-driven loops remain susceptible to host latency tails.Source: §VI · p. 12
  • Predictable latency requires static device-memory layout, fixed clocks, no steady-state CPU interaction, and a working set that fits in device memory.A configurable data-not-ready sleep interval trades polling latency for GPU utilization and power; concurrent loops must use configured SM partitions.Source: §VI · p. 12
  • The framework supports polling multiple sensors within one graph, but evaluation uses one sensor per handler and communication between distinct GPU-resident tasks remains future work.The framework also leaves synchronization between GPU-resident and CPU tasks undefined.Source: §VI · p. 12

VII. RELATED WORK

  • GPU-resident Execution makes sensing, actuation, and display I/O first-class primitives, unlike prior GPU-resident execution systems described in the paper.Those systems use mechanisms such as task queues, GPU-resident runtimes, or CUDA device graph launch.Source: §VII · p. 12
  • Where prior real-time GPU systems regulate host launches, context arbitration, or host-pipeline task release, this model moves the end-to-end path, including I/O, onto the device.The paper frames taskset scheduling as a design-time SM-partitioning problem.Source: §VII · p. 12

VIII. CONCLUSION AND FUTURE WORK

  • Across microbenchmarks and an industry-grade camera-to-display application, GPU-resident Execution tightens latency distributions and reduces end-to-end jitter by orders of magnitude on NVIDIA SoCs and GPUs.The paper attributes this shift to keeping the steady-state closed loop on the device.Source: §VIII · p. 12
  • Future work targets broader GPU-driven I/O, including FPGA-backed adapters for CAN/SPI motor control and direct SPI, GPIO, and I2C control.The paper also identifies deploying GPU-native action models such as GR00T N1 inside real-time control loops as a future direction.Source: §VIII · p. 12

Related papers

Loading 2610.13809v1…