PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Intelligent Networks via AI-Aware GPU-Native Packet Processing

Seyed Mohammad Mehdi Mirnajafizadeh, Yiwen Hu, Rhongho Jang

arXiv:2610.12139v1cs.NI

TL;DR

Inline AI networking remains constrained by host-mediated packet processing and the separation of packet handling from GPU execution. AGP makes the GPU an autonomous owner of packet processing and inference; its evaluation reports line-rate processing with lower marginal power consumption and faster network AI inference than host-scheduled bypass baselines.

  • Problem

    Inline AI networking remains constrained by a host-mediated architecture that separates packet processing from GPU execution and incurs latency, power overhead, and poor performance.

    Source: §1 · p. 1

  • Method

    AGP extends open DPDK for GPU-managed NIC queues, constructs features in GPU memory, and fuses packet I/O, feature extraction, and inference in a persistent mega-kernel.

    Source: §1 · pp. 1–2

  • Results

    AGP achieves line-rate packet processing with 6.1× lower marginal power consumption than CPU/DPDK and accelerates network AI inference by 9.3× over host-scheduled bypass baselines.

    Source: §6 · p. 6

  • Takeaways & Limitations

    GPU ownership of the data path makes the GPU a first-class network endpoint with per-flow scheduling, in-place state, and a host-free fast path.

    Source: §6 · p. 6

    The prototype targeted the mlx5 driver, and porting to other NIC families is left as future work.

    Source: §5 · p. 6

Abstract

from arXiv · show

The integration of inline Artificial Intelligence (AI) models into critical network infrastructure is fundamentally bottlenecked by the high latency and synchronization overhead of CPU-mediated packet processing. While legacy GPU offload and recent CPU-bypass frameworks attempt to bridge this gap, they remain trapped in proprietary ecosystems or still rely on the host CPU and coarse-grained batching to coordinate stateful telemetry and AI pipeline execution. In this paper, we propose AGP, a novel framework that promotes the GPU from a passive accelerator to a primary data-path controller, answering what changes architecturally when the GPU autonomously owns the complete packet-to-inference pipeline. By enabling GPU-native packet processing, contention-free in-GPU stateful aggregation, and a persistent mega-kernel for continuous AI inference, AGP removes the CPU from the critical path. Our evaluation demonstrates that native in-GPU packet processing sustains line-rate throughput while being over 6.1x more power-efficient. Furthermore, our integrated Intrusion Detection System (IDS) stress test eliminates the legacy CPU-GPU synchronization tax, reducing end-to-end latency by 7.9x (up to 35x p99) and accelerating whole-system inference throughput by 9.3x using only ~2% of the GPU's thread capacity.

1 INTRODUCTION

AGP addresses the host-mediated separation between packet processing and GPU inference by making the GPU an autonomous owner of the packet-to-inference path. Its IDS stress test reports line-rate processing, higher power efficiency and inference throughput, and reduced reliance on host resources.

Problem

  • Host-mediated packet processing separates data collection from GPU execution, requiring CPU coordination and batched PCIe transfers that add latency, power overhead, and poor performance.CPU-bypass frameworks can deliver packets directly to GPU memory but still leave state management and inference scheduling on the host.Source: §1 · p. 1

Open challenges

  • The design targets three open challenges: owning NIC queues on an open stack, updating shared telemetry state under SIMT, and triggering inference without host orchestration.The cited challenges include contention from concurrent flow updates and thread divergence during heterogeneous-header parsing.Source: §1 · p. 1

Approach

  • AGP makes the GPU an autonomous network endpoint by extending open DPDK so CUDA cores manage NIC work and completion queues in VRAM.It extracts and aggregates features in place, then fuses packet I/O, feature extraction, and inference in a persistent mega-kernel.Source: §1 · pp. 1–2

Evaluation

  • AGP sustains line-rate packet processing comparable to 16 dedicated polling cores while being over 6.1× more power-efficient, and achieves 9.3× higher inference throughput using ∼2% of GPU thread capacity.The evaluation uses a state-of-the-art IDS as an end-to-end stress test.Source: §1 · p. 2

Contributions

  • AGP combines open GPU-resident NIC queues, warp-cooperative state updates, and device-side mega-kernel scheduling to avoid proprietary stacks, atomic serialization, and CPU-GPU synchronization.Inference is triggered as individual tensors mature rather than through host-driven batched launches.Source: §1 · p. 2

2 BACKGROUND: AI DEPLOYMENT GAP

The background describes a mismatch between line-rate network control and continuous AI execution. Existing approaches accelerate parts of the pipeline but retain host orchestration or proprietary constraints.

Network-to-AI integration

  • Table 1 compares network-to-AI integration architectures; its caption identifies G# as placing payloads in GPU memory while leaving queue polling and packet-address relaying to the CPU.The broader gap is a mismatch between line-rate network control and continuous AI execution.Source: Table 1 · p. 2, §2 · p. 2

Host-orchestrated architecture

  • Classic GPU packet-processing systems, software data planes, and programmable switches still rely on CPU orchestration to prepare data and schedule inference, with repeated host-device transfers adding latency.Standard deep-learning frameworks may launch dozens of micro-kernels, each with 5 to 10 microseconds of context-switching overhead.Source: §2 · p. 2

Proprietary CPU-bypass frameworks

  • CPU-bypass frameworks move packet delivery to GPU memory but leave state tracking and inference launches host-orchestrated, while their closed data paths restrict customization across NIC generations.They also leave raw asynchronous byte streams to be converted into AI-ready tensors, with state updates subject to warp divergence and atomic contention.Source: §2 · pp. 2–3

3 THE AGP DESIGN: A UNIFIED VISION

AGP packages a GPU-owned network fast path in which the CPU supervises asynchronously while GPU-resident stages ingest packets, construct telemetry, and execute inference. Its design covers GPU queue control, stateful aggregation, and autonomous inference scheduling.

3 THE AGP DESIGN: A UNIFIED VISION

  • AGP packages a closed-loop GPU execution environment as a reusable framework that owns the entire fast path.The design responds to host-mediated packet processing and CPU-bypass systems that still leave state and inference scheduling on the host.Source: §3 · p. 3

3.1 Overview

  • The CPU initializes NIC-to-GPU access and launches a persistent kernel, then supervises asynchronously while GPU stages poll queues, parse packets, update state, form tensors, and run inference.The mega-kernel can dispatch GEMM and tensor-mapped payload inspection to Tensor Cores.Source: §3.1 · p. 3

3.2 Handing NIC Control to GPU

  • AGP places NIC queues in VRAM for persistent GPU polling, with flow-level polling distributed across RSS queues and sequence-level polling preserving packet arrival order.Cache-aware polling and device-wide synchronization ensure DMA payload visibility before processing; sequence-level queues use cooperative descriptor claims.Source: §3.2 · p. 3

3.3 GPU-Native Stateful Telemetry

  • For stateful flow telemetry, RSS keeps each flow on one receive ring and warp-cooperative register reductions combine updates before a coalesced memory write, avoiding global atomic serialization.Stateless packet parsing instead processes packets independently and lock-free; flat contiguous arrays support coalesced writes.Source: §3.3 · p. 4

3.4 The Autonomous AI Model Inference

  • AGP fuses lightweight inference within one CUDA block and uses manager warps and worker blocks to schedule multi-stage models from a VRAM-resident inference queue.This device-side execution avoids host scheduling; the design addresses the mismatch between network packet pipelines and prior mega-kernels specialized for transformer inference.Source: §3.4 · p. 4

4 CASE STUDY

The case study evaluates AGP with SketchFeature IDS as an end-to-end stress test and compares it with CPU/DPDK and NVIDIA DOCA baselines.

4.1 System Throughput and Efficiency

  • From 512-byte packets onward, all three systems reach 95.5 Gbps with four or more RX queues, saturating the 100 Gbps link.Figure 3 compares throughput across packet sizes and RX queue counts.Source: Figure 3 · p. 5, §4.1 · p. 5
  • For marginal power at saturation, CPU/DPDK adds 121 W of CPU power, while AGP adds 0.4 W of host CPU power and 20.1 W of active GPU overhead.The reported GPU baseline idle draw is approximately 61.5 W.Source: §4.1 · p. 5
  • On a CAIDA trace, AGP sustains up to 700k new flows/s at 83.5 Gbps with zero packet loss and classifies all detectable flows.The authors report this as a dynamic-traffic resilience test.Source: §4.1 · p. 5

4.2 End-to-End Latency and Inference

  • AGP’s median packet-to-score latency is 19.5 μs, versus 154.9 μs for DPDK and 211.8 μs for DOCA.Figure 4 reports the breakdown under saturated line-rate traffic with continuous inference triggers.Source: §4.2 · p. 5, Figure 4 · p. 6
  • AGP maintains 5.3M flows/s under full-system saturation, versus 0.57M for DOCA and 0.15M for CPU/DPDK.The throughput comparison is reported alongside Table 3.Source: Table 3 · p. 6, §4.2 · p. 6
  • At 60k new flows/s, AGP’s median/p99 latency is 19/24 μs, versus 438/776 μs for CPU/DPDK and 479/857 μs for DOCA.All three systems classify every detectable flow at this typical cloud arrival rate.Source: §4.2 · p. 6

4.3 System Footprint

  • With 16 RX queues, AGP uses no host CPU and at most 32 of the A100’s 108 SMs; CPU/DPDK uses 18 host cores, while DOCA has the largest VRAM footprint.AGP’s persistent kernel uses 32 thread blocks; DOCA also retains a dedicated host core for inference.Source: Table 4 · p. 6, §4.3 · p. 6

5 DISCUSSION & OPPORTUNITIES

AGP’s discussion identifies platform and driver constraints while outlining GPU-based multi-tenant networking and compiler directions.

5 DISCUSSION & OPPORTUNITIES

  • At 64-byte packets, a sub-optimal PCIe topology routes P2P traffic through the CPU root complex, creating a residual throughput gap.Platforms sharing a PCIe switch between NIC and GPU, or integrating them on one device, should close the gap; the prototype targets mlx5, with other NIC families left for future work.Source: §5 · p. 6
  • A persistent AGP pipeline could use a small SM slice as a network front-end to authenticate, parse, and dispatch remote users’ compute requests.The remaining SMs run tenant kernels without host OS intervention.Source: §5 · p. 6
  • AGP could also extend to inline semantic request routing for distributed AI serving and co-located telemetry and congestion control.Source: §5 · p. 6
  • A unified networking-and-AI compiler is proposed to reduce the complexity and error-proneness of fused GPU kernels and hardware-aware memory layouts.It could jointly compile packet-processing and AI-model graphs into optimized GPU kernels and memory layouts.Source: §5 · p. 6

6 CONCLUSION

AGP decouples packet processing from the host CPU through an in-GPU execution environment and fused mega-kernel pipeline. It achieves line-rate packet processing with lower marginal power consumption and accelerates network AI inference over host-scheduled bypass baselines.

6 CONCLUSION

  • 6.1× lower marginal power consumption than CPU/DPDK and 9.3× acceleration of network AI inference over host-scheduled bypass baselines are reported for AGP.The conclusion also reports line-rate packet processing.Source: §6 · p. 6
  • AGP’s GPU-owned data path makes the GPU a first-class network endpoint with per-flow scheduling, in-place state, and a host-free fast path.Source: §6 · p. 6
Loading 2610.12139v1…