PaperlayerView exact revision on arXiv

Source-linked AI summary

Here is a GIFT: Enforcing User Data Isolation in LLM Serving via GPU Information Flow Tracking

Jiacheng Shi, Xunjie Wang, Cheng Tan, Jinyu Gu

arXiv:2608.25431v1cs.CR

TL;DR

Sensitive user data in shared LLM serving infrastructure creates isolation and leakage concerns. GIFT tracks user-data flow with encryption-based CPU isolation and static GPU analysis, while GIFT-CC adds confidential computing; evaluations report minimal latency impact and up to 10.7% throughput overhead.

  • Problem

    Shared LLM serving frameworks handle sensitive user data, while applying classic information-flow tracking is difficult because CPU frameworks are complex, rapidly evolving, and costly to instrument.

  • Method

    GIFT combines per-user encryption for CPU-side isolation with validated static information-flow rules and decoupled tracking for GPU kernels.

  • Results

    GIFT adds at most a 5% throughput reduction, while GIFT-CC adds up to 7% overhead or up to 10.7% throughput drop with speculative decoding, with negligible latency overhead.

  • Takeaways & Limitations

    GIFT provides user-data isolation across vLLM and DistServe, and GIFT-CC extends protection to untrusted operating systems and hypervisors.

  • Takeaways & Limitations

    GIFT assumes CPU-side frameworks do not modify user-data contents; unavailable kernels may require approximate black-box rules that can sacrifice soundness.

Abstract

from arXiv · show

LLM serving frameworks process large volumes of user data--often containing sensitive information--on shared infrastructure. Ensuring isolation between users who share the same serving framework (on CPUs) and LLM operators (on GPUs) is critical for privacy protection. This paper presents GIFT, a GPU Information Flow Tracking system that enforces user data isolation in LLM serving with minimal overhead. Moreover, the design of GIFT is non-intrusive and allows CPU-side serving frameworks to evolve freely. It rests on two key insights. First, encryption-as-isolation leverages the observation that CPU components only orchestrate data flow, not content manipulation; thus, per-user encryption can provide isolation without modifying serving logic. Second, GPU kernels exhibit limited and predictable information flows, enabling static flow analysis. GIFT precomputes information flow rules for each kernel and uses decoupled flow tracking, avoiding instrumentation or GPU stalls. Furthermore, we extend GIFT to GIFT-CC, which integrates confidential computing to protect against untrusted operating systems and hypervisors (LLM service providers). Implemented on vLLM and DistServe, GIFT and GIFT-CC enforce user data isolation with a 4-10.7% throughput overhead while maintaining the same latency level.

1 Introduction

LLM serving systems process sensitive user data while rapidly evolving frameworks and GPU-level attacks make reliable isolation difficult. GIFT combines encryption-based isolation with static, decoupled GPU flow tracking and reports low performance overhead.

  • Sensitive LLM applications process private schedules, medical inquiries, and proprietary business documents.
  • GPU-level attacks can redirect access to another user’s KV cache without triggering traditional security mechanisms.
  • Traditional IFT struggles with complex, fast-evolving serving frameworks and imposes runtime overhead incompatible with high-throughput, low-latency serving.
  • GIFT uses encryption-as-isolation because CPU components orchestrate data movement while GPUs manipulate data.
  • GIFT precomputes GPU-kernel information-flow rules through static analysis and executes tracking separately from GPU computation.
  • GIFT supports vLLM and DistServe, while GIFT-CC extends protection to untrusted operating systems and hypervisors.
  • At most a 5% throughput reduction is reported for GIFT, while GIFT-CC incurs up to a 7% throughput overhead and up to a 10.7% drop with speculative decoding.

2 Motivation, Challenges, and Threat Models

LLM serving frameworks handle sensitive data on complex, evolving infrastructure where bugs and GPU-level attacks can cause cross-user leakage. The paper motivates GIFT by combining framework-independent isolation with stronger protection under different system trust assumptions.

  • CPU-side serving systems coordinate scheduling and GPU kernels, while GPUs perform computation-intensive operations such as attention and matrix multiplication.
  • Rapidly growing AI infrastructure codebases contain reported vulnerabilities and proof-of-concept exploits.
  • A vLLM vulnerability can redirect requests to another user’s KV blocks, causing Alice’s response to be exposed to Bob.
  • Classic IFT is difficult to apply because frameworks use multiple languages, evolve rapidly, lack mature multi-language tools, and can incur up to 2.6× Python overhead through instrumentation.
  • GIFT provides isolation with no observable latency increase and only a 4% throughput reduction, demonstrated on vLLM and DistServe.
  • GIFT trusts the operating system and hypervisor, whereas GIFT-CC uses confidential computing when those components are untrusted.

3 GIFT Overview

GIFT isolates users by encrypting user data outside a trusted monitor and tracking ownership throughout GPU execution. Its architecture keeps the serving framework operating on encrypted data while the monitor controls kernels, GPU memory, and plaintext exposure.

  • Per-user encryption protects prompts and derived data, with keys accessible only to the user and trusted GIFT monitor.
  • GIFT tracks ownership for GPU memory regions and enforces a single-user ownership invariant for private information during batched execution.
  • GIFT places the monitor in a protected domain to exclusively operate GPUs while the serving framework remains outside that domain.
  • The framework handles encrypted inputs and control-plane tasks, while GIFT monitor decrypts data and validates ownership metadata before kernel execution.
  • Before data returns to the untrusted framework, the monitor re-encrypts it with the corresponding owners’ keys; plaintext remains within the protected domain.
  • Encryption-as-isolation remains compatible with framework evolution because CPU-side components orchestrate computation and movement without modifying semantic data content.

4 Near-Zero-Overhead GPU IFT

GIFT reduces GPU IFT cost by recording ownership in segments, analyzing kernels statically, and tracking flows on CPUs in parallel with GPU execution. It validates kernel behavior and rejects launches that would produce unexpected ownership flows.

  • 4.2 Ownership Segmentation: Ownership segmentation records contiguous user-owned GPU memory segments instead of individual bytes or elements, reducing metadata volume and complexity.
  • 4.2 Ownership Segmentation: B records replace B × H element-level records for the representative kernel that processes each row of an input matrix.
  • 4.3 Static Flow Analysis: Static analysis exploits deterministic, lightly branching GPU kernels to generate ownership-propagation rules offline rather than instrumenting binary instructions at runtime.
  • 4.3 Static Flow Analysis: 7×B×H runtime operations are reduced to 2×B+1 for the representative kernel through three streamlined ownership checks and propagation operations.
  • 4.4 Decoupled Flow Tracking: The monitor permits only legal kernel launch orders for shared memory segments and rejects launches involving unexpected ownership mixing.
  • 4.4 Decoupled Flow Tracking: CPU-side ownership tracking runs without waiting for kernel completion, hiding tracking overhead behind GPU computation while preserving state alignment for corresponding operations.
  • 4.3 Static Flow Analysis: GIFT derives rules from IFGs, validates them with symbolic execution, and checks tensor records, shapes, aliasing, parameter labels, and owner consistency.
  • 4.3 Static Flow Analysis: Fully automated rule generation is unavailable, so GIFT currently requires manual IFG extraction and assistance translating CUDA code to C++.

5 GIFT-CC: Isolation with Confidentiality

GIFT-CC extends GIFT to outsourced settings where the operating system, hypervisor, and service provider are untrusted. It uses confidential computing and encrypted channels, with added copying overhead for tokens and KV caches.

  • GIFT-CC protects user-data confidentiality when LLM serving infrastructure, including operating systems and hypervisors, cannot be trusted.
  • Encryption-as-isolation encrypts data flowing outside TEEs, while an encrypted PCIe channel provides exclusive GPU control and protection against physical attacks.
  • Encrypted transfers between the GIFT monitor and GPUs add overhead when copying tokens and KV cache.

6 Implementation and Limitations

GIFT is implemented for vLLM and DistServe with small framework changes and support for multiple kernel families and models. Its main limitations concern CPU assumptions, unavailable kernel analyses, and semi-automated rule generation.

  • Implementation: GIFT supports vLLM and DistServe in configurations with and without confidential computing, demonstrating compatibility across representative serving frameworks.
  • Implementation: The Rust monitor uses 5.6K lines of code and 32 interfaces, while each serving framework requires fewer than 120 modified lines.
  • Implementation: A library of IFT rules covers 29 kernel families and works with models including Qwen-2.5, Llama-3, Gemma-2, OPT, GPT-2, and Phi-3.
  • Limitations: The current design assumes CPU-side frameworks do not modify user-data contents; otherwise, such operations must be delegated to the GIFT monitor.
  • Limitations: Unavailable ahead-of-time kernel analyses may require approximate black-box IFT rules, which can sacrifice soundness.
  • Limitations: IFT rule generation remains semi-automated because complete CUDA symbolic execution and comprehensive CUDA transpilation tools are unavailable.

7 Performance Evaluation

The evaluation measures end-to-end latency and throughput for GIFT and GIFT-CC across serving frameworks, models, workloads, and hardware settings. Both systems maintain low overhead, including under speculative decoding.

  • Experimental Setup: GIFT is evaluated against unmodified vLLM and DistServe, while GIFT-CC is compared with corresponding TEE-enabled baselines.Experiments cover Qwen2.5 and OPT models, ShareGPT, HumanEval, and LongBench on A800 and H100 systems.
  • Latency: 5% overhead is the maximum reported GIFT overhead on request rate under the same latency SLO on ShareGPT.The overhead on other benchmarks is reported as similar.
  • Throughput: 2.3% and 4.1% are GIFT’s maximum throughput overheads for DistServe-based and vLLM-based systems, respectively.The reported overhead sources are IFT, encryption, and CUDA API redirection; token encryption is less than 1% of total serving time per request, and API redirection is less than 0.5%.
  • GIFT-CC Latency: 7% and 4% are GIFT-CC’s maximum latency overheads relative to Baseline and Baseline-TEE, respectively.These results use TTFT and TPOT on ShareGPT on the H100 platform under the same latency SLOs.
  • GIFT-CC Throughput: 6.7% and 3.2% are GIFT-CC’s maximum throughput overheads relative to Baseline for DistServe-based and vLLM-based systems, respectively.Relative to Baseline-TEE, the corresponding overheads are 2.4% and 1.7%.
  • Speculative Decoding: 10.7% is GIFT-CC’s maximum throughput overhead relative to Baseline with speculative decoding, compared with 2.1% relative to Baseline-TEE.The larger comparison is attributed to CPU–GPU communication overhead from NVIDIA-CC.

7.2 Costs of Information Flow Tracking

The cost analysis finds that IFT time is generally smaller than GPU kernel execution time and can be overlapped through decoupled flow tracking. CPU and memory overheads also remain bounded across evaluated configurations.

  • Kernel-Level IFT Cost: IFT time is significantly lower than execution time for every evaluated GPU kernel.This result is reported for DistServe running ShareGPT with OPT-13B on an A800; Figure 12 uses a log-scale y-axis.
  • Kernel-Level IFT Cost: Decoupled flow tracking hides IFT by overlapping tracking for one kernel with execution of previously launched kernels.The paper gives DecodingStageAttention tracking overlapped with a preceding MatrixMultiply kernel as an example.
  • Parameter Size: As parameter size increases, kernel execution time rises while IFT time remains constant.Ownership for a hidden-state tensor or KV block can be read or written in one operation regardless of tensor or block size.
  • Parameter Size: For DecodingStageAttention, IFT time exceeds execution time below parameter size 2,048.The paper identifies multi-threaded IFT as a possible future reduction for this case.
  • Input Size: Across different input sizes, IFT time remains lower than execution time, although DecodingStageAttention has the longest IFT time.For this kernel, both times increase with batch size and context length, while IFT grows more slowly than execution.
  • CPU and Memory Costs: 124% and 130 MB are the maximum additional CPU usage and memory overhead reported across A800 and H100 setups.The reported CPU figure is 1.24 cores, and the memory bound is attributed to Ownership Segmentation.

7.3 Effects of Performance Optimizations

GIFT’s performance optimizations reduce synchronization and data-movement costs that would otherwise expose IFT or KV-cache overhead on critical paths.

  • Fine-Grained Concurrency Control: 12% throughput improvement comes from fine-grained concurrency control compared with naive blocking of GPU streams.The optimization avoids unnecessarily blocking prefill when decoding fetches KV cache.
  • Secure Swap Space for KV Cache: 1.9% overhead results with secure swap space optimization, versus an 11% throughput reduction without it.The optimization stores swapped KV blocks in protected monitor memory, avoiding encryption and decryption during copying between GPU and serving framework.
  • Caching Index Tables: Caching index tables prevents monitor loads from GPU memory from blocking on previously launched kernels and exposing IFT overhead on the critical path.The experiment evaluates four kernels, including DecodingStageAttention, that use index tables.

8 Security Analysis

GIFT isolates user data when the serving framework is compromised, while GIFT-CC extends confidentiality against attackers with administrative privileges and untrusted host software.

  • Security Guarantees: GIFT ensures user data isolation despite a compromised LLM serving framework, while GIFT-CC protects confidentiality against attackers with administrative privileges.The distinction reflects GIFT’s and GIFT-CC’s respective threat models.
  • Direct Access Attacks: Encrypted CPU content and VM or TEE boundaries prevent direct access to plaintext in CPU, monitor, or GPU memory.GIFT relies on VM isolation with a trusted OS and hypervisor; GIFT-CC relies on a hardware-enforced TEE boundary.
  • Confused-Deputy Attacks: Generated IFT rules cause the GIFT monitor to reject malicious kernels that mix data from different users.Continuous ownership tracking also ensures the correct encryption key is applied after sequences of GPU kernels propagate data.
  • Physical Attacks: GIFT-CC uses CPU-TEE memory encryption, GPU-HBM protections, and TEE-encrypted PCIe and NVLINK transactions against physical attack vectors.These protections are provided through NVIDIA-CC hardware features.
  • Trusted Computing Base: GIFT-CC excludes the LLM serving framework and host infrastructure from its trusted computing base, retaining a TCB with 32 exposed interfaces.The remaining TCB includes TEE hardware, software in the TEE, CUDA components, and the GIFT monitor.
  • Trusted Computing Base: Without TEE, GIFT relies on the underlying hypervisor to provide an isolated VM for the GIFT monitor.The current implementation uses vanilla KVM, while secure hypervisors could reduce the TCB.

9 Related Work

Prior privacy-preserving LLM-serving approaches rely on cryptography or trusted execution environments, while GIFT targets GPU ownership tracking and user-data isolation with a different design.

  • GPU information-flow tracking: 2.5–5.7× slowdown affects prior instrumented GPU taint tracking, whereas GIFT is designed for near-zero runtime overhead.GIFT avoids instrumenting GPU kernels by generating information-flow rules through kernel analysis and using decoupled tracking.
  • GPU information-flow tracking: GIFT constructs an information flow graph for ownership tracking, unlike prior GPU-kernel analyses aimed at fault tolerance or illegal-access prevention.The comparison distinguishes GIFT’s ownership-tracking goal from analyses such as POS and Honeycomb.
  • Privacy-protected LLM serving: Cryptographic privacy solutions face significant performance overhead and possible model modifications that can affect accuracy, limiting adoption.Prior studies are also primarily evaluated on relatively small models, such as 7B models.
  • Confidential computing: GIFT is orthogonal to TEE-based AI protection and can be combined with confidential computing as GIFT-CC to protect privacy in shared inference services.TEE-based LLM serving can approach non-confidential baseline performance, while GIFT focuses specifically on user-data isolation.
  • Confidential computing: GIFT-CC shares PCC’s privacy goal but is more portable because it is not specific to Apple’s proprietary hardware and software.The supplied passage begins a second distinction concerning PCC’s user requirements but does not provide its completion.

10 Conclusion

GIFT combines ownership segmentation, static flow analysis, and decoupled flow tracking to isolate user data in LLM serving with minimal overhead. GIFT-CC adds confidential computing for stronger protection against untrusted cloud environments.

  • GIFT: GIFT combines Ownership Segmentation, Static Flow Analysis, and Decoupled Flow Tracking to enable user data isolation with minimal performance overhead.The design is presented as the first IFT design for LLM serving systems.
  • GIFT-CC: GIFT-CC integrates confidential computing to provide stronger privacy protection against untrusted cloud environments.The extension builds on GIFT’s isolation design for shared LLM inference services.
Loading 2608.25431v1…