PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Confidential Computing Performance on NVIDIA Blackwell GPUs

Daniyal Khan, Amean Asad, Ansgar Grunseid

arXiv:2608.26575v1cs.DCcs.CR

TL;DR

Confidential inference must protect prompts, KV cache, generated tokens, and model weights without making the economics unacceptable. The paper measures paired CC-on versus CC-off inference and training runs on one B200 host, localizes costs to encrypted boundaries, and finds about 1–3% inference overhead at the correctly configured operating point. The practical conclusion is that software configuration and workload characteristics determine whether overhead stays low or rises to 30–40%.

  • Problem

    The paper asks how much throughput confidential inference costs, where the cost originates, and how to reduce it when workloads run on infrastructure controlled by others.

  • Method

    The study uses paired CC-on versus CC-off runs on one host, attributes overhead to encrypted hardware boundaries, and validates fixes with framework measurements and a microbenchmark.

  • Results

    About 1–3% throughput overhead is achievable for correctly configured confidential Blackwell inference, while stock stacks can incur 30–40% penalties from avoidable configurations.

  • Takeaways & Limitations

    Performance cost follows two axes—a host-operation cost that amortizes with batch size and an NVLink-traffic cost tied to encrypted collective share—so deployment should match software and parallelism to the workload.

  • Takeaways & Limitations

    The study measures inference and training on one eight-B200 host and does not measure security properties, attestation startup, cross-node transfer, or broader unmeasured workload ranges.

Abstract

from arXiv · show

This paper measures the performance impact of running large language model inference and training inside a Trusted Execution Environment (TEE) on NVIDIA B200 GPUs, using Intel Trust Domain Extensions (TDX) confidential VMs together with NVIDIA Confidential Computing (CC) on Blackwell GPUs. The performance impact is derived from paired confidential versus non-confidential runs on a single physical host where the only variable is the GPU CC bit and the TDX guest object in the VM launch. The main result is that confidential inference on Blackwell achieves low single-digit throughput overhead when the stack is configured correctly, at about 1-3%. Stock inference stacks incur 30 to 40% penalties due to avoidable configurations rather than the achievable operating point. The cost is not fully represented by a single number because it is governed by two independent axes, a fixed per-host-operation cost that amortizes as batch size grows and a per-NVLink-traffic cost that tracks the share of the step spent in encrypted collectives, and which of the two dominates is set by the workload and the software. We localize each cost to a specific encrypted boundary, give a microbenchmark that predicts the serving penalty to within a submission count, and end with concrete deployment guidance. GPU compute, energy draw, and usable memory capacity are unaffected by CC.

1 Introduction

The paper addresses the confidentiality exposure of hosted language-model inference by measuring how much TEE protection costs, where the cost arises, and how configuration can reduce it. It uses controlled comparisons, boundary-level attribution, mechanism validation, and end-to-end optimization across a defined performance-study scope.

  • 1.1 Motivation: Untrusted infrastructure can expose prompts, generated tokens, KV cache, and model weights, motivating hardware-rooted confidential computing.The paper frames TDX and NVIDIA CC as protections for data in use and model weights.
  • 1.1 Motivation: The central question is how much confidential inference costs, where that cost originates, and how to reduce it enough for practical use.The paper treats feasibility as established and focuses on the performance price and its causes.
  • 1.2 Contributions: The study measures CC impact under controlled conditions, attributes overhead to hardware boundaries, validates its mechanism, and tests changes that bring inference near non-confidential performance.Its stated goal is to replace uncontrolled estimates with a measured, mechanistically explained performance surface.
  • 1.2 Contributions: The evaluation reports a performance surface across framework, parallelism, concurrency, and sequence length rather than a single average penalty.Each reported number is paired with the configuration that produces it.
  • 1.3 Scope: The paper covers inference and training on one eight-B200 host under TDX and NVIDIA CC, while excluding security auditing, attestation startup, cross-node transfer, and unmeasured workload ranges.It is explicitly a performance study that takes the confidential-computing guarantee as given.

2 Background

The background defines the TEE protections and the encrypted CPU–GPU boundaries whose costs are measured. It distinguishes PCIe copies, the command channel, NVLink encryption, and CC-related capability loss, along with serving latency terms.

  • TEE Concepts: A TEE provides hardware-enforced confidentiality, integrity, and attestation for the protected execution environment.Confidentiality encrypts guest memory, integrity detects tampering, and attestation provides a signed launch measurement.
  • TEE Concepts: TDX protects the CPU-side guest while NVIDIA CC extends encryption and integrity protection across PCIe, GPU memory, and NVLink.The GPU releases session keys only after the attestation chain validates.
  • Encrypted Boundaries: PCIe bulk copies use driver-managed bounce buffers and AES-GCM, creating per-byte and per-call costs on host–device transfers.A probe observed 17,505 decrypt calls over 8.25 GiB of D2H traffic, totaling 8.44 GiB at about 517 KB per call.
  • Encrypted Boundaries: Every kernel submission crosses an encrypted GSP-RPC command channel and therefore pays a fixed control-path cost.The command channel is the host-to-GSP boundary for GPU submissions.
  • Encrypted Boundaries: NVLE encrypts cross-GPU collective traffic, so its cost scales with encrypted traffic and the wall-clock share spent in collectives.CC also blocks cuMulticast, forcing multicast-dependent kernels onto fallback paths.
  • Implementation Effects: AES hardware is available, but each per-GPU secure session is single-threaded and pinned to one host core; CC PCIe throughput reaches about 10 GB/s, or 72% of a 13.9 GB/s single-core ceiling.The bottleneck is not missing hardware crypto but limited scaling across host threads.
  • Serving Metrics: TTFT measures arrival-to-first-token delay dominated by prefill, while TPOT measures the interval between generated tokens dominated by decode.

3 Methodology

The methodology uses paired CC-on versus CC-off measurements on identical hardware and software contexts, then records configurations and measurement constraints that affect reproducibility. Cross-row comparisons are directional, but each within-row comparison is controlled.

  • 3.1 Paired Comparisons: Every measurement pairs CC-on and CC-off runs on the same disks, model bits, and GPUs, changing only the CC state and TDX guest object.CC-on combines the GPU CC-mode bit with a TDX confidential VM.
  • 3.2 System Under Test: The system under test uses an Intel Xeon 6767P host with eight NVIDIA B200 GPUs and the listed Ubuntu, QEMU, OVMF, TDX, driver, and framework versions.Models span NVFP4, FP8, AWQ, and bf16; frameworks include SGLang, vLLM, and Megatron-core with TransformerEngine.
  • 3.2 System Under Test: Cross-row comparisons are directional because configurations were collected across frameworks, versions, and weeks, whereas each within-row CC comparison is controlled.
  • 3.2 System Under Test: Table 1 lists each measured configuration by framework, model, quantization, and parallelism.It covers configurations across two serving frameworks and several versions.
  • 3.3 Measurement Constraints: Clean measurement requires rebooting between runs because stale CC GPU state can mimic utilization, throughput, and power penalties.An early uncontrolled run measured 16% on a workload that was actually 2%; all reported numbers use clean boots.
  • 3.3 Measurement Constraints: CUDA event timing and kernel profiling are disabled under CC, so timing uses %globaltimer or wall-clock throughput and attribution uses host-side tracing.Submission-level tracing runs use GPU debug mode and are not throughput anchors.

4 Summary of Results

Well-configured confidential inference on Blackwell incurs low single-digit overhead, while stock configurations can show much larger avoidable penalties. The cost separates into host-operation and encrypted NVLink-traffic components, with software and workload choices determining which dominates.

  • 1-3% overhead is achievable for confidential Blackwell inference, whereas stock stacks can incur 30-40% from avoidable configurations.The headline comparisons use paired CC-on versus CC-off runs on one host; high penalties trace to framework or workload configuration rather than the achievable operating point.
  • Two independent cost axes govern the CC tax: fixed per-host-operation overhead and per-NVLink-traffic overhead.The first amortizes with batch size; the second follows the share of wall-clock spent in encrypted collectives and can form a hardware floor.
  • Full CUDA graphs and CC framework patches reduce a single-GPU eager overlap-on penalty from 35-39% to under 1%.CC makes per-step token readback synchronous and serializes the scheduler in the expensive configuration.
  • A single-GPU workload with overlap scheduling disabled pays about 2%, representing the residual hardware floor.With no NVLink traffic, the remaining cost is the per-submission command-path overhead.
  • Decode-heavy, long-context serving reduces the relative tax because encrypted prefill all-reduce is exposed while decode hides behind memory latency.On the production MoE, the tax reaches about 3.6% at realistic output lengths and is within measurement noise at 32k context.
  • Expert and data parallelism are preferred over attention tensor parallelism, while eight-GPU CC training costs about 10-13% from encrypted collective communication.Removing attention tensor parallelism eliminates attention NVLE traffic that grows with context.
  • GPU compute and energy draw are unaffected by CC, while B200 VRAM has a fixed 700 MB carveout without changing the 183 GB usable figure.The carveout should still be included when sizing KV cache against the VRAM ceiling.

5 Cost Localization on the Data Path

The paper localizes confidential-computing overhead to encrypted PCIe, command, and NVLink paths rather than GPU computation. A synthetic probe predicts serving penalties by submission count, while production measurements reveal both launch and collective-traffic costs.

  • 5.1 Per-Operation PCIe Crypto Cost: PCIe transfers have a fixed 3 to 6 µs cipher setup for transfers of 64 KB and below, while transfers of 1 MB and above cost about 0.1 ns/byte.The crossover occurs around 256 KB to 1 MB.
  • 5.1 Per-Operation PCIe Crypto Cost: About one core of AES-GCM runs at a time per GPU, and throughput is flat while H2D degrades under host-thread oversubscription.Confidential crypto scales across GPUs, not across host threads.
  • 5.2 Command-Path Cost and the Serving-Penalty Model: Each kernel submission costs about 12 µs more under CC, whereas a bare synchronization is free.The cost comes from the encrypted GSP-RPC command path; on an eight-GPU CC partition it rises to about 31 µs per launch.
  • 5.2 Command-Path Cost and the Serving-Penalty Model: A full CUDA graph reduces the synthetic decode slowdown from 2.41× in eager mode to 1.04× by collapsing host submissions.The probe uses about 181 kernel launches per step, and GPU compute itself carries almost no CC overhead.
  • 5.2 Command-Path Cost and the Serving-Penalty Model: 2.28 ms of CC overhead per vLLM step corresponds to about 185 piecewise-split submissions, matching the synthetic prediction.The overhead is 50% of the step at batch 8 and amortizes toward 15% at production batch; crypto CPU time is 0.5%.
  • 5.3 Encrypted NVLink Bandwidth and Latency: NVLE costs about 10% of NCCL collective bandwidth, about 11% of Copy Engine bandwidth, and about 18% of SM-based bandwidth, with roughly 4× latency for small cross-GPU writes.These per-unit rates are independent of batch size, while end-to-end impact depends on the share of the step spent in cross-GPU collectives.

6 Single-GPU Overhead and Overlap Gating

Single-GPU CC overhead ranges from negligible to roughly 39%, depending primarily on whether framework scheduling preserves host-device overlap. Framework patches restore overlap, reducing the residual tax to under 1%, while compute, power, and usable memory remain essentially unaffected.

  • 6.1 Overlap Off: the Residual Hardware Floor: 2% versus 39%: the same single-GPU CC mechanism produces very different taxes depending on whether the overlap scheduler is active.With overlap disabled, the fixed per-submission cost is exposed; with overlap enabled, synchronous token readback serializes scheduling.
  • 6.1 Overlap Off: the Residual Hardware Floor: Usable VRAM remains 183 GB in both modes, while CC reserves 700 MB during initialization and may draw slightly less power at matched throughput.The reservation is a one-time protected-framebuffer cost.
  • 6.1 Overlap Off: the Residual Hardware Floor: Under overlap-off, full CUDA graphs and resident weights and KV cache leave only an approximately 11 µs per-step crypto setup cost.At concurrency 64, this is small relative to a 22 ms step.
  • 6.2 Overlap On: the Lost-Overlap Penalty: 51% higher TPOT and 58% higher TTFT occur at concurrency 64 when shipped SGLang overlap scheduling is enabled under CC.Qwen3-8B TPOT rises from 5.48 to 8.28 ms and TTFT from 259 to 409 ms.
  • 6.2 Overlap On: the Lost-Overlap Penalty: 39% lower throughput results when synchronous bounce-buffer readback serializes the scheduler and drops GPU utilization from 74% to 57%.The accompanying power decrease reflects GPU idle time rather than a hardware fault.
  • 6.3 Framework Patches: Under patched CC, the overlap-on penalty is erased almost entirely, with only a few-percent residual for long-input, long-output shapes.The asynchronous D2H worker restores compute/copy overlap; single-GPU configurations otherwise have no encrypted NVLink cost.

7 Multi-GPU Overhead and Parallelism Dependence

Multi-GPU overhead is governed by encrypted collective traffic after software removes fixed per-step host-operation costs. Tensor parallelism increases the NVLink tax, whereas alternative parallelism and right-sized tensor parallelism can keep confidential serving in the low single digits.

  • 7.1 The Two-Axis Cost Model: Two independent axes govern multi-GPU CC cost: fixed per-host-operation overhead and per-NVLink-traffic overhead.The first amortizes with batch size; the second follows the share of each step spent in encrypted collectives.
  • 7.1 The Two-Axis Cost Model: Full CUDA graphs collapse roughly 180 eager launches per step to one replay, removing repeated per-submission CC costs.The synthetic probe measures an approximately 12 µs CC tax per submission.
  • 7.3 Framework Patches: After patches recover the per-step axis, remaining multi-GPU overhead is encrypted NVLink all-reduce traffic rather than GPU idle time.Higher patched-CC utilization where throughput trails non-CC indicates encryption work on the communication path.
  • 7.4 Parallelism Strategy and the Long-Context Penalty: 9 to 14.5% overhead grows with input length under attention TP2, whereas no-TP Qwen3.5 overhead falls from 11% to 2%.Attention tensor parallelism adds encrypted prefill all-reduce traffic; without it, fixed overhead amortizes as context grows.
  • 7.5 Production Surface: At TP8, the production surface rises to about 5.4% with concurrency 64 but falls from 7.4% at 512-token input to noise at 32k context.Concurrency exposes more all-reduce traffic, while longer compute amortizes fixed per-step overhead.
  • 7.5 Production Surface: TP4 reduces the canonical penalty to 1.5% while delivering 94% of TP8 throughput.Fewer ranks reduce encrypted all-reduce traffic, and this model is not bandwidth-bound enough for eight GPUs to add much throughput.

8 Training

Training overhead on eight B200s is caused by encrypted collective communication rather than computation or memory bandwidth. The measured CC tax is about 10 to 13%, with FP8 suffering a larger relative penalty because communication occupies more of its shorter steps.

  • 8 Training: 10 to 13% is the measured eight-GPU training CC tax with Megatron-core and TransformerEngine.The comparison uses identical overlay disks and CC versus non-CC runs.
  • 8 Training: Encrypted tensor-parallel all-reduce and mixture-of-experts all-to-all communication account for the training tax.GEMM and HBM are unaffected, and utilization is identical across modes.
  • 8 Training: FP8 incurs a 1.24× CC penalty versus 1.04× for non-CC because communication becomes a larger share of its shorter computation step.Non-CC draws more power because its GPUs spend less time stalled on communication.

9 The Framework Component of the Tax

Most single-GPU CC overhead is a framework software problem caused by synchronous device-to-host readback and unsuitable timing or memory paths. Applying targeted patches reduces this avoidable cost to under 1%, while RDMA limitations remain an important deployment boundary.

  • 9.1 Framework Changes: CUDA-event timing cannot be trusted under CC, so the autotuner must use the GPU globaltimer or wall-clock throughput instead.CC restricts fine-grained device timers to avoid timing side channels.
  • 9.1 Framework Changes: Pinned host buffers provide no benefit on this CC stack because transfers use driver-managed encrypted bounce buffers.The multi-GPU fusion path also requires a multicast-free workspace because multicast is blocked under CC.
  • 9.2 Deployment Boundary: GPUDirect RDMA is blocked with GPU CC enabled, forcing cross-node transfers through CPU memory and the encrypted bounce-buffer channel.This removes the pinned-memory fast path normally used by RDMA implementations.
  • 9.2 Deployment Boundary: End-to-end confidential prefill/decode disaggregation remains unmeasured and is therefore an integration gap rather than a quantified cost.Its dependence on fast KV-cache transfer is complicated by blocked RDMA and unavailable pinned buffers.
  • 9.1 Framework Changes: 35 to 39% unpatched single-GPU overhead falls to under 1% after applying asynchronous-copy and related framework changes.The asynchronous D2H worker removes readback from the scheduler’s critical path.
  • 9.3 Residual Hardware Cost: The NVLE component remains the closer-to-hardware floor after software removes the per-step axis.For realistic decode-heavy serving, the residual encrypted prefill all-reduce is small.

10 Synthesis of the Compute Tradeoff

On Blackwell, CC leaves GPU compute, energy usage, and measured usable VRAM essentially unchanged; its performance cost is paid mainly at host-device and GPU-GPU boundaries. The multi-GPU cost is tied to encrypted NVLink traffic, while the single-GPU cost is largely an architectural software issue.

  • 700 MB of B200 VRAM is reserved for protected-framebuffer allocations, without changing the measured 183 GB usable figure.
  • Energy usage does not increase under CC, and can be lower when added latency leaves the GPU idle.
  • GPU compute adds almost no CC overhead for compute-bound prefill, GEMMs, and training compute.
  • CC costs are concentrated at host-device and GPU-GPU boundaries rather than in GPU compute.
  • Encrypted NVLink makes the multi-GPU price scale with cross-GPU traffic volume, and batching does not eliminate that cost.

11 Deployment Recommendations

Deployment should minimize host submissions and encrypted data movement while matching parallelism and traffic patterns to the workload. With full graphs, patched software, resident state, and suitable serving workloads, the remaining CC overhead is generally low single digits, although observability is host-side only.

  • Full CUDA graphs collapse submissions and avoid the roughly 12 µs per-submission CC tax that dominates decode.
  • Patched asynchronous D2H software keeps single-GPU CC tax under 1%, whereas shipped or unpatched stacks incur 30 to 40% from lost overlap.
  • Keep weights and KV cache resident on the GPU because the encrypted PCIe path runs near 10 GB/s and does not scale across host threads.
  • Right-size tensor parallelism to reduce encrypted inter-GPU traffic: the canonical MoE point costs about 3% at TP8 versus about 1.5% at TP4 while retaining 94% of TP8 throughput.
  • Decode-heavy workloads with longer outputs and context reduce CC tax, from about 14% at 256-token output to about 1% at 2048 and noise at 32k context.
  • Fully patched multi-GPU MoE serving with full graphs and decode-heavy traffic incurs about 1.5 to 3% NVLink tax at the canonical point, rising toward roughly 5% at very high concurrency.
  • CC disables GPU kernel profiling and CUDA event timing by design, so observability must use guest CPU clocks, host-side NVTX ranges, and end-to-end throughput.
  • Confidential inference remains production-viable, with measured configurations serving hundreds to thousands of tokens per second per deployment.
Loading 2608.26575v1…