PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance Study of Serverless Workloads in Confidential Virtual Machines

Rikesh Niroula, Jianchen Shan, Xiaoning Ding

arXiv:2609.04478v1cs.DC

TL;DR

Confidential serverless computing promises hardware-backed protection for sensitive workloads, but its effects on serverless memory efficiency and performance remain insufficiently understood. This study compares serverless workloads in CVMs and regular VMs across memory efficiency, warm execution, and cold-start initialization. It finds memory, VM-exit, and container-launch costs, and evaluates targeted mitigations.

  • Problem

    The effects of CVMs on serverless warm-container capacity, warm-execution latency, and cold-start delay remain insufficiently understood.

    Source: §I · p. 2

  • Method

    The paper empirically compares diverse serverless workloads in CVMs and regular VMs, examining memory efficiency, warm execution, and cold-start container initialization.

    Source: §I · p. 2

  • Results

    CVMs reduce memory consolidation and warm-container capacity, while frequent VM exits and container initialization contribute to runtime and cold-start overheads.

    Source: §I · p. 2, §VII · p. 12

  • Takeaways & Limitations

    Selective relaxation of container-level isolation among mutually trusted same-function instances can significantly reduce container launch time.

    Source: §I · p. 2, §VII · p. 12

    Guest-side polling avoids HLT-triggered exits but can waste CPU and interfere with co-located workloads; disabling PLE raises IR latency by 8% and memtier by 37%.

    Source: 3) Existing system-level mitigations… · pp. 8–9

Abstract

from arXiv · show

Confidential serverless computing is rapidly emerg- ing as a critical paradigm for application domains requiring strong confidentiality guarantees, such as healthcare, finance, and machine learning. To enable this paradigm in untrusted cloud environments, Confidential Virtual Machines (CVMs) provide isolation by encrypting the entire guest memory, and thus securing serverless workloads against host-level access and inter- ference. However, the implications of CVMs for serverless systems remain insufficiently understood. This paper presents an empirical study of serverless work- loads in CVMs, systematically covering memory efficiency and runtime overhead on both warm-starts and cold-starts. Our results show that CVMs incur substantial memory overhead because encrypted memory disables cross-VM page deduplication and reduces memory reclaimability, thereby limiting warm- container capacity under a fixed memory budget. Runtime overhead in warm-starts is workload dependent. Workloads with frequent VMEXITs, particularly idle transitions, suffer substantial slowdowns. These slowdowns are further amplified by common serverless deployment practice that couples vCPU allocation to memory size, as higher-memory configurations expose more vCPUs than some functions can use effectively. For cold starts, the study focuses on container creation, a common and major contributor to startup latency. Motivated by the memory-efficiency results, we consider the deployments in which multiple containers of the same function are consolidated within the same CVM to improve efficiency, and show that selectively relaxing certain isolation mechanisms in this setting can substan- tially reduce startup overhead. These results clarify the main performance tradeoffs of confidential serverless computing and suggest practical ways to improve efficiency and latency.

I. INTRODUCTION

The paper studies how confidential virtual machines affect serverless memory efficiency and runtime performance, addressing limited evidence about warm execution, warm-container capacity, and cold starts. Its findings identify memory, VM-exit, and container-initialization costs, along with mitigation opportunities.

I. INTRODUCTION

  • Serverless computing is used for applications including image processing, log aggregation, data transformation, and backend APIs, with scalability and fine-grained billing among its stated benefits.Source: §I · p. 1

A. Serverless Computing and Confidentiality Challenge

  • CVMs protect private guest memory from direct host access, but memory encryption disables cross-VM deduplication and protected exit handling can add performance overhead.Source: §I · p. 1

I. INTRODUCTION

  • Prior CVM studies used general-purpose workloads, leaving their effects on warm-container retention, warm-execution latency, and cold-start delay unclear.Source: §I · pp. 1–2
  • The study systematically examines memory efficiency, warm-start execution, and cold-start container initialization to characterize overheads, causes, and mitigation opportunities.Source: §I · p. 2
  • CVMs increase memory use and can reduce warm-container capacity, while VM-exit-sensitive workloads and container initialization create runtime and cold-start costs.The study also evaluates selective relaxation of container isolation for mutually trusted instances of the same function.Source: §I · p. 2

A. Serverless Computing and Confidentiality Challenge

  • Enclave-based serverless approaches can require program changes to manage data entering or leaving the enclave and to accommodate system-operation restrictions.Restricted operations include file I/O, dynamic memory allocation, and network access; developers may need to avoid or delegate them.Source: A. Serverless Computing and… · p. 2

B. Confidential Serverless Computing: System Architecture

CVMs extend hardware-backed confidentiality to the whole VM, allowing unmodified applications to execute inside a protected guest. Their protections trade off against memory efficiency and performance, including reduced warm-container capacity.

B. Confidential Serverless Computing: System Architecture

C. Confidential Serverless Computing: Overhead Challenges

1) Memory Efficiency in Serverless Deployments:

2) Runtime Overhead:

The study evaluates CVM memory efficiency and runtime performance with diverse workloads, using cloud platforms and a local testbed for comparison and root-cause analysis. Its memory study isolates deduplication and reclaimability costs while focusing on cached execution environments.

2) Runtime Overhead:

  • CVM isolation mechanisms add runtime overhead, which may be small for compute-intensive workloads but significant for workloads with frequent system operations.The paper identifies increased exit-handling complexity as the main source of this overhead.Source: 2) Runtime Overhead · p. 3

A. Experiment Design

B. Memory-Efficiency Evaluation

  • The memory evaluation compares KSM-enabled regular VMs, KSM-disabled regular VMs, and CVMs to distinguish deduplication costs from additional inefficiency due to reduced reclaimability.Source: B. Memory-Efficiency Evaluation · p. 4
  • The study examines memory overhead as both VM count and container instances per VM increase, and uses locally supplied artifacts and a consistent software stack to reduce measurement variability.Source: B. Memory-Efficiency Evaluation · p. 4, A. Experiment Design · p. 4
  • Memory measurements focus on cached warm containers and runtimes rather than transient execution memory, which is dynamic, rarely deduplicated, and typically reserved for execution.The paper notes that cached environments typically account for the dominant share of memory consumption.Source: B. Memory-Efficiency Evaluation · p. 4

C. Runtime-Overhead Evaluation

The runtime evaluation separates serverless lifecycle phases and compares regular VMs with CVMs for initialization and execution, the phases most directly tied to cold- and warm-start user latency. Controlled local experiments support root-cause analysis.

C. Runtime-Overhead Evaluation

  • The evaluation separates VM startup, remote attestation, initialization, and execution, focusing VM/CVM comparisons on initialization and execution because they directly affect user-observed cold- and warm-start latency.Source: C. Runtime-Overhead Evaluation · p. 4

D. Root-Cause Analysis Strategy

  • When slowdowns appear, the authors profile system and hardware counters, then control experimental conditions on a local testbed to identify contributing causes.Source: D. Root-Cause Analysis Strategy · p. 4

A. Evaluation Platform

  • The evaluation uses GCP Intel Sapphire Rapids servers with TDX and AMD Milan servers with SEV-SNP, plus a local HPE server for experiments.The described cloud instances include 22-vCPU and 4-vCPU Intel configurations and a 4-vCPU AMD configuration.Source: A. Evaluation Platform · pp. 4–5

B. Serverless Platform and Deployment Model

The study uses a single-node OpenFaaS deployment and a diverse benchmark set to compare serverless execution across regular VMs and CVMs. Its experiments examine memory efficiency, warm-start execution, and cold-start container initialization.

B. Serverless Platform and Deployment Model

  • OpenFaaS runs as a single-node Kubernetes deployment, with function and system pods colocated to minimize network variability.Function containers are prebuilt for warm starts; cold-start containers are run from the start.Source: B. Serverless Platform and Deployment… · p. 5

C. Workloads and Benchmarks

  • Fifteen benchmarks from SeBS, FunctionBench, and FaaSProfiler cover compute, I/O, machine-learning, and web/API workloads.Functions use OpenFaaS templates and HTTP requests, with consistent resource limits and environment variables across VM configurations.Source: C. Workloads and Benchmarks · p. 5

V. EXPERIMENTS AND FINDINGS

  • The experiments assess memory efficiency first, then warm-start execution overhead, followed by cold-start behavior.The study compares CVMs with regular VMs and examines overhead causes and mitigation opportunities.Source: §V · p. 5, A. Memory Efficiency · p. 5

A. Memory Efficiency

  • Memory measurements compare KSM-enabled regular VMs, KSM-disabled regular VMs, and CVMs, provisioning each VM with the minimum memory needed for stable execution.This separates the cost of lost cross-VM deduplication from additional inefficiency associated with reduced memory reclaimability.Source: A. Memory Efficiency · p. 5
  • Across representative functions, memory usage grows faster with VM count in KSM-disabled and CVM deployments than with KSM enabled.The comparison includes hello-world and image-recognition workloads; Fig. 3 also reports deduplication overhead and total memory for different functions.Source: 1) VM instances for same functions · p. 6, A. Memory Efficiency · p. 6, Figure 3 · p. 6

1) VM instances for same functions:

CVM memory overhead grows with deployment scale and reflects lost page sharing; it persists across different functions. Consolidating same-function containers within one CVM can reduce memory use and support warm-container capacity.

1) VM instances for same functions:

  • CVM and KSM-disabled memory curves closely align, indicating that lost cross-VM page deduplication is the dominant source of CVM memory inefficiency.Without deduplication, each VM contributes nearly the full workload footprint, while KSM-enabled execution shares pages across VMs.Source: 1) VM instances for same functions · p. 6
  • At four VMs, KSM-disabled memory overhead reaches 88% for hello-world, 104% for ResNet-50, and 128% for ResNet-152; CVM overhead reaches 120–147%.Even a single VM incurs 9–20% overhead without deduplication, and memory inefficiency increases with workload size.Source: 1) VM instances for same functions · p. 6

2) VM instances for different functions:

  • Memory inefficiency persists across three different workloads because KSM can share system-level pages across functions, whereas CVM memory usage tracks KSM-disabled execution.The result identifies lost deduplication as a platform-wide scaling issue, not only an effect of repeating identical functions.Source: 2) VM instances for different functions · p. 6

3) Implications for Warm-Container Capacity:

  • Colocating four instances in one VM reduces memory use versus four VMs, including hello-world from 4.3 GB (KSM disabled) and 2.3 GB (KSM enabled) to 1.1 GB.The study reports the same reduction pattern for ResNet-50 and ResNet-152; same-function colocation in one CVM can mitigate inefficiency and improve warm-container capacity.Source: 3) Implications for Warm-Container… · p. 7

B. Overheads Analysis in Warm Starts

Warm-start overhead under CVMs varies by workload, with some workloads showing minor slowdowns and several machine-learning inference workloads showing greater degradation. VMEXIT activity is associated with slowdowns, but exit frequency alone does not account for them.

B. Overheads Analysis in Warm Starts

  • Compute-intensive and I/O-oriented workloads typically show single-digit warm-start slowdowns under CVMs, while Image Recognition, RNN Character Generation, and CNN Image Classification slow down noticeably.Image Recognition has the largest degradation among workloads.Source: B. Overheads Analysis in Warm Starts · p. 7

1) VMEXIT-Induced Serverless Execution Overheads:

  • Workloads with larger slowdowns generally have higher VMEXIT rates; Image Recognition exceeds 300 thousand VMEXITs per second, while Chameleon has only a few thousand and minimal degradation.CNN Image Classification has around 130 thousand VMEXITs per second and a lower slowdown than Image Recognition.Source: 1) VMEXIT-Induced Serverless Execution… · p. 7, Figure 5 · p. 7
  • VMEXIT frequency alone does not explain slowdown: costly HLT exits associated with idle transitions are linked to greater degradation, while guest-side polling may help mitigate it.Video Face Detection and Logistic Regression have relatively high VMEXIT frequency without significant slowdown because many exits are low-cost types.Source: 1) VMEXIT-Induced Serverless Execution… · p. 7

2) Impact of Parallelism on Execution Overheads:

More runtime-managed threads can worsen CVM warm-start latency by increasing synchronization and idle-transition VMEXITs. Existing system-level responses face a spinning-versus-idle trade-off, while guest-side polling can interfere with co-located workloads.

2) Impact of Parallelism on Execution Overheads:

  • At eight threads, CVM slowdown reaches nearly 200%; on 22-vCPU CVMs it increases to 3.5×, while regular VMs can still improve slightly over single-thread execution.The comparison uses Image Recognition inference on Intel TDX and regular VMs.Source: Figure 6 · p. 8, 2) Impact of Parallelism on Execution… · p. 8
  • Under HLT, average inference latency is 201.4 ms on a regular VM and 347.8 ms on a CVM; POLL reduces these to 30.8 ms and 31.8 ms, respectively.POLL also reduces total VMEXITs from 3.6 million to 536 thousand on the regular VM and from 4.5 million to 679 thousand on the CVM.Source: Table III · p. 8, 2) Impact of Parallelism on Execution… · p. 8
  • Runtime frameworks may spawn threads in step with vCPU allocation, and extra threads can create more idle-active transitions and HLT-triggered exits.This is especially costly for workloads with frequent synchronization.Source: 2) Impact of Parallelism on Execution… · p. 8

3) Existing system-level mitigations and their limitations:

  • Keeping vCPUs spinning avoids costly idle transitions but trades exit overhead for wasted CPU cycles and possible interference with co-located workloads.Short spinning may fail to avoid exits, while long spinning can waste resources; existing approaches still depend on workload idle behavior.Source: 3) Existing system-level mitigations… · p. 8
  • Host halt polling is enabled by default but substantial slowdown remains; it may not adapt quickly to short-lived function phases and cannot eliminate guest-level idle-transition exits.Guest polling avoids HLT-triggered exits, but continuously spinning vCPUs can waste resources and interfere with co-located workloads.Source: 3) Existing system-level mitigations… · p. 8
  • Disabling PLE increases latency by 8% for Image Recognition and 37% for memtier in the evaluated co-located configuration.The experiment used regular VMs to isolate PLE effects from other CVM-specific performance differences.Source: Figure 7 · p. 9, 3) Existing system-level mitigations… · p. 9
  • Selecting concurrency from execution history could address idle transitions closer to the function runtime rather than reacting after they occur.The paper identifies this as a possible platform approach.Source: 3) Existing system-level mitigations… · p. 9

4) Other System-Level Effects:

The warm-start slowdown also appears on AMD SEV-SNP, while a matrix-multiplication improvement seen in the cloud was absent under controlled local conditions. NUMA placement and shared-cache contention can influence performance.

4) Other System-Level Effects:

  • AMD SEV-SNP showed around 10% slowdown versus a regular AMD VM, indicating the observed degradation is not unique to Intel TDX.The comparison used two-core instances running the Image Recognition benchmark.Source: 4) Other System-Level Effects · p. 9
  • Under controlled local conditions, CVMs showed no matrix-multiplication benefit; cross-node memory access caused 2.4% average slowdown, while shared-cache contention degraded performance by approximately 20%.The cache test colocated matrix multiplication with a cache-intensive workload on the same socket.Source: 4) Other System-Level Effects · p. 9
  • NUMA placement and LLC contention can influence cloud performance, and lower CVM adoption may leave some workloads with better cache availability or locality.Source: 4) Other System-Level Effects · p. 9

C. Characterizing Cold Start Bottlenecks

Container initialization is a major component of cold-start time, and measured runc startup varies with CPU configuration, workload contention, and parallel launches. The results point to synchronization and cgroup-write contention as important bottlenecks.

C. Characterizing Cold Start Bottlenecks

2) Impact of CPU Quota Throttling and CPU Load:

3) Launching Multiple Containers in Parallel:

  • Four concurrent runc instances on separate VMs took 52.4–56.5 ms, about 15% faster than runc alone on a four-vCPU VM; on one-vCPU VMs, four instances throttled the CPU and increased runtime.Running three instances simultaneously on a one-vCPU VM kept runtime lower by keeping the CPU active.Source: 3) Launching Multiple Containers in… · p. 10
  • Contention around WRITE|PERCPU operations during cgroup PID updates can consume up to 50% of startup time while runc makes no forward progress.KernelShark traces identify percpu_down_write and the underlying percpu_rwsem as the source.Source: 4) Cgroup Write Contention · p. 10

4) Cgroup Write Contention:

Cgroup writes can stall runc during container startup. In a restricted trust setting where same-function containers are mutually trusted inside one CVM, selectively relaxing container-level isolation mechanisms can reduce launch latency, at the cost of defense-in-depth.

4) Cgroup Write Contention:

  • Consolidating multiple instances of the same function within a single CVM improves memory efficiency even without cross-VM deduplication; the study considers relaxing selected container-level isolation mechanisms to reduce launch overhead under a restricted trust setting.Source: 5) Reducing Launch Overhead in Trusted… · p. 10

5) Reducing Launch Overhead in Trusted Settings:

  • The proposed relaxations apply only when same-function containers within a dedicated CVM are mutually trusted; different functions and tenants remain in separate CVMs.The cloud host and hypervisor remain outside the trust boundary, and the work does not evaluate microarchitectural side channels.Source: 5) Reducing Launch Overhead in Trusted… · p. 11
  • The study evaluates removing cgroup operations, bypassing Seccomp filtering, and reusing an existing container's network namespace to reduce launch overhead.These choices trade intra-CVM defense-in-depth or network isolation for faster startup.Source: 5) Reducing Launch Overhead in Trusted… · p. 11

D. Optimizations for Warm and Cold Start Performance

  • Warm-start multithreading trades parallelism benefits against synchronization and VMEXIT overhead, especially in CVMs.Source: D. Optimizations for Warm and Cold… · p. 11
  • Configuration choices can reduce launch latency, while the identified percpu_rwsem contention can be addressed with CONFIG_CGROUP_FAVOR_DYNMODS, which requires kernel reconfiguration and recompilation.Source: D. Optimizations for Warm and Cold… · p. 11
  • Skipping cgroup assignment reduces launch time by 27%; bypassing Seccomp saves approximately 30 ms, and reusing a network namespace saves approximately 50 ms.These methods are limited to the stated trust assumptions, not general-purpose recommendations.Source: D. Optimizations for Warm and Cold… · pp. 11–12

VI. RELATED WORKS

The related work spans enclave-based systems, general CVM performance studies, and confidential-serverless designs, while leaving conventional CVM-based serverless workloads underexplored. The conclusions compare CVMs with regular VMs and identify memory, runtime, and container-initialization trade-offs.

VI. RELATED WORKS

  • Earlier SGX serverless work focuses on enclave constraints, while CVM studies characterize general workloads and CC-Knative examines deployment feasibility and startup costs.A separate proposed architecture targets limitations such as startup latency, communication overhead, and low function density.Source: §VI · p. 12
  • This study instead measures conventional CVM-based serverless execution, including deduplication loss, VMEXIT interactions, and container-initialization costs, addressing a gap in work on conventional platforms.Prior serverless optimization work largely targets conventional platforms, and CVM performance for confidential workloads remains understudied.Source: §VI · p. 12

VII. CONCLUSIONS

  • Memory encryption prevents cross-VM page deduplication, increasing memory consumption with deployment scale and potentially reducing warm-container capacity.The study compares CVMs and regular VMs across fifteen benchmarks; reduced capacity can limit container reuse to mitigate cold starts.Source: §VII · p. 12
  • High synchronization and idle–active transitions generate frequent VMEXITs, while higher-memory configurations expose more vCPUs and can amplify parallelism and synchronization.Reduced hypervisor visibility also limits Pause-Loop Exiting effectiveness, increasing sensitivity to contention and scheduling inefficiencies.Source: §VII · p. 12
  • Container initialization can dominate invocation latency, and selectively relaxing unnecessary isolation for mutually trusted same-function instances can significantly reduce launch time.Source: §VII · p. 12
Loading 2609.04478v1…