PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference

Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque, Sreezon Das Gupta, Atiar Osman, Md. Abu Obaida Zishan

arXiv:2610.33475v1cs.DCcs.AI

TL;DR

Cloud LLM infrastructure faces latency and memory bottlenecks across asymmetric prefill and decode phases, while existing reviews often separate systems layers. This survey synthesizes hardware, orchestration, inference, and cloud-service architectures; it finds a shift toward specialized, heterogeneous, disaggregated execution fabrics and identifies unresolved trade-offs and evaluation boundaries.

  • Problem

    Cloud LLM serving must meet stringent TTFT and TBT objectives, while existing reviews often separate hardware, orchestration, runtime serving, and cloud resilience.

    Source: §1 · p. 4, §4.3.1 · p. 22, §5.4.4 · p. 36

  • Method

    The survey synthesizes accelerator virtualization and cluster scheduling with inference-serving techniques, cloud architectures, and cross-layer cost trade-offs.

    Source: §5.4.1 · p. 34, §5.4.4 · p. 36, §6.5.2 · p. 60

  • Results

    The review describes a shift from homogeneous accelerator clusters toward specialized, heterogeneous, disaggregated execution fabrics, with prefill-decoding phase asymmetry motivating disaggregation and related serving optimizations.

    Source: §4.3.1 · p. 22, §8 · p. 69

  • Takeaways & Limitations

    The reviewed cost-reduction techniques act at distinct, largely composable stack layers, and production deployments such as Mooncake combine several rather than relying on one alone.

    Source: §5.4.4 · p. 36

    Most surveyed evaluations use academic clusters of typically 4 to 64 accelerators or synthetic loads, limiting their representation of commercial datacenter operations.

    Source: §7.3.2 · p. 67

Abstract

from arXiv · show

The widespread deployment of foundational Large Language Models (LLM) across cloud datacenters marks a fundamental paradigm shift from stateless deep learning toward distributed, memory-intensive generative workloads. However, serving massive foundation models under stringent Service Level Objectives (SLO) exposes critical physical and system bottlenecks. These challenges include the persistent High Bandwidth Memory (HBM) wall, operational asymmetry between compute-bound prefilling and memory-bandwidth-bound decoding, explosive Key-Value (KV) cache expansion, prohibitive serverless cold starts, and hardware reliability vulnerabilities. Existing literature reviews remain layer-siloed, decoupling low-level accelerator virtualization from cluster orchestration and runtime inference, while largely overlooking defining advancements from 2024 to 2026. To address these limitations, this study presents a comprehensive, cross-layer systematic review guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 framework. We synthesize a holistic taxonomy unifying hardware virtualization, cluster-level scheduling, and runtime inference optimizations, including iteration-level continuous batching, prefill-decoding disaggregation, and hierarchical KV cache offloading. Furthermore, we evaluate resilient cloud-native service architectures, examining scale-to-zero serverless paradigms, fault-tolerant checkpointing against silent data corruption, and hardware-enforced Trusted Execution Environments (TEEs) for multi-tenant isolation. Finally, we formalize the multi-dimensional Pareto trade-offs governing throughput, latency, monetary cost, and confidentiality, outlining a forward-looking research roadmap spanning Compute Express Link (CXL) memory pooling, in-memory computing, and autonomous multi-agent control planes to guide future cloud LLM infrastructure.

1 Introduction

Cloud LLM serving combines strict latency objectives with memory-intensive, phase-asymmetric workloads and reliability and security constraints. This survey uses a PRISMA-guided, cross-layer synthesis to connect hardware virtualization, orchestration, inference, and cloud-native architectures.

1 Introduction

  • Serving LLMs under TTFT and TBT objectives is complicated by compute-bound prefilling, memory-bandwidth-bound decoding, KV-cache growth, cold starts, hardware faults, and multi-tenant security risks.The KV cache grows with sequence length and batch size, while scaling serverless instances to zero can require transferring multi-gigabyte weights and execution states.Source: §1 · pp. 4–5

1.1 Research Gaps and Motivation

  • Existing reviews are layer-siloed, often separating accelerator virtualization, cluster orchestration, runtime serving, and cloud resilience rather than connecting them.General orchestration reviews often assume stateless workloads, while reliability and security reviews abstract away serving constraints such as latency budgets and KV-cache sharing.Source: §1.1 · pp. 5–6
  • Many prior reviews also predate 2024–2026 advances, motivating a cross-layer survey that connects hardware virtualization, scheduling, runtime inference, resilience, and security.The cited advances include prefill-decoding disaggregation, CXL memory pooling, serverless cold-start reductions, and confidential accelerated computing.Source: §1.1 · p. 6

1.2 Contributions of this Survey

  • The review follows PRISMA 2020 and presents a unified taxonomy spanning accelerator virtualization, cluster scheduling, runtime-serving engines, and resilient cloud-native deployment.Its stated protocol includes formal research questions, multi-database searches, and transparent inclusion, exclusion, and quality-assessment criteria.Source: §1.2 · p. 6
  • Its technical synthesis covers continuous batching, prefill-decoding disaggregation, KV-cache reuse and offloading, serverless serving, fault tolerance, and hardware-enforced TEEs.It also maps trade-offs among throughput, latency, monetary expenditure, and privacy, and outlines research directions including CXL memory pools and decentralized multi-agent control planes.Source: §1.2 · pp. 6–7

1.3 Outline of The Study

  • The paper proceeds from related work and review methods to orchestration, inference optimization, cloud service architectures, trade-offs, and conclusions.A preliminary acronym list provides terminology for reference.Source: §1.3 · p. 7

2 Related Works

The reviewed surveys cover hardware, serving efficiency, orchestration, and reliability, but often treat these topics separately. The paper compares their scopes and limitations to motivate a cloud-systems view that connects them.

2.1 Hardware Architecture and Acceleration for LLMs

  • Hardware reviews cover accelerator architectures and energy concerns, but Kachris (2025) does not connect hardware gains to cloud orchestration, while K. Rostam et al. (2024) end coverage in December 2023.K. Rostam et al. analyze 65 publications selected from 983 and organize techniques across training, inference, and system serving.Source: §2.1 · p. 8

2.2 Efficient LLM Serving and Inference Systems

  • Serving reviews cover algorithmic and systems techniques, but Miao et al. (2025) give limited attention to cloud-specific concerns and Zhen et al. (2025) report qualitative synthesis ending in April 2025.The latter review organizes work into instance- and cluster-level strategies and notes the absence of detailed comparative experiments.Source: §2.2 · p. 8

2.3 Cloud Orchestration and Operational Automation

  • Orchestration surveys address general autoscaling or LLM-assisted operations, but do not unify how LLM workloads themselves are autoscaled and orchestrated in cloud environments.Jeong and Jeong (2025) do not account for LLM-specific memory, KV-cache, or cold-start characteristics; Zhang et al. (2025) focus on LLMs used for AIOps.Source: §2.3 · p. 9

2.4 Reliability, Safety, and Trustworthiness

  • Reliability surveys cover LLM verification and validation, bias, hallucination, and guardrails, but none situates these concerns in cloud serving, including guardrail interactions with batching or runtime monitoring within latency budgets.Huang et al. (2024) note that nondeterministic behavior and uncurated internet-scale training data limit direct applicability of conventional V&V methods.Source: §2.4 · pp. 9–10

2.5 Comparative Summary

  • Across the nine surveyed papers, coverage is often layer-specific, and review cutoffs can quickly become outdated; Table 1 summarizes their scope, methods, and self-reported limitations.The comparison also highlights the open question of integrating safety mechanisms into cloud-serving pipelines while maintaining latency and cost efficiency.Source: §2.5 · p. 10, Table 1 · p. 11

3 Methodology

The review uses a PRISMA 2020 protocol to systematically identify, assess, and synthesize peer-reviewed systems research on cloud LLMs. It defines four research questions and a structured extraction process, then discusses threats to validity.

3.1 Research Questions

  • The review asks how cloud platforms virtualize and schedule accelerators, improve inference efficiency, support cloud-native deployment, and address reliability and privacy.These four research questions align with the paper's thematic sections.Source: §3.1 · p. 12

3.2 Search Strategy and Information Sources

  • Searches covered IEEE Xplore, ACM Digital Library, SpringerLink, SpringerLink, and ScienceDirect for peer-reviewed English publications from January 2020 through February 2026.Queries combined LLM, cloud infrastructure, and systems-efficiency terms and were tailored to each library.Source: §3.2 · p. 13

3.3 Eligibility Criteria

  • Studies had to meet all four inclusion criteria, including archival peer review, cloud-systems relevance, and a concrete system or benchmark with reproducible systems-level evaluation.The criteria excluded purely linguistic or algorithmic studies without systems evaluation, non-peer-reviewed or short contributions, and work limited to standalone devices.Source: §3.3.1 · pp. 13–14, §3.3 · p. 13, §3.3.2 · p. 14

3.4 Study Selection Process and PRISMA Flow

  • Selection followed PRISMA's identification, screening, eligibility, and inclusion phases, with full-text assessment documenting exclusion reasons.The flow diagram summarizes selection across the four databases, deduplication, screening, and eligibility assessment.Source: §3.4 · pp. 14–16, Figure 1 · p. 15
  • 133 publications entered the final qualitative synthesis: 123 primary technical systems papers and 10 foundational survey and taxonomy papers.The review distinguishes these included studies from contextual citations, vendor datasheets, and industry roadmaps.Source: §3.4 · p. 16

3.5 Data Extraction and Synthesis Protocol

  • A standardized extraction matrix recorded cloud abstraction, accelerator sharing, serving and scheduling methods, reliability and security, and evaluation environments and workloads.The extracted data were synthesized qualitatively to map system design choices, compare architectural trade-offs, and build the taxonomies in Sections 4–6.Source: §3.5 · pp. 16–17

3.6.1 Internal Validity

  • Internal-validity safeguards included objective criteria, consistent searches, automated deduplication, multi-pass assessment of ambiguous cases, and structured data extraction.These procedures were intended to mitigate selection bias and subjective interpretation.Source: §3.6.1 · p. 17

3.6.2 External Validity; 3.6.3 Construct Validity

  • The review addresses temporal generalizability by emphasizing fundamental architectural abstractions and construct validity by prioritizing TTFT, TBT, and tail-latency measures under SLO constraints.It also notes that rapidly changing accelerators and cloud configurations pose an external-validity threat.Source: §3.6.2 · p. 17, §3.6.3 · p. 17

4 Orchestration and Resource Scheduling

Cloud LLM orchestration combines accelerator virtualization and sharing with cluster scheduling and serving orchestration. The section surveys hardware and software approaches for sharing GPUs across dynamic, multi-tenant workloads.

4 Orchestration and Resource Scheduling

  • Dedicated GPUs can fragment memory and compute as transformer workloads change across execution phases, motivating spatial and temporal accelerator multiplexing.The taxonomy spans GPU virtualization, cluster-level scheduling, and LLM serving orchestration.Source: §4.1 · p. 18, §4 · p. 18, Figure 2 · p. 19

4.1.1 Hardware-Level Virtualization vs. Software Spatial Multiplexing

  • MIG provides isolated GPU instances with dedicated resources and predictable QoS, but static slices and reconfiguration overhead can hinder dynamic workloads.DRL-based reconfiguration and ElasticBatch's adaptive batch sizing are approaches described for addressing that rigidity.Source: §4.1.1 · p. 18
  • MPS increases compute density and avoids MIG reconfiguration, but shared-resource contention can cause unpredictable tail latency without intelligent kernel regulation.Concurrent kernels contend for L2 caches, memory controllers, and streaming multiprocessors.Source: §4.1.1 · pp. 18–19

4.1.2 Spatio-Temporal Co-Execution and Memory Isolation

  • AntMan throttles lower-priority jobs' memory allocations and kernel launches to protect latency-critical tasks; PipeSwitch overlaps PCIe transfers with GPU computation to reduce model-switching latency from seconds to milliseconds.AlpaServe uses statistical multiplexing of model-parallel slices to improve aggregate cluster throughput while meeting latency budgets.Source: §4.1.2 · pp. 19–20

4.1.3 Serverless GPU Sharing Frameworks

  • Serverless GPU-sharing systems co-locate inference tasks, adapt memory allocation to volatile loads, and group compatible LLM tasks to reduce active power consumption.SMore opportunistically co-locates serverless inference with training jobs while respecting primary-job deadlines.Source: §4.1.3 · p. 20

4.2 GPU Resource Scheduling

LLM cluster scheduling must account for volatile, non-stationary requests and unpredictable generation lengths while balancing latency objectives, fairness, throughput, cost, and heterogeneous infrastructure constraints.

4.2.1 Workload Characterization and Request Volatility in MLaaS

  • LLM request volatility, heavy-tailed prompt and output lengths, and nondeterministic generation create load imbalances that challenge schedulers designed for fixed execution durations.The section notes these patterns in analyses of large-scale MLaaS platforms.Source: §4.2.1 · p. 20

4.2.2 SLA-Aware and Multi-Metric Cluster Schedulers

  • SLA-aware schedulers use historical execution profiles and predicted future context lengths to prioritize requests, while graph prediction helps map jobs onto heterogeneous GPU topologies.The scheduling objective spans both TTFT and TBT.Source: §4.2.2 · p. 21
  • Other approaches balance fairness, memory pressure, energy use, and cluster load through scheduling metrics, dynamic batching, DVFS, load balancing, or multi-objective reinforcement learning.Fairness-aware metrics address starvation of short interactive requests by long-prompt requests.Source: §4.2.2 · p. 21

4.2.3 Heterogeneous, Preemptible Spot Instance, and Geo-Distributed Scheduling

  • Heterogeneous-cluster scheduling profiles accelerator capabilities, while spot-instance serving uses parallel state migration and lightweight KV-cache checkpointing to recover within seconds after preemption.Geo-distributed schedulers can route requests to renewable-powered regions and reuse model state and prompt caches.Source: §4.2.3 · pp. 21–22

4.2.4 Cloud-Edge Collaborative Resource Allocation

  • Cloud-edge allocation partitions execution or offloads requests according to bandwidth and energy constraints, and some frameworks use edge SLMs for initial tokens before cloud LLM reasoning.The surveyed approaches also include security protocols and parameter sharing intended to prevent proprietary prompt leakage during offloading.Source: §4.2.4 · p. 22

4.3 LLM Orchestration

LLM orchestration coordinates serving phases, memory state, adapters, and autoscaling. The section covers disaggregated prefill and decode, multi-tenant adapter and cache management, and serverless and Kubernetes-based operations.

4.3.1 Disaggregated Prefill-Decode Serving Architectures

  • Splitwise and DistServe report up to 10× improvements in goodput while meeting strict SLA thresholds through disaggregated prefill-decode serving.The architecture separates compute-bound prefill from memory-bound decoding, transferring KV caches between specialized GPU pools; Mooncake extends this with reusable prompt-context caching.Source: §4.3.1 · pp. 22–23, Figure 3 · p. 23

4.3.2 Multi-Tenant LoRA Adapter and KV-Cache Orchestration

  • Multi-tenant LoRA orchestration loads and co-executes adapters on shared GPUs, while compression and predictive prefetching reduce adapter storage or loading overhead.Semantic prompt-prefix sharing can conserve KV memory, but shared caches are vulnerable to prompt leakage and cache side-channel attacks, motivating secure isolation.Source: §4.3.2 · p. 24

4.3.3 Serverless and Cloud-Native Kubernetes Orchestration

  • Serverless LLM serving can scale from zero to thousands of instances, but loading tens of gigabytes of model weights into GPU memory drives cold-start latency.Approaches include tiered host memory and network bandwidth, pipelined loading, O(1) host caching, proactive warm-up, and fast tensor streaming.Source: §4.3.3 · p. 24
  • Serverless orchestration also offloads idle KV caches to host RAM and predicts GPU-pool scaling, while Kubernetes tools apply LLMs to failure diagnosis and workload forecasting.KIS-S provides a GPU-aware Kubernetes inference simulator for training reinforcement-learning autoscalers.Source: §4.3.3 · pp. 24–25

4.4 Comparative Summary of Orchestration and Scheduling Frameworks

Table 3 consolidates representative cloud LLM orchestration and scheduling frameworks across their technical domains, innovations, optimization metrics, transfer media, and deployment scenarios. Its examples include prefill/decode disaggregation, KV-cache-centered designs, runtime migration, and multi-tier checkpoint loading.

4.4 Comparative Summary of Orchestration and Scheduling Frameworks

  • Table 3 compares frameworks by technical domain, architectural innovation, optimization metrics, interconnect or transfer media, and deployment scenario.It is presented as a consolidated reference for researchers and system architects.Source: §4.4 · p. 25, Table 3 · p. 25
  • Other listed designs include carbon-aware routing with model-state reuse, edge-cloud sharding and offloading, and Lyapunov-assisted DRL for split decisions.These entries are Green-Geo, EdgeShard (Zhang et al. 2025), and Splitwise-DRL (Younesi et al. 2026), respectively.Source: §4.4 · p. 26
  • Splitwise uses dedicated prefill and decode nodes, while DistServe combines disaggregation with SLA-driven allocation.The table also identifies RDMA as a transfer medium in this comparison.Source: §4.4 · p. 26
  • High-Concurrency Cloud Mooncake uses a KV-cache-centric architecture that trades storage for compute.The table associates this entry with goodput and TTFT/TBT violations.Source: §4.4 · p. 26
  • Llumnix uses dynamic runtime request migration and live load balancing.The table lists cache hit rate and throughput as optimization metrics in this portion of the comparison.Source: §4.4 · p. 26
  • ServerlessLLM uses fast multi-tier checkpoint loading and tiering; the table also lists tiered memory management with predictive autoscaling.The table groups serverless cost and latency among its optimization metrics.Source: §4.4 · p. 26

5 Inference in Cloud

Cloud LLM inference requires coordinated decisions about scaling, batching, KV-cache reuse, and cost under changing workloads and resource constraints. The reviewed work develops approaches across these areas, while identifying gaps in joint optimization and secure, production-scale evaluation.

5 Inference in Cloud

  • Inference optimization spans replica and stateful scaling, memory- and compute-aware batching, KV-cache reuse and privacy, and multi-tier cost optimization.These four pillars interact, with cost shaping scaling, batching, and caching decisions.Source: §5 · p. 27, §5.4 · p. 34

5.1.3 Stateful and Disaggregation-Aware Scaling

  • Stateful LLM scaling must account for KV-cache-bearing state, migration, and session continuity rather than treating scale-out as adding stateless replicas.The review also notes that proactive systems do not jointly optimize scaling with batching and KV-cache placement.Source: §5.1.3 · p. 29, §5.1.4 · p. 30

5.1.2 Predictive and Proactive Scaling Systems

  • Predictive autoscaling studies report gains on different metrics and traces, but their results are not directly comparable and no standardized LLM-specific scaling benchmark exists.The surveyed approaches include forecasting systems and a PPO autoscaler trained in simulation; the review notes that real-cluster transfer remains open.Source: §5.1.2 · pp. 28–29, §5.1.4 · p. 30

5.2.1 Memory-Aware Batching via KV-Cache Paging

  • PagedAttention manages KV caches in fixed-size, non-contiguous blocks, enabling vLLM to achieve near-zero cache-memory waste and 2–4× higher throughput than FasterTransformer and Orca at matched latency.Iteration-level continuous batching was pioneered by Orca; PagedAttention addresses the memory fragmentation that constrained static allocation.Source: §5.2.1 · pp. 30–31

5.2.2 Compute-Aware and Heterogeneous Batching

  • Chunked prefilling and attention-level overlap address prefill/decode phase asymmetry, while BatchCond handles computational heterogeneity and BanaServe continuously rebalances cluster-wide work.Cross-boundary offloading studies report simulation gains in data efficiency and adaptability over mainstream DRL baselines.Source: §5.2.2 · p. 31, §5.2.3 · pp. 31–32

5.2.4 Toward Jointly-Aware Batching

  • A unified batching runtime that jointly handles memory layout, continuous scheduling, phase overlap, computational heterogeneity, placement, and live rebalancing remains an open problem.Most current frameworks optimize only one or two of these dimensions.Source: §5.2.4 · p. 32

5.3.3 Reuse vs. Leakage

  • KV-cache reuse methods share prompts or conversation state across requests, but the reviewed work shows that caches can expose private inputs through software and hardware attacks.The review finds that reuse systems generally are not evaluated against these attacks, and most proposed mitigations are not tested on production-scale multi-tenant stacks.Source: §5.3.1 · pp. 32–33, §5.3.2 · pp. 33–34, §5.3.3 · p. 34

5.4.4 Composability of Cost Levers

  • Cost approaches restructure serving, compress kernels or models, or exploit cheaper hardware markets; the review finds no unified accounting of their interactions.Examples include Mooncake’s increased request capacity, compression methods, and SpotServe’s cost savings on preemptible instances; the strongest deployments can combine layers.Source: §5.4.1 · pp. 34–35, §5.4.2 · pp. 35–36, §5.4.4 · p. 36

6.1 Serverless Architecture

Serverless LLM serving promises elastic, pay-per-use deployment, but stateless ephemeral functions conflict with large, stateful models. Reviewed systems address cold starts, GPU utilization, bursty autoscaling, adapter serving, and evaluation gaps through complementary techniques.

6.1 Serverless Architecture

  • LLM serverless platforms face four recurring challenges: checkpoint-loading cold starts, coarse GPU sharing, bursty-demand autoscaling, and a lack of standardized platforms and evaluation methods.The stateless, ephemeral FaaS model conflicts with the stateful, multi-gigabyte parameters of modern LLMs.Source: §6.1 · p. 37

6.1.2 GPU Sharing and Resource Multiplexing

  • SLINFER raises serving capacity by 47%–62% through sharing alone and 86%–154% when heterogeneous CPU resources are also exploited.FaST-GShare improves throughput by 3.15× and GPU utilization by 1.34× over naive time-sharing.Source: §6.1.2 · p. 39

6.1.3 End-to-End Serverless LLM Platforms and Live Autoscaling

  • End-to-end platforms combine serverless abstractions with live scaling: DEEPSERVE scales to 64 instances within seconds, while PipeCast cuts tail TTFT by up to 5× and cost by 31.3%.BlitzScale reports up to 94% lower tail latency than ServerlessLLM and 49% lower GPU-time consumption than DistServe and vLLM under equivalent SLAs.Source: §6.1.3 · p. 39

6.1.4 Adapter-Level and Multi-Tenant Serving

  • Predictive-LoRA reduces cold-start latency by up to 68% and achieves 1.52× higher throughput and 35% lower average TTFT than S-LoRA under high concurrency.Its prediction accuracy degrades under highly irregular or adversarial traffic.Source: §6.1.4 · p. 40

6.1.5 General-Purpose Serverless DNN Inference Systems

  • General-purpose DNN serverless systems provide architectural context: INFless delivers 2–5 throughput over general-purpose serverless baselines while meeting latency SLOs.AYCI automates configuration search, but its AWS-only prototype reports limited TVM compilation gains because hardware visibility is restricted.Source: §6.1.5 · p. 40

6.1.6 Surveys and Conceptual Frameworks

  • The literature includes a survey of serverless LLM inference and a three-layer conceptual architecture, but the latter lacks standardized evaluation metrics and empirical validation.The survey identifies adaptive batching, hybrid scheduling, model partitioning, and GPU sharing as dominant technique families.Source: §6.1.6 · p. 40, §6.1.7 · p. 42

6.1.1 Cold-Start Latency Mitigation

  • Cold-start mitigation combines memory staging, network-aware model placement, pipelining, and predictive prewarming, often within systems that address multiple bottlenecks.FlashServe reports up to 49× lower cold-start latency than S3-based checkpoint loading and 3.3× than ServerlessLLM, but its caching gains erode below 256 GB host DRAM.Source: §6.1.1 · p. 38, §6.1.7 · p. 41

6.1.7 Synthesis

  • Across reviewed systems, cold-start levers are frequently combined, while scaling is shifting from container-level toward layer- or token-level elasticity.Evaluation remains largely limited to text generation on single-provider traces; prediction-based prewarming can degrade under irregular or adversarial traffic, and cross-platform benchmarks remain lacking.Source: §6.1.7 · p. 41

6.2 Fault Tolerant Service Architectures

Cloud LLM dependability spans hardware faults, statistical uncertainty, semantic errors, and cluster incidents, while stateful KV caches make conventional stateless failover costly. The review presents a defense-in-depth approach across these layers and identifies cross-layer fault propagation as an open challenge.

6.2 Fault Tolerant Service Architectures

  • A four-tier defense-in-depth hierarchy combines hardware fault mitigation, statistical reliability and verification, semantic guardrails, and AIOps remediation because failures can arise at each layer.The layers are interrelated rather than alternative designs; protecting only one cannot guarantee operational dependability.Source: §6.2 · p. 43, Figure 5 · p. 44

6.2.1 Hardware Transient Faults and Silent Data Corruption Mitigation

  • FT2 selectively protects vulnerable transformer layers using prefill-phase first-token attention profiles and lightweight checksums, while other methods detect faults through attention-kernel checksums, activation distributions, or decoding anomalies.RetryTrigger suppresses up to 95.33% of SDCs with 4.12% average execution overhead; KV-cache checkpointing and tiered or CXL-backed storage address state recovery after instance failures.Source: §6.2.1 · pp. 45–46

6.2.2 Statistical Reliability Engineering and Formal Verification

  • HIP-LLM estimates failure-free execution probability over future request sequences with uncertainty bounds, while formal runtime monitors enforce deterministic safety boundaries.Static benchmark scores do not capture production drift or confidence intervals under non-stationary traffic, and both approaches remain sensitive to input noise.Source: §6.2.2 · p. 46

6.2.3 Semantic Fault Detection and Self-Correcting Guardrails

  • Semantic Entropy flags confabulations by measuring uncertainty across semantic-equivalence clusters, while Chain-of-Verification uses a multi-stage self-correction cycle.CoVe incurs token and latency overhead; RAGVA applies verification selectively to high-stakes document chunks.Source: §6.2.3 · p. 47

6.2.4 AIOps and Autonomous Cloud Self-Healing Loops

  • LLM-driven AIOps can automate diagnosis and incident response, but deterministic admission rules and verification policies constrain potentially unsafe remediation actions.The review identifies cross-layer fault propagation and coordinated recovery across all four tiers as an open research challenge.Source: §6.2.4 · pp. 47–48

6.3 Agentic Workflows and Multi-Agent Systems

Agentic workflows introduce persistent state, variable execution trees, and asynchronous tool interactions that challenge conventional cloud-serving infrastructure. The review covers collaboration patterns, scheduling and memory needs, and operational vulnerabilities requiring safeguards.

6.3 Agentic Workflows and Multi-Agent Systems

  • Agentic workflows iterate through reasoning, actions, observations, and reflection, with tool calls and dependent subtasks making execution dynamics unlike stateless request serving.Multi-agent systems also require persistent state, dynamic execution graphs, and asynchronous communication.Source: §6.3 · p. 48

6.3.1 Architectural Paradigms: Communicative Topologies and Workflow Decomposition

  • Agent designs range from sequential role-specialized chat chains to hierarchical and decentralized collectives, with workflow decomposition used for collaborative problem solving.ChatDev uses phased agent dialogues and local reflection to review intermediate artifacts; reviewed applications also include multi-objective design exploration and network management.Source: §6.3.1 · pp. 49–50

6.3.2 Cloud Scheduling, Resource Allocation, and Memory Fabrics

  • Variable execution trees and tool-waiting periods can leave statically bound GPUs underutilized, motivating dependency-aware scheduling and dynamic resource allocation.Agent collectives also need working, episodic, and semantic memory tiers, with checkpointing to restore state after failures without repeating prior LLM calls.Source: §6.3.2 · p. 51

6.3.3 Operational Vulnerabilities: Context Drift, Cascading Errors, and Systemic Bottlenecks

  • Agentic deployments face cascading hallucinations, context drift and token growth, and prompt-injection risks through tool access, calling for context controls, privilege limits, and deterministic runtime guards.Unbounded histories can degrade reasoning and increase latency and cost; untrusted tool data can also expose credentials or alter downstream configurations.Source: §6.3.3 · pp. 52–53

6.4 AI Gateways and Traffic Routing

LLM gateways must account for variable request lengths, prefill/decode asymmetry, KV-cache locality, and competing latency SLOs. The survey presents gateways as stateful inference control planes that coordinate cache-aware routing, workload dispatch, reliability, and security.

6.4 AI Gateways and Traffic Routing

  • LLM gateways face variable request durations, compute-bound prefill, sequential memory-bandwidth-bound decoding, prefix locality, and potentially conflicting TTFT and ITL thresholds.Requests routed to identical hardware can differ in processing duration by multiple orders of magnitude; matching a cached prefix can reduce TTFT from seconds to milliseconds.Source: §6.4 · pp. 53–54

6.4.1 Prefix-Aware Dispatching and KV-Cache Centric Routing

  • Cache-aware routing exploits prefix locality and KV reuse, while distributed cache transfer and VM selection balance latency against bandwidth and monetary cost.Mooncake routes to the longest prefix match and can stream cached tensors to an idle decoder; CacheGen compresses transfers, CacheBlend reuses up to 80% of cache under variable retrieval combinations, and VM selection trades leasing costs against bandwidth.Source: §6.4.1 · pp. 54–55

6.4.2 Heterogeneous Model Cascades and QoS-Aware Dispatching

  • Mixed-workload balancing predicts queueing delay and memory fragmentation to steer prefill-heavy requests away from workers generating long token streams, helping maintain consistent ITL.SAGESERVE also forecasts prompt volumes and sequence lengths to pre-warm instances before queue buildup threatens service agreements.Source: §6.4.2 · pp. 55–56

6.4.3 Cluster Service Reliability, Ingress Protection, and Network Security

  • Node dropouts, silent packet corruption, and accelerator faults motivate gateway health checks, circuit breaking, and retries that preserve user state.Ingress defenses also use SDN flow control and container telemetry to isolate compromised pods and redirect legitimate traffic.Source: §6.4.3 · p. 56
  • The gateway has evolved into a stateful inference control plane combining cache-centric routing, heterogeneous model cascades, workload-aware dispatch, reliability checks, and ingress defense.The synthesis identifies cache reuse, matching query complexity to model capacity, proactive warming, health monitoring, circuit breaking, and intrusion detection as complementary design principles.Source: §6.4.3 · pp. 56–57

6.5 Confidential Computing and Privacy-Preserving AI

The survey reviews confidential-computing approaches for protecting model inputs, weights, and inference across cloud environments. It highlights both the performance constraints of hardware enclaves and complementary methods such as collaborative computation, prompt sanitization, and cache isolation.

6.5 Confidential Computing and Privacy-Preserving AI

  • Confidential-computing designs address risks to enterprise data and model weights from infrastructure operators, co-tenants, and third-party cloud deployment, while TEEs introduce memory-capacity and encrypted-transfer bottlenecks.The survey covers hardware enclaves, collaborative approaches, and ingress sanitization as responses to these risks.Source: §6.5 · pp. 57–58

6.5.1 Hardware-Enforced TEEs and Accelerator Enclaves

  • For LLaMA-2 models up to 70B parameters, CPU TEEs with AMX offer viable, lower-cost latency for 7B models, while 70B models require GPU TEEs despite higher acquisition costs and PCIe encryption penalties.The experiments identify CPU memory-bandwidth starvation as the constraint at 70B scale.Source: §6.5.1 · p. 59
  • PipeLLM overlaps speculative encryption and tensor staging with GPU computation, achieving near-native throughput for models up to 175B parameters while retaining hardware-enforced isolation.TZ-LLM instead restores and decrypts model layers on the fly to protect edge weights within TrustZone's limited secure memory.Source: §6.5.1 · p. 59

6.5.2 Collaborative Inference, Enclave Fine-Tuning, and Prompt Sanitization

  • Shared prefix caches can expose private conversational context through timing probes, so cache optimizations must be paired with cryptographic access controls and timing defenses.The cited analysis reports that an adversarial tenant can infer whether another tenant queried particular documents or tokens.Source: §6.5.2 · p. 60
  • Collaborative PriPTune keeps sensitive samples, activations, gradients, and adapters in CPU TEEs while offloading operations on frozen base weights; Prϵϵmpt sanitizes sensitive prompt entities before cloud transmission.The synthesis also emphasizes that hardware isolation alone does not eliminate algorithmic or shared-KV-cache side channels, motivating tenant-level cache isolation and zero-trust policies.Source: §6.5.2 · pp. 60–61

7 Discussion

The discussion frames cloud LLM infrastructure around trade-offs among compute utilization, memory and state retention, elasticity, and tenant isolation. It identifies hardware, algorithmic, orchestration, sustainability, and evidence-synthesis directions while acknowledging limits in the survey's scope and available benchmarks.

7.1.1 Compute Saturation versus Memory-Bandwidth Bottlenecks

  • Disaggregating compute-bound prefill from memory-bound decoding can reduce head-of-line blocking, but transferring multi-gigabyte KV caches can make network latency approach or exceed execution savings.The trade-off is utilization and throughput versus the cost and physical capacity of interconnects needed for low-latency state migration.Source: §7.1.1 · p. 63

7.1.2 Stateful KV Cache Retention versus Serverless Elasticity

  • Serverless scale-to-zero conflicts with stateful model weights and growing KV caches: cold starts can take tens of seconds, while retaining warm states consumes memory and constrains elasticity.The resulting compromise is among avoiding cold starts, memory rental costs, and instantaneous scale-out agility.Source: §7.1.2 · p. 63

7.1.3 Hardware Sharing Density versus Multi-Tenant Isolation and Security

  • Aggressive accelerator sharing improves hardware saturation but risks tenant interference, tail-latency jitter, side-channel exposure, and silent corruption; TEEs add encryption overhead and complicate page sharing.The core tension is multiplexing efficiency versus provable tenant isolation.Source: §7.1.3 · pp. 63–64

7.2 Future Study Prospects for LLMs in Cloud

  • Future directions include CXL memory pooling and photonic interconnects, in-memory HDC/VSA, linear-complexity sequence models, autonomous multi-agent orchestration, carbon-aware scheduling, and verifiable confidential inference.These are proposed research frontiers rather than evaluated results in the review.Source: §7.2 · p. 64, §7.2.1 · p. 64, §7.2.2 · p. 65

7.3 Limitations of this Study

  • The review is bounded by its search cutoff and peer-reviewed focus, the gap between academic testbeds and proprietary hyperscaler environments, and its deliberate exclusion of pre-training and ultra-edge platforms.Academic studies commonly use clusters of 4 to 64 accelerators or synthetic workloads, while production complexities are rarely fully modeled.Source: §7.3 · p. 66, §7.3.1 · pp. 66–67, §7.3.2 · p. 67

7.3.4 Standardized Multi-Tenant Dynamic Benchmarking Deficits

  • Benchmark heterogeneity, sparse production traces, and the lack of standardized dynamic suites limit fair cross-framework comparison; future reviews should coordinate identical-hardware, trace-driven evaluations.Proposed benchmark coverage includes multi-turn, agentic, and multimodal workloads; longitudinal comparisons should consider throughput, energy, tail-latency compliance, and dollar cost.Source: §7.3.4 · p. 67, §7.4.2 · p. 68

7.4.1 Multimodal Foundation Model Orchestration and Serving Infrastructure

  • Future reviews should address multimodal serving because image, audio, video, and embodied workloads impose asymmetric memory and compute demands that challenge text-only assumptions.The proposed review questions include cross-modal attention caching and scheduling multi-stage encoder-decoder pipelines across heterogeneous accelerators.Source: §7.4 · p. 68, §7.4.1 · p. 68

7.4.2 Longitudinal Trace-Driven Meta-Analyses and Empirical Benchmarking Surveys

  • The review calls for longitudinal, trace-driven meta-analyses on identical hardware to compare Pareto frontiers across throughput, energy, tail latency, and cost.The proposed effort aims to bridge academic prototypes and industrial deployment evidence.Source: §7.4.2 · p. 68

8 Conclusion

The conclusion presents a PRISMA-guided synthesis of cloud LLM serving, orchestration, and architecture. It maps specialized, heterogeneous execution fabrics and the mechanisms reviewed for scheduling, inference efficiency, serverless deployment, reliability, and privacy.

8 Conclusion

  • The review synthesizes contemporary cloud LLM serving and orchestration research across four research questions under the PRISMA 2020 framework.It examines mechanisms for deploying, scaling, and sustaining generative foundation models at enterprise scale.Source: §8 · p. 68
  • The surveyed systems shift from homogeneous clusters toward specialized, heterogeneous, disaggregated fabrics, using accelerator sharing, predictive consolidation, phase-aware inference, and tiered memory techniques.For reliability and privacy, the conclusion highlights silent data corruption and side-channel prompt leakage as concerns addressed by fault tolerance and hardware-enforced TEEs.Source: §8 · p. 69
  • The proposed roadmap spans CXL pooling, photonic interconnects, linear-complexity architectures, in-memory computing, autonomous multi-agent control, and carbon-aware routing.The survey presents its trade-off framework as guidance for future cloud AI infrastructure design.Source: §8 · p. 69

Related papers

Loading 2610.33475v1…