PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management

Jiaxun Lu, Xiang Zhang, Yunfeng Shao

arXiv:2609.06940v1cs.DC

TL;DR

Model choice and cache rebuilding create integration and inference costs when requests span models with different capabilities, prices, and providers. The Unified AI Gateway jointly coordinates routing, cache management, and compute placement across end, edge, and cloud resources. Its workload-level analytical simulation reports modeled TTFT speedups of 1.25×–13.28× and input-cost benefits of 1.20×–6.16×.

  • Problem

    When a request switches models, its accumulated KV cache may be unavailable or incompatible, requiring full-context prefill that can outweigh the benefits of switching.

    Source: §1 · p. 1

  • Method

    The Unified AI Gateway jointly selects models and cache actions across the End–Edge–Cloud continuum under latency, cost, quality, and resource constraints.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    1.25×–13.28× modeled TTFT speedups and 1.20×–6.16× input-cost benefits are reported across eight workload profiles.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Modeled benefits grow with reusable-cache coverage and switching frequency, provided cache transfer avoids target-side full prefill.

    Source: §8 · p. 13

    Distributed cache reuse requires co-designing consistency, mobility, placement, and routing so migration and metadata maintenance do not erase reuse gains.

    Source: §7.3 · p. 12

Abstract

from arXiv · show

Large language model (LLM) inference increasingly spans models that differ in size, capability, price, and provider. This shift creates two costs for developers. One is the integration cost of choosing among and switching between many models. The other is the inference cost of rebuilding a KV cache when it is unavailable or incompatible with the selected model. We define and analyze the Unified AI Gateway as a system setting for an edge-deployed AI traffic hub. It coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services. At request time, the gateway jointly selects a target model, an execution site, and a KV cache action under task-quality, latency, cost, and resource constraints. In parallel, background cache-management actions optimize KV cache placement, replication, retrieval, and lifecycle decisions for subsequent requests. We synthesize existing evidence on KV cache reuse, compression, cross-model mapping, distributed storage, and transfer, and discuss the remaining challenges of integrating these capabilities into one system. Across eight typical workload profiles, our workload-level analytical simulation reports TTFT speedups of 1.25$\times$--13.28$\times$ and input-cost benefits of 1.20$\times$--6.16$\times$.

1 INTRODUCTION

Model switching can make accumulated KV caches unavailable or incompatible, forcing full-context prefill that may outweigh the advantages of changing models. The Unified AI Gateway coordinates routing and cache management, with modeled benefits across eight workload profiles.

1 INTRODUCTION

  • Existing gateways and cache systems provide fragmented capabilities, but do not jointly coordinate cache handling with provider and model selection across devices and services.A model switch can make accumulated cache unavailable or incompatible, forcing full-context prefill and potentially increasing TTFT and input cost.Source: §1 · p. 1
  • 1.25×–13.28× modeled TTFT speedups and 1.20×–6.16× input-cost benefits are reported across eight workload profiles.The long-horizon coding-agent workload has TTFT speedups of 5.01×–12.87×.Source: Figure 1 · p. 1, §1 · pp. 1–2
  • The framework organizes cache management into cross-granularity reuse, adaptive compression, cross-model mapping, and distributed storage and transfer.The paper identifies compatibility and fidelity, transfer versus recomputation, and cache placement, replication, and isolation under capacity and privacy constraints as system challenges.Source: §1 · p. 2
  • The Unified AI Gateway coordinates provider and model routing with KV cache handling and compute placement across the End–Edge–Cloud continuum.Its Global Control Plane jointly selects models and cache actions under latency, cost, quality, and resource constraints.Source: §1 · p. 2

2 PRELIMINARIES AND SYSTEM SETTING

The section introduces KV caches and reuse, defines the Unified AI Gateway, and describes its End–Edge–Cloud architecture. The edge-hosted control plane coordinates model routing and cache actions across participating devices and services.

2.1 Preliminaries

  • KV caches store attention tensors so decoding can avoid recomputing full context; their size grows with context length, making storage, reuse, placement, and transfer consequential to inference.The section also describes exact-prefix, semantic, non-prefix, workflow-level, and cross-model reuse, noting that broader coverage introduces matching, calibration, correction, and fidelity costs.Source: §2.1 · p. 2

2.2 Unified AI Gateway Definition

  • The Unified AI Gateway coordinates request routing and KV cache management on the same request path, managing model-specific caches and selecting a model, execution site, and cache action.Available actions include direct reuse, cross-model mapping, compression, transfer, edge or remote prefill, and target-side re-prefill.Source: §2.2 · pp. 2–3

2.3 End–Edge–Cloud Architecture

  • The gateway uses cache metadata, resource status, network conditions, and service requirements to filter feasible actions and estimate their end-to-end cost.A valid cache can be reused or mapped; otherwise, the gateway can schedule prefill and deliver the resulting cache to the selected decode node.Source: §2.3 · pp. 3–4
  • The End–Edge–Cloud architecture places the Global Control Plane at the edge, where it coordinates model routing, cache lookup, and cache management across end devices, edge devices, and cloud services.Edge devices provide inference, prefill, and cross-model mapping; requests, results, and KV caches can flow bidirectionally among layers.Source: §2.3 · p. 3, Figure 3 · p. 4

3 GLOBAL CONTROL PLANE AND KV-CACHE WORKFLOW

The Global Control Plane jointly coordinates model choice, execution placement, and KV-cache preparation, while background management shapes the caches available to later requests. The workflow compares cache reuse, mapping, transfer, and prefill paths under end-to-end constraints.

3.1 Joint Routing, Compute Placement, and KV Cache Management

  • Figure 4 organizes five candidate paths—cache hit, cross-model mapping, remote transfer, edge prefill, and target-side re-prefill—by end-to-end cost, fidelity, and resource constraints.The paths are compared for each candidate model and execution site.Source: §3.1 · p. 4, Figure 4 · p. 5
  • The request-time decision jointly selects a model, execution site, and cache action from feasible options, balancing normalized latency and cost subject to request quality and system constraints.Background placement, replication, migration, and eviction determine which caches are available to later requests.Source: §3.1 · pp. 4–5

3.2 KV Cache Reuse and Cross-Model Mapping

  • Cache reuse avoids repeated prefill, while cross-model mapping can preserve context across model switches at the cost of transformation and fidelity constraints.Mapped caches are used only when quality loss stays within the allowed limit; otherwise compatible caches may be transferred or the target context recomputed.Source: §3.2 · p. 5, §3.1 · p. 5

3.3 Edge-Side Prefill

  • Edge devices can prefill locally when the required model and resources are available, then deliver the cache to an edge, cloud, or terminal-side decode service.A generated cache can enter the distributed pool for later reuse, mapping, compression, or transfer.Source: §3.3 · pp. 5–6

3.4 KV Cache Transfer, Compression, and Retrieval

  • Cache delivery compares transfer with target-side recomputation, accounting for bandwidth, RTT, queueing, copying, and restoration overhead.Compression reduces transfer volume; retrieval and placement determine whether a cache is available within latency, fidelity, and authorization constraints.Source: §3.1 · p. 5, §3.4 · p. 6

4 COMPONENT RESPONSIBILITIES AND REQUIRE-

The section assigns gateway responsibilities to routing, reuse, compression and transfer, mapping, and distributed storage. These components share constraints, especially transfer overhead and inference-engine memory management.

4.1 Joint Model and KV-Cache Routing

  • Routing must jointly choose the model, execution site, and cache preparation action using quality, cost, latency, access, locality, compatibility, and fidelity requirements.Online feedback can trigger a new choice when lookup fails, mapping violates quality limits, or transfer becomes slower than re-prefill.Source: §4.1 · p. 6

4.2 Cross-Granularity KV Cache Reuse

  • Reuse management spans exact-prefix lookup, semantic matching, non-prefix segments, multi-agent offset adaptation, dynamic eviction, and layer-aware indexing for hybrid-attention models.These requirements address multi-model, multisession, and multi-agent requests.Source: §4.2 · p. 6

4.3 KV Cache Compression and Transfer

  • Compression and transfer choices balance latency, storage occupancy, fidelity, and computation overhead, adapting quantization or low-rank methods to cache structure and changing link conditions.Inputs include bandwidth, RTT, and storage headroom.Source: §4.3 · p. 6

4.4 Cross-Model KV Cache Mapping

  • Cross-model translation must preserve usable context despite differences in layers, heads, dimensions, and positional encoding, with same-architecture, cross-architecture, and zero- or few-shot mapping support.Training a dedicated mapper for every model pair is impractical.Source: §4.4 · p. 6

4.5 Distributed KV Cache Management

  • Distributed storage must keep caches discoverable, authorized, and near decode destinations through geography- and demand-aware placement, efficient retrieval, bandwidth-aware transfer, and consistency maintenance.The requirements include model updates and cache invalidation.Source: §4.5 · p. 6

4.6 Control-Plane Coordination

  • Transfer couples all four component responsibilities to engine memory management: non-contiguous cache blocks create scatter-gather overhead, while compaction consumes scarce memory bandwidth.Shared wide-area links let large cache objects block smaller requests; coordination therefore needs page-aware, copy-efficient transfer and scheduling across transfer, storage, and inference.Source: §4.6 · pp. 6–7

5 EVIDENCE SYNTHESIS AND SYSTEM PROGRESS

The synthesis reviews evidence across cache reuse, compression, cross-model mapping, distributed storage and transfer, and model routing. Reported gains vary with workload and system conditions, while existing methods generally optimize individual stages rather than gateway-wide coordination.

5.1 KV Cache Reuse

  • Exact-prefix systems report 2–4× throughput at the same latency for vLLM and up to 6.4× higher throughput for SGLang, but require exact prompt-prefix matches.KVFlow reported up to 1.83× and 2.19× speedups for single and concurrent workflows.Source: §5.1 · p. 7
  • Non-prefix reuse reported 2.2–3.3× lower TTFT for CacheBlend, while semantic reuse reported a 6.25× speedup for SemShareKV on 5K-token summarization inputs.Multi-agent results include over 70% reuse and up to 7.8× prefill speedup for KVCOMM, and over 80% reuse for RelayCaching [35].Source: §5.1 · p. 7

5.2 KV Cache Compression

  • Compression spans quantization, low-rank, and mixed methods: RotateKV reported a 3.97× peak-memory reduction, while STAR-KV reported up to 20× overall compression [31].The chosen ratio must still respect fidelity and computation constraints.Source: §5.2 · p. 7

5.3 Cross-Model KV Cache Mapping

  • Cross-model mapping includes same-architecture cache sharing, [40]– and cross-architecture transformations, with C2C reporting an 8.5–10.5% average accuracy improvement on policy-semantics tasks.The gateway treats mapping as one cache action alongside model selection, placement, transfer, prefill, and re-prefill.Source: §5.3 · pp. 7–8
  • Mapping can run 2.7–25× faster than re-prefill and provide up to 2.65× TTFT speedup when it satisfies the quality limit.Its value depends on whether delivery and transformation cost less than recomputation; if mapping fails the quality criterion, cache-format compatibility determines whether direct transfer is possible.Source: §5.3 · p. 8

5.4 Distributed KV Cache Storage and Transfer

  • LMCache with vLLM reported up to 15× higher throughput and at least 2× lower latency on multi-turn workloads [5], while ObjectCache added 5.6% TTFT for a 64K-token context over local DRAM.The ObjectCache result used a 100 Gbps RoCE cluster; multi-tier storage trades capacity against access latency.Source: §5.4 · p. 8
  • Transfer systems reported up to 96% lower average KV transfer latency for FlowKV [51] and up to 9.13× lower job-completion time for KVServe.Realized gains depend on bandwidth, cache locality, and scheduling.Source: §5.4 · p. 8

5.5 Model Routing

  • Model-routing studies report cost savings above 50% and throughput gains up to 4.25×, but do not measure additional benefits from gateway-level cross-model mapping and cross-tier placement.Existing approaches coordinate resources inside datacenter serving clusters.Source: §5.5 · pp. 8–9

6 WORKLOAD-LEVEL BENEFIT ANALYSIS

The workload-level analysis models cache preparation and token billing across eight profiles under varying model-switching rates. It reports larger modeled TTFT and input-cost benefits as switching rates rise, with gains shaped by workload characteristics.

6.1 Workload Profiles and Settings

  • Eight profiles vary input/output lengths, effective cache-hit rate, switching rate, and bandwidth, including a coding-agent trace with an approximately 80K-token turn-30 context.The study evaluates switching rates of 25%, 50%, and 75%, with 8 GB/s as the reference cross-domain bandwidth.Source: §6.1 · p. 9

6.2 KV-Cache Model and Metrics

  • Same-model hits reuse cache directly, switching hits use mapping and transfer, and misses trigger target-side full prefill.Preparation speedup is the primary TTFT proxy; end-to-end speedup is a secondary diagnostic that includes output generation.Source: §6.2 · p. 10
  • The input-cost calculation assigns cached-input pricing to successful hits, including mapped switching hits, while misses use uncached pricing.The reference setting uses a 0.10 cached-to-uncached input-price ratio; background placement, replication, eviction, and migration costs are amortized and omitted from the per-request critical path.Source: §6.2 · pp. 9–10

6.3 Simulation Results and Sensitivity

  • At 25%, 50%, and 75% switching, TTFT speedups span 1.25×–5.39×, 1.49×–9.48×, and 1.74×–13.28×, respectively.Table 4 compares the modeled workload benefits across the three switching settings.Source: §6.3 · p. 10, Table 4 · p. 11
  • Input-cost ratios span 1.20×–2.72×, 1.41×–4.44×, and 1.61×–6.16× at 25%, 50%, and 75% switching, respectively.The E2E cost ratios for those settings span 1.17×–2.35×, 1.35×–3.69×, and 1.52×–5.04×.Source: §6.3 · p. 11, Table 4 · p. 11
  • For the long-horizon coding-agent profile, TTFT speedup rises from 5.01× to 12.87× as switching increases from 25% to 75%.Over the same settings, its input-cost ratio rises from 2.39× to 5.18× and its E2E cost ratio from 2.11× to 4.34×.Source: §6.3 · p. 11
  • High-hit, long-context workloads benefit most because successful cache actions avoid more prefill work and a larger uncached-input charge.Higher cache availability and switching rates produce larger ratios, while bandwidth and context length determine the attainable range.Source: §6.3 · p. 11

7 DEPLOYMENT READINESS AND RESEARCH DI-

Deployment readiness remains incomplete because gateway capabilities are fragmented, and cross-model cache mapping still faces fidelity and generalization challenges.

RECTIONS

  • Gateway deployment still requires closing gaps in the integration of current systems and assessing deployment readiness.Current support is distributed across gateway services, inference engines, and cache systems.Source: RECTIONS · p. 11

7.1 Cross-Model Semantic Alignment

  • Cross-model mappers face quality, training, or model-family constraints, while layer-wise errors can accumulate through attention and feedforward blocks.The paper calls for mapping error bounds, low-sample or training-free adaptation, and benchmarks spanning models, tasks, and languages.Source: §7.1 · p. 11

7.2 Heterogeneous Edge Hardware

Heterogeneous edge hardware limits direct transfer of GPU-oriented cache operators and layouts, while distributed placement makes consistency, mobility, and routing a joint performance concern. The section also frames TTFT speedup across bandwidth, sequence length, cache-hit rate, and model-switching rate.

7.2 Heterogeneous Edge Hardware

  • Edge hardware differences mean GPU-oriented operators and layouts cannot be transferred directly to NPU, FPGA, or ASIC platforms.Datacenter methods also assume HBM and large batches, while cache residence, eviction, and reload are more costly on edge devices.Source: §7.2 · p. 11

7.3 Distributed KV Cache Consistency

  • Stale cache metadata usually causes a miss and repeated prefill rather than an incorrect result.This motivates asynchronous refresh and weak consistency.Source: §7.3 · p. 11
  • Fig. 5 plots TTFT speedup against bandwidth and sequence length in Panel (a), and cache-hit rate and model-switching rate in Panel (b), normalized to a 1.0 equal-latency baseline.Panel (a) fixes h = 0.5 and Panel (b) fixes bandwidth at 64 Gbps.Source: Figure 5 · p. 12
  • The gateway must co-design consistency, mobility, placement, and routing so migration and metadata maintenance do not erase reuse gains.TraCT, SwiftCache, and Prefill-as-a-Service illustrate approaches to shared access, ownership reassignment, and separated cache generation and consumption [50], [76], [79].Source: §7.3 · p. 12

7.4 Security and Privacy

The section identifies security and privacy risks from storing and sharing KV caches, alongside a system-economics challenge spanning compute, storage, and network costs. It calls for controls that protect user data and economic models that account for changing prices and bottlenecks.

7.4 Security and Privacy

  • KV-cache storage and sharing create reconstruction and timing-channel risks across prefix hits, non-prefix fusion, and direct tensor access,, [82],.The gateway therefore requires authorization, user-data separation, auditing, timing-signal protection, encryption, and data-residency enforcement.Source: §7.4 · p. 12
  • Selective isolation can retain part of the KV-cache reuse benefit [84].This is the security-relevant trade-off identified alongside the listed cache-sharing risks.Source: §7.4 · p. 12

7.5 System Economics

  • Caching, compression, mapping, and migration change compute, storage, and network costs as prices and bottlenecks vary across locations and time.The section calls for gateway objectives covering hit rate, context length, cache lifecycle, bandwidth price, service quality, energy, and break-even conditions [50], [85], [86],.Source: §7.5 · p. 12

7.6 Standards and Ecosystem Support

KV-cache interoperability remains less mature than unified APIs, with differences among engines and gaps in shared interfaces and benchmarks. Transfer efficiency also depends on memory layout and compression, motivating page-aware primitives and SLO-aware scheduling.

7.6 Standards and Ecosystem Support

  • Engines differ in page size, cache layout, precision, and device affinity, making point-to-point connectors costly as model and engine versions grow.A common KV-cache data model, lifecycle and transfer events, reference implementations, and cross-vendor benchmarks are identified as needed progress.Source: §7.6 · p. 12
  • SCBench covers several cache operations but not cross-model mapping, cross-node transfer, or consistency semantics.LMCache and emerging proposals provide common control events and cache descriptions, while compression, mapping, and transfer interfaces remain open [5].Source: §7.6 · p. 12

7.7 KV Cache Transfer and Memory Management

  • PagedAttention’s non-contiguous blocks can reduce PCIe transfer efficiency, while compaction adds copies during decode.The gateway therefore needs page-aware transfer primitives and SLO-aware schedulers that jointly select reuse, compression, priority, and placement; reload, bandwidth-aware compression, streaming, and shared memory must be considered together,, [76], [79].Source: §7.7 · p. 12

7.8 KV Cache Portability and Architectural Evolution

Model design and KV-cache movement are still optimized separately, leaving model-level portability objectives less developed than token and system optimizations. The paper suggests future models could incorporate cache-friendly structures and that gateway coordination could extend beyond KV caches.

7.8 KV Cache Portability and Architectural Evolution

  • Model design and KV-cache movement are still optimized separately, with model-level portability objectives less developed than token and system optimizations [31],,,.Future models could incorporate low-rank cache structure, sparse attention, or compression regularization to make cache movement more efficient.Source: §7.8 · p. 12
  • Potential model-side directions for more efficient cache movement include low-rank cache structure, sparse attention, and compression regularization.These are proposed future possibilities rather than current integrated solutions.Source: §7.8 · p. 12
  • The same coordination principle may extend to transferable inference representations in recurrent, non-autoregressive, or diffusion architectures,,,.The paper's focus here remains KV caches.Source: §7.8 · pp. 12–13

8 CONCLUSION

The paper defines the Unified AI Gateway setting and formalizes joint model-routing and KV-cache decisions. Its workload-level analytical simulation reports TTFT and input-cost benefits, while identifying end-to-end integration as the remaining challenge.

8 CONCLUSION

  • Across eight typical workload profiles and model-switching rates from 25% to 75%, modeled TTFT speedups are 1.25×–13.28× and input-cost benefits are 1.20×–6.16×.The reported benefits grow with reusable-cache coverage and switching frequency when cache transfer avoids target-side full prefill.Source: §8 · p. 13
  • Modeled benefits grow with reusable-cache coverage and switching frequency, provided KV-cache transfer avoids target-side full prefill.Source: §8 · p. 13
  • The paper defines the Unified AI Gateway setting, formalizes joint model-routing and KV-cache decisions, and connects End–Edge–Cloud architecture to component responsibilities and requirements.Source: §8 · p. 13
  • Realizing the setting remains an end-to-end systems integration problem involving cache fidelity, distributed resources, and trustworthy coordination.Source: §8 · p. 13
Loading 2609.06940v1…