PaperlayerView exact revision on arXiv

Source-linked AI summary

Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN

Tianhang Ding, Jianchun Liu, Hongli Xu

arXiv:2608.16477v1cs.LG

TL;DR

Cellular handover can separate an active LLM request from its growing KV cache, creating a trade-off between service continuity and serving locality. Pallas proactively prepares the cache at the predicted target before handover and substantially reduces service interruption time while avoiding persistent inter-token-latency overhead.

  • Problem

    Cellular handover separates mobile users from the source-resident KV cache needed to continue LLM inference, creating tension between service continuity and serving locality.

  • Method

    Pallas overlaps target-side prefix reconstruction and source-side suffix streaming before handover, using an online scheduler to select the prefetching window.

  • Results

    Pallas substantially reduces SIT against target-side recovery baselines while avoiding the persistent ITL overhead of source-side forwarding.

  • Takeaways & Limitations

    Preparing inference state before handover makes most migration work overlap with ongoing source-side inference and token delivery.

  • Takeaways & Limitations

    Reactive recovery baselines remain limited because their transfer and recomputation work begins after handover and remains exposed as SIT.

Abstract

from arXiv · show

AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-value (KV) cache remains at the source. Retaining inference at the source preserves service continuity but persistently increases inter-token latency (ITL), whereas recovering the state at the target restores serving locality but requires KV-cache transfer, recomputation, or a combination of both only after handover, directly prolonging service interruption time (SIT). This work presents Pallas, a \textit{proactive} KV-cache migration framework that prepares the inference state at the predicted target before handover, in parallel with ongoing source-side inference and token delivery. At the preparation trigger, Pallas partitions the token sequence into a stable historical prefix and an evolving suffix. The target reconstructs the prefix through local prefill, while the source streams the KV blocks generated for the suffix. At handover, the target assembles both portions into an up-to-date KV cache and resumes decoding locally, leaving only unfinished preparation to contribute to SIT. An online scheduler selects the \textit{prefetching window}, which determines how early preparation begins before handover, based on mobility predictions and runtime telemetry. Across three LLMs and $100$--$500~\mathrm{Mbps}$ inter-gNB links, our vLLM-based prototype reduces average SIT by factors of $2.28$--$89.68$ over target-side recovery approaches and lowers average ITL by $16.0\%$--$50.0\%$ compared with source-side forwarding.

1 Introduction

Pallas addresses the continuity–locality tension in mobile LLM handover by proactively preparing inference state at the predicted target gNB before handover. It reconstructs a stable prefix and streams an evolving suffix in parallel with source-side inference, so only unfinished preparation remains on the handover critical path.

  • Motivation: Mobility separates an ongoing LLM request from its growing KV cache when the UE moves from the source to the target gNB.This separation undermines AI-RAN’s latency advantage because the state needed for continued generation remains at the source.
  • Limitations of Existing Approaches: Existing strategies trade persistent detour ITL for post-handover KV-cache recovery delay, creating a continuity–locality tension.Detour retains inference at the source but adds routing hops, while full transfer places the entire migration delay after handover.
  • Pallas Framework: Pallas treats handover as a deadline for target-side preparation, using mobility predictions to begin state migration while the UE remains connected to the source.This reframes recovery from a post-handover action into proactive preparation before handover.
  • Pallas Framework: At the preparation trigger, Pallas splits tokens into a stable prefix and evolving suffix, reconstructs the prefix locally, and streams suffix KV blocks concurrently.The source continues decoding and delivering tokens while the target prepares both portions of the eventual cache.
  • Prefetching Window: The prefetching window controls preparation timing: a short window leaves unfinished work for SIT, whereas a long window increases suffix transfer volume.Selecting the window is nontrivial because the trigger jointly affects preparation completion and incremental cache size.

2 Background and Motivation

In AI-RAN, handover separates a mobile LLM session from its growing source-side KV cache, forcing a trade-off between persistent ITL overhead and target-side recovery delay. These limitations motivate proactive preparation at the predicted target, which reconstructs stable history, tracks evolving state, and selects a prefetching window to reduce residual interruption.

  • Handover challenge: AI-RAN colocates GPU inference with serving gNBs, but handover leaves the UE at the target while its accumulated KV cache remains at the source.The target consequently lacks the state needed to immediately continue decoding.
  • Existing approaches: Source-side retention avoids KV-cache recovery but forwards every generated token through the target, persistently lengthening the serving path and degrading ITL.Detour measures this overhead as forwarding hops increase.
  • Existing approaches: 7.17 s to 29.07 s: Full-Copy SIT grows with context from 1K to 4K tokens at 300 Mbps, because transfer begins only after handover.The target cannot resume decoding during the post-handover transfer interval.
  • Existing approaches: 581 ms to 2,103 ms: Recomputation SIT rises from 1K to 4K tokens, while ctHO still reaches 2,058 ms at 4K tokens despite combining prefill and suffix transfer.Recomputation shifts recovery cost to target-side prefill; ctHO balances recomputation and transfer through prefix–suffix partitioning.
  • Proactive opportunity: Proactive preparation uses mobility predictions to prepare the target before handover while the source continues inference and token delivery.The trigger partitions the state into a stable prefix reconstructed by target-side prefill and an evolving suffix whose KV blocks are incorporated continuously.
  • Prefetching-window trade-off: Approximately 77%: increasing the prefetching window from 0 to about 1 s reduces SIT, but idle time reaches approximately 2.4 s at T_w=4 s.The window must balance residual preparation on the handover critical path against unnecessarily early target-side preparation.

3 System Overview of Pallas

Pallas proactively prepares inference state during a prefetching window, parallelizing target-side prefix reconstruction and source-side suffix transfer with ongoing inference. At handover, the target assembles the complete KV cache and resumes local decoding, avoiding the fully post-handover delays of existing recovery strategies.

  • Online scheduling: Pallas’s Scheduler Engine combines mobility predictions, context length, and runtime estimates to make online prefetching decisions.The scheduler operates at each control cycle and determines when state preparation begins.
  • Parallel state preparation: During the prefetching window, target-side prefix reconstruction, source-side decoding, and suffix transfer proceed in parallel.This overlaps state preparation with ongoing source-side inference and token delivery.
  • Handover activation: At handover, the source sends the final sequence snapshot and unsynchronized KV blocks, while the target combines the reconstructed prefix and received suffix.After all sequence positions are available, the target activates the request for local decoding.
  • Local resumption: Pallas assembles an up-to-date KV cache at the target and resumes decoding locally once prefix reconstruction and suffix synchronization finish.Only preparation that remains unfinished at handover can delay activation, as implied by the parallel workflow and handover assembly process.
  • Comparison with recovery strategies: Full-Copy and Recomputation place complete KV transfer or historical-cache reconstruction on the post-handover critical path.Pallas differs by performing state preparation before handover during the prefetching window.

4 System Design of Pallas

Pallas proactively migrates inference state by reconstructing a stable prefix at the target while streaming the evolving suffix from the source, then selecting the preparation window online to balance interruption against premature resource commitment.

  • Prefetching-window objective: The prefetching window trades unfinished recovery work at handover against unnecessarily early preparation and longer target-side state residence.Historical-prefix reconstruction and incremental-suffix transmission proceed in parallel, so the slower unfinished operation determines migration-induced delay.
  • Online window selection: Pallas selects the window by combining mobility predictions with runtime telemetry, predicting the historical-prefix length, and evaluating candidate windows within Tlimit = min(Tmax, Tremain).The candidate search explicitly captures that later triggers yield longer prefixes, whereas earlier triggers yield longer incremental suffixes.
  • Admission control: Preparation is admitted only when the predicted target has a compatible LLM instance and sufficient VRAM; otherwise, Pallas invokes its fallback policy.Model compatibility and VRAM availability are treated as admission conditions before window optimization.
  • Online window selection: The bounded grid search evaluates O(Tlimit/ΔT) candidates with scalar arithmetic in microseconds, making scheduler overhead negligible relative to control periods and inference timescales.Pallas observes resource availability but does not jointly allocate GPU cycles or inter-gNB bandwidth among concurrent migrations.
  • Parallel state preparation: At the trigger, Pallas splits the request into an immutable historical prefix and evolving suffix, prefills the prefix at the target, and streams completed suffix KV blocks asynchronously.Source-side token generation and delivery continue during preparation, while the target buffers suffix blocks for later assembly.

5 System Implementation

Pallas is implemented on vLLM 0.8.5 with approximately 1.7K lines of Python code. Its migration control and KV-cache transfer use asynchronous coordination and gRPC while interfacing directly with vLLM’s inference-state components.

  • Pallas is implemented on vLLM 0.8.5 using approximately 1.7K lines of Python code.
  • A node daemon coordinates migration through asynchronous control requests and gRPC-based KV-cache transfer.
  • Pallas interfaces directly with vLLM’s scheduler, PagedAttention block manager, and GPU KV cache without modifying model kernels.
  • The implementation uses a PagedAttention block size of 16 tokens for both single-host and cross-host experiments.

6 Performance Evaluation

Pallas consistently lowers service interruption time while preserving lower inter-token latency across fixed and time-varying network conditions. Its scheduler balances preparation timing against exposure, and both prediction availability and the two migration components materially affect performance.

  • Single-Handover Performance: 70.13–89.68×, 2.35–7.28×, and 2.28–6.76×: Pallas reduces average SIT versus Full-Copy, Recomputation, and ctHO across fixed-goodput settings.At 100 Mbps with Qwen3-32B, average SIT is 321 ms.
  • Single-Handover Performance: 16.0%–50.0%: Pallas reduces average ITL versus Detour, maintaining 28–48 ms across fixed-goodput settings.Under measured time-varying goodput traces, Pallas achieves 204–316 ms average SIT and retains its SIT advantage over target-side recovery schemes.
  • Prefetching-Window Decision: 1.64 s: Pallas selects this prefetching window for 300 Mbps, Qwen3-14B, and a 3,000-token context, yielding 227.57 ms SIT and 0.017 s early-preparation exposure.Across nine configurations, selected windows keep SIT and early-preparation exposure near their low values.
  • Multi-User Performance: Below 300 ms: Pallas maintains average SIT across all evaluated HH, HI, and joint configurations, while contention primarily affects worst-user SIT and resident tail ITL.Joint multi-user resource allocation remains outside the scope of the work.
  • Ablation: 97.93% and 33.3%: Pallas reduces SIT relative to variants without recomputation and without incremental streaming, respectively.At 300 Mbps with Qwen3-14B, SIT is 235 ms for Pallas, 12,486 ms without recomputation, and 655 ms without streaming.

7 Related Work

Prior MEC migration methods treat service state as relatively static, while AI-RAN LLM serving requires mobility support for a growing KV cache. Unlike ctHO, which begins recovery after handover, Pallas prepares the target beforehand and adaptively limits post-handover work.

  • MEC Service Migration: Prior MEC systems use decision-making, learning, mobility prediction, and collaborative optimization to determine when and where relatively static services should migrate.Active LLM requests instead maintain a KV cache that grows throughout decoding.
  • AI-RAN Mobility Support: AI-RAN motivates mobility support for edge AI workloads; AoRA colocates AI inference with RAN functions, and the AI-RAN working group identifies proactive mobility management as important.
  • Closest Related Work: ctHO combines target-side token prefill with direct KV-cache transfer and jointly optimizes their partition and backhaul allocation, but both recovery operations begin after handover.
  • Pallas Distinction: Pallas overlaps target-side historical-prefix reconstruction and incremental KV-cache streaming with source-side inference before handover, while adaptively selecting the prefetching window.This reduces the work remaining on the post-handover interruption path.

8 Conclusion

Pallas is a proactive, parallel KV-cache migration framework for mobile LLM serving in AI-RAN that moves most state preparation before handover. By overlapping target-side recomputation and incremental KV-cache streaming with source-side inference, it reduces SIT without the persistent ITL penalty of Detour.

  • 8 Conclusion: Pallas proactively migrates KV-cache state for mobile LLM serving in AI-RAN before handover.The framework performs migration in parallel rather than waiting until handover.
  • 8 Conclusion: Overlapping target-side recomputation and incremental KV-cache streaming with ongoing source-side inference moves most state-preparation work before handover.
  • 8 Conclusion: The vLLM-based prototype and trace-driven evaluation show that Pallas substantially reduces SIT versus target-side recovery baselines.
  • 8 Conclusion: Pallas avoids the persistent ITL penalty associated with Detour.
Loading 2608.16477v1…