PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Context Parallelism for Production LLM Serving

Jiarui Guo, Rongle Wang, Peijun Huang, Zongwei Lv, Ziqing Wang, Kan Liu, Tao Lan, Lin Qu, Xiaolin Wang, Tong Yang

arXiv:2609.04774v1cs.OS

TL;DR

LLM serving must handle heterogeneous, dynamic, and prefix-intensive workloads, while existing CP systems commonly rely on static configurations. Vertumnus adaptively schedules requests across CP degrees, reconfigures worker composition, and manages prefix caches, reducing TTFT and improving SLO attainment under evaluated loads.

  • Problem

    Heterogeneous request lengths, changing workloads, and prevalent prefix reuse challenge systems that do not model CP degree jointly with load and cache locality.

  • Method

    Vertumnus combines cost-based request routing across CP degrees, seconds-scale worker split and merge operations, and global prefix-cache placement and replication.

  • Results

    Vertumnus reduces mean TTFT by up to 28.1% and improves token-weighted TTFT SLO attainment by up to 13.3 percentage points over the strongest baseline.

  • Takeaways & Limitations

    Adaptive coordination of request placement, worker composition, and prefix caching improves serving capacity and SLO performance under dynamic workloads.

  • Takeaways & Limitations

    Replay-based cache statistics may understate online prefix reuse because sampled requests omit some cross-request prefix relationships.

Abstract

from arXiv · show

As LLM context windows expand and input sequences grow longer, serving systems face increasing computational and memory demands. Context parallelism (CP), which partitions the input sequence across multiple ranks to parallelize the computation, has therefore become increasingly important for efficient LLM serving. However, existing CP-enabled systems either rely on static CP configurations or adjust the CP degree only for active requests or batches. In this paper, we present Vertumnus, an adaptive CP serving system designed for heterogeneous and evolving workloads. At the request level, Vertumnus routes requests among workers with different CP degrees using a placement cost that combines predicted queuing delay, cache-aware prefill time, and GPU-time cost. At the cluster level, Vertumnus adapts the worker composition through seconds-scale split and merge operations as workload demand changes. Vertumnus further introduces a global prefix-cache management policy that coordinates cache placement and replication among workers with the same or different CP degrees, preserving cache locality as request assignments and worker composition change. Experiments on a 64-GPU cluster with public and production workloads show that, under the highest evaluated loads, Vertumnus reduces mean TTFT by up to 28.1% and improves token-weighted SLO attainment by up to 13.3 percentage points over the strongest baseline.

1 INTRODUCTION

Production LLM workloads combine heterogeneous request lengths, changing demand, and increasing prefix reuse, making fixed serving configurations inadequate. Vertumnus addresses these challenges with adaptive context-parallel scheduling, dynamic worker reconfiguration, and coordinated prefix-cache management.

  • Workload Challenges: Production workloads vary in request length, arrival rate, request composition, and prefix reuse.Requests can range from hundreds to tens of thousands of tokens, while workload conditions change over time and repeated prefixes are increasingly common.
  • Context Parallelism: Context parallelism partitions input tokens and attention computation across GPUs, reducing per-GPU memory pressure and potentially lowering TTFT for long requests.The approach adds cross-GPU communication, so its benefits depend on workload and configuration.
  • Research Gap: Existing systems often omit CP from scheduling or statically provision workers, limiting adaptation to workload evolution and worker-specific cache locality.Length-only routing can also ignore current load and prefix reuse.
  • Vertumnus: Vertumnus routes requests using predicted queuing delay, cache-aware prefill time, CP-dependent performance, and GPU-time cost.The scheduler selects among persistent workers with different CP degrees.
  • Vertumnus: Vertumnus dynamically splits and merges workers within seconds while coordinating prefix placement and replication across CP degrees.These mechanisms adapt cluster capacity and preserve cache locality as demand and assignments change.
  • Results: Vertumnus reduces mean TTFT by up to 28.1% and improves token-weighted TTFT SLO attainment by up to 13.3 percentage points.The reported results use public and production workloads.

2 BACKGROUND AND MOTIVATION

Context parallelism distributes long-sequence attention across GPUs, but its benefit depends on input length, prefix reuse, and workload dynamics. These factors motivate adaptive scheduling and worker composition for production serving.

  • Context Parallelism: Zigzag partitioning assigns paired early and late sequence chunks to CP ranks to balance their unequal attention workloads.Each rank computes local representations and uses AllGather to obtain key and value representations across ranks.
  • Context Parallelism: Context parallelism partitions input tokens and attention across GPUs, reducing per-GPU memory pressure and potentially lowering prefill latency at added communication cost.Its benefit is strongest for long inputs that can amortize communication overhead.
  • Prefix Caching: Prefix caching reuses cached KV states, leaving only the uncached suffix to undergo projection, feed-forward, and attention computation.As the cached prefix grows, less computation remains for CP while communication and synchronization costs may not decrease proportionally.
  • Workload Dynamics: Long requests comprise approximately 4% to 8% of arrivals but contribute about 32% to 54% of daily input tokens.Their greater token volume makes request count an incomplete measure of aggregate prefill demand.
  • Workload Dynamics: Arrival rates vary by more than threefold, so a worker composition effective during light load can constrain serving capacity during peaks.Larger CP degrees can reduce independent serving lanes when concurrency pressure is high.
  • Workload Dynamics: Prefix-cache hit rates vary between approximately 35% and 55%, while cache contents remain worker-specific and may require transfer or reconciliation after reconfiguration.Request placement and worker reconfiguration must therefore account for current cache state.

3 PROBLEM DEFINITION

Vertumnus models a fixed-GPU prefill pool as persistent workers with supported CP degrees, accounting for prefix reuse when estimating prefill time and TTFT. Its service objectives are mean TTFT and token-weighted TTFT SLO attainment, while decode optimization remains outside scope.

  • Each persistent CP worker has a fixed supported CP degree, occupies that many GPUs, and serves as the unit of request execution and KV-state placement.
  • The scheduler models each request’s reusable prefix, uncached suffix, and CP-dependent prefill execution time before estimating TTFT.TTFT also includes latency outside the prefill pool, including KV transfer and first-token generation.
  • Larger prefix hits reduce remaining computation, while increasing CP accelerates only computation distributed across CP ranks, making CP benefits depend on remaining work.The model reports small fitting errors across profiled CP degrees, input lengths, and prefix-cache hit ratios for Qwen3-30B-A3B.
  • Vertumnus primarily minimizes mean TTFT and also maximizes token-weighted TTFT SLO attainment, weighting requests by input length.The weighting measures the fraction of input-token demand contributed by requests meeting their TTFT requirements.
  • Decode-side optimization is outside scope because the decode pool is independently managed to meet TPOT requirements.Improved prefill efficiency may support higher request rates when sufficient decode capacity is available.

4 SYSTEM OVERVIEW

Vertumnus adapts CP at request and cluster timescales within a fixed-budget prefill pool. It routes requests across heterogeneous workers, reconfigures worker composition, and coordinates prefix-cache placement as workloads change.

  • Vertumnus operates in the prefill pool of a P/D-disaggregated system, partitioning a fixed GPU budget among persistent workers with different CP degrees.Requests are subsequently transferred to an independently provisioned decode pool.
  • At request timescales, the scheduler selects an existing eligible worker using worker state and worker-specific prefix-cache information.
  • At cluster timescales, a reconfiguration controller adjusts worker composition through split and merge operations as workload conditions change.These operations determine which CP workers should exist over longer workload windows.
  • The global prefix-cache manager adapts prefix placement to request routing and worker reconfiguration.
  • Figure 5 distinguishes request-level data and control paths from cluster-level management operations using solid and dashed arrows, respectively.

5 REQUEST-LEVEL SCHEDULING

The request-level scheduler chooses among eligible workers with different CP degrees by balancing load, cache reuse, service time, and the GPU-time cost of additional parallelism. This avoids optimizing latency, locality, or CP degree in isolation.

  • The scheduler jointly considers request length, worker load, and worker-specific prefix reuse across different CP degrees.Considering any one factor alone can produce poor aggregate serving decisions.
  • A larger CP degree may reduce prefill latency but occupies more GPUs, creating an opportunity cost that can reduce concurrent serving capacity.
  • The scheduler estimates waiting pressure from outstanding prefill workload and compares worker-specific longest reusable prefixes across candidates.The workload is used as a lightweight proxy for waiting time, while cache lengths may differ for the same request.
  • Workers are excluded when unavailable, draining, or unable to satisfy sequence-length or memory requirements; cache misses do not exclude them because missing prefixes can be recomputed.After selection, the scheduler reserves estimated prefill workload to prevent concurrent arrivals from observing stale load state.
  • The placement cost combines queuing-delay cost, cache-aware service-time cost, and an additional GPU-time cost controlled by λ.β controls emphasis on cache-aware service time, while λ controls the cost of occupying additional GPUs.
  • For short requests, limited parallel speedup typically favors smaller CP degrees, whereas compute-intensive requests may justify larger degrees when latency reduction outweighs excess GPU time.

6 CLUSTER-LEVEL RECONFIGURATION

Vertumnus adapts the cluster’s CP worker composition to evolving aggregate demand using profiled capacity, workload thresholds, and topology-compatible split or merge operations. These transitions preserve GPU count and reconcile retained cache state.

  • Capacity Profiling and Initialization: The controller initializes a heterogeneous worker composition from historical demand, request-length distribution, and profiled CP-degree capacities.It favors compositions with more large-degree workers among historically suitable configurations with sufficient capacity.
  • Runtime Reconfiguration: A sustained workload above the upper threshold triggers a split, while demand below the lower threshold triggers a merge to provide more per-request parallelism.The controller also monitors the fraction of long requests when aggregate workload remains stable.
  • Reconfiguration Operations: Each split replaces one degree-2C worker with two degree-C workers, or performs the inverse merge, preserving the aggregate GPU count.Eligible operations are prioritized by predicted drain interval and then expected KV-reconciliation traffic.
  • Reconfiguration Operations: Reconfiguration drains selected workers, redirects queued requests, completes in-flight prefill under the old geometry, and activates the new CP groups at a step boundary.Workers become schedulable only after all participating ranks adopt the new configuration epoch and metadata is updated.
  • Cache Coordination: Retained prefixes stay resident during conversion, while their ownership and addressing are reconciled after split or merge.After merging, metadata is combined and missing blocks become pending reconciliation; after splitting, prefixes are assigned by demand and available capacity.

7 PREFIX CACHE MANAGEMENT

Vertumnus manages prefix caches globally across heterogeneous CP workers by tracking logical prefixes, replica placement, and degree-specific demand. Periodic replication and reclamation preserve locality while adapting cache capacity to changing access patterns.

  • Global Prefix-Trie Management: The global prefix-cache manager tracks the logical prefix structure, replica placement, and degree-level demand across heterogeneous workers.It replicates cached prefixes within and across CP degrees and reclaims underused replicas as worker assignments and composition change.
  • Global Prefix-Trie Management: Each trie node represents a cache block, while its root-to-node path identifies the complete prefix and its replica holders across CP degrees.ReplicaCount(z,C) records how many degree-C workers hold the complete prefix ending at node z.
  • Per-Request Statistics: For each request, the manager distinguishes the globally matched prefix path from the portion actually reused at the scheduler-selected worker.It updates degree-specific access statistics and accumulates missed cache benefit when a matched prefix lacks a replica at the preferred degree.
  • Periodic Cache Management: Periodic scans generate inter-degree replication, intra-degree replication, and reclamation candidates using access and missed-demand statistics.Inter-degree replication introduces a first replica at a degree with none; same-degree replication responds to per-replica demand, while reclamation removes replicas below a threshold.
  • Replication and Reclamation: Replication copies only missing prefix blocks, choosing destinations with longer retained ancestor paths, sufficient capacity, and lower workloads.Inter-degree transfers repartition compatible KV states, and replica visibility or deletion is updated only after state confirmation.

8 IMPLEMENTATION

Vertumnus extends RTP-LLM across prefill scheduling, worker management, and prefix caching while retaining model state on GPUs and using control-plane metadata for global cache coordination. The implementation uses asynchronous KV transfers and fixed scan intervals.

  • System Integration: The implementation adds approximately 8,000 lines across the prefill engine, scheduler, worker-management layer, and prefix-cache subsystem.Decode batching and kernels are unchanged; decode-side modifications propagate CP configuration epochs and DP-rank identity.
  • Worker Reconfiguration: Each prefill rank loads model weights once and retains them while pre-established, topology-aware CP groups support in-place worker reconfiguration.Existing communicators are reused when rank membership matches required groups, and missing groups are created and warmed before serving.
  • Prefix-Cache Integration: The global prefix trie runs as an in-memory Java index in FlexLB, while workers retain native cache indexes and LRU policies for physical KV blocks.Replication and reclamation decisions run periodically, with selected transfers issued asynchronously through RTP-LLM’s KV-transfer path.

9 EVALUATION

Vertumnus is evaluated on two models across public and production workloads using heterogeneous CP configurations and production-oriented baselines. It delivers stronger latency and SLO results under high load, while reconfiguration and prefix-cache ablations show benefits from adapting worker composition and cache placement.

  • Experimental Setup: The testbed uses 64 NVIDIA H20-3e GPUs, with 40 GPUs for prefill and 24 for an independently provisioned decode pool.DeepSeek uses CP degrees {4, 8}, while Qwen3 uses {2, 4}.
  • Experimental Setup: Vertumnus is compared with homogeneous, static-threshold, vLLM, and SGLang baselines under the same serving backend or prefill budget.The baselines differ in worker composition and CP support, including homogeneous TP-4 workers for vLLM and degree-4 workers for SGLang.
  • End-to-End Serving Performance: Across two models, three workloads, and varying request rates, Vertumnus achieves the lowest or comparable mean and P90 TTFT and the highest or near-highest SLO attainment.Its advantages become clearer as baselines accumulate delays under high load.
  • End-to-End Serving Performance: At the highest Qwen3 request rates, Vertumnus reduces mean TTFT by 13.9%, 21.0%, and 28.1% on L-Eval, Tool-Agent, and Production, respectively, versus the strongest baseline.Corresponding P90 reductions are 17.8%, 55.0%, and 29.5%, while SLO attainment improves by 0.9, 13.3, and 1.1 percentage points.
  • Worker Reconfiguration: Under time-varying demand, Vertumnus switches worker compositions and reduces mean TTFT by 19.6% during rising request rates and by 5.5% and 14.5% during later long-request-heavy demand.Each split or merge takes between 1.1 and 5.1 seconds, with most variation from the drain stage.
  • Prefix-Cache Ablation: At 60 requests/s, the complete prefix-cache manager reduces mean and P90 TTFT by 10.3% and 18.7%, respectively, while modestly improving token-level prefix-cache hit rate.Replication expands cache-preserving placement choices within and across CP degrees.

10 RELATED WORK

Prior serving systems optimize execution locally, route requests or place instances at the cluster level, or dynamically adjust active execution plans. Vertumnus instead maintains persistent prefill workers with different CP degrees, reconfigures their composition, and coordinates prefix caching.

  • Existing systems optimize request execution locally through batching, execution order, and interleaving, or at the cluster level through routing, migration, and instance placement.
  • Most systems use fixed DP, TP, or PP configurations, while newer systems reconfigure model parallelism as workloads or GPU availability change.
  • LoongServe and NanoCP adapt context or sequence parallelism for active requests or batches rather than maintaining persistent worker composition across workload windows.
  • Vertumnus organizes a fixed GPU budget into persistent CP workers with different degrees, routes incoming requests among them, and reconfigures composition as aggregate demand evolves.
  • Prior prefix-cache systems reuse shared prompt states, while cache-aware and KV-state systems coordinate reuse, scheduling, worker placement, or memory tiers.

11 CONCLUSION

Vertumnus adaptively manages context parallelism for heterogeneous and changing production workloads. It combines persistent workers, adaptive composition, coordinated request placement, and global prefix-cache management, with experiments demonstrating effectiveness across diverse conditions.

  • Vertumnus adaptively manages context parallelism under heterogeneous and changing production workloads.
  • The system supports persistent workers with different CP degrees and coordinates request placement among them.
  • Vertumnus adjusts worker composition through lightweight split and merge operations at the cluster timescale.
  • A global prefix-cache manager tracks prefix demand and dynamically replicates and reclaims cached prefixes.
  • Experiments with public and production workloads demonstrate Vertumnus's effectiveness under diverse workload conditions.
Loading 2609.04774v1…