PaperlayerView exact revision on arXiv

Source-linked AI summary

TurboServe: Serving Streaming Video Generation Efficiently and Economically

Youhe Jiang, Haoxu Wang, Haotong Bao, Kai Jiang, Jianfei Chen, Jun Zhu, Fangcheng Fu, Jintao Zhang

arXiv:2606.19271v1cs.DC

TL;DR

Streaming video generation requires serving systems to handle heterogeneous session durations and fluctuating demand while meeting tight per-chunk latency targets. TurboServe jointly coordinates session placement and GPU provisioning, reducing worst-case per-chunk latency by 37.5% and GPU operating cost by 37.2% on average versus baseline configurations.

  • Problem

    Streaming video serving must manage heterogeneous session durations and sharply time-varying demand while maintaining tight per-chunk latency in multi-GPU environments.

  • Method

    TurboServe uses closed-loop scheduling that jointly coordinates migration-aware session placement and load-driven GPU autoscaling for streaming video workloads.

  • Results

    37.5% lower worst-case per-chunk latency and 37.2% lower total GPU operating cost on average versus baseline serving configurations.

  • Takeaways & Limitations

    TurboServe improves the latency-cost trade-off for streaming video generation across evaluated production traces, model sizes, and GPU configurations.

  • Takeaways & Limitations

    Experiments use an in-house Shengshu Technology cluster, with GPU operating cost reported in cloud-price-equivalent dollars.

Abstract

from arXiv · show

Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.

1 Introduction

Streaming video generation introduces persistent, stateful sessions with heterogeneous durations and sharply varying demand, challenging stateless serving systems. TurboServe jointly manages session placement and GPU provisioning, achieving lower latency and cost through online scheduling and runtime session mobility.

  • Workload shift: Streaming generation progressively emits chunks while preserving session state across active and idle periods, unlike stateless one-shot requests completed independently.Sessions can be suspended and resumed while retaining prompt context and cached temporal states.
  • Serving challenges: Long-lived sessions make static placement decisions increasingly suboptimal, causing load imbalance and potentially preventing real-time generation for some users.Session durations range from short clips to tens of minutes of iterative prompting and refinement.
  • Serving challenges: Sharp demand fluctuations create a provisioning trade-off: peak-oriented GPU capacity wastes resources during idle periods, whereas average-oriented capacity fails during bursts.Users alternate between activity bursts and idle periods as sessions arrive and complete.
  • TurboServe: TurboServe jointly controls session placement and GPU provisioning through closed-loop scheduling with migration-aware placement and load-driven autoscaling controllers.The placement controller performs event-driven min-max rebalancing under per-chunk latency constraints.
  • Evaluation: 37.5% lower worst-case per-chunk latency and 37.2% lower total GPU operating cost on average were achieved against baseline serving configurations.Evaluation used real-world Shengshu Technology production traces across multiple video generation model sizes.

2 Background and Related Work

Streaming video generation shifts video synthesis toward long-lived, interactive sessions that progressively produce chunks conditioned on prior outputs and cached state. Related work spans streaming video models, LLM serving, and autoscaling, but existing LLM systems target finite request-centric inference workloads.

  • Streaming video generation workflow: Streaming video generation serves user sessions rather than one-shot requests, progressively generating chunks from prompts and previously generated chunks with cached state.The workflow is illustrated for autoregressive streaming video models such as Self-Forcing.
  • Streaming video generation: Recent systems organize diffusion-based video generation around continuous or autoregressive generation for long videos, dynamic inputs, and interactive use cases.Examples include StreamDiffusionV2, StreamingT2V, and StreamV2V.
  • LLM serving: LLM serving systems model requests as finite decoding sequences with dynamically growing KV-cache state, whereas streaming video generation differs in state lifetime.Representative systems include vLLM, Sarathi-Serve, and AlpaServe.
  • Autoscaling for LLM serving: Autoscaling systems dynamically adjust GPU allocation, model placement, execution capacity, or checkpoint loading according to workload variation and latency objectives.These approaches improve resource efficiency under fluctuating demand but primarily target request-centric inference workloads.

3 Motivation

TURBOSERVEbase provides runtime support for concurrent chunk execution, persistent session lifecycle management, and GPU-CPU state offloading. Characterization shows that session migration addresses load imbalance, autoscaling improves efficiency under changing demand, and coordinating both motivates TurboServe’s closed-loop scheduler.

  • TURBOSERVEbase runtime: TURBOSERVEbase supports concurrent chunk execution under per-chunk latency constraints, persistent session-state management, and GPU-CPU offloading for suspension and resumption.It serves multiple sessions per GPU while preserving state across active and idle periods.
  • TURBOSERVEbase runtime: Coalesced execution batches ready sessions on each GPU and invokes the model once, improving utilization while bounding co-located sessions to meet latency targets.The runtime collects ready sessions, groups them by GPU, and writes generated chunks to the corresponding sessions.
  • Characterization study: Evolving session activity creates GPU load imbalance that raises bottleneck latency, while fixed GPU allocation causes underutilization during low demand and overutilization during high demand.The worst-case per-chunk latency is determined by the most heavily loaded GPU.
  • TURBOSERVEbase runtime: Sessions transition among execution, suspend, and terminate states, while offloading preserves suspended state in host memory and restores it before resumption.Suspension releases the GPU slot for other active sessions.
  • Characterization study: Migration balances GPU loads, autoscaling adapts capacity to workload demand, and their coordination motivates TurboServe’s unified closed-loop scheduling framework.Migration addresses placement imbalance, while autoscaling preserves latency headroom during bursts and reduces unnecessary capacity during low-demand periods.

4 TURBOSERVE Overview

TurboServe comprises four major components that coordinate workload monitoring, session placement, autoscaling, and session management. Its routine uses recent session events and workload signals to assign sessions to GPU workers and rebalance them in response to GPU status and session states.

  • Architecture: TurboServe consists of a workload detector, placement controller, autoscaling controller, and session manager.The architecture overview presents these as the system’s four major components.
  • Overall routine: The workload detector monitors arrivals, departures, and active/idle transitions using a sliding window to extract recent demand and demand variation.These workload signals feed subsequent scheduling decisions.
  • Overall routine: The placement controller assigns sessions to GPU workers using session states, GPU status, and workload signals while performing migration-aware rebalancing to reduce bottleneck latency across GPUs.The passage describes placement as part of TurboServe’s overall routine.

5 Scheduling Framework

TurboServe formulates streaming serving as an event-driven online scheduling problem that jointly controls session placement and GPU provisioning. Its closed-loop framework minimizes worst-case per-chunk latency and GPU operating cost while adapting to session-state changes and workload variation.

  • Problem formulation: The scheduler jointly assigns sessions to GPUs and determines the active GPU count M(t), with each GPU serving up to K concurrent sessions under the latency constraint.Sessions may execute on an active GPU or suspend with state offloaded to host memory; terminated sessions release all resources.
  • Problem formulation: Event-driven scheduling invokes decisions at arrivals, departures, and active–idle state transitions, observing active sessions and resource utilization at each decision epoch t.The scheduler responds when αi(t) changes, capturing users becoming active or idle.
  • Problem formulation: The multi-objective optimization minimizes GPU operating cost C(t) = cgpu · M(t) and worst-case executing-session latency L(t), subject to capacity and responsiveness constraints.Latency increases with per-GPU session co-location and may spike during migration, while user-input sessions must remain actively executed.
  • Closed-loop scheduling: The closed-loop scheduler couples a placement controller, which updates assignments and provides load feedback, with an autoscaling controller, which adjusts the GPU budget.Together, the controllers regulate the system toward a desired operating point.
  • Placement controller: The placement controller incrementally assigns new or newly active sessions and performs migration-aware min-max rebalancing to reduce bottleneck latency while limiting migration overhead.It evaluates candidate moves from the highest-latency GPU and applies only moves whose positive gain outweighs migration cost.

6 TURBOSERVE Deployment

TURBOSERVE’s deployment layer manages session state and GPU resources while its closed-loop framework updates placement and provisioning for dynamic multi-session workloads. It combines structured per-session memory, chunk-boundary GPU migration, workload-level control tuning, and serving-pool scaling procedures.

  • Control-parameter selection: Workloads are classified by sliding-window session-activation volatility, with offline cost-optimal parameters selected by workload level at runtime.Appendix A compares this procedure with an offline trace-tuned oracle for near-optimality validation.
  • Deployment mechanisms: TURBOSERVE focuses deployment on two implementation aspects: session management and GPU management.These mechanisms support efficient runtime updates to session placement and GPU provisioning.
  • Session management: Each GPU worker separates a shared model replica, isolated per-session state regions, and a session ownership table.Per-session state includes descriptors, embeddings, temporal or KV-cache states, chunk-history features, latent buffers, and output metadata.
  • Session management: TURBOSERVE migrates only the moved session’s state region through registered GPU memory buffers and one-sided GPU memory access.The target worker fetches the state from the source worker and installs it locally.
  • Session management: GPU-GPU migration occurs only at chunk boundaries, with ownership updated after state transfer and buffer verification complete.The source finishes the current chunk and freezes state before the target fetches and verifies the required buffers.

7 System Evaluation

TurboServe is evaluated on production streaming-video workloads across heterogeneous model sizes, traces, and GPU clusters, where it improves both latency and cost. Ablation and runtime analyses show that migration and autoscaling jointly provide efficient, practical online control with low scheduling and migration overhead.

  • Evaluation Setup: TurboServe is evaluated on Shengshu Technology production traces using LongLive-style models across 16 NVIDIA H20 and 64 NVIDIA B300 GPUs.The traces include heterogeneous session durations, bursty activation patterns, and time-varying active-session counts.
  • End-to-End Performance: 37.5% average reduction in worst-case per-chunk latency is achieved under matched GPU operating cost, reaching 51.6% across baseline comparisons.TurboServe consistently outperforms evaluated baselines across model sizes, traces, and cluster configurations.
  • End-to-End Performance: 37.2% average reduction in GPU operating cost is achieved under matched worst-case per-chunk latency, reaching 49.0% across baseline comparisons.TurboServe achieves the lowest serving cost among evaluated methods while maintaining the same latency target.
  • Ablation Analysis: 15.0% average higher GPU operating cost results when migration is disabled, showing that migration and autoscaling address complementary imbalance and demand-variation bottlenecks.Removing either mechanism generally degrades cost efficiency compared with the full system.
  • Scheduling Efficiency: 15 ms scheduling time on clusters with up to 64 GPUs is less than 2% of per-chunk generation time, while placement remains within 3.6% of an exhaustive-search oracle on average.The migration-aware rebalancing algorithm scales from 4 to 256 GPUs and balances near-oracle placement quality with online feasibility.
  • Runtime Overheads: 23-30 ms migration overhead, or 2%-3% of per-chunk latency, makes NCCL-based online GPU-GPU session rebalancing practical relative to generation time.The measured overhead includes buffer allocation and release, state transfer, and session ownership updates.

8 Conclusion

TURBOSERVE is presented as the first serving system for streaming video generation in multi-session, multi-GPU environments. Its closed-loop framework coordinates migration-aware session placement and load-driven GPU autoscaling to address heterogeneous session durations and time-varying demand.

  • Conclusion: TURBOSERVE targets streaming video generation workloads in multi-session, multi-GPU environments.The paper describes it as the first serving system designed specifically for this workload setting.
  • Conclusion: Its closed-loop scheduling framework jointly coordinates migration-aware session placement and load-driven GPU autoscaling.These controllers address heterogeneous session duration and time-varying user demand.
  • Conclusion: TURBOSERVE further supports runtime execution through coalesced chunk processing and session-state migration.The passage introduces these mechanisms as additional support for efficient runtime execution, though the supplied text ends during the migration description.

A Volatility-to-Parameter Mapping

TurboServe maps measured workload volatility to profiled autoscaling parameters through an offline-to-online control loop. The mapping lowers target utilization as volatility rises, preserves the latency constraint across all profiled levels, and closely tracks an offline oracle on unseen fluctuating workloads.

  • Online replacement workflow: The controller profiles a volatility-to-parameter mapping offline, then updates autoscaling parameters online by measuring, quantizing, looking up, and replacing them.Representative trace segments are grouped by volatility, and grid search selects parameters minimizing GPU cost subject to the target worst-case per-chunk latency constraint.
  • Profiling results: 0.80 to 0.25: target utilization ρ∗ decreases monotonically from volatility level 1 to level 10, reserving more GPU headroom for bursts.The profiled values form four discrete bands: 0.80, 0.65, 0.50, and 0.25.
  • Profiling results: 0.2: latency weight λℓ remains fixed across all ten volatility levels, leaving ρ∗ as the primary control axis for this trace family.The results indicate that λ would become a more important axis when adjustment through ρ∗ alone is insufficient.
  • Profiling results: 338.66 to 770.44: average GPU cost increases monotonically from volatility level 1 to level 9, roughly 2.3×, as lower ρ∗ requires more GPUs.All ten levels achieve a 100% pass rate under LSLO = 670 ms, so the configurations remain feasible while adapting cost to burstiness.
  • Comparison with an offline oracle: 668.05 ms: both the online mapping and offline oracle keep served-chunk maximum latency within the 670 ms target, with average latencies of 421.57 ms and 410.47 ms, respectively.The oracle uses future arrivals to select per-window ρ∗ values from 0.55 to 0.95, while the online mapping uses recent history and selects values from 0.60 to 0.88, yet incurs almost the same total cost.

B Workload Statistics

This section characterizes the evaluated workload traces and their experimental cluster assignments. It also defines the trace statistics used to describe arrivals, departures, and concurrent activity.

  • Trace characterization: Table 11 provides workload trace characteristics for characterization and motivation, while Table 12 reports detailed characteristics of the evaluated traces.The two tables serve complementary characterization purposes.
  • Evaluated traces: Traces T1–T3 run on the H20 cluster (Cluster 1), while traces T4–T6 run on the B300 cluster (Cluster 2).These assignments are reported for the evaluated traces.
  • Trace statistics: Arrivals count new jobs entering the system in each span, departures count jobs completing in each span, and average active counts mean concurrent jobs.These definitions describe the workload trace characteristics in Tables 11 and 12.
Loading 2606.19271v1…