Source-linked AI summary
EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models
Kunxiong Zhu, Zhihao Shu, Hangyu Zheng, Minghai Qin, Miao Yin, Gagan Agrawal, Wei Niu
TL;DR
Multimodal serving adds Encode to the Prefill-Decode pipeline, while existing systems provide limited control over Encode and downstream request flow. EAServe co-optimizes Encode-centered runtime mechanisms and deployment configuration; across image, video, and audio workloads, it reports higher goodput than NVIDIA Dynamo and vLLM under SLO constraints.
Problem
Text-only PD systems lack Encode, while existing EPD frameworks expose it as a pass-through without regulating downstream flow, leaving encode GPUs underused and downstream workers waiting.
Source: §1 · pp. 1–2Method
Results
EAServe improves goodput by up to 4.3× over Dynamo and 1.7× over vLLM across image, video, and audio workloads, with more balanced GPU utilization across EPD.
Source: §1 · p. 2Takeaways & Limitations
The runtime mechanisms adapt to load online with no per-deployment tuning.
Source: §1 · p. 2When its benefit is small, SM partitioning collapses to a single-tenant encode allocation, incurring no overhead.
Source: §3.4 · p. 7
Abstract
from arXiv · showhide
Disaggregating the two stages, Prefill and Decode, onto separate GPU pools is now a standard optimization for (text-only) LLM serving. However, multimodal LLMs (MLLMs), which add a third phase, Encode, pose new challenges for resource allocation. Encode turns images, video, or audio into embeddings that the language model can consume, yielding a three-stage Encode-Prefill-Decode (EPD) pipeline. Existing frameworks offer only partial answers: text-only PD systems lack Encode, while EPD frameworks expose it as a separate service without regulating downstream request flow. The pipeline also carries a structural resource imbalance: every request enters through Encode before downstream work can begin, yet per-request execution leaves the encode GPU severely underutilized even at high loads, starving the downstream Prefill and Decode workers. Addressing this, we reposition Encode as the control point of the EPD pipeline, exposing three tightly coupled dimensions: when work enters downstream, where prefill executes, and how the GPU is shared. We instantiate this in EAServe across two co-designed layers. Its runtime manages load-adaptive micro-batching, rate-controlled partial offload to a co-resident prefill worker, and dynamic SM partitioning for predictable co-location. The configuration layer, Hybrid Auto Selection (HAS), navigates the joint space of GPU allocation, encode batch size, and offload ratio by pruning unbalanced allocations with per-stage capacity profiling and refining the remainder through TPE-based Bayesian optimization. Evaluated on three MLLM architectures spanning image, video, and audio, EAServe delivers up to 4.3x and 1.7x higher goodput than NVIDIA Dynamo and vLLM, respectively, under identical SLO constraints, sustains more balanced and higher GPU utilization across the EPD pipeline, and reaches near-optimal configurations faster than baseline search methods.
1 Introduction
EAServe treats Encode as the control point of multimodal serving, addressing underused encode capacity and coordinating downstream request flow, prefill placement, and GPU sharing. Across image, video, and audio workloads, it reports higher goodput than NVIDIA Dynamo and vLLM under latency objectives.
1 Introduction
- MLLM serving adds modality-dependent Encode work before Prefill and Decode, making its EPD pipeline more heterogeneous and harder to schedule than text-only PD.Encode converts images, video, or audio into embeddings consumed by the language model.Source: §1 · p. 1
- Existing EPD frameworks expose Encode as a pass-through service without regulating downstream work, leaving encode GPUs underused while downstream workers wait and tail latency inflates.Per-request encoding does not saturate a modern GPU, while waiting to form larger batches carries a latency cost.Source: §1 · pp. 1–2
- EAServe repositions Encode as the pipeline control point and combines offline Hybrid Auto Selection with three runtime mechanisms for load-adaptive batching, partial prefill offload, and dynamic SM partitioning.HAS searches GPU allocation, encode batch bound, and prefill offload ratio; the runtime mechanisms adapt online without per-deployment tuning.Source: §1 · p. 2
- EAServe improves goodput by up to 4.3× over Dynamo and 1.7× over vLLM across workloads, with more balanced GPU utilization across EPD.Evaluation covers three MLLMs spanning image, video, and audio on an 8-GPU server under TTFT/TPOT targets.Source: §1 · p. 2
2 Background and Motivation
The EPD pipeline adds Encode before Prefill and Decode, with distinct stage resource profiles and SLO-aware evaluation. Encode gates downstream work but can remain underutilized, motivating coordinated dispatch, offload, and GPU sharing.
2.1 EPD Pipeline And Metrics
- EPD separates Encode, which produces modality embeddings, from Prefill, which emits the first token, and Decode, which generates the remainder.The three stages have different resource profiles, motivating independent provisioning.Source: §2.1 · p. 2
- Goodput measures the rate of requests meeting TTFT and TPOT SLO thresholds, complementing latency measurements at the mean and P99.TTFT spans arrival through the first output token; TPOT measures intervals between Decode tokens.Source: §2.1 · p. 2
2.2 Why Encode Becomes a System Bottleneck
- Encode gates downstream work, but arrival-limited batching leaves compute and HBM utilization low; simple micro-batching raises throughput and lowers TTFT while sharply raising mean TPOT.At sustained arrival rates, the growing encode queue leaves Prefill and Decode waiting; B=8 still leaves the encode GPU far from saturated.Source: §2.2 · pp. 2–3
- The system must adapt batch bounds to arrivals, offload work to local Prefill, and isolate co-located workers because these choices are coupled.The batch bound changes the SM split, the split limits offload, and stage allocation rebalances the joint configuration.Source: §2.2 · p. 3
2.3 SM Partitioning
- Software SM partitioning co-locates Encode and Prefill on disjoint SM subsets, but shared HBM bandwidth rules out placing Decode on the encode GPU.The partition ratio must track changing encode batch bounds and local-prefill load, making a static split ineffective.Source: §2.3 · pp. 3–4
3 System Design
EAServe co-designs offline configuration search with runtime mechanisms that control encode dispatch, prefill placement, and GPU sharing. HAS prunes and refines the joint deployment space, while the runtime adapts these decisions to each micro-batch.
3 System Design
- The two co-designed layers pair an offline search over GPU allocation, maximum encode batch size, and offload ratio with runtime mechanisms that realize the selected configuration.HAS selects the deployment configuration for the EPD pipeline; runtime controls when encode dispatches, where prefill runs, and how the encode GPU is shared.Source: §3 · p. 4, §3.1 · p. 4
3.1 Hybrid Auto Selection
- HAS profiles stage capacities to prune unbalanced GPU allocations, then refines survivors with end-to-end TPE trials over batch size and offload ratio.It ranks configurations by throughput, using mean TTFT and TPOT as tie-breakers; the search stops when top configurations stabilize or improvement stalls.Source: §3.1 · pp. 4–5
3.2 Load-Adaptive Micro-Batching
- The Poisson-gap micro-batcher sets g inversely with per-worker arrival rate and dispatches on batch bound, idle-gap threshold, or hard age cap.The threshold adapts without per-workload retuning; the batcher adds O(1) work per arrival and sub-microsecond timer overhead.Source: §3.2 · pp. 5–6
3.3 Rate-Controlled Partial Offload
- Deficit-controlled routing uses a counter to track target offload ratio s, keeping cumulative remote requests within one of the target count and bounding consecutive routing runs.The counter adds O(1) work per request and introduces no new tuning parameter.Source: §3.3 · p. 6
3.4 Dynamic SM Partitioning
- EAServe profiles the smallest safe encode SM share for each batch size, then gives the remainder to local prefill; when s=1, prefill is not co-located.The split is updated per micro-batch and uses a startup lookup table based on encode latency profiling.Source: §3.4 · p. 7
4 Evaluation
The evaluation compares EAServe with serving and search baselines across image, video, and audio workloads, then tests its configuration search, GPU-sharing policy, and arrival-pattern sensitivity. EAServe’s gains vary by modality and arise from both encode-aware batching and stage allocation, with local prefill and SM partitioning contributing under specific conditions.
4.1 Experimental Setup
- The main comparison uses three MLLM architectures and controlled Poisson arrivals from 1 to 5 req/s, spanning under-, near-, and over-saturated regimes.Each run processes 500 requests, and reported values are medians of three runs; Dynamo and HAS search the same configuration space.Source: §4.1 · pp. 7–8
4.2 End-to-End Comparison
- Up to 4.3× higher goodput than Dynamo and 1.7× higher than vLLM at the moderate tier across image, video, and audio.The evaluation uses five SLO tiers; the strictest tier describes the SLO frontier rather than an attainable operating point at the swept rates.Source: Figure 9 · p. 8, §4.2 · p. 8
- Dynamo’s encode GPU stays below 10% utilization, while EAServe’s co-located local prefill raises sustained utilization to roughly 80%.HAS selects local prefill for image and audio, but sets s=1 for video, where the multi-frame encoder uses more than 90% of SMs during batched forwards.Source: §4.2 · pp. 8–9, Figure 11 · p. 9
- On RTX 6000 Ada, EAServe reaches 1.95 req/s, 4.24× EPDServe and 3.36× HydraInfer; on A100, it reaches 4.91 req/s, 4.86× and 3.25×, respectively.The comparison uses LLaVA-v1.6-34B at Λ=5 req/s, with each system’s best configuration selected separately on each platform.Source: Table 5 · p. 9, §4.2 · p. 9
4.3 HAS Efficiency
- HAS’s allocation pruning reduces seven feasible allocations to three, and it finds the highest-throughput configuration within 30 trials across all three modalities.Pruned allocations had a bottleneck stage at least 20% more loaded than the best-balanced candidate at the target rate.Source: §4.3 · p. 9
- HAS reaches the 99th percentile of best-observed throughput in 1.7 h on image, 1.3 h on video, and 1.7 h on audio; Random reaches none within 30 trials.Coordinate fails to reach that percentile on image and video; Figure 12 summarizes wall-clock search time to throughput percentiles.Source: §4.3 · pp. 9–10, Figure 12 · p. 9
4.4 SM Partitioning Ablation
- Without SM partitioning, throughput drops by 9%, P99 TTFT rises by 19%, and P99 TPOT by 42% versus dynamic partitioning; static 50/50 splitting also underperforms.The Ultravox-27B ablation uses local prefill at Λ=10 req/s; dynamic partitioning adapts per batch, giving encode enough SMs near its latency knee and the remainder to local prefill.Source: §4.4 · p. 10, Table 6 · p. 10
4.5 Arrival Pattern Sensitivity
- Under Gamma CV=2 arrivals, throughput drops only 3% (1.77→1.71 req/s), while P99 TTFT rises 62% (10.2→16.5 s); under CV=0.5, P99 TTFT falls 35% (10.2→6.7 s).P99 TPOT remains approximately 71 ms across the tested arrival distributions.Source: §4.5 · p. 10, Table 7 · p. 10
5 Related Work
Prior work addresses text-only stage disaggregation, multimodal EPD serving, GPU sharing, and inference scheduling. EAServe distinguishes itself by making Encode the control point and jointly configuring encode-aware runtime mechanisms with GPU allocation and offload.
Disaggregated MLLM serving
- Unlike text-only PD systems and EPD systems that allocate Encode per request, EAServe co-designs encode-aware mechanisms and automated configuration around Encode as the pipeline entry point.NVIDIA Dynamo and HydraInfer both use EPD disaggregation, but neither provides EAServe’s joint treatment of encode batching and prefill offload.Source: §5 · pp. 10–11
Intra-GPU prefill–decode multiplexing
- Prefill–Decode multiplexing systems share phases within one LLM instance, whereas EAServe multiplexes dependent Encode and local Prefill stages of one MLLM request.This co-location adds an offload-ratio dimension jointly configured with GPU allocation and micro-batch bound.Source: §5 · p. 11
GPU sharing and SM partitioning
- General GPU-sharing mechanisms target independent workloads; EAServe’s Encode and local Prefill are dependent stages requiring pipeline-aware QoS.The paper adopts SM partitioning primitives rather than proposing new partitioning primitives.Source: §5 · p. 11
Inference batching and scheduling; automated configuration
- Unlike text-only batching and scheduling methods, EAServe’s Poisson-gap micro-batching targets Encode’s single-shot forward pass; HAS screens allocations empirically before tuning non-monotonic runtime parameters.This combines per-stage capacity profiling with end-to-end measurements and TPE-based search.Source: §5 · p. 11
6 Conclusion
EAServe treats Encode as the control point of disaggregated MLLM serving, combining offline joint configuration search with runtime mechanisms for batching, offload, and GPU sharing. Across image, video, and audio workloads on an 8-GPU server, it reports up to 4.3× higher goodput than production baselines, while identifying multi-node validation and larger search spaces as future work.
6 Conclusion
- Across image, video, and audio MLLM workloads on an 8-GPU server, EAServe delivers up to 4.3× higher goodput than strong production baselines.The design jointly searches GPU allocation, encode batch bound, and prefill offload ratio, and uses encode-aware runtime mechanisms.Source: §6 · p. 11