PaperlayerView exact revision on arXiv

Source-linked AI summary

OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, Zhihao Zhang, Xupeng Miao, Rashmi Vinayak, Zhihao Jia

arXiv:2609.14237v1cs.DCcs.AI

TL;DR

Existing ODS runtimes use fixed attention–FFN or attention–MoE splits, limiting flexible heterogeneous plans. OpWeave combines cost analysis, joint planning, and runtime execution; it reports lower serving costs on homogeneous and heterogeneous clusters while meeting TPOT SLOs.

  • Problem

    Existing ODS runtimes organize execution around fixed attention–FFN or attention–MoE splits, making flexible operator staging across heterogeneous device groups a runtime challenge.

    Source: §1 · p. 2

  • Method

    OpWeave combines an analytical cost model, regularity-aware joint deployment planning, and a vLLM-based runtime for synthesized heterogeneous ODS plans.

    Source: §9 · p. 12

  • Results

    Up to 1.78× lower serving cost than the best existing approaches on homogeneous H100 clusters and 1.89× lower than the best non-OpWeave baseline on heterogeneous clusters, while attaining TPOT SLOs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OpWeave’s evaluated plans attain latency SLOs on homogeneous and heterogeneous GPU clusters.

    Source: §1 · p. 3

    CAD stage balance can fail when GEMM weight-loading latency exceeds feasible attention latency; hybrid-layer variation can also leave stages waiting under a fixed-latency rest-of-model stage.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.

1 Introduction

OpWeave addresses gaps in ODS analysis, planning, and runtime support with a framework that synthesizes heterogeneous operator plans. Its evaluation reports lower serving costs while meeting latency SLOs.

1 Introduction

  • Colocation can mismatch decode operators to hardware and couple FFN/MoE batch sizes to attention KV-cache capacity, while heterogeneous ODS can assign stages to suitable devices and scale them independently.Attention is often memory-bandwidth-bound and underuses tensor cores, whereas FFN and MoE rely heavily on matrix multiplication.Source: §1 · p. 2
  • Existing ODS systems fix operator boundaries and lack unified characterization of when disaggregation reduces cost, while synthesized plans also require flexible staging across heterogeneous devices.The runtime gap includes pipeline orchestration and low-overhead communication across device groups.Source: §1 · p. 2
  • OpWeave’s analytical cost model characterizes and bounds homogeneous and heterogeneous ODS gains over colocated serving as sequence length, model architecture, and hardware vary.It identifies CAD’s ideal cost only when stages are balanced and communication is fully overlapped.Source: §1 · p. 2
  • The regularity-aware planner jointly optimizes operator partitioning, hardware assignment, parallelism, batching, and pipeline orchestration, reusing layer-aligned partition decisions across similar operator structures.Its partition block is the smallest repeating, layer-aligned operator pattern used to construct and reuse plans, including for hybrid-attention models.Source: §1 · p. 2
  • A vLLM-based distributed runtime executes synthesized plans with flexible operator stages and pipeline schedules across heterogeneous device groups.Workers submit computation and communication together per decoding iteration to reduce host orchestration overhead.Source: §1 · pp. 2–3
  • Up to 1.78× lower serving cost than the best existing approaches on homogeneous H100 clusters, and up to 1.89× lower than the best non-OpWeave baseline on heterogeneous clusters, while attaining TPOT SLOs.Heterogeneous OpWeave also reduces cost by up to 21.8% compared with homogeneous OpWeave.Source: §1 · p. 3

2 Background and Motivation

Decode batching benefits FFN and MoE more than attention, while growing KV-cache requirements constrain feasible batch sizes. Prior attention–FFN disaggregation pipelines stages, but their fixed split makes performance sensitive to stage balance.

2 Background and Motivation

  • Prefill processes input context in parallel and is typically compute-bound, whereas decode generates one token per autoregressive iteration and can dominate latency and cost for generation-heavy workloads.The prefill–decode distinction motivates placing the phases on separate resources for independent optimization.Source: §2 · p. 3
  • Attention remains memory-bandwidth-bound with low tensor-core utilization as batch size grows, while KV-cache growth limits feasible batches and can prevent FFN/MoE from reaching efficient batch sizes.Figure 2 illustrates utilization across batch sizes and maximum feasible batch size across sequence lengths.Source: §2 · p. 3
  • Prior AFD systems run attention and FFN/MoE on separate groups, pipeline microbatches, and overlap transfers with computation, but latency mismatches create bubbles and reduce utilization.Their fixed attention–FFN boundary leaves unresolved when heterogeneous ODS reduces serving cost and what determines its benefit.Source: §2 · p. 3

3 Theoretical Analysis

The analysis models when homogeneous and heterogeneous operator disaggregation can reduce serving cost, and identifies conditions that limit idealized core-attention disaggregation (CAD). It then shows how sequence length, hardware trade-offs, memory, and layerwise variation shape those gains and limits.

3.1 Analytical Setup

  • The setup models steady-state decode with memory-bound attention and GEMM-dominated computation, deriving costs under assumptions about compute-bound batches and hidden communication.The model includes GPU cost, bandwidth, capacity, and peak throughput, and uses a relaxed batch size alongside an exact integer batch limit for colocation.Source: §3.1 · pp. 3–4

3.2 Homogeneous Disaggregation vs. Colocation

  • When the relaxed colocated batch remains at or above the GEMM compute-bound threshold, homogeneous disaggregation has no gain; below it, gain rises with sequence length but is bounded by M/(M−Mweights).The bound reflects the model-weight footprint already consuming device memory.Source: §3.2 · p. 4

3.3 Heterogeneous vs. Homogeneous Disaggregation

  • Heterogeneous disaggregation never costs more than homogeneous disaggregation, but yields no gain when one GPU type dominates both cost per byte and cost per FLOP.When bandwidth and compute efficiencies trade off, gain is unimodal in KV-cache size, peaking only if the crossing lies within the feasible interval.Source: §3.3 · p. 5

3.4 From Bounds to Plans: CAD and Its Limits

  • CAD reaches its idealized cost only with balanced stage latencies and hidden communication, but memory and GEMM weight-loading latency can leave no feasible balance.When the GEMM weight-loading floor exceeds the maximum feasible attention latency, the latency ranges do not overlap and pipeline bubbles are unavoidable.Source: §3.4 · p. 5, Figure 6 · p. 5
  • In hybrid-attention models, CAD’s single rest-of-model stage cannot match both full- and efficient-attention layers, leaving pipeline bubbles.Full attention loads KV cache for the whole context, while efficient operators use bounded windows or compact recurrent state.Source: §3.4 · p. 5

4 Serving-Plan Formulation

The serving-plan formulation jointly selects operator stages, device groups, parallelism, batching, and pipeline execution. Partition blocks keep this search reusable across model layers, while schedule simulation accounts for stage execution and transfers under memory and TPOT constraints.

Serving-plan optimization

  • OpWeave jointly chooses operator partitions, device-group assignments, parallelism, batching, and pipeline orchestration to minimize serving cost subject to memory and TPOT constraints.The coupled choices affect execution latency, transfer overhead, memory feasibility, and replica count.Source: §4 · pp. 5, 7

Partition-block abstraction

  • A partition block is the smallest repeating, layer-aligned operator pattern; reusing one contiguous stage template per block limits partition search while accommodating hybrid layer patterns.Stages may span layer boundaries, allowing heterogeneous operators to be balanced together.Source: §4 · p. 6

Structured placement and batching

  • The planner assigns ordered device-group types and parallelism strategies to repeated stage positions, and divides each replica’s resident batch into microbatches.A structured round-robin assignment reuses one pipeline structure across partition blocks.Source: §4 · p. 6

Communication-aware latency model

  • Plan latency includes profiled stage execution and boundary-transfer costs, with schedule simulation enforcing dependencies and resource contention while allowing only feasible computation–communication overlap.The round-trip model incorporates microbatching and repeated partition blocks.Source: §4 · p. 6

Design-space expressiveness

  • The stage-template design can isolate full attention, combine efficient attention with neighboring projections, or map repeated positions to heterogeneous devices; AFD and CAD are restricted two-way templates.Colocated serving is represented as a single-stage template.Source: §4 · p. 6

5 Algorithms

OpWeave makes heterogeneous serving-plan search tractable through regularity-aware templates, network-safe per-owner frontiers, and exact branch-and-bound. These reductions narrow the search while preserving feasible plans for the configured finite search space.

5 Algorithms

  • The planner addresses combinatorial partitioning, many configurations per device-group owner, and an intractable cross-owner product with three successive search reductions.Only combinations surviving all three reductions reach the schedule simulator.Source: §5 · p. 7

5.1 Regularity-Aware Sub-Block Templates

  • Regularity-aware sub-block templates tile a legal contiguous operator template across repeated layer patterns, reducing templates while retaining layer-dependent execution costs.The approach is a structural restriction of the searched plan family, not cost-based pruning.Source: §5.1 · p. 7

5.2 Network-Safe Per-Owner Pareto Frontiers

  • Network-safe frontiers prune owner configurations only when they cannot improve any feasible complete plan, retaining per-occurrence latency vectors for layer variation.Dominance requires the same tensor-parallel degree because transfer volume varies nonmonotonically with tensor parallelism.Source: §5.2 · pp. 7–8

5.3 Exact Branch-and-Bound over Frontier Products

  • Because frontier-product enumeration can require exponentially many simulations, depth-first branch-and-bound uses optimistic bounds to discard prefixes that violate the TPOT SLO or cannot beat the incumbent.Surviving complete configurations are checked with recomputed bounds and the schedule simulator.Source: §5.3 · p. 8
  • The schedule simulator remains authoritative for survivors, making the planner exact over its configured finite hardware, batching, and template grid.The worst-case complexity remains high.Source: §5.3 · p. 8

6 System Design and Implementation

OpWeave installs synthesized plans in a distributed vLLM-based runtime whose staged engines execute across device-group replicas. It reduces fine-grained execution overhead by compiling per-rank work and batching communication.

6 System Design and Implementation

  • OpWeave is implemented on the vLLM GPU model runner, with workers loading only assigned model components and state.Inter-node activations use point-to-point NCCL, while intra-node fanout uses CUDA IPC peer copies with GPU doorbells.Source: §6 · p. 8

6.1 System Overview and Execution Flow

  • The coordinator installs replicas and plans at setup; during decoding, workers execute compiled stage schedules without coordinator or RPC intervention between stages and microbatches.Each replica runs a staged engine with one GPU worker process per GPU rank.Source: §6.1 · p. 8

6.2 Optimizations

  • Fine-grained disaggregation risks overhead from many short compute and transfer actions, so the runtime applies three execution optimizations.The figure’s distributed plane sends activations between stages and returns next-token IDs to first-stage workers.Source: §6.2 · p. 8
  • A single native submission enqueues each rank’s complete per-token sequence of launches, copies, communication rounds, and event dependencies, eliminating Python dispatch and RPCs between stages.The scheduling capacity is fixed by the planner-selected batch size, with CUDA graphs and buffers bound once.Source: §6.2 · p. 8, §6 · p. 8
  • Generated token IDs remain device-resident between iterations: terminal workers compute them on the GPU, and an NCCL all-reduce distributes them to embedding owners.Only global rank zero retains token history for eventual host-side output.Source: §6.2 · p. 8
  • The runtime packs compatible tensors and batches transfers into NCCL rounds, while sending unsharded activations once to a destination leader for local fanout to tensor-parallel ranks.Rotating leader assignment balances cross-node traffic.Source: §6.2 · pp. 8–9

7 Evaluation

Across homogeneous and heterogeneous evaluations, OpWeave finds lower-cost feasible serving plans, while reducing transfers and maintaining high occupancy. The evaluation also reports simulator fidelity, planner-search ablations, and lower network sensitivity than fixed-split baselines.

7.2 Homogeneous End-to-End Evaluation

  • Across H100 evaluations of both models, two SLOs, and three context lengths, OpWeave had the lowest serving cost in 11 of 12 settings, up to 1.78× cheaper than the best feasible baseline for Gemma-3-27B and 1.76× for Qwen3-Next-80B-A3B.Source: §7.2 · p. 9
  • OpWeave plans average 20.7 stages for Gemma-3-27B and 24 for Qwen3-Next-80B-A3B, versus 124 and 96 for both baselines, reducing mean inter-node transfer by up to 22.8×.Fewer transfers reduce decoding latency, helping meet strict TPOT SLOs where neither baseline finds a feasible plan.Source: §7.2 · p. 9, Figure 10 · p. 10
  • Mean compute occupancy exceeds 80% for both models with OpWeave, while CAD and AFD leave roughly half of hardware time unused.For Qwen3-Next at 128K context and batch size 1 on one H100, full-attention and linear-attention modules are estimated at 700 μs and 58 μs, respectively.Source: §7.2 · pp. 9–10
  • Under relaxed SLOs, OpWeave reaches larger mean per-GPU batch sizes than colocation for both models, while AFD and CAD use smaller batches than colocation for Gemma.For Gemma, the baselines’ communication latency forces smaller batches to meet the TPOT SLO.Source: §7.2 · p. 10, Figure 11 · p. 10

7.3 Simulator Fidelity

  • Across AFD, CAD, and OpWeave, simulation errors are 5.8% for Gemma-3-27B and 4.3% for Qwen3-Next-80B-A3B; Figure 12 labels signed simulation errors.Figure 12 compares simulated and measured TPOT normalized by measured TPOT.Source: Figure 12 · p. 10, §7.3 · p. 11

7.4 Heterogeneous End-to-End Evaluation

  • OpWeave achieves lower serving cost than all non-OpWeave baselines in all 12 heterogeneous settings, reaching up to 1.89× cheaper for Gemma-3-27B on H100 with A100.Across H100 with L40S, the reported maxima are 1.54× for Gemma-3-27B and 1.59× for Qwen3-Next-80B-A3B; on H100 with A100, they are 1.89× and 1.66×, respectively.Source: §7.4 · p. 11, Figure 13 · p. 11
  • Heterogeneous OpWeave is 1.04× cheaper on average than homogeneous OpWeave on H100 with L40S and 1.14× cheaper on H100 with A100.The reported maxima are 1.11× and 1.28×, respectively; measured gains can exceed the idealized bounds because those bounds assume full disaggregation of memory-bound attention from compute-bound operators.Source: §7.4 · p. 11

7.5 Ablation Study and Sensitivity Study

  • Disabling Pareto-frontier pruning increases simulator invocations by 1.85× on H100 and 3.75× on A100+H100, while disabling branch-and-bound increases considered candidates by over 45× on each setup.Together, the optimizations make search 1.45× faster on H100 and 3.35× on A100+H100 than using neither.Source: §7.5 · pp. 11–12, Table 3 · p. 11

8 Related Work

Prior LLM-serving work improves efficiency through batching, KV-cache management, caching, disaggregation, and speculative decoding. OpWeave differs from related disaggregation and heterogeneous-serving systems by searching operator-level placements within layers.

LLM Serving Optimization

  • General-purpose serving optimizations include continuous batching, paged KV-cache management, KV reuse, prefill–decode disaggregation, and speculative decoding, but do not directly address attention and GEMM bottlenecks.Source: §8 · p. 12

Operator-Level Disaggregated Serving

  • Closest ODS systems use fixed two-way attention–FFN splits, while other systems target CPU offload, distributed attention and KV capacity, or within-device overlap rather than general operator-class placement.OpWeave adds theoretical cost-gain analysis, a broader plan space, and evidence that fixed splits can fail for hybrid-attention architectures.Source: §8 · p. 12

Heterogeneous LLM Serving

  • Other heterogeneous-serving systems partition at transformer-layer boundaries or coarser, whereas OpWeave matches heterogeneous GPU types to operator classes with different bottlenecks inside a layer.Source: §8 · p. 12

9 Conclusion

OpWeave reports lower serving cost than the best feasible baseline on homogeneous and heterogeneous GPU clusters while meeting latency SLOs. Its analytical appendix models decode cost under explicit execution and communication assumptions.

9 Conclusion

  • Up to 1.78× cheaper on homogeneous GPUs and up to 1.89× cheaper on heterogeneous clusters than the best feasible baseline, while attaining latency SLOs.The conclusion presents these as OpWeave's evaluation results.Source: §9 · p. 12

A.1 Detailed analytical model

  • The analytical model sums operator costs across an ordered decode-step sequence and derives GPU cost per token by dividing step cost by batch size.Operator latency uses a roofline-style maximum of memory-load and compute time.Source: §A.1 · p. 16
  • The model assumes transfers are overlapped through microbatching and pipelining, so communication is omitted from its throughput-critical cost objective.This is an idealized steady-state assumption.Source: §A.1 · p. 16

A.2 Execution-regime assumptions

  • Decode attention is modeled as memory-bound, while GEMM operators become compute-bound above a batch-size threshold.The appendix attributes attention's memory-bound regime to loading the prior-context KV cache for each decode query.Source: §A.2 · p. 16
  • ODS can let GEMM reach the compute-bound regime by freeing its device memory from KV-cache capacity constraints and aggregating requests across attention replicas.This supports the larger effective batch size assumed in the main-text analysis.Source: §A.2 · p. 16

A.3 Derivation and interpretation of the final cost expressions

The appendix derives per-token cost expressions for colocated, homogeneous, and heterogeneous serving from operator-class approximations. It identifies KV-cache-driven batch-size reduction as a condition that can make disaggregation beneficial, subject to the model's scope.

A.3 Derivation and interpretation of the final cost expressions

  • The homogeneous and heterogeneous ODS expressions separate attention memory cost from GEMM compute cost; heterogeneous placement can match each operator class to a GPU with lower cost per byte or FLOP.The expressions assume attention is memory-bound and GEMM-dominated operators are compute-bound.Source: §A.3 · p. 17
  • Colocated serving shares memory across operators, so growing KV-cache requirements shrink feasible batch size and may keep GEMM in a memory-bound regime.In that small-batch regime, weight loading can dominate arithmetic and underuse compute capability.Source: §A.3 · p. 17
  • The idealized disaggregated cost is attainable with Core-Attention Disaggregation when stages are latency-balanced and inter-stage communication is overlapped.CAD assigns core attention kernels to the attention group and remaining GEMM-dominated operators to the other group.Source: §A.3 · pp. 17–18

A.4 Scaling approximations and GEMM threshold

  • The approximations model attention traffic as scaling with batch size and sequence length, GEMM work with active parameters, and GEMM traffic with resident weight footprint.The weight-footprint approximation corresponds to a sufficiently large batch that accesses and amortizes resident expert-weight shards.Source: §A.4 · p. 18
  • Homogeneous ODS begins improving over colocation when KV-cache pressure pushes the feasible colocated batch below the GEMM compute-bound threshold.The threshold separates compute-bound execution at or above b* from memory-bound execution below b*.Source: §A.4 · p. 18
  • These approximations provide interpretable closed-form characterizations, not a full runtime performance model; planning and end-to-end evaluation use richer model- and hardware-specific information.This bounds how the analytical expressions should be used.Source: §A.4 · p. 18

A.5 Proofs for Section 3

The appendix supplies proofs for the Section 3 theorems using the paper's stated notation and scaling approximations. Its homogeneous comparison is limited to feasible sequence lengths under a monotonic KV-cache assumption.

A.5 Proofs for Section 3

  • This appendix provides complete proofs for the theorems stated in Section 3.It uses the notation and scaling approximations introduced in Section 3.1.Source: §A.5 · p. 18
  • The homogeneous comparison is restricted to sequence lengths with b_max(s) ≥ 1.Source: §A.5 · p. 18
  • The proof assumes that per-request KV-cache memory m_kv(s) strictly increases over that feasible domain.Source: §A.5 · p. 18

A.5.1 Proof of Theorem 3.1: Homogeneous Disaggregation vs. Colocation.

The analysis characterizes homogeneous and heterogeneous disaggregation gains and their feasibility, then formalizes the serving plan across repeated operator blocks, device groups, scheduling, and memory constraints.

A.5.1 Proof of Theorem 3.1: Homogeneous Disaggregation vs. Colocation.

  • Ghom(s) = 1 when colocation is compute-bound; in the memory-bound regime it increases with sequence length and is bounded above by M/(M−Mweights).The memory-bound condition occurs when KV-cache pressure pushes the colocated batch below the GEMM saturation threshold, while disaggregation maintains compute-bound GEMM.Source: §A.5.1 · p. 19

A.5.2 Proof of Theorem 3.2: Heterogeneous vs. Homogeneous Disaggregation.

  • The heterogeneous gain is unimodal in KV traffic, peaking at the homogeneous-cost crossing when that crossing lies in the feasible interval; otherwise, the feasible gain is monotone.The analytic gain starts and approaches 1 at the low- and high-traffic extremes.Source: §A.5.1 · pp. 20–21

A.6 CAD Feasibility Condition for Uniform-Layer Models

  • For uniform-layer CAD, feasibility requires overlapping stage-latency ranges; if the condition fails, the stages cannot be balanced and pipeline bubbles are unavoidable.The attention-stage maximum is constrained by per-microbatch memory and bandwidth, while the rest-of-model stage has a weight-loading latency lower bound.Source: §A.6 · p. 21

B Detailed Serving-Plan Formulation

B.1 Model Representation and Partition

  • A partition block is the smallest layer-aligned repeating unit of the operator sequence, and the same stage template is applied to each block.The block spans the smallest repeating period of layer operator patterns.Source: §B.1 · p. 21

B.2 Device-Group Types, Replicas, and Placement

  • Device-group types combine GPU type and intra-group parallelism; their replicas share a configuration, execute the same stage set, and serve disjoint portions of the global batch.Stages are assigned to ordered device-group types using a repeating round-robin map.Source: §B.2 · pp. 21–22

B.3 Scheduling Parameters

  • The scheduling model distinguishes resident batches from microbatches and accounts for execution, transfers, resource contention, and only permitted overlap when estimating steady-state round-trip time.Transfer latency includes resharding or redistribution between replicas, and the planner evaluates the schedule with lightweight simulation.Source: §B.3 · p. 22

B.4 Memory Constraint

  • Each device-group replica must fit its assigned stage weights and resident microbatch KV cache within its effective memory capacity.The KV-cache term is zero for stages without KV-resident attention state.Source: §B.4 · p. 22

B.5 Derivation of the Cost-per-Token Objective

The appendix derives the cost-per-token objective and describes lower bounds used to prune the planner’s search before schedule simulation.

B.5 Derivation of the Cost-per-Token Objective

  • The cost-per-token objective is total replica cost rate divided by system throughput, and the global batch cancels under scale-invariant replication.The objective therefore depends on device-group configurations and per-replica resident batch sizes, not the absolute deployment scale.Source: §B.5 · p. 23

C Planner Details for Section 5; C.1 Branch-and-Bound Lower Bounds

  • Branch-and-bound uses optimistic completion bounds to prune partial assignments before invoking the schedule simulator.The bounds include mandatory serialized compute and communication work, plus a lower bound on normalized cost.Source: C Planner Details for Section 5 · p. 23, §C.1 · p. 23

C.1 Branch-and-Bound Lower Bounds

  • The network latency lower bound accumulates mandatory serialized transfer work at each endpoint and minimizes unassigned boundary costs over feasible replica counts.Independent boundary minimization keeps the estimate optimistic.Source: §C.1 · p. 23

D Notation

  • Table 4 provides the unified notation used throughout the paper.The passage identifies the table’s scope but does not specify its individual entries.Source: Table 4 · p. 24
Loading 2609.14237v1…