PaperlayerView exact revision on arXiv

Source-linked AI summary

SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao

arXiv:2609.37626v1cs.DCcs.AI

TL;DR

Attention layouts trade compute, communication, and cache capacity, while serving workloads can change shape during a run and engines have relied on draining requests to switch layouts. SPLASH uses an ownership model to introduce DOP and switch among layouts while requests continue serving. On B200 GPUs serving GLM-5.3, it improves end-to-end throughput over fixed-layout deployments.

  • Problem

    Serving workloads shift from many short requests to a few long ones, but engines fix an attention layout at launch because switching has required draining requests and restarting workers.

    Source: §1 · pp. 1–2

  • Method

    SPLASH models layouts as choices of projection placement and KV ownership, then reuses resident state and transfers missing state during ongoing inference as a scheduler changes layouts.

    Source: §1 · pp. 2–3

  • Results

    1.3–1.73× higher end-to-end serving throughput over fixed-layout deployments on B200 GPUs serving GLM-5.3.

    Source: §7 · pp. 9–10

  • Takeaways & Limitations

    DOP stores projections and histories once, and following the best feasible layout lets serving adapt as load changes.

    Source: §1 · pp. 2–3

    Near full memory, SPLASH defers a switch or holds admission until one layer of buffer headroom is available.

    Source: G LIMITATIONS · p. 23

Abstract

from arXiv · show

No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reasoning, agentic, and RL-rollout workloads make a fixed choice untenable: a batch that begins as many short requests ends as a few very long ones, so the best layout changes while the same requests run. Serving engines nevertheless fix one layout at launch, because changing it has meant draining requests and restarting workers. We present SPLASH, a serving system that switches the parallel layout of attention while requests are running. It builds on one observation: modern attention, with few or no KV heads, decouples where a request's KV cache lives from how attention weights are sharded. This has two consequences. First, layouts differ only in who owns the weights and the cache, and most of that state already sits where the next layout needs it; SPLASH reuses it, moves the rest in the background of ongoing inference, and hands off at a batch boundary, making a switch nearly free: its median overhead is under 0.51% of the step it runs in. Second, the decoupling exposes a layout that existing engines lack: Decoupled Ownership Parallelism (DOP) shards attention weights as tensor parallelism does while keeping each request's cache on a single owner as data-parallel attention does. DOP replicates neither, offers 27-60% more KV capacity than data-parallel attention, and gives the scheduler a choice when KV memory limits admission. A transition-aware scheduler follows the best of the four layouts as load changes. On B200 GPUs serving GLM-5.3, SPLASH improves end-to-end serving throughput by 1.3-1.73x over fixed-layout deployments, and the same layout regimes appear with DeepSeek-V3.2 on H200 and GLM-5.3-Flash on DCU.

1 INTRODUCTION

Serving workloads shift from many short requests to a few long ones, but attention layouts trade compute, communication, and cache capacity differently. SPLASH decouples projection placement from cache ownership, exposing DOP and enabling live layout switches while requests continue serving.

1 INTRODUCTION

  • Loads change within a batch: many short requests can give way to a few long reasoning traces, making a fixed attention layout unsuitable for the whole run.The introduction identifies reinforcement-learning rollouts as a sharp example: thousands of generations launch together, while a few long traces hold the group until they finish.Source: §1 · pp. 1–2
  • Serving engines have fixed layouts at launch because changing cache ownership requires preserving live histories and collective order, so they drain requests and restart workers.Long requests can take longest to drain precisely when switching would pay off.Source: §1 · p. 2
  • With few or no KV heads, projection sharding and request-cache ownership become independent choices, allowing layouts to trade compute, communication, and cache room differently.The paper says TP suits few requests in flight, CP suits prompts too large for one rank, and DP-attention suits many independent requests.Source: §1 · pp. 1–2
  • SPLASH introduces DOP, which shards projections while keeping each request’s history on one owner, avoiding the redundant copies in existing layouts.DOP holds the largest cache of the four layouts on a given group.Source: §1 · pp. 2–3
  • SPLASH switches among all four layouts during serving by reusing resident state and moving missing state one layer at a time behind ongoing computation.A transition-aware scheduler follows the fastest feasible layout and switches when another leads by a set margin.Source: §1 · p. 3

2 AN OWNERSHIP MODEL OF ATTENTION LAYOUTS

The ownership model describes each attention layout through two decisions: where projections live and which ranks hold each request’s KV history. It uses these choices to derive per-rank memory and the state a switch must supply.

2 AN OWNERSHIP MODEL OF ATTENTION LAYOUTS

  • Each layout sets projection placement and KV-history ownership independently; under MLA, TP’s head-sharded projections still leave a full copy of each history on every rank.Figure 2 depicts the four layouts and their different placements of projections, histories, and cross-rank traffic.Source: §2 · p. 3, Figure 2 · p. 4
  • DOP stores each projection and history once, while TP duplicates histories and DP-attention and CP duplicate projections, giving DOP the longest balanced context under a per-rank memory budget.The execution-cost ordering changes with context length, so the layout with the greatest memory capacity need not be fastest at every context.Source: §2 · p. 3
  • A switch retains blocks shared by source and destination layouts, then supplies only the destination’s missing state.The model describes each switch as a per-rank difference between the blocks already held and those needed by the destination.Source: §2 · pp. 3–4
  • Three collectives supply switch differences: discard frees unneeded blocks, all-gather collects blocks spread across ranks, and all-to-all exchanges state between request ownership and position sharding.KV all-gathers into TP, moves all-to-all between CP and request-owned layouts, or is discarded in the other five switches; its transfer volume grows with live context.Source: §2 · p. 4

3 DOP: SHARDED PROJECTIONS WITH REQUEST-OWNED KV

DOP combines sharded attention projections with request-owned KV histories, using exchanges to convert between projection and owner layouts. It avoids redundant copies and frees KV capacity, with its benefits depending on load and the cost of its exchanges.

3 DOP: SHARDED PROJECTIONS WITH REQUEST-OWNED KV

  • DOP converts between sharded projections and request-owned attention with variable-size all-to-all exchanges before and after attention.Each head attends over the complete history, so DOP’s output matches TP’s up to floating-point reduction order.Source: §3 · p. 4
  • DOP’s exchanges scale with query rows rather than context length, while its returned KV memory helps throughput when it changes admission.The exchange volume is one row per request in decode or the chunk in prefill; DOP leads for long contexts and large batches, while DP-attention leads over the middle of both axes.Source: §3 · pp. 4–5
  • For GLM-5.3 in FP8 at T = 8, DOP frees 12.69 GiB per rank for KV compared with DP-attention.Sharding the Q-B, KV-B, and O projections across all 78 layers frees memory that DP-attention spends on replicas.Source: §3 · p. 4
  • DOP shares projection shards with TP and request owners with DP-attention, making switches into DOP from either layout a pair of discards.The switch discards state the destination does not need while retaining the shared state.Source: §3 · p. 5, Figure 3 · p. 5

4 SPLASH: LIVE SWITCHING AMONG FOUR LAYOUTS

SPLASH switches attention layouts while serving continues, reusing resident state and transferring missing state layer by layer. Its scheduler chooses among feasible layouts using predicted step latency, with a margin to avoid switching near a crossover.

4.1 SWITCHING WITHIN ONE STEP

  • At a step boundary, each rank builds a per-layer manifest that either reuses and repacks destination-needed blocks or fetches missing blocks with all-gather or all-to-all.Reuse requires matching precision, quantization scales, and position convention.Source: §4.1 · p. 5
  • SPLASH transfers each layer’s missing state while the preceding layer computes, then runs that layer in the destination layout once its state arrives.Ranks switch in one step; requests retain their histories, and layer-associated indexer, position, and recurrent state move with the KV.Source: §4.1 · p. 5

4.2 TRANSITION-AWARE SCHEDULING

  • A layout must fit device memory, and switching must also satisfy a separate memory constraint; choosing batch size and admission jointly with layout remains future work.The feasible set accounts for common state, projections, live KV, scratch space, graphs, and reserve memory.Source: §4.2 · p. 6
  • The scheduler evaluates feasible layouts using their per-step latency and replaces the current layout only when the best candidate leads by a set margin.The cost model combines each layout’s per-rank loads with calibrated hardware parameters; the margin stabilizes the choice near a crossover.Source: §4.2 · p. 6

5 EVALUATION

SPLASH evaluates four attention layouts on B200 and measures their capacity, switch costs, and live-switch overhead. The preferred fixed layout varies with input length and concurrency, while SPLASH overlaps transfers with inference to keep median switching overhead below 0.51% of the destination step.

5.1 EXPERIMENTAL SETUP

  • The B200 evaluation compares TP, CP, DP-attention, and DOP, with GLM-5.3, GLM-5.3-Flash, and DeepSeek-V3.2 across B200, DCU, and H200 deployments.The main-text GLM-5.3 analysis uses 78 layers and FP8 weights and KV; B200 switch studies use B = 16 and 256K context tokens per request.Source: §5.1 · p. 6

5.2 FIXED-LAYOUT PERFORMANCE

  • At B = 16, the leading layout shifts from TP at 1K–4K input to DP-attention at 16K–64K, CP at 128K, and DOP at 256K–512K.Across the four longest-input points, DOP reaches 21,059–23,470 tokens/s, 3.4–10.6% above CP, the runner-up.Source: §5.2 · p. 7, Figure 4 · p. 7
  • At 64K and 128K input, the leading layout changes with batch size; DOP first leads at B = 128 in both sweeps.At 128K and B = 256, DOP delivers 30,724 tokens/s versus CP’s 27,677.Source: §5.2 · p. 7, Figure 4 · p. 7
  • DOP leads at eight of 32 sampled positions, while TP, DP-attention, and CP remain faster elsewhere.At 128K input and B = 1 or 4, DOP trails CP by 42.2% and 35.1%, respectively.Source: §5.2 · p. 7

5.3 KV CAPACITY AND REQUEST ADMISSION

  • On B200 at memory fraction 0.85, DOP holds 8,271,360 distinct KV tokens, 27.3% more than DP-attention and 22.7% more than CP.At 512K context, DOP keeps 16 requests resident rather than 12 for DP-attention; admission gains must be weighed against recurring activation exchange.Source: §5.3 · p. 7

5.5 LIVE SWITCHING OVERHEAD

  • Median SPLASH overhead stays below 0.51% of the destination step, and for networked switches it is 98.24–98.78% lower than blocking.Across twelve directions, SPLASH adds 0.02–11.76 ms at p50; DOP→CP is the largest at 11.76 ms, versus 667.31 ms for blocking.Source: Table 1 · p. 8, §5.5 · pp. 8–9

5.4 SWITCH PRIMITIVE COSTS

  • At B = 16 and 256K tokens per request, KV all-gathers and all-to-alls take 3.471–3.807 ms per layer, while weight all-gathers take 0.292–0.302 ms.Weight-transfer payload is independent of batch size and retained context; KV cost scales with context.Source: §5.4 · p. 8, Figure 5 · p. 8

6 RELATED WORK

Prior work addresses online parallelism changes, parallel-attention layouts, and cache management or disaggregation. SPLASH focuses on attention-layout switching within a fixed worker group and chooses layouts beneath existing serving schedules.

Live parallelism reconfiguration

  • Prior online reconfiguration systems cover different parallelism changes, but some hold the batch during switching or support only a pair of strategies.Reported switch costs range from a few hundred milliseconds for Moebius to seconds for ReMP.Source: §6 · p. 9

Parallel attention and serving schedules

  • Existing attention methods distribute projections or within-context work, while DP-attention and serving schedulers provide other parts of the serving stack.SPLASH adds DOP and selects the attention layout beneath schedules such as Orca and Sarathi-Serve.Source: §6 · p. 9

Cache representation and disaggregation

  • Cache representation, paging, quantization, eviction, and faster kernels change SPLASH’s profiles rather than its placement decision.Sparse and hybrid attention add state that the handoff moves with each layer’s KV; prefill-to-decode KV transfer shares bandwidth with SPLASH switches.Source: §6 · p. 9

7 DISCUSSION AND CONCLUSION

SPLASH treats attention layouts as choices about projection placement and KV-history ownership, enabling DOP and transitions that reuse resident state. Its evaluation finds higher throughput from following changing load regimes, while the cost model describes how layout costs shift with context.

7 DISCUSSION AND CONCLUSION

  • 1.3–1.73× higher end-to-end throughput over fixed-layout deployments comes from following the best layout; DOP leads at 256K–512K and the largest batches.DOP leads by 3.4–11.0% over the strongest alternative and holds 27.3% more KV than DP-attention; the same regimes appear on H200 and DCU.Source: §7 · p. 9
  • A complete switch adds 0.02–11.76 ms end to end, 98.24–98.78% less than a blocking switch wherever state crosses the network.One layer’s communication takes 0.29–0.30 ms for weights and 3.47–3.81 ms for KV at B = 16 and 256K; later layers hide behind computation.Source: §7 · pp. 9–10

A WEIGHT TRANSFER ACCOUNTING

  • For DeepSeek-V3, Q-B, KV-B, and O total 328 MiB per layer in BF16; at T = 8, a sharded-to-replicated step receives 287 MiB per rank.Uniform FP8 halves the parameter payload to 143.5 MiB per rank before scales; these are analytical sizes, not measured migration times.Source: A WEIGHT TRANSFER ACCOUNTING · p. 15

B PER-STEP COST MODEL

  • The stage-wise cost model accounts for arithmetic work, resident projection weights, KV reads, communication, and collective latency, with overlap limited to a budgeted fraction of attention work.The model adds dependent projection and attention costs rather than treating all work as one perfectly overlapping resource maximum.Source: B PER-STEP COST MODEL · pp. 15–16
  • At fixed query chunk size, longer histories can shift the dominant cost from weight reads or fixed communication toward attention computation or KV bandwidth.The attention-work and minimum history-read volumes grow approximately linearly with context length; length alone does not guarantee a crossover between compute and KV-bandwidth limits.Source: B PER-STEP COST MODEL · p. 16

B.1 THEORETICAL SCENARIOS FOR DCU AND H200

The DCU and H200 scenarios plot conditional latency and capacity limits under specified timing and residency assumptions. The accompanying experimental-configuration passages describe separate fixed-layout sweeps on B200, DCU, and H200 systems.

B.1 THEORETICAL SCENARIOS FOR DCU AND H200

  • The DCU scenario’s sampled nominal-input limits are 58.9k, 232.6k, 275.6k, and 402.8k for TP, DP-attention, CP, and DOP, respectively.It assumes B = 16, T = 8, 52.8 GiB per rank for attention weights and latent KV, and a separate 1 GiB transient reserve.Source: §B.1 · p. 16
  • The plotted DCU and H200 crossovers are conditional illustrations, not measured switching thresholds; the scenarios vary hardware alongside model, batch, and memory allocation.The DCU figure models GLM-5.3-Flash INT8 on BW1000 DCUs; the H200 figure models DeepSeek-V3.2 FP8 with B = 32.Source: §B.1 · pp. 16–17

C EXPERIMENTAL CONFIGURATION

  • B200 and H200 experiments ran on rented GPU servers, while all DCU experiments used BW1000 accelerators.The B200 fixed-layout sweeps used GLM-5.3 with FP8 weights, BF16 activations, and FP8 E4M3 KV; prefill used one node and decode used sixteen GPUs across two nodes.Source: C EXPERIMENTAL CONFIGURATION · p. 18, §C.1 · p. 18

C.2 DCU FIXED-LAYOUT SWEEPS

  • The DCU fixed-layout sweeps compare prefill layouts across eight input lengths from 1K to 512K and batch sizes from 1 to 512 at 32K and 64K input.Each point averages batch end-to-end time over 10 runs; Table 5 lists 136 measurements.Source: §C.2 · p. 18

C.3 H200 FIXED-LAYOUT SWEEPS

  • The H200 fixed-layout sweeps compare TP, CP, DP-attention, and DOP as prefill layouts for DeepSeek-V3.2 in a 1P1D deployment.The sweeps cover seven input lengths from 1K to 256K and multiple batch sizes at 32K and 64K; Table 6 lists 116 measurements averaged over 10 runs.Source: §C.3 · p. 18

C.4 KV CAPACITY AND DCU ADMISSION

The DCU admission confirmation compares DOP and DP-attention for a long-context batch, while the accompanying result passages identify the scope and measurement counts of fixed-layout tables.

C.4 KV CAPACITY AND DCU ADMISSION

  • The DCU admission comparison uses three confirmation runs after warmup for GLM-5.3-Flash with INT8 weights and FP8 KV at memory fraction 0.85.The long-context mixed-serving experiment complements the prefill/decode-disaggregated sweeps.Source: §C.4 · p. 18
  • 1.322× burst-throughput gain over DP-attention was measured for DOP at 240K input, 1K output tokens, and 32 requests, alongside higher mean completion latency.Input throughput was 14,846.29 versus 11,228.56 tokens/s, and mean completion latency was 516.84 versus 477.09 s.Source: §C.4 · pp. 18–19

D.1 B200 RESULTS; D.2 DCU RESULTS; D.3 H200 RESULTS

  • Table 4 reports 128 B200 measurements, Table 5 reports 136 DCU layout measurements, and Table 6 reports 116 H200 measurements.The tables report total throughput in 10^3 tokens/s; their captions identify bolded maxima at sampled positions, but these passages do not supply the table values.Source: §D.1 · p. 19, §D.2 · p. 19, §D.3 · p. 19

E DOP IMPLEMENTATION

The section describes DOP’s transfer path and the measurement protocol for switching overhead. It distinguishes logical buffer size from per-rank received bytes, which depend on the state difference between layouts.

E DOP IMPLEMENTATION

  • DOP’s transfer path synchronizes per-owner row counts, supports uneven all-to-all splits, and preserves row identity, head order, and positions.Ranks with no rows still enter both collectives to avoid deadlocks; aligned data uses a vectorized path, while unaligned data uses a scalar fallback.Source: E DOP IMPLEMENTATION · pp. 19, 23
  • The reverse exchange must retain every contribution to the sharded output projection before selecting owner rows.An allreduce or complete reduce-scatter can provide the reduction; an owner-local output GEMM instead requires compatible complete O weights and activations.Source: E DOP IMPLEMENTATION · p. 23
  • The handoff contract includes mutable or auxiliary state beyond KV, including sparse-attention metadata, recurrent state, prefix-sharing references, and CUDA-graph addresses.Recurrent state transfers after the source’s last update; graph addresses must remain stable or graphs must be recaptured.Source: E DOP IMPLEMENTATION · p. 23

F.1 COMMUNICATION AND WORKLOAD SCALING

  • Logical latent-KV buffer volume is not per-rank received bytes; received bytes follow the actual state differences in Equation 2.The inventory must separately account for rotary keys, quantization scales, page metadata, and sparse or recurrent state.Source: §F.1 · p. 23

F.2 SWITCHING OVERHEAD

  • Switching overhead is defined as the measured time minus the target-ready time for a matched execution interval.The protocol reports measured B200 execution-time p50 values and p50(p95) overheads at B = 16 and 256K context.Source: §F.2 · p. 23

G LIMITATIONS

SPLASH has deployment and workload boundaries: graph capture consumes time and memory, switching needs temporary headroom, and transfers may not fully overlap with computation.

G LIMITATIONS

  • Every supported layout needs CUDA graphs captured once at startup, with capture time and resident graph memory accounted for at deployment.CUDA graphs bind weight and KV-buffer addresses.Source: G LIMITATIONS · p. 23
  • A switch temporarily holds one layer’s buffers beyond the larger source or destination footprint, so near full memory SPLASH defers switching or admission.The required headroom is described in Equation 5.Source: G LIMITATIONS · p. 23
  • Transfers share interconnect and memory bandwidth with computation and collectives, so imperfect overlap can lengthen a switching step.Restoring replicated MLA KV when entering TP can move more bytes per layer than one layer’s computation hides.Source: G LIMITATIONS · p. 24
  • When workload regimes change faster than a switch can pay off, SPLASH keeps the current layout and forgoes the opportunity.A destination’s advantage must outlast the switch.Source: G LIMITATIONS · p. 24
  • The DOP transfer path requires equal TP and attention-DP degrees with one rank per owner.Nested CP within a DOP owner, speculative decoding, and heterogeneous workers are outside the design’s scope.Source: G LIMITATIONS · p. 24

H.1 EXPERIMENTAL SETUP

Across the supplied DCU and H200 studies, the best fixed layout varies with input length and batch size, while DOP leads in several long-context or high-concurrency regimes. The switching measurements report low overhead relative to target-ready execution, with the studies covering distinct models, hardware, and workloads.

H.1 EXPERIMENTAL SETUP

  • The DCU fixed-layout study measures GLM-5.3-Flash with DP-attention decode across input-length and batch-size sweeps, reporting end-to-end throughput including queueing, prefill, KV transfer, and decode.It covers eight input lengths from 1K to 512K and nine batch sizes at 32K and 64K input.Source: §H.1 · p. 24, §H.2 · p. 24

H.2 FIXED-LAYOUT PERFORMANCE

  • At 256K–512K input, DOP reaches 21,539–23,436 tokens/s, 10.1–11.5% above CP, and it leads at the largest batches in the stated DCU sweeps.At B = 8, 16, and 32, DOP trails DP-attention by 3.1–6.9% in both batch sweeps.Source: §H.2 · p. 24, Figure 8 · p. 24

H.3 KV CAPACITY AND REQUEST ADMISSION

  • DOP holds 1,118,656 KV tokens per owner versus DP-attention’s 940,224, a 19.0% larger pool; in the co-located run, it overtakes DP-attention between 24 and 32 requests as admission changes.At 32 requests, DOP admits four per owner while DP-attention admits three and queues one.Source: §H.3 · p. 25

H.4 SWITCH PRIMITIVE COSTS

  • On DCUs, the seven KV all-gathers and all-to-alls take 8.732–9.813 ms per layer at B = 16 and 128K tokens per request.The four weight all-gathers take 1.115–1.332 ms per layer and have payloads independent of batch size and retained context.Source: §H.4 · p. 25, Figure 9 · p. 25

I.1 EXPERIMENTAL SETUP

  • The H200 fixed-layout study compares TP, CP, DP-attention, and DOP on DeepSeek-V3.2 with one prefill and one decode instance, using end-to-end batch throughput across four sweeps.The sweeps cover seven input lengths from 1K to 256K and batch sizes up to 256 at 32K input or 128 at 64K input.Source: §I.1 · p. 27, §I.2 · p. 27

I.2 FIXED-LAYOUT PERFORMANCE

  • On H200, the leading layout changes with input length and batch size; across the three reported long-input points, DOP reaches 14,349–18,584 tokens/s, 4.2–10.3% above CP.DOP leads at seven of 29 sampled positions, while other layouts remain faster elsewhere.Source: §I.2 · p. 27, Figure 10 · p. 27

I.3 SWITCH PRIMITIVE COSTS

  • On H200, Figure 11 reports four nonzero weight all-gather costs of 0.6174–0.6526 ms per layer and seven nonzero KV collective costs of 3.6528–4.0020 ms per layer.The DOP→CP reference is 3.8870 ms for the linear Bs context trend.Source: §I.3 · p. 27, Figure 11 · p. 28

I.4 LIVE SWITCHING OVERHEAD

  • SPLASH’s H200 p50 switching overhead is below 1.39% of the corresponding ready p50; across ten network-transfer directions, it reduces p50 overhead by 98.33–98.54% relative to blocking.The reported added time is 0.02–9.65 ms at p50 and 0.07–17.07 ms at p95.Source: §I.4 · p. 28, Table 8 · p. 28

Related papers

Loading 2609.37626v1…