PaperlayerView exact revision on arXiv

Source-linked AI summary

Disaggregated Quantization: Specializing LLM Prefill and Decode

Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh

arXiv:2609.26333v1cs.LG

TL;DR

Prefill constructs the prompt KV cache, while decode consumes it to generate a response. Disaggregated quantization specializes representations, weights, and placement for these phases. Fully disaggregated quantization improves decode-heavy and prefill-heavy performance over non-disaggregated formats, and at 2-bit outperforms LUT2A16 weight-only serving by 4.5–12.5 points while delivering faster prefill computations.

  • Problem

    Prefill constructs the prompt KV cache while decode consumes it, and suitable pre-quantized decoders may not be trainable or need training during QADD.

    Source: §1 · p. 2, §2.6 · p. 6

  • Method

    Disaggregated quantization specializes linear computations and, optionally, weights and placement to prefill and decode, jointly optimizing separate phase-specific weights toward a common response objective.

    Source: §1 · p. 2, §2.4 · p. 5

  • Results

    Fully disaggregated quantization improves decode-heavy and prefill-heavy performance over non-disaggregated formats; at 2-bit, it outperforms LUT2A16 weight-only serving by 4.5–12.5 points while delivering faster prefill computations.

    Source: §3.2 · p. 8

  • Takeaways & Limitations

    Phase-aware specialization allows fast prefill, compact decode, and accurate responses to coexist for local serving.

    Source: §5 · p. 10

    Evaluations do not cover highly batched performance, multi-turn or agentic behavior, or robustness to cache-policy dependence in multi-turn use.

    Source: §5 · p. 10

Abstract

from arXiv · show

Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computation formats, weights and storage placement to both of these phases. On Qwen 3 and Gemma 3, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while matching or exceeding its accuracy at 2-3-bit decode on both decode-heavy and prefill-heavy tasks. With released Qwen3.8-27B GGUF decoders, training an NVFP4 prefiller improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro without modifying the decode checkpoint. To accommodate the additional checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. On the same 27B model, ODP delivers a 1.78x time-to-first-token speedup over the weight-only baseline at 8K prompt length in llama.cpp. We evaluate accuracy under disaggregated serving in vLLM and further validate shared-weight format disaggregation through post-training quantization on models up to 2.8T parameters.

1 INTRODUCTION

Instruction-following separates inference into prompt prefill and response decode, motivating disaggregated quantization (DQ), which specializes linear computations—and optionally weights and placement—to each phase while retaining the pretrained attention architecture.

1 INTRODUCTION

  • DQ specializes prefill and decode computations, and optionally their weights and placement, while retaining the pretrained attention architecture.The approach treats the two inference phases separately while optimizing them toward a shared response objective.Source: §1 · p. 2

1.1 HARDWARE COST

  • Long prefill workloads reuse weights across tokens and become compute-bound, whereas batch-one decode is dominated by weight loading.These different load profiles motivate compute-native formats for prefill and compact weight encodings for decode.Source: §1.1 · p. 2

1.2 CONTRIBUTIONS

  • QADD uses the SFT label mask to select phase-specific computational pathways while training them toward a common response objective.The mask selects prompt prefill versus assistant-response decode pathways, while distillation gradients reach prefill weights through prompt keys and values.Source: §1.2 · p. 3

2. Disaggregated quantization

Disaggregated quantization (DQ) specializes computation formats, weights, and storage placement across prefill and decode through complementary schemes. The section develops their training and serving mechanisms, including streamed prefill weights for single-device use.

2.5 OFFLOADED DISAGGREGATED PREFILL

  • At 8K context, ODP delivers a 1.78× time-to-first-token speedup over a weight-only baseline by streaming prefill weights from SSD and overlapping loading with computation.Two reusable device buffers support the overlap; loading overhead falls relative to compute as prompt length grows.Source: §2 · p. 3, §2.5 · p. 6

2.1 QUANTIZATION-AWARE DISTILLATION WITH DISAGGREGATION

  • QADD uses the SFT label mask to route prompt tokens through prefill and response tokens through decode, training both pathways toward the same response objective.Although the loss supervises response targets, gradients reach prefill weights through prompt keys and values consumed by decode.Source: §2.1 · p. 4

2.2 QUANTIZATION SENSITIVITY DEPENDS ON THE WORKLOAD

  • Decode-only NVFP4 quantization causes 2–4× the accuracy loss of prefill-only quantization on most decode-heavy benchmarks, while the ordering reverses on prefill-heavy tasks for seven of eight models.On decode-heavy tasks, the loss ratio reaches 7× on Gemma3-1B; on prefill-heavy tasks, prefill-only quantization causes 1.1–4.1× the loss of decode-only quantization.Source: §2.2 · p. 4
  • Full disaggregation improves low-bit accuracy on both workload types; format disaggregation primarily helps decode-heavy tasks, with its largest further gains at 2-bit decode on prefill-heavy tasks.Figure 5 reports the family-mean accuracy comparisons after quantization-aware distillation.Source: Figure 5 · p. 5

2.3 FORMAT-DISAGGREGATED QUANTIZATION REDUCES DECODE-PHASE ERROR

  • Format-disaggregated NVFP4 leaves decode activations unquantized, preserving weight storage and prefill cost while improving decode-heavy accuracy over non-disaggregated formats.For LUT formats, format disaggregation restricts NVFP4 autocast to prefill and pairs it with compact weight-only LUT decode.Source: §2.3 · p. 5

2.4 FULLY-DISAGGREGATED QUANTIZATION INCREASES PREFILL-PHASE CAPACITY

  • Fully-disaggregated quantization jointly trains separate native-NVFP4 prefill and weight-only decode checkpoints from the same unquantized model toward a common response objective.The decode checkpoint attends to the prefill-produced cache, retaining the original layer and head dimensions without attention modifications.Source: §2.4 · p. 5

2.5 OFFLOADED DISAGGREGATED PREFILL

  • ODP is mainly suited to local dense LLMs; for mixture-of-experts models, loading can remain more expensive than compute up to extremely high context lengths.The stated limitation follows from the ratio of compute cost to loading cost increasing with the fraction of active parameters.Source: §2.5 · p. 6

2.6 PREFILLERS FOR ARBITRARY WEIGHT-ONLY CHECKPOINTS

  • For an existing frozen weight-only decoder, a trained NVFP4 prefiller adapts to its representations while retaining its compressed decode weights and execution pathway.The decoder’s dequantized weights remain fixed during training, and ODP streams the prefiller without increasing device weight residency.Source: §2.6 · pp. 6–7

3 EXPERIMENTAL SETUP AND LARGE-SCALE VALIDATION

The experiments test phase-specific quantization on instruction-tuned Qwen 3 and Gemma 3, then validate format disaggregation at larger scale. Full disaggregation, trained prefillers, and offloaded prefill show different accuracy, speed, and storage trade-offs.

3.1 EXPERIMENTAL SETUP

  • Core QADD experiments distill against a frozen unquantized teacher using 100M T¨ulu 3 SFT-corpus tokens, with uniform and disaggregated runs sharing the corpus, token budget, and schedule.They use instruction-tuned Qwen 3 and Gemma 3 models; accuracy is evaluated through vLLM and NIXL, while latency is measured on DGX Spark.Source: §3.1 · p. 7

3.2 MAIN QADD-BASED RESULTS

  • Format disaggregation improves decode-heavy accuracy by 1.9 and 3.1 points for NVFP4, 4.5 and 3.7 for LUT3, and 2.5 and 1.8 for LUT2; prefill-heavy effects are under 1.3 points.On prefill, it retains NVFP4 compute speedups of up to 1.49× on Qwen3-8B and 1.67× on Gemma3-12B over BF16; on decode it is 2–3% faster than the non-disaggregated scheme.Source: §3.2 · p. 8
  • For 2-bit models, full disaggregation beats LUT2A16 weight-only serving by 4.5–12.5 points while delivering faster prefill computations.It improves accuracy on both decode-heavy and prefill-heavy tasks over non-disaggregated formats.Source: §3.2 · p. 8
  • ODP removes the device-memory overhead of prefill weights but adds SSD-bandwidth-bound time-to-first-token and slight prefill latency overhead for longer sequences.Above 16K context, offloading raises resident NVFP4 prefill latency by less than 5% on Qwen 3 and 8% on Gemma 3; full disaggregation is less useful for MoE models and when short-sequence TTFT is critical.Source: §3.2 · p. 8
  • An NVFP4 prefiller raises 1-bit decoder accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro, but slightly degrades performance at 3-bit decode.The gains transfer to visual reasoning on MMMU-Pro despite QADD training using no multi-modal examples.Source: §3.2 · p. 9
  • At 8K context, an NVFP4 prefiller cuts Qwen3.8-27B time to first token from 12.27 to 6.90 seconds, a 1.78× speedup over the weight-only baseline.Across 4K–32K contexts, speedups range from 1.38× to 1.78×, while SSD loading makes ODP slower at shorter prompts.Source: §3.2 · p. 9

3.3 LARGE-SCALE VALIDATION WITH PTQ

  • At 4-bit, format disaggregation improves point estimates in 11 of 13 model–benchmark combinations, with six significant gains and no significant degradations.The validation uses NVFP4 PTQ without retraining on models up to 2.8T parameters.Source: §3.3 · p. 9

4 RELATED WORK

Prior work addresses phase-specific precision, accelerated prefill, and adaptation around frozen quantized decoders. DQ unifies these directions, while ODP streams an additional prefill checkpoint from SSD.

4 RELATED WORK

  • DQ combines hardware-native prefill computation with compressed weight-only decode, using shared master weights or separate phase-specialized weights for joint optimization or prefill-only adaptation.Prior approaches include phase-specific weight precision, low-precision prefill with BF16 decode (Wei et al., 2026), and task-specific prefill modules around frozen decoders (Woo et al., 2026).Source: §4 · p. 9
  • Unlike FlexGen’s large-batch weight-transfer amortization (Sheng et al., 2023), ODP streams the extra prefill checkpoint and amortizes loading over prompt length without increasing device weight residency.The authors note that this may complement learned KV-cache adapters, but do not evaluate that combination (Heo et al., 2026).Source: §4 · p. 9

5 CONCLUSION AND LIMITATIONS

The conclusion presents disaggregated quantization as phase-aware specialization of computation, weights, and storage, and notes that its evaluations leave batched, multi-turn, and agentic use untested.

5 CONCLUSION AND LIMITATIONS

  • DQ specializes prefill and decode representations while keeping the pretrained attention architecture, and places prompt-only weights on SSD to make room for a specialized prefill model without increasing device weight residency.The conclusion frames the goal as allowing fast prefill, compact decode, and accurate responses to coexist in local serving.Source: §5 · p. 10
  • The evaluations cover batch-one, single-turn tasks but not highly batched performance, multi-turn or agentic behavior, or robustness to cache-policy-dependent representations.In multi-turn use, cached assistant tokens retain decode-produced KV entries, while rebuilding the cache through prefill can yield different representations for the same history.Source: §5 · p. 10

A.1 QADD SETUP

  • QADD fake-quantizes weights and activations with straight-through estimation and updates FP32 master weights using AdamW; separate-weight configurations maintain optimizer state for both weight sets.The training corpus, token budget, and optimization schedule remain unchanged between experiments.Source: §A.1 · pp. 14–15
  • QADD uses SFT labels to route prompt positions through prefill and assistant positions through decode, while response-target loss also trains prefill weights through prompt keys and values.The final prompt position uses prefill while predicting the first response token.Source: §A.1 · pp. 14–15

A.2 QADD WITH A FROZEN QUANTIZED DECODER AT 27B

  • In QADD, an NVFP4 prefill model is trained separately for each externally quantized decoder; its dequantized-BF16 decoder stays frozen, while prefill weights and normalization scales train and selected parameters remain shared and frozen.The filtered corpus contains 32,608 sequences and approximately 95M tokens, corresponding to 980 training steps. Evaluation omits MMLU-Pro, MMMU-Pro, and MATH because of benchmark overlap.Source: §A.2 · pp. 15–16

A.4 MODELS

  • Qwen3.8-27B has 64 transformer blocks, including 48 gated-linear-attention blocks and full attention in every fourth block, plus a vision tower and multimodal adapter.Per-model hyperparameters are listed in Table 6.Source: §A.2 · p. 15, §A.4 · p. 17

A.2 QADD WITH A FROZEN QUANTIZED DECODER AT 27B

  • The 27B training corpus uses 40K traces generated by the same BF16 model, excludes tool-call or incomplete examples, and contains no multimodal inputs.The filtered corpus uses an 8192-token sequence limit; examples are dropped rather than truncated.Source: §A.2 · p. 16

A.3 QUANTIZATION FORMATS

  • The quantization formats use two-level scaling; NVFP4 uses signed E2M1 values in blocks of 16 with FP8 block scales and an FP32 tensor scale, while LUT grids are optimized for a Gaussian prior.LUTb costs b+0.5 bits per element and NVFP4 costs 4.5 bits; fully disaggregated formats store two checkpoints, while ODP keeps the additional checkpoint on SSD.Source: §A.3 · p. 16

A.5 EVALUATION PROTOCOL

The appendix specifies evaluation protocols across decode-heavy, prefill-heavy, 27B, and large-model settings, then describes an ablation and supporting weight-only comparisons.

A.5 EVALUATION PROTOCOL

  • Decode-heavy QADD accuracy averages GSM8K, MATH-500, and MMLU-Pro results across applicable reasoning modes and the last five checkpoints; its bootstrap intervals capture within-run checkpoint variation, not seed or data-order variance.Qwen 3 is evaluated with reasoning enabled and disabled, while Gemma uses basic chain-of-thought mode.Source: §A.5 · p. 17
  • Prefill-heavy evaluation averages RULER task scores across four context lengths, using 500 examples per task and length, zero-shot prompts, and Qwen 3 with reasoning disabled.The same late-checkpoint and bootstrap procedure as the decode-heavy results is used.Source: §A.5 · p. 17
  • The Qwen3.8-27B evaluation uses MMLU-Pro and MMMU-Pro with reasoning enabled, one complete evaluation per format and benchmark, and the final step-980 prefiller checkpoint.All arms use vLLM on GB300; GGUF decode weights are dequantized to BF16 for evaluation, and fully disaggregated arms transfer attention caches and recurrent state between engines.Source: §A.5 · pp. 17–18

B.1 DISAGGREGATION BEYOND QUANTIZED LINEAR LAYERS

  • The beyond-linear-disaggregation ablation finds no discernible decode-heavy accuracy effect from separately quantizing embeddings, output head, and layer normalizations, which account for 26% of Qwen3-0.6B parameters.Figure 7 compares the configurations; both already specialize quantized linear weights, so the ablation does not isolate that benefit from higher prefill precision.Source: Figure 7 · p. 18, §B.1 · p. 18

A.5 EVALUATION PROTOCOL

  • Large and multimodal PTQ models are evaluated on MMLU-Pro and, where applicable, MMMU-Pro using vLLM on GB300 nodes, with disaggregated engines on separate nodes exchanging KV cache over NVLink.Four arms are reported per model; the W4A4 comparison varies whether activation quantization is applied uniformly or only during prefill.Source: §A.5 · p. 18

B.2 WEIGHT-ONLY QUANTIZATION RESULTS

  • Table 1 reports weight-only LUT3 and LUT2 schemes without NVFP4 autocast, while Figures 14 and 15 break down decode-heavy and prefill-heavy performance, respectively.The table's decode activation-quantization timings estimate the extra operation's overhead, not a complete LUT-to-NVFP4 autocast implementation.Source: §B.2 · p. 19

B.3 PARETO ANALYSIS

The Pareto analysis compares decode-heavy accuracy with encoded decode-weight size and treats prefill-checkpoint residency and timing conventions separately.

B.3 PARETO ANALYSIS

  • For LUT autocast prefill timing, BF16 is the weight-only reference and NVFP4 timings are reused without including weight-conversion overhead.Decode timings for LUT formats estimate the overhead of an added activation-quantization call, rather than a complete autocast implementation.Source: §B.3 · p. 19
  • Full disaggregation improves low-bit decode-heavy accuracy without enlarging decode weights, shifting the accuracy–decode-weight-size trade-off for Qwen 3 and Gemma 3.Figure 8 plots accuracy against encoded quantized decode-weight size; the added prefill checkpoint and its residency cost are accounted for separately in Table 1.Source: §B.3 · p. 19
  • Comparisons across model sizes exclude differing amounts of unquantized embedding and head weights.The figure's encoded decode-weight sizes do not represent all model parameters.Source: §B.3 · p. 19

B.4 INTEROPERABILITY OF PREFILLERS

The interoperability analysis exchanges trained NVFP4 prefill checkpoints across frozen Qwen3.8-27B decoders and compares them with a common RTN prefiller. Training-matched prefillers improve accuracy most at the lowest bitwidths, though cross-pairing results are not uniform.

B.4 INTEROPERABILITY OF PREFILLERS

  • On MMMU-Pro, IQ1 S decoder accuracy falls from 59.65% with its own prefiller to 52.60% with IQ1 M and 47.05% with IQ2 XXS, while IQ1 M reaches 62.49% with either its own or IQ1 S prefiller.The pattern is not universal: IQ1 S decoder MMLU-Pro accuracy rises from 61.54% to 65.02% with the IQ1 M prefiller.Source: §B.4 · p. 19
  • Against common RTN prefill, training-matched prefillers improve MMLU-Pro by 13.1, 4.2, and 1.9 points for IQ1 S, IQ1 M, and IQ2 XXS; MMMU-Pro gains are 29.4 and 9.7 points for IQ1 S and IQ1 M, while IQ2 XXS changes −0.3 points.Source: §B.4 · p. 19
  • Table 7 fixes the decode checkpoint by row and varies the prefill checkpoint by column; bold identifies the training-matched prefiller, not necessarily the highest score.Asterisks mark same-row differences from the bold cell under a two-sided exact McNemar test without multiple-comparison correction.Source: Table 7 · p. 20

MMMU-Pro

  • Table 8 reports generation-length and truncation statistics for weight-only and trained-prefiller arms, with token counts including reasoning and final-answer generation.Each arm uses one complete evaluation with the same quantized checkpoints as Table 5.Source: Table 8 · p. 20, MMMU-Pro · p. 20

B.5 GENERATION LENGTH WITH PREFILLERS

Prefillers can change response lengths as well as accuracy, and the effect varies by workload and format. On MMMU-Pro, some low-bit formats gain accuracy while generating substantially fewer tokens, but shorter typical responses do not always mean shorter mean responses.

B.5 GENERATION LENGTH WITH PREFILLERS

  • Generation lengths and truncation rates are measured for Qwen3.8-27B evaluations using the final prefill checkpoint at step 980 and a common 32,768-token generation budget.The evaluations correspond to Table 5.Source: §B.5 · p. 20
  • On MMMU-Pro, IQ1 S’s mean response length fell 54.8%, from 14,579 to 6,582 tokens, while accuracy rose from 24.4% to 59.7%; its median length fell from 10,966 to 2,393 tokens.Source: §B.5 · p. 20
  • 17.6 accuracy points: IQ1 M gains this much on MMMU-Pro while using 14.2% fewer tokens on average.The reported improvements are not bought with more generated tokens.Source: §B.5 · p. 21
  • On MMLU-Pro, seven of eight formats shorten median generations by 3–21% but increase mean length by 5–63%; on MMMU-Pro, the other six formats also increase mean length.Longer upper tails outweigh shorter typical responses on MMLU-Pro; unchanged decode weights and per-token pathways do not guarantee lower generation cost.Source: §B.5 · p. 21

C.1 PREFILL MEASUREMENTS

The prefill measurements compare BF16 and NVFP4 transformer stacks, with ODP streaming prefill weights from SSD under a cold-load and restoration protocol. The reported speedup decomposition shows that non-projection work dilutes projection gains.

C.1 PREFILL MEASUREMENTS

  • The measurements use a single NVIDIA GB10, and llama.cpp TTFT measurements are separate from the transformer-stack benchmarks.The TTFT setup uses three repetitions after one warmup, disables prompt caching, and uses Unsloth’s IQ1 S checkpoint as the native llama.cpp baseline.Source: §C.1 · p. 21
  • The timed region runs from the first transformer layer’s input through the last layer’s output, includes attention, and excludes embedding lookup, RoPE-table construction, final normalization, and the output head.BF16 uses cuBLAS; NVFP4 uses activation quantization with static global scales and shape-tuned CUTLASS or FlashInfer GEMMs, with compatible projections and activation processing fused.Source: §C.1 · p. 21
  • ODP streams prefill weights from SSD through pinned host buffers into two device slots that borrow decode-weight memory.Timing includes the first block’s cold load and ends after restoration; total SSD traffic is 3.64 + 0.20 GB for Qwen3-8B and 5.64 + 0.23 GB for Gemma3-12B.Source: §C.1 · p. 21
  • 3.0–3.4× projection speedups are diluted by attention, normalization, activation processing, and quantization.Attention accounts for 37% of NVFP4 device time on Qwen3-8B and 13% on Gemma3-12B; fused timings do not isolate activation-quantization overhead.Source: §C.1 · p. 21
  • Table 10 reports prefill transformer-stack speedup over BF16 for Qwen 3 and Gemma 3, while Table 11 reports per-component latency at S=16384.The NVFP4 and +ODP arms are measured in the same benchmark session; Table 11 groups fused operations and gives full-stack latency divided by layer count.Source: Table 10 · p. 22, Table 11 · p. 22

C.2 DECODE KERNELS

The decode-kernel measurements compare weight formats under batch-one, bandwidth-bound generation. Format-disaggregated NVFP4 skips activation quantization and slightly improves on NVFP4’s speedup over BF16, while whole-model speedups remain diluted by work that the formats do not accelerate.

C.2 DECODE KERNELS

  • Figure 9 reports end-to-end single-user decode speedup over dense BF16 on DGX Spark, including attention, normalization, and the BF16 language-modeling head.The benchmark covers NVFP4, NVFP4A16, LUT3, and LUT2 kernels.Source: §C.2 · pp. 22–23, Figure 9 · p. 23
  • Per-output-token latency is estimated from the time difference between generations of 128 and 8 output tokens, divided by their token-count difference.Measurements are end-to-end vLLM generations at batch one, with a 168-token prompt, three repetitions after warmup, prefix caching disabled, and CUDA graphs enabled.Source: §C.2 · p. 23
  • At batch one, decode is bandwidth-bound; the compared formats use 16 bits per weight for BF16, 4.5 for NVFP4, 3.5 for LUT3, and 2.5 for LUT2.NVFP4 uses an FP4×FP4 GEMM after activation quantization; NVFP4A16 keeps BF16 activations, while LUT3 and LUT2 reconstruct bit-planed weights through a shared-memory lookup table.Source: §C.2 · p. 23
  • 2.86× to 2.93× on Qwen3-8B and 3.18× to 3.27× on Gemma3-12B: format-disaggregated NVFP4 improves speedup over BF16 by skipping activation quantization.It is identical to NVFP4A16 on decode and uses a kernel tailored to the weight-only matrix-vector product.Source: §C.2 · p. 23
  • LUT2 moves 6.4× fewer weight bits than BF16 yet delivers 3.82× and 4.15× speedups, respectively.The BF16 output head and shared operations such as embedding lookup, attention, normalization, and residual adds dilute whole-model speedup; the LUT2 kernel reaches 96% of the bandwidth its weight traffic permits.Source: §C.2 · p. 23

D FULL EVALUATION RESULTS

The full-evaluation figures organize QADD breakdowns by benchmark and training step for decode-heavy tasks, and by context length and training step for prefill-heavy tasks. The supplied passages describe these figure scopes but do not report their outcomes.

D FULL EVALUATION RESULTS

Loading 2609.26333v1…