PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving

Haoyu Zheng, Fangcheng Fu, Binhang Yuan, Yongqiang Zhang, Liang Deng, Hao Wang, Yuanyuan Zhu, Xiao Yan, Jiawei Jiang

arXiv:2610.00499v1cs.LG

TL;DR

Scalar proxies discard the block-and-step structure that matters for estimating dLLM inference cost. DWS preserves that structure, with a prompt-only predictor trained coarse to fine. DWS achieves up to 2.50× lower cost MAE than common scalar-based predictors, while its scheduler reduces online chatbot end-to-end latency by up to 1.92× over FCFS.

  • Problem

    Accurate per-request inference-cost estimates guide serving request management, but output length and denoising-step totals discard heterogeneous block-step costs.

    Source: §1 · pp. 1–2

  • Method

    DWS preserves the two-dimensional structure of output blocks and denoising steps, while a prompt-only predictor uses coarse-to-fine training to predict it.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    2.50× lower cost MAE than common scalar-based predictors; for online chatbots, DWS-guided SJF reduces end-to-end latency by up to 1.92× over FCFS.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DWS separates request-dependent execution behavior from deployment-specific cost factors, supporting transfer across deployments through lightweight profiling without retraining its predictor.

    Source: §8 · p. 10

    Transfer assumes a fixed model family, block size, and decoding threshold; changing these requires retraining using activity matrices logged from routine serving.

    Source: §5 · p. 7

Abstract

from arXiv · show

As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world \textit{serving}, where request management (such as scheduling and resource allocation) relies on accurate estimation of per-request inference cost. However, common cost proxies fall short for dLLMs: output length ignores that one forward pass can unmask multiple tokens, and denoising-step count ignores the \textit{heterogeneous} per-step costs. We observe that the block-autoregressive generation mechanism induces a two-dimensional execution structure over output blocks and within-block denoising steps, whereas these proxies collapse it into a scalar, discarding information essential for characterizing the cost. Motivated by this insight, we propose the Denoising Workload Surface (DWS), which preserves this two-dimensional block-step structure as a probability surface to weight the heterogeneous per-step costs. We then design a coarse-to-fine training scheme that enables a lightweight prompt-only predictor to accurately predict the complex DWS. This predictor runs efficiently even on a single CPU core, avoiding GPU contention with the serving model. Since DWS decouples request-dependent execution behavior from deployment-specific cost factors, the predictor transfers across hardware configurations without retraining. In \textit{real-world} serving experiments, DWS reduces cost-prediction error by up to $2.50\times$ over scalar-based predictors, while the DWS-guided shortest-job-first scheduler reduces end-to-end latency by up to $1.92\times$ for online chatbots.

1 INTRODUCTION

dLLM serving needs accurate per-request cost estimates, but output length and denoising-step count each discard important workload variation. The paper introduces DWS and a prompt-only predictor to preserve and estimate that structure, then evaluates its prediction and serving benefits.

1 INTRODUCTION

  • Accurate request-cost estimates matter because scheduling, load balancing, and resource allocation rely on them; prior AR-LLM SJF work uses output length as a cost proxy.In AR generation, each generated token corresponds to one decoding forward pass, making output length a natural workload proxy.Source: §1 · p. 1
  • Output length misses that a dLLM denoising step can unmask multiple tokens, while equal-length requests showed inference times differing by up to 11.47×.The demonstration used LLaDA2.0-mini and served requests individually to avoid batching effects.Source: §1 · pp. 1–2
  • Denoising-step count is also inaccurate because it treats steps as equal despite position-dependent costs, especially across within-block step indices.The count correlates with inference time, but requests with the same number of steps can still have considerably different costs.Source: §1 · p. 2
  • DWS preserves the block-by-step execution structure and weights its fine-grained cells; a prompt-only predictor reconstructs it using separate distributions and coarse-to-fine training.The predictor is designed for CPU efficiency, and decoupling request behavior from deployment-specific cost factors supports transfer across deployments without new training data or retraining.Source: §1 · p. 2
  • Up to 2.50× lower cost MAE than common scalar-based predictors; DWS-guided SJF reduces online-chatbot end-to-end latency by up to 1.92× over FCFS.The scheduler also reduces chatbot TTFT by up to 2.22× and accelerates offline sample collection by up to 2.73× over FCFS.Source: §1 · p. 3

2 PRELIMINARIES

Serving-oriented dLLMs commonly generate autoregressive output blocks and denoise tokens within each block in parallel. Completed blocks trigger KV refresh, while blockwise termination supports variable-length generation.

2 PRELIMINARIES

  • Within each output block, dLLMs iteratively denoise masked tokens in parallel, with each denoising step performing one forward pass and revealing one or more tokens.The block size is typically fixed during inference; generation stops when a fully denoised block produces an [EOS] token, otherwise another masked block is appended.Source: §2 · p. 3
  • Variable-length block generation avoids computation on padded positions, but its request-to-request workload variation makes cost-aware scheduling beneficial.Modern serving-oriented dLLMs commonly use block-autoregressive generation.Source: §2 · p. 3
  • After a block is fully denoised, its KV states must be refreshed and cached because bidirectional attention changes those states during denoising.KV caching completed context avoids redundant computation.Source: §2 · p. 3

3 COST MODELING

The cost model focuses on prefill, KV refresh, and denoising, preserving the block-step structure to account for heterogeneous denoising costs. It combines these modeled components to estimate total inference cost.

3.1 ANATOMY OF DLLM INFERENCE COST

  • Denoising averaged 80.8% of latency, while prefill and KV refresh averaged 5.8 ± 14.4% and 6.7 ± 4.7%; the model therefore includes all three components.Other stages were small and stable, whereas prefill and KV refresh varied substantially across requests.Source: §3.1 · p. 3
  • Prefill and KV-refresh costs are modeled linearly in prompt-block count m and output-block count n, respectively, and both fits have R2 ≥ 0.999 through 32K tokens.Prefill processes each prompt block once; KV refresh occurs once per newly completed output block.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2 MODELING THE DENOISING COST

  • A fixed-size activity matrix A records which block-step cells a request reaches, separating denoising activity from each cell’s cost.Cell-cost modeling accounts for position (b, s) and prompt length m; jointly calibrated cell costs approximate denoising latency.Source: §3.2 · p. 4
  • R2 = 0.9956 and MAPE = 5.01% across 5-fold validation on 20K real traces show that the calibrated model accurately approximates denoising cost.The model jointly calibrates W and δ from request-level denoising latency.Source: §3.2 · p. 4
  • The total cost combines the modeled prefill, KV-refresh, and denoising components; m is observed from the prompt, while α, β, and K(·) depend on deployment configuration and can be profiled.The cost model retains the simpler linear models for prefill and KV refresh while representing denoising with its finer block-step structure.Source: §3.2 · pp. 4–5, §3.1 · p. 4

4 PREDICTION OF THE DENOISING ACTIVITY

DWS represents expected denoising activity as a probability surface over output blocks and within-block steps. Its predictor reconstructs this structure from prompt-derived predictions using a staged training scheme.

4.1 PREDICTION TARGET

  • DWS is the probability that each block-step cell is active, forming a surface over block-step space.The surface is the expected activity matrix D(x) conditioned on prompt x.Source: §4.1 · p. 5
  • Factorization expresses each cell’s activity as block survival multiplied by conditional within-block step survival, preserving the decoding constraints by construction.The passage states that this factorization loses no expressiveness.Source: §4.1 · p. 5

4.2 DWS PREDICTOR

  • The predictor encodes prompt semantics, predicts block-count and per-block step distributions, then converts them to survival probabilities to reconstruct DWS.The heads predict block count and denoising steps conditional on reaching each block.Source: §4.2 · p. 5
  • Two-stage training adapts the encoder with multi-horizon workload supervision before directly training the heads on reconstructed DWS.Increasing the horizon progressively exposes deeper within-block denoising behavior, while Stage B supervises the final workload target.Source: §4.2 · p. 6
  • INT8 quantization speeds prediction 2.32× on one CPU core with a 0.54% degradation in total-cost MAE; profiled cost factors then convert predicted DWS into cost.Prompt length supplies m(x) for the cost calculation.Source: §4.2 · p. 6

5 DISCUSSION

DWS retains denoising structure that scalar targets discard and supports cost estimation and serving decisions across deployments. Its stated scope assumes fixed model-family and decoding configurations.

5 DISCUSSION

  • DWS supports request-level cost-aware scheduling, block-level grouping or preemptive scheduling, and KV-cache estimation and memory allocation.The block-level uses rely on predicted within-block steps or remaining workload.Source: §5 · p. 6
  • DWS decouples request execution structure from deployment costs, allowing profiling factors to be refreshed without retraining the predictor.An unchanged DWS predictor can improve serving performance across deployments.Source: §5 · p. 6
  • Individual-request profiling isolates per-request costs, but batching and runtime factors such as KV-cache reuse may change cell costs without changing DWS or its predictor.With batch size 96, DWS-guided scheduling consistently performs best.Source: §5 · pp. 6–7
  • The cost model omits small stable stages and secondary effects such as quadratic attention; reported residuals are below inevitable prediction error.This parsimony boundary is explicitly quantified in Appendix D.Source: §5 · p. 7
  • Transfer assumes a fixed model family, block size, and decoding threshold; changing these requires retraining using activity matrices logged during routine serving.The paper presents these settings as typically static deployment configurations.Source: §5 · p. 7

6 EXPERIMENTAL EVALUATION

Across online chatbot and offline synthetic-data-generation serving, DWS-SJF improves scheduling results, while DWS improves cost-model and predictor accuracy. Evaluations use two dLLMs, multiple datasets, and disjoint predictor test prompts.

6.1 EXPERIMENTAL SETTINGS

  • Experiments use LLaDA2.0-mini and LLaDA2.0-flash, serving datasets LMSYS-Chat-1M and ShareGPT, and synthetic Alpaca; predictor train and test prompts are disjoint.Predictor and cost-model evaluations use LMSYS-Chat-1M; serving experiments include all three datasets.Source: §6.1 · p. 7

6.2 COST MODEL EVALUATION

  • DWS-full reduces overall cost MAE by 2.04× on average over Total Steps, while DWS-denoising reduces MAE by up to 1.80× across all four cost ranges.Total Steps consistently outperforms Output Length, but retains error by treating denoising steps uniformly.Source: §6.2 · p. 7

6.3 PREDICTOR EVALUATION

  • DWS achieves cost MAE 1.22–2.50× lower than scalar predictor baselines; at similar total-step MAE, it reduces cost MAE by 1.46× versus the total-step predictor.Direct Cost performs better than other scalar baselines but entangles prediction with deployment-specific costs and requires retraining across deployments.Source: §6.3 · p. 8, Table 1 · p. 8
  • Removing Stage A or multi-horizon supervision increases cost MAE by 1.70× or 1.25×; INT8 quantization improves throughput 2.32× with less than 1% cost-MAE accuracy loss.The table caption distinguishes total-cost error from denoising-structure error.Source: §6.3 · p. 8, Table 1 · p. 8

6.4 REAL-WORLD SERVING EVALUATION

  • DWS-SJF reduces online chatbot E2E latency and TTFT by up to 1.92× and 2.22× over FCFS, and performs best on both offline metrics across serving models and datasets.On LMSYS-Chat-1M, online gains over the strongest baseline are up to 1.50× and 1.65×; on LLaDA2.0-mini, it completes 1K requests 2.73× faster and 2.51× more requests within 3 minutes than FCFS.Source: Figure 7 · p. 8, §6.4 · p. 9, Table 2 · p. 9
  • Under light online load, schedulers perform similarly; as load rises, DWS-SJF grows slowest and maintains the lowest E2E latency and TTFT on both datasets.PARS-Cost (CPU) falls behind because slow prediction delays estimates and can lead the scheduler to select expensive requests.Source: Figure 7 · p. 8, §6.4 · p. 9

7 RELATED WORK

The related work describes dLLM serving systems focused on runtime execution and positions DWS as a complementary pre-inference cost signal for serving decisions.

7 RELATED WORK

  • Further discussion of dLLM inference acceleration and AR LLM serving is deferred to Appendix M.Source: §7 · p. 9
  • dLLM-Serve targets memory and resource efficiency through phase-aware execution, while Optimus and BlockServe address dynamic loads and heterogeneous convergence.The latter systems use adaptive decoding and batching.Source: §7 · p. 9
  • DiLaServe targets SLO-aware serving with deadline-aware scheduling and adaptive resource management; DWS supplies a pre-inference cost signal complementary to these runtime optimizations.Source: §7 · p. 9

8 CONCLUSION

DWS preserves the block-by-step execution structure that scalar proxies discard, supporting more faithful inference-cost estimates. The appendices validate linear prefill and KV-refresh cost models and identify heterogeneous per-step denoising costs.

8 CONCLUSION

  • DWS preserves heterogeneous computation across output blocks and denoising steps, enabling more faithful request-cost estimation and deployment adaptation through profiling.The conclusion also notes that its prompt-only predictor runs on a single CPU core.Source: §8 · p. 10

A NOTATION CONVENTIONS

  • Denoising cells are indexed by output-block b and within-block step s; starred quantities are measured costs, while unstarred quantities belong to the cost model.D(x) and its predicted value denote the prompt-conditioned target and prediction.Source: A NOTATION CONVENTIONS · pp. 13–14

B.1 COST DRIVERS

  • Prompt-block count m explains 99.87% of prefill-latency variation, while output-block count n explains 99.26% of KV-refresh variation.Interaction contributions are below 1% for both components.Source: §B.1 · p. 14

B.2 LINEAR COST MODELS

  • R² is 0.9996 for the linear prefill model and 0.9990 for the linear KV-refresh model across the supported context range.The linear forms use prompt-block count m and output-block count n, respectively.Source: §B.2 · pp. 14–15
  • For requests with at least 16 prompt blocks, prefill reaches 23.9% of latency; as output blocks grow, denoising rises from 53.4% to 88.4%.Prefill and KV refresh together account for 38.3% of latency for single-block outputs.Source: §B.2 · p. 15

C ANALYSIS OF THE PER-STEP DENOISING COST

C.1 MODEL ARCHITECTURE

The analysis attributes within-block denoising-cost variation to dynamic expert routing and context growth, while finding that the linear component models remain useful, including for long contexts. Residual cost-model errors largely preserve request ordering, and higher-order terms add limited fidelity for prefill and KV refresh.

C.1 MODEL ARCHITECTURE

  • Average forward latency rises from 6.043 ms to 8.622 ms between the first and last denoising steps, alongside increased routed-expert latency and activated experts.Routed-expert latency rises from 3.038 ms to 5.670 ms, while attention latency remains nearly unchanged; activated experts increase from 32.7 to 66.3 on average.Source: §C.1 · pp. 15–16

C.2 CONTEXT LENGTH

  • As input context grows from 256 tokens to approximately 32K, attention-kernel latency rises from 0.6091 ms to 1.4104 ms and forward latency from 6.5730 ms to 7.3447 ms.Routing mainly drives cost variation across within-block steps, while context growth changes costs across requests and output-block positions.Source: §C.2 · p. 16

C.3 IMPLICATIONS FOR SERVING COST MODELING

  • DWS applies to both dense and MoE dLLMs, with greater modeling benefit for MoE models where per-step costs are more heterogeneous.Its cell-cost profile captures position-dependent costs, while prompt-length correction and block-indexed costs model context effects.Source: §C.3 · p. 16

D.1 EFFECT OF OMITTED STAGES

  • The modeled cost achieves MAPE of 4.30%, R² = 0.9967, Spearman’s ρ of 0.9991, and 99.21% pairwise ordering accuracy.This residual analysis uses ground-truth workload inputs and includes omitted stages, which account for 6.7 ± 1.4% of latency.Source: §D.1 · pp. 16–17

D.2 PREFILL COST UNDER LONG PROMPTS

  • For 8–16K prompts, the quadratic prefill model reduces MAPE from 2.00% to 1.47%, but for 16–32K prompts the gain shrinks to 1.61% → 1.52%.The linear model remains a close fit across LLaDA2.0’s supported context range.Source: §D.2 · p. 17, Table 6 · p. 17

D.3 KV-REFRESH COST UNDER LONG OUTPUTS

  • For 16–32K outputs, quadratic KV-refresh modeling reduces MAPE by only 0.77 percentage points, so the main model retains its linear form.Linear-model MAPE is 10.17% in that range, versus 9.40% for the quadratic alternative.Source: §D.3 · pp. 17–18, Table 7 · p. 18

D.4 DENOISING COST UNDER LONG CONTEXT

The appendix tests the denoising cost model under long contexts, details online cost profiling and predictor training, and evaluates CPU quantization and workload diversity. Results support the retained model forms and show that INT8 improves single-core prediction throughput with little accuracy degradation.

D.4 DENOISING COST UNDER LONG CONTEXT

  • For 16–32K prompts, quadratic correction reduces denoising-cost MAPE from 3.57% to 2.08%, while the linear correction stays below 3.6% across ranges.For 16–32K outputs, the block-indexed baseline remains below 5.8% MAPE without an added output-length correction.Source: §D.4 · pp. 18–19, Table 9 · p. 19

E AUTOMATIC PROFILING AND REFRESH OF COST FACTORS ONLINE

  • Cost factors are calibrated from execution traces and periodically refreshed without retraining the DWS predictor; hardware changes require a new profile, while decoding-configuration changes may require retraining.Changing core decoding configuration, such as block size, reshapes the denoising structure captured by DWS.Source: E AUTOMATIC PROFILING AND REFRESH OF… · pp. 19–20

F PREDICTOR QUANTIZATION AND CPU EFFICIENCY

  • INT8 raises single-core throughput from 36.35 to 84.29 requests/s (2.32×), with final cost MAE increasing by only 0.54% versus FP32.The study adopts INT8 for downstream serving experiments; throughput does not rise monotonically as precision falls.Source: F PREDICTOR QUANTIZATION AND CPU… · p. 21, Table 10 · p. 21

G CHARACTERISTICS OF LLM SERVING WORKLOADS

  • At the same median prompt length of 42 tokens, LMSYS-Chat-1M and ShareGPT have median inference costs differing by more than 2×.LMSYS-Chat-1M and Alpaca also have means larger than their median costs, indicating right-skewed distributions.Source: Figure 10 · p. 21, G CHARACTERISTICS OF LLM SERVING… · p. 22

H PREDICTOR TRAINING DETAILS

  • The predictor uses coarse multi-horizon workload supervision before direct DWS supervision, with the multi-horizon targets capturing binned within-block step activity.Replacing multi-horizon targets with a single total-step target increases cost MAE by 1.25×.Source: H PREDICTOR TRAINING DETAILS · pp. 22–23

I ADDITIONAL SERVING RESULTS

DWS-SJF approaches Oracle-SJF across the reported offline settings, while additional serving results show gains over FCFS and MiniLM-based PARS-Cost. Encoder experiments expose a trade-off between prediction accuracy and CPU throughput, motivating MiniLM-L12 as the default.

I ADDITIONAL SERVING RESULTS

  • DWS (INT8)-SJF reduces every reported metric over FCFS across three datasets, with more modest but still positive P90 latency and TTFT gains.The authors attribute the smaller tail gains to expensive requests being deprioritized by design; with aging, average latency does not improve at the cost of starvation.Source: I ADDITIONAL SERVING RESULTS · pp. 23–24
  • 83.8%–94.3% of Oracle-SJF’s improvement over FCFS is recovered by DWS-SJF across offline settings.This also holds on unseen Alpaca and LLaDA2.0-flash, although training used LMSYS-Chat-1M traces with LLaDA2.0-mini.Source: Table 12 · p. 24, I ADDITIONAL SERVING RESULTS · p. 24
  • DWS (INT8)-SJF outperforms MiniLM-based PARS-Cost across all metrics on three datasets, indicating gains beyond using a lightweight predictor.The baseline uses the same MiniLM encoder, INT8 quantization, and single CPU core; replacing its BERT-base encoder with MiniLM generally weakens its scheduling performance.Source: I ADDITIONAL SERVING RESULTS · p. 24

J EFFECT OF PREDICTOR ENCODER

  • MiniLM-L6 raises cost MAE 27% while boosting CPU throughput 1.95×; BERT-base and Qwen3-0.6B cut MAE 5% and 9% but reach 0.30× and 0.06× MiniLM-L12 throughput.The authors select MiniLM-L12 as a balance of accuracy and CPU efficiency.Source: J EFFECT OF PREDICTOR ENCODER · p. 25

K CROSS-DEPLOYMENT EXPERIMENT

The cross-deployment experiment reuses the trained DWS predictor and refreshes only deployment-specific cost factors, retaining serving gains without retraining. The accompanying analysis explains scalar-proxy bias and the assumptions behind transfer across deployments.

K CROSS-DEPLOYMENT EXPERIMENT

  • DWS-SJF improves online and offline serving after transfer to a substantially different deployment, reusing the predictor unchanged and refreshing only its deployment-specific cost profile.The tested deployments differ in GPU architecture, tensor parallelism, CPU platform, memory hierarchy, interconnect, and software stack.Source: K CROSS-DEPLOYMENT EXPERIMENT · pp. 25–26, Table 15 · p. 26

L THEORETICAL ANALYSIS OF DWS

  • The formal transfer setup fixes the serving model, block size, and decoding configuration, so deployments share an execution distribution and differ only in profiled cost factors.The analysis also notes that changing the serving model within a family can shift the target DWS.Source: L THEORETICAL ANALYSIS OF DWS · p. 26

L.1 BIAS OF SCALAR PREDICTION TARGETS

  • A total-step proxy is unbiased only when its constant matches the prompt-specific effective mean step cost, which varies across requests.The output-length proxy additionally assumes a fixed number of denoising steps per generated block.Source: §L.1 · p. 27
  • With four denoising steps, two traces cost 20 ms and 29 ms; the total-step proxy ties them, while output length reverses their true cost order.The example uses four blocks with one step each versus one block with four steps.Source: §L.1 · p. 27
  • A constant rescaling of a direct-cost predictor generally cannot remove deployment bias, whereas DWS requires only re-profiling deployment-specific factors.The rescaling works in the stated example only when the new deployment uniformly rescales the original cost factors.Source: §L.1 · p. 27

L.2 PROOF OF THE ERROR-CONTROL BOUND

The analysis establishes properties of the DWS prediction objective and factorized representation, while qualifying transfer when the execution distribution changes. It also points to block-level preemption as a possible application of DWS predictions.

L.2 PROOF OF THE ERROR-CONTROL BOUND

  • The true DWS is the unique minimizer of the Brier risk, whose excess risk decomposes into the mean squared distance between predicted and true DWS.The risk and predicted-cost setup is defined for a predictor that maps prompts to DWS matrices.Source: §L.2 · p. 28
  • The error bound assumes a fixed serving model and decoding configuration; a changed execution distribution adds a DWS-shift term, with limited measured impact within one model family.The reported comparison reuses a predictor trained on LLaDA2.0-mini on LLaDA2.0-flash without retraining.Source: §L.2 · p. 28

L.3 EXPRESSIVE COMPLETENESS OF THE FACTORIZED DWS

  • The factorized parameterization represents every execution-induced DWS, and this completeness suffices for cost estimation without modeling the joint distribution of activity matrix A.The cost depends on A through its cell-wise activity probabilities.Source: §L.3 · pp. 28–29

M ADDITIONAL RELATED WORK

  • DWS’s block-level predictions could support preemptive scheduling at block boundaries, where no partially denoised state needs to be preserved.The passage presents DWS-guided preemptive scheduling as future work.Source: M ADDITIONAL RELATED WORK · p. 29
Loading 2610.00499v1…