PaperlayerView exact revision on arXiv

Source-linked AI summary

Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware

Farhana Amin, Sabiha Afroz, Mona Moghadampanah, Dimitrios S. Nikolopoulos

arXiv:2608.23807v1cs.AI

TL;DR

Existing dLLM serving work lacks measurement of concurrent real-hardware behavior, which matters because AR-serving assumptions may not apply. This paper characterizes LLaDA-8B-Instruct with D2F on a single H200 across GSM8K and HumanEval, finding discrete unpredictable costs, CPU-dominated latency, and a 16.0× throughput gain from step-level sharing.

  • Problem

    Concurrent serving behavior for masked diffusion language models has not been sufficiently characterized under real hardware load, despite differing from autoregressive decoding.

  • Method

    The paper uses measurement-driven single-GPU characterization and derives scheduling principles from measured dLLM service-time and dispatch behavior.

  • Results

    16.0× throughput improvement results from sharing one forward pass per denoising step over advancing requests independently.

  • Takeaways & Limitations

    Efficient dLLM serving requires step-level parallelism, with admission and eviction designed around an already shared forward pass rather than copied AR-serving assumptions.

  • Takeaways & Limitations

    The appendix notes that the scheduling simulation’s 8-batch sample was not converged: a 32-batch replication shifted estimated service capacity by about 24 percent.

Abstract

from arXiv · show

Masked diffusion language models (dLLMs) can in principle generate text faster than autoregressive (AR) models, since they denoise many tokens at once. Recent systems have begun building serving infrastructure for dLLMs, but none first measure how these models behave under real, concurrent serving load. Serving systems built without this grounding risk carrying over assumptions from AR serving that may not hold for dLLMs. We characterize dLLM serving to close this gap, using LLaDA-8B-Instruct with a D2F (Discrete Diffusion Forcing) LoRA adapter on a single NVIDIA H200 GPU, evaluated on GSM8K and HumanEval. We report three findings. First, request difficulty, the number of denoising steps a request needs, is discrete rather than continuous: requests fall into 11 fixed step-count levels (178 + 29k), and no signal we test predicts the level before generation starts (best R2 = 0.150). Second, benchmarks with short generation budgets below 320 tokens understate serving variance, since requests are cut off before the latency spread appears. Third, only 24% of single-request wall-clock time is GPU computation; the rest is CPU-side dispatch overhead. Batching mainly helps by amortizing this overhead: sharing one forward pass per denoising step improves throughput by 16.0x at batch size 16 over a per-request-dispatch baseline. We also argue structurally that output quality should not degrade with batch size, stating three assumptions this rests on; we measure 74 to 76% GSM8K accuracy at single-request scale. Finally, we derive a batch-timeout rule for fixed-fill synchronized batching under Poisson arrivals. Together, these results show that serving diffusion language models needs parallelism at the level of each denoising step, which differs from AR serving in how admission and eviction interact with an already shared forward pass.

I. INTRODUCTION

This paper characterizes masked diffusion language model serving under concurrent load, showing that AR-serving assumptions do not directly apply. It identifies discrete, unpredictable request costs, CPU dispatch dominance, and the need for step-level parallelism.

  • Motivation: Masked diffusion serving does not share AR decoding’s independent per-request update property, so unchanged continuous batching can reduce throughput.AR requests advance mostly independently, whereas masked diffusion requests require different denoising-step counts and can use one shared forward pass per step.
  • Serving Design: Step-level sharing amortizes dispatch overhead by advancing active slots through one shared forward pass and refilling slots when requests finish.This slot-based pattern contrasts with independent per-request dispatch and targets the computation structure of block-parallel decoding.
  • Contribution: The paper empirically characterizes dLLM serving rather than building another serving system, linking measurements to mechanisms and design principles.It reports discrete difficulty, batch-size-related dispatch cost, host-side latency dominance, short-budget variance underestimation, and a batch-timeout rule.
  • Scope: The single-GPU characterization generalizes workload-level properties to shared facilities but does not measure co-location contention, preemption, or multi-GPU serving.The batch-timeout rule is therefore a single-GPU, single-tenant building block rather than a complete cluster scheduling policy.
  • Difficulty: Request difficulty is discrete and cannot be predicted at admission time, so scheduling must treat per-request cost as unpredictable.The proposed explanation is that new blocks start only at fixed completion checkpoints.

2. Short budgets understate variance (§III-D)

Short generation budgets can hide the latency variation that appears at natural completion lengths. The serving cost also contains substantial CPU-side dispatch work, making step-level batching important for amortization.

  • Short budgets understate variance: Benchmarks must reach natural completion length, approximately 320 tokens here, because truncation before completion hides the real latency spread.The stated mechanism is that early truncation prevents requests from exhibiting their full serving-time differences.
  • Dispatch overhead: CPU dispatch dominates single-request latency because masking, threshold checks, and synchronization run once per denoising step.Sharing one forward pass per step amortizes this fixed control-loop cost across a batch.
  • Batching implication: Independent per-request batching pays dispatch overhead separately for each request and step, whereas step-level sharing pays it once for the batch.The paper derives a batch-timeout rule from measured service-time statistics to manage the resulting batching trade-off.
  • Mechanism: D2F exposes serving variability through fixed block and token confidence thresholds that govern block starts and token unmasking.These thresholds make the decoding path’s serving behavior observable rather than hidden.

B. Experimental Setup

Experiments use D2F-augmented LLaDA-8B-Instruct on one NVIDIA H200 GPU across GSM8K and HumanEval. GSM8K requests exhibit discrete denoising tiers, near-perfect latency prediction from step count, and substantial D2F latency variance.

  • Experimental Setup: The experiments use LLaDA-8B-Instruct with a D2F LoRA adapter on a single NVIDIA H200 GPU, evaluating GSM8K and HumanEval.Generation uses sampling temperature 0.2 and the reported software stack includes PyTorch 2.12.1, CUDA 13.0, transformers 4.49.0, and peft 0.19.1.
  • Experimental Setup: Latency is measured end to end from tokenization through detokenization, with torch.cuda.synchronize() around timed regions.Scheduling percentiles come from Monte Carlo simulation over measured service times; other reported values are direct GPU measurements.
  • Difficulty is Discrete, Not Continuous: 11 denoising-step values—178, 207, 236, and so on up to 468—appear for 100 GSM8K requests, spaced 29 steps apart.The fixed thresholds are τadd = 0.5 and τdecode = 0.9, and the pattern follows the block-addition rule.
  • Difficulty is Discrete, Not Continuous: r = 0.9997 measures the relationship between denoising step count and end-to-end latency across the 11 difficulty levels.The accompanying statistics are R2 = 0.9994 and p < 10^-99.
  • High-Variance Serving Workload: CV = 0.272 and a 2.64× max-to-min latency ratio indicate substantial GSM8K serving variability under a 512-token ceiling.Separate measurement passes produced CV estimates from 0.256 to 0.283, consistent with sampling variability at n = 100.
  • High-Variance Serving Workload: D2F increases serving-latency variance relative to the base model, with CV = 0.272 versus CV = 0.100 on identical GSM8K prompts.The comparison is reported under a 512-token generation ceiling.

C. Difficulty Cannot Be Predicted Before Generation Starts

Request difficulty is discrete and cannot be predicted usefully before generation begins. Short generation budgets also hide serving variance, while the same cross-request variability pattern appears on GSM8K and HumanEval.

  • Difficulty prediction: 11 fixed difficulty outcomes cannot be predicted reliably from signals available before generation starts.The tested signals achieve R2 values of 0.011, 0.004, 0.023, and 0.150, all below the 0.5 scheduling-usefulness floor.
  • Difficulty prediction: Difficulty unpredictability is structural because all requests initially have the first block fully masked, leaving the denoising path unseen.The discreteness of the outcomes therefore does not make admission-time scheduling easier.
  • Budget sensitivity: 76.0% GSM8K accuracy is reached at 512 tokens with 3% truncation, compared with 48.0% accuracy and 83% truncation at 256 tokens.At 1024 tokens, accuracy is 73.0% with 1% truncation; average completion length is roughly 320 tokens.
  • Budget sensitivity: Short budgets understate serving variance because truncation masks latency differences before requests reach natural completion length.Variance rises most sharply between 256 and 512 tokens once truncation no longer dominates.
  • Cross-task generalization: Across-request latency varies roughly 36× more than within-request latency on both GSM8K and HumanEval.HumanEval shows within-request CV = 0.0096 versus across-request CV = 0.345, while GSM8K shows 0.0073 versus 0.260.

IV. SYSTEM DESIGN: STEP-LEVEL DISPATCH COST SCALING

Masked diffusion serving benefits from sharing one forward pass across requests at each denoising step, rather than dispatching separately per request. Synchronized batching therefore scales near-linearly, while the paper’s per-request-dispatch baseline incurs batch-size-proportional overhead.

  • Synchronized batching: Synchronized batching runs one shared forward pass per denoising step, while the baseline runs one independent forward pass per request at every step.Requests are padded to a common shape, but each still samples its own tokens independently after the shared pass.
  • Baseline scope: The per-request-dispatch baseline is not the strongest real-system comparison because continuous batching already shares a forward pass across active requests.The paper uses this baseline to isolate dispatch overhead and identifies slot-based batching as future work.
  • Synchronized batching: 16.0× higher throughput is achieved by synchronized batching at batch size 16 versus single-request throughput.Throughput reaches 1.939 requests/s at batch size 16, compared with 0.121 requests/s for a single request.
  • Step-level dispatch cost scaling: 8.1× relative overhead appears at batch size 8 for per-request dispatch, following 2.4× at B = 2 and 5.1× at B = 4.The measured overhead tracks batch size because independent advancement uses B forward passes per denoising step, versus one for joint advancement.
  • Step-level dispatch cost scaling: Step-level sharing is built into masked-diffusion decoding: one forward pass can advance every request in a batch through a denoising step.A slot-based design would preserve shared passes while refilling completed requests from a queue.

C. Where the Time Actually Goes

Single-request profiling finds that most wall-clock time is CPU-side dispatch rather than GPU computation. Batching helps mainly by amortizing the per-step control-loop cost across requests.

  • Wall-clock breakdown: 75.6% of 10.28 seconds of wall-clock generation time is CPU-side overhead, while 24.4% is GPU kernel time.The GPU component is 2.511 seconds, measured by summing raw kernel-execution durations.
  • Wall-clock breakdown: The CPU-side control loop tracks block state, builds attention masks, checks thresholds, and synchronizes forward passes.These operations account for the dispatch overhead shown in the single-request breakdown.
  • Amortization mechanism: Synchronized batching pays the control-loop cost once per denoising step, whereas per-request dispatch pays it once per request per step.This is why batching primarily amortizes fixed CPU dispatch overhead rather than improving GPU utilization.
  • Implications: CPU dispatch, not GPU computation, is the identified bottleneck, directing optimization toward CPU-GPU integration and dispatch elimination.The paper points to tightly coupled systems such as GH200-class designs and techniques such as CUDA Graphs.

D. Forward-Pass Launch Overhead: A CUDA Graphs Micro-Experiment

The micro-experiment shows that CUDA graph capture reduces forward-pass CPU dispatch overhead, while broader quality and scheduling results provide context for serving trade-offs.

  • CUDA graph capture: 48.6%: CUDA graph capture reduced isolated forward-pass CPU dispatch time from 19.4 ms to 10.0 ms.The logits remained numerically identical, with a maximum absolute difference of 0.0.
  • CUDA graph capture: 235 forward-pass calls occurred during the profiled generation, independently confirmed by aten::embedding operation counts.The exact 235-of-235 match validates the call count used for extrapolating dispatch savings.
  • CUDA graph capture: 28.5%: capturing all 235 forward passes would remove 2.22 s from 7.77 s of total dispatch overhead.The remaining 71.5% is attributed to Python control flow for mask construction, threshold evaluation, and block-state bookkeeping.
  • CUDA graph capture: CUDA graph capture is a partial fix because it covers roughly a quarter of per-step dispatch overhead, while control-loop restructuring addresses the larger remainder.The paper characterizes graph capture as a low-risk practical intervention rather than a complete solution.
  • Quality and serving context: 74.0% exact-match accuracy was measured in one GSM8K single-request run at gen_length=512, versus 76.0% in a separate run.The two-point difference is attributed to run-to-run variation under temperature > 0 sampling; 22 of 24 errors completed normally rather than truncating.

VI. LIMITATIONS AND FUTURE WORK

The paper’s scope is limited by its hardware, model, evaluation design, small scheduling sample, and incomplete comparisons; several validations remain future work.

  • Scope: The experiments use LLaDA base and D2F on one NVIDIA H200 GPU, with task and configuration generalization tested separately.Joint testing across configurations and tasks, and evaluation on other hardware, remain open.
  • Scheduling analysis: 24%: a later 32-batch scheduling run shifted estimated capacity relative to the analysis based on 8 measured batches.The paper therefore treats the scheduling estimates as preliminary.
  • Scheduling analysis: The reported latency percentiles should be treated as preliminary because they rely on n = 8 measured service times pending a larger sample.Other single-run results may carry similar uncertainty because repeated trials were not used.
  • Comparisons: The paper does not compare against existing dLLM inference engines such as dInfer.A direct comparison would strengthen the scheduling-rule evaluation, although the contribution is workload characterization rather than a competing system.
  • Configuration: The measured tier structure uses one threshold setting, τadd = 0.5 and τdecode = 0.9, so its constants may be configuration-specific.The paper identifies validation under other settings as necessary before generalizing the tier constants.
  • Future work: Quality under batch sizes 8 and 16 and a fully correct slot-based design remain unvalidated future work.Further testing of the batch-timeout stability rule is also listed as future work.

VII. RELATED WORK

Prior work accelerates individual dLLM requests or builds serving components, whereas this paper supplies measurement-grounded evidence about concurrent workload behavior and step-level batching.

  • dLLM acceleration: Caching and sparsity methods such as dKV-Cache, Fast-dLLM, dLLM-Cache, and FlashDLM primarily reduce single-request latency by reusing denoising-step computation.The paper distinguishes this focus from concurrent serving characterization.
  • dLLM serving systems: dInfer, DiLaServe, Sangam, and dLLM-Serve address batched inference, scheduling, resource partitioning, or memory management for concurrent dLLM serving.The paper positions its contribution as complementary rather than as another complete serving system.
  • Measurement-grounded characterization: The paper fills a measurement gap by examining difficulty structure, predictability, and benchmark sensitivity under concurrent load.These workload properties are presented as prerequisites for understanding why serving behavior occurs.
  • AR serving contrast: AR continuous batching relies on request-parallel decoding, but dLLMs more efficiently share computation across requests at each denoising step.The paper reports that applying AR-style continuous batching without modification can reduce throughput.
  • Paper contribution: 76%: CPU-side dispatch overhead dominates single-request wall-clock time, compared with 24% GPU kernel time.This finding explains why step-level sharing amortizes a fixed dispatch cost and yields a 16.0× throughput improvement.

APPENDIX A PROFILER MEASUREMENT METHODOLOGY

The appendix documents reproducible measurement procedures, including synchronized timing, direct kernel-trace accounting, experiment workflow, and the hardware and software environment.

  • Profiler validation: Profiler accounting was unreliable: prof.key_averages() doubled estimated GPU compute, and long-lived wrapper markers inflated self-time around Python loops.Neither issue was fixed by calling key_averages() once or removing the wrapper marker.
  • Profiler validation: Raw kernel durations from the exported Chrome trace matched PyTorch’s printed GPU total to three decimal places across repeated runs.The appendix reports this trace-based total as the ground-truth profiling result.
  • Environment: All GPU-bound measurements use LLaDA-8B-Instruct with D2F on one NVIDIA H200; HumanEval is used only for cross-task validation.Scheduling percentiles come from Monte Carlo simulation over measured service times, while other reported numbers are direct GPU measurements.
  • Reproducibility: Timed regions are bracketed by torch.cuda.synchronize(), and the artifact emits per-request CSVs, aggregate JSON summaries, and Chrome-trace JSON profiling output.The runtime is a single-tenant, single-GPU Linux process without distributed execution.
  • Experiment workflow: The workflow characterizes service time, measures batching and GPU kernels, runs scheduling Monte Carlo simulation, and evaluates GSM8K exact-match accuracy.The simulation uses service-time data from the preceding measurements.

C. Installation

The paper evaluates dLLM serving through single-request characterization, batching, profiling, scheduling simulation, and correctness experiments. Results show discrete difficulty tiers, unpredictable admission-time cost, strong synchronized-batching gains, CPU-dominated latency, and variable single-request accuracy.

  • Evaluation and Expected Results: 11 discrete denoising-step values occur on GSM8K, with step count predicting latency almost perfectly.The reported correlation is r > 0.99.
  • Evaluation and Expected Results: R2 < 0.15 for every tested early signal, so none is useful for admission-time scheduling.Tested signals include confidence, entropy, and prompt length.
  • Evaluation and Expected Results: GPU kernel time makes up well under half of single-request wall-clock latency, while host-side dispatch overhead dominates.The profiling experiment tracks both CPU and CUDA activity.
  • Evaluation and Expected Results: Mean latency under fixed-fill batching is U-shaped in utilization ρ, with its minimum near ρ ≈ 0.6 to 0.8.The scheduling simulation uses measured service-time samples and Poisson arrivals.
  • Evaluation and Expected Results: GSM8K exact-match accuracy is 65 to 80% for LLaDA-8B with D2F at gen length ≥ 512 under single-request inference.The range reflects run-to-run variation from temperature-above-zero decoding.

F. Experiment Customization

The experiments expose adjustable generation, batching, and sampling settings while balancing statistical precision against GPU time. Reproduction is limited by a non-converged scheduling sample, an unvalidated slot-based design, and incomplete batched quality evaluation.

  • Experiment Customization: Generation-length budget, block size, confidence thresholds, batch size, and sampling temperature are the main adjustable settings.Reducing experiment sample size trades statistical precision for shorter GPU time.
  • Limitations: A later 32-batch replication shifted estimated service capacity by about 24%, indicating the original 8-batch sample had not converged.Larger service-time samples are recommended before treating percentile estimates as precise.
  • Limitations: The slot-based batching design was not fully correct, so its discussion remains analytical rather than empirically validated.The design mixes shared forward passes with independent per-request completion.
  • Limitations: Direct empirical quality evaluation at batch sizes 8 and 16 was not completed within the artifact’s scope.The paper instead gives a structural argument for quality preservation at those batch sizes.
  • Limitations: The appendix restates the main paper’s open items with concrete consequences for reproduction and extension.It is intended to remain consistent with the main paper’s Limitations section.
Loading 2608.23807v1…