PaperlayerView exact revision on arXiv

Source-linked AI summary

DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification

Rongjian Chen, Minxian Xu, Zhengxin Fang, Kejiang Ye, Chengzhong Xu

arXiv:2609.37532v1cs.DCcs.AI

TL;DR

DScale addresses verification padding, rejected candidates, and changing prefixes under fixed-shape graphs in high-concurrency block-diffusion decoding. It combines path-aware tiles, score-guided prefix allocation, and fixed-address graph integration. Across the evaluated models and workloads, it improves geometric-mean throughput over DFlash, DSpark, and Domino.

  • Problem

    At high concurrency, block-diffusion decoding faces verification padding, rejected candidates, and variable prefixes that are difficult to execute with fixed-shape graphs.

    Source: §I · pp. 1–2

  • Method

    DScale retains the DFlash drafter and full draft length, using path-aware tiles, a separate predictor to allocate prefixes within half-capacity verification, and fixed-address graphs for changing boundaries.

    Source: §I · p. 2

  • Results

    Across four datasets and seven concurrencies, geometric-mean throughput improves over DFlash, DSpark, and Domino by 43.9%, 22.2%, and 24.4% on Qwen3-8B, and 48.8%, 37.7%, and 32.0% on Qwen3-4B.

    Source: B. Throughput Results · p. 7

  • Takeaways & Limitations

    Cumulative ablations show throughput rising with each added mechanism, while budget adjustment raises accepted-token retention from about 71% to over 91%.

    Source: §I · p. 2

    The evaluation retains the selected DFlash drafters’ 16-slot blocks; other block widths remain for future adaptation and evaluation.

    Source: §VI · p. 11

Abstract

from arXiv · show

Growing large language model applications demand efficient inference. At high concurrency, block-diffusion speculative decoding suffers from verification padding, rejected candidates, and incompatibility between variable prefixes and fixed-shape graphs. Uniform truncation sacrifices acceptable tokens. We present DScale, preserving drafter architecture, weights, and full draft length. A separate 112K-parameter predictor requires neither confidence calibration nor hardware speed-curve preparation. Path-aware tiles reduce padding. Dynamic verify-length (DVL) allocation packs scored prefixes into half the native verification capacity. Fixed-address workspaces propagate changing boundaries through verification and acceptance while reusing captured graphs. On A100-40GB with tensor parallelism 1, Qwen3-8B and Qwen3-4B cover four datasets and concurrency 8-32, reusing each target's frozen predictor. Geometric-mean throughput gains across these configurations are respectively 43.9% and 48.8% over DFlash, 22.2% and 37.7% over DSpark, and 24.4% and 32.0% over Domino, with lower request latency. Cumulative ablations show that adding the three mechanisms successively increases geometric-mean throughput, while budget adjustment improves accepted-token retention. GPU profiling shows that complete decode-step time on GSM8K decreases by 30.8-52.5% relative to DFlash

I. INTRODUCTION

DScale targets wasted verification work and the difficulty of compactly verifying unequal accepted prefixes while preserving full-length DFlash drafting. It combines path-aware tiles, score-guided budget allocation, and fixed-address graph integration, improving throughput over three baselines across the evaluated models and workloads.

I. INTRODUCTION

  • DScale retains DFlash’s drafter architecture, weights, and full draft length, using a separate 112K-parameter predictor without confidence calibration or hardware speed-curve profiling.The predictor allocates prefixes within half of each verification graph’s native candidate-slot capacity.Source: §I · p. 2
  • Path-aware tile routing selects precompiled attention tiles for prefill and verification, removing padding beyond the verify width while preserving prefill reuse.Source: §I · p. 2
  • Dynamic verify-length allocation uses predictor scores to select per-request prefixes under a halved shared budget, while flat-pack compaction retains useful long prefixes without shortening drafting.Source: §I · p. 2
  • A fixed-shape compact workspace carries changing request boundaries through decision and verification graphs, avoiding fixed-width padding and CPU re-slicing.Source: §I · p. 2
  • Across four datasets and concurrency 8–32, DScale raises geometric-mean throughput over DFlash by 43.9–48.8% and lowers aggregate mean E2E latency by 31.4–33.1%.Cumulative ablations show throughput rising with each added mechanism; budget adjustment raises accepted-token retention from about 71% to over 91%.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

The background identifies verification-tile waste and request-level variation in accepted prefixes as obstacles to efficient DFlash serving. Draft-side signals correlate with acceptance, motivating request-specific prediction without retraining the drafter.

II. BACKGROUND AND MOTIVATION

  • At concurrency 16, target verification accounts for 83.3% of summed GPU-kernel time in the profiled native DFlash workload.The profile attributes 10.9% to the draft graph and 5.7% to kernels outside the graphs.Source: §II · p. 2

A. Verify Inherits Prefill’s Tile

  • The A100 verify tile has 128 rows for each request’s 16 slots, leaving 112 rows, or 87.5%, masked; adding requests does not fill those rows.Distinct KV contexts and causal boundaries prevent requests from sharing a tile.Source: A. Verify Inherits Prefill’s Tile · p. 2

B. Request-Level Acceptance Variability and Verification Budget Allocation

  • On GSM8K at concurrency 8, width 8 lowers throughput by 15% and mean progress from 5.92 to 4.96 tokens; width 4 lowers progress to 3.37.Uniform widths below 16 all yield lower throughput than full-width verification, while width 14 reduces selected slots by only 12.5%.Source: B. Request-Level Acceptance… · p. 3
  • Mean Top1–Top2 draft-logit gap correlates positively with accepted draft length: Spearman 0.741 on Qwen3-8B and 0.722 on Qwen3-4B.The gap is available before target verification, motivating request-level prefix prediction.Source: B. Request-Level Acceptance… · p. 3
  • DScale trains a separate 112K-parameter predictor on draft logits, hidden-state features, and batch context, preserving the drafter without STS calibration or SPS profiling.Source: B. Request-Level Acceptance… · p. 3

C. Adapting Dynamic Prefixes to Fixed Verification Graphs

This section motivates reducing verification capacity while retaining full-length drafting, then outlines how DScale routes and packs request-specific prefixes into graph execution. Its compact workspace carries dynamic boundaries through verification and acceptance.

C. Adapting Dynamic Prefixes to Fixed Verification Graphs

  • Shortened logical prefixes alone do not shrink graph execution: padding them to DFlash’s reserved width leaves GEMM and projection dimensions unchanged.The graph must process fewer candidate rows to realize the savings.Source: C. Adapting Dynamic Prefixes to Fixed… · p. 3
  • Halving the candidate-slot capacity to 8N is motivated by mean progress of 6.09, but the long tail argues against assigning every request the same eight slots.The halving applies to verification graph slots, not request capacity or draft length.Source: C. Adapting Dynamic Prefixes to Fixed… · pp. 3–4
  • As packed prefix lengths change, request boundaries move; CPU re-slicing requires GPU length readback and host preparation that can delay verification.Equal-width padding instead expands compact rows back to the original width.Source: C. Adapting Dynamic Prefixes to Fixed… · p. 4

A. Overview

  • DScale’s request flow batches active requests, drafts full-length blocks, predicts and allocates prefixes, then packs them into a fixed-address workspace for target verification.The target verifier uses a path-aware tile router; graph integration shares the workspace through acceptance and state commit.Source: A. Overview · p. 4

B. Path-Aware Tile Routing

DScale routes attention tiles by execution path and allocates predictor-scored, request-specific prefixes within a halved verification budget while retaining full-length drafting. The packed prefixes support compact verification without padding requests back to equal width.

B. Path-Aware Tile Routing

  • Verification uses a 16-row tile for queries up to the 16-token verify width, while prefill and longer queries retain the hardware-selected 128-row tile.The tile height fits the maximum verify width without extra inactive rows; smaller tiles would split key-value reuse across blocks.Source: B. Path-Aware Tile Routing · p. 4

C. DVL Budget Allocation

  • DVL selects unequal prefixes under a shared budget of 8N rows, then packs them contiguously using prefix-sum offsets.The allocator grows the highest-scoring next slot or shrinks the lowest-scoring tail until the budget is met, preserving anchored prefixes.Source: C. DVL Budget Allocation · pp. 4–6
  • The acceptance predictor uses draft-position confidence features and batch context to estimate prefix scores before verification.A 1735–64–15 MLP produces scores for draft positions; cumulative minima enforce nonincreasing prefix scores, and rounded score sums initialize request lengths.Source: C. DVL Budget Allocation · pp. 4–5
  • With initial lengths [5, 10, 7, 9] and budget 32, DVL grows the first request to produce lengths [6, 10, 7, 9] and offsets [0, 6, 16, 23].The figure also illustrates shrinking a different request as an alternative budget adjustment.Source: C. DVL Budget Allocation · p. 6, Figure 6 · p. 6
  • On GSM8K at concurrency 8, the predictor, allocator, and packer together account for 3.08% of Qwen3-8B and 4.08% of Qwen3-4B complete decode-step time.These figures measure combined GPU execution, not separate per-module costs.Source: C. DVL Budget Allocation · p. 6

D. Graph Integration for Ragged Decisions

DScale carries changing request boundaries through compact verification and acceptance using fixed-address workspaces, allowing captured graphs to be reused without restoring equal-width padding. State updates and finished-request removal occur outside graph capture.

D. Graph Integration for Ragged Decisions

  • Fixed-address workspace arrays retain their shape and address across steps while carrying changing packed boundaries through verification and acceptance.Verification derives request boundaries from lengths and offsets, then uses persistent cache-mapping metadata to gather committed context.Source: D. Graph Integration for Ragged… · pp. 6–7
  • For greedy decoding, DScale captures decision and verification pipelines for bucketed request capacities, then stages inputs and replays the graphs each step.GPU reads are ordered after decision execution and prior cache writes; state commit remains outside capture.Source: D. Graph Integration for Ragged… · pp. 6–7
  • Verification reads each request’s committed context and masks rows beyond its packed prefix length, while the prefix verifier accepts candidates and supplies a successor token.At the first rejection, the successor is drawn from the target distribution with the rejected candidate removed and probabilities renormalized.Source: D. Graph Integration for Ragged… · p. 7

A. Experimental Setup

The benchmarks evaluate Qwen3-8B and Qwen3-4B on an A100-40GB at tensor parallelism 1 across four math and code datasets. Comparisons use a concurrency sweep and matched request protocols across four SGLang-based systems.

A. Experimental Setup

  • The main benchmarks use an A100-40GB GPU at tensor parallelism 1 to evaluate Qwen3-8B and Qwen3-4B.Generation temperature is 0.Source: A. Experimental Setup · p. 7
  • The four workloads are GSM8K, MATH500, HumanEval, and MBPP, covering math problems and code-generation tasks.Measured runs use 32 warmup requests followed by 512 requests, with fixed request manifests and an output cap of 2048 tokens.Source: A. Experimental Setup · p. 7
  • Concurrency ranges from 8 to 32 in seven settings, with five-repetition medians and aggregate throughput ratios computed as geometric means over dataset–concurrency cells.Each configuration uses matched seeds and prompts.Source: A. Experimental Setup · p. 7
  • Four SGLang-based systems are compared under the same request protocol, with DFlash and DScale sharing the same released drafter checkpoint for each target.DSpark and Domino use their respective target-matched draft checkpoints; tokenization, prompt formatting, stopping criteria, output length, and temperature are held fixed.Source: A. Experimental Setup · p. 7

B. Throughput Results

DScale achieves higher geometric-mean token throughput than DFlash, DSpark, and Domino across the four datasets and seven concurrency settings for both target models. Gains over DFlash grow with concurrency, while dataset-level gains vary.

B. Throughput Results

  • 43.9%, 22.2%, and 24.4% geometric-mean throughput gains over DFlash, DSpark, and Domino, respectively, on Qwen3-8B; Qwen3-4B gains are 48.8%, 37.7%, and 32.0%.These ratios are computed across the four datasets and seven concurrency settings.Source: B. Throughput Results · p. 7, Figure 7 · p. 8
  • Against DFlash, gains rise from 23.2% on Qwen3-8B and 24.0% on Qwen3-4B at c = 8 to 59.6% and 62.3%, respectively, at c = 32.Four-dataset geometric-mean gains increase with concurrency on both models.Source: B. Throughput Results · pp. 7–8
  • Across GSM8K, MATH500, HumanEval, and MBPP, gains over DFlash are 40.0%, 44.1%, 46.5%, and 45.3% on Qwen3-8B, and 44.9%, 51.4%, 50.7%, and 48.5% on Qwen3-4B.Each value is a geometric mean of paired throughput ratios over seven concurrencies.Source: B. Throughput Results · p. 8
  • The same predictors support sampling and tensor parallelism without retraining, with geometric-mean gains over DFlash of 45.2% and 45.8% under TP1 at temperature 1, and 43.9–55.8% under TP2.The TP2 range covers temperatures 0 and 1.Source: B. Throughput Results · p. 8

C. End-to-End Latency

DScale reduces end-to-end request latency relative to DFlash, DSpark, and Domino on both target models. The reported reductions cover mean, p95, and p99 latency across four datasets and seven concurrency settings.

C. End-to-End Latency

  • Mean E2E latency reductions relative to DFlash, DSpark, and Domino are 31.4%, 18.4%, and 20.2% on Qwen3-8B, and 33.1%, 27.1%, and 24.7% on Qwen3-4B.Reductions use geometric means of DScale-to-baseline ratios of five-repetition medians across four datasets and seven concurrencies.Source: C. End-to-End Latency · p. 8
  • Relative to DFlash, p95 and p99 latency reductions are 30.7% and 29.3% on Qwen3-8B, and 32.9% and 31.9% on Qwen3-4B.Fig. 8 shows lower cross-dataset medians than all three baselines at every concurrency for mean, p95, and p99.Source: C. End-to-End Latency · p. 8

D. Component Ablation

The ablation adds tile routing, dynamic verify-length allocation, and graph integration successively, with throughput gains at each stage. Additional evaluations report increased accepted-token retention and shorter decode intervals, alongside allocator costs.

D. Component Ablation

  • Tile routing adds 14.7% and 20.8% throughput over the DFlash control on Qwen3-8B and Qwen3-4B; adding DVL yields a further 11.8% and 3.9%, and graph integration another 9.8% and 15.2%.These are geometric means across seven concurrencies.Source: D. Component Ablation · p. 9
  • With all three mechanisms, cumulative geometric-mean throughput gains over the DFlash control reach 40.8% on Qwen3-8B and 44.6% on Qwen3-4B.The ablation holds the runtime, target, drafter, and request set fixed on GSM8K; the control disables all three optimizations.Source: D. Component Ablation · pp. 8–9
  • Score-guided budget adjustment raises accepted-token retention from 71.6% to 91.3% on Qwen3-8B and from 70.5% to 91.6% on Qwen3-4B.The evaluation includes score-guided redistribution and budget expansion.Source: D. Component Ablation · p. 9
  • On GSM8K, DScale decode intervals are 30.8–52.5% shorter than DFlash intervals in the TP1 profile.The profile averages 700 step intervals after discarding 40, at concurrency 8, 16, and 32.Source: D. Component Ablation · p. 9
  • At concurrency 8, the Acceptance predictor, Budget allocator, and Flat-buffer packer together account for 4.08% of step time on Qwen3-4B and 3.08% on Qwen3-8B.Predictor timing includes feature extraction, shared Top1 selection, and input preparation, but excludes the LM-head.Source: D. Component Ablation · p. 9

E. Graph-Pool Memory

DScale combines adaptive verification with graph-aware execution and reports throughput gains over three baselines. Its evaluation also reports lower graph-pool memory and retains the selected DFlash drafters’ 16-slot blocks.

VI. CONCLUSION

  • Fig. 9’s five-repetition-median GSM8K ablation successively adds tile routing, DVL, and graph integration.The conclusion says ablations quantify the mechanisms’ gains and costs.Source: Figure 9 · p. 10, §VI · p. 11

E. Graph-Pool Memory

  • DScale retains 832 MiB on Qwen3-8B and 724 MiB on Qwen3-4B, respectively; these amounts are below DFlash by 20.2% and 11.3%, respectively.They are below DSpark by 27.1% and 26.1%, and below Domino by 42.4% and 36.5%, respectively.Source: E. Graph-Pool Memory · p. 10

V. RELATED WORK

  • Unlike KV-cache, scheduling, and sparse-attention methods, DScale allocates verification budgets while preserving target computation and adapting verification tiles and graph interfaces.The comparison concerns methods that optimize stored context, schedule serving resources, or select attention connections.Source: §V · pp. 10–11

VI. CONCLUSION

  • 43.9–48.8% over DFlash, 22.2–37.7% over DSpark, and 24.4–32.0% over Domino in geometric-mean throughput, with lower request latency.The evaluation covers two target models, four datasets, and concurrency 8–32 on A100 with TP1.Source: §VI · p. 11
  • The evaluation retains the selected DFlash drafters’ 16-slot blocks, leaving other block widths for future adaptation and evaluation.DScale combines path-aware tiles, score-guided half-capacity verification, and fixed-address graphs with dynamic request boundaries.Source: §VI · p. 11

Related papers

Loading 2609.37532v1…