FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

1 INTRODUCTION

• Masked dLLMs start from a fully masked, fixed-length sequence and iteratively choose positions to unmask and token values to place there.A common strategy selects the most confident token position and samples from its marginal distribution.Source: §1 · p. 1
• Each unmasking changes both the dLLM logits and guidance vector, requiring a dLLM forward pass and reward-model forward and backward passes.These computations can be large inference-time bottlenecks.Source: §1 · pp. 1–2
• Existing parallel decoders were developed for unguided generation and do not establish when guidance can be reused or which tokens remain safe to unmask together.The resulting research question is whether reward-guided dLLM inference can be accelerated while preserving generation quality.Source: §1 · p. 2
• FastGuide is up to 4.4× faster than sequential guided decoding while retaining similar reward model scores across three reward benchmarks and two dLLMs.It combines guidance reuse with autoregressive token unmasking and sparse dLLM recomputation.Source: §1 · p. 2

Masked dLLM inference

• Masked dLLMs iteratively denoise a partially masked sequence, predicting categorical distributions at masked positions and choosing which positions and values to unmask.Confidence-based decoding may select a position whose predicted distribution has minimum entropy.Source: §2 · p. 3

Parallel decoding

• Parallel decoding proposes k tokens together, approximating their true joint distribution with a product of per-token marginal distributions.This approximation is known as the joint-marginal mismatch.Source: §2 · p. 3

Related work

• Existing parallel decoders use confidence, logit stability, or attention graphs to select tokens, but do not address guidance-vector recomputation or validate selection rules for guided distributions.External planners offer another approach, whereas FastGuide uses parallel techniques for reward guidance and autoregressive dLLM decoding without external models.Source: §2 · p. 3

Reward guidance for alignment

• Gradient-based reward guidance steers dLLM logits with a per-token vector derived by completing a masked sequence, scoring it with a reward model, and differentiating the reward.Prior work has focused on improving completion inputs and has largely evaluated sequential decoding, while this paper focuses on accelerating reward guidance.Source: §2 · p. 3

Related work

• FastGuide’s overview combines guidance caching and autoregressive sparse dLLM recomputation, then defers tokens whose recomputed confidence falls below a threshold.For the illustrated sparse update, selected positions attend to the full sequence using updated K/V within windows and cached K/V elsewhere.Source: Figure 2 · p. 4

3.1 HYBRID DECODING: GUIDANCE CACHING AND DLLM RECOMPUTATION

• FastGuide amortizes one guidance computation across k candidates while recomputing dLLM logits after each unmasking.This hybrid combines parallel-style guidance reuse with autoregressive token selection.Source: §3 · p. 4, §3.1 · p. 5

3.1.1 COUNTERFACTUAL ANALYSIS OF HYBRID DECODING

• Cached guidance changes high-confidence token distributions less, while guidance increases co-proposed token incompatibility from 1.04 to 2.47 average pairs per prompt.The fraction of prompts with at least one incompatible pair rises from 58% to 90%.Source: §3.1.1 · p. 5

3.1 HYBRID DECODING: GUIDANCE CACHING AND DLLM RECOMPUTATION

• Guidance caching with dLLM recomputation retains most quality, whereas reusing dLLM logits degrades quality with or without guidance caching.The comparison varies reuse versus recomputation of each term within a candidate group.Source: Figure 3 · p. 5, §3.1 · p. 5

3.1.2 SPARSE DLLM RECOMPUTATION

• Sparse recomputation updates attention and feed-forward outputs near the latest committed token and next candidate, reducing each pass from O(L^2) to O(wL).Queries at those positions attend to the full sequence, using cached keys and values elsewhere.Source: §3.1.2 · pp. 5–6

3.2 ADAPTIVITY VIA CONFIDENCE DEFERRAL

• Confidence deferral commits a candidate when its recomputed confidence reaches threshold τ; otherwise it stays masked for reconsideration after fresh logits and guidance.The first candidate is always committed, so each step makes progress.Source: §3.2 · p. 6
• Each FastGuide step uses one dLLM forward pass, one guidance computation, and k − 1 sparse recomputation passes, versus k of each full computation for sequential decoding.Source: §3.2 · p. 6

4 EXPERIMENTS

• The experiments evaluate three reward benchmarks using Dream-7B-Instruct and LLaDA-8B-Instruct, comparing guided parallel decoders with sequential guidance and Best-of-N.Quality is measured with Top@1, Seq Gap, and LMUnit; inference time is reported as wall-clock seconds per generation.Source: §4 · pp. 6–7

4.1 MAIN RESULTS

• FastGuide is 2.8–4.4× faster than sequential guided decoding across three benchmarks and two models, while remaining within 0.75 reward points of sequential guidance.It also reduces inference time to 0.9–1.3× that of Best-of-N while retaining a Top@1 improvement of 0.1–0.65 reward points over Best-of-N.Source: Table 1 · p. 7, §4.1 · p. 8
• FastGuide matches or improves on DAPD across every benchmark and model, with its largest gain reaching +1.1 Top@1 on RM-Bench with Dream-7B.Fast-dLLM, KLASS, and EB-sampler lose substantial performance relative to sequential decoding; Figure 7 shows FastGuide's small Seq Gap across nearly all categories.Source: §4.1 · p. 8, Figure 7 · p. 8
• In an RM-Bench math example, guided DAPD changes a correct arithmetic step from 82 to 52 and propagates the error, whereas FastGuide recomputes the candidate distribution and recovers the correct answer.The figure marks incorrect tokens in red.Source: §4.1 · pp. 8–9, Figure 8 · p. 9

4.2 ABLATION STUDIES

• Confidence deferral trades speed for quality: with τ=0, unmasking eight tokens per step is fastest but degrades quality, while deferring more tokens approaches sequential-decoding quality at higher inference time.More deferrals require additional guidance computation steps and full dLLM forwards.Source: Figure 6 · p. 8, §4.2 · p. 9
• Sparse dLLM recomputation approximates exact recomputation by token-distribution TV distance, and both sparse and exact recomputation change many token values during decoding.A sparse-recomputation window of 2 recovers a significant proportion of sequential guided decoding quality; the reported quality is largely agnostic to candidate-set order.Source: §4.2 · p. 9

5 CONCLUSION

• FastGuide combines guidance caching with sparse dLLM recomputation to efficiently recompute each proposed token distribution autoregressively within a decoding step.The conclusion reports speedups up to 4.4× over sequential guided decoding while retaining similar generation quality.Source: §5 · p. 9
• The authors find that guidance can be reused across diffusion timesteps, but can worsen token incompatibilities in parallel decoding.They describe this as the first comprehensive study of accelerating training-free gradient-based reward guidance for dLLMs and the first evaluation of parallel decoders under reward guidance.Source: §5 · p. 9

A EXTENDED RELATED WORK

• Existing guided dLLM methods require reward-model forward and backward passes at every diffusion timestep, while sampling-based alternatives can require many decoded sequences or particles.Training-based alignment methods require retraining when the reward changes.Source: A EXTENDED RELATED WORK · p. 14
• Guidance reuse in masked dLLMs is position- and timestep-dependent, unlike continuous diffusion's timestep-scheduled reuse; high-confidence positions have stable guided token distributions.The related-work discussion also identifies guidance's interaction with joint-marginal mismatch as a distinct observation.Source: A EXTENDED RELATED WORK · p. 14
• Prior parallel decoders use auxiliary structure, self-verification, or revocable unmasking, but the paper says these methods had not been evaluated under gradient-based reward guidance.Their existing objectives or mechanisms do not cleanly address guided-distribution consistency, recomputation, and token compatibility.Source: A EXTENDED RELATED WORK · p. 15
• FastGuide approximates recomputed token distributions with sparse forward passes and acts before unmasking, reselecting token values and deferring low-confidence candidates rather than accepting, rejecting, or remasking drafts.The paper says the approximation is validated empirically.Source: A EXTENDED RELATED WORK · p. 15

B.1 BENCHMARKS

• The paper evaluates JudgeBench, RM-Bench, and Reward-Bench-2, covering categories including reasoning, coding, mathematics, instruction following, and safety.The benchmarks assess distinctions such as objectively correct versus subtly incorrect answers, substantive response quality versus style, and valid ties among multiple answers.Source: §B.1 · pp. 15–16

B.2 HYPERPARAMETERS

• Generation uses length 128 and temperature 0.7; reward guidance uses three reward-model backpropagation steps per diffusion timestep.For tokenizer-nonoverlapping tokens between LLaDA and the Skywork reward models, the embedding is set to zero.Source: §B.2 · p. 16
• Baseline hyperparameters were tuned separately because existing parallel-decoding baselines were not designed for guided decoding; FastGuide's candidate pool is k=8 in the high-quality regime and k=16 in the high-throughput regime.Each baseline was tuned for highest quality at throughput comparable to FastGuide, using Top@1 on 64 RM-Bench prompts.Source: §B.2 · p. 16, Table 2 · p. 17
• Fast-dLLM unmasks every token whose confidence exceeds threshold c, while EB-Sampler unmasks the longest low-entropy prefix within budget γ, with γ=0 recovering sequential decoding.The EB-Sampler sweep covers γ∈[0.1,16].Source: §B.2 · pp. 16–17
• KLASS requires confidence above c and distributional KL stability below ϵ for h steps, but published settings often default to sequential decoding under guidance and can stop accepting tokens altogether.The paper attributes this to guided token distributions changing drastically and to the remaining masked positions predicting end-of-sequence.Source: §B.2 · pp. 16–17
• DAPD builds an attention-based dependency graph and unmasks an independent set, with its threshold scheduled from 0.005 to 0.05 and scaled during tuning.Once fewer than half the tokens remain masked, it also unmasks tokens with confidence above 0.9.Source: §B.2 · p. 17
• LMUnit scores responses from 1 to 5 using five fixed unit tests and reports their mean; the judge prompt requests a score without written rationale.LMUnit is used as an independent quality assessment rather than the primary metric.Source: §B.2 · p. 17, Figure 10 · p. 17

B.3 EXPERIMENTAL DETAILS FOR SECTION 3

• Section 3 diagnostics use Dream-7B, a 0.6B Skywork reward model, and 250 RM-Bench prompts; tests compare guidance reuse and recomputation and measure incompatible token proposals.The incompatibility test compares guided and unguided proposals from the same state, counting pairs whose PMI is below −1.Source: §B.3 · pp. 17–18

C FASTGUIDE PSEUDOCODE

• Each hybrid step computes a full guided pass, then verifies candidate tokens one at a time using cached K/V states and sparse attention recomputation, deferring tokens below threshold τ.The attention window is centered at aj −1 for Dream and at aj for LLaDA.Source: C FASTGUIDE PSEUDOCODE · p. 18

D.1 GUIDANCE CACHING WITH OTHER REWARD GUIDANCE ALGORITHMS

• On JudgeBench and RM-Bench, FastGuide retains the corresponding sequential method's performance with both Expectation and APS guidance.Expectation uses expected token embeddings as reward-model input; APS samples from the token distribution to construct that input.Source: §D.1 · p. 19, Table 3 · p. 19

D.2 FASTGUIDE WITH DIFFERENT REWARD MODELS

• Across the tested reward models, FastGuide similarly retains sequential-decoding performance; with Skywork-Llama-3.1-8B, it reduces inference time from 88.1 to 17.9 seconds, a 4.9× speedup.The 8B reward model was run on a second GPU; the other two reward-model blocks used one GPU.Source: §D.2 · p. 19, Table 4 · p. 20

D.3 MEASURING AVERAGE REWARD MODEL PERFORMANCE AND THROUGHPUT

• FastGuide attains the highest Avg@4 among parallel decoders on both models and all three benchmarks, staying within 1.1 points of sequential decoding.It essentially matches sequential decoding on Reward-Bench-2; the strongest baseline, DAPD, trails sequential decoding by up to 2.14 points.Source: §D.3 · p. 20, Table 5 · p. 21, Table 7 · p. 23
• With k = 16 candidates, FastGuide's Seq Gap remains within 1.20 on all three benchmarks, while its throughput rises only from 4.4–5.7 to 5.5–8.0 tokens per step.Inference time is essentially unchanged because candidates below the recomputed-confidence threshold are deferred.Source: §D.3 · p. 20, Table 6 · p. 22

D.4 EFFECT OF BATCHING

• FastGuide decodes batches of the same prompt in 5.7 seconds per generation at B = 4 versus 15.3 seconds at B = 1, and its speedup over sequential guided decoding grows from 2.9× to 3.4×.Sparse-over-exact recomputation speedup grows from 1.21× to 1.89× across those batch sizes.Source: §D.4 · p. 24, Figure 11 · p. 24

D.5 SPARSE DLLM VERIFICATION WINDOW SIZE ABLATION

• A window size of 2 recovers a significant proportion of sequential guided decoding quality, and sparse recomputation approximates exact recomputation at the same k.The comparison uses Top@1 reward-model score across window radii w and removes confidence deferral, unmasking all k candidates per step.Source: §D.5 · p. 24, Figure 12 · p. 25
• The remaining quality gap to sequential decoding is attributed to fixing the tokens to unmask, not to sparse recomputation.Confidence deferral is proposed to add adaptivity to sampling.Source: §D.5 · p. 24
• The sweep varies recomputation window radius w on RM-Bench with confidence deferral removed, so every step unmasks all k candidates.Source: Figure 12 · p. 25

D.6 VERIFICATION ORDER ABLATION

• Candidate verification order does not meaningfully affect quality when the dLLM is reconditioned on each previously unmasked token.The ablation removes confidence deferral and unmasks exactly k = 4 tokens per step.Source: §D.6 · p. 25, Table 8 · p. 25
• The ordering ablation isolates verification order by fixing k = 4 and disabling confidence deferral.Source: §D.6 · p. 25, Table 8 · p. 25

D.7 EFFECT OF GREEDY TOKEN SELECTION

• Greedy selection improves Top@1 by 0.30 and LMUnit by 0.05, with slightly higher throughput.The comparison uses the full RM-Bench test set on Dream-7B with k = 8.Source: §D.7 · p. 25, Table 9 · p. 25

D.8 EXTENDED COMPUTATIONAL ANALYSIS

• FastGuide uses fewer FLOPs than other methods because guidance caching reduces repeated guidance computation and sparse recomputation costs less than a full forward pass.In the k = 16 regime, more deferred tokens lead to slightly increased compute.Source: §D.8 · p. 25, Table 10 · p. 26
• FastGuide does not increase peak GPU memory across the compared methods; memory is dominated by diffusion- and reward-model weights.Table 10 reports peak memory and total compute per generation on RM-Bench.Source: §D.8 · p. 25, Table 10 · p. 26
• The RM-Bench timing breakdown distinguishes full dLLM forwards, guidance computation, and sparse dLLM forwards at batch size 4 on Dream-7B.Sparse forwards occur k − 1 times per hybrid decoding step; full dLLM and guidance computation occur once per step.Source: §D.8 · pp. 25–26, Table 11 · p. 26

• Across chat and safety examples, guided DAPD produces incompatible tokens and repeated numbering, while FastGuide remains coherent and fluent with higher reward-model scores.Examples include the malformed “5.emonial” and a repeated list number.Source: §D.9 · p. 26, Figure 14 · p. 28, Figure 15 · p. 29
• Confidence deferral suppresses repeated or low-confidence proposals for later context, revising words and sentence endings and improving final quality at the cost of inference time.Examples show repeated “mindful” proposals deferred after one is unmasked, and other deferred words later replaced.Source: §D.9 · p. 26, Figure 16 · p. 29, Figure 17 · p. 30
• Guided DAPD can produce an incorrect final answer while most of its derivation remains masked, unlike coherent unguided DAPD generation.Source: §D.9 · p. 26, Figure 13 · p. 27