PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaTE: Accelerating Diffusion LLMs via Dynamic Token Execution

Minghan Jiang, Jiayi Wang, Shuaiting Li, Haibin Shen, Kejie Huang

arXiv:2610.11284v1cs.ARcs.AI

TL;DR

dLLM decoding enables parallel refinement, but locally coupled tokens can require repeated revisions and additional denoising steps. DynaTE dynamically adapts execution to token states with selective pruning, local refinement, and hardware support for variable workloads. Across two dLLMs, it reports speedups and energy-efficiency gains over prior dLLM accelerators and Jetson AGX Orin.

  • Problem

    Locally coupled tokens in parallel dLLM decoding can require repeated revisions and additional sequence-wide denoising steps, while AR decoding has limited parallelism from sequential token dependencies.

    Source: §I · pp. 1–2

  • Method

    DynaTE adapts execution to evolving token states with token-wise pruning, FLDD-guided local refinement, and hardware for variable workloads.

    Source: §I · p. 2

  • Results

    2.05–2.78× speedup and 2.99–3.93× higher energy efficiency over evaluated prior dLLM accelerators; 2.55× speedup and 6.07× higher energy efficiency over Jetson AGX Orin.

    Source: §I · p. 2

  • Takeaways & Limitations

    Algorithmic optimizations yield lower gains on a fixed systolic array than when coupled with the proposed datapath, reconfigurable array, and MSM dataflow.

    Source: 7) Overall Ablation · p. 12

Abstract

from arXiv · show

Diffusion-based LLMs (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs by enabling bidirectional parallel refinement, alleviating the sequential decoding bottleneck of AR generation. However, their parallel iterative refinement mismatches AR accelerators optimized for sequential decoding and their discrete token generation differs from DiT accelerators designed for continuous denoising. Recent dLLM accelerators have explored workload-specific optimizations to reduce vocabulary processing overhead and redundant computation across denoising iterations. However, these approaches retain all tokens in parallel execution, despite varying token refinement utility and execution requirements. This paper presents DynaTE, a hardware--software co-design architecture that dynamically adapts accelerator execution to evolving token states during dLLM decoding. DynaTE first enables adaptive token execution by skipping low-utility token computation, while a dimension-reconfigurable PE array maintains high utilization under varying active-token patterns. Second, DynaTE exploits dynamic token dependencies through FLDD to refine a small number of locally dependent tokens within the current iteration, reducing the overall number of denoising iterations, while a Merge--Split--Merge dataflow hides the resulting serial overhead. Third, a streaming vocabulary engine interleaves multiple token streams from the LM head to accommodate irregular output variations caused by selective token computation and uneven vocabulary-selection demands. Evaluated on two representative dLLMs, DynaTE achieves 2.05--2.78$\times$ speedup and 2.99--3.93$\times$ higher energy efficiency over state-of-the-art dLLM accelerators, while delivering 2.55$\times$ speedup and 6.07$\times$ higher energy efficiency over Jetson AGX Orin.

I. INTRODUCTION

DynaTE adapts dLLM accelerator execution to evolving token states, addressing execution mismatches and token-level variation in refinement utility and dependencies. It combines selective computation with hardware mechanisms for dynamic workloads and reports speedups and energy-efficiency gains over prior dLLM accelerators and Jetson AGX Orin.

I. INTRODUCTION

  • Existing AR accelerators target sequential single-token generation, DiT accelerators target continuous latent updates, and prior dLLM accelerators retain all tokens in parallel.dLLM decoding instead performs vocabulary projection and sampling during iterative refinement.Source: §I · p. 1
  • DynaTE adapts accelerator execution to evolving token states through token-wise pruning, selective local refinement, and hardware support for variable workloads.The design also includes a streaming multi-token vocabulary engine that avoids full-logit HBM transfers.Source: §I · p. 2
  • Token utility-aware execution skips low-benefit Transformer computation, while a dimension-reconfigurable PE array adjusts execution granularity for varying active-token counts.The array reconfigures execution between token and model dimensions to improve utilization across participation patterns.Source: §I · p. 2
  • FLDD identifies transient local dependencies for selective intra-iteration refinement, and an MSM dataflow integrates that serial work with parallel execution.This targets additional sequence-wide denoising steps caused by locally coupled tokens.Source: §I · p. 2
  • A streaming multi-token vocabulary engine handles irregular token-wise outputs and reduces unnecessary data movement during multi-token generation.It processes vocabulary outputs as streams rather than requiring full-logit HBM transfers.Source: §I · p. 2
  • 2.05–2.78× speedup and 2.99–3.93× higher energy efficiency over evaluated prior dLLM accelerators; 2.55× speedup and 6.07× higher energy efficiency over Jetson AGX Orin.Evaluated on LLaDA-8B and Dream-7B across five benchmarks.Source: §I · p. 2

II. BACKGROUND & MOTIVATION

The section introduces iterative dLLM decoding and Fast-dLLM’s approximate KV reuse, then contrasts dLLM execution with AR and DiT accelerators. Its workload characterization identifies changing token utility, sparse transient dependencies, and irregular vocabulary demand as opportunities for dynamic execution.

A. dLLM basics

  • dLLMs reconstruct text over denoising iterations by recomputing tokens in parallel and remasking low-confidence tokens, enabling bidirectional context modeling and parallel decoding.Fast-dLLM adds part-wise approximate KV caching across adjacent diffusion steps, reusing attention activations based on temporal similarity.Source: A. dLLM basics · p. 3
  • Token-level variation in refinement utility and local dependencies creates opportunities beyond part-wise decoding and KV-cache reuse.These variations offer further opportunities to reduce denoising computation.Source: A. dLLM basics · p. 3

B. Execution Mismatch with Existing Accelerators

  • AR accelerators assume sequential single-token generation and appended states, whereas dLLMs update and revisit multiple positions, making fixed single-token execution inefficient.The active workload changes during generation.Source: B. Execution Mismatch with Existing… · p. 3
  • DiT accelerators densely update a fixed layout of continuous latents, while dLLMs produce vocabulary distributions and make discrete commit or remask decisions.The downstream dataflow therefore differs from vocabulary-wide processing and token-state control.Source: B. Execution Mismatch with Existing… · p. 3

C. Workload Characterization and Design Opportunities

  • Many unresolved positions gain little from repeated updates, and the high-gain positions change across iterations, so a fixed token mask cannot capture the redundancy.Runtime identification of low-utility positions can adapt active computation.Source: C. Workload Characterization and… · p. 3
  • Strong local dependencies occur at isolated token pairs and iterations, so serial refinement should target temporary hotspots rather than the whole block.A nearby token’s updated context can make short serial refinement more effective than another fully parallel iteration.Source: C. Workload Characterization and… · p. 4
  • Full N × V logits are written to and read from HBM each iteration, while irregular Top-32 candidate arrivals can leave independent vocabulary-selection streams underutilized.Interleaving streams can smooth aggregate demand; vocabulary candidates and normalization statistics can instead be extracted before HBM transfer.Source: C. Workload Characterization and… · p. 4

III. ALGORITHM METHODOLOGY

DynaTE prunes tokens predicted to offer little refinement benefit, then safeguards the choice using attention received from active queries. Pruning skips query-side attention and FFN work while preserving context needed by active tokens.

A. Token-wise Query-Pruning Strategy

  • DynaTE identifies masked tokens with low expected refinement utility from prior-iteration distribution statistics and decoded-token positions.The candidate set is formed within the current decoding block.Source: A. Token-wise Query-Pruning Strategy · p. 4
  • Query-skip bypasses selected tokens’ query-side attention and FFN computations while retaining their keys and values for active tokens.The pruned token’s hidden state passes unchanged to the next layer, and its keys and values are recomputed at each layer.Source: Figure 4 · p. 4, A. Token-wise Query-Pruning Strategy · p. 4
  • Candidate tokens’ average attention mass is accumulated layer by layer as a safeguard against pruning tokens influential to active queries.The probabilities are accumulated during streamed attention computation without storing the full attention matrix or adding off-chip accesses.Source: A. Token-wise Query-Pruning Strategy · p. 4
  • Pruning remains in effect through the rest of the current denoising iteration and is reassessed in the next iteration.A token is newly pruned at the next layer only when it has low predicted refinement utility and receives little attention from active queries.Source: A. Token-wise Query-Pruning Strategy · p. 4

B. Fast Local Dependency Detection for Selective Serial Refinement

Fast Local Dependency Detection (FLDD) uses neighboring positions’ cross-iteration distribution changes to find anchors for local serial refinement. It targets opportunities to refine locally with low overhead, rather than exhaustively detecting dependencies.

B. Fast Local Dependency Detection for Selective Serial Refinement

  • FLDD detects local dependencies because parallel predictions share pre-update context, so newly resolved tokens cannot immediately update neighboring positions.Without that within-iteration update, locally coupled tokens may undergo repeated revisions and add sequence-wide denoising steps.Source: B. Fast Local Dependency Detection for… · p. 4
  • FLDD compares neighboring positions’ cross-iteration Top-32 truncated KL divergences; similar divergence magnitudes suggest a potential local dependency.This comparison is one signal used to identify anchors for local serial refinement.Source: B. Fast Local Dependency Detection for… · pp. 4–5
  • FLDD also compares position i’s Top-4 candidate sets across consecutive iterations, with small overlap indicating substantial candidate change.The design uses K = 4 for this comparison.Source: B. Fast Local Dependency Detection for… · p. 5
  • FLDD combines the distribution-change and candidate-overlap signals with current Top-32 truncated entropy to score local dependency.For unresolved positions, lower entropy indicates probability mass concentrated among fewer candidates, not cross-iteration stability.Source: B. Fast Local Dependency Detection for… · p. 5
  • When a position exceeds the FLDD threshold, it becomes an anchor whose neighbor is refined using the updated anchor’s KV context.The parallel path retains the pre-anchor context; FLDD aims to reduce denoising iterations without exhaustive dependency detection.Source: B. Fast Local Dependency Detection for… · p. 5

C. Adaptive Activation Precision

DynaTE adapts activation precision to decoding progress and module sensitivity, while its PE array changes execution dimensions to handle token pruning and serial refinement. The design also places computation, memory, streaming, and control components under coordinated runtime control.

C. Adaptive Activation Precision

  • DynaTE profiles module quantization sensitivity across decoded-token ratios, then uses a lookup table and runtime controller to select W8A8 or W8A4 precision.The profiles cover QKV projections, the LM head, and FFN projections; a shared splittable MAC executes one W8A8 or two W8A4 operations in parallel.Source: C. Adaptive Activation Precision · p. 5

IV. HARDWARE OPTIMIZATION

  • The architecture comprises a computation core, memory hierarchy, streaming processing unit, and control logic, with iteration control configuring pruning, serial refinement, array shape, and precision.The Global Controller manages the overall denoising flow.Source: §IV · p. 5

A. Reconfigurable Computation Dimension

  • With 2,048 MAC units fixed, the dimension-reconfigurable array supports 64 × 32, 128 × 16, 256 × 8, and 512 × 4 configurations.The array has eight 64×4 subarrays and changes effective matrix dimensions for attention and FFN operations under token pruning and serial refinement.Source: A. Reconfigurable Computation Dimension · p. 5

B. Splittable MAC Unit

DynaTE shares MAC datapaths across activation-precision modes and streams irregular LM-head outputs through multi-token vocabulary selection. Interleaving token streams smooths variable candidate arrivals while the engine produces sampling results and reusable distribution statistics.

B. Splittable MAC Unit

  • A shared splittable MAC executes either one W8A8 operation or two independent W8A4 operations without duplicating the accumulator datapath.The design shares multiplier and adder resources across the precision modes.Source: B. Splittable MAC Unit · p. 6

C. Streaming Multi-Token Vocabulary Processing Engine

  • The vocabulary engine reshapes valid LM-head outputs into 4–32 logit lanes, restores token order, and converts bursty array outputs into a uniform vocabulary stream.The lanes are packed into a 32-bank elastic buffer before row scanning.Source: C. Streaming Multi-Token Vocabulary… · p. 6
  • The Top-32 selector warm-starts each token’s threshold from its previous candidate IDs, filters streamed logits, and revalidates buffered candidates against the latest threshold.This warm start reduces comparison pressure while preserving exact Top-32 selection.Source: C. Streaming Multi-Token Vocabulary… · pp. 6–7
  • Per-token survivor buffers feed shared tournament-pipeline cores, which prioritize high-pressure and long-waiting streams before round-robin service.Interleaving streams fills the dependency interval between updates to the same Top-32 table, smoothing arrivals and improving pipeline utilization.Source: C. Streaming Multi-Token Vocabulary… · p. 7
  • The engine derives Top-32 results and shared distribution statistics for sampling, FLDD, and subsequent token pruning while eliminating full-logit HBM transfers.The statistics include cross-iteration KL divergence and Top-4 overlap; Top-32 results support sampling.Source: C. Streaming Multi-Token Vocabulary… · p. 7

D. Merge-Split-Merge Dataflow

MSM integrates FLDD-selected local serial refinement into parallel dLLM decoding after the anchor is committed. It combines the serial and parallel paths while adapting PE dimensions for GEMV work.

D. Merge-Split-Merge Dataflow

  • MSM interleaves serial refinement with parallel generation after the FLDD-selected anchor is committed, conditioning local refinement on the updated anchor.The design addresses poor PE utilization for GEMV refinement and stalls to parallel token generation when refinement runs independently.Source: D. Merge-Split-Merge Dataflow · pp. 7–8
  • For QKV projection, the merge stage batches the N parallel tokens with the updated anchor and its neighbor-refinement token.Attention A uses the pre-anchor context; Attention B reuses the parallel path’s KV context with the anchor entries replaced by updated KV, propagating the updated anchor through layers to condition neighbor refinement.Source: D. Merge-Split-Merge Dataflow · p. 8
  • The split attention outputs are merged into an (N + 2)-token batch for FFN computation, while Attention B’s GEMV operations use the dimension-reconfigurable PE array.The PE array adapts its computation dimensions to improve GEMV utilization.Source: D. Merge-Split-Merge Dataflow · p. 8
  • Only the locally refined neighbor replaces its corresponding parallel prediction, and MSM mitigates pipeline stalls while enabling dimension-adaptive GEMV execution.Algorithm 2 describes FLDD-guided anchor selection and local refinement through MSM.Source: D. Merge-Split-Merge Dataflow · p. 8

V. EVALUATION

The evaluation compares DynaTE with commercial and accelerator baselines, describes its hardware and simulation setup, and tests two dLLMs across five benchmarks.

1) Hardware Baselines:

  • DynaTE is compared with Jetson AGX Orin, a matched fixed-shape systolic-array baseline, and dLLM-OPU and DART normalized to equivalent MAC count.Fixed-SA matches DynaTE’s technology, frequency, compute capacity, on-chip storage, and off-chip bandwidth.Source: 1) Hardware Baselines · p. 8

2) DynaTE Implementation:

  • DynaTE uses 2,048 splittable MAC units in a dimension-reconfigurable array and a streaming vocabulary engine, modeled with a cycle-accurate simulator.The implementation is in TSMC 28-nm technology at 1 GHz; Table I summarizes the hardware configuration.Source: 2) DynaTE Implementation · p. 8, Table I · p. 9

3) Models and Benchmarks:

  • The evaluation uses LLaDA-8B and Dream-7B with part-wise decoding, parts of 32 tokens, and up to 32 denoising iterations per part, across five reasoning, code, and knowledge benchmarks.The benchmark suite comprises GSM8K, MATH, HumanEval, MBPP, and MMLU.Source: 3) Models and Benchmarks · pp. 8–9

B. Design Space Exploration

Design-space experiments select buffer and tournament-core configurations, then assess token pruning and FLDD’s iteration savings and dependency coverage.

1) Survivor Buffer Depth:

  • Increasing survivor-queue depth from two to eight entries per token rapidly reduces backpressure; depth 16 adds little benefit while nearly doubling buffer area.The exploration reports both buffer-full backpressure and buffer area.Source: 1) Survivor Buffer Depth · p. 9, Figure 11 · p. 9

2) Tournament Core Count:

  • Two shared five-stage tournament cores sharply reduce Top-32 latency versus one, while four or eight add little benefit as engine area increases.The design therefore shares two cores across active token contexts.Source: 2) Tournament Core Count · p. 9, Figure 12 · p. 9

1) Token-Pruning Effectiveness:

  • 26.1–40.0% fewer Transformer MACs are executed across evaluated workloads with token-wise pruning, averaging 32.7% for LLaDA-8B and 34.2% for Dream-7B versus no pruning.Attention-mass verification prevents influential tokens from being pruned.Source: 1) Token-Pruning Effectiveness · p. 9, Figure 13 · p. 9

2) FLDD Effectiveness:

  • 21.4–34.3% fewer denoising iterations are required with FLDD, averaging 30.1% for LLaDA-8B and 26.5% for Dream-7B, with local refinements averaging 14.3% of baseline iterations.With MSM, the refinements incur an 8.54% per-iteration latency overhead.Source: 2) FLDD Effectiveness · pp. 9–10, Figure 14 · p. 10
  • Adjacent tokens make up 71% of oracle dependency pairs on LLaDA-8B and 65% on Dream-7B; FLDD captures 45.2% and 39.1%, respectively.FLDD is intended as a low-overhead heuristic for useful local refinement opportunities, not exhaustive dependency coverage.Source: 2) FLDD Effectiveness · p. 10, Figure 15 · p. 10

3) Generation Quality:

DynaTE’s evaluation reports end-to-end speed and energy gains, generation-quality changes, and results for its reconfigurable array, MSM dataflow, hardware costs, and streaming vocabulary engine.

3) Generation Quality:

  • With all optimizations enabled, average and maximum generation-score reductions are 0.41 and 0.7 points, respectively, across both models and all benchmarks.Token pruning reduces scores by at most 0.6 points, FLDD changes them by at most 0.5 points, and adaptive quantization reduces them by at most 0.3 points.Source: 3) Generation Quality · p. 10, Table II · p. 10

2) Reconfigurable-Array Effectiveness:

3) MSM Dataflow Effectiveness:

  • MSM reduces per-iteration latency by 39.4% versus independently executed serial refinement and limits local refinement to an 8.54% increase over parallel-only execution.The normalized latency breakdown compares MSM with independent serial refinement.Source: 3) MSM Dataflow Effectiveness · p. 10, Figure 18 · p. 11

1) End-to-End Performance and Energy:

4) Area and Energy Breakdown:

  • Iteration-adaptive precision reduces PE-array energy by 32.6% versus uniform W8A8, while supporting multiple precision modes increases MAC area by only 6.17%.The streaming process unit contributes 3.2% of logic area and 3.9% of logic energy; the MAC array and original PE datapath remain the dominant contributors.Source: Figure 19 · p. 10, Table III · p. 10, 4) Area and Energy Breakdown · p. 11

5) Streaming Vocabulary Processing:

6) Multi-token Scheduler Effectiveness:

DynaTE adapts execution to token-level dynamics through scheduling, pruning, local refinement, and streaming vocabulary processing. The reported evaluations show gains over prior accelerators, while ablations and related-work comparisons describe the roles and distinctions of its design.

6) Multi-token Scheduler Effectiveness:

  • Interleaving independent token contexts raises tournament-core utilization from about 20% to 97.6%, while Prior-ID seeding reduces logits admitted to the tournament tree by 91.68%.A single token can issue only one tournament update every five cycles because of the pipeline dependency.Source: 6) Multi-token Scheduler Effectiveness · p. 11, Figure 21 · p. 12

7) Overall Ablation:

  • Cumulative speedup rises from 1.26× with mixed precision to 2.26× with token pruning, 3.07× with FLDD, and 4.32× with streaming vocabulary processing.On a fixed systolic array, mixed precision, pruning, and FLDD reach cumulative speedups of 1.07×, 1.51×, and 2.08×; the paper attributes the lower gains to the importance of coupling algorithmic optimizations with hardware and dataflow.Source: 7) Overall Ablation · pp. 11–12, Figure 22 · p. 12

8) Comparison with Prior dLLM Accelerators:

  • Table IV reports average results across two models and five benchmarks at 256-token generation length; DynaTE improves speed by 2.05–4.32× and energy efficiency by 2.99–5.28× compared with three accelerators.DART, SA, and DynaTE use the same HBM configuration, while dLLM-OPU retains its native FPGA memory system and decoding pipeline.Source: 8) Comparison with Prior dLLM… · p. 12, Table IV · p. 12

VI. RELATED WORK

  • Related work distinguishes DynaTE from DART [18], dLLM-OPU, SureLock, and prior array and Top-K designs through streaming, utility-based pruning, and token-driven reconfiguration.DynaTE streams logits from the LM head instead of transferring full logits through HBM, targets low-utility masked tokens, and interleaves candidates from multiple token streams.Source: §VI · pp. 12–13

VII. CONCLUSION

  • Across two dLLMs and five benchmarks, DynaTE delivers 2.05–2.78× speedup and 2.99–3.93× higher energy efficiency over state-of-the-art dLLM accelerators, plus 2.55× speedup and 6.07× higher energy efficiency over Jetson AGX Orin.The conclusion attributes these results to token-wise pruning, FLDD-guided refinement, reconfigurable execution, and streaming vocabulary processing.Source: §VII · p. 13
Loading 2610.11284v1…