Source-linked AI summary
TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
Zhiben Chen, Youpeng Zhao, Yang Sui, Jun Wang, Yuzhang Shang
TL;DR
Deploying MoE diffusion language models on resource-constrained devices is difficult because expert routing can create substantial CPU computation and GPU-CPU I/O bottlenecks. TIDE periodically refreshes and reuses GPU experts using an optimized interval, achieving up to 1.4× and 1.5× throughput improvements on LLaDA2.0-mini and LLaDA2.0-flash, respectively.
Problem
MoE diffusion inference can activate many experts per block, making GPU-CPU I/O and CPU computation bottlenecks that require coordinated scheduling.
Method
TIDE periodically refreshes and reuses GPU experts based on temporally stable activation patterns, optimizing the refresh interval to reduce I/O and CPU computation.
Results
Up to 1.4× and 1.5× throughput improvements are achieved on LLaDA2.0-mini and LLaDA2.0-flash, respectively, in a single GPU-CPU system.
Takeaways & Limitations
TIDE improves end-to-end MoE diffusion decoding throughput without accuracy loss and requires no model training.
Takeaways & Limitations
TIDE relies on adjacent-step activation similarity, which decreases across farther-apart steps, and assumes experts are split between GPU and host memory.
Abstract
from arXiv · showhide
Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive (AR) models, offering better hardware utilization and bidirectional context through parallel block-level decoding. However, as dLLMs continue to scale up with mixture-of-experts (MoE) architectures, their deployment on resource-constrained devices remains an open challenge. Existing AR-based methods often incur either prohibitive I/O overhead or significant compute bottlenecks. In this work, we propose TIDE, a novel resource-efficient inference system that leverages the temporal stability of expert activations during the diffusion process within the block. Specifically, we leverage the temporal stability of expert activations during the diffusion process within the block and introduce an interval-based expert refresh strategy that updates the expert placement in an I/O-aware fashion. To ensure optimal performance, we formulate the inference scheduling as a mathematical programming problem, solving for the optimal interval that minimizes I/O traffic and CPU computation. Most importantly, TIDE is a lossless optimization that requires no model training, providing a "free lunch" acceleration for dLLM inference. In a single GPU-CPU system, we demonstrate that TIDE achieves up to 1.4$\times$ and 1.5$\times$ throughput improvements over prior baselines on LLaDA2.0-mini and LLaDA2.0-flash models, respectively.
1 Introduction
TIDE addresses the challenge of deploying MoE dLLMs on resource-constrained platforms by exploiting cross-step stability in expert routing. Its training-free, lossless interval-based scheduling reduces unnecessary expert migration while preserving model accuracy.
- Motivation: MoE dLLM deployment is difficult because each denoising step activates a larger, more diverse expert set than an AR step, creating massive CPU-GPU I/O traffic.Routing computation to CPU experts is an alternative, but modern CPU execution can be orders of magnitude slower than GPU execution.
- Contributions and evaluation: TIDE provides training-free, lossless MoE-dLLM inference for resource-constrained environments, improving system efficiency without any accuracy drop.The method is evaluated on LLaDA2.0 models in a single GPU-CPU system against previous baselines.
- Key insight: TIDE reuses GPU experts across adjacent denoising steps by applying interval-based expert refreshes, exploiting similar expert activation patterns within a block.The strategy aims to maintain a high GPU expert hit rate while reducing expert migration overhead.
- System design: At refresh steps, TIDE promotes CPU experts with the most token hits to GPU memory within its budget; at skipped steps, it reuses the current placement asynchronously.This separates expert placement updates from steps that can reuse the existing GPU-CPU arrangement.
- Optimization: TIDE determines the optimal refresh interval by modeling latency overheads and solving a constrained mathematical programming problem with an analytical model.The formulation optimizes expert placement scheduling to avoid unnecessary I/O overhead.
2 Related Work
Related work positions dLLMs as diffusion-based generative models that iteratively denoise data, while highlighting inference-efficiency efforts and the deployment challenges of MoE architectures. Existing optimization directions include compression, caching, and efficient decoding, whereas MoE models trade reduced per-token computation for substantial memory demands.
- Diffusion Large Language Models (dLLMs): dLLMs generate outputs through iterative denoising, extending diffusion modeling from image and video generation to language applications.
- Inference Optimization for dLLMs: Inference optimization for dLLMs has explored model compression, improved caching, and efficient decoding strategies, particularly for dense models.
- Mixture-of-Expert (MoE): MoE architectures increase parameter capacity through multiple FFNs while activating only a subset of experts per token to reduce effective computation.
- Mixture-of-Expert (MoE): Efficient MoE deployment remains challenging in resource-constrained settings because these models have a large memory footprint.
3 Methodology
TIDE addresses resource-constrained MoE-dLLM inference by exploiting temporally stable expert activations to refresh GPU-resident experts at intervals selected through an I/O–CPU computation tradeoff. The method preserves model outputs without training or accuracy degradation.
- Resource-Constrained MoE Inference: 46B+ parameters in Mixtral-8x7B require over 94 GB of FP16 GPU VRAM, exceeding a single H100 80 GB GPU.This motivates placing some expert weights in host memory.
- Temporal Expert Locality: 0.985 mean within-block cosine similarity shows that adjacent denoising steps produce highly similar expert activation patterns.This temporal locality makes short-window reuse of expert-placement decisions possible.
- Interval-Based Expert Refresh: TIDE refreshes the GPU-resident expert set every τ steps by promoting host experts with the highest token hits, while skipped steps perform no expert migration.The policy partitions each block into refresh and skipped steps.
- Refresh-Interval Optimization: TIDE minimizes total expert migration and CPU execution costs by selecting an interval τ that balances I/O traffic against increasing CPU computation.The CPU cost is modeled as LatCPU(τ) ≈ CCPU · T · B · f(τ), where f(τ) is monotonically increasing.
- Lossless Inference: TIDE preserves model outputs because it changes neither MoE routing or model weights nor parallel decoding, making the optimization lossless and training-free.Each token is assigned the same experts as in GPU-only execution, with no accuracy degradation.
4 Experiments
Experiments evaluate TIDE on two LLaDA2.0 MoE models using sanitized MBPP, comparing it with CPU-computation and per-denoising-step offloading baselines. TIDE delivers the highest throughput, while optimized expert-refresh intervals consistently outperform alternative interval choices across decoding configurations.
- Experimental Setup: Experiments use LLaDA2.0-mini and LLaDA2.0-flash, each with 256 FFN experts and top_k = 8, evaluated on sanitized MBPP.The models are evaluated under the LLaDA2.0 architecture.
- Baselines and Metrics: TIDE is compared with Fiddler’s static expert placement and Mixtral-Offload’s expert offloading at every denoising step.The comparison targets inference efficiency using decode throughput and related system metrics.
- Throughput Results: 1.2∼1.4× higher throughput over Mixtral-Offload is achieved consistently by TIDE across the evaluated settings.The improvement is attributed to accounting for diffusion-specific expert activation patterns, which prior methods do not consider.
- Impact of the Refresh Interval τ: Optimized τ outperforms τ = 1 and random interval choices across models, GPU expert budgets, and block lengths.The study uses GPU expert budgets of 64/128, block lengths of 32/64, confidence threshold 0.95, and generation length 1024.
- Impact of the Refresh Interval τ: The interval-based strategy avoids redundant expert-weight I/O transfers, especially when GPU expert budgets are higher.This mechanism explains TIDE’s advantage over the per-step offloading baseline in the interval study.
- Sensitivity Studies: TIDE consistently achieves the highest throughput across varying block sizes, GPU expert budgets, and confidence thresholds.The sensitivity study covers block sizes 32∼128, GPU expert budgets 32∼128, and confidence thresholds 0.7∼0.95 on LLaDA2.0-mini.
5 Conclusion
TIDE is a resource-efficient, I/O-aware inference system for MoE-based diffusion language models that periodically refreshes expert placement using an optimized interval. In a single GPU-CPU system, it improves decode throughput by up to 1.4× on LLaDA2.0-mini and 1.5× on LLaDA2.0-flash.
- 5 Conclusion: TIDE uses an interval-based expert refresh strategy that periodically updates expert placement by exploiting expert activation patterns during decoding.This avoids redundant I/O transfer and CPU computation.
- 5 Conclusion: TIDE determines the optimal refresh interval by solving an optimization problem to improve end-to-end decode throughput in resource-constrained scenarios.
- 5 Conclusion: 1.4× and 1.5× throughput improvements are achieved on LLaDA2.0-mini and LLaDA2.0-flash, respectively, in a single GPU-CPU system.
A Appendix: Limitation
The work has two limitations: it analyzes expert activation patterns only within blocks and evaluates on limited hardware platforms.
- Limitations: The framework examines expert activation patterns only within each block, leaving further block-level similarity analysis for potential performance improvements.The authors identify this scope as a straightforward choice and suggest broader analysis of the MoE-dLLM decoding procedure as future work.
- Limitations: The evaluation uses limited hardware platforms, with future work needed to explore AMD GPUs and ARM CPUs.The passage specifically names AMD GPUs and ARM CPUs as unexplored platforms.