PaperlayerView exact revision on arXiv

Source-linked AI summary

DIET: Deletion-response Expert Trimming for Video Diffusion Transformers

Jiachang Zhang, Teng Hu, Bohao Feng, Songhang Shen, Wenqiang Wang, Hongqian Deng, Ran Yi

arXiv:2609.37829v1cs.AI

TL;DR

Video DiT-MoEs retain costly expert banks, and existing static pruning criteria do not capture re-routing after deletion. DIET measures deletion responses to select experts and allocate layer budgets; at 50% retention, official VBench Total rose from 0.7941 to 0.8115. The approach halves the checkpoint footprint and supports single-GPU deployment, though cross-architecture generalization remains open.

  • Problem

    Video DiT-MoEs retain their full expert-bank storage despite sparse computation, while static pruning criteria miss layer-level re-routing after expert deletion.

    Source: §1 · p. 2

  • Method

    DIET uses one all-expert calibration and frozen-state replay to form deletion-response signatures, then applies directional ODL and regression-guided inter-layer budget search.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 6

  • Results

    At 50% expert retention without retraining, official VBench Total rose from 0.7941 to 0.8115, while the checkpoint footprint fell from 57 GB to 30 GB.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    The reduced checkpoint enables a model replica to reside entirely on a single 48 GB workstation GPU without multi-GPU sharding overheads.

    Source: §4.1 · p. 7

    The findings use one representative open-source DiT-MoE backbone, so cross-architecture generalization remains open.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video diffusion transformers (DiTs) increasingly adopt mixture-of-experts (MoE) architectures to reduce active computation, but their full expert storage remains costly. Existing one-shot pruning criteria mainly rely on static activation or routing statistics and cannot capture layer-level re-routing after expert deletion. We introduce DIET, a training-free expert pruning framework based on deletion responses. A single all-expert calibration pass records expert outputs and router states for matched conditional and unconditional tokens. Candidate deletions are then replayed from cached tensors, requiring no additional model forward passes. The resulting deletion-response signatures characterize each expert by the changes induced when it is removed. DIET selects retained experts by minimizing Overall Diversity Loss (ODL), which preserves directional coverage in signature space, and combines intra-layer local search with an inter-layer regression-guided budget search to allocate experts across layers. On LingBot-Video 30B-A3B, pruning 50% of experts (6,144 to 3,072) reduces the checkpoint from 57 GB to 30 GB and enables single-card deployment on a 48 GB GPU without fine-tuning. Under a fixed 284-case VBench protocol, the VBench Total increases from 0.7941 to 0.8115. Across tested retention budgets, DIET consistently outperforms competitive pruning baselines adapted from large language models.

1 INTRODUCTION

Video DiT-MoEs reduce active computation but retain costly expert banks, while existing pruning criteria miss the layer-level re-routing caused by deletion. DIET measures deletion responses and uses them to select retained experts and allocate budgets across layers.

1 INTRODUCTION

  • Existing pruning criteria rely on static statistics or isolated reconstruction, missing post-deletion re-routing and the non-additive geometry of diffusion representations.Repeatedly evaluating multi-expert deletions is prohibitively expensive at this scale.Source: §1 · p. 2
  • DIET records expert outputs and router states in one calibration pass, then replays all 6,144 layer-local deletions with zero additional model forward passes.It pools the resulting responses into signatures and uses Overall Diversity Loss for intra-layer selection and regression-guided search for inter-layer budgets.Source: §1 · p. 2
  • 50% expert pruning on LingBot-Video 30B-A3B reduces the checkpoint from 57 GB to 30 GB and raises official VBench Total from 0.7941 to 0.8115.DIET also consistently outperforms ported LLM pruning baselines across tested budgets.Source: §1 · p. 2
  • Figure 1 depicts the pipeline from grouped routing and cached deletion replay to signature-based retention and regression-guided layer allocation.The allocation is optimized against uniform references at different retention ratios.Source: Figure 1 · p. 3

2 RELATED WORK

MoE scaling has moved from language models into video diffusion, where sparse routing reduces active computation but leaves the full expert bank in device memory. Prior pruning approaches use static scores, reconstruction, or interaction models, while DIET targets exact grouped re-routing and directional coverage for video DiT-MoEs.

2.1 MIXTURE-OF-EXPERTS: FROM LANGUAGE TO VIDEO DIFFUSION

  • Video DiT-MoEs inherit sparse MoE scaling, but routing sparsity reduces active FLOPs without reducing the expert bank's device-memory footprint.Large video models may consequently require multi-GPU sharding just to hold inactive weights.Source: §2.1 · pp. 3–4

2.2 MOE PRUNING AND RETAINED EXPERT SELECTION

  • Static scalar scores rank experts using routing and activation statistics, but do not capture the re-routing triggered by deletion.Examples include routing frequency and router-weighted activations (Lasby et al., 2026), frequency–variance interactions (Zhang et al., 2026), and unified signals (Liu et al., 2026b).Source: §2.2 · p. 4
  • Re-evaluation methods such as NAEE (Lu et al., 2024) replay cached layer inputs and outputs, but combinatorial replay is costly for fine-grained video DiTs and strict reconstruction can miss directional manifold coverage.The cited methods were primarily designed for language MoEs without grouped routing.Source: §2.2 · p. 4
  • DIET measures exact grouped re-routing at frozen calibration states without extra forward passes and frames retained-set selection as geometric coverage over deletion signatures.The gap is that language-oriented objectives do not transfer cleanly to diffusion denoising trajectories, classifier-free guidance, and non-additive response manifolds.Source: §2.2 · p. 4

3 METHODOLOGY

DIET separates expert pruning into measuring each deletion's layer response, choosing retained experts within layers, and allocating retention budgets across layers. It uses frozen-state replay, directional Overall Diversity Loss, and a regression-guided allocation loop validated with end-to-end generation.

3 METHODOLOGY

  • Each expert deletion changes grouped routing and redistributes gate weight, so raw routing frequency and independent expert scores cannot reliably identify layer-level pruning choices.DIET therefore measures re-routing directly, then uses signatures for per-layer selection and a separate inter-layer allocation decision.Source: §3 · pp. 4–5

3.1 DELETION-RESPONSE SIGNATURES VIA FROZEN STATE REPLAY

  • A single all-expert calibration records router and expert-output states, allowing 6,144 leave-one-out deletions to be replayed with tensor arithmetic and zero additional model forward passes.The naive alternative would require at least 6,144 forward passes across 48 layers and 128 experts per layer.Source: §3.1 · p. 5
  • Deletion-response signatures capture each expert's layer-output perturbation, including discrete token re-routing and gate re-normalization; CFG responses are computed on guidance-weighted outputs.Case-wise token-averaged perturbation vectors are concatenated into fixed-dimensional signatures.Source: §3.1 · p. 5

3.2 SELECTING RETAINED EXPERTS BY OVERALL DIVERSITY LOSS

  • ODL selects retained experts by matching each pruned expert to its nearest surviving signature and summing cosine-distance penalties, preserving directional coverage rather than magnitude.Summing matched distances avoids cancellation between opposing responses; local search and annealing refine each layer's retained set.Source: §3.2 · p. 6

3.3 INTER-LAYER BUDGET ALLOCATION

  • Because layers have non-uniform pruning sensitivity, DIET fits a regularized linear surrogate from per-layer retention counts to benchmark scores and proposes allocations under the global budget and score constraints.Proposed allocations are checked with full end-to-end video generation and official benchmark evaluation; reported results are empirical, not surrogate predictions.Source: §3.3 · p. 6

4 EXPERIMENTS

DIET improves VBench quality at 50% expert retention while halving checkpoint size, and outperforms the tested pruning baselines. Its analyses show that quality depends on post-deletion responses and layer-specific retention, not simply preserving routing or reconstructing outputs.

4.1 MAIN RESULTS

  • At 50% retention, VBench Total rises from 0.7941 to 0.8115, while the checkpoint shrinks from 57 GB to 30 GB and fits on one 48 GB GPU.The pruned model is evaluated without retraining; the reported gains cover both visual Quality and Semantic scores.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 COMPARISON EXPERIMENT

  • Across retention regimes, DIET establishes the Pareto-optimal frontier; at 50%, its 0.8115 Total exceeds HTS’s 0.7921 by 1.94 points.DIET also remains strongest under uniform layer allocations; the searched comparison measures the full pipeline, including regression-guided allocation.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 ANALYSES

  • Routing is skewed but has no dead-capacity partition: the top 10% of experts handle 23.1% of tokens, while normalized routing entropy is 0.955.This long tail means frequency-based pruning can remove moderately active, functionally critical experts.Source: §4.3 · p. 8
  • DIET’s best-performing mask also causes the largest routing perturbation, retaining 3.25 of 8 originally selected experts per token and reallocating 1.25 units of gating weight.The analysis argues that strict output reconstruction is misaligned with discrete rerouting and nonlinear interactions across transformer blocks; DIET instead preserves directional coverage.Source: Figure 2 · p. 7, §4.3 · pp. 8–9
  • Retention in early, middle, and deep blocks is associated with different visual dimensions, and non-uniform allocation contributes a 1.2-point gain over uniform allocation.Early blocks correlate with spatial coherence and scene composition, middle blocks with dynamics and appearance, and deep blocks with color fidelity and temporal consistency.Source: §4.3 · p. 9

4.4 ABLATION STUDIES

  • At 50% retention, static routing frequency drops Total by 6.30 points, while isolated outputs or router affinities drop it by 2.32–3.02 points; additive single-deletion response minimization drops it by 5.44 points.Low-rank alternatives also trail DIET: D-optimal volume selection by 1.08 points and PCA energy retention by 2.66 points; joint responses depart from linear superposition, with cosine similarity falling to 0.66 at the operating point.Source: §4.4 · p. 9, Table 3 · p. 9
  • Single-swap local search and simulated annealing recover tail directions missed by greedy initialization, while uniform inter-layer budgets cost 1.19 points.The ablation attributes measurable gains to both retained-set refinement and regression-guided allocation.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION AND LIMITATIONS

DIET compresses a deployed video diffusion transformer without post-pruning fine-tuning by replaying cached states and selecting experts for directional functional coverage. The conclusion reports a smaller checkpoint and higher observed VBench Total, while noting limits in downstream drift tracking and cross-architecture evidence.

5 CONCLUSION AND LIMITATIONS

  • DIET halves the expert bank without fine-tuning or weight updates; frozen-state replay evaluates deletions without extra model forwards, and ODL plus regression-guided allocation reduces the checkpoint to single-GPU scale.For LingBot-Video 30B, the checkpoint falls from 57 GB to 30 GB, and official VBench Total is observed to rise from 0.7941 to 0.8115.Source: §5 · pp. 9–10

A APPENDIX OVERVIEW

  • The appendix covers calibration and synthesis protocols, compute, budget search, selected-mask structure, deployment, robustness, routing, response geometry, ablations, and statistical analysis.It also lists the model architecture and grouped-routing formulation.Source: A APPENDIX OVERVIEW · p. 14

B ARCHITECTURAL FORMULATION AND GROUPED ROUTING

  • Grouped routing selects experts from activated groups; after pruning, logits for removed experts are masked, surviving gates are renormalized, and tokens can be rerouted.The backbone uses 48 layers, 128 experts per layer, four groups, two activated groups, and top-8 routing.Source: B ARCHITECTURAL FORMULATION AND… · p. 14

C EXPERIMENTAL AND CALIBRATION PROTOCOLS

A single all-expert calibration sweep captures routing and expert outputs for replaying deletions, with subsequent counterfactuals requiring no additional model forwards. Evaluation uses paired fixed-seed video generations and an official 284-case VBench protocol; calibration and search costs are front-loaded.

C EXPERIMENTAL AND CALIBRATION PROTOCOLS

  • One instrumented sweep over 120 cases records expert outputs and router states for 368,640 matched conditional–unconditional token pairs, enabling all 6,144 single deletions and joint deletions to be replayed without further model evaluations.The capture evaluates all 128 candidate experts per token, about 16× active sparse-forward compute, but incurs that overhead once.Source: C EXPERIMENTAL AND CALIBRATION PROTOCOLS · p. 15
  • Videos use 240 × 416 resolution, 121 frames, 40 denoising steps, and shared fixed random seeds; official VBench scoring covers 16 dimensions on a fixed 284-case protocol.Calibration prompts are excluded, and missing evaluation pairs are not imputed.Source: C EXPERIMENTAL AND CALIBRATION PROTOCOLS · p. 15

D COMPUTE BUDGET

  • The 120-case calibration sweep costs about 24 GPU-hours, while 87 end-to-end budget-search evaluations cost about 4 × 10^2 GPU-hours; per-mask optimization runs on CPU.After calibration, cached-state counterfactuals avoid the repeated model forwards that exhaustive leave-one-out simulation would require.Source: D COMPUTE BUDGET · p. 15

E REGRESSION-GUIDED LAYER BUDGET SEARCH

The regression-guided search proposes layer allocations from a regularized surrogate and validates candidates end to end. Its selected 50% allocation generalizes across partitions, while the sweep, structural checks, and qualitative and routing analyses characterize the resulting mask.

E REGRESSION-GUIDED LAYER BUDGET SEARCH

  • A regularized linear surrogate maps layer-retention counts to dimension scores and proposes candidates, each of which is validated through full end-to-end synthesis and official scoring.The surrogate is fitted from 87 evaluation trajectories over 48 layer covariates; reported benchmark results use unfiltered completed-case aggregates.Source: E REGRESSION-GUIDED LAYER BUDGET SEARCH · pp. 15–16
  • The selected allocation improves VBench Total by +0.0200 on shared cases and +0.0142 on held-out cases, aligning with the full-manifest improvement of +0.0176.The shared-partition 95% CI is [+0.0018, +0.0429] (n = 139); the held-out CI is [−0.0008, +0.0298] (n = 138).Source: E REGRESSION-GUIDED LAYER BUDGET SEARCH · p. 16, Table A1 · p. 16

F LAYER-BUDGET ALLOCATION SWEEP

G STRUCTURAL PROPERTIES OF THE SELECTED MASK

H EXTENDED QUALITATIVE COMPARISONS

I ROUTING CONCENTRATION AND ACTIVATION ENTROPY

  • Neither CFG branch shows collapsed routing: conditional and unconditional average Gini coefficients are 0.328 and 0.364, with normalized entropies of 0.960 and 0.950, respectively.The top-decile token shares are 0.220 for the conditional branch and 0.243 for the unconditional branch.Source: I ROUTING CONCENTRATION AND ACTIVATION… · p. 24, Figure A8 · p. 24

J ROBUSTNESS TO CALIBRATION SET SIZE

Smaller calibration subsets produce masks with lower overlap against the 120-case reference, while their measured objective values remain close to the best full-calibration solution. The differences are considered against substantial solver-seed variation.

J ROBUSTNESS TO CALIBRATION SET SIZE

K ANALYSIS OF RESPONSE-SPACE GEOMETRY AND OBJECTIVES

Deletion-response signatures are near-orthogonal and high-dimensional, with sparse aligned pairs providing the principal exploitable redundancy. End-to-end comparisons favor DIET’s contextual response-based selection over low-rank and isolated-feature alternatives.

K ANALYSIS OF RESPONSE-SPACE GEOMETRY AND OBJECTIVES

L RESPONSE GEOMETRY ACROSS ARCHITECTURE DEPTH

M IMPLEMENTATION AND PORTING OF LLM MOE BASELINES

The appendix documents adapted pruning baselines, evaluates the selected mask at native resolution, and describes compact-checkpoint verification and deployment. The 50% mask preserves reported performance at 480p while reducing stored weights enough for a single 48 GB GPU.

M IMPLEMENTATION AND PORTING OF LLM MOE BASELINES

  • The adapted baseline suite uses shared calibration tensors, evaluation protocols, and layer budgets, with baseline selections repaired when needed to satisfy grouped-routing feasibility.The ports include router-frequency, REAP, SHAPE, HTS, TB-Coverage, and NAEE-style reconstruction criteria.Source: Table A5 · p. 28, M IMPLEMENTATION AND PORTING OF LLM… · pp. 28–29

N RESOLUTION TRANSFERABILITY

  • The pruned mask scores 0.8179 Total at native 480p versus 0.8115 for the same mask at 240p, with no systematic qualitative artifacts reported.The paired difference between resolutions is +0.0066 (95% CI [−0.0087, +0.0203], n = 279); the two checks differ in mask, case set, and run configuration.Source: N RESOLUTION TRANSFERABILITY · p. 30, Figure A14 · p. 30, Figure A15 · p. 31

O MODEL SLICING AND COMPACT CHECKPOINT ASSEMBLY

  • Physical compaction slices expert tensors while preserving routing semantics, and its runtime agrees with the offline simulator at mean token-wise relative output error 3.4×10^-3.Across 48 layers and 40 scenarios, the residual is attributed to cached-value quantization; ungrouped top-k routing instead raises output divergence to 0.23.Source: O MODEL SLICING AND COMPACT CHECKPOINT… · p. 31

P PHYSICAL MEMORY AND DEPLOYMENT FOOTPRINT

Q ADDITIONAL ABLATIONS

The ablation shows that summed-response selection degrades performance, consistent with single-expert responses failing to capture joint deletion effects.

Q ADDITIONAL ABLATIONS

  • The summed-response norm proxy causes a 5.44-point degradation, with spatial coherence falling from 0.63 to 0.33.This ablation isolates the effect of using an additive proxy for selection.Source: Q ADDITIONAL ABLATIONS · p. 32
  • The observed non-additivity means additive norm metrics fail to model multi-expert removal dynamics.The ablation links this failure to the degradation from summed-response selection.Source: Q ADDITIONAL ABLATIONS · p. 32
  • Joint-deletion cosine similarity to summed single-deletion responses falls from 1.00 at k = 1 to 0.66 at k = 64, while relative reconstruction error reaches 0.95.Figure A16 shows the sharp decline in similarity as deletion cardinality grows.Source: Figure A16 · p. 32, Q ADDITIONAL ABLATIONS · p. 32

R PAIRED STATISTICAL SIGNIFICANCE ANALYSIS

Paired bootstrap analysis reports DIET’s quality differentials against the unpruned model and ported pruning baselines. The accompanying geometric discussion motivates directional coverage because additive norms can miss unique expert directions.

R PAIRED STATISTICAL SIGNIFICANCE ANALYSIS

  • DIET’s paired differential against the unpruned baseline is ∆Total = +0.0176 (95% CI [+0.0057, +0.0306], n = 277).The improvement remains significant when dynamic degree is excluded and the remaining dimensions are re-normalized (+0.0145, 95% CI [+0.0039, +0.0249], n = 262).Source: R PAIRED STATISTICAL SIGNIFICANCE… · p. 32, Figure A17 · p. 33
  • Paired differentials are significant against Routing Frequency (+0.0632), SHAPE (+0.0642), REAP (+0.0349), TB-Coverage (+0.0390), and NAEE-Style Recon (+0.0274), but not against HTS (+0.0180; 95% CI [−0.0013, +0.0377]).The listed significant comparisons have 95% confidence intervals above zero; the HTS interval includes zero.Source: R PAIRED STATISTICAL SIGNIFICANCE… · p. 33, Figure A17 · p. 33
  • Across retention budgets, margins over ported criteria persist at 20% retention and compress toward parity at 80%, while DIET’s advantages over those baselines concentrate in high-level semantic dimensions.Figure A18 presents the 50% dimension-wise comparisons; Figures A19 and A20 show the 20% and 80% cases.Source: R PAIRED STATISTICAL SIGNIFICANCE… · p. 33, Figure A18 · p. 33, Figure A19 · p. 34

S GEOMETRIC SURROGATE FORMULATIONS AND FUNCTIONAL REDUNDANCY

  • Overall Diversity Loss measures the directional discrepancy between an excised expert’s signature and its nearest surviving counterpart, independent of signature scale.The discussion frames ODL as a geometric surrogate for retaining functional capacity.Source: S GEOMETRIC SURROGATE FORMULATIONS AND… · pp. 34–35
  • In near-orthogonal signature manifolds, norm minimization favors low-norm experts, risking removal of directionally unique experts despite their modest activation scales.Signature norm correlates strongly with activation energy (r = 0.91); the discussion associates directional coverage with protecting specialized compositional semantics.Source: S GEOMETRIC SURROGATE FORMULATIONS AND… · p. 35

Related papers

Loading 2609.37829v1…