Source-linked AI summary
Importance-Aware Low-Rank Distillation of Diffusion Transformers
Denis Zavadski, Sebastian Heid, Damjan Kalšan, Stefan Roth, Carsten Rother
TL;DR
Large DiTs are difficult to deploy because their parameter and memory footprints remain after step distillation, while naive low-rank compression can fail in LLMs. The paper introduces SVDtrunc, which allocates low-rank compression by block importance and fine-tunes with modular distillation. Across benchmarks, it achieves a favorable quality–parameter trade-off and remains robust after temporal distillation, although aggressive compression can cause domain drift and loss of distinctive concepts.
Problem
Large DiTs remain costly to deploy, while the sensitivity of DiTs to low-rank approximation is unclear given catastrophic failures reported for naive LLM compression.
Method
SVDtrunc probes block sensitivity, allocates rank reduction non-uniformly under a global budget, and combines projection-level SVD with modular knowledge distillation and rectified-flow training.
Results
Across benchmarks, SVDtrunc achieves a favorable quality–parameter trade-off, with gradual degradation and robustness even after temporal distillation.
Takeaways & Limitations
Projection-level low-rank compression is a promising continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models.
Takeaways & Limitations
At aggressive compression levels, generated images can drift stylistically and lose faithful reproduction of distinctive characters or concepts.
Abstract
from arXiv · showhide
Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to FLUX.dev across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: https://vislearn.github.io/SVDtrunc/
1 Introduction
The paper studies efficient DiT deployment through low-rank compression, finding that truncated SVD degrades DiT quality smoothly rather than catastrophically. SVDtrunc uses block-importance estimation, non-uniform rank allocation, and distillation to preserve quality under compression.
- DiT scaling creates substantial parameter and memory requirements, motivating reductions in diffusion evaluations and model parameters.
- Unlike reports for LLMs, truncated SVD preserves DiT perceptual fidelity with gradual quality loss, including in training-free compression.Projection-matrix compression is more graceful than coarse architectural modification.
- SVDtrunc estimates block importance through controlled compression probing and allocates rank reduction non-uniformly under a global parameter budget.The method preserves critical components while compressing redundant ones more aggressively.
- The approach combines importance-aware rank allocation with modular knowledge distillation and rectified-flow training without altering the architecture.
2 Related Work
Diffusion Transformers improve text-to-image quality but impose substantial memory and runtime costs. Existing compression methods reduce steps or alter architecture, whereas this work uses importance-aware low-rank compression that preserves the original architecture and degrades gracefully under aggressive compression.
- Text-to-Image Generators: Recent DiT models scale from 8 to 20 B parameters, creating substantial memory and runtime constraints for tuning and deployment.Reducing computational and memory cost has become a central research topic.
- Temporal Distillation: Step distillation reduces multi-step generation to 4–8 steps and lowers latency, but leaves parameter count and VRAM footprint unchanged.These methods compress the generation trajectory rather than the backbone architecture.
- Structural Compression: Structural compression removes neurons, heads, residual branches, or blocks, reducing representational capacity at a coarse granularity.Under aggressive compression, these irreversible interventions may cause abrupt performance degradation.
- SVDtrunc Overview: SVDtrunc ranks blocks by comparing images from individually compressed probe models with the original model’s generations, then assigns block-specific compression factors.The procedure uses quality-based block importance to guide the final compressed model.
- Low-Rank Compression: Unlike LLMs, DiTs tolerate truncated SVD of projection matrices with graceful degradation at aggressive compression rates while preserving the original architecture.LLM evidence reports that naive truncated SVD can cause catastrophic failure and may require carefully tuned truncation schedules.
3 Importance-Aware Low-Rank Distillation
SVDtrunc compresses DiT blocks according to their estimated importance, preserving block structure while allocating more compression to less important components. The compressed student is then fine-tuned with rectified-flow training and modular knowledge distillation.
- Importance-Aware Low-Rank Compression: Truncated SVD compresses projection matrices while preserving individual block functionality instead of removing entire Transformer blocks.The method retains the original block structure and removes the least dominant singular values from linear projections.
- Block Importance Analysis: Block importance is estimated through controlled compression probing that compares generated-image quality after compressing each block.Images are generated with identical prompts and seeds, then evaluated against the original outputs using a quality metric.
- Block Importance Analysis: Pixel-wise similarity is unsuitable for ranking compressed blocks because visual changes in pose or appearance do not necessarily indicate lower image quality.The method therefore favors quality measures that are less tied to exact pixel correspondence.
- Importance-Aware Low-Rank Compression: Compression ratios vary by block importance, with less important blocks compressed more aggressively under a global parameter budget.Blocks outside the selected compression subset remain unchanged, while selected blocks receive a linearly decreasing compression schedule based on importance.
- Fine-Tuning: The student is fine-tuned using the rectified-flow objective together with modular knowledge distillation from the uncompressed teacher.Distillation aligns teacher and student velocity predictions and intermediate block representations, while the combined loss includes feature, knowledge-distillation, and rectified-flow terms.
4 Experiments
The experiments show that projection-level truncated SVD compresses DiTs more gracefully than block pruning, remains effective without fine-tuning, and benefits from importance-aware allocation under stronger budgets.
- Redundancy and compression behavior: Truncated SVD preserves functional behavior more gracefully than pruning, with mean CMMD of 0.23 ± 0.65 versus 1.18 ± 3.36 for pruning.Single-block compression produces substantially smaller degradation across all probed blocks, while pruning can catastrophically fail in early blocks.
- Training-free compression: At approximately 70% parameters, SVDtrunc drops only R = 9.86%, whereas EcoDiff drops R = 85.50%.This comparison uses training-free compression and shows stable generation quality for SVDtrunc under a moderate parameter budget.
- Comparison to state of the art: At 68% parameter count, SVDtrunc-m achieves near-full performance with R = 0.75% across GenEval, HPSv2, and DPG; at 57%, SVDtrunc-s remains competitive with R = 2.6%.The medium model exceeds competing methods at comparable or larger budgets, while the small model uses substantially fewer parameters.
- Progressive compression: For P ≥ 0.68, performance remains nearly indistinguishable from the uncompressed model with < 1% average quality reduction, then degrades gradually as compression increases.The progressive-compression results cover global parameter budgets P ∈[0.9, 0.4].
- Qualitative results and domain drifts: Stronger compression can cause stylistic domain drift and reduce faithful reproduction of distinctive characters or concepts despite stable general visual quality.Examples include shifts from cartoonish to realistic styles and loss of identities such as the tower of Pisa, Bugs Bunny, Darth Vader, or the Minions.
- Ablations: Importance-aware allocation matches uniform and random schedules at moderate compression but consistently improves quality retention at 57% and 49% budgets.All schedules degrade smoothly, while importance awareness becomes increasingly beneficial under stronger constraints.
5 Summary and Conclusion
The paper finds that DiTs tolerate projection-level low-rank approximation, with gradual rather than catastrophic degradation under substantial parameter reduction, including without training.
- Summary and conclusion: DiTs exhibit substantial tolerance to projection-level truncated SVD, with generative performance degrading gradually rather than catastrophically even without fine-tuning.The result establishes a favorable quality–parameter trade-off across benchmarks.
A Block-Wise SVD Compression
The compression scheme targets projection modules, which contain most block parameters, while preserving the original block structure and non-linearities.
- Compression target: Most block parameters reside in projection modules, while the remaining components have no or negligible parameter counts and remain untouched.This motivates compressing projections rather than removing entire blocks.
- Compression behavior: SVDtrunc preserves most original performance even at parameter budgets of ≤60%, with graceful degradation as compression increases.The reported average quality reduction R aggregates GenEval, HPSv2, and DPG relative to the corresponding uncompressed model.
- Architecture preservation: SVDtrunc preserves internal block transformations and interactions by compressing projections without replacing or removing complete Transformer blocks.Each Transformer block uses a block-specific compression ratio c.
B Additional Quantitative Results
SVDtrunc compresses diffusion transformers across model scales while preserving benchmark performance and reducing runtime. Across GenEval, DPG, and HPSv2, compression reaches 50% of the original parameter budget with only around a 5% average performance drop.
- SVDtrunc is evaluated on FLUX.dev across compression levels, with PPCL reported as the strongest competing method at the common 68% parameter budget.
- 424 ms for FLUX.1-dev falls to 380 ms for SVDtrunc-m and 354 ms for SVDtrunc-s per generation step.These correspond to relative runtimes of 89.6% and 83.4%, respectively.
- GenEval mainly degrades on object counting for FLUX.dev at 57% compression, whereas PixArt-based compression primarily affects positional generation.Other tasks remain relatively stable in the reported detailed results.
- SVDtrunc successfully compresses both large and small transformer models down to 50% of the original parameter budget with only around a 5% average performance drop across GenEval, DPG, and HPSv2.Detailed results show balanced degradation across subcategories, with no task or category emerging as a systematic failure mode.
- Per-tensor FP8 quantisation remains compatible with SVDtrunc, increasing average quality reduction from 0.75 to 1.87 for SVDtrunc-m and from 2.60 to 3.57 for SVDtrunc-s.
C Training Specifications
The supplementary training setup specifies iterative compression schedules and reports the computational cost of probing and distillation. Detailed HPSv2 and DPG results indicate graceful degradation across categories under compression.
- Higher compression levels are obtained iteratively by further compressing and retraining models trained at lower compression levels.The schedules specify maximum compression cm and slope α for different compression levels.
- SVDtrunc probing requires approximately 175 parallelisable H200 GPU hours, while distillation adds 126 GPU hours for SVDtrunc-m and 63 additional GPU hours for SVDtrunc-s.
- SVDtrunc shows consistent and graceful degradation across all HPSv2 categories without any single category degrading disproportionately.
- SVDtrunc maintains competitive performance across all DPG categories with graceful degradation under stronger compression.
D Ablations
Ablations show that distributing compression across more blocks helps under aggressive budgets, while the choice of fine-tuning objective materially affects quality retention.
- Influence of the Number of Compressed Blocks: At 40–50% compression, distributing rank reduction across more blocks improves quality retention compared with compressing fewer blocks more severely.At moderate compression levels of 20–30%, performance differences remain below one percentage point across configurations.
- Training Specifications: Training details report the number of compressed blocks, maximum compression cm, compression slope α, and training steps for each parameter-budget transition.
- Influence of the Number of Compressed Blocks: Under stronger constraints, broader compression distribution better preserves performance, indicating that redundancy is distributed throughout the Transformer backbone.
- Influence of the Fine-Tuning Objective: A rectified-flow-only objective produces an overall performance decrease of R = 2.12% at a 68% parameter budget.
- Influence of the Fine-Tuning Objective: MKD-only and combined fine-tuning yield averaged quality reductions of R = 0.64% and R = 0.93%, respectively.The combined objective is used in the main experiments to preserve consistency with native rectified-flow training while maintaining competitive benchmark performance.
E Domain Drift and Memorisation loss
Strong compression can gradually shift generated images toward different stylistic domains and reduce memorisation of specific concepts, even when visual coherence remains.
- Under strong compression, generated outputs occasionally drift toward different stylistic domains while remaining visually coherent.The drift does not necessarily correspond to reduced visual quality.
- The drift appears gradually as compression increases, with anime or cartoon-like outputs often moving toward more realistic appearances.
- Strong compression can reduce preservation of parameters supporting instance-specific memorisation more than those encoding broadly useful image-synthesis capabilities.
F Additional Qualitative Examples
Qualitative examples across parameter budgets show gradual, visually subtle degradation under strong compression, with high-quality images and diverse compositions largely preserved. More aggressive compression mainly introduces minor fine-detail losses, style drift, and failures to reproduce iconic instances.
- Additional Qualitative Examples: 80%–39% parameter budgets still produce high-quality images with diverse compositions, while degradation remains gradual and visually subtle.The reported changes at stronger compression levels are mainly minor losses in fine details.
- Additional Qualitative Examples: Figures S5 and S6 provide additional qualitative examples across different parameter budgets.
- Domain Drift Examples: Aggressive compression can shift manga or cartoonish styles toward more realistic images.
- Memorisation Loss Examples: Aggressive compression can prevent reproduction of iconic instances such as the tower of Pisa or Bugs Bunny.