Source-linked AI summary
DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution
Zheng-Peng Duan, Jiawei Zhang, Xin Jin, Ziheng Zhang, Zheng Xiong, Dongqing Zou, Jimmy S. Ren, Chun-Le Guo, Chongyi Li
TL;DR
Real-ISR requires realistic high-resolution recovery from degraded low-resolution images, but adapting powerful DiT diffusion models to this task remains non-trivial. DiT4SR integrates LR information into DiT attention and uses cross-stream convolution for local guidance, achieving superior Real-ISR performance in extensive experiments.
Problem
Real-ISR is an ill-posed restoration problem, and the paper asks whether advanced DiT-based diffusion models can provide suitable generative priors for it.
Method
DiT4SR integrates an LR Stream into DiT attention for bidirectional interaction and injects LR guidance into the Noise Stream through cross-stream convolution.
Results
DiT4SR achieves superior performance in Real-ISR, with extensive experiments reporting competitive or top performance across evaluated datasets and non-reference metrics.
Takeaways & Limitations
The designs demonstrate the potential of adapting large-scale DiT models for high-quality image restoration.
Takeaways & Limitations
Global joint attention alone is insufficient for Real-ISR because it may overlook local information needed to restore fine details and preserve image fidelity.
Abstract
from arXiv · showhide
Large-scale pre-trained diffusion models are becoming increasingly popular in solving the Real-World Image Super-Resolution (Real-ISR) problem because of their rich generative priors. The recent development of diffusion transformer (DiT) has witnessed overwhelming performance over the traditional UNet-based architecture in image generation, which also raises the question: Can we adopt the advanced DiT-based diffusion model for Real-ISR? To this end, we propose our DiT4SR, one of the pioneering works to tame the large-scale DiT model for Real-ISR. Instead of directly injecting embeddings extracted from low-resolution (LR) images like ControlNet, we integrate the LR embeddings into the original attention mechanism of DiT, allowing for the bidirectional flow of information between the LR latent and the generated latent. The sufficient interaction of these two streams allows the LR stream to evolve with the diffusion process, producing progressively refined guidance that better aligns with the generated latent at each diffusion step. Additionally, the LR guidance is injected into the generated latent via a cross-stream convolution layer, compensating for DiT's limited ability to capture local information. These simple but effective designs endow the DiT model with superior performance in Real-ISR, which is demonstrated by extensive experiments. Project Page: https://adam-duan.github.io/projects/dit4sr/.
1. Introduction
Real-ISR must recover realistic high-resolution details from degraded low-resolution images, motivating the use of powerful DiT priors. DiT4SR adapts DiT by integrating LR information bidirectionally and adding local feature injection.
- Motivation: Real-ISR recovers high-resolution images from degraded low-resolution inputs while requiring both degradation removal and perceptually realistic detail generation.The task is ill-posed, so prior knowledge helps avoid ambiguous results and degraded restoration quality.
- Motivation: DiT-based diffusion models motivate a new Real-ISR design because their attention-based architecture supports bidirectional interaction between visual features and text tokens.The paper specifically asks whether advanced DiT-based diffusion models can be adopted for Real-ISR.
- Existing approach: SD3-ControlNet duplicates MM-DiT blocks and directly adds LR-derived hidden states to the Noise Stream, restricting interactions between LR and generated information.This ControlNet-style approach was inherited from UNet-based architectures and may not exploit DiT-specific characteristics.
- DiT4SR: DiT4SR integrates an LR Stream into the original DiT blocks, enabling bidirectional interaction between LR guidance and the diffusion process.Duplicated Noise Stream modules process the LR input inside each MM-DiT block, allowing both streams to evolve together.
- DiT4SR: A cross-stream convolution layer injects LR guidance into the Noise Stream, compensating for DiT’s limited ability to capture local information.The contribution list identifies this convolution as the mechanism for restoring local details that attention may miss.
2. Related Work
Real-ISR methods face complex degradations and an ill-posed inverse problem, while large-scale text-to-image models provide generative priors for restoration. DiT4SR builds on DiT models to enable fuller LR-feature interaction.
- Image Super-Resolution: Convolutional and transformer ISR methods struggle with Real-ISR because real-world degradations are complex and the inverse problem is ill-posed.Prior work addressed degradation modeling with BSRGAN and Real-ESRGAN, while other methods explored generative approaches.
- Diffusion-based Restoration: Large-scale text-to-image models are used for Real-ISR because their pretrained generative priors can guide restoration from low-resolution inputs.StableSR and DiffBIR build on UNet-based Stable Diffusion and inject LR information through ControlNet-like conditioning.
- Diffusion Transformers: MM-DiTs integrate text and image modalities through attention, allowing the two streams to fully interact within DiT-based generative models.PixArt-α, SD3, and Flux are cited as large-scale T2I models built on DiT architectures.
- Diffusion Transformers: DiT4SR extends this DiT advantage to Real-ISR by incorporating an LR stream into DiT blocks for interaction with the original features.The method is built on the DiT-architectured SD3 model.
3. Methodology
DiT4SR adapts DiT for Real-ISR by integrating an LR Stream into the original MM-DiT blocks, enabling bidirectional guidance and adding convolutional local-detail injection. LR residual connections preserve guidance across depth while the convolution compensates for DiT’s limited local information capture.
- Architecture: Unlike SD3-ControlNet, DiT4SR integrates LR information into the original DiT attention rather than processing it in duplicated blocks.This design addresses the one-way information flow caused by directly injecting LR embeddings into the Noise Stream.
- Architecture: DiT4SR adds an LR Stream to the original MM-DiT block, allowing LR, noise, and text representations to interact during diffusion.The LR image is encoded into latent space and represented as LR image tokens alongside noisy image and text tokens.
- LR Integration in Attention: Joint attention concatenates projections from the noise, LR, and text streams, with newly created LR projections initialized to zero.Zero initialization lets LR influence grow progressively during training while initially preserving the pretrained model’s behavior.
- LR Integration in Attention: Bidirectional attention lets the Noise Stream receive LR guidance while the LR Stream adapts to the Noise Stream, producing more context-aware guidance.Attention visualizations show interactions between corresponding noisy-image and LR-image token positions.
- LR Integration in Attention: An LR Residual preserves LR guidance across deeper attention blocks, where interactions between LR and noise tokens otherwise decay.The shortcut directly connects the LR attention input and output to improve guidance consistency.
- LR Injection between MLP: A zero-initialized 3×3 depth-wise convolution injects intermediate LR-stream features into the Noise Stream between the MLPs.The convolution adds local information that global joint attention and linear injection may miss, improving fine-structure recovery.
4. Experiments
DiT4SR is evaluated on four real-world datasets against GAN-, UNet-diffusion-, and DiT-based methods using quantitative, qualitative, and user-study comparisons. It achieves competitive results on DrealSR and RealSR, strong performance on RealLR200 and RealLQ250, and favorable realism and fidelity judgments.
- Datasets and Metrics: Experiments cover DrealSR, RealSR, RealLR200, and RealLQ250, all evaluated at ×4 scaling.RealLR200 and RealLQ250 lack corresponding ground-truth images, motivating non-reference evaluation.
- Comparison Methods: DiT4SR is compared with GAN-based, UNet-based diffusion, and DiT-based Real-ISR methods.The DiT-based comparison includes DreamClear and SD3-ControlNet.
- Quantitative Results: DiT4SR closely matches or exceeds strong competitors on DrealSR and RealSR, while achieving top performance across all non-reference metrics on RealLR200 and RealLQ250.The authors describe the latter results as overwhelming performance and high-quality restoration.
- Qualitative Comparisons: Qualitative comparisons show better clarity and more abundant details under severe blur, with advantages in fine structures and fidelity.The reported fine-structure advantage includes architectural structures and distinguishes DiT4SR from SD3-ControlNet.
- User Study: A user study with 80 volunteers compares DiT4SR against SeeSR, DiffBIR, SUPIR, and DreamClear on 60 images using realism and fidelity judgments.Participants viewed the LR input, DiT4SR output, and a randomly selected competing result; the reported results favor DiT4SR.
5. Ablation Study
The ablation study evaluates LR Integration, LR Residual, and LR Injection, showing that the components support guidance adjustment, fidelity, and local-detail restoration. Removing them produces metric declines, artifacts, or content distortions.
- Experimental Setup: The ablation study evaluates variants on RealLQ250 with MUSIQ and MANIQA under identical training settings.Variants remove LR Integration, LR Residual, or LR Injection, while another replaces convolution with a linear layer.
- LR Integration: Removing LR Integration prevents LR information from progressively adjusting to provide more accurate guidance for complex degradations.The study examines this component by excluding the LR Stream from attention computation while retaining other components.
- LR Residual: Removing LR Residual lowers MUSIQ and MANIQA and introduces artifacts by destabilizing LR Stream evolution, reducing content fidelity.The residual directly preserves consistency of LR guidance in deeper DiT blocks.
- LR Injection: Removing LR Injection causes a slight metric decline and noticeable content distortions because attention alone lacks sufficient local guidance.The reported distortion is especially visible in the eye region.
6. Conclusion
DiT4SR adapts large-scale DiT models to Real-ISR by integrating LR embeddings into DiT attention and injecting LR guidance through convolution. The resulting bidirectional interaction and local guidance support superior Real-ISR performance.
- Contribution: DiT4SR is presented as one of the pioneering efforts to adapt a large-scale DiT model for Real-ISR.The conclusion frames the work as demonstrating DiT's potential for high-quality image restoration.
- Architecture: Unlike ControlNet, DiT4SR integrates LR embeddings into DiT's original attention mechanism, enabling bidirectional information flow between LR and generated latents.This design allows LR guidance to interact directly with the diffusion process.
- Local Guidance: A cross-stream convolution layer injects LR-guided information into the generated latent and compensates for DiT's weak local feature-capturing ability.The conclusion links this modification to the method's Real-ISR performance.
- Conclusion: Extensive experiments demonstrate superior DiT4SR performance in Real-ISR and motivate further use of DiT for high-quality image restoration.The stated conclusion remains within the demonstrated Real-ISR scope.