Source-linked AI summary
FreeU: Free Lunch in Diffusion U-Net
Chenyang Si, Ziqi Huang, Yuming Jiang, Ziwei Liu
TL;DR
Diffusion U-Net’s internal contributions are underexplored, particularly the differing roles of its backbone and skip connections in denoising. The paper introduces FreeU, which reweights these features during inference, and reports improved generation quality across image and video models without additional training or fine-tuning.
Problem
The internal properties of diffusion U-Nets remain largely underexplored, including how their backbone and skip connections contribute to denoising.
Method
FreeU strategically reweights backbone and skip-connection feature contributions during inference using two modulation factors.
Results
FreeU substantially improves generation quality across existing diffusion models for image and video generation without additional training or fine-tuning.
Takeaways & Limitations
FreeU can be integrated into existing diffusion models by adjusting two scaling factors during inference.
Takeaways & Limitations
Indiscriminately amplifying all backbone channels can oversmooth textures by compromising high-frequency details, so scaling is confined to half the channels.
Abstract
from arXiv · showhide
In this paper, we uncover the untapped potential of diffusion U-Net, which serves as a "free lunch" that substantially improves the generation quality on the fly. We initially investigate the key contributions of the U-Net architecture to the denoising process and identify that its main backbone primarily contributes to denoising, whereas its skip connections mainly introduce high-frequency features into the decoder module, causing the network to overlook the backbone semantics. Capitalizing on this discovery, we propose a simple yet effective method-termed "FreeU" - that enhances generation quality without additional training or finetuning. Our key insight is to strategically re-weight the contributions sourced from the U-Net's skip connections and backbone feature maps, to leverage the strengths of both components of the U-Net architecture. Promising results on image and video generation tasks demonstrate that our FreeU can be readily integrated to existing diffusion models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion, to improve the generation quality with only a few lines of code. All you need is to adjust two scaling factors during inference. Project page: https://chenyangsi.top/FreeU/.
1. Introduction
The paper examines underexplored internal properties of diffusion U-Nets, finding distinct denoising roles for backbone and skip features. FreeU reweights these contributions during inference to improve generation quality without additional training or fine-tuning.
- Motivation: Diffusion U-Net properties remain largely underexplored despite U-Nets being used to iteratively predict removable noise during denoising.The paper frames this internal analysis as a gap in existing diffusion-model research.
- Fourier-domain analysis: Low-frequency components change gradually during denoising, whereas high-frequency components vary more substantially across iterations.The analysis uses Fourier-domain views of progressive denoising to distinguish these dynamics.
- U-Net analysis: The U-Net backbone primarily contributes to denoising, while skip connections introduce high-frequency features into the decoder.Skip connections help recover fine-grained information but can weaken the backbone’s denoising capabilities during inference.
- FreeU: FreeU balances backbone and skip-connection contributions with two inference-time modulation factors.Backbone scaling strengthens denoising, while skip scaling addresses texture oversmoothing caused by backbone amplification.
- Evaluation: FreeU improves generated-output quality across existing image and video diffusion models without additional training or fine-tuning.Evaluated foundations include Stable Diffusion, DreamBooth, ReVersion, ModelScope, and Rerender.
2. Methodology
FreeU analyzes how diffusion U-Net components contribute to denoising and reweights backbone and skip features during inference. It amplifies backbone features, attenuates skip features, and uses structure-related and spectral modulation to improve generation quality without retraining.
- Diffusion U-Net denoising: A time-conditional U-Net denoising model reverses diffusion by predicting and removing noise from noisy data.The denoising process recovers clean data through learned inverse diffusion operations.
- U-Net component analysis: The U-Net backbone primarily contributes to denoising, whereas skip connections introduce high-frequency features into the decoder.Skip and backbone features are concatenated at each decoder stage.
- Scaling-factor analysis: Increasing backbone scaling factor b enhances image quality and suppresses generated high-frequency components, while skip scaling s has negligible influence on synthesis quality.The observed suppression links stronger backbone features to increased denoising capability.
- FreeU framework: FreeU applies two inference-time modulation factors: b amplifies backbone feature maps, while s attenuates skip feature maps before concatenation.The method improves generation quality without additional training or fine-tuning.
- Backbone feature scaling: Structure-related scaling adapts backbone amplification per sample using feature characteristics, aligning enhancement with structural information to mitigate oversmoothed textures.The backbone factor map is derived from the average feature map along the channel dimension.
- Texture preservation: Because indiscriminate backbone amplification compromises high-frequency details, FreeU confines scaling to half the backbone channels and selectively diminishes low-frequency skip components in the Fourier domain.The modified skip features are concatenated with modified backbone features for subsequent decoder layers.
3. Experiments
Experiments show that FreeU improves image and video generation across foundational and downstream diffusion models, while Fourier and feature-map analyses indicate stronger denoising and structural information.
- Experimental setup: FreeU integrates with Stable Diffusion, DreamBooth, ReVersion, ModelScope, and Rerender during inference without additional training or fine-tuning.The method adds backbone and skip feature factors while preserving the established models’ prescribed settings.
- Text-to-image: FreeU improves Stable Diffusion outputs in entity portrayal, fine-grained details, realistic poses, and relationships between entities.Examples include correcting rooftop irregularities, rabbit appearance, animal poses, and a cat riding a motorcycle.
- Analysis: Ablations examine backbone and skip scaling factors, while additional visualizations compare FreeU samples, Fourier amplitudes, and U-Net feature maps.The supplied figures cover SDXL, ModelScope, DreamBooth, ReVersion, Rerender, and the scaling-factor studies.
- Text-to-video: FreeU improves ModelScope videos through clearer motion, richer detail, and semantic alignment, including corrected viewpoint and object representations.Most participants preferred videos generated with FreeU in the text-to-video evaluation.
- Downstream tasks: FreeU enhances DreamBooth realism, ReVersion’s relational representations, and Rerender’s video detail and realism in downstream applications.Reported examples include correcting action-figure anatomy, relational artifacts, and sunglasses-related video artifacts.
- Analysis: FreeU reduces high-frequency information during denoising and produces feature maps with more pronounced structural information.These observations align with the method’s intended denoising effect while preserving intricate details.
4. Conclusion
FreeU improves diffusion-model sample quality without additional computational costs by re-weighting U-Net backbone and skip-connection features. The method leverages their distinct roles and integrates across diffusion foundation models and downstream tasks.
- FreeU substantially improves diffusion-model sample quality without incurring additional computational costs.
- FreeU strategically re-weights U-Net skip-connection and backbone feature maps to capitalize on their complementary strengths.The backbone primarily contributes to denoising, while skip connections predominantly introduce high-frequency features into the decoder.
- The method improves sample quality across a wide range of text prompts and random seeds.
- FreeU can be seamlessly integrated into various diffusion foundation models and their downstream tasks.