Source-linked AI summary
Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Models
Jinjin Zhang, Qiuyu Huang, Junjie Liu, Xiefan Guo, Di Huang
TL;DR
Direct 4K image synthesis lacks comprehensive datasets and fine-detail evaluation while remaining computationally challenging. Diffusion-4K introduces the Aesthetic-4K benchmark and wavelet-based fine-tuning for latent diffusion models, with experiments reporting effective photorealistic 4K generation and prompt adherence, especially with SD3 and Flux.
Problem
Direct training and assessment of 4K images remain limited, while conventional metrics inadequately evaluate fine details in ultra-high-resolution synthesis.
Method
Diffusion-4K combines the Aesthetic-4K dataset and benchmark metrics with wavelet-based fine-tuning that emphasizes high-frequency details while preserving low-frequency approximations.
Results
Diffusion-4K demonstrates effective photorealistic 4K image synthesis and superior performance, with advantages in fine-detail metrics and preference evaluations.
Takeaways & Limitations
The framework supports 4K training and generation across latent diffusion models, including SD3 and Flux, while enabling comprehensive assessment of image quality and detail.
Abstract
from arXiv · showhide
In this paper, we present Diffusion-4K, a novel framework for direct ultra-high-resolution image synthesis using text-to-image diffusion models. The core advancements include: (1) Aesthetic-4K Benchmark: addressing the absence of a publicly available 4K image synthesis dataset, we construct Aesthetic-4K, a comprehensive benchmark for ultra-high-resolution image generation. We curated a high-quality 4K dataset with carefully selected images and captions generated by GPT-4o. Additionally, we introduce GLCM Score and Compression Ratio metrics to evaluate fine details, combined with holistic measures such as FID, Aesthetics and CLIPScore for a comprehensive assessment of ultra-high-resolution images. (2) Wavelet-based Fine-tuning: we propose a wavelet-based fine-tuning approach for direct training with photorealistic 4K images, applicable to various latent diffusion models, demonstrating its effectiveness in synthesizing highly detailed 4K images. Consequently, Diffusion-4K achieves impressive performance in high-quality image synthesis and text prompt adherence, especially when powered by modern large-scale diffusion models (e.g., SD3-2B and Flux-12B). Extensive experimental results from our benchmark demonstrate the superiority of Diffusion-4K in ultra-high-resolution image synthesis.
1. Introduction
Diffusion-4K addresses the limited exploration of direct 4K synthesis by introducing a benchmark and wavelet-based fine-tuning framework for detailed, photorealistic generation.
- Most latent diffusion models train and generate at 1024×1024, while direct 4K synthesis remains computationally demanding and underexplored.Existing 4K approaches emphasize efficiency but neglect high-frequency details and rich textures.
- Aesthetic-4K provides a curated ultra-high-resolution dataset with GPT-4o-generated captions and metrics for both holistic quality and fine details.The benchmark combines FID, Aesthetics and CLIPScore with GLCM Score and Compression Ratio.
- Wavelet-based fine-tuning emphasizes high-frequency components while preserving low-frequency approximations for 4K image synthesis.The method is designed for direct training with photorealistic 4K images and is applicable to latent diffusion models.
- The method supports multiple latent diffusion models, including SD3-2B and Flux-12B, at 4096 × 4096 resolution.Experiments use open-sourced models capable of training and generating photorealistic images at 4096 × 4096.
- Extensive experiments report effective and generalized 4K synthesis, particularly with large-scale diffusion transformers such as SD3 and Flux.The reported effectiveness concerns ultra-high-resolution image synthesis using the proposed benchmark and method.
2. Related work
Related work develops latent diffusion and transformer-based text-to-image systems, but most models remain centered on 1024 × 1024 generation. Recent approaches pursue direct 4K synthesis mainly through efficiency improvements and specialized compression or fusion strategies.
- Latent diffusion architectures: Latent diffusion models perform diffusion in compressed VAE latent space, and DiT applies transformer architectures within this setting.The related-work passages identify transformer scalability through models including PixArt, SD3, Flux and Playground.
- Text-to-image conditioning: Text encoders support prompt coherence, with recent models such as SD3 and Flux combining CLIP and T5-XXL for text understanding.Earlier systems use CLIP or T5-XXL separately, while newer approaches combine both.
- Ultra-high-resolution synthesis: Advanced latent diffusion models typically train and synthesize at 1024 × 1024 because higher resolution introduces quadratic computational costs.Training-free fusion approaches have also been developed for 4K generation from existing latent diffusion models.
- Ultra-high-resolution synthesis: PixArt-Σ generates images close to 4K using efficient token compression, while Sana supports 1024 × 1024 through 4096 × 4096 synthesis with a deep-compression VAE.These approaches target more efficient direct ultra-high-resolution generation and training.
3. Methods
Diffusion-4K combines the Aesthetic-4K benchmark with wavelet-based fine-tuning for direct photorealistic 4K synthesis. The benchmark evaluates both holistic quality and fine details, while the method decomposes latent features into frequency components for training.
- Aesthetic-4K Benchmark: Aesthetic-4K provides a benchmark for direct ultra-high-resolution synthesis using curated images, GPT-4o captions, and qualitative and quantitative evaluation.Its evaluation combines holistic metrics with measures designed for fine details in 4K images.
- Aesthetic-4K Benchmark: The dataset contains 12,015 curated training images and 2,781 evaluation images, with an Aesthetic-Eval@4096 subset of 195 images.The training images have median dimensions of 4128×4640, while the evaluation images have median dimensions of 2983×3613.
- Aesthetic-4K Benchmark: GLCM Score and Compression Ratio complement FID, Aesthetics, and CLIPScore by assessing local texture richness and image detail.GLCM Score averages entropy-based measurements over 64-sized local patches, while Compression Ratio uses JPEG compression at quality 95.
- Wavelet-based Fine-tuning: Wavelet-based fine-tuning enables direct training with photorealistic 4096×4096 images across latent diffusion models.The approach uses a partitioned VAE with down-sampling factor F = 16 to reduce memory consumption and avoid out-of-memory issues.
- Wavelet-based Fine-tuning: The training objective applies a discrete wavelet transform to predicted and target quantities, jointly optimizing low-frequency structure and high-frequency details.Haar wavelets decompose features into LL, LH, HL, and HH sub-bands, and the reconstruction objective can be substituted to support different diffusion models.
4. Experiments
Experiments evaluate Diffusion-4K with partitioned VAEs, WLF fine-tuning, standard and fine-detail metrics, qualitative comparisons, preference studies, and ablations. Results support 4K reconstruction, generation, prompt adherence, aesthetics, and fine-detail quality.
- Experimental setup: SD3-2B and Flux-12B are evaluated with F = 16 partitioned VAEs, standard metrics, and proposed GLCM Score and Compression Ratio metrics.The evaluation covers reconstruction, image quality, prompt adherence, and fine details at high resolution.
- Partitioned VAE: The partitioned VAE resolves the original VAE's out-of-memory issue without retraining or fine-tuning, preserving latent-space consistency for subsequent 4K fine-tuning.Qualitative results demonstrate effective 4K image reconstruction.
- Qualitative synthesis: Diffusion-4K synthesizes qualitative 4K images at varied aspect ratios and supports direct photorealistic generation beyond the 4096 × 4096 fine-tuning resolution.The reported qualitative examples use Flux-12B.
- Preference study: WLF achieves a superior win rate in both human and GPT-4o preference evaluations for visual aesthetics, prompt adherence, and fine details.The preference study compares Flux outputs with and without WLF.
- Caption ablation: GPT-4o-generated captions improve Aesthetic-Eval@4096 results for both SD3 and Flux compared with original LAION-5B captions.The ablation links higher-quality captions with improved image synthesis and prompt coherence.
- Ablation studies: WLF outperforms fine-tuning without WLF on CLIPScore, Aesthetics, GLCM Score, and Compression Ratio.The ablation attributes gains across visual aesthetics, prompt adherence, and high-frequency details to WLF.
5. Conclusion
The conclusion presents Diffusion-4K as a framework combining the Aesthetic-4K benchmark with wavelet-based fine-tuning for ultra-high-resolution text-to-image generation. Qualitative and quantitative results support photorealistic 4K training and generation with models such as SD3 and Flux.
- Conclusion: Diffusion-4K combines the Aesthetic-4K benchmark and wavelet-based fine-tuning for ultra-high-resolution text-to-image generation.The framework is designed for training state-of-the-art latent diffusion models such as SD3 and Flux.
- Conclusion: Qualitative and quantitative results demonstrate effective training and generation of photorealistic 4K images.
PixArt-Sigma
The figure compares PixArt-Σ and Diffusion-4K using identical prompts, placing PixArt-Σ images on the left and Diffusion-4K images on the right. The highlighted patches emphasize fine-detail differences.
- Comparison layout: Diffusion-4K images appear on the right and PixArt-Σ images on the left under identical prompts.
- Fine-detail comparison: Yellow and red patches identify corresponding fine-detail regions in the comparison.The passage states that the comparison demonstrates Diffusion-4K's superiority in fine details.
7. Comparisons
Diffusion-4K is compared with direct ultra-high-resolution methods and shown qualitatively across diverse prompts, aspect ratios, seeds, and text-rendering cases. Wavelet-based latent fine-tuning preserves structure while enriching fine details.
- Performance Comparisons: Diffusion-4K consistently produces impressive fine details under identical prompts, highlighting superiority over PixArt-Σ in 4K generation.PixArt-Σ generates only 3840 × 2160 images, while the comparison uses images from PixArt-Σ and Sana's official websites.
- Qualitative Evaluation of WLF: Wavelet-based latent fine-tuning decomposes latents into high- and low-frequency components to refine details while preserving overall structure.The decomposition is presented as making fine-tuning efficient and precise.
- Qualitative Evaluation of WLF: Images generated with WLF exhibit richer details than images generated without WLF.
- More Results of 4K Image Synthesis: Generated 4K images demonstrate visual aesthetics, text-prompt adherence, and fine details across different text prompts.
- More Results of 4K Image Synthesis: Varying aspect ratios, random seeds, and spelled texts illustrate diversity in the synthesized 4K images.
8. Aesthetic-4K Dataset
Aesthetic-4K is presented as a high-quality, diverse 4K image-text dataset whose resolution substantially exceeds earlier open-source data. Its samples and statistics document both visual richness and textual coverage.
- Dataset Statistics: Aesthetic-Train has median height and width of 4128 and 4640 pixels, compared with 1615 and 1801 pixels for PixArt-30k.The LAION-Aesthetics dataset contains approximately 0.03% 4K images, underscoring the scarcity of such data in open-source datasets.
- Dataset Statistics: Aesthetic-4K provides histograms of image height and width to characterize its resolution distribution.
- Dataset Statistics: Caption word clouds visualize the textual distribution of the Aesthetic-4K dataset.
- Qualitative Samples: Training samples pair ultra-high-resolution images with precisely generated GPT-4o captions, illustrating dataset diversity and richness.
9. More Details
The paper details 4K preprocessing, latent fine-tuning, scalable-model training, and evaluation procedures. Additional figures and tables document captioning, dataset statistics, reconstruction, efficiency, and human assessment.
- Training Details: Images are resized so the shorter dimension is 4096, randomly cropped to 4096 × 4096, and normalized using mean and standard deviation 0.5.
- Training Details: The partitioned VAE compresses pixel space into a latent space with F = 16.
- Training Details: Latent diffusion models are optimized with the wavelet-based latent fine-tuning objective while text encoders and the partitioned VAE remain fixed.Text embeddings are pre-computed, and the default DiT patch size is P = 2 for SD3-2B and Flux-12B.
- Training Efficiency: Fine-tuning a 12B diffusion model requires approximately 2,000 A100 GPU hours.The training details cover SD3-2B and Flux-12B at 4096 × 4096 resolution.
- Reconstruction: Reconstruction results at 512 × 512 showcase the partitioned VAE's image-handling capability.
- Evaluation Details: FID, Aesthetics, and CLIPScore are computed using images resized to a fixed low resolution.
- Human Evaluation: Five participants rate extracted patches from 1 to 10 for visual details, with averages used to calculate SRCC and PLCC.Compression Ratio correlations use its reciprocal.
- Human Evaluation: Human preference evaluation uses 112 prompts and ten participants rating visual aesthetics, prompt adherence, and fine details.