Source-linked AI summary
Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, Devi Parikh
TL;DR
Pre-trained text-to-image models can generate broad visual concepts but often struggle with aesthetic quality, motivating post-training aesthetic alignment. The paper introduces quality-tuning, which fine-tunes broadly pretrained models on a few thousand exceptionally high-quality images; Emu improves visual appeal while retaining conceptual generality, and the recipe also works across other architectures.
Problem
Pre-trained text-to-image models generate broad visual concepts but do not always produce highly aesthetic images, creating a need for post-training aesthetic alignment.
Method
The paper pre-trains a latent diffusion model on 1.1 billion image-text pairs, then quality-tunes it with a few thousand hand-picked exceptionally high-quality images.
Results
Emu improves visual appeal without compromising generality across visual concepts, and quality-tuning also works for pixel diffusion and masked generative transformer models.
Takeaways & Limitations
A small, exceptionally high-quality fine-tuning set can substantially improve text-to-image aesthetics, with image quality prioritized over quantity.
Takeaways & Limitations
Human-evaluation results may not fully reflect real-world usage because aesthetic judgments are subjective and noisy, while pre-training deficiencies can persist for insufficiently learned objects.
Abstract
from arXiv · showhide
Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text. However, these pre-trained models often face challenges when it comes to generating highly aesthetic images. This creates the need for aesthetic alignment post pre-training. In this paper, we propose quality-tuning to effectively guide a pre-trained model to exclusively generate highly visually appealing images, while maintaining generality across visual concepts. Our key insight is that supervised fine-tuning with a set of surprisingly small but extremely visually appealing images can significantly improve the generation quality. We pre-train a latent diffusion model on $1.1$ billion image-text pairs and fine-tune it with only a few thousand carefully selected high-quality images. The resulting model, Emu, achieves a win rate of $82.9\%$ compared with its pre-trained only counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred $68.4\%$ and $71.3\%$ of the time on visual appeal on the standard PartiPrompts and our Open User Input benchmark based on the real-world usage of text-to-image models. In addition, we show that quality-tuning is a generic approach that is also effective for other architectures, including pixel diffusion and masked generative transformer models.
1. Introduction
The paper proposes quality-tuning: a second training stage that uses a very small set of exceptionally aesthetic images to align text-to-image outputs with high visual quality while preserving conceptual generality. The approach produces Emu and extends beyond latent diffusion to pixel diffusion and masked generative transformers.
- Training recipe: Quality-tuning adds a quality-learning stage after broad knowledge learning to restrict generated outputs to a high-quality, aesthetically pleasing domain.Knowledge learning typically uses hundreds of millions of image-text pairs, while quality learning focuses on aesthetic alignment.
- Training recipe: A couple of thousand exceptionally high-quality image-text pairs can substantially improve aesthetics without compromising the range of visual concepts the model generates.The authors caution that prioritizing quantity over quality may reduce generated-image quality.
- Emu: The method fine-tunes an LDM pretrained on 1.1 billion image-text pairs using a few thousand hand-picked, exceptionally high-quality images.Selection follows photography principles including composition, lighting, color, resolution, focus, and storytelling.
- Generality: Quality-tuning is presented as a generic approach that also works with pixel diffusion and masked generative transformer architectures.
- Emu: Emu is a quality-tuned latent diffusion model that significantly outperforms the publicly available state-of-the-art SDXLv1.0 on visual appeal.
2. Related Work
Text-to-image models use diffusion, autoregressive or masked transformers, and GANs, but they do not always produce highly aesthetic images. Prior fine-tuning methods target personalization, control, or editing, whereas this work focuses on generic aesthetic alignment.
- Text-to-image generation includes pixel- and latent-space diffusion, autoregressive and masked transformers, and generative adversarial networks.
- Existing text-to-image fine-tuning methods adapt models to subjects or styles, add conditioning controls, or enable instruction-based image editing.
- This work highlights fine-tuning as a way to promote aesthetic alignment generically across a wide range of visual domains.
- Like language-model fine-tuning, a relatively small but high-quality dataset can improve text-to-image output quality.
3. Approach
The approach separates knowledge learning from quality-tuning: a large-scale latent diffusion model is followed by supervised fine-tuning on a small, manually curated set of exceptionally aesthetic images. Architectural changes improve reconstruction quality, while early stopping preserves visual-concept generality.
- 3. Approach: The recipe uses knowledge learning followed by quality-tuning, with a couple thousand exceptionally high-quality images as the fine-tuning dataset.The authors emphasize that quality matters more than quantity and that fully automated curation is difficult.
- 3. Approach: The quality-tuning strategy also applies to pixel diffusion and masked generative transformer models.
- 3.1. Latent Diffusion Architecture: The latent diffusion model generates 1024 × 1024 images using an autoencoder for latent embeddings and a U-Net for denoising.
- 3.1. Latent Diffusion Architecture: Increasing the autoencoder latent channel size from 4 to 16 improves reconstruction quality by preserving more fine detail, especially in small objects.Fourier Feature Transform preprocessing and adversarial loss further improve reconstruction performance.
- 3.2. Pre-training: The model is pre-trained on 1.1 billion images with progressively increasing resolutions and a final-stage noise offset of 0.02.The progressive schedule targets high-level semantics at lower resolutions and finer details at the highest resolutions.
- 3.3. High-Quality Alignment Data: Curation combines standard automated filters, additional size and aspect-ratio checks, domain balancing, and manual aesthetic assessments.Human criteria include composition, lighting, color and contrast, subject-background arrangement, storytelling, and exceptional overall quality.
- 3.3. High-Quality Alignment Data: The filtering process retains 2000 exceptionally high-quality images, captions them, and upsamples below-target-resolution examples when necessary.
- 3.4. Quality-Tuning: Quality-tuning is intended to restrict outputs toward the high-quality subset while retaining the pre-trained model’s generality across visual concepts.Fine-tuning too long on the small dataset causes overfitting and degradation in visual-concept generality, motivating early stopping.
4. Experiments
The experiments evaluate Emu against its pre-trained counterpart and SDXLv1.0 using human judgments of visual appeal and text faithfulness across broad prompt sets. They also test whether quality-tuning transfers across architectures and benefits from very small high-quality datasets.
- Quality-Tuning vs Pre-training: 82.9% and 91.2% of the time, Emu was preferred over the pre-trained model for visual appeal on PartiPrompts and OUI Prompts, respectively.The corresponding text-faithfulness preferences were 36.7% and 47.9%.
- Quality-Tuning vs Pre-training: Emu’s improvements broadly apply to non-photorealistic styles, including sketches and cartoons, without observed degradation across visual concepts.The evaluation covered large prompt sets spanning various domains and categories.
- Quality-Tuning Other Architectures: Quality-tuning significantly improves visual appeal and text faithfulness for both pixel diffusion and masked generative transformer models.Both architectures were evaluated on one-third of randomly sampled PartiPrompts after quality-tuning on 2000 images.
- Ablation Study: A quality-tuned model using 100 high-quality images reaches a 60% win rate against SDXL, up from 24.8%.The dataset-size ablation reports results for randomly sampled subsets of 100, 1000, and 2000 images.
5. Limitation
The paper notes limits in human evaluation, small-scale fine-tuning, and text-to-image models generally, while showing examples and transfer beyond latent diffusion.
- Limitations of Human Evaluation: Human evaluations may not fully reflect real-world usage because aesthetic judgments are subjective and noisy.Different prompts, annotators, or guidelines may produce different results.
- Examples: Generated examples illustrate Emu outputs across everyday objects, people, animals, food, and scenes.
- Scope: Quality-tuning is reported as effective beyond latent diffusion, including pixel diffusion and masked generative transformer architectures.
- Limitations of Small-Scale Fine-Tuning: Quality-tuning does not resolve weaknesses inherited from pre-training, such as difficulty generating objects insufficiently learned beforehand.
- Limitations of Text-to-Image Models in General: Text-to-image models may generate biased, misleading, or offensive outputs despite fairness, safety, and red-teaming efforts.
6. Conclusion
The paper concludes that manually selecting highly aesthetic images is a key way to improve text-to-image visual appeal without sacrificing visual-concept generality. It builds Emu and reports that quality-tuning also benefits multiple model architectures.
- Conclusion: Manually selected, highly aesthetic images are identified as a key ingredient for improving generated image aesthetics.
- Conclusion: A few hundred to thousand fine-tuning images improved visual appeal without compromising the generality of depicted visual concepts.
- Conclusion: Emu is presented as a latent diffusion model for high-quality image synthesis and evaluated against SDXLv1.0 and the pre-trained latent diffusion model.
- Conclusion: Quality-tuning improved latent diffusion, pixel diffusion, and masked generative transformer models.