PaperlayerView exact revision on arXiv

Source-linked AI summary

RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths

Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, Ping Luo

arXiv:2305.18295v5cs.CV

TL;DR

RAPHAEL targets imprecise text–image alignment in diffusion generators, where conventional cross-attention can inadequately preserve textual concepts. It stacks space- and time-MoE layers with edge-supervised learning, producing stronger reported image quality and alignment than prior models, including a zero-shot FID-30k of 6.61 on COCO. The paper also identifies misleading or false information as a potential negative societal impact before API release.

  • Problem

    Existing text-to-image diffusion models require more precise text–image alignment because classic cross-attention often fails to preserve textual concepts adequately.

  • Method

    RAPHAEL stacks space-MoE and time-MoE layers within an edge-supervised text-conditional diffusion model to route concepts to image regions and diffusion timesteps.

  • Results

    RAPHAEL outperforms Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2 in reported image quality, aesthetics, style switching, FID-30k, and human evaluation, achieving 6.61 zero-shot FID-30k on COCO.

  • Takeaways & Limitations

    RAPHAEL supports artistic generation across diverse styles and extensions using LoRA, ControlNet, and SR-GAN.

  • Takeaways & Limitations

    Using the RAPHAEL API to create misleading or false information is identified as a potential negative societal impact, with prompt filtering planned before public release.

Abstract

from arXiv · show

Text-to-image generation has recently witnessed remarkable achievements. We introduce a text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by stacking tens of mixture-of-experts (MoEs) layers, i.e., space-MoE and time-MoE layers, enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting a particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. Secondly, a single model with three billion parameters, trained on 1,000 A100 GPUs for two months, achieves a state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers of image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https://raphael-painter.github.io/.

1 Introduction

RAPHAEL addresses coarse text–image alignment in existing diffusion models by routing concepts through image regions and diffusion timesteps with stacked MoE layers. Experiments report stronger artistry, fidelity, style switching, benchmark performance, and extensibility than prior approaches.

  • Motivation: Existing text-to-image diffusion models need more precise text–image alignment because classic cross-attention often preserves textual concepts inadequately.The coarse integration of text descriptions into visual representations compromises generated results.
  • Method: RAPHAEL stacks space-MoE and time-MoE layers so different concepts are depicted in specific image regions at different diffusion timesteps.This configuration produces billions of diffusion paths, with each path acting as a painter for a concept, region, and timestep.
  • Method: Edge-supervised learning further enhances the generated images’ quality and aesthetic appeal.
  • Results: RAPHAEL achieves a zero-shot FID-30k score of 6.61 on COCO and supports extensions with LoRA, ControlNet, and SR-GAN.The reported model uses three billion parameters and was trained on 1,000 A100 GPUs.

2 Notation and Preliminary

This section introduces generative diffusion modeling, text conditioning through U-Net cross-attention, and RAPHAEL’s framework for assigning text concepts to image regions. The framework combines standard attention components with space- and time-specific expert processing.

  • DDPM: A generative model p(x) is trained to represent the underlying distribution of a collection of N images.
  • DDPM: Forward diffusion progressively adds Gaussian noise to source images according to a T-step noise schedule.At timestep t, the noisy image x_t can be sampled directly from the Gaussian distribution.
  • DDPM: The reverse process uses a denoising neural network to estimate additive Gaussian noise and progressively recover image structure.DDPM training and sampling are described as stable through score estimation across timesteps.
  • RAPHAEL Framework: RAPHAEL’s examples depict rich details and semantics across human-written prompts and diverse artistic styles.
  • RAPHAEL Framework: Each RAPHAEL transformer block combines self-attention, cross-attention, space-MoE, and time-MoE components.Space-MoE targets concepts in image regions, while time-MoE handles diffusion timesteps.
  • U-Net with Text Prompts: A U-Net incorporates text prompts through an encoder and cross-attention over image and text tokens.The cross-attention matrices Q, K, and V are computed from intermediate image representations and encoded text.

3 Our Approach

RAPHAEL combines space- and time-MoE layers to route text concepts to image regions and diffusion timesteps, creating billions of concept-specific diffusion paths. It further uses learned routing, attention-derived spatial masks, and edge supervision to improve concept representation and image detail.

  • 3.1 Space-MoE and Time-MoE: RAPHAEL uses 16 transformer blocks, each combining self-attention, cross-attention, space-MoE, and time-MoE layers.Space-MoE handles text concepts at image regions, while time-MoE handles different diffusion timesteps.
  • 3.1 Space-MoE and Time-MoE: Space-MoE routes each text token and its binary image-region mask to an expert, then averages the expert outputs.The mask indicates the region corresponding to each token, and the output combines the routed experts.
  • 3.1 Space-MoE and Time-MoE: A learned gate assigns each text token exclusively to one space expert, with random noise added to prevent mode collapse.The argmax routing preserves one-expert assignment without increasing computational complexity.
  • 3.1 Space-MoE and Time-MoE: 616 spatial routes arise from 16 space-MoE layers with 6 experts each, and different paths are associated with distinct textual concepts.A classifier achieves over 93% accuracy on open-world adjectives, while COCO object categories show analogous path specialization.
  • 3.1 Space-MoE and Time-MoE: Time-MoE layers automatically learn timestep assignments through an additional gate network rather than relying on hand-crafted assignments.They are placed before each space-MoE layer to specialize processing across the diffusion trajectory.
  • 3.2 Edge-supervised Learning: Edge-supervised learning predicts image edges from attention maps and combines focal edge loss with denoising loss to preserve detailed features.Edge supervision is paused for large timesteps because attention maps become vague, and the combined objective is L = Ldenoise + Ledge.

4 Experiments

RAPHAEL is evaluated against recent text-to-image models on image quality, text-image alignment, style switching, ablations, and runtime. Results report strong COCO and human-evaluation performance, while ablations expose module contributions and computational trade-offs.

  • Comparisons: RAPHAEL receives higher user preference rates for image quality and prompt conformity than DALL-E 2, SD XL, ERNIE-ViLG 2.0, and DeepFloyd on ViLG-300.The study reports preference rates with 95% confidence intervals.
  • Comparisons: RAPHAEL achieves 6.61 zero-shot FID-30k on MS-COCO 256 × 256, surpassing Stable Diffusion, Imagen, ERNIE-ViLG 2.0, and DALL-E 2.FID measures image quality and diversity using 30,000 validation images.
  • Ablation Study: All evaluated RAPHAEL modules contribute effectively; removing space-MoE lowers CLIP performance and increases FID at the same guidance weight.Space-MoE shifts the optimal guidance weight from 3.0 to 4.5.
  • Ablation Study: The ablation selects α = 0.2 and Tc = 500, with Tc terminating edge-supervised learning at large diffusion timesteps.The reported choice of Tc is intended to support effective texture learning.
  • Ablation Study: Increasing the number of experts improves FID but adds computation through space experts; six experts reduce inference speed by 24% while improving fidelity.Complexity stops growing after all available experts are deployed and remains faster than Imagen and eDiff-I in the reported setting.

5 Related Work

Related work spans mixture-of-experts methods and text-to-image generation, covering expert partitioning across modalities and successive GAN, autoregressive, and diffusion-based approaches.

  • Mixture-of-Experts: Mixture-of-Experts partitions model parameters into distinct subsets called experts and has been applied to language, visual, and multimodal models.Related efforts also target faster MoE training or inference.
  • Text-to-Image Generation: Text-to-image generation has progressed from GANs to autoregressive transformers that model text and image tokens as sequences.Recent work also integrates textual conditioning into diffusion models.

6 Conclusion

RAPHAEL is a text-conditional diffusion model using space-MoE and time-MoE within edge-supervised learning to align textual concepts with image regions and produce artistic images. Experiments report stronger FID-30k and ViLG-300 performance than prior approaches, while the authors note a risk of misleading or false generated images.

  • Conclusion: RAPHAEL combines space-MoE, time-MoE, and edge-supervised learning to improve alignment between textual concepts and image regions.The model uses a large-scale mixture of diffusion paths to generate highly artistic images.
  • Conclusion: RAPHAEL surpasses Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2 on FID-30k and the ViLG-300 human-evaluation benchmark.The model can also be extended using LoRA, ControlNet, and SR-GAN.
  • Limitations and Potential Negative Societal Impact: The RAPHAEL API could be used to create misleading or false information, so the authors plan prompt filtering before public API release.

7 Appendix

The appendix details RAPHAEL’s U-Net architecture, its space- and time-MoE components, routing mechanisms, and timestep-specialized expert behavior.

  • Configuration: The appendix provides RAPHAEL’s hyper-parameters and values in Table 2.The table is identified as the source for the model’s hyper-parameter settings.
  • Time-MoE routing: The Time Gate Network routes features to a time expert according to the diffusion timestep.The router selects an expert from a set of time experts using the timestep representation.
  • Architecture: RAPHAEL’s U-Net contains transformer blocks with self-attention, cross-attention, space-MoE, and time-MoE components.The space-MoE depicts text concepts in image regions, while the time-MoE handles diffusion timesteps.
  • Time-MoE routing: Random noise is added to the time-routing logits to prevent mode collapse.The routing function applies argmax over a softmax of the timestep embedding’s feed-forward output plus noise.
  • Time-MoE analysis: In the first transformer block, one time expert processes the noisiest images across the initial 59% of DDIM steps, while other experts handle the final 41%.This division indicates specialization by noise level across diffusion timesteps.

7.4 Related Work

The related-work appendix situates RAPHAEL within mixture-of-experts research and the evolution of text-to-image generation methods.

  • Scope: The paper reviews prior work from two perspectives: mixture-of-experts and text-to-image generation.These perspectives frame RAPHAEL’s architectural and task-specific context.
  • Mixture-of-Experts: Mixture-of-Experts models partition parameters into experts and dynamically route each input to selected experts during forward propagation.Routing may activate all experts or a sparse subset, depending on the model design.
  • Mixture-of-Experts: MoE methods have been applied beyond language processing, including visual models and multimodal transformers.Related research also targets faster MoE training and inference.
  • Text-to-Image Generation: Text-to-image generation synthesizes images from natural-language descriptions and has progressed from GAN-based methods toward transformer-based generative models.The review names DALL-E, CogView, and Make-A-Scene as sequence-to-sequence approaches.

7.5 More Details on Routers of Space-MoE

The appendix examines how RAPHAEL’s space-MoE routes concepts and reports strong route-category associations across COCO categories and verbs.

  • Route analysis: A classifier achieves 94.3% accuracy for COCO categories and 97.5% for randomly generated verbs based on diffusion routes.The investigation uses 100 samples for each category and verb and XGBoost as the classifier.
  • Route analysis: Diffusion routes are closely associated with COCO categories and 50 verbs when using 16 space-MoE layers with six experts each.The visualization presents routes from network input to output.
  • Visualizations: Figures 10 and 11 visualize diffusion routes, generated images, and attention maps.These figures connect route behavior with prompt-associated outputs and attention patterns.
  • Prompt concepts: The appendix lists adjectives and verbs used in the route and prompt analyses.The adjective list includes stylistic descriptors, while the verb list includes actions such as dance, paint, and swim.
  • Comparisons: The appendix compares RAPHAEL with DALL-E 2, Midjourney v5.1, Stable Diffusion XL, ERNIE ViLG 2.0, and DeepFloyd using the same prompts.Highlighted prompt words identify concepts that human artists want preserved in generated images.

7.7 More Images Generated by RAPHAEL

The appendix provides additional examples of RAPHAEL-generated artistic images across varying text prompts and visual styles.

  • Additional examples: Figures 14–18 show more RAPHAEL-generated cases across varying text prompts and styles.The examples extend the image gallery beyond the main results.

7.8 Extension to LoRA, ControlNet, and SR-GAN

The paper extends RAPHAEL to LoRA, ControlNet, and SR-GAN, with detailed experimental settings provided in the corresponding figure captions.

  • LoRA: LoRA experiments compare RAPHAEL and Stable Diffusion after fine-tuning on 28 images for spider-man prompts across multiple settings.RAPHAEL preserves the prompt concepts, whereas Stable Diffusion produces compromised results.
  • LoRA: LoRA experiments fine-tune RAPHAEL and Stable Diffusion on 32 images using prompts that vary a boy’s accompanying concepts.Only RAPHAEL preserves the requested concepts across the prompt variants.
  • ControlNet: ControlNet uses canny edges extracted from a reference image as an additional constraint for RAPHAEL on ox and game-icon prompts.The evaluated prompt groups vary color, object attributes, and watercolor or monochrome styles.
  • SR-GAN: SR-GAN enhances the resolution of RAPHAEL-generated images to 4096×6144 pixels.This result is shown in two experiments, with detailed settings provided in the figure captions.
Loading 2305.18295v5…