Mixture of Diffusers for scene composition and high resolution image generation

1 Introduction

• Text-conditioned diffusion often represents scene elements accurately but handles requested relationships and object positions vaguely.The introduction contrasts semantic accuracy with weak adherence to composition instructions.Source: §1 · p. 2
• Direct high-resolution diffusion is impractical because neural-network computation and GPU RAM requirements scale with image size.The passage gives a computational cost of O(N^2 log N^2) for an N × N image.Source: §1 · p. 2
• Mixture of Diffusers runs several diffusion processes simultaneously on one canvas and harmonizes their contributions to a shared image.The processes use the noise-predicting neural network.Source: §1 · p. 2
• Different processes can use different prompts in separate canvas regions to place objects or create smooth transitions between styles.The introduction presents these as benefits of the method.Source: §1 · p. 2
• Sharing one neural network keeps memory use essentially at the largest region's single-process requirement, supporting high-resolution generation on low-memory GPUs.The memory footprint is described as essentially the same as that required by the largest affected region.Source: §1 · p. 2
• Mixture of Diffusers is compatible with SDEdit, allowing image-and-text conditioning for image-to-image generation and outpainting.The method can condition a region on a guide image.Source: §1 · p. 2
• The method can use any pre-trained diffusion model, including latent-space models when pixel locations can be mapped approximately to latent locations.Source: §1 · p. 2

2.1 Diffusion models

• A forward Markov process gradually adds Gaussian noise; a learned reverse process uses Gaussian transitions to generate samples from noise.The reverse transition parameters depend on the current sample and step, and can be learned by optimizing an evidence lower bound.Source: §2.1 · pp. 3–4

2.2 Learning a diffusion model by predicting noise

• Noise-prediction training estimates the noise in a partially corrupted sample, while sampling repeatedly removes estimated noise; scheduler choice is separate from the noise model.The DDPM scheduler requires T iterations, while DDIM and LMS are cited as alternatives that still use noise predictions.Source: §2.2 · pp. 4–5

2.3 Diffusion models for image generation

• Image-generation systems progressed from ADM classifier guidance to text-conditioned GLIDE [5], cascaded DALL-E 2 [7] and Imagen [8], and latent diffusion.DALL-E 2 uses cascaded diffusion to produce and upscale images; latent diffusion operates on image representations compressed by an autoencoder.Source: §2.3 · pp. 5–6

2.4 Controlling composition in the generation process

• Composition-control methods range from guide-image img2img with a realism–faithfulness trade-off, through masked inpainting and outpainting and prompt-conditioned inpainting [5], to region-linked words in eDiff-I [32].GLIDE inpainting still uses a prompt describing the whole picture, whereas eDiff-I links selected prompt words to canvas regions.Source: §2.4 · p. 6

3.1 Method description

• Mixture of Diffusers combines noise predictions from models using different prompts on rectangular canvas regions into one global prediction.The regional predictions are mixed with per-pixel weights normalized across models.Source: §3.1 · p. 7
• Reusing one noise-prediction network lets model calls run sequentially, so multiple diffusion processes need not increase the model memory footprint.Calls can instead be batched when memory allows, for better GPU efficiency.Source: §3.1 · p. 7
• Overlapping regions create smooth prompt transitions, while constant weights can leave visible boundary artifacts.The method proposes Gaussian weights that taper toward region edges to favor smoother transitions.Source: §3.1 · pp. 7–8

3.2 Adaption to latent spaces

• For latent-space models, pixel-region coordinates are divided by the decoder upscaling factor to select corresponding latent regions; canvas indices must be multiples of that factor.The mapping assumes each latent location gives rise to a U × U pixel region.Source: §3.2 · p. 8
• The latent mapping is approximate: decoder self-attention can let latent values influence other pixels, but the experiments report that ignoring this does not seem to pose a practical problem.The approximation is motivated by the decoder’s local convolutions, especially for sufficiently large regions.Source: §3.2 · p. 8

4.1 Image composition

• Mixture of Diffusers locates specified objects in almost all samples, though horizons can become incoherent where a large object crosses model regions.Stable Diffusion often misses described objects and can produce samples that seem split into several images.Source: §4.1 · p. 10

4.2 High resolution image generation

• A 1408×896 image was generated from six overlapping regions, and adding a center region gives flexible rectangular placement for specific objects.The paper notes this image size was not feasible with one Stable Diffusion model on the hardware used.Source: §4.2 · p. 10
• 4K images (3840×2160) were generated on a 16GB GPU using 88 models arranged in an 8×11 grid.Each model covers a 640×480 region, with overlap between neighboring regions.Source: §4.2 · p. 10, Figure 8 · p. 13

4.3 Smooth style transitioning

• Using the same base prompt while varying only style descriptions across regions produces smooth transitions between styles.The example uses two overlapping regions per style to create a slower transition and clearer style expression.Source: §4.3 · p. 13

4.4 Conditioning on images

• A guide image can condition part of the canvas while a diffusion model guides the full image toward a prompt, producing outpainting with an overall consistent result.Figure 10 uses a guide image at the bottom of the canvas with 60% strength.Source: §4.4 · p. 14, Figure 10 · p. 15
• Image guidance supports iterative creation by fixing part of an earlier image while resampling noise or changing prompts for the rest.The Figure 11 example generates a postcard in three steps.Source: §4.4 · p. 14, Figure 11 · p. 15
• At 4K, a sketch guiding the full canvas preserves overall structure while region prompts add patterns.Figure 12 shows this with 25% guide strength.Source: §4.4 · p. 16, Figure 12 · p. 16

• The method combines several diffusion models on one canvas, offering finer composition control, high-resolution generation within a small GPU RAM budget, and image-guided generation.Source: §5 · p. 16
• Each diffusion model is restricted to a rectangular region, limiting the region shapes available for model placement.Source: §5 · p. 16
• The authors suggest masking, similar to inpainting techniques, as a possible future way to influence free-form regions.Source: §5 · p. 16

A Additional image composition results

• Figures 13 and 14 show random, unfiltered samples from Stable Diffusion 1.4 and Mixture of Diffusers.The paper reports that Mixture of Diffusers better represents the user's intended composition.Source: A Additional image composition results · p. 19

B Additional 4K results

• Figures 15, 16, and 17 present additional 4K image-generation samples.Source: B Additional 4K results · p. 19

C Additional outpainting results

• Mixture of Diffusers replicates the uncropping experiments in, extending a guide image in both horizontal directions.Figures 18 and 19 show two examples.Source: C Additional outpainting results · p. 19

D Mean of the diffusion process posterior for a noise-predicting model

• The derivation of the posterior mean is provided fully for clarity and completeness, although it is hinted at in.Source: D Mean of the diffusion process… · p. 19
• From Equation 10, the section expresses x0 using xt and ϵθ(xt), then substitutes that expression into Equation 4 to derive the next result.Source: D Mean of the diffusion process… · p. 19