DemoFusion: Democratising High-Resolution Image Generation With No $$$

1. Introduction

• High-resolution model training requires substantial hardware, data, and energy investment, and resulting models are increasingly placed behind paywalls.The paper gives Stable Diffusion 1.5 training at 512^2 as an example: over 20 days on 256 A100 GPUs.Source: §1 · p. 2
• DemoFusion extends open-source SDXL from 1024^2 generation to 4×, 16×, or greater resolution without additional training, using a consumer-grade RTX 3090 GPU.Directly prompting SDXL for higher-resolution images fails, while its patch inferences suggest prior knowledge of localized high-resolution content.Source: §1 · p. 2
• Figure 2 contrasts direct SDXL prompting, patch inference, MultiDiffusion, and DemoFusion; the caption reports global semantic coherence for DemoFusion.MultiDiffusion avoids edge effects but lacks global context for semantic coherence.Source: Figure 2 · p. 2, §1 · p. 2
• DemoFusion uses Progressive Upscaling, Skip Residual, and Dilated Sampling to improve global semantic coherence and local detail in high-resolution generation.Progressive Upscaling iterates through an upsample-diffuse-denoise loop; Skip Residual maintains consistency between low- and high-resolution images; Dilated Sampling adds denoising paths with broader perception.Source: §1 · p. 2
• High-resolution generation takes longer, partly because progressive upscaling requires more passes; intermediate low-resolution results can support prompt iteration before full generation.The highest-resolution pass dominates cost, while memory use remains low enough for consumer-grade GPUs.Source: §1 · pp. 2–3

2. Related Work

• Current LDMs are limited to 512^2 or 1024^2 generation; super-resolution did not recover desired detail from low-resolution outputs, and fine-tuning carries non-negligible cost.Cascaded Diffusion Models were capped at 256^2 images in the cited work.Source: §2 · p. 3
• MultiDiffusion and SyncDiffusion offer training-free generation, but can produce repetition and structural distortion for specific objects.MultiDiffusion fuses overlapping denoising paths, while SyncDiffusion constrains consistency between paths using gradient descent.Source: §2 · p. 3
• The paper reports that DemoFusion performs better than SCALECRAFTER, whose results somewhat degrade model performance and lack expected local detail.SCALECRAFTER adapts diffusion models for higher resolutions by dilating convolution kernels at specific layers.Source: §2 · p. 3

3.1. Preliminaries

• MultiDiffusion extends LDMs to high-resolution panoramas by independently denoising overlapping latent crops and averaging overlaps, but patches lack awareness of global context.Source: §3.1 · pp. 3–4

3.2. Progressive Upscaling

• Progressive Upscaling divides generation into phases that upsample the previous image, add noise, and denoise, compensating for interpolation while gradually adding local detail.Source: §3.2 · p. 4

3.3. Skip Residual

• To avoid choosing one example-specific intersection timestep, Skip Residual reuses noise-inversed prior-phase representations, guiding early global structure before fading for local refinement.Source: §3.3 · p. 5

3.4. Dilated Sampling

• Dilated Sampling creates global denoising paths alongside local paths; Gaussian prefiltering reduces graininess, with its effect fading to avoid blurring the final image.Source: §3.4 · p. 5

4.1. Comparison

• The comparison includes SDXL, MultiDiffusion, SDXL+BSRGAN, and SCALECRAFTER; quantitative evaluation uses FID, IS, CLIP Score, and 1×-patch FIDcrop and IScrop.The evaluation uses 1K randomly sampled LAION-5B captions, and inference time is measured on an RTX 3090 GPU.Source: §4.1 · pp. 5, 7
• DemoFusion ranks first or second across all metrics and significantly outperforms SDXL+BSRGAN on crop-based FIDcrop and IScrop.Table 1 reports the quantitative comparison; the prose notes that DemoFusion may score slightly lower than SDXL+BSRGAN on FID and IS as resolution increases.Source: Table 1 · p. 7, §4.1 · p. 7
• At 4096², DemoFusion produces richer local details than SDXL+BSRGAN while maintaining global semantic coherence, unlike the compared baselines’ limitations.MultiDiffusion tends toward repetitive content; SCALECRAFTER can produce incoherent objects and repetitive local patterns, while super-resolution mainly sharpens and smooths input images.Source: §4.1 · p. 7

4.2. Ablation Study

• The ablation tests all component combinations at 3072²; removing all three yields repetitive content, while adding them progressively improves consistency with original-resolution images.Skip Residual helps preserve overall structure, Dilated Sampling adds globally perceptive denoising paths, and Progressive Upscaling helps address artificial noise from large-scale upscaling.Source: Figure 5 · p. 7, §4.2 · p. 8

5. Limitations and Opportunities

• Overlapped denoising paths and progressive upscaling increase computational load and inference time.The authors note that progressive upscaling requires more passes.Source: §5 · p. 8
• As a tuning-free framework, DemoFusion’s performance correlates directly with the underlying LDM; with SD1.5 and SD2.1 it remains effective, but results are less astonishing than SDXL.Local irrational content may appear in sharp close-up images, and small repetitive content may remain in background regions.Source: §5 · p. 8
• Mini-batching original-size denoising paths prevents the expected exponential memory increase, and low-resolution previews can support rapid prompt iteration.The authors also identify training a bespoke LDM for a DemoFusion-like framework as a promising direction to explore.Source: §5 · p. 8

6. Conclusion

• DemoFusion combines Progressive Upscaling, Skip Residual, and Dilated Sampling to pursue global semantic coherence and rich local detail in higher-resolution generation.The conclusion describes the framework as tuning-free and plug-and-play with open-source GenAI models.Source: §6 · p. 8
• The paper presents DemoFusion as evidence that existing LDMs can generate images above their training resolutions, without additional training or prohibitive inference memory requirements.The authors state that this may help democratize high-resolution image generation.Source: §6 · p. 9

Appendix

• Figure 8 illustrates diffusing the upsampled latent to different time-steps during progressive upscaling, with 1,000 training time-steps.Source: Figure 8 · p. 11
• Algorithm 1 illustrates DemoFusion's image-synthesis process, while Table 2 presents an ablation study of Progressive Upscaling, Skip Residual, and Dilated Upsampling.Source: A. Pseudo Code · p. 11, Table 2 · p. 11

• Unless otherwise stated, experiments use SDXL with a 50-step DDIM scheduler and guidance scale 7.5; crop size is 128 × 128 for SDXL.Source: B. Implementation Details · p. 11
• Crop stride is half the crop size, and each crop position receives random vertical and horizontal perturbations capped at one-sixteenth of its respective crop dimension.Source: B. Implementation Details · p. 11
• The implementation sets scale factors α1, α2, and α3 to 3, 1, and 1, respectively, and decreases the Gaussian filter standard deviation from 1 to 0.01.High-resolution decoding uses tiled decoding, with a larger feature range sampled around each tile to eliminate seams.Source: B. Implementation Details · p. 11
• SDXL's coarse-cropping condition supplies patch coordinates, but the authors observed no significant effect from its presence or absence.For varying aspect ratios, the longer side is aligned with the highest resolution.Source: B. Implementation Details · p. 11

C.1. Diffusing to Different Time-step t

• Larger denoising time-steps lose more information and weaken global perception, whereas smaller ones introduce stronger noise from upsampling.Source: §C.1 · p. 11
• Algorithm 1 begins with random-noise initialization and denoising, then repeats upsampling and diffusion in later phases.Source: §C.1 · p. 12
• The algorithm includes skip residuals, crop sampling, local and global denoising paths, and fusion of the local and global paths.Source: §C.1 · p. 12
• Figure 10 presents results at different strides for images generated at 9× resolution; Figure 11 compares SDXL and DemoFusion inference time and memory demands.Source: Figure 10 · p. 12, Figure 11 · p. 12

C.2. Quantitative Results of Ablation Study

• The ablation study reports quantitative results at a single resolution: 4096².Source: §C.2 · p. 12

C.3. Effects of Scale Factors α1, α2, and α3

• Cosine-descending weights emphasize skip residuals, dilated sampling, and Gaussian filtering early in denoising; separate scale factors control the decline rate for each component.Source: §C.3 · pp. 12–13
• The parameter effects interact: α1 = 1 produces artificial upsampling noise, while α2 = 1 can make images grainy and α2 = 5 can weaken global perception and allow content repetition.The authors report that excessive Gaussian filtering over-smooths latent representations, while too little can yield repetition and graininess; they use grid search to find an optimal parameter combination.Source: §C.3 · p. 13

C.4. Effect of Stride Sizes dh and dw

• At dh = h and dw = w, DemoFusion retains good global coherence; half-size strides were chosen to balance performance and efficiency.With smaller strides, the authors report no noticeable seams.Source: §C.4 · p. 13

C.5. Resource Demands of DemoFusion

• DemoFusion achieves high-resolution generation on limited computational resources, with a little more time cost than an assumed high-resolution SDXL under the current framework.The paper notes that SDXL itself cannot generate valid content above 1024²; the resource comparison assumes a hypothetical high-resolution SDXL.Source: §C.5 · pp. 13–14

D.1. The Progressive Upscaling Process

• Outputs across progressive phases show global consistency at different resolutions, which the authors identify as addressing content repetition.Figure 12 displays outputs at each generation phase.Source: §D.1 · p. 14

D.2. More Landscape Samples

• 4096² samples provide additional qualitative comparison results in Fig. 14.The caption notes that local details have been zoomed in.Source: Figure 14 · p. 15
• 3072² examples show DemoFusion combined with ControlNet in Fig. 15.The caption recommends viewing the images zoomed in.Source: Figure 15 · p. 16
• Real images are upscaled to 3072² in Fig. 16.The caption recommends viewing the images zoomed in.Source: Figure 16 · p. 16
• 8192 × 4096 samples represent 64× upscaling from the initial 512 × 1024 resolution.The samples are presented as additional performance examples in Fig. 13.Source: §D.2 · pp. 14, 16

D.3. More Comparison Results

• DemoFusion provides better local details than SDXL+BSRGAN [39].The authors say these results are consistent with those in the main text.Source: §D.3 · p. 16
• DemoFusion better preserves SDXL performance during upscaling than SCALECRAFTER [7].The authors say these results are consistent with those in the main text.Source: §D.3 · p. 16

E.1. Combining with ControlNet

• Combining DemoFusion with ControlNet [40] enables controllable high-resolution generation using Canny-edge or human-pose conditions.Examples are shown at 3072² resolution.Source: §E.1 · p. 16

E.2. Upscaling Real Images

• For real-image upscaling, DemoFusion replaces phase 1's output with representations encoded from real images.The authors avoid calling this super-resolution because outputs tend toward the base LDM's latent data distribution, making the process more like image generation based on a real image.Source: §E.2 · p. 16

F. Prompts Used in This Paper

• The paper's prompts are drawn from the internet or generated by ChatGPT, and are listed by figure.The supplied prompt lists cover figures in the main text and appendix.Source: F. Prompts Used in This Paper · pp. 16–18