PaperlayerView exact revision on arXiv

Source-linked AI summary

ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models

Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, Ying Shan

arXiv:2310.07702v1cs.CV

TL;DR

Generating beyond training resolution can preserve image definition but produces repeated objects and unreasonable structures. The paper adjusts convolutional receptive fields during inference and adds methods for ultra-high-resolution generation. It reports better metric scores than direct inference and attention scaling across its experiment settings.

  • Problem

    Pre-trained diffusion models can preserve image definition at higher resolutions, but direct generation produces repeated objects and unreasonable structures.

    Source: §1 · p. 2

  • Method

    The paper dynamically adjusts convolutional receptive fields during inference and adds dispersed convolution and noise-damped classifier-free guidance for ultra-high-resolution generation.

    Source: §1 · pp. 2–3

  • Results

    The method achieves better metric scores than direct inference and attention scaling across all reported experiment settings, with plausible structures and detailed textures at unseen resolutions.

    Source: §4.1 · p. 8

  • Takeaways & Limitations

    The authors conclude that pre-trained diffusion models can generate higher-resolution images without sacrificing image definition, although object structures worsen under direct inference.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this work, we investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. In addition, the generated images should have arbitrary image aspect ratios. When generating images directly at a higher resolution, 1024 x 1024, with the pre-trained Stable Diffusion using training images of resolution 512 x 512, we observe persistent problems of object repetition and unreasonable object structures. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. As a new perspective, we examine the structural components of the U-Net in diffusion models and identify the crucial cause as the limited perception field of convolutional kernels. Based on this key observation, we propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference. We further propose the dispersed convolution and noise-damped classifier-free guidance, which can enable ultra-high-resolution image generation (e.g., 4096 x 4096). Notably, our approach does not require any training or optimization. Extensive experiments demonstrate that our approach can address the repetition issue well and achieve state-of-the-art performance on higher-resolution image synthesis, especially in texture details. Our work also suggests that a pre-trained diffusion model trained on low-resolution images can be directly used for high-resolution visual generation without further tuning, which may provide insights for future research on ultra-high-resolution image and video synthesis.

1 INTRODUCTION

Generating beyond a diffusion model’s training resolution preserves image definition but can produce repeated objects and implausible structures. The paper attributes repetition to convolutional receptive fields and proposes inference-time changes for higher-resolution generation.

1 INTRODUCTION

  • Direct inference on 512 × 1024 and 1024 × 1024 images from a 512 × 512-trained model produces repeated objects, despite preserving image definition.The authors note that repetition becomes more severe as image size increases.Source: §1 · pp. 1–2
  • Multi-Diffusion and SyncDiffusion address window overlap, while attention scaling and these joint-diffusion approaches do not resolve the repetition issue.SyncDiffusion uses an anchor window to improve Multi-Diffusion’s style consistency; the paper says these methods focus on overlap smoothness.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The paper proposes inference-time re-dilation to increase receptive fields, plus dispersed convolution and noise-damped classifier-free guidance for ultra-high-resolution generation.The introduction describes these methods as requiring no optimization or tuning cost.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The authors identify limited convolutional receptive fields, rather than attention-token quantity, as the primary cause of object repetition.They report that larger inference images worsen repetition and that changing convolution to dilated convolution removes repetition but can introduce local-edge problems.Source: §1 · pp. 1–2
  • The method generates 4096 × 4096 images, 16× the training resolution, and is evaluated across diffusion models and a text-to-video model.The authors report varying image resolutions and aspect ratios in their evaluations.Source: Figure 2 · p. 3, §1 · p. 3

2 RELATED WORK

The related work situates text-to-image diffusion models within the broader development of image synthesis and notes their resolution limits. It contrasts existing high-resolution adaptation approaches with the paper’s network-structure perspective.

2.1 TEXT-TO-IMAGE SYNTHESIS

  • Latent diffusion models use a compact latent space to improve efficiency, and open-source Stable Diffusion models offer high sample quality and creativity.The passage places these models in the development of diffusion-based image generation.Source: §2.1 · p. 3
  • Figure 3 depicts test-time insertion of spaces into fixed pre-trained kernels, fractional dilation through interpolated features, and dispersed kernels with structure- and pixel-level calibration.The caption says structure-level calibration adapts to a larger input feature dimension, while pixel-level calibration preserves local information processing.Source: Figure 3 · p. 4
  • Stable Diffusion 2.1 and SD XL remain limited to their training resolutions, motivating mechanisms for generating 2K or 4K images.The passage gives 512² for SD 2.1 and 1024² for SD XL.Source: §2.1 · p. 4

2.2 HIGH-RESOLUTION SYNTHESIS AND ADAPTATION

  • Prior high-resolution methods train from scratch, fine-tune, or adapt without training; the cited tuning-free approaches leave repetition unresolved or require extra user conditions.The paper proposes a tuning-free network-structure approach to address repetition without those extra conditions.Source: §2.2 · p. 4

3 METHOD

The method targets convolutional receptive-field mismatch when pretrained diffusion U-Nets are run above training resolution. It combines inference-time re-dilation with kernel dispersion and noise-damped guidance.

3.1 PROBLEM FORMULATION AND MOTIVATION

  • The authors formulate training-free adaptation from a low-resolution pretrained diffusion model to higher-resolution output, and investigate convolutional receptive-field limits as a source of repetition.Increasing convolutional receptive fields across all U-Net blocks corrected object counts, while attention re-dilation produced results indistinguishable from direct inference.Source: §3.1 · pp. 4–5

3.2 RE-DILATION

  • Re-dilation adjusts convolutional dilation only at inference, with fractional scales supported by stretching feature maps and with schedules varying across layers and timesteps.Progressively decreasing dilation from deep to shallow layers and from noisier to less noisy steps performed better than a fixed factor.Source: §3.2 · p. 5

3.3 CONVOLUTION DISPERSION

  • Convolution dispersion enlarges pretrained kernels through calibration, and can be combined with re-dilation to expand the perceptual field without periodic subsampling artifacts.The calibration matches outputs on interpolated features and on the original feature map.Source: §3.3 · pp. 5–6

3.4 NOISE-DAMPED CLASSIFIER-FREE GUIDANCE

  • At 4× height and width, outer-block re-dilation supports small-object structures but harms denoising, motivating guidance that combines base and re-dilated predictions.The erroneous noise predictions in the two re-dilated estimates are similar and vanish in their difference, leaving information useful for correct object structures.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate higher-resolution generation across Stable Diffusion versions, scales, aspect ratios, and a text-to-video model. The authors report better metrics than training-free image-generation baselines, texture advantages over a super-resolution model, and improvements from each proposed component.

4 EXPERIMENTS

  • Evaluation covers SD 1.5, SD 2.1, and SD XL 1.0 at four higher-resolution settings, with Laion-5B prompts and FID/KID measures.The settings span 4×, 6.25×, 8×, and 16× the training pixel count; samples use 1:1 aspect ratio except where otherwise specified.Source: §4 · p. 7

4.1 EVALUATION

  • Our method achieves better metric scores than Direct-Inf and Attn-SF across all experiment settings, with plausible structures and detailed textures at unseen resolutions.Visual comparisons cover 4×, 8×, and 16× settings across three Stable Diffusion models.Source: §4.1 · p. 8, Figure 5 · p. 8
  • Against the pretrained SD 2.1 upscaler, our method produces finer texture details, although its FID and KID are slightly worse.The comparison evaluates 4× and 16× generation; at 16×, 300 user responses are used to measure texture-definition preference.Source: §4.1 · pp. 8–9, Figure 6 · p. 8, Table 2 · p. 9

4.2 ABLATION STUDY

  • Removing progression, dispersed convolution, and noise-damped classifier-free guidance drops FID by 16.68, 27.96, and 33.93, respectively.The ablation evaluates SD 1.5 at 16× resolution (2048²); the components also improve detail, object-structure fidelity, and final-image noise, respectively.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 APPLY ON VIDEO DIFFUSION MODELS

  • Applied to pretrained LVDM, the method generates higher-resolution videos without image-definition degeneration; evaluation uses FVD and KVD on 2,048 Webvid-10M videos.The supplied passage reports video results in Table 3 but gives no metric values.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSIONS

The paper investigates sampling above the training resolution of pretrained diffusion models, where direct sampling can preserve definition but causes severe object repetition. It identifies convolution as critical and evaluates re-dilation, dispersed convolution, and noise-damped guidance for higher-resolution image and video generation.

5 CONCLUSIONS

  • Direct higher-resolution sampling can preserve image definition but encounters severe object repetition.The paper explores the receptive fields of Stable Diffusion U-Net components and identifies convolution as critical to higher-resolution sampling.Source: §5 · p. 10
  • Dynamic re-dilation is proposed to remove repetition, alongside dispersed convolution and noise-damped classifier-free guidance for ultra-high-resolution generation.Source: §5 · p. 10
  • Evaluations demonstrate the methods’ effectiveness across different text-to-image and text-to-video models.Source: §5 · p. 10

A.1 MODEL LAYERS IN OUR METHOD

The appendix specifies where re-dilated and dispersed convolutions are used in U-Net blocks. It also identifies convolution layers that are excluded from those operations.

A.1 MODEL LAYERS IN OUR METHOD

  • Attention projection and convolution shortcut layers are excluded from re-dilation and dispersion because their convolution kernels are 1×1.Source: §A.1 · p. 13, Table 5 · p. 13
  • The U-Nets of SD 1.5, SD 2.1, and SD XL 1.0 share a similar convolution-layer layout, and the text-to-video model’s spatial architecture matches SD.Source: §A.1 · p. 13
  • The first and last U-Net convolutions, conv in and conv out, are not modified because they do not contribute to generating image contents.Table 5 lists convolution layers in a U-Net block and notes whether each layer exists in particular blocks.Source: §A.1 · p. 13, Table 5 · p. 13

A.2 HYPERPARAMETERS

The appendix describes the hyperparameter settings for re-dilated and dispersed blocks and for noise-damped guidance. It refers readers to figures and tables for the detailed block configurations.

A.2 HYPERPARAMETERS

  • Experiments use the pretrained models’ default classifier-free guidance scales: 7.5 for SD 1.5 and SD 2.1, and 5.0 for SD XL 1.0.Source: §A.2 · p. 14
  • The settings tables specify which U-Net blocks use re-dilated and dispersed convolutions, with separate dilation scales for each.SD 2.1 uses settings similar to SD 1.5; SD 1.5 settings are listed for brevity, and SD XL settings are shown in figures.Source: §A.2 · p. 14, Table 6 · p. 14, Table 7 · p. 14
  • For noise-damped guidance, selected outskirt blocks are returned to their original form to construct a predictor with strong denoising capability.The selected blocks are listed as noise-damped blocks; Figure 9 provides reference block names and locations.Source: Figure 9 · p. 13, §A.2 · p. 14

A.3 SYNCHRONIZE STATISTICS BETWEEN TILES IN GROUPNORM

For decoding images larger than 2048 × 2048, the VAE requires enormous VRAM, so tiled decoding is used. Synchronizing GroupNorm statistics across tiles addresses their color-tone differences.

  • For images larger than 2048 × 2048, the VAE requires enormous VRAM and is usually unsuitable for a personal GPU.Source: §A.3 · p. 15
  • Tiled decoding can create abrupt tile-border changes and inconsistent color tones; overlap interpolation addresses border changes but not the tone differences.Source: §A.3 · p. 15, Figure 10 · p. 15
  • Computing GroupNorm mean and standard deviation across all tiles, rather than only the current tile, efficiently eliminates color-tone differences.Source: §A.3 · p. 15, Figure 10 · p. 15

B RE-DILATED ATTENTION

The re-dilated attention experiment preserves the training token quantity but does not resolve generated-image structure; re-dilating convolutional kernels does.

  • Re-dilated attention splits the feature map into four slices at 4× training resolution, keeping each attention token sequence at the training length.Source: B RE-DILATED ATTENTION · p. 15
  • Keeping attention token quantity at the training level does not resolve the generated image’s structural issue.Source: B RE-DILATED ATTENTION · p. 15
  • Applying re-dilation to convolutional kernels produces a structurally correct image, supporting convolutional kernels as the source of repetition.Source: B RE-DILATED ATTENTION · p. 15

C OTHER VISUALIZATIONS

The paper provides additional generated results at arbitrary aspect ratios and sizes for SD 2.1, SD 1.5, and SD XL 1.0.

  • Figure 12 shows additional results with the method and SD 2.1 at arbitrary aspect ratios and sizes.Source: Figure 12 · p. 16
  • Figure 13 shows additional results with the method and SD 1.5 at arbitrary aspect ratios and sizes.Source: Figure 13 · p. 17
  • Figure 14 shows additional results with the method and SD XL 1.0 at arbitrary aspect ratios and sizes.Source: Figure 14 · p. 18
Loading 2310.07702v1…