PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization

Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, Lei Zhang

arXiv:2308.14469v4cs.CV

TL;DR

Real-ISR and personalized stylization require realistic outputs without losing faithful pixel-level structure, a weakness in existing diffusion-based approaches. PASD adds pixel-aware and degradation-robust conditioning to pre-trained diffusion, and experiments report effectiveness across enhancement and stylization tasks, while fidelity remains difficult for heavily degraded inputs.

  • Problem

    Existing diffusion-based methods often fail to preserve faithful pixel-wise image structures, while extra skip connections require image-space training and limit latent-space applications such as stylization.

  • Method

    PASD uses pixel-aware cross attention, degradation removal, high-level information, and an adjustable noise schedule to guide a pre-trained stable diffusion model.

  • Results

    PASD demonstrates effectiveness and flexibility across realistic image restoration and personalized stylization tasks, producing diverse stylization results with highly consistent semantic contents.

  • Takeaways & Limitations

    Switching the base diffusion model lets PASD support personalized stylization without additional training and provides flexible perception-fidelity control during inference.

  • Takeaways & Limitations

    PASD may fail to reproduce faithful details when inputs are heavily degraded or semantic information is inaccurate, while fidelity-perception balance remains unresolved.

Abstract

from arXiv · show

Diffusion models have demonstrated impressive performance in various image generation, editing, enhancement and translation tasks. In particular, the pre-trained text-to-image stable diffusion models provide a potential solution to the challenging realistic image super-resolution (Real-ISR) and image stylization problems with their strong generative priors. However, the existing methods along this line often fail to keep faithful pixel-wise image structures. If extra skip connections between the encoder and the decoder of a VAE are used to reproduce details, additional training in image space will be required, limiting the application to tasks in latent space such as image stylization. In this work, we propose a pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR and personalized image stylization. Specifically, a pixel-aware cross attention module is introduced to enable diffusion models perceiving image local structures in pixel-wise level, while a degradation removal module is used to extract degradation insensitive features to guide the diffusion process together with image high level information. An adjustable noise schedule is introduced to further improve the image restoration results. By simply replacing the base diffusion model with a stylized one, PASD can generate diverse stylized images without collecting pairwise training data, and by shifting the base model with an aesthetic one, PASD can bring old photos back to life. Extensive experiments in a variety of image enhancement and stylization tasks demonstrate the effectiveness of our proposed PASD approach. Our source codes are available at \url{https://github.com/yangxy/PASD/}.

1 Introduction

Real-world restoration must recover perceptually realistic details despite complex degradations, while existing GAN and diffusion approaches struggle with artifacts, limited detail, or pixel-level faithfulness. PASD addresses these challenges with pixel-aware conditioning and flexible diffusion controls for Real-ISR and personalized stylization.

  • Real-world images contain mixtures of low resolution, blur, noise, and other acquisition degradations, while fidelity-focused restoration often produces over-smoothed details.
  • GAN-based Real-ISR can produce promising results but remains limited in rich realistic detail reproduction and may introduce unpleasant visual artifacts.
  • PASD introduces pixel-aware cross attention to perceive local pixel-level information without skipped connections, while degradation removal reduces unknown degradation effects during diffusion.
  • High-level classification, detection, and captioning information extracted from the input can further boost Real-ISR performance.
  • Compared with ControlNet, PASD preserves input structure better at the pixel level, whereas ControlNet produces clear content inconsistency.
  • An adjustable noise schedule improves flexibility across perception-fidelity trade-offs, and switching to a personalized base model enables stylization without additional training.

2 Related Work

Related work spans fidelity-oriented and adversarial Real-ISR, diffusion-prior restoration, stylization, and old-photo restoration. PASD builds on pre-trained latent diffusion while targeting pixel-level restoration and a simpler, flexible workflow.

  • Realistic Image Super-Resolution: Deep-learning super-resolution methods often over-smooth details because they optimize fidelity objectives such as PSNR and SSIM, motivating perceptual Real-ISR.
  • Realistic Image Super-Resolution: GAN-based Real-ISR methods can generate unnatural artifacts because adversarial training is unstable and remain difficult to extend with additional details.
  • Realistic Image Super-Resolution: Diffusion-prior methods leverage pre-trained text-to-image models for Real-ISR, while PASD specifically targets pixel-level restoration of image details and textures.
  • Personalized Stylization: Personalized stylization methods have used optimization or image-to-image frameworks, but prior approaches often require a style image or lack pairwise training data.
  • Personalized Stylization: Old-photo restoration involves unknown mixed degradations and prior multi-stage latent translation methods can be unstable and difficult to use.
  • Diffusion Probabilistic Models: Latent diffusion models extend diffusion generation into latent space with reduced computational costs, providing the foundation for PASD's approach.

3 Pixel-Aware Stable Diffusion Network

PASD augments a pretrained Stable Diffusion model with degradation-aware features, pixel-aware conditioning, adjustable noise, and high-level image information. These components target faithful Real-ISR structures while supporting personalized stylization through model replacement.

  • PASD combines a pretrained Stable Diffusion model with degradation removal, high-level information, pixel-aware conditioning, and adjustable noise modules.The degradation removal module extracts degradation-insensitive low-level control features, while the architecture adds several complementary conditioning mechanisms.
  • The degradation removal module uses multiscale features and intermediate RGB supervision so diffusion can focus on realistic detail recovery.An L1 loss compares each scale's reconstruction with the corresponding HQ-image pyramid decomposition, and this module is required only for Real-ISR.
  • ControlNet-style feature addition can produce structure inconsistencies, motivating PACA for pixel-level information without skipped connections.Skipped connections also require image-space training, limiting use in latent-space tasks such as image stylization.
  • PACA treats the unencoded conditional feature map as pixel-aligned context, allowing diffusion features to perceive original image structures.The conditional feature length equals the total number of latent-feature pixels, while remaining outside the VAE latent space preserves image structures.
  • ANS mixes the low-quality latent with Gaussian noise to adjust residual-signal strength and provide a flexible perception-fidelity trade-off.The method addresses train-test discrepancy in Stable Diffusion's terminal noise schedule; severe degradations can still leave discrepancy-related harm.
  • Replacing the base diffusion model with personalized or aesthetic models enables cartoonization and old-photo restoration without pairwise stylization training data.The approach decouples stylization generation from pixel-to-pixel mapping and can improve old-photo quality and aesthetics simultaneously.

4 Experiments

PASD is evaluated across Real-ISR, cartoonization, and old photo restoration, including quantitative metrics, visual comparisons, user studies, and ablations. The experiments examine its adjustable noise schedule, comparative performance, model reuse across stylization tasks, and component contributions.

  • Experimental settings: The experiments evaluate Real-ISR on synthetic and real-world datasets, while cartoonization reuses the Real-ISR-trained PASD with stylized base models and old photo restoration uses aesthetic base models.Real-ISR training uses DIV2K, Flickr2K, OST, and FFHQ-derived pairs; testing includes synthetic and real-world data, cartoonization uses 100 FFHQ and Flickr2K images, and old photo restoration uses 100 Internet-collected photos.
  • Evaluation: Evaluation combines perceptual and fidelity metrics with user studies in which 15 volunteers select preferred outputs among competing methods.Real-ISR reports FID, LPIPS, DISTS, NIQE, MUSIQ, and QAlign, with PSNR and SSIM provided for reference; stylization uses FID, MUSIQ, and QAlign.
  • Adjustable noise schedule: Increasing ¯αa raises PSNR but lowers QAlign, exposing a flexible perception-fidelity trade-off; the authors use ¯α900 = 0.1189 as a practical balance.Visual results show higher ¯αa improves fidelity while producing less realistic details.
  • Realistic image super-resolution: Across three test datasets, PASD achieves the best QAlign scores, while diffusion methods generally outperform GAN-based methods on perception metrics but not PSNR/SSIM.PASD also receives the most rank-1 votes in the Real-ISR user study and generates more realistic details in visual comparisons.
  • Personalized stylization: PASD produces cleaner cartoonization, preserves details such as human hair, and supports varied stylization by switching personalized base models without additional training.Most subjects preferred PASD in the cartoonization user study.
  • Old photo restoration: PASD is preferred in old photo restoration, achieves the best results on all three reported indices, and better recovers vivid details such as human hair.The method reuses the pretrained PASD model while replacing the base model with an aesthetic one.
  • Ablation studies: Ablations show that removing PACA causes color and structural inconsistency, removing degradation removal causes dirty outputs and worse metrics, and omitting high-level information or negative prompts reduces quality.The full model balances clean-smooth and detailed-dirty outputs and achieves the best FID score in the ablation comparison.

5 Conclusion and Limitation

PASD is proposed for realistic image restoration and personalized stylization, combining pixel-aware diffusion with an adjustable perception–fidelity trade-off. The authors report broad effectiveness while identifying degradation severity and semantic accuracy as limitations.

  • PASD targets realistic image restoration and personalized stylization with a pixel-aware diffusion network.
  • PACA enables pixel-level local-structure perception without skipped connections, supporting robust and perceptually realistic Real-ISR results.
  • An adjustable noise schedule provides a flexible perception–fidelity trade-off during inference.
  • Switching to a personalized base model produces diverse stylization results with highly consistent semantic contents.
  • PASD may fail to reproduce faithful details when inputs are heavily degraded or semantic information is inaccurate.The authors suggest more robust degradation estimation and more precise semantic extraction as future improvements.

Supplementary Materials

The supplementary materials add visual comparisons across Real-ISR, stylization, old-photo restoration, and colorization, alongside details on noise-schedule handling. They report realistic fine-detail recovery, pixel-wise detail preservation during stylization, semantic-aware restoration, and reduced color bleeding.

  • The supplement provides additional Real-ISR, stylization, old-image restoration, and image-colorization results.
  • Realistic Image Super-Resolution: PASD comparisons against RealESRGAN, SwinIR, StableSR, DiffBIR, and SeeSR show adjustable pixel-level guidance, realistic fine details, and fewer visual artifacts.
  • Various Stylization Results: Switching to Disney 3D, Oil painting, or Shinkai personalized base models enables stylization without additional training while preserving pixel-wise image details.
  • Old Image Restoration: PASD comparisons with Real-ESRGAN, FeMaSR, StableSR, and DiffBIR show better recovery of semantic-aware and photo-realistic details in old images.
  • Image Colorization: For image colorization, the standard adjustable noise schedule cannot directly compensate RGB residual signals from grayscale inputs, so the schedule is rescaled.The supplement gives the scaled schedule as α′_t=(√ᾱ_t−√ᾱ_N)/(√ᾱ_1−√ᾱ_N), with α′ described as the scaled α.
  • Image Colorization: With zero SNR, epsilon prediction becomes trivial, so PASD is finetuned using a rescaled noise schedule and ν loss and prediction.
  • Image Colorization: For colorization, PASD is reported to generate more photo-realistic and vivid results while significantly alleviating color bleeding versus DeOldify, BigColor, CT2, and DDColor.
Loading 2308.14469v4…