PaperlayerView exact revision on arXiv

Source-linked AI summary

Palette: Image-to-Image Diffusion Models

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, Mohammad Norouzi

arXiv:2111.05826v2cs.CVcs.LG

TL;DR

Image-to-image translation involves challenging multimodal inverse problems, and it was unclear whether diffusion models could provide a versatile general framework. Palette applies conditional diffusion to colorization, inpainting, uncropping, and JPEG restoration, achieving strong results without task-specific customization while supporting standardized evaluation and multi-task modeling.

  • Problem

    It was unclear whether diffusion models could rival GANs as a versatile and general framework for image manipulation across image-to-image tasks.

  • Method

    Palette is a conditional image-to-image diffusion framework evaluated across colorization, inpainting, uncropping, and JPEG restoration, with studies of denoising loss, architecture, and standardized evaluation.

  • Results

    Palette outperforms strong GAN and regression baselines across four tasks without task-specific architecture, hyper-parameter, loss, or auxiliary-objective changes.

  • Takeaways & Limitations

    A generalist multi-task Palette model performs as well or better than task-specific counterparts, while L2 denoising provides greater sample diversity than L1.

  • Takeaways & Limitations

    Palette requires many refinement steps during generation, resulting in significantly slower inference than GAN-based models.

Abstract

from arXiv · show

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io for an overview of the results.

1 INTRODUCTION

Palette tests whether conditional diffusion models can provide a versatile image-to-image framework across challenging tasks. It achieves high-fidelity results without task-specific customization while revealing effects of denoising loss and self-attention.

  • Motivation: Diffusion models address image-to-image tasks as multimodal inverse problems by modeling the conditional distribution of output images given inputs.Examples include colorization, inpainting, super-resolution, and depth estimation.
  • Motivation: Diffusion models had achieved strong results in speech synthesis, class-conditional ImageNet generation, and super-resolution, but their versatility for image manipulation remained unclear.The paper investigates whether they can rival GANs as a general image-manipulation framework.
  • Contributions: Palette delivers high-fidelity outputs on colorization, inpainting, uncropping, and JPEG restoration without task-specific architecture, hyper-parameter, or loss changes.It outperforms task-specific and strong regression baselines using an identical neural architecture.
  • Multi-task modeling: A single generalist Palette model outperforms a task-specific JPEG model and performs competitively on the other tasks.The multi-task model is trained on colorization, inpainting, and JPEG restoration.
  • Empirical findings: L2 and L1 denoising losses produce similar sample-quality scores, but L2 yields more diverse samples while L1 produces more conservative outputs.Removing self-attention from Palette’s U-Net also hurts performance.

2 RELATED WORK

Related work spans GAN-based translation, emerging diffusion approaches, restoration and uncropping methods, and relatively under-explored multi-task training. Palette extends conditional diffusion across diverse image-to-image tasks without task-specific specialization.

  • Prior image-to-image methods: GANs dominate image-to-image translation but can require auxiliary objectives, be domain-specific, and lack output diversity.These limitations are noted for inpainting and uncropping applications.
  • Diffusion models: Diffusion models have shown strong results in image generation, audio synthesis, super-resolution, unpaired translation, and image editing.Palette builds conditional diffusion models on these advances.
  • Inpainting and uncropping: Conditional diffusion models trained on large datasets reliably address both inpainting and uncropping across image domains.Uncropping is more challenging than inpainting because it generates open-ended content with less context.
  • Task-specific context: Palette removes JPEG compression artifacts at quality factors as low as 5, whereas prior GAN-based methods were restricted to quality factors above 10.It also performs colorization without task-specific auxiliary classification losses.
  • Multi-task training: Multi-task training remains relatively under-explored in image-to-image translation, with prior work focusing mainly on enhancement tasks and smaller modular networks.Palette takes a first step toward multi-task diffusion models for a wider variety of tasks.

3 PALETTE

Palette is a conditional image-to-image diffusion model that denoises noisy target images using source images and noise-level information. Its design uses a modified U-Net, with L2 selected for greater sample diversity.

  • Model formulation: Image-to-image diffusion models represent conditional distributions p(y|x), where both x and y are images.For example, x may be grayscale and y may be color.
  • Denoising objective: Palette trains a neural network to denoise a noisy target image given the source image and a noise-level indicator.The loss is defined over the denoising prediction.
  • Denoising objective: L1 denoising produces significantly lower sample diversity than L2, so Palette adopts L2 to capture the output distribution more faithfully.L1 may instead help reduce potential hallucinations in some applications.
  • Architecture: Palette uses a modified 256×256 class-conditional U-Net without class conditioning and with source-image conditioning by concatenation.The architecture is inspired by recent diffusion-model designs.

4 EVALUATION PROTOCOL

The paper proposes standardized evaluation for difficult image-to-image tasks, addressing inconsistent datasets and task-specific metrics. It combines ImageNet-based benchmarks, automated quality and diversity measures, and controlled human evaluation.

  • Evaluation motivation: Prior evaluation uses inconsistent datasets and task-dependent criteria, including FID, human judgments, PSNR, and SSIM.Many tasks lack standardized test sets, while pixel metrics can favor blurry regression outputs on hallucination-heavy tasks.
  • Datasets and protocol: The proposed protocol benchmarks inpainting, uncropping, and JPEG restoration on ImageNet, with Places2 evaluation retained for inpainting and uncropping.It introduces the ImageNet ctest10k and Places2 places10k subsets.
  • Automated metrics: Automated evaluation uses Inception Score, Fréchet Inception Distance, top-1 ResNet-50 Classification Accuracy, and Perceptual Distance.Pairwise SSIM and LPIPS are also used for sample diversity on some tasks.
  • Human evaluation: Human evaluation uses 2-alternative forced-choice trials to compare model outputs with natural images and reports the resulting fool rate.The protocol measures whether humans can discriminate generated outputs from natural images.

5 EXPERIMENTS

Palette is evaluated across four image-to-image tasks using a unified setup without task-specific engineering. It achieves strong results across these tasks, while ablations examine architecture, diversity, and multi-task training.

  • 5 EXPERIMENTS: Palette applies one RGB-image framework to colorization, inpainting, uncropping, and JPEG restoration without task-specific hyper-parameter tuning, architecture customization, or auxiliary losses.The tasks involve distinct inverse problems, including open-ended uncropping and nonlinear JPEG artifact removal.
  • Colorization: Palette establishes a new state of the art in colorization, outperforming Pix2Pix, PixColor, and ColTran by a large margin.FID, IS, and CA indicate outputs close to the original images, while human evaluation improves ColTran’s fool rate by more than 10%.
  • Inpainting: Palette outperforms DeepFillv2, HiFill, and Co-ModGAN by a large margin across inpainting datasets and mask configurations.For 20–30% free-form masks, Palette’s FID scores are extremely close to those of the original unmasked images.
  • Uncropping: Palette outperforms Boundless and InfinityGAN on ImageNet and Places2 uncropping, reaching a 40% human-evaluation fool rate versus 25% and 15%.Repeated uncropping produces realistic, coherent panoramas after eight applications and appealing zoom-out sequences.
  • JPEG restoration: Palette outperforms a regression baseline for JPEG restoration across quality factors, with a widening gap as quality factor decreases.At quality factor 5, Palette produces sharper images while the regression model produces blurry outputs.
  • Self-attention in diffusion model architectures: Global self-attention improves inpainting performance over fully convolutional alternatives, whereas local self-attention performs worse than fully convolutional alternatives.Palette requires 0.8 sec./image for 1000 test images on a TPUv4, and sampling is slower than GAN models.
  • Sample diversity: L2 and L1 denoising losses yield similar FID scores, but L2 produces more diverse samples and L1 produces more conservative outputs.Lower SSIM and higher LPIPS indicate greater diversity; the authors leave comparison with diversity-focused GAN techniques to future work.
  • Multi-task learning: A generalist Palette model outperforms the task-specific JPEG specialist but slightly lags behind task-specific models on inpainting and colorization.The models used the same number of training steps, and the authors expect multi-task performance to improve with more training.

6 CONCLUSION

Palette is presented as a simple, general framework for image-to-image translation that achieves strong results across four challenging tasks and supports effective multi-task training.

  • Palette achieves strong results on colorization, inpainting, uncropping, and JPEG restoration, outperforming strong GAN and regression baselines.
  • The framework produces diverse and high-fidelity outputs without task-specific customization or optimization instability.
  • A multi-task Palette model performs just as well or better than task-specific counterparts.

Supplementary Material for Palette: Image-to-Image Diffusion Models

Palette learns a conditional reverse diffusion process that recovers target images from noisy inputs, using iterative denoising from Gaussian noise at inference.

  • Diffusion Models: Diffusion models add Gaussian noise through a forward Markov process and use reverse denoising for generation.The forward process runs for T iterations, with the terminal state designed to approximate Gaussian noise.
  • Learning: Palette conditions its neural network on the input image, noisy target image, and current noise level while predicting the noise vector.The model’s objective is also known as L_simple and is equivalent to maximizing a weighted variational lower bound on likelihood.
  • Inference: At inference, Palette starts from pure Gaussian noise and performs T iterative refinement steps using the learned reverse process.The forward prior is constructed so p(y_T) approximates a standard normal distribution.
  • Reverse Process: Each reverse iteration uses the network’s estimate of the clean image to parameterize the reverse-process mean, with variance set to 1 − α_t.The resulting update resembles one step of Langevin dynamics, with f_θ estimating the data log-density gradient.

B IMPLEMENTATION DETAILS

The implementation uses fixed training and diffusion settings across tasks, while task-specific details define data representations, masks, and crops.

  • Training Details: All models use batch size 1024, 1M training steps, Adam with learning rate 1e-4, 10k warmup steps, and 0.9999 EMA.The reported checkpoint is the 1M-step model, and no task-specific hyper-parameter tuning is performed.
  • Diffusion Hyper-parameters: Training uses a linear noise schedule from 1e-6 to 0.01 over 2000 steps, while inference uses 1000 refinement steps with a 1e-4 to 0.09 schedule.Alpha conditioning permits inference-time tuning over noise schedules and refinement steps.
  • Task Specific Details: Colorization uses grayscale inputs and trains Palette to predict full RGB images from randomly cropped and resized 256×256 images.The RGB parameterization follows prior colorization work.
  • Task Specific Details: Inpainting combines free-form and rectangular masks, fills masked regions with Gaussian noise, and restricts L_simple training and inference to masked regions.Rectangular masks cover 10% to 40% of the image, while free-form masks are sampled with 60% probability.

C.1 Colorization

Colorization is evaluated on ImageNet using standard benchmark metrics and controlled human judgments comparing Palette with established baselines.

  • Evaluation: Models are evaluated using the first 5000 ImageNet validation images, with the next 5000 serving as the FID reference distribution.The protocol follows ColTran’s implementation; an additional ctest10k ImageNet benchmark is reported in Table C.1.
  • Human Evaluation: Human evaluation asks subjects to identify which of two side-by-side images appears to come from a camera.Subjects view images for either 3 or 5 seconds, and comparisons include PixColor, ColTran, Regression, and Palette.
  • Human Evaluation: Palette’s fool rate is close to 50% and higher than every baseline in all reported colorization cases.Shorter inspection times produce somewhat higher fool rates, and the regression baseline also exceeds PixColor and ColTran.

C.2 Inpainting

The supplementary inpainting and uncropping evaluations use controlled image configurations and compare Palette variants with prior methods and human judgments.

  • Inpainting Evaluation: Inpainting results use 256×256 center-cropped images and fixed image-mask pairs across models to ensure fair comparisons.For HiFill and Co-ModGAN, 512×512 center crops are used because those methods are primarily trained at that resolution.
  • Inpainting Evaluation: Palette inpainting models are trained either on ImageNet alone or on a uniform mixture of ImageNet and Places2.Results are categorized by mask type and covered area, including free-form and rectangular configurations.
  • Human Evaluation: Human-evaluation figures report fool rates for ImageNet colorization and Places2 uncropping under 3-second display conditions.The figures are labeled by task and dataset, with separate Set-I and Set-II fool-rate conditions.
  • Inpainting Results: Palette trained only on ImageNet is slightly worse than the mixed-data model on Places2, despite never training on Places2 images.The supplementary comparison also reports a significant performance drop for HiFill with larger masks.

C.3 Uncropping

Palette is trained on combined Places2 and ImageNet data but evaluated on right-half uncropping for fair comparison, outperforming existing uncropping methods in automated and human evaluations.

  • Experimental setup: Palette is trained on combined Places2 and ImageNet data, while fair comparison evaluates extending only the right half of images.The model is also trained for extension in all directions or one direction.
  • Comparison with existing methods: Palette significantly outperforms Boundless on the top-50 Places2 categories used by that baseline.Boundless is trained only on the top-50 Places2 categories.
  • Comparison with existing methods: Palette is significantly better than InfinityGAN on scenery images in automated metrics.The comparison uses scenery categories from the Places2 validation and test sets.
  • Human evaluation: Palette achieves significantly higher fool rates than Boundless and InfinityGAN across all uncropping human-evaluation runs.When raters inspect images longer, baseline fool rates worsen considerably, whereas Palette’s remain approximately similar.
  • Evaluation protocol: The evaluation framework reports FID, Inception Score, Perceptual Distance, and Classification Accuracy using specified ImageNet and Places2 reference distributions.ImageNet uses the 10,000-image ctest10k subset, while FID references the full validation set; Places2 uses places10k and 36.5k validation images.

D LIMITATIONS

Palette’s main limitations are slow diffusion sampling, restricted input-resolution generality, and implicit biases that require attention before practical deployment.

  • Sampling efficiency: Palette uses 1k refinement steps for generation, making inference significantly slower than GAN-based models.The paper notes that reducing refinement steps is an active research direction and leaves applying proposed techniques to future work.
  • Input resolution: Group normalization and self-attention prevent Palette from generalizing to arbitrary input resolutions.The paper identifies fine-tuning and patch-based inference as possible directions for adapting such models.
  • Deployment: Palette, like other generative models, suffers from implicit biases that should be studied and mitigated before deployment.This limitation is presented as a practical concern for using the model in practice.
Loading 2111.05826v2…