Source-linked AI summary
Pixel Recursive Super Resolution
Ryan Dahl, Mohammad Norouzi, Jonathon Shlens
TL;DR
High-magnification super resolution is underspecified because one low-resolution input can match multiple plausible high-resolution images, making pixel-independent prediction prone to blurry averaging. The paper uses a probabilistic PixelCNN-based model with a deep conditioning network to model dependencies among output pixels and reports stronger human-rated realism than regression baselines.
Problem
High-magnification super resolution is underspecified because small inputs omit high-frequency details and may correspond to multiple plausible high-resolution images.
Method
The paper jointly models a PixelCNN prior and a deep conditioning convolutional network to generate high-resolution pixels recursively given a low-resolution input.
Results
27.9% of the time, samples from the probabilistic model fooled human observers, compared with a 50% chance rate.
Takeaways & Limitations
Human evaluation supports diverse, sharp samples as a useful target for underspecified super resolution, while pSNR and SSIM do not correlate with human judgment at large magnification ratios.
Takeaways & Limitations
The model can suffer from the conditioning network being ignored because correlations with previous high-resolution pixels may be stronger than correlations with the low-resolution input.
Abstract
from arXiv · showhide
We present a pixel recursive super resolution model that synthesizes realistic details into images while enhancing their resolution. A low resolution image may correspond to multiple plausible high resolution images, thus modeling the super resolution process with a pixel independent conditional model often results in averaging different details--hence blurry edges. By contrast, our model is able to represent a multimodal conditional distribution by properly modeling the statistical dependencies among the high resolution image pixels, conditioned on a low resolution input. We employ a PixelCNN architecture to define a strong prior over natural images and jointly optimize this prior with a deep conditioning convolutional network. Human evaluations indicate that samples from our proposed model look more photo realistic than a strong L2 regression baseline.
1. Introduction
At high magnification, super resolution is underspecified because missing details admit multiple plausible reconstructions. The paper proposes a probabilistic model that generates diverse, realistic high-resolution samples and evaluates them with human judgments.
- 1. Introduction: High-magnification super resolution must synthesize plausible novel details because fine information is absent from very small inputs.The paper focuses on particularly small inputs and large magnification ratios, where variations in objects, viewpoints, illumination, and occlusions become relevant.
- 1. Introduction: Human observers could distinguish real images from sophisticated regression-based outputs, while the proposed samples fooled observers 27.9% of the time.The comparison uses a 50% chance rate for the human evaluation task.
- 1. Introduction: The paper characterizes underspecified super resolution as a multimodal prediction problem.A single low-resolution image may correspond to multiple plausible high-resolution images.
- 1. Introduction: The proposed probabilistic model produces diverse, plausible, non-blurry high-resolution samples by modeling multiple possible outputs.It is trained end-to-end with a log-likelihood objective and synthesizes 32×32 faces from 8×8 inputs.
- 1. Introduction: The paper argues that pSNR and SSIM fail to capture sample quality in underspecified super resolution.Human evaluation is used to assess perceptual quality beyond traditional super-resolution metrics.
- 1. Introduction: The model builds on ResNet and PixelCNN and emphasizes diversity and human-rated sample quality.The paper describes a probabilistic model jointly developed with a deep conditioning network and evaluates generated samples through human studies.
2. Related work
Prior super-resolution methods range from interpolation and image-prior techniques to CNNs, perceptual losses, and adversarial training. PixelCNN offers an explicit probabilistic alternative that models dependencies among image pixels, while GANs face instability and mode-collapse concerns.
- 2. Related work: Interpolation methods are simple and widely used but often produce blurry outputs because linear models lack expressive input-output dependencies.Richer priors and low-level image statistics improve these methods, but the related work also discusses their limitations.
- 2. Related work: CNN-based methods learn layered abstractions directly from images, with prior work using MSE, deeper residual networks, and perceptual losses.Perceptual-loss approaches compare predictions and ground truth through pretrained CNN activations or optimize through gradient propagation.
- 2. Related work: Adversarial losses can produce compelling high-frequency detail compared with traditional Lp losses.The related work describes super-resolution networks trained against discriminators to distinguish synthesized images from real ones.
- 2. Related work: GANs face unstable training and mode collapse, which can reduce the diversity of generated samples.The paper identifies these as drawbacks of adversarial approaches.
- 2. Related work: PixelRNN and PixelCNN model images as ordered sequences whose later pixels depend on previously observed pixels.PixelCNN uses log-likelihood training and has demonstrated strong predictive ability on benchmarks including CIFAR-10 and MNIST.
3. Probabilistic super resolution
The paper formulates super resolution as learning a conditional distribution over high-resolution images given low-resolution inputs. It shows that pixel-independent objectives cannot coordinate multimodal details, motivating recursive pixel models that capture dependencies among output pixels.
- 3. Probabilistic super resolution: The model learns pθ(y | x) from paired low- and high-resolution images by maximizing conditional log-likelihood.The dataset consists of low-resolution inputs and corresponding ground-truth high-resolution outputs.
- 3. Probabilistic super resolution: The multimodal failure of independent objectives motivates models that jointly represent output-pixel dependencies, although alternatives such as Gaussian mixtures may make inference expensive.The paper presents this dependency requirement as the central challenge for realistic non-blurry outputs.
- 3. Probabilistic super resolution: Pixel-independent models factor the conditional distribution into separate predictions for each output pixel.The paper discusses Gaussian and multinomial models for continuous and discrete pixel values.
- 3. Probabilistic super resolution: With fixed variance, Gaussian conditional log-likelihood reduces to minimizing MSE, yielding a unimodal isotropic multivariate Gaussian.The CNN transformation C_i(x) supplies the estimated mean for output pixel y_i.
- 3. Probabilistic super resolution: Pixel-independent multinomial predictions can express per-pixel multimodality but cannot coordinate dependencies across pixels.They cannot choose coherently between alternative locations for an edge or other image structure.
- 3.2. Synthetic multimodal task: On the MNIST corners dataset, L2 regression blends digits from two possible locations, while per-pixel cross-entropy produces sharper but still non-stochastic predictions.The dataset randomly places digits in either the top-left or bottom-right corner, never both.
4. Pixel recursive super resolution
The model addresses super resolution as a multimodal prediction problem by generating pixels autoregressively while conditioning on the low-resolution input. A PixelCNN prior and convolutional conditioning network are jointly optimized to produce diverse, sharp high-resolution samples.
- Motivation: The lack of conditional independence between predicted pixels is a significant failure mode for previous probabilistic objectives.Jointly modeling dependencies avoids treating each output pixel as an independent stochastic prediction.
- Autoregressive model: The joint distribution is factorized by ordering image pixels, conditioning each output dimension on the input and previously generated pixels.This chain-rule formulation retains flexible conditional dependencies while keeping inference straightforward.
- Autoregressive model: PixelCNN captures multimodality and pixel correlations using masked convolutions that predict each pixel from earlier pixels.The model is designed to represent multiple plausible image configurations rather than average them together.
- Architecture: For pixel i, the conditional distribution is a softmax over the sum of conditioning logits from A_i(x) and prior logits from B_i(y_<i).The conditioning network receives the low-resolution image, while the prior network uses preceding high-resolution pixels.
- Architecture: The late-fusion architecture combines a PixelCNN prior network for serial pixel dependencies with a convolutional conditioning network for global low-resolution structure.This design explicitly addresses the tendency of the autoregressive model to ignore the low-resolution conditioning signal.
- Training: The networks are jointly trained by maximizing conditional log likelihood with cross-entropy losses, including an additional conditioning-network loss.The added loss counters optimization behavior in which the powerful autoregressive decoder ignores the conditioning network.
- Sampling: Sampling proceeds pixel by pixel, feeding previously sampled values back into the network; greedy decoding and temperature-controlled sampling provide alternative decoding strategies.Color channels are generated sequentially, and decreasing temperature concentrates the distribution toward its mode.
- Implementation details: The implementation uses a ResNet-like conditioning CNN with transposed convolutions and a 20-block gated PixelCNN, producing distributions over 256 color-channel values.The final output applies a softmax over the summed activations of the conditioning and prior networks.
5. Experiments
Experiments evaluate pixel recursive super resolution on 8×8-to-32×32 CelebA and LSUN Bedroom images against nearest-neighbor, ResNet L2, GAN, and interpolation baselines. The model generates diverse samples, whose perceptual quality depends on sampling temperature and is better captured by human judgments than conventional image-similarity or likelihood metrics.
- Super resolution samples: Pixel recursive samples capture dataset structure and produce multiple plausible high resolution images for the same low resolution input.The samples differ in qualitative features while remaining conditioned on the input.
- Super resolution samples: Tuning the sampling temperature to τ between 0.9 and 0.8 improves sample quality, whereas greedy decoding produces overly smooth images with line artifacts.Temperature controls the concentration of the predictive distribution; τ = 1.0 can introduce undesired high frequency content.
- Quantitative evaluation of image similarity: pSNR and SSIM did not track perceived quality: bicubic interpolation scored relatively well despite poor samples, while negative log likelihood likewise favored perceptually worse outputs.Figure 6 notes that bicubic upsampling and greedy decoding have the best log-probability despite poor image quality.
- Quantitative evaluation of image similarity: The pixel recursive model achieved consistency comparable to L2 regression and bicubic interpolation, superior to GAN consistency, and better than nearest-neighbor consistency.These results indicate diverse outputs remained largely constrained by the low resolution input and were not simple nearest-training-sample copies.
- Perceptual evaluation with humans: 11.0% and 27.9% of pixel recursive samples fooled humans for faces and bedrooms, respectively, outperforming the GAN across all temperatures.The L2 regression fooled humans 24% of the time and the GAN 8.5% of the time; 50% represents perfect confusion.
6. Conclusion
The paper targets highly underspecified high-magnification super resolution, where missing high-frequency details require plausible predictions rather than simple recovery. Its probabilistic model produces sharp, diverse outputs and human studies find them more plausible than strong baselines, while standard metrics may fail to reflect that quality.
- 6. Conclusion: High-magnification super resolution is dramatically underspecified because high-frequency details are absent from small inputs.The method enlarges 8×8 images to sharp 32×32 images.
- 6. Conclusion: The fully probabilistic pixel recursive method produces non-blurry outputs by modeling plausible missing content and outperforms strong regression and adversarial baselines.The conclusion frames this as a model for the multimodal high-magnification regime.
- 6. Conclusion: Human evaluations find pixel recursive samples more plausible, while pSNR and SSIM do not correlate with human judgment at large magnification ratios.This conclusion follows the reported mismatch between automated metrics and perceptual assessments.
A. Hyperparameters for pixel recursive super resolution model.
Table 2 lists the hyperparameters used for both datasets, with different B values for LSUN bedrooms and cropped CelebA faces.
- LSUN bedrooms use B = 10, while cropped CelebA faces use B = 6.
B. Samples from models trained on LSUN bedrooms
The LSUN bedroom sample comparison includes Input, Bicubic, ResNet L2, recursive-model samples at τ values, Truth, Nearest N., and GAN [11].
- The samples compare bicubic interpolation, ResNet L2, three τ settings, ground truth, nearest-neighbor output, and GAN [11].
C. Samples from models trained on CelebA faces
The CelebA face sample comparison includes Input, Bicubic, ResNet L2, recursive-model samples at τ values, Truth, Nearest N., and GAN.
- The samples compare bicubic interpolation, ResNet L2, three τ settings, ground truth, nearest-neighbor output, and GAN.
D. Samples images that performed best and worst in human ratings.
The human study presents images rated best and worst, with comparisons against ground truth shown below the images.
- The figure shows the best- and worst-rated images from the human study.
- Fractions below the images indicate how often a person chose that image over the ground truth.
- The displayed samples are labeled either Ours or Ground Truth.