Source-linked AI summary
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
Mehdi S. M. Sajjadi, Bernhard Schölkopf, Michael Hirsch
TL;DR
Single-image super-resolution is ambiguous, and pixel-wise objectives can produce blurry, texture-poor images despite strong PSNR. The paper combines feed-forward fully convolutional networks with adversarial, perceptual, and texture-transfer losses to create realistic textures. It reports state-of-the-art quantitative and qualitative results, while acknowledging imperfect pixel matching and residual artifacts.
Problem
SISR lacks a unique high-resolution solution, while MSE-based evaluation correlates poorly with human-perceived image quality and favors blurry reconstructions.
Method
The method uses fully convolutional neural networks with adversarial training, perceptual losses, and a texture-transfer loss to produce realistic textures.
Results
The approach achieves state-of-the-art results by both quantitative and qualitative measures, including PSNR and perceptual metrics.
Takeaways & Limitations
The trained model generates full-color images in a single forward pass at competitive speeds while improving perceptual texture quality.
Takeaways & Limitations
Outputs can look realistic without matching ground truth pixel-wise, and adversarial artifacts are reduced but not fully eliminated.
Abstract
from arXiv · showhide
Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack high-frequency textures and do not look natural despite yielding high PSNR values. We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixel-accurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.
1. Introduction
Single-image super-resolution is ill-posed, and high magnification makes missing high-frequency information especially damaging: conventional results become blurry and unnatural. The paper instead combines fully convolutional networks, adversarial training, perceptual losses, and texture synthesis to improve realism while retaining strong quantitative performance.
- SISR maps one low-resolution image to many possible high-resolution images, making high magnification ratios particularly challenging.
- Loss of high-frequency information makes textured regions blurry, overly smooth, and unnatural at large downsampling factors.
- Pixel-wise MSE correlates poorly with human perception, and its mean solution produces unnatural-looking reconstructions.
- The proposed method combines a fully convolutional network with texture synthesis, adversarial training, and perceptual losses to produce realistic textures at large magnification ratios.
- Training with suitable loss combinations achieves state-of-the-art results in both PSNR and perceptual metrics.
2. Related work
Related work spans interpolation, exemplar-based, dictionary-based, and neural approaches to SISR, with deep CNNs defining PSNR-based state of the art. Perceptual and adversarial methods target sharper images, while this work adds explicit local texture-statistics matching to improve realism and reduce artifacts.
- SISR methods include interpolation, exemplar-based, dictionary-based, and neural-network approaches.
- Deep CNNs form the PSNR-based state of the art, but MSE training tends to produce blurry outputs lacking high-frequency textures.
- Perceptual losses shift optimization toward higher-level feature space and can produce sharper results despite lower PSNR values.
- Texture-synthesis networks had been constrained to producing a single texture, while adversarial networks had only limited application to super-resolution.
- Unlike a similar perceptual-adversarial approach, this work explicitly matches local texture statistics to improve realism and reduce visually implausible artifacts.
3. Single image super-resolution
SISR estimates a high-resolution image from a low-resolution observation after non-injective downsampling, so the inverse problem admits many valid outputs. Euclidean-loss optimization favors the mean of possible textures, explaining high PSNR alongside blurry, texture-poor reconstructions.
- A high-resolution image is downsampled with a fixed scaling factor, and SISR seeks an approximate inverse that estimates it from the low-resolution image.
- Non-injective downsampling makes SISR highly ill-posed because many estimated high-resolution images can produce the same low-resolution input.
- Recent neural approaches approximate the inverse by minimizing Euclidean loss between the estimate and the ground-truth image.
- Euclidean-loss models can achieve excellent PSNR while producing blurry images lacking the original high-frequency textures.
- When high-frequency information is lost, Euclidean loss selects the average of possible images, whereas advanced losses yield more realistic but non-pixel-perfect reproductions.
4. Method
EnhanceNet combines fully convolutional processing, residual learning, perceptual and texture losses, and adversarial training to produce sharper, more realistic super-resolved images. The method addresses artifacts and instability through architectural choices and loss combinations evaluated on ImageNet examples.
- Architecture: The fully convolutional architecture supports arbitrary input sizes at a fixed scaling factor and requires only one network pass during inference.Convolutional layers enable a single model to process arbitrary-size inputs, while feed-forward operation improves inference efficiency.
- Architecture: Residual learning predicts the difference between the bicubic-upsampled input and the high-resolution output, stabilizing training and reducing color shifts.The residual formulation also avoids requiring the network to learn the identity function for the low-resolution input.
- Losses: Perceptual loss compares generated and ground-truth images in VGG-19 feature space rather than enforcing pixel-wise agreement.Using feature activations allows outputs that are not pixel-perfect but have similar low- and high-level representations.
- Losses: Texture matching computes a style-transfer loss patch-wise during training so generated images reproduce locally similar textures without requiring matching textures at inference.A 16×16-pixel patch size provided the best balance between faithful texture generation and overall perceptual quality.
- Adversarial training: Adversarial training uses a generator that maps low-resolution images to realistic high-resolution estimates and a discriminator that distinguishes real from generated images.In SISR, the generator input is an LR image rather than a random vector, and its output is a high-resolution estimate.
- Results: At 4× super-resolution, ENet-PAT produces more realistic textures and photorealistic images than models using simpler loss combinations, while some alternatives show blur, artifacts, or unnatural textures.ENet-P produces checkerboard artifacts, ENet-PA produces unnatural textures, and ENet-PAT is described as close to the original high-resolution images.
5. Evaluation
The evaluation compares loss combinations, reconstruction metrics, visual results, object-recognition performance, and human judgments. Texture-aware adversarial training improves perceptual realism, while PSNR favors pixel-accurate ENet-E reconstructions.
- 5.1. Effect of different losses: The evaluation tests the architecture with different combinations of loss functions before conducting quantitative and qualitative comparisons.The loss ablations are reported in Fig. 4 and Tables 2–3, with additional scaling-factor results in the supplementary material.
- 5.1. Effect of different losses: Perceptual loss alone produces slightly sharper results than ENet-E but introduces artifacts without adding realistic texture details.The loss provides no incentive to generate realistic textures by itself.
- 5.1. Effect of different losses: Texture loss reduces adversarial noise, halos, and sharpening artifacts while enabling ENet-PAT to generate locally meaningful textures, sometimes closely matching ground truth at 4x magnification.ENet-PA adds high-frequency details but can produce unpleasing noise in smooth regions and random edge sharpening.
- 5.2.1 Quantitative results by PSNR: ENet-E achieves the highest PSNR, whereas ENet-PAT produces more perceptually realistic images despite lower PSNR because its reconstructions are not pixel-accurate.The authors state that SSIM and IFC also fail to capture the perceptual quality of the results as well as their alternative evaluations.
- 5.2. Comparison with other approaches: ENet-PAT is the only compared model reported to produce significantly sharper images with realistic textures in the Set14 zebra comparison.The comparison highlights textures and repeating patterns absent from the low-resolution input, while ENet-E and prior methods retain blur in unsupported high-frequency regions.
- 5.2.1 Quantitative results by PSNR: ENet-E achieves state-of-the-art PSNR results on all evaluated datasets, while object-recognition performance is proposed as a complementary perceptual metric.The object-recognition evaluation applies restoration models to degraded images before feeding reconstructions into a pretrained recognition model.
6. Discussion, limitations and future work
EnhanceNet combines Euclidean or adversarial, perceptual, and texture-transfer losses to achieve strong quantitative and qualitative results, while retaining limitations from ill-posed super-resolution and adversarial artifacts.
- The proposed architecture achieves state-of-the-art results using either Euclidean loss or a combination of adversarial, perceptual, and texture-transfer losses.
- The model processes full-color images in a single forward pass at competitive speeds.
- ENet-PAT produces realistic images but does not match ground-truth images pixel by pixel.
- Adversarial training sometimes introduces output artifacts that texture loss greatly reduces but does not fully eliminate.
- On a BSD100 image, the model extends a visible pattern into smooth regions by hallucinating frequently occurring textures.
- Future work targets real-time high-resolution inference and possible video super-resolution through faster networks and temporal consistency.
— Supplementary —
The supplementary section is represented only by a brief reference to other sections of the main paper.
- The supplementary material refers readers to sections in the main paper.
4 Additional details on the method
The method uses patch-based texture matching within an adversarial convolutional architecture, with a 16×16 patch size selected as the best balance between texture fidelity and perceptual quality.
- A 16×16-pixel patch size provides the best balance between faithful texture generation and overall perceptual quality.
- Using 4×4 patches causes artifacts in textured regions, while 128×128 patches average different texture statistics and also produce artifacts.
- The adversarial discriminative network uses 3×3 convolutional filters and reduces inputs to 4×4 spatial dimensions before classification.
- The adversarial network uses leaky ReLU activations and strided convolutions instead of pooling layers.
5 Further evaluation of results
Further evaluation shows that EnhanceNet adds localized high-frequency detail and realistic textures, but perceptual quality is not fully captured by standard metrics and training choices affect stability.
- ENet-PAT generates fine high-frequency textures in textured regions while preserving smooth areas such as skies and house fronts.
- Combining adversarial and texture losses with Euclidean loss is less stable than using perceptual loss, with perceptual quality oscillating during training.
- EnhanceNet is visually sharper than Bruna et al. and avoids jagged edges in the reported comparison.
- At 4x super-resolution, ENet-PAT produces plausible sharper textures comparable in quality to current state-of-the-art models at 2x, while running at 9/18ms per image on Set5/Set14.
- PSNR, SSIM, and IFC do not correctly capture the perceptual quality of ENet-PAT’s results.
- A user survey evaluated perceptual quality by asking subjects to select the image most similar to a target, with each subject viewing up to 30 images.
- Replacing perceptual loss with Euclidean loss produces sharp but jagged edges, overly smooth textures, and substantially harder training.
6 Specialized training datasets
Specialized training data strongly affects ENet-PAT’s ability to synthesize realistic details, especially for faces. Face-focused training improves facial results, while perceptual evaluations favor ENet-PAT overall despite failures on some images.
- Face-specific training: ENet-PAT trained on MSCOCO struggles to reproduce realistic faces at high scaling factors, and many subjects preferred ENet-E’s blurrier output.Human perception is highly sensitive to small changes in facial appearance.
- Face-specific training: Training ENet-PAT on CelebA with identical hyperparameters yields significantly better performance on face images.The comparison attributes the improvement to specialized training data rather than retuned loss parameters.
- Qualitative comparisons: At 4x super-resolution, ENet-PAT produces sharper images with realistic textures despite having significantly less information than a prior state-of-the-art method evaluated at 2x.The comparison uses 93.75% missing pixels for ENet-PAT versus 75% for the 2x method.
- Qualitative comparisons: At 2x super-resolution on a Set5 butterfly image, ENet-PAT is sharper, recreates small details better, and fully reconstructs a central vertical white line.Image quality increases from RAISR to ENet-E and then ENet-PAT in the shown comparison.
- Metric comparison: ENet-PAT’s PSNR, SSIM, and IFC results do not capture the perceptual quality of its outputs.The paper specifically notes low SSIM values and IFC scores that roughly follow PSNR.
- Perceptual evaluation: 91.0% of survey selections favored ENet-PAT over ENet-E across 843 votes from 49 responses.The survey measured which output looked more similar to a target image.