PaperlayerView exact revision on arXiv

Source-linked AI summary

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi

arXiv:1609.04802v5cs.CVstat.ML

TL;DR

At large upscaling factors, pixel-based super-resolution objectives leave texture detail and perceptual fidelity unresolved. SRGAN combines adversarial and perceptual content losses, and MOS testing found its 4× reconstructions more photo-realistic than state-of-the-art reference methods.

  • Problem

    At high upscaling factors, SR reconstruction typically lacks texture detail, while MSE and PSNR capture perceptually relevant differences only weakly.

    Source: §1 · p. 1

  • Method

    SRGAN combines a discriminator-based adversarial loss with a perceptual content loss based on VGG feature maps to generate super-resolved images.

    Source: §1 · p. 2, §2.2.1 · p. 5

  • Results

    MOS testing found SRGAN reconstructions at 4× considerably more photo-realistic than state-of-the-art reference methods.

    Source: §1.2 · p. 4, §5 · p. 9

  • Takeaways & Limitations

    PSNR and SSIM fail to assess image quality accurately with respect to the human visual system, supporting MOS testing for perceptual performance.

    Source: §4 · p. 7

    Hallucinating finer detail may be less suited to medical or surveillance applications, and perceptually convincing text or structured-scene reconstruction remains challenging.

    Source: §4 · p. 8

Abstract

from arXiv · show

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

1. Introduction

The introduction frames high-factor SISR as a perceptual-detail problem that pixelwise MSE and PSNR do not capture, then presents SRGAN’s adversarial, VGG-feature loss and deep residual approach alongside SRResNet’s distortion-metric results.

1. Introduction

  • At high upscaling factors, pixelwise MSE and PSNR can favor smooth reconstructions that lack texture and are not photo-realistic.The introduction notes that PSNR does not necessarily track perceptual quality.Source: §1 · pp. 1–2

1.1.1 Image super-resolution

  • SISR approaches range from fast but overly smooth filtering and example-based methods to CNNs, including models using perceptual losses [33] [5].CNN approaches also learned upscaling filters directly to improve accuracy and speed [11] [48] [57].Source: §1.1.1 · pp. 2–3

1.1.2 Design of convolutional neural networks

  • Deeper CNNs can model more complex mappings but are harder to train; batch normalization, residual blocks, and skip connections support training, while learned upscaling filters improve SISR accuracy and speed.Skip connections avoid requiring the network to represent the identity mapping with convolutional kernels.Source: §1.1.2 · p. 3

1.1.3 Loss functions

  • For recovering lost high-frequency texture, minimizing pixel-wise MSE averages plausible solutions into typically overly-smooth reconstructions with poor perceptual quality [42] [33] [13] [5].Dosovitskiy and Brox [13] combine neural-network feature-space Euclidean losses with adversarial training; Johnson et al. [33] and Bruna et al. [5] use VGG feature maps instead of pixel-wise error, obtaining perceptually more convincing results for super-resolution and artistic style-transfer.Source: §1.1.3 · p. 3

1.2. Contribution

  • SRGAN combines a very deep ResNet with GAN training and a perceptual loss based on VGG feature maps, which are more invariant to pixel-space changes [38].Its discriminator encourages solutions that are perceptually hard to distinguish from HR reference images.Source: §1 · p. 2, §1.2 · p. 4
  • SRResNet sets a new 4× SR state of the art by PSNR and SSIM, while SRGAN achieves a large-margin MOS state of the art for photo-realistic 4× SR on three public benchmarks.SRResNet uses a 16-block ResNet optimized for MSE; SRGAN uses a new perceptual loss with VGG feature-map content loss.Source: §1.2 · p. 4

2. Method

SRGAN estimates high-resolution images from low-resolution inputs using a CNN generator trained with a perceptual objective combining content and adversarial losses. During training, low-resolution inputs are made by Gaussian-filtering and downsampling their high-resolution counterparts.

2. Method

  • Training pairs comprise Gaussian-filtered, downsampled low-resolution images and their high-resolution counterparts, which are available only during training.A feed-forward CNN generator maps low-resolution inputs to estimated high-resolution images and is optimized with an SR-specific loss.Source: §2 · p. 4

2.1. Adversarial network architecture

  • The generator uses residual blocks with 3×3 convolutions, batch normalization, and PReLU, then upsamples with two trained sub-pixel convolution layers.An alternating discriminator distinguishes generated super-resolved images from real images; its network has eight convolutional layers, followed by two dense layers and a sigmoid output.Source: §2.1 · p. 4

2.2. Perceptual loss function

  • The perceptual loss is a weighted sum of content and adversarial losses, designed to assess perceptually relevant characteristics of reconstructed images.The discriminator-based adversarial formulation encourages generated solutions to resemble real images and lie on the natural-image manifold, unlike pixel-wise error minimization.Source: §2.1 · p. 4, §2.2 · p. 5

2.2.1 Content loss

  • MSE optimization can yield high PSNR yet smooth, perceptually unsatisfying textures; SRGAN instead measures Euclidean distance between reconstructed and reference images’ activations in pretrained VGG19.The VGG loss uses feature maps from ReLU activations after convolutional layers and before max-pooling layers.Source: §2.2.1 · p. 5

2.2.2 Adversarial loss

  • For better gradient behavior, the generator adversarial loss minimizes −log DθD(GθG(ILR)), where the discriminator output is the probability that the reconstruction is a natural high-resolution image.This generative loss encourages the generator to fool the discriminator and favor solutions on the natural-image manifold.Source: §2.2.2 · pp. 5–6

3. Experiments

The experiments compare 4× super-resolution methods using benchmark similarity measures and human ratings. SRResNet leads on PSNR/SSIM, while SRGAN achieves the strongest reported perceptual results on BSD100 and SRGAN-VGG54 excels in MOS on Set14.

3.1. Data and similarity measures

  • Evaluation used Set5, Set14, and BSD100, with 4× upscaling and PSNR/SSIM computed on the cropped luminance channel.The protocol removed a 4-pixel border from center-cropped images; reference outputs came from published online materials for methods including SRCNN, SelfExSR, and DRCN [34].Source: §3.1 · p. 6

3.2. Training details and parameters

  • Training used 350,000 ImageNet images, bicubic 4× downsampling, and MSE-trained SRResNet initialization for GAN generator training.Networks were trained on an NVIDIA Tesla M40; SRResNet used 10^6 updates, and SRGAN variants used two 10^5-update stages at learning rates 10^-4 and 10^-5. The generator had 16 residual blocks and supported arbitrary image sizes.Source: §3.2 · p. 6

3.3. Mean opinion score (MOS) testing

  • Twenty-six raters scored super-resolved images from 12 methods on a 1–5 scale, rating 1,128 instances each across Set5, Set14, and BSD100.Raters were calibrated against nearest-neighbor and original high-resolution images; a pilot found good test-retest reliability and no significant differences for repeated images.Source: §3.3 · p. 6

3.4. Investigation of content loss

  • On Set14, SRGAN-VGG54 significantly outperformed other SRGAN and SRResNet variants in MOS; no loss was significantly best on Set5.MSE yielded the highest PSNR but produced smoother, less convincing results; the authors attributed some SRGAN-MSE artifacts to competition with adversarial loss and observed a trend toward better texture detail with φ5,4 than φ2,2. The ablation compared SRGAN-MSE, SRGAN-VGG22 using lower-level features [68], and SRGAN-VGG54 using deeper, higher-level features [68] [65] [40].Source: §3.4 · p. 7, Table 1 · p. 7

3.5. Performance of the final networks

  • SRResNet set a new PSNR/SSIM state of the art across three benchmarks, while SRGAN beat all reference methods by a large MOS margin on BSD100.The MOS differences on BSD100 were highly significant except for SRCNN versus SelfExSR.Source: §3.5 · p. 7

4. Discussion and future work

The discussion confirms SRGAN’s superior perceptual performance in MOS testing and argues that PSNR and SSIM do not reliably reflect human-perceived image quality. It highlights tradeoffs in content loss, network depth, efficiency, and application-specific suitability.

4. Discussion and future work

  • MOS testing confirmed SRGAN’s superior perceptual performance, while PSNR and SSIM fail to assess image quality accurately with respect to human vision [56].The work prioritized perceptual quality over computational efficiency and was not optimized for real-time video SR, unlike Shi et al. [48].Source: §4 · p. 7
  • Deeper networks were beneficial, unlike the finding of Dong et al., and networks with B > 16 could further improve SRResNet at the cost of longer training and testing times.The authors speculate that the ResNet design contributes to the performance of deeper networks.Source: §4 · pp. 7–8
  • Deeper SRGAN variants became increasingly difficult to train because of high-frequency artifacts.This limitation qualifies the potential performance gains from increasing network depth.Source: §4 · p. 8
  • The lSR VGG/5.4 content loss yielded the most perceptually convincing results, which the authors attribute to deeper layers’ more abstract features [68] [65] [40].They speculate that deeper feature maps represent content while the adversarial loss focuses on texture; Figure 6 compares the reconstruction variants and reference images.Source: §4 · p. 8, Figure 6 · p. 8
  • The ideal loss depends on the application: hallucinated detail may be unsuitable for medical or surveillance imagery, while text and structured-scene reconstruction remains challenging.Future work includes content losses that describe spatial content while being more invariant to pixel-space changes.Source: §4 · p. 8

5. Conclusion

The conclusion presents SRResNet as a PSNR state-of-the-art model on public benchmarks and SRGAN as a more photo-realistic approach for 4× upscaling, supported by MOS testing.

5. Conclusion

  • SRResNet sets a new state of the art on public benchmark datasets when evaluated using PSNR.The conclusion also highlights limitations of PSNR-focused super-resolution.Source: §5 · p. 8
  • MOS testing found SRGAN reconstructions at 4× upscaling considerably more photo-realistic than those from state-of-the-art reference methods.SRGAN augments the content loss with an adversarial loss trained through a GAN.Source: §5 · p. 9

A. Supplementary Material

The supplementary material provides 4× reconstructions from the paper’s methods for zoomed-in comparison and download.

A. Supplementary Material

  • The original low- and high-resolution images and 4× reconstructions from bicubic, SRResNet, and SRGAN variants are available for download and best compared zoomed in.The listed variants are SRResNet-MSE, SRResNet-VGG22, SRGAN-MSE, SRGAN-VGG22, and SRGAN-VGG54.Source: A. Supplementary Material · p. 11

A.1. Performance (PSNR/time) vs. network depth

For 4× SR on BSD100, adding a skip-connection substantially improves PSNR without substantially affecting inference time. The authors select 16 residual blocks as an accuracy–speed tradeoff, although deeper networks continue to offer benefits.

A.1. Performance (PSNR/time) vs. network depth

  • The evaluation uses 16 residual blocks with a skip-connection as a tradeoff between accuracy and speed, including training time.Accuracy gains slowly saturate beyond 16 blocks, but deeper networks still provide a clear benefit.Source: §A.1 · p. 12
  • A skip-connection yields substantial PSNR gains with no substantial inference-time impact.The comparison uses the architecture described in Figure 4 and measures PSNR on BSD100 for 4× SR.Source: Figure 7 · p. 12, §A.1 · p. 12
  • Inference time grows approximately linearly with the number of residual blocks, regardless of whether a skip-connection is used.Time was averaged over 100 reconstructions of a 64×64 low-resolution image on an NVIDIA M40 GPU.Source: Figure 7 · p. 12, §A.1 · p. 12

A.2. Evolution of Generator during SRGAN training

Figure 8 documents the SRGAN generator’s evolution during training for 4× upscaling, with an SRResNet initialization and a staged learning-rate schedule.

A.2. Evolution of Generator during SRGAN training

  • The generator was initialized with SRResNet weights for the 4× upscaling training shown in Figure 8.Source: Figure 8 · p. 13
  • Training used a 10^-4 learning rate for the first 100k iterations, then 10^-5 for another 100k.Source: Figure 8 · p. 13

A.3. Mean opinion score (MOS) testing

MOS testing found SRGAN’s perceptual advantage on Set14 and BSD100, while Set5 results were inconclusive. The study used blinded, randomized ratings from calibrated human raters.

A.3. Mean opinion score (MOS) testing

  • Twenty-six calibrated, blinded raters scored 4× reconstructions from Set5, Set14, and BSD100 on a 1–5 scale, producing 29,328 randomized ratings.Raters were calibrated so nearest-neighbor reconstructions should score 1 and original high-resolution images 5.Source: §A.3 · p. 14
  • On Set14 and especially BSD100, SRGAN significantly outperformed compared state-of-the-art methods in MOS, with scores closer to original high-resolution images; Set5 results were inconclusive.Figure 9 summarizes MOS-score distributions, and Figure 10 shows average ordinal ranks.Source: Figure 9 · p. 14, Figure 10 · p. 14, §A.3 · p. 14

A.4. Set5 - Visual Results

Figure 11 presents Set5 visual results for 4× upscaling with bicubic interpolation, SRResNet, and SRGAN.

A.4. Set5 - Visual Results

  • Figure 11 shows Set5 results for 4× upscaling using bicubic interpolation, SRResNet, and SRGAN.Source: Figure 11 · p. 15

A.5. Set14 - Visual Results · A.6. BSD100 (five random samples) - Visual Results

The visual-results sections present 4× upscaling comparisons for Set14 and five random BSD100 samples using bicubic interpolation, SRResNet, and SRGAN alongside the original images.

A.5. Set14 - Visual Results

  • The Set14 visual-result labels identify bicubic, SRResNet, SRGAN, and original images.Source: §A.5 · pp. 16–18
  • Figure 12 presents a Set14 visual comparison of bicubic interpolation, SRResNet, SRGAN, and original images at 4× upscaling.Source: Figure 12 · p. 16
  • Figure 13 presents a Set14 visual comparison of bicubic interpolation, SRResNet, SRGAN, and original images at 4× upscaling.Source: Figure 13 · p. 17
  • Figure 14 presents a Set14 visual comparison of bicubic interpolation, SRResNet, SRGAN, and original images at 4× upscaling.Source: Figure 14 · p. 18

A.6. BSD100 (five random samples) - Visual Results

  • The BSD100 visual-result labels identify bicubic, SRResNet, SRGAN, and original images.Source: §A.6 · p. 19
  • Figure 15 presents five random BSD100 samples compared using bicubic interpolation, SRResNet, and SRGAN at 4× upscaling.Source: Figure 15 · p. 19

Related papers

Loading 1609.04802v5…