Source-linked AI summary
Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination
Jae Woong Soh, Gu Yong Park, Junho Jo, Nam Ik Cho
TL;DR
Distortion-focused SISR often misses realistic fine detail, while methods that enhance perceptual quality can introduce unnatural artifacts. This paper models the natural HR manifold and constrains super-resolved outputs to it. The authors report more natural results with fewer artifacts than other perception-oriented methods.
Problem
Distortion-oriented SISR often fails to recover realistic fine textures and details, while perception-oriented methods can generate unnatural artifacts.
Source: §1 · p. 2Method
The method uses a CNN-based natural manifold discriminator and explicit HR-space modeling to constrain outputs to the natural manifold.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors expect deeper, heavier generators and a better NMD classifier to bring more naturalness and realistic details.
Source: §9 · p. 8
Abstract
from arXiv · showhide
Recently, many convolutional neural networks for single image super-resolution (SISR) have been proposed, which focus on reconstructing the high-resolution images in terms of objective distortion measures. However, the networks trained with objective loss functions generally fail to reconstruct the realistic fine textures and details that are essential for better perceptual quality. Recovering the realistic details remains a challenging problem, and only a few works have been proposed which aim at increasing the perceptual quality by generating enhanced textures. However, the generated fake details often make undesirable artifacts and the overall image looks somewhat unnatural. Therefore, in this paper, we present a new approach to reconstructing realistic super-resolved images with high perceptual quality, while maintaining the naturalness of the result. In particular, we focus on the domain prior properties of SISR problem. Specifically, we define the naturalness prior in the low-level domain and constrain the output image in the natural manifold, which eventually generates more natural and realistic images. Our results show better naturalness compared to the recent super-resolution algorithms including perception-oriented ones.
1. Introduction
Distortion-oriented SISR often blurs fine detail, while perception-oriented methods can introduce unnatural artifacts. The paper proposes modeling the natural HR manifold explicitly to produce more natural, realistic super-resolved images.
1. Introduction
- Distortion-oriented CNNs produce blurry results without fine details because their outputs average possible HR images.SISR is ill-posed: multiple HR images can correspond to one LR image.Source: §1 · pp. 1–2
- Perception-oriented methods seek better perceptual quality, but can generate unpleasant, unnatural artifacts alongside reconstructed details.The introduction cites perceptual loss, SRGAN, and EnhanceNet as examples.Source: §1 · p. 2
- The proposed NatSR constrains outputs to the natural manifold using a low-level domain prior, aiming for realistic textures and natural details with high perceptual quality.The paper reports fewer artifacts than other perception-oriented methods.Source: §1 · p. 2
- The paper models SISR and the desirable HR space, then divides that space into natural and unnatural manifolds.Its natural manifold discriminator is a CNN-based component.Source: §1 · p. 2
- The authors also introduce FRSR, a distortion-oriented CNN with fractal residual learning that achieves results comparable to recent CNNs.The paper separately proposes NatSR as its perception-oriented method.Source: §1 · p. 2
2. Related Work
Earlier SISR methods used image and domain priors, while CNN approaches implicitly inherit priors from training data and typically optimize distortion. Perception-oriented work addresses missing detail, but also raises challenges around realism and semantic conditioning.
2.1. Single Image Super-Resolution
- CNN-based SISR methods implicitly adopt image and domain priors from training data and are generally discriminative, distortion-oriented methods aimed at higher PSNR.Earlier approaches explored interpolation, edge and gradient features, non-local similarity, and dictionary-learning sparse coding.Source: §2.1 · p. 2
2.2. Perception Oriented Super-Resolution
- Distortion-oriented results often lack high-frequency detail; perception-oriented methods instead use feature-space or texture losses and generate a candidate from the multi-modal HR space.The cited examples include SRGAN [22], EnhanceNet [29], and SFT-GAN [37].Source: §2.2 · p. 3
- SFT-GAN [37] conditions the feature space on target-pixel semantic categories to restrict the generated pixels.The frequency-domain illustration introduces the LR-HR relationship and SISR.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Modeling the SISR
SISR models an LR image as a low-pass-filtered, downsampled HR image, but many HR images can correspond to the same LR image. This one-to-many mapping makes recovery an ill-posed problem.
3. Modeling the SISR
- An LR image is defined as the low-pass-filtered and downsampled version of an HR image.The relation is shown in the frequency domain, with possible HR images corresponding to the same LR image.Source: §3 · p. 3
- Different HR images can correspond to one LR image, making SISR an ill-posed one-to-many problem.The possible HR images can share low-frequency components while differing in their high-frequency parts.Source: §3 · p. 3
- SISR is usually modeled as finding the conditional likelihood p(IHR|ILR), and its one-to-many nature favors a generative model over a discriminative one.Source: §3 · p. 3
4. Natural Manifold Discrimination
The section partitions the possible HR space into blurry, noisy, and natural regions, then trains a discriminator to distinguish natural images from the two unnatural regions.
4.1. Designing Natural Manifold
- The possible HR space V is divided into blurry A, noisy B, and natural N sets; distortion-oriented outputs average candidates in V, while refined training sets exclude low-gradient patches.The excluded patches implicitly constrain candidate outputs away from blurry images.Source: §4.1 · p. 3
- Set A models blurry images as a convex combination of interpolated LR and original HR, with α controlling the boundary between A and N.The interpolation uses the same low-pass filter and upsampling with zero insertion.Source: §4.1 · pp. 3–4
- Set B models noisy images by adding high-frequency noise n to HR, with standard deviation σ.Source: §4.1 · p. 4
- Noise is injected in the frequency domain using 2D-DCT; the ×4 bicubic-filter coefficients are shown, and experiments use 8 × 8 DCT with noise in the last row and column.Source: §4.1 · p. 4, Figure 4 · p. 4
4.2. Natural Manifold Discriminator
- The natural manifold discriminator is a CNN classifier trained to distinguish N from A ∪ B; its VGG-style architecture ends with a convolution layer and global average pooling instead of fully connected layers.Training uses sigmoid binary cross entropy.Source: §4.2 · p. 4, Figure 5 · p. 4
- Training starts at α = 0.5 and σ = 0.1, updates them after validation accuracy exceeds 95%, and ends at α = 0.8 and σ = 0.0044.Source: §4.2 · p. 4
5. Natural and Realistic Super-Resolution
NatSR combines fractal residual learning with a residual dense block and trains the generator with reconstruction, naturalness, and adversarial losses. The naturalness loss uses a pretrained natural manifold discriminator, while the adversarial training uses RaGAN.
5.1. Network Architecture
- NatSR uses fractal residual learning for mid- and long-path skips and a residual dense block for short-path connections.The network also adopts residual scaling in the residual dense block.Source: §5.1 · p. 5, Figure 6 · p. 6
5.2.1 Reconstruction Loss
- The reconstruction loss is mean absolute error between the ground-truth and super-resolved images.The authors do not use perceptual loss because their experiments found it caused undesirable artifacts; they instead use NMD to boost high-frequency details.Source: §5.2.1 · p. 5
5.2.2 Naturalness Loss
- The naturalness loss uses a pretrained NMD and log-scale penalty to push its output for the super-resolved image toward 1.The log-scale is used to boost gradients.Source: §5.2.2 · p. 5
5.2.3 Adversarial Loss
- NatSR employs RaGAN adversarial training, whose discriminator compares real and generated images relative to their respective averages.The discriminator's stated motivation is to measure whether an image is closer to real HR images than generated SR images on average.Source: §5.2.3 · p. 5
5.2.4 Overall Loss
- The overall NatSR training objective is a weighted sum of the defined loss terms.The distortion-oriented FRSR baseline uses reconstruction loss alone, whereas NatSR uses the full loss with λ1 = 1 and λ2 = λ3 = 10^-3.Source: §5.2.4 · p. 5
6. Discussion and Analysis
The analysis reports that NatSR receives NMD scores close to the natural-HR target and generally satisfies the modeled LR-HR consistency condition. SRGAN, by contrast, is reported at about 36 dB for LR consistency.
6.1. Effectiveness of Proposed Discriminator
- NatSR's NMD scores are close to 1, while distortion-oriented methods score almost 0 and HR and bicubic score 1 and 0, respectively.The authors interpret the results as supporting their natural-manifold model and NMD's discrimination of natural and unnatural images.Source: §6.1 · p. 6, Table 1 · p. 6
6.2. Study on the Plausibility of SR Images
- On Set5, NatSR mostly lies in the modeled feasible set V, while SRGAN reaches about 36 dB RGB-PSNR for the downsampled SR-to-LR comparison.The study uses this comparison to assess whether downsampling a super-resolved image returns an image similar to the LR input.Source: §6.2 · p. 6, Table 2 · p. 7
7. Experimental Results
The experiments train and test ×4 super-resolution models on standard datasets, using distortion metrics for distortion-oriented models and no-reference metrics for perceptual quality. FRSR is comparable to or better than other distortion-oriented methods, while NatSR is competitive but not best on both no-reference metrics.
7.1. Implementation details
- Training uses DIV2K and tests on Set5, Set14, BSD100, and Urban100, with scaling factor ×4.The training set comprises 800 images, with 100 validation and 100 test images.Source: §7.1 · p. 7
7.2. Evaluation Metrics and Comparisons
- The authors use PSNR and SSIM for distortion-oriented models and NIQE and NQSR as rough references for perceptual quality.They note that it is questionable whether NR-IQA measures perfectly reflect human perceptual quality.Source: §7.2 · p. 7
7.3. FR-IQA Results
- FRSR shows comparable or better average PSNR/SSIM results than other distortion-oriented methods, while EDSR has the best result.The authors describe FRSR as competent when considering the number of parameters.Source: §7.3 · p. 7
7.4. NR-IQA Results
- NatSR is not best on either BSD100 no-reference metric but is comparable to other perception-oriented methods and original HR.SRGAN scores best on NIQE and NQSR; NatSR is second on NIQE and ranks below SRGAN and ENet on NQSR.Source: §7.4 · p. 7, Figure 8 · p. 7
8. Subjective Assessments
The subjective assessment includes a MOS test on the DIV2K validation set and visual comparisons across super-resolution methods. The reported visual examples contrast blurry distortion-oriented outputs with more detailed perceptual outputs, while noting unnatural or insufficiently natural details in some methods.
8.1. Mean Opinion Score (MOS)
- The MOS test uses the DIV2K validation set and evaluates SFT-GAN with a semantic segmentation mask for fair comparison.Implementation details are deferred to supplementary material.Source: §8.1 · p. 7
8.2. Visual Comparisons
- Visual comparisons show blurry distortion-oriented outputs and better detail from perception-oriented methods, but ENet looks cartoony and SFT-GAN lacks natural building details.The authors also report that NatSR has the least distortion among perception-oriented methods in Figure 1.Source: §8.2 · p. 8, Figure 9 · p. 8
9. Conclusion
The proposed SISR approach uses a CNN-based natural manifold discriminator to narrow the target space and hallucinate natural, realistic textures. Experiments report gains for distortion-oriented FRSR and perceptually plausible results for perception-oriented NatSR.
9. Conclusion
- The CNN-based natural manifold discriminator narrows the target space to the natural manifold.The SR generator uses residual dense blocks and fractal residual learning, with a loss that supports distortion-oriented or perception-oriented operation.Source: §9 · p. 8
- FRSR shows considerable gains over models with similar parameters, while NatSR produces perceptually plausible results compared with other methods.The authors expect deeper, heavier networks and a better classifier to bring more naturalness and realistic details.Source: §9 · p. 8