Source-linked AI summary
Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution
Jie Liang, Hui Zeng, Lei Zhang
TL;DR
GAN-based SISR can recover rich details, but unstable adversarial training often adds perceptually unpleasant artifacts. The paper proposes locally discriminative learning to separate artifacts from realistic details during training, and reports improved reconstruction accuracy and perceptual quality across synthetic and real-world datasets.
Problem
GAN-based SISR can generate rich details, but unstable adversarial training often introduces perceptually unpleasant artifacts alongside them.
Method
LDL discriminates GAN-generated artifacts from realistic details using local statistics and generates an artifact map to regularize adversarial training.
Results
LDL outperforms existing GAN-SR methods quantitatively and qualitatively, improving reconstruction accuracy and perceptual quality on synthetic and real-world tasks.
Takeaways & Limitations
LDL is a simple plug-in framework that suppresses visual artifacts while preserving rich realistic details across different GAN-SR models and tasks.
Takeaways & Limitations
LDL remains limited in discriminating artifacts in heavily aliased regions, with artifacts persisting around dense windows.
Abstract
from arXiv · showhide
Single image super-resolution (SISR) with generative adversarial networks (GAN) has recently attracted increasing attention due to its potentials to generate rich details. However, the training of GAN is unstable, and it often introduces many perceptually unpleasant artifacts along with the generated details. In this paper, we demonstrate that it is possible to train a GAN-based SISR model which can stably generate perceptually realistic details while inhibiting visual artifacts. Based on the observation that the local statistics (e.g., residual variance) of artifact areas are often different from the areas of perceptually friendly details, we develop a framework to discriminate between GAN-generated artifacts and realistic details, and consequently generate an artifact map to regularize and stabilize the model training process. Our proposed locally discriminative learning (LDL) method is simple yet effective, which can be easily plugged in off-the-shelf SISR methods and boost their performance. Experiments demonstrate that LDL outperforms the state-of-the-art GAN based SISR methods, achieving not only higher reconstruction accuracy but also superior perceptual quality on both synthetic and real-world datasets. Codes and models are available at https://github.com/csjliang/LDL.
1. Introduction
GAN-based SISR can generate sharp, rich details but unstable adversarial training often introduces perceptually unpleasant artifacts. LDL addresses this conflict by discriminating artifacts from realistic details and regularizing training, improving both perceptual quality and reconstruction accuracy.
- Motivation: Pixel-wise losses can achieve high PSNR and SSIM but generally fail to produce rich image details.These limitations motivate perceptual-quality-oriented SISR methods.
- Motivation: GAN-based SISR offers the potential to recover sharp images with rich details, but unstable adversarial training often introduces unpleasant visual artifacts.The central challenge is preserving realistic details while suppressing artifacts.
- Region analysis: Fine-scale type B and type C regions are difficult to restore because signal aliasing removes much of their high-frequency information.Type B outputs can remain perceptually realistic despite pixel differences, whereas type C outputs can contain overshoot pixels and distorted structures.
- Proposed approach: LDL uses local residual variance to distinguish unpleasant artifacts from realistic details and constructs a pixel-wise artifact-probability map.The refined map is used to penalize artifacts without affecting realistic details.
- Results: LDL can be plugged into off-the-shelf GAN-SR methods and improves them quantitatively and qualitatively on synthetic and real-world SISR tasks.The paper reports clear improvements against state-of-the-art methods.
2. Related work
SISR methods balance signal fidelity against perceptual quality: pixel-wise and structural losses tend toward faithful but blurry outputs, while GAN-based methods recover details but can introduce artifacts. LDL regularizes adversarial training by explicitly separating artifacts from realistic details.
- Overview: SISR methods are broadly divided into signal fidelity-oriented and perceptual quality-oriented approaches according to their losses and training strategies.This distinction organizes the related work discussed by the paper.
- Signal fidelity-oriented methods: Pixel-wise losses often produce blurry averages of possible HR solutions, while SSIM preserves local structures but struggles to reproduce fine details.RCAN is cited as achieving strong reconstruction accuracy through deep networks, residual connections, and channel attention.
- Perceptual quality-oriented methods: Perceptual-quality methods use feature-space losses or GANs to improve the visual detail of SISR outputs.GANs generate desired images by discriminating between image distributions.
- GAN-SR limitations: Existing GAN-SR methods can produce unpleasant artifacts because adversarial training is unstable.The paper identifies this as a key issue across GAN-SR work.
- Trade-off and proposed direction: SISR faces a perception-distortion trade-off, and suppressing artifacts can limit detail generation under existing training strategies.LDL addresses this dilemma by explicitly discriminating artifacts from realistic details during adversarial training.
3. Methodology
LDL addresses unstable GAN-SR training by distinguishing artifacts from realistic details using residual statistics, then refining the resulting penalty map with temporal stabilization. This improves training stability, especially for challenging fine-scale regions.
- GAN-SR induced visual artifacts: GAN-SR combines reconstruction, perceptual, and adversarial losses, but adversarial updates can move solutions along many directions and generate artifacts with details.The resulting optimization is unstable because one low-resolution input may correspond to many plausible high-resolution images.
- GAN-SR induced visual artifacts: Type C regions with fine-scale regular structures show the greatest optimization instability, whereas type A regions are stable and type B regions are more uncertain.The comparison uses MAD between intermediate outputs separated by 5000 iterations.
- Discriminating artifacts from realistic details: LDL estimates an artifact probability map from the local variance of residuals between the SISR output and high-resolution ground truth.The primary map uses a local window size n = 7; residual distributions in texture-like type B regions are more random than in artifact-prone type C regions.
- Discriminating artifacts from realistic details: A patch-level variance σ scales the primary map, reducing over-penalization in smooth and texture regions while retaining artifact responses in type C regions.The reported σ values for type A, B, and C are 0.25, 0.39, and 0.67, respectively.
- Loss and learning strategy: LDL uses an exponential moving average model to stabilize training and refines the map by penalizing only pixels where the current model's residual is no smaller than the averaged model's residual.This removes locations corresponding to fine textures and desirable edges from the refined penalty map.
- Loss and learning strategy: With the same RRDB backbone, LDL produces much smaller MAD and MAD variation than ESRGAN, particularly for type B and type C patches.The comparison indicates improved stability in model learning.
4. Experimental results
Across three backbone families and synthetic and real-world settings, LDL improves perceptual quality and reconstruction accuracy while reducing artifacts and preserving realistic details.
- Experiment setup: LDL is evaluated with SRResNet, RRDB, and SwinIR backbones against representative GAN-SR methods on synthetic and real-world SISR tasks.Experiments use six benchmarks and assess perceptual quality with LPIPS, DISTS, and FID, alongside PSNR and SSIM reconstruction metrics.
- Quantitative comparison: LDL improves perceptual quality and reconstruction accuracy on most benchmarks under all three backbones.The gains span LPIPS, DISTS, FID, PSNR, and SSIM rather than being confined to one evaluation axis.
- Quantitative comparison: PSNR +0.3 ∼0.5dB and SSIM +0.01 over the second best method are reported for SRResNet+LDL across all benchmarks.SRResNet+LDL also outperforms SFTGAN and SRGAN on most perceptual-quality metrics.
- Quantitative comparison: LPIPS improves from 0.1099 to 0.1011 on DIV2K validation when RRDB-based LDL is compared with SPSR.Against USRGAN, LDL also achieves better reconstruction accuracy on all benchmarks while improving perceptual indexes.
- Qualitative comparison: Visual comparisons show fewer artifacts and more high-fidelity details in fine-scale aliasing structures and regular patterns than competing RRDB-based methods.Examples include lines on windows and grids on bridges.
- Real-world SISR: On real-world SISR, LDL suppresses artifacts and encourages sharp details in dense windows while improving fine details in twigs.RealESRGAN+LDL changes only the proposed Lartif loss relative to RealESRGAN, and qualitative comparisons are made without ground truth.
- Ablation study: Ablation results show successive gains from the artifact map, globally scaled map, refined map, and EMA model during testing.The artifact map improves both perceptual quality and reconstruction accuracy over the baseline, while later components further improve performance.
- Limitations: LDL remains limited in discriminating artifacts in heavily aliased regions, with residual artifacts around dense windows in an example.The authors identify more effective artifact-discrimination and detail-generation designs as an open possibility.
5. Conclusion
The paper proposes locally discriminative learning to distinguish visual artifacts from realistic details during GAN-SR training. Across synthetic and real-world tasks, LDL is reported to improve GAN-SR results quantitatively and qualitatively.
- 5. Conclusion: LDL generates an artifact map to penalize visual artifacts without sacrificing realistic details and can be plugged into different off-the-shelf GAN-SR models.The strategy is applied to both synthetic and real-world SISR tasks.
- 5. Conclusion: Extensive experiments on widely used datasets show that LDL outperforms existing GAN-SR methods quantitatively and qualitatively.