PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Dynamic Convolutional Network for Super-Resolution with Variational Degradations

Yu-Syuan Xu, Shou-Yao Roy Tseng, Yu Tseng, Hsien-Kai Kuo, Yi-Min Tsai

arXiv:2004.06965v1eess.IVcs.CV

TL;DR

Most SISR methods assume fixed degradation combinations or require separate networks for different variations, limiting their fit to variable degradation settings. UDVD uses dynamic convolutions in a unified network, and the paper reports favorable or comparable results on synthetic and real images. Its evaluation adopts a non-blind setting with given degradation parameters.

  • Problem

    Many SISR methods assume a fixed combination of degradations, while approaches training individual networks for different combinations do not provide one network for wide-ranging variations.

    Source: §1 · p. 1

  • Method

    UDVD is a unified non-blind SISR network that uses dynamic convolutions to accommodate cross-image and spatial degradation variations.

    Source: §1 · p. 2

  • Results

    The experiments show favorable or comparable UDVD performance on both synthetic and real images.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper concludes that one UDVD network handles a wide range of degradation variations, including cross-image and spatial variations, for real-world images.

    Source: §5 · p. 8

    The evaluation uses a non-blind setting with given degradation parameters; extending to blind settings would require a degradation-estimation method prepended to UDVD.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Deep Convolutional Neural Networks (CNNs) have achieved remarkable results on Single Image Super-Resolution (SISR). Despite considering only a single degradation, recent studies also include multiple degrading effects to better reflect real-world cases. However, most of the works assume a fixed combination of degrading effects, or even train an individual network for different combinations. Instead, a more practical approach is to train a single network for wide-ranging and variational degradations. To fulfill this requirement, this paper proposes a unified network to accommodate the variations from inter-image (cross-image variations) and intra-image (spatial variations). Different from the existing works, we incorporate dynamic convolution which is a far more flexible alternative to handle different variations. In SISR with non-blind setting, our Unified Dynamic Convolutional Network for Variational Degradations (UDVD) is evaluated on both synthetic and real images with an extensive set of variations. The qualitative results demonstrate the effectiveness of UDVD over various existing works. Extensive experiments show that our UDVD achieves favorable or comparable performance on both synthetic and real images.

1. Introduction

The paper addresses the limits of fixed-degradation SISR networks by proposing UDVD, a unified network for cross-image and spatial degradation variations. Its dynamic convolutions use per-pixel, content-adaptive kernels, and the introduction reports qualitative adaptation to varying degradations where fixed-setting methods struggle.

1. Introduction

  • Fixed-degradation SISR methods are limited in practical cases with multiple degradations, while image-specific networks require retraining when degradations vary across images.SRMD handles multiple variations in one network but assumes predefined degradation types; SFTMD addresses a limited set of blind degradations.Source: §1 · p. 1
  • The proposed method is presented as accommodating cross-image and spatial variations; in the illustrated comparison, it adapts to varied regions where RCAN and ZSSR show unsatisfying quality.The introduction reports similar quality in red patches and adaptation in green, purple, and blue patches.Source: §1 · pp. 1–2
  • Dynamic convolution uses per-pixel kernels generated by a parameter-generating network, adapting to different inputs after training, unlike fixed, content-agnostic standard kernels.The paper presents this flexibility as a way to handle degradation variations.Source: §1 · p. 2
  • UDVD achieves favorable or comparable performance on synthetic and real images, according to the introduction’s summary of extensive experiments.The paper also introduces two types of dynamic convolutions and multistage loss to refine images through consecutive dynamic convolutions.Source: §1 · p. 2

2. Related Work

Prior SISR work advanced CNN architectures and image reconstruction, while most methods assumed fixed degradations. Work on multiple degradations includes image-specific training, single-network approaches, and dynamic kernels.

Single Image Super-Resolution

  • CNN-based SISR progressed from SRCNN’s three-layer architecture to deeper and recursive networks, then to direct LR-to-HR mapping and larger or iterative architectures.FSRCNN and ESPCNN used transpose or sub-pixel convolution; EDSR and RDN used larger models, DBPN used iterative up/downsampling, and RCAN added channel attention.Source: §2 · p. 2

Multiple Degradations

  • For multiple degradations, Shocher et al. trained image-specific networks, used GANs, SRMD handled multiple degradations in one network, and SFTMD addressed a subset of blind degradations.The paper describes SRMD as its most directly comparable approach and reports favorable performance against it on synthetic and real images.Source: §2 · p. 2

Dynamic Kernel Network

  • Dynamic kernels generated by parameter-generating networks have been used in denoising, video super-resolution, and video interpolation.This paper uses a fully convolutional backbone to predict per-pixel kernels for SISR with variational degradations.Source: §2 · p. 2

3. Proposed Method

UDVD addresses non-blind SISR with blur, noise, and downsampling by combining feature extraction with a refinement network built around dynamic convolutions. Its dynamic blocks can preserve resolution or integrate upsampling, and training minimizes losses at intermediate outputs.

3.1. Problem Formulation

  • The non-blind degradation model covers blur, additive noise, and downsampling, using isotropic Gaussian blur, AWGN, and bicubic downsampling.The degradation parameters can be controlled to synthesize varied degradations for training.Source: §3.1 · pp. 2–3

3.2. Unified Dynamic Convolutional Network for Variational Degradation (UDVD)

  • UDVD combines feature extraction with a refinement network that enhances and upsamples the image using extracted high-level features.The refinement network allocates dynamic blocks for feature transformation.Source: §3.2 · p. 3
  • The refinement network predicts per-pixel kernels in one path and a residual image in another, combining their outputs to form each dynamic block's image.A sub-pixel convolution aligns the resolutions between the paths.Source: §3.2 · p. 4

3.3. Types of Dynamic Convolutions

  • Dynamic convolution uses per-pixel kernels to weight nearby pixels, while its upsampling variant creates an r×r output block from each input patch.With upsampling, the output resolution is r times the input resolution, and weights are shared across channels.Source: §3.3 · p. 4

3.4. Multistage Loss

  • Multistage loss compares each dynamic block's intermediate image with the HR target and minimizes the sum of losses across blocks.The loss function can include L2 or perceptual loss.Source: §3.4 · p. 4

4. Experiments

The experiments compare UDVD configurations and evaluate performance on synthetic and real images under fixed, varied, and spatially varying degradations. Results include dynamic-kernel visualizations and comparisons with existing methods.

4.1. Datasets and Training Setups

  • Training uses synthesized degradations, 48×48 LR patches, corresponding HR patches at scales 2–4, and RGB training with PSNR and SSIM evaluated on the Y channel.Training images are drawn from DIV2K and Flickr2K; blur width and AWGN noise level are uniformly sampled from specified ranges.Source: §4.1 · p. 5

4.2. Comparison of UDVD Configurations

  • Among the tested configurations, dynamic blocks improve on the feature-extraction baseline, the UD configuration performs better than its counterparts, and multistage loss is effective.The reported configuration comparison is on Set5 with scale factor 3, kernel width 1.3, and noise level 15.Source: §4.2 · p. 5, Table 1 · p. 5

4.3. Visualizing Dynamic Kernels

  • The second dynamic block generates kernels that adapt to image content and degradation, with the adaptation varying among degradations regardless of content.Figure 4 contrasts kernels learned from an LR image, kernels learned under spatially varying blur and noise, and their absolute difference.Source: §4.3 · p. 5, Figure 4 · p. 5

4.4. Experiments on Synthetic Images

  • UDVD achieves better PSNR than IRCNN and SRMD across tested multiple-degradation settings, including the hardest case of kernel width 2.6 and noise level 50.The evaluation considers scales 2, 3, and 4, blur widths 0.2, 1.3, and 2.6, and noise levels 15 and 50.Source: §4.4 · p. 5, Table 2 · p. 6
  • Across Set5, Set14, and BSD100, UDVD constantly outperforms SRMD under spatially varying blur and noise settings.Kernel width and noise level increase from left to right across the synthesized images.Source: Table 3 · p. 6
  • In the scale-3 spatially varying blur and noise examples, UDVD produces sharper, clearer reconstructions than SRMD.Figure 5 shows zebra in the first row and bamboo in the second.Source: §4.4 · p. 6, Figure 5 · p. 7
  • On noise-free tests, UDVD is comparable to SRMDNF and SFTMD in most cases, while one UDVD model is competitive on BI and superior on DN.The noise-free tests report an advantage over both methods at large scale factors on BSD100; the BI/DN comparison uses scale factor 3.Source: Table 4 · p. 7, §4.4 · p. 7, Table 5 · p. 8

4.5. Experiments on Real Images

  • For real images, UDVD reconstructs sharper, clearer results than the compared methods, and different degradation estimates yield favorable results on different patches.Real-image degradation parameters are selected by manual grid search because ground-truth degradations are unavailable.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion

The conclusion presents UDVD as a single network for varied real-world degradations and reports favorable or comparable performance on synthetic and real images.

5. Conclusion

  • UDVD combines two types of dynamic convolutions with multistage loss to refine images across consecutive dynamic convolutions, and experiments report favorable or comparable performance on synthetic and real images.The paper describes the single network as handling cross-image and spatial degradation variations.Source: §5 · p. 8

Related papers

Loading 2004.06965v1…