PaperlayerView exact revision on arXiv

Source-linked AI summary

Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution

Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, Mingkui Tan

arXiv:2003.07018v4cs.CV

TL;DR

Single-image SR must infer HR images from ambiguous LR inputs and often lacks paired data for real-world degradations. The paper introduces dual regression, which closes the LR-to-HR mapping with an HR-to-LR reconstruction constraint and enables learning from real-world LR data. Experiments on paired and unpaired data report superiority over existing methods, while the theoretical analysis is limited to paired data.

  • Problem

    Existing SR methods face an ill-posed LR-to-HR mapping and difficult adaptation when paired data or the real-world degradation process is unavailable.

  • Method

    Dual regression jointly learns LR-to-HR super-resolution and an HR-to-LR mapping that reconstructs LR inputs, forming a closed-loop constraint usable with LR data.

  • Results

    Experiments on paired and unpaired real-world SR data demonstrate superiority over the considered baseline methods.

  • Takeaways & Limitations

    The scheme supports SR improvement on paired data and adaptation to real-world LR data without corresponding HR images.

  • Takeaways & Limitations

    The paper theoretically analyzes the proposed generalization bound only for paired data; unpaired-data analysis is left for future work.

Abstract

from arXiv · show

Deep neural networks have exhibited promising performance in image super-resolution (SR) by learning a nonlinear mapping function from low-resolution (LR) images to high-resolution (HR) images. However, there are two underlying limitations to existing SR methods. First, learning the mapping function from LR to HR images is typically an ill-posed problem, because there exist infinite HR images that can be downsampled to the same LR image. As a result, the space of the possible functions can be extremely large, which makes it hard to find a good solution. Second, the paired LR-HR data may be unavailable in real-world applications and the underlying degradation method is often unknown. For such a more general case, existing SR models often incur the adaptation problem and yield poor performance. To address the above issues, we propose a dual regression scheme by introducing an additional constraint on LR data to reduce the space of the possible functions. Specifically, besides the mapping from LR to HR images, we learn an additional dual regression mapping estimates the down-sampling kernel and reconstruct LR images, which forms a closed-loop to provide additional supervision. More critically, since the dual regression process does not depend on HR images, we can directly learn from LR images. In this sense, we can easily adapt SR models to real-world data, e.g., raw video frames from YouTube. Extensive experiments with paired training data and unpaired real-world data demonstrate our superiority over existing methods.

1. Introduction

Existing SR methods face an ill-posed LR-to-HR mapping problem and difficulty adapting when paired data or the degradation process is unavailable. The proposed dual regression scheme constrains the mapping through LR reconstruction and supports learning from real-world LR data.

  • The LR-to-HR mapping is ill-posed because infinitely many HR images can downscale to the same LR image.This creates a very large function space, making it difficult to learn mappings that produce sharp textures.
  • Real-world SR is challenging because paired LR-HR data may be unavailable and real LR images may not match Bicubic-degraded training data.Directly applying existing models can cause severe adaptation problems and poor performance.
  • Dual regression adds an LR reconstruction constraint that estimates the downsampling kernel and reduces the space of possible LR-to-HR mappings.The super-resolved output is downsampled to reconstruct the input LR image, forming a closed loop.
  • Because dual regression does not depend on HR images, the scheme can learn from unpaired real-world LR data such as raw YouTube video frames.The paper presents this as a way to adapt SR models to real-world data.
  • Experiments on paired SR tasks and unpaired real-world data demonstrate the effectiveness of the proposed scheme.The contribution summary explicitly covers both paired and unpaired settings.

2. Related Work

Prior work improves supervised SR architectures and develops unpaired or blind SR methods, but these approaches differ in how they use paired and real-world data. The proposed approach is positioned as adapting SR with both data types.

  • Supervised super-resolution: Supervised SR methods include interpolation-based and reconstruction-based approaches, including DBPN and RCAN.DBPN iteratively produces LR and HR images, while RCAN uses channel attention in a deep model.
  • Unsupervised super-resolution: Unsupervised SR methods seek to generate HR images without paired data, including CinCGAN and blind SR approaches for unknown degradations.These methods address settings where paired training data are unavailable.
  • Unsupervised super-resolution: Unlike methods that discard paired synthetic data, dual regression adapts SR models using both real-world LR data and paired synthetic data.The paper frames this combined use as its distinction from the cited unsupervised approaches.

3. Proposed Method

The proposed method jointly learns primal LR-to-HR and dual HR-to-LR regressions, using their closed loop to constrain super-resolution mappings. Training applies this scheme to paired data and, through an adaptation algorithm, combines unpaired real-world with paired synthetic data.

  • 3. Proposed Method: Dual regression augments LR-to-HR super-resolution with an inverse mapping from super-resolved images back to LR images.The additional constraint is intended to reduce the otherwise very large space of possible mappings.
  • 3. Proposed Method: The primal task predicts HR images from LR inputs, while the dual task predicts the original LR image from an HR image.The dual mapping can be regarded as estimating the underlying downsampling kernel.
  • 3. Proposed Method: The primal and dual tasks form a closed loop because D(P(x)) should be close to x when P(x) is the correct HR image.This reconstruction constraint provides additional supervision and narrows the function space.
  • 3. Proposed Method: For paired samples, the training objective combines primal and dual regression losses with a weight λ controlling the dual loss.Both losses are defined as ℓ1-norm losses.
  • 3. Proposed Method: The adaptation algorithm samples unpaired real-world data and paired synthetic data, then updates the primal and dual models iteratively.The procedure loads pretrained models and alternates optimization of the primal and dual objectives.

11 end

DRN adapts super-resolution to unpaired real-world LR data by learning degradation through dual regression while retaining paired synthetic supervision. Its training combines both data types and forms a closed loop that reconstructs LR images from super-resolved outputs.

  • Training Method: The dual regression mapping learns degradation from unpaired real-world LR data without requiring corresponding HR images.Paired synthetic data are also incorporated to preserve HR reconstruction performance.
  • Architecture: DRN uses a U-Net-style architecture with primal and dual networks, linking shallow and deep features through shortcut connections.The dual network is represented by red lines in the architecture, while downsampling and upsampling feature maps occupy the two halves.
  • Training Method: The objective combines unpaired LR samples with paired synthetic samples, training the model end-to-end through the dual regression loss.Each iteration samples m unpaired real-world examples and n paired synthetic examples.
  • Training Method: Setting the unpaired-data ratio to ρ = 30% gives the best reported results and supports adaptation across various unpaired datasets.The ratio can be adjusted by changing the number of paired synthetic samples.
  • Differences from CycleGAN based SR Methods: Unlike CycleGAN-based SR, DRN reduces the mapping space by reconstructing LR images and exploits paired synthetic data alongside real-world unpaired data.CycleGAN-based methods are described as discarding the readily obtainable paired synthetic data.

4. More Details

The paper builds DRN from U-Net-style primal and lightweight dual networks, then analyzes its generalization and compares performance against model size. The dual scheme is reported to tighten the generalization bound and improve SR predictions, while DRN-L achieves the best empirical results.

  • Architecture Design of DRN: DRN combines a U-Net-style primal network with a lightweight dual network that learns downsampling from super-resolved images.The dual model uses two convolution layers and LeakyReLU, with lower computation cost than the primal model.
  • Architecture Design of DRN: For scale factor s, the primal network uses log2(s) downsampling and upsampling blocks built from residual channel attention blocks.The architecture therefore uses two blocks for 4× upscaling and three blocks for 8× upscaling.
  • Limitations: The generalization analysis is provided for paired data, whereas the unpaired-data case is left for future theoretical investigation.Definitions, proofs, and lemmas are placed in the supplementary material.
  • Theoretical Analysis: Theorem 1 bounds the expected loss of the dual regression scheme using the empirical Rademacher complexity of its joint primal-dual function space.The bound applies when the loss is bounded in [0, C].
  • Experiments: DRN-S delivers promising 4× SR performance with few parameters, while DRN-L achieves the best results after increasing channels and layers.The comparison is reported for models evaluated on Set5.
  • Theoretical Analysis: The dual regression scheme has a smaller generalization bound than the primal feed-forward scheme and helps the primal model achieve more accurate SR predictions.The stated comparison follows from the lower capacity of the joint dual function space.

5. Experiments

Experiments evaluate the method on paired benchmark data and unpaired real-world data, including synthetic degradations and YouTube video frames. The proposed models achieve strong quantitative and visual results, while DRN-Adapt improves adaptation to unknown degradations.

  • Paired Image Super-Resolution: DRN-S uses about 5M parameters, while DRN-L uses about 10M and achieves the best reported performance among considered methods for 8× SR.DRN-L is comparable with the considered methods for 4× SR.
  • Paired Image Super-Resolution: For both 4× and 8× SR, the proposed model produces sharper edges and shapes than baselines, which often produce blurrier results.The visual comparisons are reported in Figure 5.
  • Unpaired Real-World Data: Unpaired-data experiments evaluate adaptation on LR images generated with Nearest and BD degradations, plus raw YouTube video frames without corresponding HR images.The adaptation model is called DRN-Adapt.
  • Unpaired Real-World Data: DRN-Adapt consistently outperforms supervised methods across all evaluated datasets for 8× SR and improves over the Nearest-kernel recovery baseline.Table 2 reports PSNR and SSIM for the adaptation comparison.
  • Unpaired Real-World Data: On YouTube frames, DRN-Adapt produces sharper and clearer textures, whereas supervised baselines contain mosaics and CinCGAN produces distortion and noise.Because no ground-truth HR images are available, the comparison is visual.

6. Further Experiments

Further experiments isolate the contribution of dual regression and examine the adaptation algorithm's hyperparameters and unpaired-data ratio. The results support the scheme's reconstruction benefit and identify practical trade-offs for λ and ρ.

  • Ablation Study on Dual Regression Scheme: Adding dual regression improves PSNR on all five benchmark datasets in the 4× ablation study.The study attributes the improvement to an additional constraint that reduces the mapping-function space.
  • Hyper-parameter λ: Increasing λ from 0.001 to 0.1 strengthens dual-regression supervision, but λ values of 1 or 10 overwhelm primal regression and reduce performance.The paper sets λ = 0.1 as the practical trade-off.
  • Adaptation Algorithm: Using unpaired-data ratios ρ of 30%, 50%, or 70% improves performance over the ρ=0% baseline.The paper sets ρ=30% in practice because it obtains the best performance.

7. Conclusion

The paper proposes dual regression for both paired and unpaired super-resolution settings. Experiments on paired and unpaired data report superiority over the considered baseline methods.

  • Conclusion: The proposed dual regression scheme reconstructs LR images to constrain the mapping space and improve SR performance on paired data.The same scheme is applied to real-world unpaired data such as YouTube raw video frames.
  • Conclusion: Extensive experiments on paired and unpaired data demonstrate superiority over the considered baseline methods.

Supplementary Materials for “Closed-loop Matters: Dual Regression Networks

The supplementary materials provide theoretical analysis, architectural and implementation details, additional ablations, and further reported results.

  • Supplementary Materials: The supplement derives the dual-regression generalization error bound and gives additional details about the DRN architecture and training procedures.It covers both paired-data and unpaired-data SR training.
  • Supplementary Materials: Additional ablation studies and further results are included in the supplementary materials.

A. Theoretical Analysis

The analysis formalizes dual learning through joint primal–dual losses and shows that its constrained function space yields tighter generalization bounds than standard supervised learning. The proposed DRN implements this closed loop with scale-specific primal and dual models and is evaluated across paired and unpaired settings.

  • Generalization framework: Dual learning optimizes bidirectional primal and dual tasks using a combined loss over predictions and reconstructed inputs.The expected loss is defined as LP(P(x), y) + λLD(D(P(x)), x).
  • Generalization framework: For finite function spaces, larger sample size and smaller function space improve the generalization guarantee.The theorem explicitly links generalization to sample size and function-space size.
  • Generalization framework: For infinite function spaces, the bound depends on empirical Rademacher complexity, sample size, and the loss upper bound.The bound is stated for losses bounded in [0, C] and includes the dual-learning complexity term.
  • Generalization framework: The dual hypothesis space has lower capacity than the standard supervised space, giving a smaller generalization bound for the dual model.The analysis states B(P, D) ≤ B(P) under the same number of samples.
  • DRN architecture: DRN uses two primal blocks and two dual models for 4× SR, and three primal blocks and three dual models for 8× SR.The architecture is organized around scale-specific closed loops.
  • Experimental setup: The experiments train on paired DIV2K/Flickr2K data and also construct unpaired synthetic and real-world data for adaptation studies.The unpaired setting includes LR raw video frames from YouTube.

D.2. Effect of the Dual Regression on HR Data

Experiments examine whether dual regression needs an HR-domain constraint and how degradation choices affect adaptation, while visual comparisons assess supervised and real-world results. The reported findings indicate comparable HR-domain performance at lower computation and sharper visual outputs than state-of-the-art methods.

  • Effect on HR data: DRN-S achieves comparable performance to its HR-dual variant while requiring only half the computation.The comparison evaluates dual regression on LR data versus regression on both LR and HR images.
  • Degradation methods: DRN-Adapt obtains similar results when paired synthetic data use Bicubic or Nearest degradation and evaluation uses BD data.The experiment varies the degradation kernel used to create paired synthetic data.
  • Visual comparisons: Visual comparisons cover 4× and 8× supervised SR, plus 8× adaptation on real-world YouTube video frames.Figures C, D, and E correspond to these settings.
  • Visual comparisons: The models produce sharper edges and clearer textures than state-of-the-art methods in the reported visual comparisons.This conclusion is stated for the supervised and real-world unpaired visual results.
Loading 2003.07018v4…