PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution

Alice Lucas, Santiago Lopez Tapia, Rafael Molina, Aggelos K. Katsaggelos

arXiv:1806.05764v2cs.CV

TL;DR

Video super-resolution needs to reconstruct high-resolution video from low-resolution observations, while prior MSE-based approaches can remain blurry and perceptual losses had not been applied to VSR. The paper introduces VSRResNet within a GAN framework, adds feature- and pixel-space regularization, and uses PercepDist to evaluate perceptual quality. VSRResFeatGAN is reported to outperform current state-of-the-art models quantitatively and qualitatively, though adversarial training can introduce noise that remains to be reduced.

  • Problem

    Video super-resolution must reconstruct high-resolution frames from low-resolution video, but MSE-based approaches can remain blurry and GAN or feature-based losses had not been applied to VSR.

  • Method

    The paper combines VSRResNet with GAN training and Charbonnier distance regularizers in feature and pixel spaces, evaluating perceptual quality with PercepDist.

  • Results

    VSRResFeatGAN outperforms current state-of-the-art SR models quantitatively and qualitatively, and its PercepDist results align with visually pleasing frames.

  • Takeaways & Limitations

    PercepDist can provide a perceptual-quality comparison that is more consistent with qualitative VSR results than PSNR and SSIM alone.

  • Takeaways & Limitations

    Adversarial training can introduce noise into estimated frames, motivating constraints that enforce consistency with the mathematical VSR formulation.

Abstract

from arXiv · show

Video super-resolution (VSR) has become one of the most critical problems in video processing. In the deep learning literature, recent works have shown the benefits of using adversarial-based and perceptual losses to improve the performance on various image restoration tasks; however, these have yet to be applied for video super-resolution. In this work, we propose a Generative Adversarial Network(GAN)-based formulation for VSR. We introduce a new generator network optimized for the VSR problem, named VSRResNet, along with a new discriminator architecture to properly guide VSRResNet during the GAN training. We further enhance our VSR GAN formulation with two regularizers, a distance loss in feature-space and pixel-space, to obtain our final VSRResFeatGAN model. We show that pre-training our generator with the Mean-Squared-Error loss only quantitatively surpasses the current state-of-the-art VSR models. Finally, we employ the PercepDist metric (Zhang et al., 2018) to compare state-of-the-art VSR models. We show that this metric more accurately evaluates the perceptual quality of SR solutions obtained from neural networks, compared with the commonly used PSNR/SSIM metrics. Finally, we show that our proposed model, the VSRResFeatGAN model, outperforms current state-of-the-art SR models, both quantitatively and qualitatively.

I. INTRODUCTION

Video super-resolution reconstructs high-resolution frames from low-resolution video, but conventional MSE-trained approaches can produce conservative, blurry results. This paper extends GANs and feature-based losses to VSR, introducing VSRResNet and VSRResFeatGAN and evaluating perceptual quality with PercepDist.

  • Video super-resolution estimates high-resolution frames from observed low-resolution frames for high-resolution display.
  • MSE training provides reasonable VSR solutions but can be conservative and fail to fully exploit deep networks’ potential.
  • GANs and feature-based losses had improved image restoration, but had not yet been applied to video super-resolution.
  • The paper introduces VSRResNet, reframes it adversarially, adds feature-based losses, and obtains VSRResFeatGAN.
  • PercepDist is used alongside PSNR and SSIM to assess perceptual quality, with VSRResFeatGAN reported to sharpen frames more than existing VSR networks.

II. RELATED WORK

Prior VSR systems commonly rely on MSE-based training and, in some cases, motion compensation, which can yield blurry estimates. The paper proposes a deeper residual architecture that learns from motion without input motion compensation, then extends it with adversarial and perceptual losses.

  • Existing VSR models use analytical preprocessing, direct mappings, recurrent networks, or motion-compensation strategies.
  • MSE-guided training in prior systems can produce estimated high-resolution frames that remain fairly blurry.
  • A. The VSRResNet architecture: VSRResNet uses a chain of residual blocks totaling 34 convolution operations to increase model capacity while avoiding vanishing gradients.
  • A. The VSRResNet architecture: VSRResNet fuses features across time steps and learns the final high-resolution solution through fifteen residual blocks.
  • A. The VSRResNet architecture: The network is trained without motion compensation so it can extract useful motion information, while also reducing computational time.
  • B. The proposed adversarial system: VSRResNet is incorporated into an adversarial framework with perceptual losses to form VSRResFeatGAN.

B. The proposed adversarial system

The proposed adversarial system trains a generator to super-resolve a center high-resolution patch from a sequence of low-resolution frames. It uses VSRResNet as generator and a discriminator that predicts whether an output patch is real.

  • The GAN adapts adversarial training to VSR by super-resolving high-resolution center patches from low-resolution input sequences.
  • The generator receives a sequence of low-resolution frames, while the discriminator guides generation toward outputs that resemble real high-resolution patches.
  • The adversarial objective compares real high-resolution center frames with generator outputs using trainable discriminator and generator parameters.
  • The generator is fixed as the VSRResNet architecture, and the discriminator consists of three convolutional layers, a fully connected layer, and a sigmoid.
  • The discriminator’s sigmoid output represents the probability that an input patch is real.

C. Adding feature-space and pixel-space distance as regularizers

The proposed VSRResFeatGAN regularizes adversarial training with Charbonnier distances in pixel and VGG feature spaces, combining these with adversarial loss to produce the final model.

  • Adversarial loss alone can produce strong high-frequency artifacts, including ringing patterns around frame edges.
  • The model adds pixel-space and feature-space distances between generated and ground-truth frames as regularizers.The pixel term constrains content similarity, while the feature term compares representations learned by a VGG network.
  • The Charbonnier loss uses ϵ = 0.001 and approximates the l1 loss; the authors report improved GAN learning behavior over l2 regularization.
  • The feature-space distance uses VGG feature maps from the network’s third and fourth convolutional layers.
  • The total objective weights feature, adversarial, and pixel losses with α > 0, β > 0, and α+β < 1, producing the VSRResFeatGAN model.

IV. EXPERIMENTS

The experiments evaluate VSRResNet and VSRResFeatGAN against state-of-the-art DNN-based VSR models using a synthesized HR/LR video dataset.

  • The training dataset uses 53 Myanmar video sequences, with 6 sequences reserved for testing.
  • The raw 4K Myanmar videos are downsampled by four to produce 960 × 540 frames for the synthesized dataset.

B. Pre-training of the generator architecture

Because generator learning is more difficult than discrimination and poor generator outputs can destabilize GAN training, the authors pre-train VSRResNet with MSE before adversarial training.

  • The generator is pre-trained with the traditional MSE loss before adversarial training to start GAN optimization from a reasonable point.
  • This initialization is intended to support proper convergence because artifact-laden generator patches can make discrimination trivial and cause GAN training failure.
  • VSRResNet pre-training runs for 100 epochs with ADAM, batch size 64, and learning rate 0.001 reduced tenfold at epochs 50 and 75.
  • The same pre-training hyperparameters are used for scale factors 2, 3, and 4.

A. Evaluation of the effect of depth in VSRResNet

The evaluation compares the deeper VSRResNet with the shallower VSRNet and finds that increasing depth substantially improves video super-resolution quality across scale factors. Qualitative examples likewise show sharper VSRResNet outputs.

  • Evaluation setup: The experiment evaluates VSRResNet against VSRNet to assess the effect of network depth.Both architectures extract spatial information from five input frames before subsequent convolutional processing.
  • Depth comparison: VSRResNet outperforms VSRNet by a large margin across all scale factors on the Myanmar test dataset.The comparison uses PSNR and SSIM.
  • Qualitative comparison: VSRResNet produces sharper super-resolved frames than VSRNet in selected qualitative regions.The comparison is illustrated for scale factor 3.

C. Comparison with state-of-the-art DNNs

The evaluation compares VSR models on VidSet4 using conventional distortion metrics and PercepDist, alongside qualitative frame comparisons. VSRResNet leads on PSNR/SSIM at scales 2 and 3, whereas VSRResFeatGAN is favored for perceptual quality across scale factors.

  • Evaluation metrics: PercepDist measures distance between ground-truth and super-resolved frames, with smaller values indicating better perceptual quality.The metric is computed using a pretrained CNN and its evaluation code.
  • Quantitative comparison: VSRResNet surpasses state-of-the-art neural VSR systems in PSNR and SSIM for scale factors 2 and 3 without input motion compensation.Its performance decreases somewhat at scale factor 4, where large downscaling removes helpful motion details.
  • Perceptual comparison: VSRResFeatGAN outperforms VSRResNet and the compared state-of-the-art VSR models consistently across all scale factors according to PercepDist.The comparison includes SPMC-VSR, VESPCN, and Temporal Adaptive Net, as well as still-image SR models.
  • Qualitative comparison: VSRResFeatGAN produces sharper, less blurry frames than competing VSR and still-image SR models in the reported qualitative comparisons.Examples include sharper calendar text, cars, and leafy trees.
  • Qualitative comparison: Adversarial training introduces visible distortions, including dot-like patterns on architectural lines in the city frame.The paper notes that these artifacts coexist with increased sharpness and high-frequency detail.
  • Metric interpretation: Lower PSNR/SSIM values for VSRResFeatGAN do not align fully with its qualitative sharpness, supporting PercepDist as a more suitable perceptual comparison metric.The authors report that low PercepDist values are consistent with visually pleasing results.

D. Training Observations

The training observations examine loss-function and initialization choices used to improve adversarial VSR training. They focus on replacing l2 distance with Charbonnier loss and transferring weights across scale factors.

  • Loss design: The Charbonnier loss is introduced to measure distances between estimated and ground-truth patches in both pixel and feature space.The comparison replaces the Charbonnier term in Equation 3 with l2 distance.
  • Overall effect: These design choices improve adversarial training and produce frames with higher perceptual quality.The paper evaluates their effects through discriminator and generator losses and qualitative frame comparisons.

1) The effect of the Charbonnier loss during training:

The Charbonnier-loss experiment compares it with l2 distance during adversarial training. Charbonnier produces more stable discriminator behavior and better handling of small details.

  • Training stability: Charbonnier loss provides a more stable alternative to l2 distance during adversarial training.The comparison plots discriminator loss functions under controlled replacement of the loss term.
  • Effect on details: Charbonnier loss is more robust to small details, making the discriminator’s task slightly more difficult.This increased difficulty facilitates subsequent learning by the generator network.

2) Transfer Learning from VSRResFeatGAN-u2 to VSRResFeatGAN-u4:

The transfer-learning experiment initializes scale-factor-4 VSRResFeatGAN training with weights learned at scale factor 2. This attenuates adversarial distortions and improves perceptual quality.

  • Transfer-learning effect: Transferring weights from scale factor 2 attenuates the strong dot-like pattern produced by adversarial loss at scale factor 4.The comparison is shown in Figure 13.
  • Training dynamics: Weight transfer makes the discriminator’s task harder and is associated with smaller generator loss values during training.The authors interpret this as evidence that the generator produces higher-perceptual-quality patches.
  • Practical outcome: For challenging scale factors, transferring weights from a smaller scale factor can yield frames with fewer high-frequency distortions.The paper reports this as a benefit of appropriately transferring VSRResFeatGAN parameters.

VI. CONCLUSION

The paper finds that architectural and loss-function choices improve VSR performance and perceptual quality, while Perceptual Distance can rate sharp outputs more favorably than PSNR. The authors also identify adversarially introduced noise as an avenue for further improvement.

  • Conclusion: Architectural and loss-function choices substantially improve VSR performance while allowing the network to use motion information without explicit motion compensation.The approach trains a deep residual network with GAN losses and Charbonnier distance in feature and pixel spaces.
  • Conclusion: GAN and Charbonnier losses enable VSRResFeatGAN to produce frames with significantly higher perceptual quality.The model is compared using PSNR, SSIM, and Perceptual Distance.
  • Conclusion: Perceptual Distance can favor sharp frames that PSNR would classify as blurry, providing a perceptual-quality comparison alongside PSNR and SSIM.This highlights a difference between conventional distortion metrics and perceptual evaluation.
  • Conclusion: Qualitative evaluations compare VSRResFeatGAN with VSRResNet and existing VSR or image-SR models on Myanmar and VidSet4 sequences.The comparisons include scale factor 4 outputs and zoomed-in regions across calendar, foliage, walk, and city sequences.
  • Conclusion: Replacing MSE with Charbonnier loss during adversarial training produces a more stable training procedure.With MSE, the discriminator quickly separates super-resolved patches from ground truth, limiting generator learning.
Loading 1806.05764v2…