PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Privileged Information for Efficient Image Super-Resolution

Wonkyung Lee, Junghyup Lee, Dohyung Kim, Bumsub Ham

arXiv:2007.07524v1cs.CV

TL;DR

Efficient SISR models often trade reconstruction quality against computational cost and memory. This paper distills knowledge from an HR-input teacher into an LR-input student using imitation and feature-distillation losses, substantially improving FSRCNN and other SR methods.

  • Problem

    CNN-based SISR methods can require substantial memory and computational resources, motivating efficient models without sacrificing reconstruction performance.

  • Method

    The framework trains an HR-input teacher with imitation and reconstruction losses, then transfers affordable intermediate decoder features to an LR-input student through feature distillation.

  • Results

    The framework substantially improves FSRCNN and other SISR methods, including VDSR, IDN, and CARN.

  • Takeaways & Limitations

    Ground-truth HR images can serve as privileged information for transferring useful reconstruction knowledge to efficient SISR students.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have allowed remarkable advances in single image super-resolution (SISR) over the last decade. Most SR methods based on CNNs have focused on achieving performance gains in terms of quality metrics, such as PSNR and SSIM, over classical approaches. They typically require a large amount of memory and computational units. FSRCNN, consisting of few numbers of convolutional layers, has shown promising results, while using an extremely small number of network parameters. We introduce in this paper a novel distillation framework, consisting of teacher and student networks, that allows to boost the performance of FSRCNN drastically. To this end, we propose to use ground-truth high-resolution (HR) images as privileged information. The encoder in the teacher learns the degradation process, subsampling of HR images, using an imitation loss. The student and the decoder in the teacher, having the same network architecture as FSRCNN, try to reconstruct HR images. Intermediate features in the decoder, affordable for the student to learn, are transferred to the student through feature distillation. Experimental results on standard benchmarks demonstrate the effectiveness and the generalization ability of our framework, which significantly boosts the performance of FSRCNN as well as other SR methods. Our code and model are available online: https://cvlab.yonsei.ac.kr/projects/PISR.

1 Introduction

CNN-based SISR has improved reconstruction quality but often demands substantial computation and memory. The paper proposes using HR images as privileged information in a teacher–student distillation framework to improve efficient SISR models.

  • CNN-based SISR methods improve reconstruction quality but often require substantial computational cost and memory, complicating deployment on devices such as televisions and mobile phones.
  • The framework treats ground-truth HR images as privileged information to extract complementary features, including high-frequency details, for SISR.
  • The teacher receives HR images while the student receives corresponding LR images, unlike conventional knowledge distillation, and both reconstruct HR images.
  • An encoder trained with imitation loss extracts compact HR features, while the teacher decoder transfers intermediate features to the student through feature distillation.
  • The proposed framework uses FSRCNN as a hardware-friendly student and reports significant performance improvements for FSRCNN, VDSR, IDN, and CARN.

2 Related work

Related work addresses SISR quality, efficiency, network compression, and privileged-information learning. The proposed framework combines these directions by distilling knowledge from an HR-trained teacher to an LR-input student.

  • SISR methods use deeper, wider, recursive, residual, memory, feedback, cascaded, multi-branch, or grouped-convolution architectures to improve reconstruction or efficiency.
  • Many efficient SR architectures reduce runtime or memory but can be difficult to implement in hardware because they are specially designed for SR.
  • Knowledge distillation transfers representational ability from a large teacher to a small student, using soft outputs or intermediate features.
  • SRKD applies feature distillation to SISR with teacher and student networks that both use LR images, whereas this framework gives the teacher HR inputs.
  • Learning using privileged information uses extra training-time information that is unavailable at test time; here, ground-truth HR images serve that role without additional test-time cost.

3 Method

The framework uses a teacher that learns from HR privileged information and a same-capacity student that reconstructs HR images from LR inputs. It transfers reconstruction capability and adaptive intermediate features through initialization and feature distillation.

  • Teacher–student framework: The teacher learns from ground-truth HR images, while its decoder and the student share the same architecture.The teacher reconstructs HR images from compact encoder features; the student instead takes LR images as input.
  • Teacher training: The teacher combines HR reconstruction with an imitation loss that makes its encoder output close to the LR image.The reconstruction term encourages useful compact features, while the imitation term restricts encoder representational power.
  • Teacher training: The combined teacher objective avoids a trivial LR-copying solution because encoder parameters receive both reconstruction and imitation updates.The two terms are balanced by λT.
  • Student training: The student is initialized with the teacher decoder’s weights, transferring reconstruction capability before student training on LR inputs.The student uses FSRCNN as its hardware-friendly architecture and is further trained with reconstruction and distillation losses.
  • Feature distillation: Feature distillation maximizes a variational lower bound on mutual information between teacher and student intermediate features.A multivariate Laplace model uses location and scale parameters to approximate the conditional feature distribution.
  • Feature distillation: The learned scale adaptively reduces distillation where the student cannot benefit, making transferred features affordable despite different inputs.The scale parameter increases when distillation is not beneficial, while the location map minimizes teacher–student feature distance.

4 Experiments

Experiments evaluate the framework on standard super-resolution benchmarks and analyze its components and compact features. Ablations show that feature distillation, the hourglass teacher, decoder initialization, and imitation loss improve student performance, while compact features retain high-frequency information useful for reconstruction.

  • Experimental setup: The framework is trained on 800 DIV2K LR-HR pairs and evaluated on Set5, Set14, B100, and Urban100 using luminance-channel PSNR and SSIM.LR images are synthesized by bicubic downsampling, with cropped and augmented HR-LR patches used for training.
  • Ablation studies: Feature distillation improves the student over the FSRCNN baseline in the Set5 2× ablation.Table 1 reports average PSNR for student and teacher variants, with FSRCNN reproduced without distillation as the baseline.
  • Ablation studies: The hourglass teacher improves the student by narrowing the teacher-student performance gap, despite reducing the teacher’s own performance by 19.9dB.The resulting student reaches 37.22dB versus 37.19dB for the preceding variant.
  • Ablation studies: Initializing the student with decoder weights provides a useful starting point and transfers the teacher’s reconstruction capability.The ablation attributes the student improvement to inherited initialization from the teacher decoder.
  • Analysis on compact features: Compact features retain more high-frequency components than LR images and have magnitude spectra more similar to HR images, especially at high frequencies.The teacher decoder outperforms the same-architecture student, indicating that HR-derived compact features contain useful reconstruction information.
  • Analysis on compact features: The imitation loss makes LR-image and compact-feature distributions more similar while preserving useful feature content for distillation.The ablation reports that adding imitation loss further improves PSNR and eases student learning.

4.4 Results

Results compare the proposed student models with efficient and state-of-the-art SISR methods across datasets and scale factors. The framework consistently improves FSRCNN and benefits several other architectures while maintaining a favorable quality-efficiency trade-off and better qualitative reconstruction.

  • Quantitative comparison: The student model outperforms FSRCNN by a large margin across all scale factors despite using the same network architecture.Table 2 reports PSNR and SSIM for 2×, 3×, and 4× upsampling on Set5, Set14, B100, and Urban100.
  • Quantitative comparison: 13K parameters, 6.0G operations, and 0.83ms runtime achieve 30.24dB average PSNR on Urban100 for factor 2.The comparison is against DRCN’s 30.75dB using 1,774K parameters, 17,974.3G operations, and 233.93ms.
  • Efficiency comparison: The trade-off plot encodes operations on one axis, average Set5 PSNR on the other, parameter count by circle size, and efficiency by background color.White indicates high efficiency and black indicates low efficiency.
  • Generalization across architectures: The framework improves student versions of nearly all evaluated SR methods, except IDN at scale factor 4 on Set5.The paper highlights significant gains for FSRCNN and VDSR and further improvements for cost-effective methods.
  • Qualitative comparison: Student reconstructions improve qualitatively over baselines by removing artifacts and recovering small-scale structures and textures on Urban100 and Set14.Examples include sculpture borders, windows, iron railings, and tablecloth patterns.

5 Conclusion

The framework uses ground-truth HR images as privileged information for SISR and substantially improves FSRCNN and other methods. Component analyses demonstrate its effectiveness, while future work will explore distillation losses tailored to the model.

  • The proposed distillation framework leverages ground-truth HR images as privileged information for SISR.
  • The framework substantially improves the performance of FSRCNN and other SISR methods.
  • Component-level analyses demonstrate the effectiveness of the framework.
  • Future work will explore distillation losses specific to the model to further improve performance.

Learning with Privileged Information for Efficient Image Super-Resolution Supplement

The supplementary material analyzes scale-map sizes and imitation loss, reports ablations and large-dataset comparisons, and provides additional qualitative results for student networks matching various SR architectures.

  • The supplement analyzes different scale-map sizes and the imitation loss used in the framework.
  • It includes ablation studies with comparisons on a large dataset.
  • It presents additional qualitative results for student networks sharing architectures with various SR methods.

1 The size of scale map

The scale-map analysis compares FSRCNN student networks with different scale-map sizes on Set5 at 2× using average PSNR. Performance is best when the scale-map and teacher feature-map sizes match.

  • The analysis compares FSRCNN student networks with different scale-map sizes using average PSNR on Set5 at 2×.
  • The best performance occurs when the scale-map size matches the teacher feature-map size.This matching adaptively controls distillation across elements of the teacher feature map.
  • Matching these sizes adaptively controls the extent of distillation on each teacher feature-map element.

2 Imitation loss

The imitation-loss analysis evaluates teacher variants and distillation frameworks across benchmark settings. The loss supports weight transfer, but excessive weighting removes the benefit of privileged information and reduces performance.

  • The imitation loss encourages HR-derived compact features to approach LR counterparts, facilitating student-network initialization.
  • Removing the imitation loss decreases the performance gain, indicating that it is crucial for weight transfer.
  • The experiments compare student networks across framework variants and distillation losses on Set5 and B100.
  • A λT value of 1 causes a 0.08dB decrease compared with the third-row result by forcing compact features to match LR images.
  • With excessive λT, the framework becomes self-distillation and no longer benefits from privileged information.

3 Ablation studies

The ablation analysis shows that progressively adding the framework’s components increases average PSNR, while the proposed privileged-information framework consistently outperforms conventional knowledge distillation on Set5 and B100.

  • Adding each component of the framework gradually increases average PSNR for student networks on B100.The analysis uses variants evaluated on Set5 and B100.
  • The proposed framework consistently achieves better PSNR than conventional knowledge distillation on Set5 and B100.The comparison uses different distillation losses and attributes the advantage to distilling features from ground-truth HR images.

4 Qualitative results

Across Set14, B100, and Urban100, student networks trained with the framework produce sharper and more accurate reconstructions than their corresponding baselines, including for FSRCNN, FSRCNN-L, IDN, CARN, and VDSR.

  • Student networks trained with the framework consistently show better visual results than the original FSRCNN-L, FSRCNN, IDN, CARN, and VDSR networks.The comparisons cover reconstruction examples on Set14, B100, and Urban100.
  • The framework improves reconstruction of sharp boundaries, small-scale structures, textures, and straight lines across the qualitative examples.Examples include alphabet characters, windows, railings, zebra patterns, ceilings, and bridges.
  • Compared with FSRCNN-L, the model reconstructs straight lines and object boundaries more accurately on Set14 and Urban100.The figure reports average PSNR/SSIM in parentheses and compares the model with FSRCNN-L.
  • Compared with FSRCNN, the model reconstructs small-scale structures, straight lines, and object boundaries more accurately on Set14, Urban100, and B100.The comparison covers 3× and 4× super-resolution settings.
  • Compared with IDN, the model more accurately reconstructs repetitive patterns, straight lines, and object boundaries on B100 and Urban100.The comparison covers 2× and 3× super-resolution settings.
  • Compared with CARN, the model more accurately reconstructs small-scale structures, repetitive patterns, and straight lines on Urban100 and B100.The comparison covers 2×, 3×, and 4× super-resolution settings.
  • Compared with VDSR, the model more accurately reconstructs small-scale structures, straight lines, and object boundaries on B100 and Ser14.The figure reports average PSNR/SSIM for 2×, 3×, and 4× settings.
Loading 2007.07524v1…