PaperlayerView exact revision on arXiv

Source-linked AI summary

ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic

Xiangtao Kong, Hengyuan Zhao, Yu Qiao, Chao Dong

arXiv:2103.04039v1cs.CV

TL;DR

Large-image super-resolution is costly because computational and memory demands grow quadratically with input size, motivating sub-image processing. ClassSR classifies sub-images by restoration difficulty and routes them to SR branches of different capacities, saving up to 50% computational cost for several SR networks on DIV8K.

  • Problem

    Super-resolution on 2K–8K images has high memory and computational costs, making sub-image decomposition and acceleration necessary for real-time implementation.

  • Method

    ClassSR jointly trains a Class-Module and an SR-Module to classify sub-images by restoration difficulty and process each with a corresponding network branch using Class-Loss and Average-Loss.

  • Results

    ClassSR saves 50%, 47%, 48%, and 50% computational cost for CARN, SRResNet, and RCAN on DIV8K, respectively.

  • Takeaways & Limitations

    ClassSR provides a data-characteristic-based acceleration pipeline that can be combined with previous methods and applied to other low-level vision tasks.

  • Takeaways & Limitations

    The evaluation uses FLOPs rather than device-dependent running time, although the authors report that both have the same trend for ClassSR branches.

Abstract

from arXiv · show

We aim at accelerating super-resolution (SR) networks on large images (2K-8K). The large images are usually decomposed into small sub-images in practical usages. Based on this processing, we found that different image regions have different restoration difficulties and can be processed by networks with different capacities. Intuitively, smooth areas are easier to super-solve than complex textures. To utilize this property, we can adopt appropriate SR networks to process different sub-images after the decomposition. On this basis, we propose a new solution pipeline -- ClassSR that combines classification and SR in a unified framework. In particular, it first uses a Class-Module to classify the sub-images into different classes according to restoration difficulties, then applies an SR-Module to perform SR for different classes. The Class-Module is a conventional classification network, while the SR-Module is a network container that consists of the to-be-accelerated SR network and its simplified versions. We further introduce a new classification method with two losses -- Class-Loss and Average-Loss to produce the classification results. After joint training, a majority of sub-images will pass through smaller networks, thus the computational cost can be significantly reduced. Experiments show that our ClassSR can help most existing methods (e.g., FSRCNN, CARN, SRResNet, RCAN) save up to 50% FLOPs on DIV8K datasets. This general framework can also be applied in other low-level vision tasks.

1. Introduction

ClassSR accelerates super-resolution on large images by classifying decomposed sub-images by restoration difficulty and routing them to networks with different capacities. Experiments show substantial computational savings while maintaining performance across representative SR networks.

  • Large-image SR becomes costly because CNN memory and computation grow quadratically with input size, motivating sub-image decomposition for 2K–8K inputs.
  • Different image regions require different network complexities: smooth areas are easier to restore than complex textures.
  • About 60% of 32 × 32 LR sub-images are smooth in DIV8K, compared with 30% in DIV2K, suggesting greater acceleration potential for large images.
  • ClassSR jointly classifies sub-images by restoration difficulty and applies corresponding SR branches using Class-Loss, Average-Loss, and Image-Loss during training.
  • ClassSR is designed as a data-characteristic-based acceleration framework that is orthogonal to network compression and applicable to other low-level vision tasks.

2. Related work

Related work spans lightweight SR architectures and region-aware restoration strategies. ClassSR differs by using a general classification-and-routing framework to assign sub-images to SR branches according to restoration difficulty.

  • 2.2. Light-weight SR Networks: Lightweight SR methods reduce computation through architectural choices such as late upscaling, pyramidal processing, group convolution, feature splitting, and pixel attention.
  • 2.2. Light-weight SR Networks: ClassSR is presented as a general framework that accelerates existing SR methods from tiny to large networks rather than designing only another lightweight architecture.
  • 2.3. Region-aware Image Restoration: Region-aware restoration methods include clustering filters, semantic feature transforms, and reinforcement-learning-based paths for different image regions.
  • 2.3. Region-aware Image Restoration: ClassSR uses classification to determine each region’s processing path, distinguishing it from prior region-aware approaches.

3. Methods

ClassSR classifies decomposed sub-images by restoration difficulty and routes each to an appropriately sized SR branch. Joint training uses reconstruction, confidence, and balancing losses so simpler regions use smaller networks while preserving reconstruction quality and improving efficiency.

  • 3.1. Observation: High-PSNR sub-images are generally smooth, whereas low-PSNR sub-images contain complex textures, supporting difficulty-based classification.
  • 3.1. Observation: FSRCNN(16) nearly matches FSRCNN-O(56) on simple sub-images, while FSRCNN(36) roughly matches the base network on medium sub-images.
  • 3.2. Overview of ClassSR: ClassSR decomposes a large LR image into overlapping sub-images, classifies each into M classes, selects the maximum-probability branch, and recombines the outputs.
  • 3.3. Class-Module: The lightweight Class-Module uses five convolution layers, average pooling, and a fully connected layer to generate each sub-image’s probability vector.
  • 3.4. SR-Module: The SR-Module contains independent branches based on the target SR network, with reduced complexity controlled primarily by convolutional channel counts.
  • 3.5. Classification Method: During training, all branches contribute through probability-weighted outputs; Image-Loss preserves reconstruction, Class-Loss sharpens decisions, and Average-Loss prevents collapse to the most complex branch.
  • 3.7. Joint Training: After joint finetuning, sub-images are assigned across different branches while performance and efficiency improve simultaneously.
  • 3.8. Discussion: ClassSR is designed as a general, data-characteristic-based acceleration framework that can complement existing SR networks.

4. Experiments

Experiments evaluate ClassSR on large-image SR settings, showing reduced computation while preserving or improving reconstruction quality. Ablations and visual results examine training behavior, branch assignment, and applicability beyond SR.

  • Experimental setup: ClassSR uses DIV2K-derived training patches and DIV8K-based Test2K, Test4K, and Test8K data, with 32 × 32 LR sub-images during evaluation.Testing combines overlapping super-resolved patches by averaging and measures PSNR alongside average FLOPs.
  • Experimental setup: ClassSR evaluates FSRCNN, CARN, SRResNet, and RCAN using representative network scales and three SR branches.Branches are formed by reducing channels, with FSRCNN receiving a layer-specific channel adjustment.
  • Main results: 50% FLOPs are saved on Test8K for most ClassSR methods, with lower computational cost and generally better performance than the original networks.The reported acceleration ranges from 70% to 50%, and Test8K shows nearly two-times acceleration across methods.
  • Training behavior: 61%, 23%, and 16% of Test8K sub-images use FSRCNN branches with 16, 36, and 56 channels, reducing overall FLOPs from 468M to 236M.Training curves show PSNR increasing as FLOPs decrease during Class-Module and joint training.
  • Qualitative results: ClassSR produces visual effects comparable to the original networks, with smooth transitions between adjacent sub-images.The visual comparison uses 4K images with ×4 super-resolution and 128 × 128 super-resolved sub-images.
  • Ablation studies: Removing Class-Loss causes non-convergent PSNR and FLOPs curves, while removing Average-Loss assigns inputs to the most complex branch.These ablations support complementary roles for the two losses in avoiding random or biased branch assignment.
  • Additional task: ClassSR adapts to denoising, where it improves DnCNN PSNR at lower computational cost but saves less computation than in SR.The denoising setting contains fewer simple sub-images than SR.

5. Conclusion

The conclusion presents ClassSR as a classification-and-super-resolution pipeline for accelerating learning-based SR on 2K–8K images. It emphasizes data-dependent branch selection, reported acceleration across datasets, and extension to other low-level vision tasks.

  • Conclusion: ClassSR classifies sub-images by restoration difficulty and routes them to branches with different network capacities.The framework uses a Class-Module, SR-Module, and two novel loss functions.
  • Conclusion: ClassSR is reported to accelerate almost all learning-based SR methods on large images from 2K to 8K.The conclusion describes this as the framework’s supported scope.
  • Conclusion: Images with more simple regions, such as DIV8K images, save more FLOPs under ClassSR.The conclusion links the amount of saved computation to the prevalence of simple regions.
  • Conclusion: ClassSR can also be applied to other low-level vision tasks.The conclusion states this extension without specifying a broader task range.

Data Characteristics

The supplied passages identify the paper’s authors and their institutional affiliations. They also provide the corresponding contact information.

  • Authors: Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong are listed as authors.
  • Affiliations: The authors are affiliated with the Key Laboratory of Human-Machine Intelligence-Synergy Systems.
  • Affiliations: The affiliation is at the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences.
  • Contact: The listed contact address uses the siat.ac.cn domain for the authors’ email accounts.

Abstract

The supplementary file provides additional details and experiments for ClassSR. It covers computation-cost reductions, ablations, and qualitative results.

  • Supplementary contents: The supplementary file presents more details and experimental results for ClassSR with existing SR networks.
  • Supplementary contents: Additional experiments illustrate how ClassSR reduces computation cost.
  • Supplementary contents: The supplementary file includes further ablation-study experiments.
  • Supplementary contents: Additional qualitative results are provided to show the effectiveness of ClassSR.

1. Details of ClassSR with Existing SR networks

ClassSR instantiates branches of different capacities for several existing SR networks and routes sub-images according to restoration difficulty. For ClassSR-FSRCNN, this routing reduces FLOPs from 468M to 236M while retaining the original network as the hardest branch.

  • Branch design: ClassSR uses branches with controlled network scales for FSRCNN, CARN, SRResNet, and RCAN.The branch architectures are documented separately for each base network.
  • Evaluation: The reported PSNR comparisons use original networks trained on all DIV2K data and ClassSR branches trained on corresponding simple or complex datasets.These comparisons are provided for FSRCNN, CARN, SRResNet, and RCAN.
  • Classification and computation: 468M to 236M: ClassSR-FSRCNN reduces overall FLOPs while routing Test8K sub-images across three branches.The three branches have different computational complexities.

2. Additional Experiments

Additional experiments examine loss weighting, training-data range, gradient-based classification, resolution effects, and the relationship between FLOPs and running time. They show that classification behavior and acceleration depend on training conditions and image resolution.

  • Loss weighting: w2 = 1 achieves the best trade-off between PSNR and FLOPs, while w2 = 2 causes a significant performance decrease.The experiment fixes w1 = 2000 and w3 = 6 while varying the Class-Loss weight.
  • Training data: ClassSR should be trained with data covering a large range of PSNR values rather than only simple or complex samples.Training only on simple or complex data produces worse PSNR and can increase computation through misclassification of normal data.
  • Alternative classification: Gradient-FSRCNN obtains lower PSNR than ClassSR-FSRCNN and the original networks when gradient thresholds 279.52 and 556.79 define three classes.The thresholds are computed from average gradients of training sub-images.
  • Resolution effects: ClassSR has greater computational significance on higher-resolution images because decomposition produces more sub-images with a higher probability of simple regions.The resolution comparison uses images with the same contents at different resolutions.
  • Runtime measure: FLOPs and running time follow the same trend because ClassSR branches are derived from the original networks by reducing layers or channels.FLOPs are used because they are device-independent and widely recognized.

3. More Qualitative Results

Qualitative experiments show that ClassSR can reduce computation while preserving visual results relative to the original networks. The benefit is smaller for images dominated by complex regions.

  • Qualitative evaluation: ClassSR methods obtain better performance with lower computation cost in the reported qualitative evaluations.The evaluations use FLOPs and PSNR alongside visual results.
  • Content-dependent savings: Images with more relatively simple regions yield larger computation savings, whereas highly complex images are assigned mostly to the most complex network.For DIV2K-0821, ClassSR saves little computation because almost all sub-images use the most complex network.
  • Visual consistency: ClassSR produces the same visual effects as the original networks without incoherence between adjacent sub-images.The comparison is shown in the additional visual results.

Test8K-1430 (8K) Test8K-1459(8K)

The visual results compare ClassSR-FSRCNN with the original network for ×4 super-resolution, with sub-images classified as simple, medium, or hard. Colors indicate the assigned difficulty classes.

  • Test8K visual classification: ClassSR-FSRCNN classifies sub-images into simple, medium, and hard classes shown in green, yellow, and red, respectively.The classification is illustrated for ×4 super-resolution results.
Loading 2103.04039v1…