PaperlayerView exact revision on arXiv

Source-linked AI summary

ENAF: A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution

Duong M. Nguyen, Tuan Nghia Nguyen, Xuan Truong Nguyen

arXiv:2608.15349v1cs.CVcs.AIeess.IV

TL;DR

Large-image SISR incurs substantial memory and computation costs, while patch difficulty is imperfectly matched to execution capacity. ENAF uses multiple early exits, PSNR-based patch scoring, and adaptive fusion, saving 24%-55% FLOPs across evaluated datasets and backbones.

  • Problem

    Large-image SISR has substantial memory and computation costs, while patch-based execution adds overhead despite differing patch restoration difficulties.

  • Method

    ENAF combines multiple early exits with a tiny PSNR estimator, blank vector generator, and adaptive patch fusion to select computation paths.

  • Results

    24%-55%, 43%-52%, 43%-48%, and 31%-52% FLOPs are saved on F2K, Test2K, Test4K, and Test8K, respectively, across evaluated SISR backbones.

  • Takeaways & Limitations

    ENAF provides a computation-performance trade-off across model scales and computational budgets for efficient large-image super-resolution.

  • Takeaways & Limitations

    Evaluation fixes PSNR and compares mean FLOPs, with inference trade-offs controlled by the hyperparameters γ and η.

Abstract

from arXiv · show

To accelerate single image super-resolution (SISR) networks on large images (2K-8K), many recent approaches decompose an image into small patches and dynamically determine an execution path according to its difficulty (referred to as a dynamic network). To quantify the hardness of a patch, they mainly rely on a handcrafted assessment score, e.g., edge, which weakly associates a patch's texture with the computational complexity of a SISR model. To address the problem, we introduce ENAF - a dynamic network for SISR with an adaptive patch fusion. Built on top of a backbone, ENAF incorporates multiple early exits (EEs) to tackle the over-parameterized SISR model. More importantly, ENAF plugs a tiny network that estimates PSNR to associate data texture with a computation cost at an EE. Based on the scores, ENAF effectively assigns image patches to an exit, enhancing the quality-complexity trade-off. Extensive experiments on common datasets with popular SISR backbones demonstrate the effectiveness of ENAF in various settings. The source code is provided in https://github.com/nmduonggg/ENAF

1. Introduction

Large-image SISR becomes costly because modern CNNs have memory and computation costs that grow quadratically with input size, while patch decomposition introduces overlap overhead. ENAF addresses this with multiple early exits, a PSNR estimator, a blank vector generator, and adaptive patch fusion for difficulty-aware patch processing.

  • Motivation: Modern CNN-based SISR algorithms have huge memory footprints and computation costs that grow quadratically with input size.The challenge is especially important for 2K, 4K, and 8K images.
  • Motivation: Patch-based execution reduces memory requirements but incurs computation overhead because neighboring patches overlap.Reported tile and overlap settings include {128,8}, {64,4}, and {32,2}.
  • ENAF: ENAF is a dynamic SISR network with adaptive patch fusion built on a backbone and multiple early exits.The early exits target over-parameterized SISR models.
  • ENAF: A tiny PSNR estimator predicts PSNR to associate patch texture with computation cost at an early exit.This replaces reliance on a handcrafted difficulty assessment alone.
  • ENAF: ENAF combines PSNR-estimator scores with a blank vector generator and trains adaptive patch fusion to assign patches effectively.The blank vector generator targets patches with little detail, which account for nearly 13% of large input images.

2. Background and Motivations

SISR has advanced through increasingly deep CNN architectures, but large-image applications face rapidly growing computation and memory costs. Dynamic networks therefore adapt processing to patch difficulty, while lightweight and sparse approaches target the resulting redundancy and expense.

  • CNN-based SISR: CNN-based SISR evolved from SRCNN’s three-layer network toward deeper or skip-connected architectures such as VDSR, SRResNet, and SRDenset.These developments improved SISR performance through increased depth, residual blocks, and dense blocks.
  • Efficiency motivation: Large-image SISR is costly because CNN memory and computation grow quadratically with input size.An SR-CNN with large inputs and feature maps requires ×1.5 computation workload and ×10 external memory bandwidth relative to a classification network.
  • Efficiency motivation: The increasing complexity of SISR models has motivated lightweight designs using methods such as neural architecture search and parameter pruning.The supplied passage identifies TPSR as a NAS-based network and mentions combining NAS with parameter pruning.
  • Dynamic super-resolution: Dynamic networks decompose large images into patches and select execution paths with different capacities according to each patch’s restoration difficulty.This strategy is used in super-resolution tasks because patches within a large image can have different processing requirements.
  • Redundancy reduction: Other SISR methods exploit fine-grained spatial redundancy through sparse masks, sparse convolution, or frequency-domain branches.SMSR applies sparsity across spatial and channel dimensions, while FAD uses high-frequency signals and a computationally intensive branch.

3. Methodology · 3.1. ENAF Architecture

ENAF is a dynamic, patch-based super-resolution architecture that splits low-resolution images into fixed-size patches, processes them through a network, and merges the resulting high-resolution patches. Its design combines multiple early exits, PSNR estimation, blank-vector generation, and adaptive patch execution or fusion.

  • 3.1. ENAF Architecture: ENAF splits a low-resolution image into N fixed-size 32×32 patches for dynamic super-resolution processing.Each patch x_i is processed by a network F(.) to generate a corresponding high-resolution patch y_i.
  • 3.1. ENAF Architecture: The generated high-resolution patches are merged to produce the final high-resolution image.The underlying SR network typically combines feature-extraction encoders with sub-pixel or deconvolution layers.
  • 3.1. Architecture: The network prediction is represented through the backbone’s encoder stages and its final upsampling mapping.The supplied formulation describes the original predictor as the final mapping after the encoder sequence.
  • 3.1. ENAF Architecture: ENAF comprises a backbone with multiple early exits, a PSNR estimator, a blank vector generator, and adaptive patch selection or fusion.These components address differing computational requirements across image patches.
  • 3.1.1 Backbone with Multi Early Exits: ENAF modifies SR backbones into multi-exit networks by placing K−1 additional learnable mappings along the inference graph.The original backbone predictor is the K-th mapping, while bicubic interpolation is the 0-th execution path.
  • 3.1.1 Backbone with Multi Early Exits: At each early exit, a deconvolution layer f_j follows the intermediate representation produced by the j-th encoder.The early exits are indexed by j ∈ [0, L).
  • 3.1.2 PSNR Estimator: The PSNR estimator U_j generates an early-exit signal from a sub-image to determine whether restoration is easy or difficult.This estimator is designed to provide more flexible patch fusion than strict hardness-group assignment.

3.2. ENAF Training

ENAF uses a two-stage training strategy: it first optimizes the backbone, then trains early-exit predictors while preserving the backbone’s prior knowledge. Afterward, a frozen-backbone PSNR estimator learns to predict each exit’s PSNR for hardness evaluation.

  • Backbone pre-training: The backbone is first pre-trained conventionally with an L1 loss to establish general knowledge and preserve performance consistency with the original backbone.This phase follows the standard training procedure for the main backbone.
  • Early-exit training: The second stage trains early-exit predictors using the first stage’s knowledge, deprioritizing the already optimized final exit while preserving learned weights.The multi-exit loss in Eq. (6) focuses on developing the early exits.
  • Early-exit training: 10^-8: the backbone uses a small learning rate in the second stage to prevent prior-knowledge distortion and performance degradation.This corresponds to assigning a smaller loss weight to the last exit.
  • PSNR estimator training: After the two-stage process, ENAF adds a PSNR estimator, freezes the super-resolution backbone, and trains the estimator to predict each exit’s PSNR using a hardness-evaluation loss.The PSNR value is computed by a function P(a, b) over the generated outcome and reference image pair.

3.3. ENAF Inference and Execution Path Selection

ENAF splits a large low-resolution image into equal-sized patches, estimates PSNR for each, and selects execution paths using predicted quality and precomputed computation costs. It also uses a blank-patch mechanism to route minimally textured patches toward simple processing while mitigating estimator bias.

  • Patch-wise inference: ENAF divides a large LR image into equal-sized patches, estimates their PSNR, and combines the resulting SR patches into the final image.Each exit’s computation cost is precalculated before inference.
  • Blank-patch handling: For blank patches, bicubic processing can outperform all backbone exits while requiring minimal computation.These patches contain minimal texture, motivating a dedicated route to a simple exit.
  • Blank-patch handling: The blank vector primarily prevents biased PSNR-estimator patterns for the simple branch by adjusting only the potentially biased predictor.This mechanism addresses the estimator’s tendency to assume that smaller exits perform worse.
  • Blank-patch handling: ENAF detects blankness with Laplacian edge information and uses a blank threshold of 10 to construct a blank vector.The vector has a non-zero first index only when the patch satisfies the blankness condition.
  • Computation-aware fusion: The computation-aware fusion favors execution paths with higher estimated PSNR and lower computational effort.The fusion incorporates normalized path costs and estimated PSNR values, with η controlling the performance–computation tradeoff.

4. Experiments · 4.1. Settings

ENAF is evaluated on large-image super-resolution using multiple backbones, standardized datasets and metrics, and fixed-PSNR FLOPs comparisons. Training uses staged optimization with patch preprocessing and augmentation.

  • 4.1.1 Training Datasets and Backbones: ENAF is built on FSRCNN, CARN, and SRResNet backbones representing small, medium, and large network sizes.The DIV2K training set uses images 0001–0800, which are decomposed into small patches and augmented through random rotation and flipping.
  • 4.1.2 Training Configurations: The first training phase pretrains the SR backbone for 30 epochs with batch size 16 using L1 loss and Adam optimization.The stated optimizer settings are η1 = 0.9 and β2 = 0.999.
  • 4.1.2 Training Configurations: Cosine annealing dynamically adjusts learning rates from 10^-3 toward a minimum of 10^-8, with reductions after each epoch.This schedule is part of the backbone pre-training phase.
  • 4.1.3 Benchmark and Evaluation Metric: Evaluation covers four large-size test images or datasets: Flickr2K, Test2K, Test4K, and Test8K.Test2K, Test4K, and Test8K are derived from DIV8K by downsampling.
  • 4.1.3 Benchmark and Evaluation Metric: Performance and computation are measured with Peak Signal-to-Noise Ratio (PSNR) and FLOPs.These are the two common metrics adopted for evaluation.
  • 4.1.3 Benchmark and Evaluation Metric: Figure 4 compares ENAF backbones with prior patch-based dynamic SISR methods in ×4 super-resolution on Test2K and Test4K examples.The figure states that ENAF’s main backbone achieves equivalent or higher PSNR than the original and SOTA methods.
  • 4.1.3 Benchmark and Evaluation Metric: Comparisons fix PSNR and measure the FLOPs required to achieve that value under standard protocols.Unless stated otherwise, FLOPs are averaged over all 32×32 low-resolution patches with ×4 super-resolution across the test set.

4.2. Results

ENAF consistently achieves a better computation–performance tradeoff than ClassSR and ARM across multiple SISR backbones and datasets. Its adaptive exit selection and blank-vector generation reduce computation while preserving restoration quality, especially for lightweight networks.

  • Overall comparison: ENAF consistently outperforms ClassSR and ARM by requiring less computation to achieve the same restoration performance.The comparison covers FSRCNN, CARN, and SRResNet backbones on conventional datasets.
  • Overall comparison: 4-12% of the original FLOPS is reduced by ENAF-L compared with ARM across all evaluated SISR backbones.The same efficiency trend also holds for the medium and small ENAF versions across multiple datasets.
  • Overall comparison: ENAF preserves performance through shared encoders that learn different mappings without significantly degrading super-resolution quality.Unlike ClassSR’s loss interference and ARM’s single parameter set for all subnets, ENAF supports scaling to more exit levels.
  • Trade-off analysis: ENAF-FSRCNN and ENAF-SRResNet achieve higher restoration performance with less computation than ARM in PSNR-FLOPS comparisons.These results indicate superior performance-cost tradeoffs and more effective execution-path selection, with especially considerable gains for FSRCNN.
  • Trade-off analysis: The blank vector helps lightweight FSRCNN prefer bicubic processing in simple cases, improving restoration with only slight performance differences.This addresses cases where lightweight learned networks struggle to outperform bicubic interpolation.

4.3. Ablation Study

The ablation study shows that the blank vector reduces ENAF’s computational cost, while the PSNR estimator and fusion parameter η are evaluated for accuracy and computation control. Removing blankness prior increases computation by roughly 5%–15% to achieve similar PSNR, and η primarily affects FLOPS below 1.0.

  • Blank Vector Ablation: Removing the blank vector increases ENAF-FSRCNN computation by around 5% to 15% to reach similar PSNR.The “-NB” variants use only PSNR and FLOPS during inference, without prior knowledge of patch blankness.
  • Blank Vector Ablation: The blank vector’s computational savings become more pronounced at higher image resolutions because larger images yield more sub-images.This supports biased simple predictors more effectively when patch count increases.
  • PSNR Estimator Accuracy: The PSNR estimator is assessed using MAE between predicted and actual PSNR across execution depths, with ARM-LUT as the comparison method.The evaluated levels are mapped to Bicubic, EE1–EE3 for ENAF and Bicubic, Subnet-S–Subnet-L for ARM-LUT.
  • Fusion Constraint Analysis: η has a negative correlation with FLOPS because it controls the computation-related constraint in ENAF’s fusion strategy.Across three backbones on Test8K, FLOPS drops sharply as η increases from 0 to 1.0, then remains nearly constant through η = 2.0.

5. Conclusion

ENAF accelerates large-image super-resolution with a dynamic network containing multiple early exits and adaptively selects inference paths using a tiny PSNR estimator and blank vector generator.

  • Conclusion: ENAF uses multiple early exits to accelerate large image super-resolution.The approach is designed as an effective dynamic network for practical efficient super-resolution.
  • Conclusion: A tiny PSNR estimator and blank vector generator enable adaptive execution-path selection during inference.The passage describes this strategy as simple yet effective.
  • Conclusion: ENAF achieves a favorable computation-performance trade-off across model scales and computational budgets.This trade-off supports applications requiring efficient super-resolution.
Loading 2608.15349v1…