PaperlayerView exact revision on arXiv

Source-linked AI summary

N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution

Haram Choi, Jeongmin Lee, Jihoon Yang

arXiv:2211.11436v3cs.CV

TL;DR

Swin-based SR is limited by plain WSA’s local receptive field, while many SR methods are computationally intensive. The paper introduces N-Gram context, applies it in NGswin and SwinIR-NG, and reports competitive efficient performance for NGswin and state-of-the-art lightweight SR results for SwinIR-NG.

  • Problem

    Plain WSA overlooks broad regions because its receptive field is limited, and many deep learning SR methods require intensive computation.

  • Method

    The paper defines N-Gram context as interacting neighboring Swin local windows using sliding-WSA, and builds NGswin with hierarchical encoding and an SCDP bottleneck.

  • Results

    SwinIR-NG outperforms current best lightweight SR approaches and establishes state-of-the-art results, while NGswin achieves competitive results with an efficient structure.

  • Takeaways & Limitations

    N-Gram context extends Swin-based SR beyond individual windows and is reported to improve NGswin, SwinIR, and HNCT.

  • Takeaways & Limitations

    Ablation studies on other benchmarks were omitted from the main paper because of page limits and placed in the appendix.

Abstract

from arXiv · show

While some studies have proven that Swin Transformer (Swin) with window self-attention (WSA) is suitable for single image super-resolution (SR), the plain WSA ignores the broad regions when reconstructing high-resolution images due to a limited receptive field. In addition, many deep learning SR methods suffer from intensive computations. To address these problems, we introduce the N-Gram context to the low-level vision with Transformers for the first time. We define N-Gram as neighboring local windows in Swin, which differs from text analysis that views N-Gram as consecutive characters or words. N-Grams interact with each other by sliding-WSA, expanding the regions seen to restore degraded pixels. Using the N-Gram context, we propose NGswin, an efficient SR network with SCDP bottleneck taking multi-scale outputs of the hierarchical encoder. Experimental results show that NGswin achieves competitive performance while maintaining an efficient structure when compared with previous leading methods. Moreover, we also improve other Swin-based SR methods with the N-Gram context, thereby building an enhanced model: SwinIR-NG. Our improved SwinIR-NG outperforms the current best lightweight SR approaches and establishes state-of-the-art results. Codes are available at https://github.com/rami0205/NGramSwin.

1. Introduction

The paper addresses Swin-based super-resolution’s limited receptive field and intensive computation by introducing N-Gram context and efficient architectures. NGswin targets efficient SR, while SwinIR-NG applies the context to existing Swin-based models and achieves state-of-the-art lightweight SR results.

  • Motivation: Plain window self-attention has a limited receptive field, while many leading SR networks require intensive computation.The paper motivates reducing operations for practical deployment while retaining roughly million-parameter-scale models.
  • N-Gram context: N-Gram context models interactions among neighboring local windows through sliding-WSA, expanding WSA’s receptive field for degraded-pixel recovery.Channel-reducing group convolution produces uni-Gram embeddings to reduce the complexity of N-Gram interaction.
  • NGswin: NGswin combines hierarchical encoder stages, an asymmetrically small decoder, NSTBs, and an SCDP bottleneck for efficient SR.The SCDP bottleneck takes multi-scale encoder outputs, and the architecture targets ×2, ×3, and ×4 tasks.
  • Model improvements: The N-Gram context improves other Swin Transformer-based SR models, including SwinIR-light and HNCT.The paper presents this as a second track alongside the design of NGswin.
  • Results: SwinIR-NG outperforms the compared lightweight SR methods and establishes state-of-the-art results on lightweight SR.The cited introduction passages state the headline result without reporting a numerical metric.

2. Related Work

Prior efficient SR methods reduce computation with architectural, attention, or pruning strategies, while prior N-Gram uses span context in language and selected vision settings. This paper distinguishes its image formulation by modeling bidirectional 2D interactions among local windows in a single image.

  • Efficient SR: Efficient SR methods have used cascading residual blocks, information distillation, lattice filters, channel-reducing Transformers, CNN–Swin combinations, pruning, and group-wise multi-scale attention.The related-work passage lists CARN, IMDN, LatticeNet, ESRT, SwinIR-light, HNCT, SRPN-Lite, and ELAN-light as examples.
  • N-Gram: Language-model N-Gram represents consecutive characters or words and captures longer sentence context statistically.The passage also notes N-Gram embeddings, recurrent networks, attention, and BERT-style encoders as ways to model N-Gram information.
  • N-Gram in image: Unlike earlier vision uses, this paper’s N-Gram represents bidirectional 2D information among neighboring local windows in a single image.The paper defines a uni-Gram as a non-overlapping Swin local window and an N-Gram as a larger window containing neighboring uni-Grams.

3. Methodology

The methodology expands Swin's limited receptive field through N-Gram interactions among neighboring local windows, then integrates this context into an efficient hierarchical SR architecture. NGswin combines sliding-WSA with multi-scale encoding, an SCDP bottleneck, and a small decoder to balance reconstruction context and computation.

  • Problem verification: Plain WSA can produce distorted reconstructions because its shallow-layer receptive field is confined to a local window and cannot use surrounding patterns.The paper links this limitation to homogeneous deeper-layer feature patterns and mismatched neighboring regions.
  • N-Gram definition: N-Gram defines each Swin local window as a uni-Gram and combines neighboring windows into larger forward, backward, or bi-directional contexts.For N = 2, neighboring windows are selected at lower-right or upper-left positions relative to the target window.
  • NGswin architecture: NGswin uses an asymmetric U-Net with three hierarchical encoder stages, patch-merging, an SCDP bottleneck, a small decoder, and reconstruction module.The SCDP bottleneck receives multi-scale outputs from the shallow module and encoder stages, while patch-merging reduces feature resolution and attention computation.
  • NSTB: NSTB implements N-Gram context with scaled-cosine window self-attention, post-normalization, and sliding computation over uni-Gram embeddings.The default local window size is M = 8, and the implementation computes forward and backward N-Gram features with padding for boundary regions.
  • N-Gram interaction: Sliding-WSA lets neighboring uni-Gram embeddings interact before window partitioning, expanding WSA's receptive field for recovering degraded pixels.The method uses channel-reducing group convolution to form uni-Gram embeddings, reducing the cost of N-Gram interaction.

4. Experiments

Experiments evaluate training and benchmark settings, efficient SR performance, N-Gram ablations, and architectural choices. NGswin and SwinIR-NG achieve strong results, while ablations clarify efficiency and reconstruction trade-offs.

  • Experimental Setup: NGswin was trained on DIV2K image pairs with L1 loss and evaluated using PSNR and SSIM on five benchmark datasets.Evaluation used Y-channel metrics with bicubic MATLAB degradation.
  • Efficient SR Results: NGswin outperformed previous leading efficient SR models on all benchmarks, with PSNR margins up to 0.3dB over SRPN-Lite and 0.19dB over FMEN.The comparison also reported training data, Mult-Adds, and parameter counts.
  • Lightweight SR Results: SwinIR-NG established state-of-the-art lightweight SR results on all benchmarks against current lightweight methods.Its reduced-channel ×4 variant remained better than the compared models with the fewest computations.
  • N-Gram Ablation: N-Gram context produced especially strong gains on ×3 and ×4 tasks and Urban100 or Manga109, reaching a PSNR margin up to 0.24dB over the non-N-Gram model.The ablation also found generally increased SSIM and robustness on Urban100 and Manga109.
  • N-Gram Interaction: Bi-directional N-Gram interaction with sliding-WSA gave the best performance-efficiency trade-off among tested interaction settings.Sliding-WSA also computed correlations within each N-Gram, unlike sliding-window convolution.
  • Architecture Ablation: SCDP preserved performance when hierarchical encoder stages were added by using outputs from multiple resolutions, whereas extra stages alone reduced performance.The comparison involved models with the same number of NSTBs but different stage configurations.

5. Conclusion

The paper introduces N-Gram context for Transformer-based low-level vision and applies it to efficient SR models. NGswin achieves competitive results, while SwinIR-NG reaches state-of-the-art lightweight SR performance.

  • Conclusion: N-Gram context expands Swin Transformer receptive fields through sliding-WSA, enabling NGswin, SwinIR, and HNCT to address broad-region limitations.The paper also uses uni-Gram embeddings and reports reduced operations through its architectural components.
  • Conclusion: NGswin combines the proposed components to achieve competitive results against previous leading SR methods.The conclusion attributes efficiency to the hierarchical encoder, small decoder, and uni-Gram embedding.
  • Conclusion: SwinIR-NG establishes state-of-the-art results among the compared lightweight SR methods.The authors suggest future applications to denoising, deblurring, and broader Transformer architectures.

A.1. Sequentially Reflected Window Padding

Sequentially reflected window padding supplies reflected neighboring windows for forward and backward N-Gram features. The method preserves the input data size and therefore requires no extra Mult-Adds operations.

  • Padding Design: For N-Gram size N, sequentially reflected window padding uses (N −1) rows and columns of neighboring windows at the lower-right and upper-left boundaries.The padding generates forward and backward N-Gram features from uni-Gram embeddings.
  • Padding Design: The padding lets uni-Grams interact with reflected neighbors instead of trivial zero-padding values.The same padding size is applied on opposite sides for the forward and backward features.
  • Efficiency: Sequentially reflected window padding adds no extra Mult-Adds because it provides the same number of 32-bit float inputs as zero padding.Figure A illustrates the uni-Gram embedding before padding and the two directional padded features for N=2.
  • Directional Interaction: Sliding-WSA uses shared weights for the bi-directional N-Gram features.The forward and backward features are formed using the corresponding reflected padding arrangements.

A.2. Within-Stage Residual Connections

The decoder uses residual connections within stages and receives a fused contribution from the SCDP bottleneck and the first encoder stage. This design differs from CARN’s dense local cascading.

  • Residual Connections: Within-stage residual connections link NSTBs and patch-merging layers in the encoder and decoder, except at the third encoder and decoder stages.The design uses residual connections rather than CARN-style dense connections within stages.
  • Decoder Mapping: The input to the k-th decoder NSTB is denoted z_k−1 dec, and its mapping function is F_k dec.The decoder mapping is described for 1≤k≤K_dec.
  • Decoder Mapping: The decoder input combines the SCDP bottleneck output z_scdp with the last NSTB output from the first encoder stage.This is expressed as z_dec = z_scdp + z_K1 enc1 before the decoder processing described in the appendix.
  • Reconstruction Module: The reconstruction-module diagram distinguishes channel changes from Pixel Shuffle scale factors using separate parenthetical annotations.The figure specifies that a → b denotes network-dimension change, while r denotes the upscaling factor.

A.3. Reconstruction Module

The reconstruction module is the only component that differs across the ×2, ×3, and ×4 models, mainly through output channels and pixel-shuffling scale.

  • The ×2, ×3, and ×4 models differ only in their reconstruction modules.The module varies the first convolution’s output channels and the pixel-shuffler’s scale factor.

B. Experimental Setup Details and Findings

The experiments evaluate learning strategies and implementation settings for NGswin, while emphasizing that the reported strategy findings are suggestions rather than universal rules.

  • The study presents its learning-strategy findings as helpful considerations for future research, not absolute truths.The findings are summarized in Table A.
  • Model Architecture: NGswin uses six, four, four, and six NSTBs across its three encoder stages and decoder.The network dimension is 64, FFN hidden dimension 128, window size 8, and N-Gram size 2.
  • For ×3 and ×4 training, warm-start, data-derived standard-deviation normalization, de-normalization before loss, half learning-rate decay, and no regularization performed better in the reported comparisons.Table A lists these choices among the better strategies for NGswin.
  • Cosine learning-rate decay underperformed in the authors’ observations, which they hypothesize may reflect faster learning-rate reduction and underfitting.They also report that decay points set too early or too late reduced performance.

C. Other Ablation Studies

Additional ablations examine Swin Transformer versions, padding, warm-start training, and benchmark-wide consistency, alongside visual comparisons with and without N-Gram context.

  • Swin Transformer Version: SwinV2 outperforms SwinV1 for NGswin’s super-resolution tasks in the reported ablation.The authors attribute this to scaled-cosine attention limiting domination by particular pixels.
  • Warm-start: The authors report that warm-start training is superior to scratch training in both time resources and performance for super-resolution.This comparison is made for SwinIR-NG.
  • Other Benchmarks: Results across the full benchmark ablations consistently show positive effects from each proposed or employed approach.The additional results are collected in Table C.

E. Discussions and Limitations

The discussion positions NGswin as the most operation-efficient model and SwinIR-NG as the strongest-performing lightweight model, while identifying parameter, operation, dataset, and task-scope boundaries.

  • Performance and Efficiency: NGswin has the fewest Mult-Adds operations, whereas SwinIR-NG presents the best performance among the compared lightweight SR methods.The parameter sizes of NGswin and SwinIR-NG are 4.04MB and 4.74MB, respectively.
  • NGswin: SCDP compensation helps offset the performance loss associated with NGswin’s hierarchical encoder.The SCDP bottleneck uses multi-scale encoder outputs.
  • NGswin: Removing patch-merging layers would increase operations by about 17G for the ×4 task in the specified encoder stages.This estimate concerns eight NSTBs in the second and third encoder stages.
  • SwinIR-NG: SwinIR-NG uses slightly more operations than SwinIR-light and has 4.74MB of parameters, while establishing state-of-the-art lightweight SR results.The authors explicitly identify its higher parameters and operations than ELAN-light as a limitation.
  • Scope and Other Tasks: The experiments focus on super-resolution of bicubic low-resolution images, while blind SR and other restoration tasks remain outside the primary study.Classification and ST-VQA are shown only as secondary visualizations.
  • Scope and Other Tasks: Visualizations suggest sharper SwinIR-NG outputs may support CIFAR10 classification and improve scene-text detection, but they do not compare against other models.The CIFAR10 and ST-VQA figures are explicitly described as secondary provisions.
Loading 2211.11436v3…