Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary

1. Introduction

• Single-image super-resolution estimates a high-quality image from a degraded low-quality image, an ill-posed task that can extend the capabilities of cost-effective sensors and outdated equipment.Each low-resolution image may correspond to many possible high-resolution images.Source: §1 · p. 1
• Transformers have advanced image SR, but local-window attention limits receptive fields, while enlarging windows increases computational burden.The introduction also notes that general-purpose computations do not account for image content.Source: §1 · pp. 1–2
• The auxiliary token dictionary supplies prior information to image tokens and summarizes image-wide information through cross-attention.The dictionary's auxiliary tokens represent common image structures and support linear complexity in image size.Source: §1 · p. 2
• Category-based self-attention groups image tokens by similarity to dictionary tokens, allowing long-range interactions beyond local-window boundaries.This uses similar image parts rather than rectangular local windows.Source: §1 · p. 2
• The combined token-dictionary and category-based attention model achieves superior super-resolution results over existing state-of-the-art methods.The introduction states that this is achieved without substantially increasing model complexity.Source: §1 · p. 2

2. Related Works

• CNN architectures improved image SR through learned LR-to-HR mappings and architectural changes including deeper networks, recursive structures, and residual blocks.Later work also incorporated channel attention and sparse attention.Source: §2 · p. 2
• Transformer SR methods use shifted-window, group-wise multi-scale, sparse, and anchored attention, alongside pretraining, ConvFFN, and larger windows to expand receptive fields.The passage describes these as approaches aimed at improving performance.Source: §2 · p. 2
• The paper proposes TDCA for external priors and AC-MSA for long-range dependencies, combining them with window-based attention to integrate local, global, and external information.Source: §2 · p. 2

3.1. Motivation

• Dictionary learning motivates the method: it models local image priors, while quadratic self-attention encourages window limits that constrain receptive fields.The method uses their connection to incorporate dictionary learning into the Transformer framework.Source: §3.1 · p. 3

3.2. Token Dictionary Cross-Attention

• TDCA uses a learned token dictionary to select similar keys and combine their values, injecting external prior information into image features.The method uses normalized cosine similarity and SoftMax to form the attention weights.Source: §3.2 · pp. 3–4

3.3. Adaptive Dictionary Refinement

• ADR initializes a dictionary in the first layer, then refines it using enhanced image tokens to combine external priors with image-specific information.The refined dictionary summarizes information across the image and implicitly carries category information.Source: §3.3 · pp. 4–5

3.4. Adaptive Category-based Attention

• AC-MSA groups tokens by their most-similar dictionary item, splits uneven categories into fixed-size sub-categories, and attends within groups to connect similar features globally.The sub-category division improves parallelism while preserving a global receptive field through random shuffling.Source: §3.4 · pp. 5–6

3.5. The Overall Network Architecture

• ATD combines TDCA, AC-MSA, and shifted-window attention in parallel, summing their features before later layers and pixel shuffle produce the HR estimate.The three attention branches leverage external, global, and local features, respectively.Source: Figure 3 · p. 5, §3.5 · p. 6

4.1. Experimental Settings

• Ablations use rescaled ATD-light trained for 250k DIV2K iterations; the full ATD uses six blocks, 128 dictionary tokens, and sub-category size ns = 128.Evaluation for the ablations uses Set5, Urban100, and Manga109.Source: §4.2 · p. 6, §4.1 · p. 6

4.2. Ablation Study

• TDCA and ADR together improve results by 0.11 dB on Urban100 and 0.13 dB on Manga109; the final model with AC-MSA reaches 26.51 / 30.98 dB on those benchmarks.The ablation models were trained for 250k iterations on DIV2K and evaluated on Set5, Urban100, and Manga109.Source: §4.2 · pp. 6, 8, Table 1 · p. 6
• Random category-based attention beats the baseline, while adaptive category-based attention adds 0.05–0.08 dB over random categorization.The authors attribute the additional gain to more precise categorization and partitioning.Source: Table 2 · p. 6, §4.2 · p. 8
• AC-MSA improves significantly as sub-category size ns reaches 128, then gains slowly; increasing dictionary size initially adds 0.06 −0.08 dB, but M = 96 degrades performance.The final model uses ns = 128 to balance performance and computational resource consumption.Source: Table 3 · p. 6, §4.2 · p. 8

4.3. Comparisons with State-of-the-Art Methods

• 0.20–0.25 dB PSNR gains over HAT on Urban100 are reported for ATD across zooming factors at comparable parameter size.The comparison covers the classical SR task.Source: Table 4 · p. 7, §4.3 · p. 8
• 0.45 dB gains on ×4 Manga109 over OmniSR accompany better ATD-light results on all benchmark datasets at similar model size.The comparison is for the lightweight SR task.Source: Table 5 · p. 7, §4.3 · p. 8
• Visual comparisons show ATD recovering sharper edges and cleaner textures with fewer artifacts, while category visualizations group visually or semantically similar pixels.The examples include severely degraded inputs and category maps for trees, shrubs, roofs, and smooth textures.Source: §4.3 · p. 8, Figure 4 · p. 9, §4.5 · p. 9

4.4. Model Size and Computational Burden Analysis.

• ATD achieves better SR results than HAT at comparable model size and complexity, and outperforms CAT-A by up to 0.22 dB with 10% more parameters and FLOPs.The model-size analysis compares restoration accuracy, parameter count, and FLOPs.Source: Table 6 · p. 8, §4.4 · p. 9

5. Conclusion

• ATD-SR achieves state-of-the-art results on single-image super-resolution benchmarks.The paper also reports ablations demonstrating the effectiveness of its token dictionary, adaptive refinement, and adaptive category-based attention.Source: §5 · p. 9
• Adaptive dictionary refinement uses the preceding layer’s similarity map to fit learned dictionary priors to each input image.The dictionary provides external supplementary information for estimating missing high-quality details.Source: §5 · p. 9
• Category-based self-attention groups input features and connects similar structures across the image beyond local-window boundaries.The approach uses the external prior embedded in the token dictionary to categorize features.Source: §5 · p. 9

Supplementary Material

• The supplement covers ATD and ATD-light training details, followed by AC-MSA illustrations and additional model visualizations.These topics are presented in Sections A and B, respectively.Source: Supplementary Material · p. 12

A. Training Details

• ATD trains on DF2K, using 64 × 64 LR patches for 300k iterations before 96 × 96 patches for another 250k iterations.For ×3 and ×4, training starts from the well-trained ×2 model and uses only the second-stage finetuning; warm-up iterations are used at each stage.Source: A. Training Details · p. 12
• ATD-light trains on DIV2K for 500k iterations at ×2, then finetunes ×3 and ×4 for 250k iterations from the ×2 model.Unlike ATD, it does not use the larger patch size; the remaining training settings are kept the same as ATD.Source: A. Training Details · p. 12

B.1. More Visualization of AC-MSA.

• Pixels are sorted and classified into categories by attention-map value, then each category is flattened and concatenated sequentially.Misassignments to the same sub-category are limited to at most M = 128, much less than the number of sub-categories HW/ns, and are reported to have almost no performance impact.Source: §B.1 · p. 12
• Visual examples show the Categorize operation grouping similar textures in both natural and cartoon images.Only a few categories are visualized for each image for simplicity.Source: §B.1 · p. 12

• ATD and ATD-light both yield better visual results than the compared methods.The comparisons cover classical methods for ATD and lightweight methods for ATD-light.Source: §B.2 · p. 12, Figure B.3 · p. 14, Figure B.4 · p. 15
• ATD recovers sharper edges in img011 and img027, where other methods remain blurry.Source: §B.2 · p. 12
• ATD-light reconstructs the black blocks in Donburakokko more accurately as rectangles matching the ground truth.Source: §B.2 · p. 12
• The categorization examples show similar-texture areas such as sky, grass, and roof grouped into the same category.Pixels in each category are flattened and sorted for further subdivision.Source: Figure B.2 · p. 13