PaperlayerView exact revision on arXiv

Source-linked AI summary

On Efficient Transformer-Based Image Pre-training for Low-Level Vision

Wenbo Li, Xin Lu, Shengju Qian, Jiangbo Lu, Xiangyu Zhang, Jiaya Jia

arXiv:2112.10175v2cs.CV

TL;DR

Low-level image-processing datasets are typically small, and principled analysis of how pre-training acts in these tasks has been lacking. The paper studies transformer pre-training strategies and their representational effects, finding task-specific performance effects and advantages for multi-related-task pre-training. It also reports state-of-the-art models for multiple low-level tasks.

  • Problem

    Low-level image-processing datasets typically contain only a few thousand images, but principled analysis of how pre-training acts in these tasks has been lacking.

    Source: §1 · p. 1

  • Method

    The paper evaluates transformer pre-training strategies across low-level tasks and diagnoses their effects on internal representations using centered kernel alignment.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Pre-training has task-specific effects, yielding significant improvements on SR and deraining but limited gains on denoising, while multi-related-task pre-training performs best among the tested methods.

    Source: §6 · pp. 14–15

  • Takeaways & Limitations

    Multi-related-task pre-training exhibits potential for exploiting image priors more efficiently than using larger pre-training datasets.

    Source: §6 · p. 15

    The study focuses on pre-training with synthesized data, and EDT uses fixed downsampling although adaptive downsampling based on degradation degrees may be a better choice.

    Source: §5 · p. 14

Abstract

from arXiv · show

Pre-training has marked numerous state of the arts in high-level computer vision, while few attempts have ever been made to investigate how pre-training acts in image processing systems. In this paper, we tailor transformer-based pre-training regimes that boost various low-level tasks. To comprehensively diagnose the influence of pre-training, we design a whole set of principled evaluation tools that uncover its effects on internal representations. The observations demonstrate that pre-training plays strikingly different roles in low-level tasks. For example, pre-training introduces more local information to higher layers in super-resolution (SR), yielding significant performance gains, while pre-training hardly affects internal feature representations in denoising, resulting in limited gains. Further, we explore different methods of pre-training, revealing that multi-related-task pre-training is more effective and data-efficient than other alternatives. Finally, we extend our study to varying data scales and model sizes, as well as comparisons between transformers and CNNs-based architectures. Based on the study, we successfully develop state-of-the-art models for multiple low-level tasks. Code is released at https://github.com/fenglinglwb/EDT.

1 Introduction

The paper addresses the lack of principled analysis of how pre-training works in low-level vision, where task datasets are typically small. It studies efficient transformer pre-training, finding task-specific representational effects and reporting gains from multi-related-task training and models with state-of-the-art performance across several tasks.

1 Introduction

  • The paper identifies a gap in principled analysis of pre-training for low-level vision, where image-processing datasets typically contain only a few thousand images.Source: §1 · p. 1
  • The study uses centered kernel alignment to diagnose how pre-training strategies affect internal representations across image-processing tasks.Source: §1 · p. 2
  • Pre-training improves performance by introducing different degrees of local information, treated as an inductive bias, to intermediate layers.The reported representational effects differ by task: pre-training mainly changes higher layers in SR and deraining, with limited impact on denoising.Source: §1 · p. 2
  • Multi-related-task pre-training brings more improvements across downstream tasks and is more data-efficient than purely increasing data scale.Source: §1 · p. 2
  • Both transformers and CNNs benefit from pre-training, while transformers obtain greater improvements and provide higher-performance baselines with fewer parameters.Source: §1 · p. 2
  • The proposed pre-trained models achieve state-of-the-art performance on super-resolution, denoising, and deraining.The paper presents a systematic study of pre-training effects and strategies in low-level vision.Source: §1 · p. 2

2 Encoder-Decoder-Based Transformer

EDT combines convolutional encoders and decoders with a transformer body, using different input paths to model long-range interactions at low resolution. The paper reports strong task results and provides four model variants for different scenarios.

2 Encoder-Decoder-Based Transformer

  • EDT improves ×4 SR by 0.49dB on Urban100 over IPT, while its 11.6M parameters are 10.0% of IPT's 115.6M and it uses 200K images, 15.6% of IPT's pre-training data.Source: §2 · p. 3
  • For level-50 Gaussian denoising, EDT uses 38 GFLOPs for 192×192 inputs versus SwinIR's 451 GFLOPs, and runs in 51.9ms versus 271.9ms.Source: §2 · p. 3

2.1 Overall Architecture

  • EDT combines convolutional encoders and decoders with a transformer body, processing high-resolution and low-resolution inputs through different paths.High-resolution inputs are downsampled and later upsampled; SR inputs retain their size, with an additional convolutional upsampler before the output.Source: §2.1 · p. 4
  • The transformer body models long-range relations at low resolution for computational efficiency, while skip connections support fast convergence during training.Source: §2.1 · p. 4

2.2 Architecture Variants

  • EDT-T, EDT-S, EDT-B, and EDT-L vary in channel, stage, and head counts in the transformer body.The reported parameter counts and FLOPs are for denoising at 192 × 192 size.Source: §2.2 · p. 4, Table 1 · p. 4

3 Study of Image Pre-training

The study evaluates transformer pre-training across low-level vision tasks, using CKA to examine internal representations and compare training strategies. It finds that multi-related-task pre-training is more effective and data-efficient than simply scaling single-task data, while pre-training’s representational effects vary by task and model size.

3.1 Pre-training on ImageNet

  • The study compares single-task, unrelated-task, and highly related-task pre-training, then fine-tunes using the task-specific encoder and decoder with the shared transformer body.The multi-task setups use a multi-encoder, multi-decoder, shared-body architecture; ImageNet is used for pre-training, typically with 200K images.Source: §3.1 · pp. 4–5

3.2 Centered Kernel Alignment

  • CKA compares network-layer representations using Gram matrices, with minibatch estimators used to reduce the memory burden of computing it across a dataset.The study uses batches of 300 and iterates over the test dataset 10 times.Source: §3.2 · p. 5

3.3 Representation Structure of EDT

  • SR and deraining representations form staged groups, while denoising is more uniform; single-task pre-training changes higher SR and deraining layers more than denoising layers.Denoising body layers are more similar to the second SR group than the third, and denoising representations change little with pre-training.Source: §3.3 · pp. 5–6

3.4 Single- and Multi-Task Pre-training

  • SR models progress from more local information in early layers to global information in higher layers, and multi-related-task pre-training produces the strongest reported performance improvements.Related-task pre-training changes more global representations than unrelated-task pre-training; the latter is attributed mainly to representation mismatch between unrelated tasks.Source: §3.4 · pp. 7–8

3.5 Effect of Data Scale on Pre-training

  • With multi-related-task pre-training on 200K images, EDT-B achieves new state of the arts on all benchmarks despite a smaller data scale and much fewer training iterations.Single-task pre-training scales data from 50K to 400K and doubles iterations at 400K so the data can be fully functional, increasing training burden.Source: §3.5 · p. 9

3.6 Effect of Model Size on Pre-training

  • Larger EDT models generally gain more from single-task pre-training, while the third CKA group grows with model size and dominates EDT-L’s map, suggesting redundancy.The reported gains include already strong EDT-L models.Source: §3.6 · p. 10, Figure 8 · p. 10

3.7 EDT v.s. ConvNets with Pre-training

  • EDT-B’s early layers resemble RRDB and RCAN more than its higher layers, and it obtains greater or comparable pre-training improvements with fewer parameters.The authors suggest transformers’ use of global information may explain their superiority over ConvNets.Source: Figure 9 · p. 10, §3.7 · p. 11, Figure 10 · p. 11

4 Experiments

Experiments evaluate the pre-training strategy across super-resolution, denoising, and deraining. Results vary by task: SR and deraining show strong gains, while denoising gains are limited.

4 Experiments

  • Experiments use multi-related-task pre-training for ×2, ×3, and ×4 SR, Gaussian denoising at g15, g25, and g50, and light- and heavy-rain deraining.The section covers classical and lightweight SR; lightweight SR generally refers to models with fewer than 1M parameters.Source: §4 · p. 11, §4.1 · p. 11

4.1 Super-Resolution Results

  • Up to 0.46 dB and 0.45 dB gains on Urban100 and Manga109, respectively, show pre-training improves classical SR across ×2, ×3, and ×4 scales.EDT-B also gains nearly 0.1 dB over SwinIR on multiple datasets without pre-training.Source: §4.1 · pp. 11–12, Table 4 · p. 12
  • EDT-T achieves the best lightweight SR results on all benchmark datasets, with nearly 0.2 dB and 0.4 dB improvements on Urban100 and Manga109 across scales.SwinIR uses 64 × 64 training patches versus EDT-T's 48 × 48 patches and initializes ×3 and ×4 models from its ×2 model.Source: §4.1 · p. 12, Table 5 · p. 13

4.2 Denoising Results

  • Pre-training yields only 0.02–0.11 dB gains in denoising, which the authors relate to sufficient task-specific training data and little change in internal representations.The authors suggest Gaussian denoising may not need a large training dataset.Source: §4.2 · pp. 12–13
  • EDT-B performs better at high noise levels but slightly worse at low levels; removing downsampling improves low-noise results, while EDT-B uses 8.4% of SwinIR's FLOPs.The authors attribute the low-noise difference possibly to downsampling and describe EDT as balancing performance and computational complexity.Source: §4.2 · pp. 13–14, Table 6 · p. 13

4.3 Deraining Results

  • EDT-B outperforms IPT by 0.52 dB on light rain and RCDNet by 2.66 dB on heavy rain.The comparison is reported on Rain100L and Rain100H, respectively.Source: §4.3 · p. 14

5 Limitations

The study focuses on pre-training on synthesized data and identifies fixed downsampling as a limitation of its encoder-decoder design.

5 Limitations

  • The study is limited to synthesized data and fixed downsampling; future work should examine real-world settings, more tasks, video processing, and adaptive downsampling by degradation degree.The authors note adaptive downsampling may be preferable for images with different degradation degrees.Source: §5 · p. 14

6 Conclusion

The conclusion finds that pre-training affects low-level tasks differently and improves intermediate representations by incorporating more local information. It also identifies multi-related-task pre-training as promising and more efficient than using larger pre-training datasets.

6 Conclusion

  • Pre-training significantly improves SR and deraining but yields limited gains on denoising, with stronger intermediate representations incorporating more local information.The authors describe multi-related-task pre-training as having potential to dig out image priors and as more efficient than using larger datasets.Source: §6 · pp. 14–15
  • Multi-related-task pre-training is more data-efficient than simply using larger pre-training datasets.The conclusion also reports examining effects of data scale and model size and comparing transformers with ConvNets.Source: §6 · p. 15

A Network Architecture

EDT combines a convolutional encoder and decoder with a transformer body that processes inputs at task-appropriate resolutions. Its shifted crossed local attention balances receptive-field growth and computational cost.

A Network Architecture

  • EDT combines convolutional encoder and decoder blocks with a transformer body; high-resolution inputs use additional downsampling and upsampling operations.The low-resolution path processes inputs at their original size, while the high-resolution path enables transformer modeling at reduced resolution.Source: A Network Architecture · p. 20, Figure A.1 · p. 20
  • The implementation sets the horizontal and vertical window dimensions using (h, w) = (6, 24), with the orientations reversed.The paper states that shifted crossed windows increase the effective receptive field rapidly.Source: Figure A.3 · p. 21, A Network Architecture · p. 22
  • Shifted crossed local attention is designed to balance receptive-field growth against the computational cost of global stripe attention.The design draws on shifted or halo windows and horizontal and vertical stripe attention.Source: A Network Architecture · pp. 20–21
  • The attention module splits channels between horizontal and vertical local-window attention, then projects their fused outputs; successive blocks shift the windows.The two window orientations use sizes (h, w) and (w, h).Source: A Network Architecture · pp. 21–22
  • Anti-FFN is introduced to eliminate possible blocking from window partitioning, using a depth-wise convolution with a 5×5 kernel.The block applies the anti-blocking operation between linear and activation operations.Source: A Network Architecture · p. 22

B Study on the Network Architecture

Larger and more receptive-field-efficient attention windows improve SR performance, while the anti-blocking FFN’s depth-wise kernels vary from low-pass filters in lower layers to diverse local-context filters in higher layers.

Window Size

Kernels in Anti-blocking FFN

  • Lower-layer Anti-FFN kernels resemble low-pass filters, while higher-layer kernels learn diverse local contexts.The lower-layer filters act like anti-aliasing filters against possible blocking from window splitting; removing the anti-blocking design causes nearly a 0.1 dB drop on both Urban100 and Manga109 in ×2 SR.Source: B Study on the Network Architecture · pp. 22–23

C Datasets

The experiments use ImageNet pre-training and task-specific datasets for SR, denoising, and deraining, with separate training and fine-tuning schedules.

C Datasets

  • SR uses DIV2K and Flickr2K for training and five benchmarks for evaluation; denoising and deraining use their own specified training and evaluation sets.Pre-training uses 200K ImageNet images unless otherwise specified; deraining excludes 546 Rain100H training images sharing background contents with test images.Source: C Datasets · pp. 23–24

D Training Details

  • Experiments use eight RTX 2080Ti GPUs except multi-task pre-training and denoising without downsampling, which use eight V100 GPUs; all models use L1 loss.Source: D Training Details · p. 24
  • Pre-training runs for 500K iterations, while fine-tuning and scratch-training schedules vary by task.Fine-tuning lasts 500K iterations for SR, 800K for denoising, and 200K for deraining; scratch-training settings differ, including a 1600K-iteration denoising setup without downsampling.Source: D Training Details · pp. 24–25

E Single- and Multi-Task Pre-training

Multi-related-task pre-training improves results across low-level tasks, while representation changes after fine-tuning vary by task. Full ImageNet pre-training yields state-of-the-art EDT-B results across benchmarks, and visual comparisons report richer textures and more regular structures.

E Single- and Multi-Task Pre-training

F Before and After Fine-tuning

  • Fine-tuning mainly changes higher-layer SR representations, has little effect on denoising, and changes deraining representations little when its fine-tuning set contains only hundreds of low-resolution images.For deraining, pre-training changes representations more clearly than fine-tuning, and the pre-trained model outperforms other methods by a large margin.Source: F Before and After Fine-tuning · pp. 26–27, Figure E.8 · p. 26

G Full ImageNet Pre-training

  • With full ImageNet pre-training, EDT-B sets a new state of the art on all benchmark datasets; this setting uses 1.28M images.The paper presents this as an exploration of the upper bound for single-task pre-training.Source: G Full ImageNet Pre-training · p. 27, Table G.2 · p. 27

H Visual Comparison with SOTA Methods

Loading 2112.10175v2…