PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers for Single Image Dehazing

Yuda Song, Zhuqing He, Hui Qian, Xin Du

arXiv:2204.03883v1cs.CV

TL;DR

Image dehazing lacks Transformer designs tailored to recovering haze-free images, despite the importance of the task and CNNs’ recent dominance. The paper proposes DehazeFormer by modifying Swin Transformer components and evaluates multiple variants across datasets. Its models achieve strong benchmark results, including over 40 dB PSNR on SOTS indoor, alongside a realistic remote-sensing dataset for non-homogeneous haze.

  • Problem

    Vision Transformers have not brought new dimensions to image dehazing, whose goal is to estimate latent haze-free images from hazy images.

  • Method

    DehazeFormer modifies Swin Transformer with RescaleNorm, ReLU, reflection-based shifted windows, parallel convolution, soft reconstruction, and SK fusion.

  • Results

    DehazeFormer substantially outperforms contemporaneous methods with lower overhead, while DehazeFormer-L exceeds 40 dB PSNR on SOTS indoor.

  • Takeaways & Limitations

    The proposed variants work across evaluated datasets, including non-homogeneous haze in the realistic remote-sensing RS-Haze dataset.

Abstract

from arXiv · show

Image dehazing is a representative low-level vision task that estimates latent haze-free images from hazy images. In recent years, convolutional neural network-based methods have dominated image dehazing. However, vision Transformers, which has recently made a breakthrough in high-level vision tasks, has not brought new dimensions to image dehazing. We start with the popular Swin Transformer and find that several of its key designs are unsuitable for image dehazing. To this end, we propose DehazeFormer, which consists of various improvements, such as the modified normalization layer, activation function, and spatial information aggregation scheme. We train multiple variants of DehazeFormer on various datasets to demonstrate its effectiveness. Specifically, on the most frequently used SOTS indoor set, our small model outperforms FFA-Net with only 25% #Param and 5% computational cost. To the best of our knowledge, our large model is the first method with the PSNR over 40 dB on the SOTS indoor set, dramatically outperforming the previous state-of-the-art methods. We also collect a large-scale realistic remote sensing dehazing dataset for evaluating the method's capability to remove highly non-homogeneous haze.

I. INTRODUCTION

Image dehazing estimates haze-free images from hazy observations, but haze impairs visibility and can undermine machine-vision reliability. DehazeFormer adapts Swin Transformer designs for dehazing and reports strong performance with lower overhead while introducing a realistic remote-sensing dataset.

  • Haze reduces visibility, affects object judgment, and can impair traffic safety and machine-vision reliability.
  • Image dehazing estimates the latent haze-free image from an observed hazy image.
  • Prior-based methods constrain the ill-posed problem but can produce unrealistic images and artifacts when scenes violate their assumptions.
  • DehazeFormer replaces unsuitable Transformer components with RescaleNorm, ReLU, reflection-based shifted windows, and convolutional spatial aggregation.
  • Our small model defeats FFA-Net with only 25% #Param and 5% computational cost, while the large model is the first method over 40 dB on SOTS indoor.
  • The paper collects a large-scale realistic remote-sensing dehazing dataset because existing non-homogeneous haze datasets are too small and unlike natural scenes.

II. RELATED WORKS

Prior-based dehazing can fail outside empirical assumptions, while learning-based methods and Transformer architectures motivate DehazeFormer. The paper combines a modified U-Net with specialized fusion, reconstruction, normalization, and attention-related components.

  • Related dehazing methods: Prior-based methods can output unrealistic results when scenes do not satisfy their empirical statistics, whereas learning-based methods have recently dominated dehazing.
  • Dehazing datasets: Synthetic and real dehazing datasets provide training and evaluation data, with synthetic datasets generated from haze-free images and depth maps.
  • Vision Transformers: Vision Transformers address limitations of convolutional architectures through token-based processing, while Swin Transformer uses windowed attention for linear computational cost.
  • DehazeFormer architecture: DehazeFormer uses a modified 5-stage U-Net whose convolutional blocks are replaced by DehazeFormer blocks.
  • Training: DehazeFormer trains on paired hazy and haze-free images using only an L1 loss.
  • DehazeFormer architecture: The architecture includes SK fusion and soft reconstruction in place of concatenation fusion and global residual learning.

B. Rescale Layer Normalization

LayerNorm can discard relationships between image patches and harm dehazing reconstruction. DehazeFormer instead normalizes the whole feature map and restores its mean and standard deviation through RescaleNorm.

  • B. Rescale Layer Normalization: LayerNorm normalizes image-patch tokens separately, discarding relative brightness and contrast between patches.The paper links these statistics to image brightness and contrast.
  • B. Rescale Layer Normalization: LayerNorm produces block artifacts in a simple autoencoder trained to reconstruct one image.The autoencoder uses patch embedding, normalization, and patch reconstruction layers.
  • B. Rescale Layer Normalization: RescaleNorm normalizes the entire feature map and reintroduces the mean and standard deviation lost during normalization.The saved statistics are transformed into learned scaling and bias parameters at the residual block’s end.
  • B. Rescale Layer Normalization: The normalization layer before the MLP is removed because doing so hardly worsens performance.This also avoids computing mean and standard deviation during inference as required by LayerNorm.

C. Nonlinear Activation Function with Simple Inversal

The paper argues that activation invertibility matters for image dehazing, where GELU performs poorly relative to simpler alternatives. It proposes SoftReLU as a smooth, controllable approximation to ReLU.

  • C. Nonlinear Activation Function with Simple Inversal: GELU performs worse than ReLU and LeakyReLU in image dehazing, which the paper attributes to its difficult inversion and stronger nonlinearity.GELU is non-monotonic, while ReLU and LeakyReLU are piecewise linear and easier to invert.
  • C. Nonlinear Activation Function with Simple Inversal: SoftReLU is a simple smooth approximation to ReLU formed as an excess between GELU and ReLU.Its shape parameter α controls the approximation; α = 0 gives ReLU, while α = 0.1 mimics GELU.
  • C. Nonlinear Activation Function with Simple Inversal: LeakyReLU performs similarly to ReLU, better than SoftReLU and GELU, while SoftReLU performs better than GELU.These comparisons come from the paper’s activation-function ablation studies.

D. Shifted Window Partitioning with Reflection Padding

Cyclic-shift windows create smaller masked windows at image edges, which is undesirable for dehazing because edge pixels matter. The proposed reflection-padding scheme preserves full-window aggregation and supplements window attention with convolutional neighborhood aggregation.

  • D. Shifted Window Partitioning with Reflection Padding: Cyclic shift produces smaller edge windows through masked attention, reducing the available spatial context at image boundaries.The paper considers edge information as important as center information for image dehazing.
  • D. Shifted Window Partitioning with Reflection Padding: Reflection padding avoids masking and prevents unreasonable inter-patch interactions during shifted window partitioning.The scheme introduces additional computational cost relative to cyclic shift.
  • D. Shifted Window Partitioning with Reflection Padding: The edge-region overhead becomes proportionally smaller as processed images become larger.The paper notes that dehazing networks typically process images larger than training patches.
  • D. Shifted Window Partitioning with Reflection Padding: The spatial aggregation scheme combines window attention with convolution over the pre-partition feature map.Convolution aggregates neighborhood information without considering window partitioning, while attention remains within each window.
  • D. Shifted Window Partitioning with Reflection Padding: Reflection padding is used only when shifted window partitioning is required, and convolution can be depth-wise or a ConvBlock.Some blocks also omit MHSA and RescaleNorm.

F. Implementation Details

DehazeFormer is implemented as five size variants with stage-specific attention and convolution choices. Training uses cropped patches, variant-dependent batch sizes and learning rates, while RS-Haze extends evaluation to realistic non-homogeneous remote-sensing haze.

  • F. Implementation Details: DehazeFormer provides five variants: tiny (-T), small (-S), basic (-B), middle (-M), and large (-L).These variants differ in their detailed architecture specifications.
  • F. Implementation Details: The attention ratio specifies the percentage of blocks containing MHSA, with those blocks placed at the end of each stage.Small models use DWConv (K = 5), whereas large models use ConvBlock (K = 3).
  • F. Implementation Details: Training randomly crops images into 256 × 256 patches and uses variant-dependent mini-batch sizes and initial learning rates.The mini-batch sizes are {32, 16, 16, 16, 8}, and the initial learning rates are {4, 2, 2, 2, 1} × 10^-4 for {-T, -S, -B, -M, -L}.
  • F. Implementation Details: RS-Haze is a larger and more realistic synthetic remote-sensing dehazing dataset for highly non-homogeneous haze.Remote-sensing imagery provides a practical non-homogeneous dehazing setting because its haze is highly non-homogeneous.
  • F. Implementation Details: The remote-sensing haze model allows spatially varying medium thickness and wavelength- and haze-condition-dependent transmission.The formulation treats the medium thickness d(x) as spatially varying but consistent over channels.

B. Synthesis Pipeline

The synthesis pipeline models non-homogeneous haze using cirrus-channel information, corrects atmospheric light estimates, and adjusts haze-dependent scattering and dense-haze information loss.

  • Transmission-map extraction: The cirrus channel generates a transmission map that characterizes spatially non-homogeneous natural haze.The method uses the reflectance of channel 9 to generate t1(x).
  • Transmission-map extraction: A 0.1% linear stretch removes the cirrus channel’s dark level so the maximum transmission does not remain below 1.Without correction, the dark level acts like additional homogeneous haze.
  • Atmospheric-light correction: Atmospheric-light correction refines the estimated atmospheric light used in the synthesis pipeline.Figure 6 compares atmospheric lights before and after correction and shows three synthesis samples.
  • Haze-dependent scattering: The exponent γ(x) is modeled as a cubic function of haze reflectance and clipped to [0, 4] to avoid outliers.The fitted coefficients are a0 = 6.537, a1 = −27.465, a2 = 41.224, and a3 = −21.547.
  • Dense-haze synthesis: A decay factor ξ = 1.25 attenuates haze-free-image information so sufficiently dense synthesized haze can completely remove that information.This addresses residual information that otherwise remains under dense haze in the imaging model.

C. Dataset Details

The study uses RESIDE benchmarks alongside a remote sensing dataset built from multispectral imagery, cirrus-channel haze information, and cropped image patches.

  • RS-Haze construction: RS-Haze is constructed from 76 atmospheric-corrected haze-free multispectral images and 108 cloudy images used to generate transmission maps.The source images come from Landsat-8 Level 1 data products.
  • RS-Haze construction: The dataset contains 6000 haze-free 512×512 multispectral patches and 1500 cirrus-channel patches with distributions similar to natural haze.Each haze-free image generates nine synthetic hazed images.
  • Benchmark datasets: The experiments use the RESIDE dataset and the authors’ RS-Haze dataset for image dehazing evaluation.RESIDE includes RESIDE-V0, RESIDE-Standard, and RESIDE-β, with ITS, OTS, and SOTS among its commonly used subsets.

1) RESIDE-Full:

The RESIDE-Full evaluation trains indoor and outdoor models separately on ITS and OTS, measures computational overhead, and reports broad gains for DehazeFormer variants.

  • Experimental setup: Indoor models train on 13,990 ITS image pairs and outdoor models on 313,950 OTS image pairs, then test on 500 indoor or outdoor SOTS pairs.The setup follows FFA-Net and evaluates indoor and outdoor scenes separately.
  • Experimental setup: The mixed RESIDE-6K setup uses 3,000 ITS and 3,000 OTS training pairs, resized to 400 × 400, with a 1,000-image SOTS-mix test set.All models are retrained using L1 loss for 1,000 epochs.
  • Evaluation measures: Overhead is measured using parameter count and multiply-accumulate operations, with MACs measured on 256 × 256 images.The reported measures are #Param and MACs.
  • Quantitative comparison: DehazeFormers overall outperform the compared baselines in the quantitative results.The comparison is reported in Table IV across the RESIDE experimental settings.
  • Interpretation of setups: RESIDE-Full indoor performance primarily measures high-frequency handling, outdoor performance reflects convergence speed, and RESIDE-6K measures stability and low-frequency extraction.RS-Haze is described as measuring semantic-feature extraction.

1) RESIDE-Full:

DehazeFormer achieves strong indoor and remote-sensing results, while the authors identify resolution mismatch and haze heterogeneity as important evaluation factors.

  • RESIDE-Full: DehazeFormer-B surpasses all baselines in PSNR and SSIM on the SOTS indoor set, while DehazeFormer-L exceeds 40 dB PSNR.The authors identify this as the first reported method exceeding 40 dB PSNR on SOTS indoor.
  • RESIDE-Full: Outdoor SOTS results may reflect convergence speed because the training set contains more than 300,000 sample pairs and models may not have converged.The authors state that performance on this set still has substantial room for improvement.
  • RESIDE-6K: CNN-based networks perform worse than DehazeFormers on RESIDE-6K, which the authors associate with inconsistent high-frequency distributions between resized training images and test images.The authors connect this setting to convolution’s high-frequency filtering and attention’s low-frequency filtering properties.
  • RS-Haze: On RS-Haze, DehazeFormer-S surpasses FFA-Net among the baselines, while remote-sensing results show higher PSNR but lower SSIM than other setups.The authors attribute the pattern to more monotonous scenes and highly non-homogeneous haze.
  • Qualitative results: In qualitative SOTS-mix comparisons, DehazeFormer-S restores clearer images while retaining texture and color information with the least haze residual.Comparative methods often fail where haze density varies significantly.

2) RS-Haze:

On RS-Haze, DehazeFormer-S better reproduced color and detail under thin haze and removed dense haze more effectively than competing networks. Ablations further show that normalization and shifted-window design choices materially affect performance, although results vary across datasets.

  • RS-Haze: DehazeFormer-S better reproduced color and detail than GCANet, PFDN, and FFA-Net under thin haze on RS-Haze.AOD-Net produced severe artifacts, while the competing methods were less effective at color and detail reproduction.
  • RS-Haze: DehazeFormer-S removed dense haze on RS-Haze, whereas the other evaluated networks produced apparent artifacts.The comparison specifically highlights the water-surface area in the third row of Fig. 8.
  • Ablation Study: DehazeFormer-T outperformed DehazeFormer-A, while using fewer parameters and lower computational cost than the ablation model.The comparison was conducted on the RESIDE-Full indoor scene, with DehazeFormer-A constructed specifically for ablation studies.
  • Ablation Study: Avoiding inter-patch relativity loss and reintroducing feature-map statistics improved network performance, with normalization more critical for MHSA than MLP.The study removes the normalization layer before MLP because it has no significant performance impact and requires nontrivial standard-deviation computation.
  • Ablation Study: Reflection-padding shifted windows moderately improved performance, whereas zero padding caused a significant drop by introducing meaningless tokens.The proposed scheme added negligible computational cost on 256 × 256 images.
  • Ablation Study: Activation functions strongly affected performance: ReLU and LeakyReLU were similar, while SoftReLU caused significant degradation and GELU was difficult to invert.These findings came from replacing nonlinear activations throughout the network.

4) Parallel conv:

The paper finds that convolutional spatial aggregation is most effective when placed in parallel with attention, while soft reconstruction and selective-kernel fusion provide additional improvements. These design choices contribute to DehazeFormer’s superior performance across datasets.

  • Parallel conv: Parallel convolution dramatically improved performance, but its placement was critical within the Transformer block.Convolution before the MLP brought only minor gains, while parallel placement with attention performed better.
  • Parallel conv: Convolution parallel to attention outperformed convolution parallel to MHSA because both operations aggregate spatial information in the same feature space.DWConv supplies static learnable aggregation weights, whereas attention supplies dynamic all-positive aggregation weights.
  • Other components: Soft reconstruction improved performance more than expected, while SK fusion delivered only a minor gain but offered lower overhead than concatenation fusion.The authors interpret soft constraints on the prior as beneficial.
  • Conclusion: DehazeFormer achieved superior performance on several datasets after incorporating improvements to normalization, activation, window partitioning, and spatial aggregation.The conclusion identifies these changes as adaptations of Swin Transformer for image dehazing.
Loading 2204.03883v1…