PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Polarization Image Restoration with General Restoration Priors

Chenggong Li, Jinhao Liu, Caiyun Wu, Yidong Luo, Junchao Zhang, Degui Yang

arXiv:2609.30728v1cs.CVeess.IV

TL;DR

Real-world polarization acquisition is vulnerable to degradations that complicate restoration, and polarization training data have limitations. The paper combines pretrained general restoration priors with polarization-specific modeling in an all-in-one framework. It reports stronger fidelity than general all-in-one models and several task-specific methods, while noting compute overhead and marginal gains over task-specific experts.

  • Problem

    Real-world polarization acquisition is vulnerable to blur, low illumination, and noise, whose effects are amplified in derived polarization parameters and challenge accurate restoration.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    The method uses normalized Stokes representations in a dual-branch architecture, with pretrained general restoration priors and mixture-of-experts in the intensity branch and adaptive transfer to the polarization branch.

    Source: §1 · p. 3

  • Results

    Under the all-in-one setting, the method substantially outperforms general all-in-one models and surpasses several task-specific methods in fidelity.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    PolarCDD provides a benchmark covering 11 composite-degradation settings to support all-in-one polarization restoration research.

    Source: §1 · p. 3

    Gains over task-specific expert models remain marginal, and the polarization branch adds computational overhead, especially with Transformer-based backbones.

    Source: §A.9 · p. 19

Abstract

from arXiv · show

Polarization imaging captures distinctive surface and geometric cues that benefit a wide range of vision tasks. However, real-world polarization acquisition is often affected by multiple coupled degradations, making image restoration essential for practical polarization vision. Existing methods are largely tailored to specific degradations and remain constrained by the limited scale and quality of polarization data. To address these limitations, we develop an all-in-one polarization restoration framework for diverse and composite degradations. We first study the impact of different polarization representations on restoration performance and identify the normalized Stokes representation as an effective choice for separating intensity and polarization information. Accordingly, we devise a dual-branch architecture that separates intensity and polarization modeling. To overcome the limitations of polarization-specific training, the intensity branch leverages pretrained general restoration priors and a mixture-of-experts extension for composite degradations, while its restoration knowledge is adaptively distilled into the symmetric polarization branch via a cross-domain feature transform. In addition, we establish a composite-degradation polarization benchmark to support all-in-one restoration research. Extensive experiments on public datasets and our proposed benchmark demonstrate the effectiveness of the proposed method.

1 Introdution

Polarization restoration is needed because real-world acquisition degradations compromise polarization images, while existing methods face data and model-design limits. The paper proposes an all-in-one framework that combines general restoration priors with polarization-specific modeling and introduces a composite-degradation dataset.

1 Introdution

  • Blur, low illumination, and noise make polarization images vulnerable to real-world degradation, limiting practical applications and motivating restoration.Polarization cues include reflectance, surface roughness, and geometry, and benefit downstream vision tasks.Source: §1 · pp. 1–2
  • Existing polarization datasets lag behind general restoration data, representation choices remain under-studied, and all-in-one restoration across diverse degradations is largely unexplored.Polarization deblurring relies mainly on synthetic blur kernels, while the acquisition process restricts dataset scale and diversity.Source: §1 · pp. 2–3
  • The framework uses normalized Stokes representations and separate intensity and polarization branches, with pretrained restoration priors, mixture-of-experts, and adaptive cross-domain transfer.The intensity branch also feeds complementary polarization cues back through zero-convolution modulation.Source: §1 · p. 3
  • PolarCDD provides a dataset with 11 degradation settings spanning blur, low light, noise, mosaic artifacts, and their combinations.The dataset is introduced to support all-in-one and composite polarization restoration research.Source: §1 · p. 3

2 Related Work

Prior work spans task-specific polarization restoration and general restoration models for diverse degradations. The paper identifies limited polarization data, task-specific designs, under-studied representations, and unexplored all-in-one restoration as open gaps.

2 Related Work

  • Polarization restoration includes intensity-guided and dual-domain methods, plus task-specific approaches to denoising and demosaicking; a recent dual-domain design applies across restoration tasks (Zhou et al. [2026]).Source: §2 · p. 3
  • Despite this progress, polarization restoration remains constrained by limited data and task-specific designs; representation choices have little systematic study, and all-in-one restoration remains largely unexplored.Source: §2 · p. 3
  • General restoration has moved toward unified models using prompts, degradation-aware representations, mixture-of-experts routing, and broader generative priors, primarily for conventional RGB images.Source: §2 · pp. 3–4

3 METHODOLOGY

The method selects normalized Stokes representations for restoration and introduces a composite-degradation dataset and dual-domain network. Its design transfers pretrained intensity restoration priors to polarization features while also feeding polarization cues back into intensity features.

3.1 Preliminaries

  • Polarization measurements at multiple angles yield Stokes-derived intensity, DoLP, and AoP; degradation and noise can amplify errors in polarization parameters, so restoration must recover intensity and estimate them accurately.The degraded observation is modeled using arbitrary degradation operators and noise.Source: §3.1 · p. 4

3.2 Representation Exploration

  • Normalized Stokes [S0, ˜S1, ˜S2] gives the best overall polarization accuracy across three tasks; its DoLP-weighted AoP encoding also improves reliable-region MAEr over [S0, p, cos(2ϕ), sin(2ϕ)].The representation decouples polarization from intensity, and DoLP reduces the influence of noise in low-DoLP regions.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 Composite Degradation Polarization Image Dataset

  • PolarCDD combines low light, motion blur, noise, and mosaic artifacts into 11 degradation settings, each with 50,000 training images and 100 test images.Noise includes synthetic and real observations, while low-light data are captured using a DoT system.Source: §3.3 · p. 5

3.4.1 Overview

  • A U-Net processes normalized Stokes inputs through stacked IPRBs, using pretrained RGB restoration weights for intensity and textual embeddings to condition restoration.The architecture also reduces the polarization branch's block count by half for efficiency.Source: §3.4.1 · p. 5, Figure 3 · p. 6

3.4.2 IPRB

  • Within each IPRB, separate branches process representations; the intensity branch uses four pretrained InstructIR experts with top-1 selection, while polarization streams are concatenated before attention.The branches use NAFNet-style processing to reduce interference from distribution differences across domains.Source: §3.4.2 · p. 6

3.4.3 IP-ATM

  • IP-ATM gates transferable features to modulate polarization representations with intensity-derived affine parameters, then uses zero-convolution modulation to feed polarization cues back into intensity features.The zero initialization starts the reverse interaction in an identity-like state, preserving pretrained restoration behavior at optimization's start.Source: Figure 4 · p. 6, §3.4.3 · pp. 6–7

4 Experiment

The method produces cleaner polarization restorations and, under the AIO setting, outperforms general AIO models while surpassing several task-specific methods in fidelity. Ablations support bidirectional interaction, selective intensity-to-polarization transfer, pretrained restoration priors, and the MoE extension.

4.1 Implementation Details

  • Training uses 200K iterations on either mixed single-degradation datasets or PolarCDD, with L1 supervision for intensity and Stokes/DoP and cosine loss for AoP.The two model settings use training and test sets drawn from the same dataset.Source: §4.1 · p. 8

4.2 Comparison with State-of-the-Art Methods

  • Under the AIO setting, the method substantially outperforms general AIO models and surpasses several task-specific methods in fidelity.Qualitative comparisons also report cleaner results, sharper structural details, and fewer noise artifacts.Source: §4.2 · p. 9

4.3 Ablation Study

  • Bidirectional interaction improves intensity metrics with polarization-to-intensity modulation and polarization metrics with intensity-to-polarization transfer; gating and stronger modulation also matter.The ablation reports clearer polarization feature structure and reduced feature noise after interaction, without transferring intensity-specific responses.Source: Table 4 · p. 8, §4.3 · p. 9, Figure 6 · p. 9
  • Removing pretrained restoration priors substantially reduces performance, while the MoE extension improves all metrics and robustness to degradations outside the pretraining distribution.The passage names demosaicking as an example of an out-of-distribution degradation.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents a unified network for diverse polarization degradations, pairing normalized Stokes representations with separate intensity and polarization modeling. It reports effective results on public datasets and a composite-degradation benchmark.

5 Conclusion

  • Normalized Stokes representations are advantageous for polarization restoration, particularly under low-intensity conditions and in reliable polarization regions.The conclusion connects this choice to an intensity-polarization decoupled dual-branch architecture.Source: §5 · p. 10
  • The network introduces general restoration priors into the intensity domain and selectively transfers suitable intensity features into the polarization branch.The conclusion frames these choices as addressing limitations in existing polarization data distributions.Source: §5 · p. 10
  • The authors establish a composite-degradation polarization dataset and report effectiveness on public datasets and their proposed benchmark.Source: §5 · p. 10

A Appendix

The appendix supplies implementation and analysis material for the proposed network, dataset, and experiments.

A Appendix

  • The appendix covers network architecture, dataset construction, and implementation details for compared methods.Source: A Appendix · p. 13
  • It analyzes model complexity, restoration-prior choices, polarization-branch design, and the mixture-of-experts mechanism.Source: A Appendix · p. 13
  • It presents loss-formulation ablations, additional qualitative results, and a discussion of limitations and future directions.Source: A Appendix · p. 13

A.1 Network Architecture Details

The architecture processes intensity and polarization representations in separate branches, with cross-domain interaction and intensity-side restoration controls.

A.1 Network Architecture Details

  • The intensity branch extends a NAFNet-style backbone with four experts initialized from pretrained InstructIR weights for composite-degradation handling.The experts are mixed with the main branch through a convolutional layer; the MoE module is used in the decoder.Source: §A.1 · p. 13
  • Each IPRB uses separate branches for the representations, which are processed independently and interact across domains through IP-ATM.The polarization representations are temporarily aggregated during attention, then split for independent processing and merged again for information transfer.Source: §A.1 · p. 13
  • Textual restoration guidance modulates only intensity features, while IP-ATM propagates the restoration mode to the polarization domain.A fixed prompt is used for each degradation setting, and embeddings can be precomputed offline so the text encoder is removed at inference.Source: §A.1 · p. 13

A.2 PolarCDD Dataset Construction

PolarCDD combines four basic polarization degradations and their composites into 11 settings, using DoT images and both synthesized and captured degradation data. Its construction controls how degradations are combined and separates training and test scenes.

A.2 PolarCDD Dataset Construction

  • DoT images are used to isolate degradation effects, especially mosaic artifacts; 256 × 256 crops train the model and 512 × 512 crops test it, with no scene shared across splits.Most source images exceed 1024 × 1024 resolution.Source: §A.2 · pp. 13–14
  • Motion blur is synthesized by averaging frames warped along pixel-wise paths derived from random camera-motion trajectories.Low-light data are captured at four exposure levels; noise combines Gaussian synthesis and real-noise data, while mosaic observations are synthesized with a forward imaging operator.Source: §A.2 · pp. 14–15
  • 11 settings cover low light, blur, noise, and mosaic artifacts, both individually and in specified combinations.The combinations include low light+blur+mosaic and blur+noise+mosaic.Source: Figure 7 · p. 14, §A.2 · p. 15
  • Composite samples are built by selecting normal-exposure or low-light scenes, then applying blur synthesis, noise addition, and mosaic generation.Extra blur or noise is omitted for real-noise images, and extra noise is omitted for low-light scenes.Source: §A.2 · p. 15

A.3 Compared Methods Implementation Details

The comparison methods are reproduced from official implementations or paper configurations, with training time and GPU memory kept comparable where possible. The baselines span common restoration architectures, prompt- and instruction-conditioned models, and expert or polarization-specific designs.

A.3 Compared Methods Implementation Details

  • NAFNet, Restormer, and CDIR use normalized-Stokes inputs, with the supplied configurations specifying their backbone depths and channel or attention designs.NAFNet and CDIR take nine channels; Restormer uses MDTA and GDFN blocks.Source: §A.3 · p. 15
  • PromptIR, InstructIR, and AdaIR represent prompt-, instruction-, and adaptive-frequency-based restoration baselines.InstructIR processes four RGB polarization angles independently, while PromptIR and AdaIR use nine-channel inputs and outputs.Source: §A.3 · p. 15
  • MoCE-IR uses frequency-domain mixture-of-experts adapters, while SSMD jointly processes four-angle images and Stokes features.MoCE-IR processes four RGB angles independently with four experts and top-k=1 routing.Source: §A.3 · p. 15

A.4 Complexity Analysis

The proposed dual-branch MoE model has the largest parameter count, but its convolution-dominated backbone retains competitive memory use, computation, and inference speed. Transformer attention can impose substantial computational and memory overhead despite relatively few parameters.

A.4 Complexity Analysis

  • The proposed model has the largest parameter count, yet maintains competitive memory consumption, computational cost, and inference speed.The authors attribute this balance to its convolution-dominated backbone despite the dual-branch and MoE design.Source: §A.4 · p. 15, Table 6 · p. 16
  • Transformer-based Restormer and SSMD have relatively few parameters, but their attention operations introduce substantial computational and memory overhead.Complexity measurements use 512 × 512 images on an NVIDIA RTX 4090 GPU.Source: §A.4 · p. 15, Table 6 · p. 16
  • The complexity comparison reports GPU memory, FLOPs, and latency, with nine-channel normalized-Stokes input adaptation denoted “9ch.”The caption specifies the measurement image size and GPU.Source: Table 6 · p. 16

A.5 Analysis of Restoration Priors

Across the additional visual evaluations, the method improves restoration on real-world blur, mosaic, and composite degradations, while ablations examine its priors, polarization branch, losses, and MoE. The authors also identify marginal gains over task-specific experts and computational cost as limitations.

A.5 Analysis of Restoration Priors

  • Pretrained NAFNet restores real-world polarization motion blur better visually than a polarization-only expert, and transferring its priors further improves results under real-world degradations.The authors select InstructIR as the restoration prior for its performance, generalization, and lightweight backbone that is easier to extend.Source: §A.5 · p. 16, Figure 8 · p. 16

A.6 Polarization Branch Design

  • Halving the polarization branch's block count causes only a 0.1 dB PSNRp drop with nearly unchanged MAE, while the tiny branch performs consistently worse.Removing IP-ATM from the full polarization branch causes a substantial drop, falling below both half and tiny variants.Source: §A.6 · p. 17, Table 8 · p. 17

A.7 Loss Function Ablation

  • Adding Stokes and polarization losses improves polarization metrics while maintaining intensity performance, whereas removing the angle loss substantially degrades intensity metrics.The selected loss configuration supervises intensity and polarization parameters derived analytically from normalized-Stokes predictions.Source: §A.7 · p. 17, Table 9 · p. 18

A.8 MoE Analysis

  • MoE experts are all used with distinct specialization patterns, and MoE consistently improves all metrics under AIO3; on PolarCDD, gains are limited for intensity metrics but improve polarization-related metrics.The MoE extension helps adapt to demosaicking, which is absent from the pretrained restoration prior, without disrupting that prior.Source: §A.8 · pp. 18–19, Figure 9 · p. 18, Table 10 · p. 18

A.9 Limitations and Future Work

  • The method's gains over task-specific expert models remain marginal, and its polarization branch adds computational overhead, especially with Transformer-based backbones.The authors describe the increased computational cost as a trade-off for stronger restoration performance.Source: §A.9 · p. 19
  • Restoration priors remain constrained by the distribution and scale of general restoration datasets compared with large-scale vision corpora.The authors identify incorporating large-scale generative priors as future work, noting instability with a lightweight polarization branch and deployment costs when scaling it to Stable Diffusion capacity.Source: §A.9 · p. 19

A.10 Additional Visual Results

  • On real-world PV blur, real-world polarization mosaics, and composite degradations, the method produces cleaner or more effective visual restorations than compared approaches.For the mosaic dataset, visual quality is comparable to task-specific experts; joint low-light enhancement and demosaicking also better recover texture while reducing noise in polarization parameters.Source: §A.10 · p. 19, Figure 11 · p. 20, Figure 12 · p. 21
Loading 2609.30728v1…