PaperlayerView exact revision on arXiv

Source-linked AI summary

LLPR: Location-aware learning and physics-based reconstruction for raindrop removal from a single image

Zewei He, Xingyu Liu, Xing Luo, Guizhong Fu, Zixuan Chen, Yu Chen, Jinlei Li, Zhe-Ming Lu

arXiv:2609.30758v1cs.CV

TL;DR

Raindrops can obscure and distort background scenes, and directly recovering the clear image is highly ill-posed when the raindrop components are unknown. LLPR combines training-only location-aware learning with physics-based reconstruction and cycle loss. The paper reports state-of-the-art performance and generality across its evaluated methods and datasets.

  • Problem

    Recovering a clear background from raindrop-degraded images is highly ill-posed when the raindrop components are unknown, and existing learned methods may fail under specific conditions.

    Source: §I · p. 1, C. Physics-based reconstruction scheme · p. 4

  • Method

    LLPR teaches the encoder raindrop locations with a training-only branch and estimates transparency and raindrop layers for physics-based reconstruction, with cycle-loss supervision.

    Source: §I · p. 2

  • Results

    LLPR variants using DRSformer and Restormer are reported to achieve state-of-the-art performance, with experiments demonstrating the framework's effectiveness and generality.

    Source: 1) Quantitative comparisons · p. 7, §V · p. 9

  • Takeaways & Limitations

    The framework can be combined with existing restoration networks, and its location-aware branch adds no resources during testing because it is removed at inference.

    Source: §I · p. 2

Abstract

from arXiv · show

Raindrops can cause occlusion and distortion in the background scenes due to their adherence to windows or camera lenses. Existing raindrop removal methods concentrate on designing sophisticated CNN or Transformer architectures to recover distorted and missing texture. In this paper, we try to integrate location information and physical model into off-the-shelf CNN or Transformer architectures to help improve their performance. Specifically, we notice that existing methods deploy a preprocessing sub-network to generate a binary or soft mask to indicate the raindrop location, which will increase the network parameters and computational complexity. In contrast, a location-aware learning branch is embedded to teach the encoder in the training phase with the capability of perceiving the position of the raindrops. Note that this location-aware learning branch can be removed during the inference process (achieving performance improvements at no cost). Furthermore, instead of directly reconstructing the raindrop-free image (i.e., background scene), we devise a physics-based reconstruction scheme to first learn the transparency matrix and the raindrop layer. The latent background layer is then reversely derived based on the physical model. By combining the above-mentioned components, we propose our location-aware learning and physics-based reconstruction (LLPR) framework for this challenging ill-posed problem. We also collect a real-world raindrop-degraded image dataset, which is challenging for single-image raindrop removal (SIRR) methods. Extensive experimental results demonstrate the effectiveness and generality of our LLPR framework, achieving superior performance against state-of-the-art SIRR methods. The code will be made available upon acceptance.

I. INTRODUCTION

Raindrops obscure and distort background scenes, while existing removal methods either rely on direct restoration or use location priors that add computational cost. LLPR adds training-only location learning and physics-based reconstruction to existing restoration networks, and the paper reports strong performance and generalization to wild images.

I. INTRODUCTION

  • Raindrops on windows and lenses obscure and distort background scenes, degrading image quality and potentially affecting downstream vision tasks.The paper notes that such capture conditions have become increasingly common with smartphones and outside-mounted surveillance cameras.Source: §I · p. 1
  • Existing approaches either directly restore images or locate raindrops before restoration, with prior extraction and integration adding computational cost.The paper also notes that direct-restoration methods neglect the physical model and may fail on complex, varied real-world images.Source: §I · p. 2
  • Physics-based reconstruction estimates transparency A and raindrop effect R, then uses them to derive the raindrop-free prediction under the formation model.A and R are learned without supervision, while only the derived prediction is supervised; the design softly constrains their correlation.Source: §I · pp. 1–2
  • A cycle loss uses estimated A and R to supervise the rainy input image and is described as an effective self-supervised strategy.The framework is presented as compatible with existing restoration networks including DRSformer and Restormer.Source: §I · p. 2
  • LLPR(DRSformer) is reported to outperform state-of-the-art raindrop-removal methods on real-world wild images and to generalize strongly to them.The framework can be combined with existing restoration networks rather than requiring a new CNN or Transformer module.Source: §I · p. 2
  • A training-only location-aware branch teaches the encoder to perceive raindrop positions and can be removed at inference without added testing resources.Unlike prior location-aware methods, the auxiliary branch is independent of the backbone network.Source: §I · p. 2

II. RELATED WORK

The related-work section groups single-image raindrop-removal approaches into multi-task, restoration, and task-specific methods. Multi-task methods aim to handle several weather degradations with one trained model, using different encoder-decoder designs.

II. RELATED WORK

  • Existing single-image raindrop-removal approaches are categorized as multi-task, restoration, or task-specific methods.The section then begins with multi-task methods.Source: §II · p. 2

A. Multi-task methods

  • Multi-task methods aim to handle multiple weather degradations, such as rain, raindrops, and haze, with one trained model.Source: A. Multi-task methods · p. 2
  • Examples include All-in-One's task-specific encoders with a shared decoder, a transformer encoder-decoder for varied weather, and a unified framework trained in two stages.The two-stage strategy explores weather-general and weather-specific features.Source: A. Multi-task methods · p. 3

B. Restoration methods

The section reviews restoration approaches and introduces LLPR’s U-net-like backbone and training-only location-aware branch. The branch teaches encoder features to perceive raindrop positions without adding inference-time computation.

B. Restoration methods

  • General restoration methods use versatile architectures across degradation types, including Transformer, dual-residual, multi-axis MLP, and wavelet-diffusion approaches.The surveyed methods train degradation-aware models with a shared architecture for different degradation types.Source: B. Restoration methods · p. 3

C. Task-specific methods

  • Task-specific methods target raindrops alone or jointly address raindrops and rain streaks, using attention, masks, cascades, or Transformer designs.Examples include binary or soft raindrop masks and multi-scale attention, as well as unified rain-streak and raindrop systems.Source: C. Task-specific methods · p. 3

A. Basic architecture

  • LLPR uses a four-level U-net-like encoder-decoder backbone with off-the-shelf restoration blocks rather than designing sophisticated Transformer blocks.Three down-sampling and three corresponding up-sampling operations connect the four levels.Source: A. Basic architecture · p. 3

B. Location-aware learning branch

  • The location-aware branch trains encoder features to perceive raindrop positions, using GT–I as supervision, then is removed after training to avoid added inference time.Earlier location-map approaches use additional sub-networks and can add computation during training or testing.Source: B. Location-aware learning branch · pp. 3–4

C. Physics-based reconstruction scheme

Directly predicting a raindrop-free image is ill-posed because the transparency matrix and raindrop layer are not predefined. LLPR estimates those components and derives the background image through a physics-based reconstruction scheme.

C. Physics-based reconstruction scheme

  • Direct prediction of the raindrop-free image is highly ill-posed because the transparency matrix A and raindrop layer R are not predefined.The paper notes that learned models relying on particular architectures or extracted cues may fail under specific conditions.Source: C. Physics-based reconstruction scheme · p. 4
  • LLPR estimates A and R, then derives the background image B, supervising the derived image rather than the individual components.The authors say this introduces physical priors and softly constrains the correlation between A and R.Source: C. Physics-based reconstruction scheme · pp. 4–5
  • The authors state that explicitly modeling raindrop formation can improve removal under actual conditions.They use mean absolute error to update the parameters.Source: C. Physics-based reconstruction scheme · p. 5

D. Cycle loss function

The cycle-loss design synthesizes a degraded image from estimated raindrop components and uses its agreement with the input to help learn those components.

D. Cycle loss function

  • Given estimated A and R, the method synthesizes a raindrop-degraded image, addressing the input image as well as the raindrop-free target.The passage introduces the synthesis equation after noting that prior supervision focuses on the raindrop-free image.Source: D. Cycle loss function · p. 5
  • When A and R are sufficiently accurate, the synthesized image can approximate the input, so cycle loss on that image facilitates learning A and R.The cycle loss supervises the synthesized image against the observed input.Source: D. Cycle loss function · p. 5
  • The implementation sets λ1 to 1 and λ2 to 0.1 as weights for balancing the losses.Source: D. Cycle loss function · p. 5

E. Test-wild: raindrop-degraded images in rainy weather

Test-wild is a 226-image collection of naturally formed raindrops captured in rainy weather through windows and windshields. It is intended for testing and has no aligned ground-truth images.

E. Test-wild: raindrop-degraded images in rainy weather

  • Unlike prior datasets with synthetic drops or water sprayed onto glass, Test-wild contains naturally formed raindrops captured in actual rainy weather.The authors distinguish it from Qian’s real-world dataset, which used a relatively ideal, flat-glass setup with a 2–5 cm camera distance.Source: E. Test-wild: raindrop-degraded images… · p. 5
  • Test-wild is intended solely for testing and does not include corresponding aligned ground truths.Source: E. Test-wild: raindrop-degraded images… · p. 5
  • 226 images were captured behind vehicle windshields or glass windows on rainy days, with varied camera poses, angles, and distances.The image acquisition used Sony ILCE-7RM4A and Nikon D7100 cameras and several iPhones.Source: E. Test-wild: raindrop-degraded images… · p. 5

IV. EXPERIMENTS

The experiments use Qian’s paired raindrop dataset, two test splits, and a newly collected real-world test set. They evaluate image quality with PSNR and SSIM and report the training and augmentation setup.

1) Datasets:

  • On Qian’s dataset, LLPR uses 861 image pairs for training and the rest for testing, comprising Test-a (58 pairs) and Test-b (249 pairs); it is also tested on collected Test-wild.Source: 1) Datasets · pp. 5–6

2) Evaluation metrics:

  • PSNR and SSIM measure image quality, with both metrics calculated on the luminance channel of YCbCr for fair comparison.Source: 2) Evaluation metrics · p. 6

3) Implementation details:

  • Models use AdamW and train for 300K iterations, with 256 × 256 random crops and horizontal and vertical flips for augmentation.The initial learning rate is 3e−4 for the first 100K iterations and then follows cosine annealing to 1e−6.Source: 3) Implementation details · p. 6

B. Ablation study

Ablations assess the framework’s components and the location-branch attachment point. Each component improves PSNR in the reported DRSformer example, while level 4 gives the best location-branch placement.

1) Overall:

  • 0.13 dB, 0.17 dB, and 0.08 dB PSNR gains are reported for physics-based reconstruction, location-aware learning, and cycle loss, respectively, with DRSformer.The SSIM trajectories on Test-a visually demonstrate the designs’ effectiveness.Source: Table I · p. 6, 1) Overall · p. 6, Figure 5 · p. 6

2) Position of location-aware learning branch:

  • Level 4 gives the best location-branch performance because it contains the richest features; deploying the branch at all four levels reduces performance.The authors attribute that drop to possible disruption of gradient backpropagation from multiple location losses.Source: 2) Position of location-aware learning… · p. 6

3) Supervision of the learned location map M:

The ablations compare supervision choices for the learned location map and location-learning styles. Binary masks perform worse and miss slight raindrops, while the proposed branch adds little training-time overhead and none at testing.

3) Supervision of the learned location map M:

  • Binary-mask supervision harms performance and misses many slight raindrops, leading to unsatisfactory removal in those regions.The paper contrasts this with using the difference between the ground truth and input as the location-map label.Source: 3) Supervision of the learned location… · pp. 6–7

4) Location learning styles:

  • Both the proposed location-learning branch and the AGAN-style subnetwork improve PSNR, but the proposed branch adds few training parameters and uses significantly less training GPU memory.The proposed branch is removed at testing, so it adds no testing-phase resources.Source: 4) Location learning styles · p. 7

C. Comparisons with state-of-the-art methods

LLPR variants built on DRSformer and Restormer achieve state-of-the-art results on Test-a and Test-b while retaining low computational cost. On Test-wild, the method is favored in an expert study and outperforms the compared methods on the reported no-reference metrics.

1) Quantitative comparisons:

2) Qualitative comparisons:

LLPR(DRSformer) produces stronger qualitative results than prior methods on the reported test sets, including real-world rainy scenes. The conclusion reports that LLPR variants achieve superior performance against state-of-the-art methods.

2) Qualitative comparisons:

  • On Test-a and Test-b, LLPR(DRSformer) recovers sharper background contours and results closer to ground truth than alternatives, and is visually favored on Test-wild.The authors note imperceptible block artifacts in IDT results.Source: 2) Qualitative comparisons · p. 9

V. CONCLUSION

  • The LLPR framework achieves superior performance against state-of-the-art methods, with reported effectiveness and generality.The authors trained variants using DRSformer and Restormer blocks.Source: §V · p. 9

Related papers

Loading 2609.30758v1…