PaperlayerView exact revision on arXiv

Source-linked AI summary

Restoring Without Forgetting: Continual Learning Across Image Degradations

Alif Ashrafee, Bartosz Krawczyk

arXiv:2608.23799v1cs.CVcs.AIcs.LG

TL;DR

Multi-degradation restoration is typically trained in closed-world settings, but deployed systems face new degradations sequentially without retaining past data or knowing degradation labels. RwF uses isolated lightweight adapters on a frozen backbone with unsupervised routing, achieving stronger continual performance and transfer across synthetic and real benchmarks. Its scope includes explicit challenges in low-light routing and out-of-distribution rain severity.

  • Problem

    Existing restoration approaches assume joint access to all degradation data or risk catastrophic forgetting when adapting sequentially without historical data.

  • Method

    RwF freezes a pretrained backbone, adds a lightweight low-rank adapter for each degradation, and uses prototype matching to route unlabeled test inputs.

  • Results

    RwF improves final average PSNR by up to 15.25 dB over naive sequential fine-tuning on a five-domain benchmark and transfers to eleven real-degradation benchmarks.

  • Takeaways & Limitations

    The framework provides a parameter-efficient continual restoration baseline that preserves prior degradation performance while supporting unsupervised inference-time routing.

  • Takeaways & Limitations

    Routing is weakest for LOL-v1 at 46.7% accuracy, while Rain100H reaches only 16.92 dB because its rain severity exceeds the synthesis distribution.

Abstract

from arXiv · show

Recent progress in image restoration has converged on all-in-one architectures that jointly handle multiple degradations within a single network. These methods are effective on static benchmarks but target a closed-world setting that assumes simultaneous access to every target degradation at training time. In practice, degradations are encountered sequentially as field-deployed systems progressively face new environmental conditions, and historical training data is often unavailable due to privacy or storage constraints. Accommodating a new degradation then requires either retraining on the union of all prior data, which is often costly or infeasible, or fine-tuning, which causes catastrophic forgetting. We formulate multi-degradation image restoration as a continual domain-incremental learning problem, in which degradations arrive incrementally and prior data is unavailable. Our proposed Restoring without Forgetting (RwF) framework learns a lightweight adapter for each new degradation, eliminating forgetting by construction at a fraction of the cost of dedicated per-domain networks. To isolate degradation learning from dataset variation, we construct a benchmark spanning five degradation domains under shared image content. At test time, an unsupervised routing mechanism identifies the appropriate restoration path for unknown inputs without requiring domain labels. Across the five-domain sequence, RwF improves final average PSNR over naive sequential fine-tuning by 15.25 dB and 11.83 dB on the Restormer and NAFNet backbones, respectively. The framework transfers to eleven canonical real-degradation benchmarks (3,465 images) at 89.5% routing accuracy with only a +0.94 dB oracle PSNR gap, establishing, to our knowledge, the first systematic baseline for continual multi-degradation image restoration.

1 Introduction

The paper reframes multi-degradation restoration as continual domain-incremental learning, where degradations arrive sequentially without past data or test-time labels. RwF addresses this setting with isolated lightweight adapters and unsupervised routing, supported by a controlled benchmark.

  • Field-deployed systems encounter new degradations sequentially, while historical data may be unavailable because of storage, privacy, or licensing constraints.
  • All-in-one restoration assumes simultaneous access to every target degradation, making extension to new degradations costly or infeasible when prior data cannot be retained.
  • RwF freezes a pretrained restoration backbone and adds lightweight low-rank adapters for each degradation, isolating degradation-specific parameters to prevent forgetting.
  • RwF improves final average PSNR over naive sequential fine-tuning by up to 15.25 dB across the five-domain sequence.
  • The paper frames multi-degradation restoration as domain-incremental learning, a setting described as largely unexplored in continual learning for pixel-level generative tasks.
  • RwF routes unknown test inputs to the appropriate adapter through prototype matching without requiring degradation labels.
  • The benchmark spans five degradation domains over shared image content and includes canonical real-degradation evaluations to isolate degradation learning from dataset variation.

2 Related Work

Prior restoration methods largely use task-specialized or jointly trained networks, while continual domain-incremental learning has focused mainly on discriminative benchmarks. The paper positions exemplar-free parameter isolation with frozen backbones as a route toward continual pixel-level restoration.

  • Modern restoration systems use attention and transformer architectures to establish strong baselines across denoising, deblurring, dehazing, deraining, and low-light enhancement.
  • All-in-one methods jointly train one network on multiple degradations but assume that the full degradation set and its data are available in advance.
  • Domain-incremental learning studies sequential adaptation without domain identity at inference, commonly using regularization, replay, or parameter isolation.
  • Replay introduces storage and privacy costs at high resolution, motivating exemplar-free parameter isolation over frozen pretrained backbones.
  • Continual learning remains overwhelmingly discriminative, with pixel-level generative tasks such as image restoration largely absent from the literature.

3 Methodology

RwF formulates multi-degradation restoration as domain-incremental learning with sequentially revealed degradations, unavailable prior data, and no degradation labels at inference. It isolates each degradation through lightweight adapters on a frozen denoising backbone, uses shared-content data for identifiability, and routes inputs to degradation-specific paths.

  • Problem Formulation: Multi-degradation restoration is modeled as sequential domain-incremental learning, where each domain is induced by a distinct degradation operator.Domains arrive one at a time as paired clean and degraded images.
  • Problem Formulation: Prior domains and samples are unavailable during later training, while the degradation index is withheld at test time.This separates the setting from joint all-in-one training and replay-based continual learning.
  • Pretrained Backbone and Degradation-Isolated Learning: A frozen denoising-pretrained backbone supplies shared restoration features, while new degradations are learned through compact trainable modules.The denoising weights remain fixed throughout and the scheme transfers to NAFNet using its released denoising weights.
  • Benchmark Construction: Shared clean content across synthetic domains makes degradation the distinguishing factor, reducing confounding between degradation operators and dataset-specific content.The benchmark applies multiple forward models to one DIV2K corpus, so all domains share identical clean source images.
  • Restoration Paths via Low-Rank Adaptation: Each low-rank adapter uses a rank-r bottleneck residual correction, with Wdown and Wup comprising the trainable parameters and Wup initialized to zero.The zero initialization makes each inserted adapter initially act as the identity.
  • Restoration Paths via Low-Rank Adaptation: Network-spanning restoration paths place one adapter at every backbone block so coordinated corrections can invert each degradation.A path is the complete set of per-block adapters associated with one degradation.
  • Prototype-Based Degradation Routing: Disjoint parameter subspaces make cross-domain gradients vanish, yielding exactly zero forgetting by construction.At inference, prototypes formed from frozen-backbone embeddings route unknown inputs by cosine similarity to the matching path.
  • Continual Learning Evaluation Protocol: Continual-learning evaluation uses a lower-triangular matrix of restoration quality after each training stage, from which final quality, incremental quality, and forgetting are summarized.The forgetting summary uses F = 0 to denote perfect retention.

Experiments

Experiments evaluate RwF across synthetic continual sequences, real-degradation transfer, routing ablations, and qualitative restoration. RwF preserves performance across domains while requiring lightweight adapters and domain-agnostic inference.

  • Experimental setup: RwF freezes each pretrained backbone and trains lightweight adapter paths for D2–D5, while sequential fine-tuning and regularization baselines update shared parameters.The comparison uses Restormer and NAFNet with identical training hyperparameters; only the unfrozen parameters differ.
  • Continual learning comparison: 28.87 dB final average PSNR on Restormer and 24.03 dB on NAFNet represent gains of 15.25 dB and 11.83 dB over sequential fine-tuning, respectively.RwF reports zero forgetting and narrows the gap between final average performance and per-domain peaks to under 1 dB on both backbones.
  • Real-degradation transfer: 89.5% routing accuracy across 3,465 images leaves a +0.94 dB oracle PSNR gap on eleven real-degradation benchmarks without retraining or recalibration.Seven benchmarks exceed 90% routing accuracy, and Kodak24 noise plus Rain100H reach 100%.
  • Real-degradation transfer: Misrouting is graceful because adjacent degradation operators share restoration primitives, so an incorrect path can still apply a related correction.A D4 hazy input misrouted to the D3 derain path remains largely restored.
  • Routing ablations: 79.7% routing accuracy is achieved with separately normalized GAP plus spatial standard deviation at encoder stage 1, outperforming deeper or simpler aggregation choices.Stage-1 readout reaches 79.7%, while the bottleneck reaches 58.8%; separately normalized GAP + std reaches 79.7%, versus 73.9% for GAP alone.
  • Qualitative analysis: On a shared scene, sequential fine-tuning restores only the latest domain at 25.05 dB and collapses to 3.78 dB on D4, whereas RwF produces 23.84–35.15 dB across all five domains.The qualitative comparison uses domain-agnostic routing, which correctly selects the appropriate restoration path for every input shown.

5 Conclusion

RwF frames multi-degradation restoration as continual domain-incremental learning and combines isolated low-rank adapters with unsupervised routing. It improves sequential-restoration performance and transfers to real-degradation benchmarks, while routing remains weaker for globally characterized degradations such as low light.

  • RwF freezes a denoising-pretrained backbone, adds isolated low-rank adapter paths, and uses unsupervised prototype matching to route inputs.
  • 15.25 dB is the reported maximum improvement in final-stage average PSNR over naive sequential fine-tuning across Transformer and convolutional backbones.
  • 89.5% routing accuracy was achieved across eleven canonical real-degradation benchmarks containing 3,465 images.
  • +0.94 dB was the oracle PSNR gap reported for transfer to the eleven real-degradation benchmarks.
  • Routing accuracy degrades when degradation signatures are primarily global rather than spatially localized, particularly for low-light enhancement.
Loading 2608.23799v1…