PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Restore More: Continual Capability Expansion for Pretrained Image Restoration Models

Hu Gao, Yulong Chen, Lizhuang Ma

arXiv:2608.30305v1cs.CV

TL;DR

Existing restoration systems struggle to support newly arriving degradations without extra models or retraining on historical data. RestoreMore expands a frozen pretrained restorer with residual experts and bi-level routing, and experiments show consistent acquisition of new abilities alongside preserved or improved prior capabilities.

  • Problem

    Restoration models commonly assume a fixed capability set, while new degradations require additional models or access to historical data for joint retraining.

  • Method

    RestoreMore freezes the pretrained restorer as a capability anchor and incrementally adds lightweight residual experts selected through capability-oriented bi-level routing.

  • Results

    Extensive experiments across diverse restoration benchmarks show that RestoreMore consistently acquires new restoration abilities while preserving and often improving existing capabilities.

  • Takeaways & Limitations

    The framework supports progressive capability expansion by reusing accumulated restoration knowledge for new tasks and subsequent expansions.

  • Takeaways & Limitations

    Existing expansion assumes historical training images are neither stored nor revisited, and previously learned components remain frozen during incremental stages.

Abstract

from arXiv · show

Image restoration models are typically trained with a fixed set of capabilities. When new restoration requirements emerge, existing solutions usually train additional models or jointly retrain the original model with both new and historical data. Instead of designing another restoration backbone, we investigate how a trained restorer can continually acquire new capabilities without forgetting those learned previously. We propose RestoreMore, a continual capability-expansion framework that preserves the pretrained restoration model as a frozen capability anchor and learns residual expansion modules for newly arriving degradations. RestoreMore introduces a capability-oriented bi-level routing mechanism at multiple feature stages. The first routing level identifies restoration capabilities relevant to the current input, while the second selects and combines a sparse set of complementary degradation experts. This design enables newly introduced tasks to selectively reuse historical restoration knowledge and progressively enriches the expert bank available for subsequent restoration tasks. Extensive experiments on a wide range of restoration benchmarks demonstrate that RestoreMore consistently acquires new restoration abilities while preserving and improving previously learned capabilities.

1 INTRODUCTION

Image restoration traditionally assumes a fixed capability set, making newly arriving degradations costly to support. RestoreMore instead expands a frozen pretrained restorer with routed residual experts while preserving prior capabilities.

  • Motivation: Existing restoration models are typically optimized for predefined degradations and perform poorly when the degradation changes.
  • Motivation: Supporting new degradations commonly requires either additional task-specific models or joint retraining with historical and new datasets.Additional models increase storage and deployment costs, while joint retraining requires access to previous data.
  • Related gap: Prior continual restoration methods mainly mitigate forgetting within restricted task families rather than preserving arbitrary pretrained pathways or reusing complementary knowledge across stages.
  • RestoreMore: RestoreMore freezes the pretrained model as a restoration anchor and adds lightweight residual experts for newly arriving degradations.A null expert preserves the original pathway when no additional correction is needed.
  • RestoreMore: Capability-oriented bi-level routing first retrieves relevant capabilities and then sparsely combines complementary experts at selected feature stages.This allows new tasks to reuse accumulated restoration knowledge across degradation types.
  • Results: Extensive experiments show that RestoreMore acquires new restoration abilities while preserving and improving previously learned capabilities.

2 RELATED WORK

Image restoration and continual-learning research provide important foundations, but existing approaches remain constrained by fixed capability sets or restricted task families. RestoreMore addresses this gap through frozen-backbone expansion and stage-wise knowledge reuse.

  • Image restoration: Prior-driven restoration uses explicit structural assumptions or degradation models, while deep learning has shifted the field toward data-driven representation learning.Manually defined assumptions can be insufficient for complex and spatially varying degradations.
  • Image restoration: Existing restoration methods remain mainly fixed-capability systems that require another model or retraining with new and historical datasets when unsupported degradations appear.
  • Continual learning: Continual-learning methods commonly use regularization, replay, optimization constraints, or architectural expansion to mitigate catastrophic forgetting.
  • Continual restoration: Existing continual restoration methods focus on restricted task families and do not explicitly preserve arbitrary pretrained pathways or reuse complementary knowledge across degradation types and feature stages.
  • RestoreMore: RestoreMore freezes the pretrained restorer and progressively expands its capability set through lightweight residual experts and capability-oriented bi-level routing.

3 METHOD

RestoreMore expands an already trained restorer without changing its backbone: the frozen original pathway is retained while lightweight residual modules add capabilities for sequentially arriving degradations. Capability-oriented routing then selects relevant historical capabilities and sparse complementary experts at feature stages, enabling knowledge reuse without revisiting historical data.

  • 3.1 PROBLEM FORMULATION: Continual expansion starts from a pretrained restorer with capability set S0 and sequentially adds previously unsupported degradations using new paired datasets Dt.Historical training images are neither stored nor revisited after one-time capability indexing.
  • 3.2 OVERALL PIPELINE: The pretrained backbone and previously learned expansion components remain frozen, while lightweight components are introduced only for newly arriving degradations.This preserves the original restoration pathway during capability expansion.
  • 3.2 OVERALL PIPELINE: Each expansion module constructs a degradation-sensitive descriptor from spatial statistics and frequency characteristics before routing.The descriptor uses channel-wise spatial average, standard deviation, and Fourier characteristics projected by a lightweight network.
  • 3.3 CAPABILITY-ORIENTED BI-LEVEL ROUTING: The first routing level selects the Top-M relevant restoration capabilities from accumulated capability prototypes and routers.Multiple selected capabilities can provide complementary restoration knowledge, and routing is performed separately at feature stages.
  • 3.4 EXPERIMENTS: RestoreMore is compared with original pretrained models and joint retraining across CNN-, Transformer-, and Mamba-based restorers.The comparison includes RestoreMore trained only with newly arriving task data.
  • 3.3 CAPABILITY-ORIENTED BI-LEVEL ROUTING: For each selected capability, query–key matching retrieves complementary experts, with only the Top-K experts activated and their outputs sparsely composed.New experts can be appended as expert–key pairs without modifying historical router structures; a null expert can apply no additional correction.

4 EXPERIMENTS

Experiments show that RestoreMore expands restoration capabilities across architectures and tasks while preserving historical performance. It also generalizes to real-world degradations, unseen degradation types and severities, with ablations supporting the roles of its frozen anchor and routing components.

  • Quantitative comparison: RestoreMore outperforms joint retraining across ALGNet, Perceive-IR, and StarIR, despite using only newly arriving-task datasets.Average PSNR/SSIM improves from 31.98/0.925 to 33.26/0.942 for ALGNet, from 33.53/0.937 to 34.10/0.946 for Perceive-IR, and from 33.54/0.936 to 34.14/0.947 for StarIR.
  • Capability preservation: RestoreMore preserves historical capabilities while acquiring new ones, reducing ALGNet’s original-task degradation from 4.08 dB to 0.31 dB.ALGNet retains 33.18 dB on deblurring versus 29.41 dB after mixed-data retraining, while adding four restoration capabilities; Perceive-IRR also improves several historical tasks.
  • New-task performance: On newly introduced tasks, RestoreMore exceeds joint retraining across evaluated architectures, including a 1.71 dB dehazing gain for ALGNet.Perceive-IR and StarIR also show 1.94 dB and 1.81 dB deblurring gains over their corresponding joint-training baselines.
  • Qualitative comparison: RestoreMore produces visually cleaner restoration across five tasks, suppressing rain and snow artifacts while preserving structures, contrast, boundaries, and textures.Qualitative comparisons report clearer dehazing, sharper deblurring, and less smoothing in denoising.
  • Generalization: RestoreMore improves real-world generalization and maintains strong performance on unseen degradation types and noise severities.Perceive-IRR reaches 27.55 dB PSNR, 0.369 LPIPS, and 0.903 SSIM on RealRain-1k-L; it achieves the best results across unseen-type metrics and three of four unseen-noise settings.
  • Ablation and robustness: Ablations show that the frozen capability anchor and both routing levels are complementary, with complete RestoreMore reaching 33.26 dB average PSNR.The complete model outperforms mixed-data retraining by 1.28 dB and the strongest partial variant by 0.57 dB; performance remains stable across four task orders.

5 CONCLUSION

RestoreMore preserves a pretrained restorer as a frozen capability anchor while incrementally adding lightweight residual expansion modules and bi-level routing for new degradations. Experiments across diverse benchmarks and architectures show acquisition of new abilities while preserving, and often improving, existing ones.

  • RestoreMore freezes the pretrained restorer and incrementally adds lightweight residual expansion modules for newly arriving degradations.
  • Its bi-level router first identifies relevant restoration capabilities, then sparsely selects complementary degradation experts at multiple feature stages.
  • The expanding expert bank reuses accumulated restoration knowledge for new tasks and becomes available for subsequent capability expansion.
  • Extensive experiments across diverse restoration benchmarks and model architectures show consistent acquisition of new restoration abilities while preserving, and often improving, existing capabilities.

AI USE STATEMENT

The authors used generative AI for manuscript editing and scientific figure preparation, while retaining responsibility for the final scientific content. AI assistance did not generate or modify the underlying experimental data or numerical results in Fig. 5.

  • Generative AI tools assisted manuscript editing by improving language, grammar, and readability without changing the underlying scientific content.
  • AI also assisted with Fig. 5 preparation using only the authors’ experimental values, without generating or modifying its underlying data or numerical results.
  • The authors manually reviewed and verified all AI-assisted outputs and accept responsibility for the final manuscript, figures, results, claims, and conclusions.

A.1 INITIALIZATION OF THE CAPABILITY BANK

RestoreMore initializes its capability bank from the pretrained model through a one-time, non-updating indexing procedure, then prepares prototypes and routers for initial and arriving degradations. After initialization, historical raw training images are discarded, so later stages use only newly arriving degradation data.

  • A one-time indexing procedure constructs stage-wise capability representations for the initial capability set without updating the pretrained restoration backbone.
  • Samples from each initial degradation are forwarded through F0 to extract stage-wise degradation descriptors, while each projection ψl is calibrated and then frozen.
  • The capability router is initialized during indexing, with the null expert serving as the default residual option for capabilities already represented by F0.
  • New-degradation prototypes are initialized from current training features, updated by exponential moving averaging, and frozen when their stage is complete.
  • After initial indexing, historical raw training images are discarded and subsequent incremental stages use only the newly arriving degradation dataset.

A.2 WHY QUERY–KEY ROUTING SUPPORTS CONTINUAL EXPANSION

Query–key routing avoids expanding historical router output layers as the expert bank grows. It preserves previously learned components while allowing new experts to accumulate and serve future restoration tasks.

  • A standard MoE router’s output dimension grows with the expert bank, so adding experts would require expanding historical routers.
  • RestoreMore separates capability routing from expert indexing using fixed-dimensional capability queries and expert routing keys.
  • New capabilities append expert–key pairs while previously learned routers and experts remain structurally unchanged.
  • The shared query–key space makes newly introduced experts candidates for future restoration tasks.

A.3 OPTIMIZATION AND COLD-START STRATEGY

At each incremental stage, RestoreMore freezes the pretrained backbone and historical capability components while optimizing the newly introduced capability. A cold-start routing strategy guarantees early training access to the current capability and expert while retaining selective reuse of historical components.

  • The pretrained backbone and all historical capability components remain frozen at incremental stage t.
  • RestoreMore always includes the current capability in the Top-M capability set during training.
  • The newly introduced expert is forced into its capability router’s Top-K set during training.
  • The cold-start strategy gives new routing components sufficient optimization before their scores become reliable.
  • The remaining selected capabilities and experts dynamically reuse previously learned restoration knowledge without historical image replay or output-level distillation.

A.4 EXPERT ARCHITECTURE AND BACKBONE INSTANTIATION

RestoreMore adds lightweight residual spatial adapters to major feature stages and uses sparse routing to reuse degradation knowledge without changing the original backbone’s internal operations. Experiments evaluate expansion from task-specific and all-in-one pretrained restorers across multiple restoration capabilities, with routing becoming more specialized at deeper stages while preserving earlier abilities.

  • Expert architecture: Each degradation expert is a lightweight bottleneck spatial adapter with channel reduction, depthwise convolution, channel restoration, and zero-initialized residual scaling.
  • Backbone instantiation: The same residual expansion interface attaches to CNN-, Transformer-, and Mamba-based restorers without modifying their internal operations.
  • Sparse computation: O(MKCE) bounds dominant expert computation, replacing dense O(tCE) evaluation when M and K are fixed.
  • Sparse computation: Capability-prototype and query–key matching costs grow with the capability bank but remain cheaper than spatial expert evaluation.
  • Evaluation settings: The experiments test task-specific-to-more and all-in-one-to-more expansion across deraining, desnowing, dehazing, deblurring, and denoising.
  • Routing behavior: Routing is dispersed in shallow stages but increasingly diagonal and task-aligned in deeper stages, while expert activations become more localized.
  • Routing behavior: Each task activates only a small subset of accumulated capabilities and experts, supporting sparse computation and reduced interference as tasks grow.
  • Continual expansion results: After deraining expansion, ALGNet reaches 36.28 dB on deraining while retaining 33.14 dB on deblurring, a 0.35 dB decrease from its original 33.49 dB.
Loading 2608.30305v1…