Source-linked AI summary
Benchmarking RAW and RGB Restoration in Image Signal Processors
Zihao Lu, Radu Timofte, Marcos V. Conde
TL;DR
The paper benchmarks RAW pre-ISP and RGB post-ISP restoration around fixed ISPs to test whether placement or training alignment matters more. Benchmark-trained RAW models outperform generic RGB models in most configurations, while target-ISP-trained RGB restoration performs best in a pipeline-specific experiment, indicating no universally preferable domain.
Problem
Existing restoration methods largely operate on RGB images, although cameras transform RAW measurements through an ISP, leaving limited evidence about restoration placement around a fixed imaging pipeline.
Method
The benchmark compares pre-ISP RAW and post-ISP RGB restoration across four smartphone device groups, two learned ISP proxies, three degradation regimes, and representative restoration methods.
Results
Benchmark-trained RAW restoration outperforms generic pre-trained RGB restoration in most evaluated configurations, whereas target-ISP-trained RGB restoration performs best in the Vivo X90 pipeline-specific experiment.
Takeaways & Limitations
Robust final-image quality depends on matching restoration training distributions to the distribution induced at the model’s insertion point in the target imaging pipeline.
Takeaways & Limitations
The generic comparison is not training-matched, and the RAW and RGB models optimize losses in different domains, so the results do not determine either domain’s intrinsic performance ceiling.
Abstract
from arXiv · showhide
Modern cameras transform RAW sensor measurements into sRGB images through an image signal processor (ISP). We benchmark two placements for blind restoration around a fixed ISP: (A) pre-ISP restoration in the RAW domain and (B) post-ISP restoration in the sRGB domain. The benchmark covers four smartphone device groups, two learned ISPs, three degradation regimes--noise, blur, and joint noise and blur--, and several representative RAW and RGB restoration models. Our results show that placement alone does not determine performance. The RAW restoration strategy outperforms the best generic RGB restoration models. However, RGB restoration models trained considering the ISP transformations, achieve the best overall performance. Our novel benchmark demonstrates that the image reconstruction performance strongly depends on the alignment between the restoration model and the target imaging pipeline. We consequently recommend reporting restoration placement and ISP-aware supervision as key experimental factors. Our code is available at https://github.com/mv-lab/AISP
1 Introduction and Motivation
The paper examines whether restoration should occur before or after a fixed ISP, separating restoration placement from training-distribution alignment. It benchmarks this comparison across multiple devices and degradation settings to clarify why observed rankings differ.
- Motivation: RGB restoration must handle noise and blur after nonlinear, device-specific ISP transformations, rather than direct sensor measurements.Tone mapping can amplify darker-region noise and create complex degradation interactions.
- Research gap: Existing comparisons confound restoration domain with degradation synthesis, training data, camera pipeline, and evaluation protocol.The paper therefore isolates whether differences arise from domain choice or alignment with the target pipeline.
- Framework: The framework treats the ISP as a fixed black box and compares pre-ISP RAW restoration with post-ISP RGB restoration under distinct training regimes.Generic models are not trained on target-ISP outputs, whereas ISP-aware models use examples from that output distribution.
- Evaluation scope: The benchmark spans four smartphone groups, three degradation levels, and real-world examples to evaluate restoration around fixed ISPs.The degradation levels are denoising, deblurring, and joint restoration.
- Main finding: Generic pre-trained RGB models struggle with ISP-transformed degradations, while target-ISP-trained RGB models can recover degradations and pipeline-specific artifacts.This finding qualifies universal claims that either RAW or RGB restoration is intrinsically preferable.
2 Related Work
Prior restoration research has advanced in RGB and RAW domains, but systematic comparisons around fixed smartphone ISPs remain limited. The paper focuses on modular restoration and distribution mismatch rather than jointly learning the ISP.
- RGB restoration: Deep RGB restoration includes task-specific methods for denoising, deblurring, and super-resolution, alongside all-in-one networks for multiple degradations.These methods benefit from large datasets and mature architectures.
- Positioning: The paper instead evaluates modular restoration around fixed ISPs, avoiding the need to jointly train inaccessible proprietary camera pipelines.This framing addresses practical constraints of paired RAW–RGB data and non-differentiable ISPs.
- RGB restoration: Generic RGB restoration faces distribution mismatch because tone mapping and device-specific processing alter noise, blur, and content interactions.This motivates both restoration before the ISP and target-ISP training after it.
- RAW restoration: RAW restoration has shown benefits for denoising, deblurring, and super-resolution, but prior work does not systematically compare RAW pre-processing with RGB post-processing for smartphone ISPs.Existing methods also commonly assume downstream ISP knowledge or jointly train restoration and ISP components.
3 Methodology
The methodology constructs controlled RAW degradations, evaluates restoration around fixed learned ISP proxies, and measures performance across devices and three degradation levels. The ISP experiments establish how degradation severity affects RGB output quality before restoration.
- Dataset: The dataset contains 2,092 training images and 47 test images from Vivo X90 Pro, iPhone XS, Samsung S9/S21, and Google Pixel 7–9.All methods and degradation levels use the same split.
- RAW preprocessing: RAW inputs are normalized with sensor-specific black levels and bit depths, packed into four-channel RGGB data, and cropped into 512×512×4 patches.These patches correspond to 1024×1024×3 RGB regions.
- Degradation protocol: The benchmark evaluates denoising, deblurring, and joint blur-and-noise restoration using synthetic degradation models.Noise-only, blur-only, and combined degradations are defined separately across three levels.
- Degradation protocol: Training randomizes degradation parameters, applies blur with probability 0.5, and samples noise types with equal probability.The protocol uses diverse noise profiles, PSFs, and blur kernels.
- Neural ISP models: Two learned neural ISPs are trained for camera-specific RAW-to-sRGB rendering and then kept fixed as controlled pipeline proxies.The proxies use curated aligned RAW–sRGB pairs whose targets come from corresponding phone pipelines.
- ISP sensitivity: On Vivo X90, ISP v1 falls from 22.60 dB on clean inputs to 19.37 dB under joint noise and blur.The corresponding drops are 1.08, 1.70, and 3.23 dB for noise, blur, and joint degradation.
4 Experimental Results
The experiments compare RAW restoration before a fixed ISP with generic and target-ISP-trained RGB restoration after it. Benchmark-trained RAW models outperform generic RGB models, while target-ISP-trained RGB restoration reverses the ranking in the pipeline-specific comparison.
- Experimental setup: The benchmark evaluates pre-ISP RAW restoration and post-ISP RGB restoration around fixed ISPs, with Strategy B split into generic and ISP-aware training regimes.Strategy A restores RAW inputs before the ISP; Strategy B restores its RGB outputs, and ISP-aware training uses outputs from the same fixed ISP evaluated at test time.
- RAW restoration: NAFNet and MOFA outperform the other evaluated RAW restoration methods while maintaining reasonable parameter counts and computational complexity.These two methods are therefore selected for subsequent evaluation.
- Generic RGB restoration: Benchmark-trained RAW models outperform generic pre-trained RGB models across the evaluated degradation levels, but this comparison is not training-matched.Generic RGB models use conventional RGB degradations rather than outputs from the target ISPs, whereas RAW models use benchmark-generated RAW degradations.
- Generic RGB restoration: 3.50 dB: NAFNet+ISPv1 reaches 25.99 dB versus 22.49 dB for ISPv1+MiOIR at Level 2 on Samsung S9.The gap generally widens for joint degradation at Level 3, where conventional RGB training transfers less well to ISP-transformed artifacts.
- RAW restoration: 0.60 dB: On Vivo X90 at Level 1, NAFNet+ISPv1 reaches 22.00 dB, compared with 22.60 dB for the clean baseline after degradation lowers performance to 21.52 dB.This comparison illustrates how RAW restoration can shield the ISP from input degradations.
- Efficiency: The compact RAW models use approximately 1–2 million parameters, versus 15–30 million for the pre-trained RGB baselines, favoring mobile deployment.The parameter comparison accompanies the reported RAW-restoration improvements.
- Target-ISP training: Target-ISP-trained RGB restoration exceeds sensor-specific Strategy A across all three degradation levels and can restore artifacts present after ISP processing.This reversal supports training-distribution alignment rather than domain alone as the explanation for the observed ranking.
- Limitations: The benchmark’s broad generalization claims are limited because synthetic degradations cannot capture every smartphone sensor and lens property, and the test split contains 47 images.The generic comparison is also not training-matched, and CCM-aware or target-ISP-aware RAW objectives are not investigated.
5 Conclusion
The benchmark shows that restoration performance depends on alignment between the restoration model, training distribution, and target imaging pipeline rather than placement alone. Benchmark-trained RAW restoration beats generic RGB restoration, while target-ISP-trained RGB restoration performs best in the pipeline-specific experiment.
- Benchmark-trained RAW restoration outperforms generic pre-trained RGB restoration in most evaluated configurations.
- Target-ISP-trained RGB restoration performs best in the Vivo X90 pipeline-specific experiment.
- Matching restoration training to the distribution induced at the model’s insertion point is central to robust final-image quality.