Source-linked AI summary
CycleISP: Real Image Restoration via Improved Data Synthesis
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao
TL;DR
Real-image denoising lacks affordable paired data, and AWGN-based synthesis does not capture signal-dependent noise transformed by camera ISPs. CycleISP learns bidirectional sRGB–RAW mappings to synthesize realistic training pairs, achieving state-of-the-art results on real-noise benchmarks and extending to stereoscopic color matching.
Problem
Real paired datasets are expensive to collect, while AWGN-based synthetic data poorly represents signal-dependent camera noise transformed by the imaging pipeline.
Method
CycleISP learns device-agnostic transformations between sRGB and RAW spaces and uses them to synthesize clean/noisy pairs for both domains.
Results
CycleISP achieves state-of-the-art performance on DND and SIDD real-noise benchmarks, with about 5× fewer parameters than the previous best RAW-denoising method.
Takeaways & Limitations
The framework supports realistic RAW and sRGB denoising data synthesis and also applies to color matching in stereoscopic cinema.
Takeaways & Limitations
RAW2RGB is initially learned from original camera RAW images rather than RGB2RAW outputs, because the mapping is trained independently.
Abstract
from arXiv · showhide
The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However, for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synthetic data that is usually generated with a widespread assumption of additive white Gaussian noise (AWGN). While the CNNs achieve impressive results on these synthetic datasets, they do not perform well when applied on real camera images, as reported in recent benchmark datasets. This is mainly because the AWGN is not adequate for modeling the real camera noise which is signal-dependent and heavily transformed by the camera imaging pipeline. In this paper, we present a framework that models camera imaging pipeline in forward and reverse directions. It allows us to produce any number of realistic image pairs for denoising both in RAW and sRGB spaces. By training a new image denoising network on realistic synthetic data, we achieve the state-of-the-art performance on real camera benchmark datasets. The parameters in our model are ~5 times lesser than the previous best method for RAW denoising. Furthermore, we demonstrate that the proposed framework generalizes beyond image denoising problem e.g., for color matching in stereoscopic cinema. The source code and pre-trained models are available at https://github.com/swz30/CycleISP.
1. Introduction
Real-image denoising is limited by costly paired-data collection and synthetic AWGN that poorly represents camera noise. CycleISP addresses this by modeling the imaging pipeline bidirectionally to synthesize realistic RAW and sRGB training pairs.
- Motivation: Collecting real noisy/clean image pairs is expensive, cumbersome, and difficult because motion and camera-specific noise complicate acquisition.Ground-truth construction requires repeated captures, while lighting, scene motion, and sensor differences introduce mismatches.
- Motivation: Synthetic AWGN-trained denoisers perform well on synthetic data but generalize poorly to real camera images.Real camera noise differs substantially from the AWGN assumption used in common synthetic datasets.
- Motivation: Real sensor noise is signal-dependent and transformed by the ISP into correlated, potentially non-Gaussian noise, making realistic sRGB synthesis difficult.The paper therefore argues for models that account for the imaging pipeline rather than using uniform AWGN.
- Approach: CycleISP learns a device-agnostic transformation between sRGB and RAW spaces without requiring camera parameters.This avoids dependence on proprietary or device-specific ISP information.
- Contributions: The framework synthesizes clean/noisy paired data in both RAW and sRGB spaces and supports denoising, CycleISP learning, and realistic noise generation.Its contributions also include a dual-attention CNN and applications beyond denoising.
2. Related Work
Image denoising spans classical self-similarity methods and CNN-based approaches, but realistic camera-noise modeling remains difficult. CycleISP addresses this by modeling the camera ISP to synthesize noise for RAW and sRGB denoising.
- Classical methods: Classical denoising methods modify transform coefficients or average neighboring pixels, while non-local methods exploit self-similarity in natural images.These approaches include DCT and wavelet processing, neighborhood averaging, contour-aware filtering, and patch-based redundancy.
- Noise modeling: Large-scale real-noise datasets are prohibitively expensive, so denoising research commonly relies on synthetic data generated with AWGN.AWGN-designed algorithms do not effectively remove noise from real images.
- Noise modeling: A realistic RAW noise model combines signal-dependent shot noise with signal-independent read noise.The ISP then transforms this sensor noise into spatio-chromatically correlated, potentially non-Gaussian noise.
- CycleISP: CycleISP models the camera ISP in forward and reverse directions before synthesizing realistic datasets for RAW and sRGB denoising.Its two-stage scheme first learns the bidirectional pipeline and then uses it for noise-data synthesis.
3. CycleISP
CycleISP learns device-agnostic forward and reverse camera-ISP mappings, then uses RAW-domain noise injection to synthesize realistic noisy pairs in RAW and sRGB spaces.
- Framework: CycleISP uses RGB2RAW and RAW2RGB CNN branches to model the camera imaging pipeline in reverse and forward directions.The branches are trained independently before joint fine-tuning connects them.
- RGB2RAW Network Branch: The RGB2RAW branch converts sRGB images into demosaicked RAW estimates without requiring camera-specific parameters.It learns to invert tone mapping, gamma correction, color correction, white balance, and other ISP transformations.
- RAW2RGB Network Branch: The RAW2RGB branch packs Bayer measurements into four RGGB channels and maps clean RAW images to clean sRGB outputs.The input is the original camera RAW image during independent training, rather than the RGB2RAW output.
- Color Attention Unit: A color correction branch extracts color information from blurred sRGB images, while structural content and fine texture come from the main RAW2RGB pathway.The Gaussian kernel has empirically set standard deviation 12, and strong blurring limits this branch to color information.
- RRG: Recursive Residual Group: Recursive residual groups contain dual attention blocks that recalibrate features through channel and spatial attention to suppress less useful features.Channel attention models inter-channel dependencies, while spatial attention computes a feature-location map.
- Joint Fine-tuning of CycleISP: Joint fine-tuning feeds RGB2RAW outputs into RAW2RGB and combines the two reconstruction losses with weight β.RAW2RGB receives gradients from its own sub-loss, whereas RGB2RAW receives gradients from both sub-losses.
4. Synthetic Realistic Noise Data Generation
CycleISP generates realistic clean/noisy training pairs in RAW and sRGB spaces by modeling camera processing and injecting noise before rendering to sRGB.
- CycleISP addresses the gap between synthetic Gaussian-noise data and real camera data by synthesizing realistic noise pairs for RAW and sRGB denoising.
- The denoising network uses the same structure for RAW and sRGB inputs, differing only in input and output handling.
- The RGB2RAW branch converts clean sRGB images to clean RAW images, where shot and read noise are injected to create paired RAW data.
- The RAW2RGB branch converts synthetic noisy RAW images into noisy sRGB images, producing corresponding clean/noisy sRGB pairs.
5. Denoising Architecture
The proposed denoising architecture uses one network design for both RAW and sRGB images, adapting the input and output representations to each domain.
- The network denoises both RAW and sRGB images with the same structure, changing only how inputs and outputs are handled.
- For sRGB denoising, the network maps 3-channel sRGB inputs to 3-channel sRGB outputs.
- For RAW denoising, the network receives a 4-channel packed noisy image concatenated with a 4-channel noise-level map.
6. Experiments
Experiments on DND and SIDD show strong RAW and sRGB denoising performance, generalization across datasets, and an extension to stereoscopic-cinema color matching.
- Datasets: The evaluations use DND and SIDD datasets containing RAW and sRGB data from consumer and smartphone cameras.
- RAW Denoising: CycleISP consistently outperforms learning-based and conventional methods on RAW DND and SIDD benchmarks while using approximately 5× fewer parameters than the previous best method.The model also improves SIDD PSNR by 6.89 dB over BM3D.
- RAW Denoising: The denoising model removes real noise, including low-frequency chroma and defective-pixel noise.
- sRGB Denoising: For sRGB denoising, CycleISP gains 0.33 dB on DND and 0.81 dB on SIDD over RIDNet.
- sRGB Denoising: Compared with other methods, CycleISP produces clean, artifact-free sRGB results while preserving image details.
- Generalization Test: When trained on DND and evaluated directly on SIDD, the proposed synthesis method yields an approximately 1 dB gain over UPI.
- Color Matching For Stereoscopic Cinema: In stereoscopic cinema, CycleISP uses target-view color information to generate color-matched results that are perceptually more faithful than three competing techniques.
7. Conclusion
The paper concludes that CycleISP synthesizes realistic paired data in both image spaces, supports state-of-the-art real-noise denoising, and extends to stereoscopic color matching.
- CycleISP converts sRGB images to RAW data and back, enabling realistic clean/noisy pair synthesis for RAW and sRGB denoising.
- Training denoising networks with CycleISP achieves state-of-the-art performance on the DND and SIDD real-noise benchmarks.
- The framework also applies to color matching in stereoscopic cinema.