Source-linked AI summary
DeepISP: Towards Learning an End-to-End Image Processing Pipeline
Eli Schwartz, Raja Giryes, Alex M. Bronstein
TL;DR
Low-light mobile cameras are constrained by hardware and capture conditions, motivating a learned replacement for conventional ISP processing. DeepISP jointly maps raw low-light inputs through low- and high-level processing to final images using paired smartphone data. It achieves better reported visual quality than the manufacturer ISP and strong PSNR results on joint denoising and demosaicing.
Problem
Low-light mobile cameras need algorithms to compensate for limited pixel size, optics, and exposure stability, while conventional ISP tasks are typically engineered independently.
Method
DeepISP is an end-to-end deep model that jointly learns low-level corrections and higher-level global restoration from raw images, using a two-stage architecture and quadratic color transformation.
Results
DeepISP images scored 7% higher than the manufacturer ISP in human evaluation of full images and 0.7% below equivalent well-lit images; joint denoising and demosaicing exceeded prior solutions by 0.72dB PSNR.
Takeaways & Limitations
The results support learned joint processing as a candidate for replacing more of a camera's conventional ISP while sharing information across pipeline stages.
Abstract
from arXiv · showhide
We present DeepISP, a full end-to-end deep neural model of the camera image signal processing (ISP) pipeline. Our model learns a mapping from the raw low-light mosaiced image to the final visually compelling image and encompasses low-level tasks such as demosaicing and denoising as well as higher-level tasks such as color correction and image adjustment. The training and evaluation of the pipeline were performed on a dedicated dataset containing pairs of low-light and well-lit images captured by a Samsung S7 smartphone camera in both raw and processed JPEG formats. The proposed solution achieves state-of-the-art performance in objective evaluation of PSNR on the subtask of joint denoising and demosaicing. For the full end-to-end pipeline, it achieves better visual quality compared to the manufacturer ISP, in both a subjective human assessment and when rated by a deep model trained for assessing image quality.
I. INTRODUCTION
DeepISP targets the limitations of low-light mobile cameras by replacing independently engineered ISP stages with an end-to-end learned pipeline. It jointly addresses low-level restoration and higher-level enhancement using raw inputs and paired training data.
- Mobile phones require capable image-processing algorithms because small pixels, limited optics, and handheld operation constrain low-light capture.
- An ISP combines low-level demosaicing, denoising, and sharpening with higher-level image adjustment and color correction, typically using separate engineered algorithms.
- Deep learning can jointly solve multiple image-processing tasks, enabling feature sharing across stages rather than treating each task independently.
- The authors contribute a low-level model achieving state-of-the-art results for joint denoising and demosaicing.
- DeepISP is an end-to-end model that accepts raw images and outputs final high-perceptual-quality images.
- The released S7 ISP dataset pairs raw and processed JPEG images from normal-exposure and shorter-exposure captures of the same scenes.
II. THE DEEPISP NETWORK
DeepISP uses a two-stage fully convolutional architecture that first performs local low-level corrections and then applies a learned global color transformation. Its quadratic transformation is designed to model raw-to-processed image mappings more effectively than a linear one.
- II. THE DEEPISP NETWORK: DeepISP jointly learns low-level corrections and higher-level global image restoration in an end-to-end fashion.
- II. THE DEEPISP NETWORK: The two stages extract low-level and higher-level features, perform local and global corrections, and share computed features across tasks.
- Low-level stage: The low-level stage uses convolutional blocks that preserve spatial dimensions and begin from an RGB image produced by bilinear demosaicing.
- Low-level stage: Residual images are added at each low-level block, giving later layers both the current image estimate and computed features.
- High-level stage: The high-level stage estimates transformation parameters from low-level features and applies them to the first-stage corrected image.
- High-level stage: The global correction uses a pixel-wise quadratic transformation with W ∈ R3×10, selected because linear regression inadequately approximates real raw-to-processed mappings.
- High-level stage: Using a global transformation limits the architecture's ability to learn classical local HDR tone mapping, although the authors report sufficient performance and improved convergence and stability for their task.
B. Loss
The loss design differs by task: PSNR-oriented low-level restoration uses ℓ2 loss, while the full ISP uses Lab-domain ℓ1 and MS-SSIM to emphasize perceptual quality.
- For joint denoising and demosaicing, the model uses ℓ2 loss and evaluates performance with PSNR.
- The full ISP uses a combination of ℓ1 and MS-SSIM because PSNR can penalize global color deviations that do not reduce perceptual quality.
- Full-ISP loss is computed in Lab space after differentiable RGB-to-Lab conversion, with ℓ1 applied to all channels and MS-SSIM only to luminance.
- This combination targets global corrections through ℓ1 and local corrections through MS-SSIM, while luminance-only structural evaluation avoids requiring color channels to converge first.
A. Evaluation
DeepISP is evaluated on joint denoising and demosaicing using MSR data, with results reported against competing methods on Panasonic and Canon test sets.
- The evaluation uses Panasonic MSR images for training and reports performance on disjoint Panasonic and Canon test sets.The mosaiced images contain realistic noise with standard deviation σ ∈[1, 10].
- The low-level network uses 20 residual blocks and bilinear interpolation to produce the RGB input during preprocessing.Only the low-level stage is used because joint denoising and demosaicing requires local image modifications.
- The comparison table reports PSNR for joint denoising and demosaicing, with competing results taken from [2].
- Figure 3 provides visual restoration examples for the joint denoising and demosaicing task.
- DeepISP achieves the best joint denoising and demosaicing results on both Panasonic and Canon MSR test sets.Compared with SEM, improvements are 0.38dB linear and 0.72dB sRGB on Panasonic, and 0.61dB linear and 1.28dB sRGB on Canon.
B. Choosing hyper-parameters
The hyper-parameter study examines depth, filter count, receptive field, and residual connections, showing deeper networks and wider layers improve performance until convergence.
- Each additional 3 × 3 convolution layer enlarges the receptive field to 2N + 1 × 2N + 1 neighboring input pixels.More residual blocks also provide additional parameters and nonlinear units for learning complicated functions.
- The joint denoising and demosaicing model removes artifacts introduced by bilinear interpolation in visual examples.
- PSNR improves with network depth but reaches convergence or diminishing returns at around 16 residual layers.
- For a 20-layer network, performance increases with filter count and converges at around 64 filters per layer.Parameters scale quadratically with filter count but only linearly with layer count.
- Effect of skip connections: Skip connections stabilize training, whereas removing them produces unstable training and prevents convergence to the original model’s performance.Very deep networks are susceptible to vanishing or exploding gradients because backpropagation spans many layers.
A. S7-ISP Dataset
The S7-ISP dataset pairs low-light and well-lit smartphone captures in raw and JPEG formats, using controlled exposure changes across diverse scenes.
- Capture procedures used a tripod and a custom Android application, but complete absence of motion could not be guaranteed outside a laboratory setting.
- Figure 4 illustrates removal of blue Moiré artifacts from a far building that remain in SEM’s output.
- Low-light images use the camera’s automatic settings except for an exposure time set to one quarter of the automatic value.The closest supported discrete exposure time is selected.
- The dataset contains 110 scenes split into 90 training, 10 validation, and 10 test scenes.Images have 3024 × 4032 resolution, providing many training patches despite the small scene count.
- Scenes include indoor and outdoor environments under sunlight and artificial light, yielding over 20 thousand non-overlapping 256 × 256 patches.
B. Mean Opinion Score
Full-pipeline quality is assessed with human mean opinion scores and a learned image-quality model because objective metrics are difficult to define.
- Human evaluators rate full images from 1 (bad) to 5 (excellent) to produce a mean opinion score for each image.
- A learned model trained to estimate human evaluations also scores image quality, with outputs normalized to the range [1, 5].
C. DeepISP Evaluation
DeepISP is evaluated as a low-light raw-to-JPEG pipeline, against both manufacturer processing and well-lit references. The experiments also test black-box ISP imitation and show that sharing features between low- and high-level stages improves image quality.
- Evaluation scope: The dataset lacks motion artifacts, although learning from shorter-exposure images may help mitigate such artifacts.This limits direct coverage of a capability expected from a real camera ISP.
- Evaluation protocol: The evaluation used MOS from Amazon Mechanical Turk for both random 512 × 512 patches and full images, alongside a deep-learning image-quality rating.Each image version received 200 ratings: 100 for patches and 100 for the full image.
- End-to-end visual quality: 4.02 MOS for DeepISP full images exceeded Samsung S7 ISP’s 3.74 and approached the well-lit reference’s 4.05.DeepIQA scores followed the same ordering: 3.92 for DeepISP, 3.72 for Samsung’s ISP, and 4.02 for well-lit scenes.
- ISP imitation: DeepISP learned to mimic Samsung’s ISP from raw inputs paired with processed JPEG outputs, producing pleasant images close to the ground truth in details.The ground-truth images define the upper bound of expected performance in this setting.
- Shared features: Severing the connection between low- and high-level stages degraded image quality despite keeping the same layer and parameter budget.Without shared features, the model often failed to generate good-looking colors.
V. CONCLUSION
DeepISP is an end-to-end learned alternative to independently engineered ISP stages, jointly processing raw inputs into visually compelling images. It improves reported perceptual and objective results, while motion handling, HDR, noise adaptation, and agreed end-to-end quality metrics remain open issues.
- Contribution: DeepISP jointly learns the full image-processing pipeline, sharing information across tasks that are traditionally performed independently.The approach is presented as a step toward replacing a camera’s full ISP with a learned network.
- Future scope: Current exclusions include camera shake or blur removal, HDR options, and adaptation to varying noise levels.These capabilities are identified as requiring further exploration before broader ISP replacement.
- Perceptual evaluation: 7% higher full-image human ratings than the manufacturer ISP and 0.7% below well-lit images demonstrate strong perceptual performance.DeepIQA showed similar trends to the human evaluation.
- Objective evaluation: 0.72dB PSNR over prior methods on Panasonic MSR and 1.28dB PSNR when training on Panasonic and testing on Canon show strong joint denoising-and-demosaicing performance.The cross-dataset result is reported as evidence of generalization.
- Evaluation limitations: No agreed objective metric exists for general end-to-end image quality, although DeepIQA scores were well-correlated with human perception in this study.The authors suggest learned quality models as a possible evaluation or training component.