Source-linked AI summary
Deep White-Balance Editing
Mahmoud Afifi, Michael S. Brown
TL;DR
Incorrect white balance is hard to correct after camera ISPs apply nonlinear rendering to sRGB images. The paper trains an end-to-end DNN with one encoder and multiple decoders for AWB, indoor, and outdoor outputs. It reports state-of-the-art results across datasets and supports both correction and user-controlled white-balance manipulation.
Problem
Nonlinear ISP rendering makes sRGB images with incorrect white balance difficult to correct, while such errors also affect photographic and computer-vision applications.
Method
An end-to-end DNN uses a single encoder and multiple decoders to generate correct AWB, indoor, and outdoor white-balance renderings from sRGB images.
Results
The method achieves state-of-the-art results for post-capture white-balance correction and sRGB manipulation across several datasets.
Takeaways & Limitations
The framework directly corrects wrongly white-balanced sRGB images and lets users adjust their apparent white-balance color temperature.
Abstract
from arXiv · showhide
We introduce a deep learning approach to realistically edit an sRGB image's white balance. Cameras capture sensor images that are rendered by their integrated signal processor (ISP) to a standard RGB (sRGB) color space encoding. The ISP rendering begins with a white-balance procedure that is used to remove the color cast of the scene's illumination. The ISP then applies a series of nonlinear color manipulations to enhance the visual quality of the final sRGB image. Recent work by [3] showed that sRGB images that were rendered with the incorrect white balance cannot be easily corrected due to the ISP's nonlinear rendering. The work in [3] proposed a k-nearest neighbor (KNN) solution based on tens of thousands of image pairs. We propose to solve this problem with a deep neural network (DNN) architecture trained in an end-to-end manner to learn the correct white balance. Our DNN maps an input image to two additional white-balance settings corresponding to indoor and outdoor illuminations. Our solution not only is more accurate than the KNN approach in terms of correcting a wrong white-balance setting but also provides the user the freedom to edit the white balance in the sRGB image to other illumination settings.
1. Introduction and related work
White balance normalizes illumination early in the camera ISP, but later nonlinear rendering makes post-capture correction of sRGB images difficult. The paper proposes an end-to-end multi-decoder DNN for correcting and manipulating white balance directly in sRGB images.
- Motivation: White balance normalizes scene illumination so objects retain consistent colors across lighting conditions.It is applied early to raw-RGB data by the camera’s ISP.
- Motivation: After white balancing, camera-specific ISP operations render raw-RGB data into sRGB, often using nonlinear color manipulations.These operations may also reflect aesthetic photographic preferences.
- Problem: Incorrectly selected white balance produces strong color casts that are difficult to correct after an image is rendered into sRGB.Traditional diagonal correction is insufficient because of the ISP’s nonlinear rendering.
- Motivation: sRGB white-balance editing matters for photography and computer vision, where incorrect white balance can affect recognition, scene understanding, and color augmentation.Prior work reported effects resembling an untargeted adversarial attack on DNN models.
- Related work: The method addresses the need to reverse or emulate complex camera rendering without requiring the original unprocessed raw-RGB image.Earlier approaches included radiometric calibration and exemplar-based correction using large collections of rendered images.
- Approach: The proposed framework uses one encoder and multiple decoders to produce correct AWB, indoor, and outdoor white-balance renderings from sRGB inputs.Users can blend indoor and outdoor outputs to adjust aesthetic white-balance properties.
2. Deep white-balance editing
The method models sRGB white-balance editing as an encoder/decoder problem, using one encoder and multiple target-specific decoders rather than reconstructing raw RGB. It trains on rendered white-balance examples and combines neural outputs with global polynomial color mapping for full-resolution results.
- Method formulation: The framework approximates the camera pipeline by encoding an sRGB input into a latent representation and decoding it into a selected target white-balance image.The target image is generated directly rather than explicitly reconstructing and re-rendering the original raw-RGB values.
- Multi-decoder architecture: Three decoders generate AWB, Incandescent, and Shade outputs, covering correct scene lighting plus indoor and outdoor illumination settings.Incandescent and Shade correspond to approximately 2850 K and 7500 K, respectively.
- Multi-decoder architecture: The network uses a four-level U-Net encoder with multi-scale skip connections and three four-level decoders.Convolutional channel counts double across levels, from 24 at the first level to 192 at the fourth.
- Training phase: Training uses the Rendered WB dataset, with 12,000 training images and three ground-truth renderings per image for AWB, Shade, and Incandescent settings.The dataset contains approximately 65K sRGB images, while separate partitions provide testing data.
- Training phase: The model minimizes an L1 loss over reconstructed and ground-truth patches, with geometric augmentation and additional randomly colored raw-RGB renderings used during training.The authors also report that squared L2 loss worked well for the task.
- Testing phase: For efficient full-resolution output, the network processes resized images and fits a global polynomial color mapping from an 11-dimensional RGB kernel space.The authors found generating the image and then fitting the mapping more robust than directly estimating polynomial coefficients because coefficient errors can create noticeable color errors.
3. Results
The method achieves state-of-the-art results for sRGB white-balance correction and manipulation across several datasets, with quantitative and qualitative comparisons against prior approaches. Its shared-encoder, multi-decoder design supports multiple target white-balance settings and reduces parameters while improving results over separate vanilla U-Nets.
- The method achieves state-of-the-art results for post-capture white-balance correction and manipulation across several datasets.
- Quantitative results: 34,169 unseen sRGB images were used to evaluate white-balance correction across Set 1-Test, Set 2, and Cube+.The evaluation reports MSE, MAE, and △E 2000 along with quartiles.
- Quantitative results: 40,222 testing images show that the method outperforms the KNN-WB emulator across all evaluation metrics for white-balance manipulation.
- Qualitative results: Qualitative comparisons report lower △E 2000 errors and greater similarity to ground-truth images than competing methods.The comparisons cover white-balance correction and manipulation, including target settings such as Incandescent, Daylight, Shade, and Fluorescent.
- Qualitative results: Internet-image tests show compelling results compared with other methods and commercial photo-editing software, including images with strong color casts.
- Comparison with a vanilla U-Net: A single shared encoder with multiple decoders produces target white-balance settings and gives better results with fewer parameters than three separate vanilla U-Nets.The architecture targets AWB, indoor, and outdoor settings; additional examples show re-rendering from arbitrary input settings.
4. Conclusion
The paper presents an end-to-end deep learning framework for editing the white balance of sRGB camera-rendered images, including direct correction of wrong settings and manual adjustment across a wide range of color temperatures.
- A single-encoder, multiple-decoder DNN is trained end-to-end to edit white balance in sRGB images.The architecture directly operates on camera-rendered images.
- The framework directly corrects images captured with wrong white-balance settings.
- Users can manually adjust outputs to appear as if rendered with a wide range of white-balance color temperatures.
- Quantitative and qualitative results demonstrate effectiveness against recent data-driven methods.