PaperlayerView exact revision on arXiv

Source-linked AI summary

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Mahmoud Afifi, Michael S Brown

arXiv:1912.06960v1cs.CV

TL;DR

Incorrect WB can create strong color casts that yield unpredictable results for DNNs trained on datasets biased toward correctly balanced images. The paper introduces realistic WB-error augmentation and evaluates it alongside preprocessing WB correction. Both approaches improve results on CIFAR-10, CIFAR-100, and ADE20K, while external classification tests report a ∼6% accuracy reduction on images modified with incorrect WB settings.

  • Problem

    The paper examines how incorrect WB affects DNN classification and segmentation, since real applications encounter such images while web-derived datasets are biased toward correctly balanced images.

    Source: §1 · p. 1

  • Method

    The paper introduces augmentation that emulates realistic WB errors and evaluates preprocessing training and testing images with a WB correction algorithm.

    Source: §1 · pp. 1–2

  • Results

    Both WB correction and proposed augmentation improve results across CIFAR-10, CIFAR-100, and ADE20K; external tests show accuracy is reduced by ∼6% on incorrectly WB-modified images versus properly WB images.

    Source: §6 · p. 8, §5.4 · p. 8

  • Takeaways & Limitations

    The proposed WB augmentation method may be useful for other DNN tasks that use image augmentation.

    Source: §6 · p. 8

    Training with a fixed number of iterations did not fully exploit the additional augmented training images.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

There is active research targeting local image manipulations that can fool deep neural networks (DNNs) into producing incorrect results. This paper examines a type of global image manipulation that can produce similar adverse effects. Specifically, we explore how strong color casts caused by incorrectly applied computational color constancy - referred to as white balance (WB) in photography - negatively impact the performance of DNNs targeting image segmentation and classification. In addition, we discuss how existing image augmentation methods used to improve the robustness of DNNs are not well suited for modeling WB errors. To address this problem, a novel augmentation method is proposed that can emulate accurate color constancy degradation. We also explore pre-processing training and testing images with a recent WB correction algorithm to reduce the effects of incorrectly white-balanced images. We examine both augmentation and pre-processing strategies on different datasets and demonstrate notable improvements on the CIFAR-10, CIFAR-100, and ADE20K datasets.

1. Introduction

The paper examines how incorrect white balance (WB), a global image manipulation, affects DNN classification and segmentation. It introduces realistic WB-error augmentation and evaluates it alongside WB correction, reporting improvements on CIFAR-10, CIFAR-100, and ADE20K.

1. Introduction

  • Incorrect WB can produce unpredictable DNN results because web image collections are biased toward correctly balanced images, despite incorrect WB being unavoidable in real applications.Strong color casts are often discarded by users, contributing to that bias.Source: §1 · p. 1
  • The paper evaluates WB errors in image classification and semantic segmentation and introduces a color augmentation method that emulates realistic WB degradation.It also examines preprocessing training and testing images with a WB correction method.Source: §1 · pp. 1–2
  • Both proposed WB augmentation and preprocessing correction improve results on color-constancy-error inputs across CIFAR-10, CIFAR-100, and ADE20K.Source: §1 · p. 2

2. Related Work

The related work explains the camera pipeline behind WB, describes the difficulty of correcting WB errors in finished sRGB images, and contrasts those errors with existing augmentations. It positions the paper's camera-based augmentation and preprocessing as approaches that target incorrect WB without changing the DNN architecture or training regime.

2. Related Work

  • WB is applied to raw-RGB values to estimate scene illumination, after which camera image-signal processors apply nonlinear finishing operations to produce sRGB images.Source: §2 · p. 2
  • Correcting WB in sRGB is non-trivial because nonlinear camera finishing must be undone, and the needed metadata is rarely available for web images.WB-sRGB estimates a polynomial correction matrix for an sRGB image.Source: §2 · p. 2
  • Existing color augmentations can create unrealistic colors and do not reproduce camera WB casts; prior simplified-pipeline augmentation omitted WB errors after nonlinear processing.Source: §2 · p. 3
  • The paper's augmentation and preprocessing target images directly and require no changes to DNN architecture or training regime.Source: §2 · p. 3

3. Effects of WB Errors on Pre-trained DNNs

Testing pre-trained models on images with simulated white-balance settings shows performance losses in both classification and semantic segmentation. The same scene’s DNN attention also changes with its white-balance setting.

Motivation

  • Changing the WB setting considerably alters a DNN’s attention to the same scene.Figure 3 illustrates two settings applied to the same image.Source: §3 · p. 3, Figure 3 · p. 3

Evaluation setup

  • The evaluation applies simulated WB errors to validation images because creating a newly labeled test set would be an enormous task.The models were trained on the original ImageNet and ADE20K training sets.Source: §3 · p. 3

Classification

  • In most ImageNet classification cases, incorrect white balance causes an accuracy drop of approximately 10%.The evaluation uses five color temperatures and two photo-finishing styles per validation image.Source: §3 · p. 3, Table 1 · p. 4

Semantic Segmentation

  • On ADE20K, incorrectly white-balanced images cause a performance drop of over 2% for semantic segmentation.The reported measures are intersection-over-union (IoU) and pixel-wise accuracy (pxl-acc).Source: Table 2 · p. 4, §3 · p. 4

4. Proposed Method to Emulate WB Errors

The method emulates incorrect white balance directly in sRGB by learning transformations from paired images, then retrieves and combines transformations from color-similar training images to create variants at different color temperatures and photo-finishing styles.

4. Proposed Method to Emulate WB Errors

  • The pipeline applies a retrieved transformation to an input sRGB image to emulate a target color temperature and photo-finishing style without access to its original raw-RGB image.The approach performs this emulation directly in sRGB, building on the WB-sRGB correction framework.Source: §4 · p. 4, §4.5 · p. 5

4.1. Dataset

  • The training data contains 1,797 correctly white-balanced sRGB images, each paired with ten images rendered at five color temperatures and two photo-finishing styles.The temperatures are 2850K, 3800K, 5500K, 6500K, and 7500K; the styles are Camera Standard and Adobe Standard.Source: §4.1 · p. 4

4.2. Color Mapping

  • A polynomial fit in a nine-dimensional RGB feature space maps each correct-WB image to its corresponding target image using a nonlinear 3×9 color transformation matrix.The feature mapping includes RGB channels, pairwise products, and squared channels; ten matrices are computed per training image.Source: §4.2 · p. 4

4.3. Color Feature

  • Each training image is represented by an RGB-uv projected color histogram compressed by PCA into a 55-dimensional feature vector.The compact feature is stored alongside the image’s transformations and the PCA coefficients and bias.Source: §4.3 · p. 4

4.4. KNN Retrieval

  • For a new input, the method uses L2 distance between compact color features to retrieve transformation matrices from the nearest k training images.Retrieval is based on color distribution, not scene content.Source: §4.4 · pp. 4–5

4.5. Transformation Matrix

  • The retrieved transformations are weighted using a radial-basis-function parameter and combined into a final matrix for rendering the target-temperature image.The weighted matrix is applied to produce the re-rendered image.Source: §4.5 · p. 5

5. Experiments

The experiments test three robustness strategies—WB correction at inference, correction of training and test images, and WB augmentation—against standard color jittering on classification and segmentation tasks. Across CIFAR-10, CIFAR-100, and ADE20K, WB augmentation and WB correction improve performance on images with WB errors.

5.1. Experimental Setup

  • The experiments compare inference-time WB correction, correction of both training and test images, and WB augmentation against RGB and HSV jittering.Inference-time correction assumes correctly white-balanced training images; WB-sRGB is used because diagonal-based correction gives inadequate results on sRGB images.Source: §5 · pp. 5–6, §5.1 · p. 6

5.2. Network Training

  • The study trains SmallNet and AlexNet for CIFAR classification and fine-tunes SegNet for ADE20K segmentation, using original, WB-corrected, or color-augmented training images.Models were trained for equal iteration counts, so the additional augmented images’ full potential was not exploited.Source: §5.2 · pp. 6–7

5.3. Results on Cat-1

  • WB-augmented models achieved the best results on expanded CIFAR-10 and CIFAR-100 tests containing strong color casts.Each expanded test set contained 100,000 images after WB emulation.Source: §5.3 · p. 7, Table 3 · p. 7
  • SegNet achieved its best ADE20K validation results with WB augmentation, measured by pixel-wise accuracy and IoU.Figure 7 provides qualitative examples of segmentation with and without color augmentation under different WB settings.Source: §5.3 · p. 7, Table 4 · p. 7, Figure 7 · p. 8

5.4. Results on Cat-2

  • Cat-2 evaluates 15,098 camera-pipeline-rendered images with varied WB settings, while Cat-1 expands dataset tests using synthetic WB emulation.Cat-2 has fewer images with provided ground-truth labels; the Cat-1 testing images use the same WB-emulation method as training augmentation.Source: §5 · p. 6, §5.4 · p. 7
  • On Cat-2, top-1 accuracy fell by ∼6% on testing images modified with incorrect white balance versus properly white-balanced images using in-camera AWB; best accuracies came from preprocessing training and testing images or end-to-end WB augmentation.Source: §5.4 · p. 8, Table 5 · p. 8

6. Conclusion

The paper examines how computational color-constancy errors affect DNN classification and segmentation, and introduces WB augmentation that mimics those errors. WB correction and training with WB-augmented images improve results on CIFAR-10, CIFAR-100, and ADE20K.

6. Conclusion

  • WB correction and training with WB-augmented images improve DNN results on CIFAR-10, CIFAR-100, and ADE20K.The paper introduces augmentation designed to mimic WB errors and examines their impact on classification and semantic segmentation.Source: §6 · p. 8

Related papers

Loading 1912.06960v1…