PaperlayerView exact revision on arXiv

Source-linked AI summary

Learned Perceptual Image Enhancement

Hossein Talebi, Peyman Milanfar

arXiv:1712.02864v1cs.CV

TL;DR

L1 and L2 losses used for image enhancement do not necessarily produce perceptually compelling results, and reference-based similarity losses lack an implicit prior for image quality. The paper uses a human-preference-trained NIMA quality predictor alongside data fidelity to train enhancement operators. The authors report perceptually improved enhancement results, including higher average NIMA scores for detail enhancement methods on FiveK test images.

  • Problem

    L1 and L2 losses can yield perceptually inferior enhancement, while reference-based similarity losses lack implicit prior information on perceptual image quality.

    Source: §1 · p. 1

  • Method

    The paper combines a reference-based data-fidelity term with a differentiable NIMA score trained on human aesthetic ratings as a perceptual loss for enhancement.

    Source: §1.1 · p. 1, §2 · p. 3

  • Results

    The authors report perceptually superior detail, tone, and color, and higher average NIMA scores with lower variance for their loss on FiveK detail-enhancement results.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    A human-preference-based quality score can guide image-enhancement algorithms toward perceptually improved results in the tasks studied.

    Source: §1.1 · p. 1, §5 · p. 6

Abstract

from arXiv · show

Learning a typical image enhancement pipeline involves minimization of a loss function between enhanced and reference images. While L1 and L2 losses are perhaps the most widely used functions for this purpose, they do not necessarily lead to perceptually compelling results. In this paper, we show that adding a learned no-reference image quality metric to the loss can significantly improve enhancement operators. This metric is implemented using a CNN (convolutional neural network) trained on a large-scale dataset labelled with aesthetic preferences of human raters. This loss allows us to conveniently perform back-propagation in our learning framework to simultaneously optimize for similarity to a given ground truth reference and perceptual quality. This perceptual loss is only used to train parameters of image processing operators, and does not impose any extra complexity at inference time. Our experiments demonstrate that this loss can be effective for tuning a variety of operators such as local tone mapping and dehazing.

1. Introduction

Supervised enhancement models trained with L1 or L2 can produce perceptually inferior results, while existing similarity losses require references and lack an implicit image-quality prior. The paper introduces NIMA as a human-preference-based perceptual loss for enhancement and describes AVA as its large-scale aesthetic-rating dataset.

1. Introduction

  • L1 and L2 losses are optimization-friendly but can yield perceptually inferior enhancement, while reference-based similarity losses lack an implicit image-quality prior.Feature-space perceptual losses have succeeded in image transformation, but their use in per-pixel optimization remained to be investigated.Source: §1 · p. 1

1.1. Our contributions

  • NIMA learns photo aesthetics and quality from a large-scale human-rated dataset, then serves as a perceptual loss for enhancement tasks.The paper focuses on lighting, color, tone, contrast, and sharpness enhancement.Source: §1.1 · p. 1
  • The proposed loss combines data fidelity with a NIMA-based perceptual score, and experiments report perceptually superior detail, tone, and color in photos.The authors also report that the quality score improves when training with the proposed loss.Source: §1.1 · pp. 1–2

1.2. A Large-Scale Database for Aesthetic Visual Analysis (AVA) [27]

  • AVA contains 255,000 images rated aesthetically by amateur photographers, with ratings gathered across nearly 900 photography challenges.Visual inspection associates high scores with good lighting, tone, contrast, and sharpness; noise and blur are typically rated poorly.Source: §1.2 · p. 2
  • AVA ratings also reflect image semantics and composition: visually dull scenery is rated low, while professional framing and composition can increase appeal.Source: §1.2 · p. 2

2. Perceptual Loss

The perceptual loss combines reference-image fidelity with an AVA-trained NIMA quality penalty. NIMA predicts rating distributions, trained with EMD, and its mean quality score supplies the perceptual term.

2. Perceptual Loss

  • The loss adds an AVA-trained NIMA perceptual penalty to a reference-image distance, with γ controlling the penalty’s strength.The reference distance can use L1, L2, or a hybrid such as Huber.Source: §2 · p. 3

2.1. NIMA: Neural Image Assessment

  • NIMA predicts ten AVA quality-score outputs using classifier architectures; Inception-v2 produced the best aesthetic and quality-score predictions among those tested.Its final layer uses average pooling followed by ten neurons, and fully convolutional layers allow arbitrary-sized inputs and backpropagation to pixels.Source: §2.1 · p. 3, Figure 3 · p. 3

2.1.1 Training NIMA

  • NIMA training estimates each image’s human-rating probability mass function over the ten AVA score buckets.The ground-truth distribution is formed from user ratings assigned to each image.Source: §2.1.1 · p. 3
  • NIMA uses normalized Earth Mover’s Distance to train on ordered score buckets, penalizing rating errors according to the distance between classes.The predicted probabilities are softmax-normalized, and the training framework sets the class-distance norm to 2.Source: §2.1.1 · p. 3
  • The perceptual loss uses the mean of NIMA’s predicted rating distribution as the image’s quality score.That score is defined as the weighted sum of the ten score buckets and is shown in example plots in Figure 4.Source: §2.1.1 · p. 4, Figure 4 · p. 4

3. Enhancement CNN

The enhancement network uses CAN, whose dilated convolutions aggregate global context through expanding receptive fields. Its padding is modified to avoid artifacts from highly dilated zero-padded layers.

3. Enhancement CNN

  • CAN’s dilated convolutions expand receptive fields through the network, aggregating global context for each output pixel and allowing arbitrary-sized inputs without rescaling.The architecture is chosen for global enhancements such as tone and contrast.Source: §3 · p. 4, Figure 5 · p. 4
  • The enhancement CAN has ten convolutional layers with 32 feature maps, using 3 × 3 intermediate kernels and a 1 × 1 final kernel.Dilation increases exponentially by layer except in the last layer; leaky ReLU is used except at the output.Source: §3 · p. 4
  • Unlike the cited CAN implementation, intermediate feature maps are symmetrically padded to avoid artifacts from highly dilated zero-padded layers.The final layer is not dilated.Source: §3 · p. 4

4. Experimental Results

The experiments compare NIMA quality predictors and enhancement models trained with a perceptual loss. The loss improves visual enhancement results and yields higher average, lower-variance NIMA scores across FiveK test images.

4.1. NIMA Results

  • NIMA(Inception-v2) shares the highest AVA two-class accuracy with, while NIMA(VGG16) and NIMA(Inception-v2) outperform [19] in rank correlation; Inception-v2 was selected considering performance and complexity.Unlike, which applies multiple VGG16 networks to image patches, NIMA(Inception-v2) requires one network pass.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2. Enhancement Results

  • Enhancement models were trained and tested on FiveK, with L2 fidelity, γ = 0.0001, and Adam training for 5 × 10^6 steps.Training images were rescaled to 480 × 640; large γ values may cause unexpected distortions, while small values may yield barely visible improvements over L2.Source: §4.2 · p. 5
  • Compared with and [5], perceptual loss better preserves dark and bright detail and dehazing color saturation; FiveK NIMA scores also have higher average and lower variance.Figure 6 reports the score statistics for the input, reference, CAN(L2), and CAN(L2+NIMA).Source: §4.2 · p. 5, Figure 6 · p. 5

5. Conclusions

The conclusion presents a differentiable perceptual loss built on a no-reference quality predictor trained on human-rated images. Visual comparisons report better preservation of image detail than local tone mappers [28] and [5].

5. Conclusions

  • The loss combines a differentiable, human-rater-trained no-reference quality predictor with image-enhancement training.The authors identify exploring other applications of the loss as future work.Source: §5 · pp. 5–6
  • Compared with local tone mappers [28] and [5], the results better preserve detail in dark and bright regions.The conclusion points to comparisons in both dark and bright regions.Source: Figure 7 · p. 6, Figure 8 · p. 7

Related papers

Loading 1712.02864v1…