PaperlayerView exact revision on arXiv

Source-linked AI summary

Loss Functions for Neural Networks for Image Processing

Hang Zhao, Orazio Gallo, Iuri Frosio, Jan Kautz

arXiv:1511.08861v3cs.CV

TL;DR

The paper addresses the limited attention given to loss functions in neural-network image restoration, where ℓ2 is the default despite weak alignment with human image-quality perception. It compares differentiable alternatives across restoration tasks and proposes a novel loss combining ℓ1 and MS-SSIM. Better loss choices improve results with the same architecture, and the proposed loss outperforms the alternatives under the considered metrics.

  • Problem

    Image-restoration research has focused on network architectures while generally using ℓ2, although ℓ2 correlates poorly with human-perceived image quality.

  • Method

    The paper compares ℓ1, ℓ2, SSIM, and MS-SSIM across three restoration tasks and defines a novel differentiable loss combining ℓ1 and MS-SSIM.

  • Results

    The proposed loss outperforms other networks in the considered traditional and perceptually motivated image-quality metrics.

  • Takeaways & Limitations

    Loss selection has a strong impact on image-restoration quality even when the network architecture remains unchanged.

Abstract

from arXiv · show

Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems. The impact of the loss layer of neural networks, however, has not received much attention in the context of image processing: the default and virtually only choice is L2. In this paper, we bring attention to alternative choices for image restoration. In particular, we show the importance of perceptually-motivated losses when the resulting image is to be evaluated by a human observer. We compare the performance of several losses, and propose a novel, differentiable error function. We show that the quality of the results improves significantly with better loss functions, even when the network architecture is left unchanged.

I. INTRODUCTION

The paper argues that image-processing research has overlooked the loss layer while relying on ℓ2, despite its poor alignment with human-perceived image quality. It evaluates alternative losses and reports improved restoration quality without changing the network architecture.

  • I. INTRODUCTION: Image-processing research has focused primarily on architectures, while the loss layer has remained comparatively neglected and ℓ2 the de facto standard.The paper frames loss selection as an underexplored factor in neural-network image restoration.
  • I. INTRODUCTION: ℓ2 correlates poorly with human-perceived image quality because it does not model the human visual system’s sensitivity to local image characteristics.Its assumptions include noise effects independent of local image content and white Gaussian noise.
  • I. INTRODUCTION: The study compares ℓ2, ℓ1, SSIM, and MS-SSIM across super-resolution, JPEG artifact removal, and joint denoising plus demosaicking.The selected perceptual metrics are established and differentiable for backpropagation.
  • I. INTRODUCTION: Perceptually motivated losses and ℓ1 outperform ℓ2 on the considered restoration use cases, while the proposed loss combines advantages of ℓ1 and MS-SSIM.The paper also analyzes convergence properties and implements the studied loss layers for Caffe.
  • I. INTRODUCTION: Networks trained with ℓ2 produce visible splotchy artifacts in flat regions during joint denoising and demosaicking.Figure 1 highlights this artifact as a practical consequence of the loss choice.

B. Evaluating image quality

The paper reviews why traditional ℓ2 and PSNR poorly reflect human image-quality judgments and motivates differentiable perceptual alternatives. It connects these metrics to the loss layer and illustrates their practical relevance in restoration outputs.

  • B. Evaluating image quality: ℓ2 and PSNR do not correlate well with human perception because they fail to capture intricate characteristics of the human visual system.This mismatch motivates evaluating alternative image-quality measures.
  • B. Evaluating image quality: SSIM evaluates image similarity using the human visual system’s sensitivity to changes in local structure.MS-SSIM extends this analysis across scales related to factors such as image-to-observer distance.
  • B. Evaluating image quality: SSIM-based indexes had not previously been adopted to train neural networks, motivating their investigation as loss functions.The paper focuses on reference-based, differentiable measures suitable for backpropagation.
  • B. Evaluating image quality: More recent HVS-based indexes can outperform SSIM and MS-SSIM, but their greater complexity or nondifferentiability can hinder optimization use.Examples include IW-SSIM, VIF, GMSD, and FSIM/FSIMc.
  • B. Evaluating image quality: In joint denoising and demosaicking, ℓ2 strongly attenuates noise in flat regions but produces visible splotchy artifacts.The paper attributes this behavior to ℓ2’s error weighting conflicting with the HVS’s sensitivity to variations in textureless regions.

A. The ℓ1 error

The paper introduces ℓ1 as an alternative loss whose error weighting and convergence behavior may reduce artifacts relative to ℓ2. It reports visible improvements, while noting that ℓ1 remains imperfect.

  • A. The ℓ1 error: ℓ1 does not over-penalize larger errors like ℓ2, so the two losses may exhibit different convergence properties.This motivates retraining the same network with ℓ1 instead of ℓ2.
  • A. The ℓ1 error: For each pixel, the ℓ1 loss derivative depends on that pixel’s processed and ground-truth values, with zero derivative contributions from other pixels.The derivative is undefined at zero, so the paper uses sign(0) = 0.
  • A. The ℓ1 error: ℓ1 removes the splotchy sky artifacts visible with ℓ2, but residual artifacts remain in the sky and near building boundaries.Thus, ℓ1 improves over ℓ2 without producing fully optimal results.

B. SSIM

SSIM provides a perceptually motivated loss by evaluating local image structure rather than only pixelwise error. Its neighborhood-based computation requires special handling near patch boundaries, but convolutional processing enables central-pixel training with distributed gradients.

  • SSIM is motivated by the human visual system’s sensitivity to local structure, making it suitable when networks should produce visually pleasing images.
  • SSIM computes local statistics through Gaussian filtering, with means and standard deviations depending on the neighborhood around each pixel.
  • SSIM loss requires a neighborhood as large as the Gaussian-filter support, so it cannot be calculated in some boundary regions of a patch.
  • The convolutional network approximates patch loss using the center pixel while still back-propagating error to pixels within the Gaussian support.

C. MS-SSIM

MS-SSIM extends SSIM across multiple scales to reduce sensitivity to the choice of a single Gaussian-filter scale. The paper approximates the multiscale computation with several full-resolution Gaussian scales to make training faster.

  • Smaller Gaussian scales can reintroduce splotchy artifacts, whereas larger scales can preserve noise near edges.
  • MS-SSIM analyzes image structure using a dyadic pyramid with M levels, rather than relying on one SSIM scale.
  • The MS-SSIM loss and its derivatives can be approximated from the loss at the center pixel of each patch.
  • To reduce computational cost, the method replaces pyramid construction with M Gaussian-filter values on the full-resolution patch, each half the preceding value.

D. The best of both worlds: MS-SSIM + ℓ1

MS-SSIM preserves high-frequency contrast, while ℓ1 better preserves colors and luminance. The proposed combination is designed to capture both properties, using a weighted sum with empirically selected α = 0.84.

  • MS-SSIM preserves high-frequency contrast better, whereas ℓ1 preserves colors and luminance but produces less contrast.
  • The paper combines MS-SSIM and ℓ1 to capture the complementary strengths of both loss functions.
  • The combined loss uses a weighted sum, with α = 0.84 selected empirically to roughly balance the two terms.
  • The MS-SSIM component weights ℓ1 error using Gaussian contributions based on each pixel’s influence on the central pixel’s MS-SSIM.

IV. RESULTS

The experiments compare loss functions for joint demosaicking plus denoising and validate the findings on super-resolution and JPEG artifact removal. Across visual and quantitative analyses, MS-SSIM+ℓ1 performs best overall, while ℓ1 consistently improves on ℓ2.

  • The study focuses on joint demosaicking plus denoising and confirms its findings on super-resolution and JPEG artifact removal.
  • The network uses a fully convolutional CNN trained with six cost functions on noisy RGB patches generated from 700 MIT-Adobe FiveK images.
  • Visually, ℓ2 produces splotchy flat-region artifacts, SSIM variants produce edge noise, MS-SSIM dulls colors, and MS-SSIM+ℓ1 generates the best results.
  • The quantitative comparison includes CNNs trained with different losses and BM3D, evaluated using traditional and perceptually inspired image-quality indexes.
  • ℓ1 yields superior average quality for every evaluated image-quality metric compared with ℓ2, including ℓ2 and PSNR evaluation.
  • MS-SSIM+ℓ1 remains superior to ℓ2+MS-SSIM across all indexes considered, while ℓ2+MS-SSIM improves perceptual metrics over ℓ2 alone.

B. Super-resolution

The super-resolution experiments compare restoration losses across representative visual results, emphasizing artifacts produced by ℓ2 and the broader evaluation of alternative metrics.

  • ℓ2 produces grating artifacts on the wing’s black stripes and around the girl’s face.
  • JPEG de-blocking comparisons examine artifact removal around edges and in relatively flat regions where blocketization is more apparent.
  • The experiments use stride 7 rather than stride 8 because it better removes JPEG artifacts while producing sharper images.
  • Additional numerical and visual results are provided in the supplementary material.

V. DISCUSSION

The discussion examines why losses differ in optimization and visual quality, highlighting trade-offs in SSIM-based metrics and the scope of the analysis.

  • A. Convergence of the loss functions: Networks trained with different losses can outperform networks optimized directly for the evaluated loss, indicating that convergence affects final performance.The authors relate this behavior to smoothness and local convexity, with ℓ2 more prone to local minima than ℓ1.
  • A. Convergence of the loss functions: Alternating losses can improve ℓ2 optimization, but the resulting visual quality depends on the switching order and may retain splotchy artifacts.Training with ℓ1 first and ℓ2 afterward achieves the lowest reported ℓ2 loss, while the reverse order produces results visually similar to ℓ1 alone.
  • B. On the performance of SSIM and MS-SSIM: SSIM performance depends on σG: smaller values improve edges but worsen flat regions, whereas larger values reduce splotchy artifacts in flat areas.The support size changes how the metric responds to noise near edges and across image regions.
  • C. Visual and quantitative comparisons: The visual comparisons report that Mix addresses ℓ2 splotchy artifacts and SSIM-based color changes in denoising plus demosaicking, and outperforms ℓ2 and ℓ1 for JPEG de-blocking.The table caption states that quality metrics average SSIM-family and FSIM values across the three color channels.
  • B. On the performance of SSIM and MS-SSIM: MS-SSIM reduces noise around edges but does not eliminate color changes in flat bright regions, reflecting SSIM’s limited sensitivity to uniform bias.The contrast-based term reduces the impact of bias when the background is bright.
  • Scope and limitations: The paper analyzes SSIM and MS-SSIM within neural-network image restoration rather than defining a new general-purpose image-quality metric.It also notes that SSIM-based color-image losses are approximations because the metrics were originally designed for grayscale images.

C. Are we committing the inverse crime?

The authors assess whether synthetic-data generation may create an inverse crime and test robustness by varying the Gaussian-filter standard deviation.

  • C. Are we committing the inverse crime?: Synthetic ground truth is necessary for the paper’s numerical evaluations, but the authors caution that synthetic-data design can affect the validity of inverse-problem results.For super-resolution, low-resolution inputs are generated by low-pass filtering and down-sampling original images.
  • C. Are we committing the inverse crime?: 0.9829 to 0.9752: Mix’s average MS-SSIM drops when the Gaussian-filter standard deviation changes from 3 to 5 pixels.Networks trained with ℓ1 and ℓ2 show similar drops, reaching 0.9744 and 0.9748 respectively.

VI. CONCLUSIONS

The conclusions emphasize loss-layer choice as an overlooked determinant of image-restoration quality. Across restoration tasks, the proposed loss improves results while retaining an efficient fully convolutional network design.

  • VI. CONCLUSIONS: The paper focuses on replacing the de facto ℓ2 loss with alternative and novel losses for super-resolution, JPEG artifact removal, and joint denoising plus demosaicking.The analysis uses both traditional and perceptually motivated image-quality metrics.
  • VI. CONCLUSIONS: The proposed loss outperforms the other tested losses and enables the joint denoising and demosaicking network to outperform CFA-BM3D.The conclusion attributes this result to the proposed loss while noting the network is fully convolutional and requires no aggregation step.
  • VI. CONCLUSIONS: Alternating a related loss such as ℓ1 can help an ℓ2-trained network find a better solution for ℓ2.The authors connect this observation to the convergence properties of the losses.
  • VI. CONCLUSIONS: SSIM is evaluated near edges under Gaussian noise and under uniform bias in bright regions, illustrating distinct sensitivity patterns.Figure 10 specifically compares support effects near an edge with reduced sensitivity to bright-region bias.
  • VI. CONCLUSIONS: The implementation of the described layers is made available to the research community.
Loading 1511.08861v3…