Source-linked AI summary
Understanding SSIM
Jim Nilsson, Tomas Akenine-Möller
TL;DR
SSIM is widely used for image-quality assessment, but its mathematical behavior has received limited deep scrutiny. The paper analyzes its factors and shows unexpected, sometimes undefined, and nonintuitive results in synthetic and realistic cases. It concludes that SSIM should be used cautiously or replaced because it may distort image-quality assessment.
Problem
SSIM’s widespread use has outpaced deep scrutiny of its mathematical and perceptual properties, making the validity of its image-quality interpretations important to examine.
Method
The paper reviews SSIM and mathematically inspects its luminance, variance, and correlation factors using synthetic and realistic image examples.
Results
SSIM can produce unexpected, invalid, and counterintuitive results, including undefined outputs for some parameter settings.
Takeaways & Limitations
SSIM can lead image-quality assessment and neural-network training in the wrong direction, so the paper recommends using it with care or preferably replacing it.
Takeaways & Limitations
Perceptual phenomena such as contrast masking apply within one image’s frame of reference, not directly to variations between compared images.
Abstract
from arXiv · showhide
The use of the structural similarity index (SSIM) is widespread. For almost two decades, it has played a major role in image quality assessment in many different research disciplines. Clearly, its merits are indisputable in the research community. However, little deep scrutiny of this index has been performed. Contrary to popular belief, there are some interesting properties of SSIM that merit such scrutiny. In this paper, we analyze the mathematical factors of SSIM and show that it can generate results, in both synthetic and realistic use cases, that are unexpected, sometimes undefined, and nonintuitive. As a consequence, assessing image quality based on SSIM can lead to incorrect conclusions and using SSIM as a loss function for deep learning can guide neural network training in the wrong direction.
NVIDIA
Figure 1 shows how SSIM maps reference–test image comparisons into visual similarity maps and separate luminance, chrominance, and structure components. The examples include cases where SSIM responses diverge from visible differences.
- White denotes SSIM = 1, black denotes SSIM = 0, and negative SSIM values map from red toward green in the heatmap.
- The figure separates reference images, test images, SSIM images, and the luminance, chrominance, and structure components.
- SSIM detects dark text but hardly reacts to chrominance differences and misses large areas on a red door and window shutter.
- Edges can receive SSIM values close to one despite large errors across the test image, while visually similar Einstein images can receive large errors when highly zoomed-in.
1 Introduction
The paper introduces SSIM as a widely used index, then reviews its formulation and implementation assumptions. It focuses on the local luminance, contrast, and structure factors and their aggregation into MSSIM.
- SSIM has been widely adopted as an image-quality index, with the original paper receiving over 20,000 Google Scholar citations.
- The paper reviews SSIM and reports that it can produce unexpected or invalid results in simple use cases and real image pairs.
- The input color space is unspecified, so the analysis assumes images are encoded in approximately gamma-encoded sRGB and viewed directly on screen.
- SSIM computes Gaussian-weighted local statistics over 11×11-pixel patches with σ = 1.5, then forms luminance, contrast, and structure components.
- MSSIM pools per-pixel SSIM values over the entire image to produce a mean image-level value.
- The standard simplification sets α = β = γ = 1 and C3 = C2/2, while constants are parameterized using the image range and K1, K2.
2 The History of SSIM
The paper places SSIM within a history of related image-quality indices, extensions, and comparisons with MSE and PSNR. This history also motivates questioning SSIM’s perception-based interpretation.
- UQI is essentially SSIM without stabilizing constants, while MS-SSIM evaluates contrast and structure across five low-pass-filtered, downsampled scales.
- SSIM has been extended to complex wavelets, video, spherical projections, and medical images, with some extensions designed for translation or rotation tolerance.
- MSSIM is the average SSIM value over an image pair, whereas MS-SSIM is the multi-scale variant of SSIM.
- Prior evaluations report that MS-SSIM performs well for noisy Monte Carlo rendered images, while broader user-study evaluations find contradictory results for several metrics including SSIM.
- SSIM and UQI are not mathematical metrics because they do not claim to satisfy the triangle inequality, although 1 − c(x, y)s(x, y) does.
- Prior work found close mathematical relationships between SSIM*, local MSE, and PSNR, including an approximately linear relation for similar-luminance images with SSIM* in [0.2, 0.8].
- These relationships challenge claims that SSIM is perception-based because MSE is not a perception-based metric.
3 Mathematical Properties
The paper mathematically analyzes SSIM’s luminance, contrast, and structure factors, deriving their minima and examining how the index behaves in synthetic and perceptual examples. These analyses expose nonintuitive, misleading, and sometimes undefined outcomes.
- Minimum Values of the SSIM Factors: SSIM’s luminance, contrast, and structure components can each attain distinct minima under specific image statistics.The analysis minimizes l, c, and s separately using means, variances, and covariance.
- Minimum Values of the SSIM Factors: The SSIM range is (−1, 1], and all three component minima are independent of image range L.Examples in Figure 2 realize the minima for luminance, contrast, and structure.
- Minimum Values of the SSIM Factors: A low-dot-pitch 128/255 image versus a black-white checkerboard can receive near-zero SSIM despite differences being hard for human observers to discern.This example achieves the contrast minimum and conflicts with the reported visual experience.
- Minimum Values of the SSIM Factors: With the full SSIM expression, negative structure values combined with non-integer γ can produce complex-valued results.The structure denominator remains positive, but covariance may be negative; without proper interpretation, implementations can yield undefined results or complex numbers.
- Perceptual Properties: Weber’s law and contrast masking apply to stimuli within one image’s frame of reference, not directly to variations between compared images.This limits the basis for interpreting SSIM’s luminance and contrast factors as perceptually motivated.
- Perceptual Properties: SSIM’s luminance factor can exaggerate small changes near black while remaining insensitive to relatively large changes near white.Figure 3’s black and white comparison curves predict the misleading results observed in Figure 4.
4 Evaluation
The evaluation shows that SSIM can disagree sharply with visible image differences across grayscale, color, gradient, and complex-image examples, and can sometimes produce negative or undefined results.
- 4.1 Luminance: MSSIM misjudges constant grayscale images, assigning low quality to nearly indistinguishable dark pairs and high similarity to visibly different pairs.For black versus 2/255 images, the grayscale difference is 2/255 although MSSIM is low; another row reports similarity despite visible differences.
- 4.1 Luminance: The luminance component’s quadratic normalization denominator exaggerates differences near black, making SSIM poorly aligned with perceived luminance differences.The authors state that the l component explains the misleading results in the grayscale experiments.
- 4.2 Color: SSIM can report extremely high similarity for visibly different color images because RGB-to-grayscale conversion maps distinct colors to the same grayscale value.The color examples reduce red, green, or blue components until the error is approximately 0.99 while SSIM remains extremely high.
- 4.3 Gradients: For mirrored gradients, MSSIM becomes negative at smaller resolutions even though the compared gradients remain similar, reversing the expected perceived-error relationship.The s component falls from 0.86 at 256×256 to −0.10 at 64×64 and −0.90 at 16×16; the authors attribute the negative values to this component.
- 4.3 Gradients: Negative s values combined with non-integer exponents can make SSIM complex or undefined, even for simple distortions.The paper identifies this issue in the two bottom rows of the gradient experiment and notes that such cases are possible for simple distortions.
- 4.4 Complex Images: Complex-image tests reveal false positives, false negatives, edge-related high similarity, and dark-region errors that can dominate the final SSIM output.Examples include missed chrominance and intensity variations, high SSIM near contrast edges, and negative s-component values.
5 Conclusions
The paper concludes that SSIM can produce counterintuitive results despite being useful in some cases, challenging broader beliefs about its capabilities. The authors recommend cautious use or replacement of SSIM and motivate improved image-difference metrics.
- SSIM can generate counterintuitive results in many cases, although it remains useful in some applications.The authors argue that its expanded perceived capabilities exceed the original scope of the work on which it was based.
- The findings open research toward improved metrics for human detection of differences in synthetic, rendered, and natural images.
- The analysis derives minima, ranges, and normalization for SSIM’s luminance, variance, and correlation factors.These properties produce nonintuitive behavior for low luminance values and visually similar but regularly differing local pixel distributions.
- SSIM’s shortcomings may disproportionately affect neural networks used for Monte Carlo rendering, denoising, and reconstruction.Small training variations could interact adversely with the index’s documented weaknesses.
- The authors encourage graphics researchers to use SSIM with care or preferably replace it, because its use may distort or bias image quality assessment.They identify the difference evaluator for alternating images (LIP) as one step toward replacement.