Source-linked AI summary
Image Quality Assessment: Unifying Structure and Texture Similarity
Keyan Ding, Kede Ma, Shiqi Wang, Eero P. Simoncelli
TL;DR
Existing full-reference IQA methods are overly sensitive to texture resampling because they compare aligned pixels or responses, despite human observers judging resampled textures as similar. DISTS transforms images with a VGG-based multi-scale representation and combines texture and structure similarity, with parameters optimized against human ratings and texture-resampling tolerance. It correlates well with human quality judgments, remains tolerant of texture substitution and mild geometric distortions, and performs competitively in texture classification and retrieval.
Problem
Pointwise or convolution-response comparisons in existing full-reference IQA models are poorly suited to visual textures whose repeated elements vary in spatial arrangement.
Method
DISTS uses a pre-trained VGG representation, global feature-map means for texture statistics, and a combined structure-texture similarity measure optimized for human ratings and texture resampling.
Results
DISTS predicts human quality ratings on textures and natural photographs, tolerates texture resampling and mild geometric distortions, and performs well in texture classification and retrieval.
Takeaways & Limitations
DISTS unifies structure and texture similarity in a full-reference IQA method with built-in tolerance to texture resampling.
Takeaways & Limitations
DISTS remains highly non-convex, with abundant saddle points and plateaus, and recovering images from random noise requires many more iterations.
Abstract
from arXiv · showhide
Objective measures of image quality generally operate by comparing pixels of a "degraded" image to those of the original. Relative to human observers, these measures are overly sensitive to resampling of texture regions (e.g., replacing one patch of grass with another). Here, we develop the first full-reference image quality model with explicit tolerance to texture resampling. Using a convolutional neural network, we construct an injective and differentiable function that transforms images to multi-scale overcomplete representations. We demonstrate empirically that the spatial averages of the feature maps in this representation capture texture appearance, in that they provide a set of sufficient statistical constraints to synthesize a wide variety of texture patterns. We then describe an image quality method that combines correlations of these spatial averages ("texture similarity") with correlations of the feature maps ("structure similarity"). The parameters of the proposed measure are jointly optimized to match human ratings of image quality, while minimizing the reported distances between subimages cropped from the same texture images. Experiments show that the optimized method explains human perceptual scores, both on conventional image quality databases, as well as on texture databases. The measure also offers competitive performance on related tasks such as texture classification and retrieval. Finally, we show that our method is relatively insensitive to geometric transformations (e.g., translation and dilation), without use of any specialized training or data augmentation. Code is available at https://github.com/dingkeyan93/DISTS.
1 BACKGROUND
Full-reference IQA methods often improve on MSE but remain tied to pointwise or aligned comparisons, making them poorly suited to visually similar texture samples with different feature arrangements. Texture analysis therefore motivates statistical representations that capture appearance beyond exact spatial correspondence.
- Limitations of Existing IQA: SSIM and related perceptual IQA methods rely on image alignment, so they are highly sensitive to different samples or cropped regions of the same texture.The precise arrangement of texture features can change substantially without a comparable perceptual change.
- Limitations of Existing IQA: Most knowledge-driven full-reference IQA models compare pixels or convolution responses point by point, limiting their ability to handle visual textures.Visual textures contain repeated elements whose locations, sizes, colors, or orientations may vary spatially.
- Texture Similarity: Different images of the same texture can look nearly identical to humans while differing substantially in pixel intensities.This mismatch is especially relevant for spatially randomized repeated elements.
- Texture Analysis: Computational texture features such as Gabor responses and local binary patterns classify textures effectively but correlate poorly with human texture-similarity judgments.More sophisticated statistics, including correlations of complex wavelet coefficients, have shown greater power for texture synthesis.
2 THE DISTS INDEX
DISTS transforms images with an injective, differentiable VGG-based representation, models texture through spatial feature-map means, and combines texture and structural comparisons into a perceptual distance.
- 2.1 Initial Transformation: DISTS begins with a VGG16-based, multi-scale transformation designed to be continuous, differentiable, and injective.The architecture replaces max pooling with weighted ℓ2 pooling and retains the input image as a zeroth-layer feature map.
- 2.1 Initial Transformation: Injectivity preserves distinct image inputs in the representation, enabling the resulting quality measure to satisfy the identity of indiscernibles.The paper contrasts this property with surjective or information-discarding transformations used by other IQA methods.
- 2.3 Perceptual Distance Measure: The method combines texture and structural terms across the representation to capture both tolerance to texture resampling and sensitivity to structural distortions.The recovery experiment optimizes IQA distortion measures from JPEG-compressed and white-Gaussian-noise initializations, illustrating the role of the distance in perceptual optimization.
- 2.2 Texture Representation: Spatial means of VGG feature maps provide a 1,475-statistic parametric model that can synthesize visually similar textures.Matching statistics through conv5_3 captures both early-layer intensity and color information and later-layer shape and structure information.
- 2.2 Texture Representation: The texture model uses gradient descent to synthesize an image whose feature-map means match those of a target texture.The synthesis objective compares the spatial averages of corresponding channels in the target and synthesized representations.
- 2.3 Perceptual Distance Measure: The perceptual distance separates texture and structure measurements because feature-map responses retain spatial dependencies that make independent-error norms inappropriate.Texture is measured from global means, while structure is measured from global correlations, following the form of SSIM.
3 EXPERIMENTS
Experiments evaluate DISTS on conventional and texture quality prediction, texture classification and retrieval, and robustness to geometric transformations. Ablations identify global SSIM-like measurements, texture-resampling training, and architectural choices as important contributors.
- Quality Prediction: DISTS was trained on KADID and evaluated on LIVE, CSIQ, and TID2013 using PLCC, SRCC, and KRCC.A four-parameter function was fitted before computing PLCC.
- Quality Prediction: DISTS performed favorably against classic and CNN-based IQA models, though MAD, FSIMc, and GMSD achieved the best overall results across the three databases and metrics.On TID2013, DISTS was nearly linear in MOS after fitting.
- Quality Prediction: On BAPPS, DISTS achieved performance comparable to LPIPS despite not being trained on BAPPS or a similar database.BAPPS includes synthetic, CNN-based, and real-world image-processing distortions, with human judgments collected through 2AFC experiments.
- Texture Classification and Retrieval: DISTS achieved the best retrieval performance on both CBT and GBT, while GBT performance was slightly lower than CBT, indicating the importance of color information.Errors were primarily associated with textures containing noticeable inhomogeneities.
- Geometric Invariance and Ablation: DISTS showed high robustness across geometric transformations and distortions, attributed to ℓ2 pooling, global quality measurements, and optimization for texture-resampling invariance.The ablation study found that global SSIM-like distance outperformed Euclidean and local SSIM-like distances, while texture-related training improved geometric invariance but slightly reduced standard-IQA performance.
4 CONCLUSIONS
DISTS unifies structure and texture similarity to predict human quality ratings across textures and natural photographs, while tolerating texture resampling and mild geometric distortions. The method also performs well on texture classification and retrieval, but its fixed linear combination and non-convex optimization leave practical limitations.
- Empirical conclusions: DISTS unifies structure and texture similarity, predicts human quality ratings on textures and natural photographs, and remains robust to mild geometric distortions.The conclusions summarize its built-in tolerance to texture resampling and robustness to mild geometric changes.
- Empirical conclusions: DISTS also performs well on texture classification and retrieval.
- Model properties: DISTS uses global means of pre-trained VGG convolution responses to form a relatively low-dimensional statistical characterization of texture appearance.The representation is supported by texture synthesis results and is described as a universal parametric texture model.
- Limitations: The authors identify local distortion measurement as future work, potentially allowing adaptive selection between structure and texture terms instead of fixed weights.
- Limitations: DISTS remains highly non-convex, and recovery from random noise with stochastic gradient descent requires more iterations than SSIM.Increasing the structure-term weight can accelerate convergence, but achieving reasonable texture invariance constrains the learned weights.
- Applications: DISTS-based optimization can improve visual quality in denoising, deblurring, super-resolution, and compression applications.The authors describe these as initial results and report noticeable improvements in visual quality.