PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphIQA: Learning Distortion Graph Representations for Blind Image Quality Assessment

Simeng Sun, Tao Yu, Jiahua Xu, Wei Zhou, Zhibo Chen

arXiv:2103.07666v4cs.MM

TL;DR

BIQA needs distortion representations that capture both relationships among distortion types and distributions across distortion levels. GraphIQA represents each distortion as a DGR, learns it with TDN and FPN, and reports state-of-the-art performance across synthetic and authentic benchmark datasets.

  • Problem

    Previous BIQA representations do not effectively model relationships between distortions or sample distributions across levels within a distortion type.

  • Method

    GraphIQA represents each distortion as a graph and uses TDN for type relationships and FPN for distributional level prediction.

  • Results

    GraphIQA achieves state-of-the-art performance on benchmark datasets with synthetic and authentic distortions.

  • Takeaways & Limitations

    The learned DGR can serve as distortion prior knowledge for known distortions and help infer the influence of unknown distortions on perceptual quality.

Abstract

from arXiv · show

A good distortion representation is crucial for the success of deep blind image quality assessment (BIQA). However, most previous methods do not effectively model the relationship between distortions or the distribution of samples with the same distortion type but different distortion levels. In this work, we start from the analysis of the relationship between perceptual image quality and distortion-related factors, such as distortion types and levels. Then, we propose a Distortion Graph Representation (DGR) learning framework for IQA, named GraphIQA, in which each distortion is represented as a graph, i.e., DGR. One can distinguish distortion types by learning the contrast relationship between these different DGRs, and infer the ranking distribution of samples from different levels in a DGR. Specifically, we develop two sub-networks to learn the DGRs: a) Type Discrimination Network (TDN) that aims to embed DGR into a compact code for better discriminating distortion types and learning the relationship between types; b) Fuzzy Prediction Network (FPN) that aims to extract the distributional characteristics of the samples in a DGR and predicts fuzzy degrees based on a Gaussian prior. Experiments show that our GraphIQA achieves the state-of-the-art performance on many benchmark datasets of both synthetic and authentic distortions.

I. INTRODUCTION

GraphIQA addresses limitations in distortion representations for BIQA by modeling distortion types and levels as graph structure. Its DGR framework uses TDN and FPN to learn type relationships and level distributions for downstream IQA.

  • Motivation: Existing distortion-classification representations may not distinguish distortion levels or remain robust for authentic distortion with uncertain distortion types.These limitations motivate modeling both distortion type and level.
  • Core idea: GraphIQA represents each distortion type as a graph whose node distribution encodes different distortion levels.The framework models type and level as a hierarchical structure.
  • Core idea: DGR learning contrasts graphs from different distortions while modeling internal sample distributions to learn distortion representations.The two-stage framework pre-trains distortion representations before fine-tuning them for the target IQA dataset.
  • Network design: TDN embeds each DGR into a compact code for type discrimination, while FPN extracts sample-distribution characteristics and predicts fuzzy distortion levels.TDN uses contrastive relationships between distortion types, and FPN uses a Gaussian prior for fuzzy prediction.
  • Scope and outcome: GraphIQA is designed for synthetic, authentic, and multiply distorted IQA tasks and is reported to achieve state-of-the-art performance.The learned DGR is intended to provide distortion prior knowledge across downstream IQA settings.

II. RELATED WORK

Related work establishes the need for general-purpose BIQA under mixed or unspecified distortions and motivates graph-based representation learning. GraphIQA extends distortion-aware representation learning by modeling type relationships and distortion levels rather than treating distortion factors as a plane model.

  • Blind Image Quality Assessment: Distortion-specific BIQA can be accurate when distortion types are known, but authentic datasets often contain mixed and unspecified distortions.This limitation motivates general-purpose BIQA methods.
  • Blind Image Quality Assessment: Deep general-purpose BIQA methods increasingly learn distortion-aware features, including score distributions and auxiliary distortion-type representations.Prior methods use pre-training, probabilistic quality representations, and type-identification tasks.
  • Motivation: GraphIQA targets both relationships between distortion types and relationships between distortion levels through a discriminate-then-rank representation strategy.The framework is presented as a representation-learning approach for both factors.
  • Motivation: Unlike plane-model classification approaches, GraphIQA models distortion-related factors as a graph and learns relationships between types to support generalization to unseen types.The graph formulation is intended to capture structured relationships rather than only class labels.
  • Graph Representation Learning: Graph representation learning encodes non-Euclidean data with relationships and interdependencies, using learned embeddings and neighborhood aggregation.Prior graph methods include GCNs, GATs, and GraphSAGE.

III. METHODOLOGY

GraphIQA models distortion type and level as a hierarchy: each distortion becomes a graph whose node distribution captures level-related structure. The framework is motivated by observed differences in distortion-type score distributions, ordered levels, and content-dependent quality changes.

  • Motivation: Images with different content can undergo different MOS changes across levels, while scores at each level generally follow a Gaussian-like distribution.These observations motivate modeling both content-dependent variation and level distributions.
  • Motivation: Different distortion types produce distinct IQA-score distributions, while higher distortion levels generally correspond to lower scores.Kadid-10k contains 10,125 images spanning 25 distortion types and 5 distortion levels.
  • Motivation: Distortion type and level form a hierarchy in which quality assessment first considers type and then the level distribution within that type.The paper relates this hierarchy to how distorted images are analyzed and uses it to motivate graph representations.
  • Distortion Graph Representation: Each DGR represents one distortion type, with nodes representing sample features and edges representing relationships between samples.The graph is formulated as G_k = (V_k, E_k).
  • Distortion Graph Representation: Node embeddings are optimized from CNN features, while vector-valued edge embeddings expand pairwise similarity into a 3D adjacency representation.The edge dimension C_E is set smaller than the feature dimension C to reduce computational complexity.

C. Domain Graph Optimization

GraphIQA optimizes distortion graphs through complementary networks that distinguish distortion types and model level distributions. TDN uses graph-level contrastive structure, while FPN predicts content-aware fuzzy levels from Gaussian distributions.

  • Type Discrimination Network: TDN aggregates node and edge information into a compact DGR code and applies triplet loss to contrast same-type and different-type distortions.The triplet objective learns subtle type differences and relationships intended to improve robustness beyond training distortion types.
  • Type Discrimination Network: Triplet loss pulls the anchor toward a same-type DGR and separates it from a different-type DGR using L2 distance and a margin.The positive and negative codes correspond to same-type and different-type distortion graphs, respectively.
  • Fuzzy Prediction Network: FPN predicts distortion levels while accounting for content-induced uncertainty by sampling from a Gaussian prior with predicted mean and scale.The reparameterization trick enables differentiable training of the sampling process.
  • Fuzzy Prediction Network: FPN combines direct node embeddings with edge embeddings averaged across neighboring nodes to estimate distortion levels.The hyper-predictor is trained with mean square error, and the full model minimizes a weighted combination of losses.

D. Finetune and Inference

During finetuning, GraphIQA concatenates node and self-loop edge embeddings to regress IQA scores, using a small two-layer network. Authentic distortion prediction uses a Gaussian prior, and inference supports arbitrary batch sizes.

  • Finetune: GraphIQA concatenates node and self-loop edge embeddings, then feeds them into a two-layer regression module for IQA score prediction.The self-loop edge embedding contains distortion-level priors learned during pre-training.
  • Finetune: For authentic distortions, GraphIQA uses a Gaussian prior to handle unknown distortion types before finetuning on MOS/DMOS with MSE.
  • Inference: GraphIQA supports any input batch size during inference because it can already infer the DGR.

A. Experiments Setting

Experiments pre-train GraphIQA on large synthetic distortion datasets and evaluate it across authentic, synthetic, and multiple-distortion targets. Evaluation uses SRCC and PLCC, with specified augmentation and optimization settings.

  • Architecture: Figure 4 distinguishes GraphIQA architectures for pre-training and finetuning.
  • Datasets: Pre-training uses Kadid-10k or Kadis-700k, each covering 25 distortion types and 5 distortion levels.Kadis-700k contains 700,000 distorted images, while Kadid-10k provides validation for hyper-parameter selection.
  • Datasets: Target evaluation includes two authentic datasets, two synthetic datasets, and one multiple-distortion dataset.The datasets are KonIQ-10k, LIVEC, LIVE, CSIQ, and LIVEMD.
  • Metrics: SRCC measures prediction monotonicity and PLCC measures prediction accuracy; both range from −1 to 1, with higher values indicating better performance.
  • Implementation: GraphIQA is pre-trained with 224×224 random crops, λ=0.25, triplet-loss margin 0.1, Adam, 350000 steps, and batch size 32.

B. DGR Performance Evaluation

The evaluation examines DGR clustering, unseen-distortion interpretability, and leave-one-distortion-out IQA performance. Results indicate that DGRs encode both distortion types and ordered distortion levels while supporting interpretable combinations.

  • Visualization and Clustering Evaluation: DGRs cluster distortion types more distinctly than DBCNN representations in visualization experiments.
  • Visualization and Clustering Evaluation: DGR node embeddings cluster by distortion level and follow a regular ordering pattern; most clustering metrics exceed 60%.The evaluation uses homogeneity, completeness, and V-measure on Kadid-10k.
  • Interpretability for unseen distortions: For unseen combinations, BLUR+JPEG is closest to BLUR and JPEG, while BLUR+WN is closer to noise than BLUR+JPEG is.DBCNN representations are described as not interpretable for the same comparison.
  • Leave-One Evaluation: Leave-one-distortion-out cross-validation tests IQA performance with one distortion type held out and the remaining types used for training.

C. Comparison with the State-of-the-arts

GraphIQA is compared with state-of-the-art BIQA methods across single-database, individual-distortion, cross-dataset, and enhanced-image evaluations. It outperforms all compared methods on synthetic and multiple-distortion datasets and remains competitive elsewhere.

  • Single Database Evaluations: GraphIQA outperforms all compared methods on the LIVE, CSIQ, and LIVEMD datasets.The comparison uses SRCC and PLCC across multiple datasets.
  • Single Database Evaluations: On authentic datasets KonIQ-10k and LIVEC, GraphIQA achieves comparable performance to methods designed for authentic distortions.No authentic or multiple-distortion data are used during pre-training.
  • Individual Distortion Types: Without MOS/DMOS annotations during pre-training, GraphIQA remains comparable on some individual distortion types and improves after target-dataset finetuning.
  • Generalization Evaluation: Cross-dataset tests show GraphIQA achieves comparable performance with DBCNN and HyperIQA on synthetic and authentic dataset pairs.
  • Generalization Evaluation: GraphIQA performs better on the de-hazed dataset and comparably on the de-raining dataset relative to Res50.These enhanced-image datasets were unseen during pre-training.

D. Ablation Study

The ablation study evaluates GraphIQA’s components and embedding dimensions against a ResNet50 baseline on synthetic and authentic IQA datasets. Combining node and edge embeddings with graph-based representation learning provides the strongest reported configuration.

  • Model Components: GraphIQA combines a pre-trained backbone, node embedding, and edge embedding to outperform the ResNet50 baseline in the ablation study.Using node or edge embedding separately is weaker than combining both, especially for edge embedding.
  • Model Components: FPN performs better than a softmax classifier for distortion-level prediction because level estimation is treated as regression rather than classification.
  • Model Components: ImageNet pre-training helps on authentically distorted images but performs worse than expected on synthetically distorted images.
  • Embedding Dimensions: Edge embedding size 16 and node embedding size 256 produce the best reported linear-evaluation performance.The tested edge sizes are 1, 16, 32, 64, and 96; node sizes are 32, 64, 128, 256, and 300.

E. Experiments on architecture and hyper-parameters.

Additional experiments examine architecture, training duration, model complexity, and triplet-loss margins. The results identify settings that improve representation quality while exposing a generalization risk from excessive pre-training.

  • Architecture: Three fully connected layers in the Node Builder achieve the best performance among the tested architectures on KonIQ.
  • Training Duration: Longer pre-training initially improves distortion discrimination and representation on synthetic and authentic datasets, but excessive training causes overfitting to synthetic distortions.This overfitting leads to poor generalization on unknown distortion types.
  • Complexity Analysis: GraphIQA contains 34.9M parameters in total, while TDN and FPN are excluded during fine-tuning and inference, reducing the active amount to 29.9M.
  • Triplet-Loss Margin: A triplet-loss margin of 0.2 yields the best linear-evaluation result among the tested margins.The margin controls how the network discriminates distortion types, while soft-margin loss avoids truncation.

V. CONCLUSION

GraphIQA integrates graph representation learning into IQA to learn distortion graph representations. The paper concludes that these representations capture distortion characteristics and internal structure while motivating more complex graphs and broader applications.

  • GraphIQA learns distortion graph representations that capture each distortion’s characteristics and internal structure.
  • The learned DGRs can provide prior knowledge for known distortions and help infer the perceptual-quality influence of unknown distortions.
  • Future work considers more complex graph structures, interpretable IQA, and applications to image restoration such as denoising and deblurring.
Loading 2103.07666v4…