PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning

Kwan-Yee Lin, Guanxiang Wang

arXiv:1804.01681v1cs.CV

TL;DR

NR-IQA must predict distorted-image quality when the corresponding reference is usually absent, making comparison-based assessment ill-posed. The paper hallucinates a reference constrained by the distorted image, feeds their discrepancy to a quality regressor, and uses adversarial and ranking guidance. Evaluated on four benchmarks, the method reportedly outperforms previous state-of-the-art NR-IQA methods by significant margins.

  • Problem

    NR-IQA lacks the undistorted reference normally used to quantify perceptual discrepancy, making quality prediction ill-posed.

  • Method

    The method generates a quality-aware hallucinated reference, computes discrepancy with the distorted image, and guides regression through adversarial and implicit-ranking mechanisms.

  • Results

    The method was evaluated on LIVE, CSIQ, TID2008, and TID2013 and significantly outperformed previous state-of-the-art NR-IQA methods.

  • Takeaways & Limitations

    Hallucinated-reference discrepancy provides compensatory information for NR-IQA without requiring extra annotations or artificial prior knowledge.

  • Takeaways & Limitations

    The approach assumes a pre-trained quality regression model trained from scratch with an IQA dataset.

Abstract

from arXiv · show

No-reference image quality assessment (NR-IQA) is a fundamental yet challenging task in low-level computer vision community. The difficulty is particularly pronounced for the limited information, for which the corresponding reference for comparison is typically absent. Although various feature extraction mechanisms have been leveraged from natural scene statistics to deep neural networks in previous methods, the performance bottleneck still exists. In this work, we propose a hallucination-guided quality regression network to address the issue. We firstly generate a hallucinated reference constrained on the distorted image, to compensate the absence of the true reference. Then, we pair the information of hallucinated reference with the distorted image, and forward them to the regressor to learn the perceptual discrepancy with the guidance of an implicit ranking relationship within the generator, and therefore produce the precise quality prediction. To demonstrate the effectiveness of our approach, comprehensive experiments are evaluated on four popular image quality assessment benchmarks. Our method significantly outperforms all the previous state-of-the-art methods by large margins. The code and model will be publicly available on the project page https://kwanyeelin.github.io/projects/HIQA/HIQA.html.

1. Introduction

NR-IQA predicts distorted-image quality without a corresponding undistorted reference, making the task ill-posed. The proposed method hallucinates a reference and uses perceptual discrepancy to guide quality regression, achieving strong benchmark performance.

  • Motivation: NR-IQA predicts perceptual quality from distorted images, but typically lacks the reference needed for direct comparison.This missing reference makes quality prediction highly challenging and contributes to a performance bottleneck.
  • Approach: The Hallucination-Guided Quality Regression Network generates a high-resolution hallucinated scene and uses its discrepancy map with the distorted image for regression.The discrepancy map captures pixel-wise differences that guide quality prediction.
  • Motivation: Image distortions arise across the lifecycle and include noise, compression artifacts, transmission errors, and exposure problems.
  • Approach: A Quality-Aware Generative Network combines texture-feature and quality-feature similarity to produce qualified hallucinated references.A quality-aware perceptual loss complements texture similarity with quality similarity.
  • Approach: An IQA-Discriminator and implicit ranking relationship guide generator learning and suppress negative effects from poor hallucinations.These mechanisms connect hallucination quality to robust regression behavior.
  • Evaluation: The method was evaluated on LIVE, CSIQ, TID2008, and TID2013, outperforming prior state-of-the-art NR-IQA methods by significant margins.Ablation studies also examine the effectiveness of individual components.

2. Related Work

Prior NR-IQA work advances feature representation through classic methods and deep neural networks. This paper instead compensates for absent reference information without extra annotation or prior knowledge, using quality-tailored adversarial learning.

  • NR-IQA: Recent NR-IQA methods increasingly use deep neural networks to improve quality-feature representation beyond classic approaches.
  • NR-IQA: The proposed approach compensates for absent reference information without extra data annotation or prior knowledge.The authors position this as increasing flexibility and feasibility relative to other methods.
  • Generative Adversarial Network: The adversarial learning scheme introduces an IQA-discriminative network because the quality regressor, rather than image realism alone, is the ultimate target.The discriminator is tailored to image-quality assessment.

3. Our Approach

The framework generates hallucinated references from distorted images, then uses them with discrepancy information and adversarial refinement to support quality regression. Its generator combines pixel-wise and perceptual constraints, while feature representations provide structure, texture, and distortion information.

  • Framework overview: The framework comprises a generative network G, discriminative network D, and regression network R that jointly produce references and quality scores.G generates hallucinated references, D refines them adversarially, and R uses distorted images with discrepancy maps and implicit ranking guidance.
  • Framework overview: The discrepancy map represents pixel-wise differences between distorted images and hallucinated references, while the IQA-Discriminator suppresses harmful hallucinations.The map is distinct from the FR-IQA error map, which compares distorted and true reference images.
  • Quality-Aware Generative Network: G generates a high-resolution hallucinated reference conditioned on the distorted image to compensate for the absent true reference.The generator is based on a stacked hourglass architecture.
  • Quality-Aware Generative Network: Pixel-wise loss promotes holistic content, whereas feature-space perceptual loss produces sharper local results.The perceptual constraint uses feature transformations, with intermediate features supplying structure and texture information.
  • Quality-Aware Generative Network: The perceptual constraint can overlook hard cases because semantic features are invariant to input noise, motivating distortion-sensitive features from a pretrained quality regression network.The regression network's intermediate representations capture distortion information for quality similarity measurement.

3.2. IQA-Discriminative Network

The IQA-Discriminator adapts adversarial learning to the regression objective by distinguishing hallucinations according to their influence on predicted quality. This design stabilizes optimization and directs the generator toward hallucinations beneficial to regression.

  • 3.2. IQA-Discriminative Network: Adversarial learning is introduced to generate realistic high-frequency details when distortions remove structure or texture information.The target cases include local block-wise distortions, transmission errors, and severe distortion levels.
  • 3.2. IQA-Discriminative Network: Unlike conventional GAN discrimination, D labels hallucinations by whether they positively or negatively influence regression network R.A hallucination that improves R's precision is treated as real; otherwise it is treated as fake.
  • 3.2. IQA-Discriminative Network: The influence label is defined from the quality regression loss relative to the ground-truth quality score and threshold ε.The formulation uses the loss as an explicit index of G's impact on R.
  • 3.2. IQA-Discriminative Network: This influence-based discrimination penalizes only negatively influential samples and acts as a relaxation strategy for stabilizing adversarial learning.G is optimized to generate hallucinated scenes that benefit R.
  • 3.2. IQA-Discriminative Network: The generator's overall objective combines three loss components weighted by μ1, μ2, and μ3.The cited passage identifies the trade-off parameters but does not specify their numerical values.

3.3. Hallucination-Guided Quality Regression Network

The hallucination-guided regressor pairs distorted images with discrepancy maps and fuses high-level generator features to exploit perceptual differences. Generator and regressor training are mutually reinforcing, while fusion helps stabilize quality prediction when hallucinations vary in eligibility.

  • Discrepancy Map: The regression network receives distorted images and discrepancy maps, using the map as prior information about distortion appearance.The discrepancy map is defined as the absolute difference between the distorted image and generated hallucinated reference.
  • Discrepancy Map: G supplies auxiliary hallucinated references during R training, while R helps G generate better hallucinations during G training.The two networks are mutually correlated and can reinforce each other.
  • High-level Semantic Fusion: Qualified hallucinations help R explore correct perceptual discrepancies, whereas unqualified hallucinations can introduce bias by narrowing distortion information improperly.This motivates a constrained scheme for stabilizing the quality regression process.
  • High-level Semantic Fusion: High-level semantic fusion combines generator encoder features from the second stack with features extracted by R before its fully connected layers.A linear projection aligns dimensions, and concatenation forms the fusion term.
  • High-level Semantic Fusion: The regression loss lets generator semantics participate in R optimization and uses implicit ranking relationships in G to guide adaptive quality prediction.If G is optimal, connections from the projected generator features toward R's fully connected layers may approach identity mappings.

3.4. Training Strategy

The generator and regression sub-networks are differentiable and can be trained end-to-end, but the method uses alternating optimization to mutually reinforce generation and quality regression.

  • Both G and R are differentiable, enabling end-to-end training.
  • An alternative training strategy is used to better optimize generation and quality regression mutually.The complete procedure is specified in supplementary Algorithm 1.

3.5. Weakly-Supervised Quality Assessment

The weakly supervised extension addresses cross-dataset quality assessment by using hallucinated scenes as a shared reference medium and separating generator training from regression training.

  • 3.5. Weakly-Supervised Quality Assessment: Dataset gaps in quality definitions, distortion types, and levels can cause overfitting and poor generalization for models trained on one dataset.Existing nonlinear score mappings across dataset distributions may introduce bias.
  • 3.5. Weakly-Supervised Quality Assessment: The generator encodes differences between differently distorted images as an implicit ranking relationship.
  • 3.5. Weakly-Supervised Quality Assessment: Hallucinated scenes provide a universal medium across datasets because they are constrained on distorted images and serve as reference information for image quality.
  • 3.5. Weakly-Supervised Quality Assessment: The framework’s detachable training process offers an alternative for training the regression component across datasets.

3.6. Implementation Details

Experiments use four benchmark databases, patch-based training and testing, and SROCC and LCC as evaluation criteria.

  • Training setup: Training uses randomly sampled 256 ×256 pixel patches with rotation and flip augmentation.
  • Testing setup: Whole-image quality scores are computed by averaging predictions from overlapped testing patches.
  • Datasets: Experiments cover LIVE, CSIQ, TID2008, and TID2013 benchmark datasets.
  • Evaluation Metrics: SROCC measures monotonic agreement, while LCC measures linear correlation between ground-truth and predicted quality.

4.1. Comparisons with the state-of-the-arts

Across within-dataset and cross-dataset evaluations, the proposed method outperforms state-of-the-art NR-IQA methods and shows generalization across databases.

  • Single dataset evaluations: 13% relative improvement over RankIQA is achieved on the entire TID2013 dataset across all distortion types.
  • Single dataset evaluations: On LIVE, the method improves over BIECON by 2%, over PQR by 2% SROCC and 1% LCC, and over RankIQA by 0.1%.
  • Single dataset evaluations: The method achieves the highest reported performances on both TID2008 and CSIQ.Detailed results are provided in the supplementary material.
  • Oracle analysis: Using ground-truth references as an oracle outperforms all methods across all datasets, indicating potential gains from improved hallucinated information.
  • Cross-dataset evaluations: Training on LIVE and testing on TID2008 produces promising cross-dataset results, demonstrating the approach’s generalization ability.The test uses JPEG, JPEG2K, WN, and BLUR subsets with logistic regression matching DMOS to MOS.
  • Cross-dataset evaluations: Aggregating more IQA datasets for generator training yields higher SROCC and LCC on LIVE, approaching the oracle.Generator training is compared across LIVE, TID2008, and TID2008 plus TID2013.

4.2. Ablation study

Ablations on TID2008 show that hallucinated-reference compensation, quality-aware perceptual loss, adversarial learning, and multi-level semantic fusion each improve quality prediction over the distorted-image baseline.

  • Ablation setup: The baseline uses a modified Res-18 network with only distorted images, and components are compared using SROCC and LCC.The ablation study evaluates the proposed components on the entire TID2008 dataset.
  • Hallucinated reference compensation: 0.859 SROCC and 0.870 PLCC result from adding hallucinated-reference compensation, improving the baseline by 14% and 8%, respectively.The BL+HCM model pairs hallucinated references with distorted images as regression inputs.
  • Quality-aware perceptual loss: Quality-aware perceptual loss adds a further 0.5% improvement on SROCC and 2% on LCC.The loss is applied during training of the hallucination model through feature matching with respect to quality similarity.
  • Adversarial learning: Original adversarial learning improves SROCC and LCC by 3% each, while the proposed adversarial method adds a further 2% SROCC and about 1% LCC improvement.The comparison is between BL+HCM+QPL+ADV and BL+HCM+QPL+QADV.
  • Multi-level semantic fusion: Multi-level semantic fusion produces the highest reported values, reaching 0.941 SROCC and 0.949 LCC.The mechanism fuses generator feature maps from stack two with same-sized feature maps in the quality regression network.

5. Conclusion

The paper addresses the ill-posed nature of NR-IQA with a hallucination-guided quality regression network that captures discrepancy between distorted and hallucinated images. Its quality-aware generation and IQA-discriminator operate adversarially in an end-to-end model without extra annotations or artificial prior knowledge.

  • 5. Conclusion: The proposed network captures perceptual discrepancy between distorted and hallucinated images to predict perceptual quality.It is presented as a new perspective for addressing the ill-posed nature of NR-IQA.
  • 5. Conclusion: Quality-aware generation and a specially designed IQA-discriminator produce hallucinations through adversarial learning.The network can be trained end-to-end without extra annotations or artificial prior knowledge.
Loading 1804.01681v1…