PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment

Tianhe Wu, Kede Ma, Jie Liang, Yujiu Yang, Lei Zhang

arXiv:2403.10854v3cs.CV

TL;DR

Prior MLLM-IQA work has focused largely on human quality-description datasets, while challenges remain in annotation and evaluation. This study tests combinations of psychophysical procedures and NLP prompts alongside difficult-sample selection; it finds that prompting needs vary by model and that fine-grained and multiple-image analysis remain difficult.

  • Problem

    Prior MLLM-IQA work primarily benchmarks human quality descriptions, but balanced annotation, subjective inconsistency, and comparison of model outputs with reference descriptions remain challenging.

    Source: §1 · p. 3

  • Method

    The study compares nine psychophysics-by-NLP prompting systems and uses diversity- and uncertainty-aware difficult-sample selection to test MLLMs in full-reference and no-reference IQA.

    Source: §1 · p. 3

  • Results

    Different MLLMs require different optimal prompting systems, and MLLMs including GPT-4V still have room to improve at fine-grained quality discrimination and multiple-image analysis.

    Source: §1 · pp. 3–4, §5 · p. 14

  • Takeaways & Limitations

    Chain-of-thought results suggest integrating IQA into broader, higher-level tasks using physical, geometrical, and semantic information.

    Source: §1 · p. 4

    Prompts are not optimized, and the sampler depends on human quality scores, restricting it to existing human-rated IQA datasets.

    Source: §4.4 · p. 14

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) have experienced significant advancement in visual understanding and reasoning, their potential to serve as powerful, flexible, interpretable, and text-driven models for Image Quality Assessment (IQA) remains largely unexplored. In this paper, we conduct a comprehensive and systematic study of prompting MLLMs for IQA. We first investigate nine prompting systems for MLLMs as the combinations of three standardized testing procedures in psychophysics (i.e., the single-stimulus, double-stimulus, and multiple-stimulus methods) and three popular prompting strategies in natural language processing (i.e., the standard, in-context, and chain-of-thought prompting). We then present a difficult sample selection procedure, taking into account sample diversity and uncertainty, to further challenge MLLMs equipped with the respective optimal prompting systems. We assess three open-source and one closed-source MLLMs on several visual attributes of image quality (e.g., structural and textural distortions, geometric transformations, and color differences) in both full-reference and no-reference scenarios. Experimental results show that only the closed-source GPT-4V provides a reasonable account for human perception of image quality, but is weak at discriminating fine-grained quality variations (e.g., color differences) and at comparing visual quality of multiple images, tasks humans can perform effortlessly.

1 Introduction

The paper systematically tests prompting MLLMs for IQA and considers their potential as flexible, interpretable, text-driven assessors. It highlights challenges in human-description benchmarks and reports that fine-grained and multiple-image quality analysis remain difficult.

1 Introduction

  • IQA seeks computational predictions of image quality as perceived by the Human Visual System, and MLLMs' visual understanding and reasoning motivate their study for this task.The paper frames potential benefits in relation to MLLMs' visual encoders and multimodal capabilities.Source: §1 · p. 2
  • MLLMs are presented as potentially useful for IQA through improved accuracy, robustness, text-conditioned flexibility, and descriptive outputs.The text identifies cross-task visual information, mild geometric transformations, texture resampling, and detailed quality descriptions as examples.Source: §1 · pp. 2–3
  • Human-description benchmarks [54] [66] raise challenges in obtaining balanced annotations, handling subjective inconsistency, and comparing model outputs with reference descriptions.The paper notes that established human-rated IQA datasets are available for assessing this perceptual task.Source: §1 · p. 3
  • The study combines three psychophysical testing procedures with three NLP prompting strategies, then selects difficult samples using diversity and uncertainty.It evaluates three open-source and one closed-source MLLMs in full-reference and no-reference settings across structural and textural distortions, geometric transformations, and color differences.Source: §1 · p. 3
  • Different MLLMs require different optimal prompting systems, and MLLMs including GPT-4V still have room to improve at fine-grained quality discrimination and multiple-image analysis.Chain-of-thought results suggest integrating IQA into broader, higher-level tasks using physical, geometrical, and semantic information.Source: §1 · pp. 3–4

2 Related Work

Related work reviews expert IQA models and prior efforts to benchmark or fine-tune MLLMs for IQA.

2.1 Expert Models for IQA

  • Full-reference IQA models often require aligned reference and test images, while their treatment of color can be superficial.The paper notes potential difficulty with mild geometric transformations and texture resampling, as well as perceptual color differences.Source: §2.1 · p. 4
  • No-reference IQA models rely heavily on human-annotated training data, which can weaken generalization to wider image and distortion ranges.This dependence applies to both knowledge-driven and data-driven approaches.Source: §2.1 · p. 4
  • Expert IQA models' numerical scores limit their flexibility and interpretability by omitting other input formats and detailed perceptually relevant outputs.Source: §2.1 · p. 4

2.2 MLLMs for IQA

  • DepictQA [66] uses a double-stimulus comparison, whereas Q-Bench uses a single-stimulus quality classification; prior work reports limited human-perception replication by open-source MLLMs.The passage contrasts these results with proprietary GPT-4V [59].Source: §2.2 · p. 4
  • Instruction-tuned open-source MLLMs can revert to template-like quality descriptions, motivating a systematic exploration of prompting on human-rated datasets.The paper presents this exploration as a step back from benchmarking and fine-tuning.Source: §2.2 · p. 5

3 Prompting MLLMs for IQA

The section combines psychophysical testing procedures with NLP prompting strategies to create nine systems for MLLM-based IQA, then selects informative difficult samples to evaluate them more efficiently.

3.1 Prompting Strategies from Psychophysics

  • The single-stimulus method requests a score for one test image, whereas the double-stimulus method compares image pairs and aggregates preferences into rankings.In FR, the reference image accompanies the test image or pair; NR scoring uses no reference, and NR pairs may contain different image content.Source: §3.1 · p. 5, Figure 2 · p. 6
  • The multiple-stimulus method presents a list of L images for partial rankings, which are converted into pairwise preferences and aggregated.In FR, the list shares visual content and is accompanied by a reference; these restrictions do not apply in NR.Source: §3.1 · p. 6, Figure 2 · p. 6

3.2 Prompting Strategies from NLP

  • Standard prompting directly requests scores or comparisons, chain-of-thought elicits attribute-by-attribute reasoning, and in-context prompting supplies human-scored exemplars.The figure illustrates these three NLP strategies for GPT-4V in the NR scenario.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
  • Crossing three psychophysical procedures with three NLP strategies yields nine candidate prompting systems, and different MLLMs may perform best with different systems.The paper cites variation in training data, architectures, optimization pipelines, and alignment strategies as context for this expectation.Source: §3.1 · p. 5, §3.2 · pp. 6, 8

3.3 Computational Procedure for Difficult Sample Selection

  • The sample-selection procedure targets difficult cases, sample diversity, and consistency with the majority of human judgments while limiting costly MLLM evaluation.It uses expert IQA models as proxies to identify cases where predictions diverge from human scores, because gMAD is not applicable under the affordable-inference assumption.Source: §3.3 · p. 8
  • In the FR procedure, reference-image selection combines expert-model error normalized by human-score variance with a diversity measure; λ trades off these terms.Human quality scores and their standard deviations supply the normalization, and ε prevents division by zero.Source: §3.3 · p. 8
  • For NR-IQA, the procedure modifies the selection objective, while the FR procedure also selects difficult distorted images for each chosen reference.The supplied equations show variance-normalized expert-model error and a diversity term in the FR and NR objectives.Source: §3.3 · p. 8
  • Variance-normalized sampling shifts the empirical distribution of sample standard deviations toward zero, suggesting more consistent human quality judgments.The comparison holds sample diversity constant.Source: §3.3 · p. 9, Figure 4 · p. 9

4 Experiments

The experiments evaluate nine prompting systems across image-quality attributes and full-reference and no-reference settings, then test optimally prompted models on difficult samples and discuss study limitations.

4.1 Experimental Setups

  • The experiments cover four full-reference datasets and three no-reference datasets spanning structural and textural distortions, geometric transformations, texture similarity, color differences, and artifacts.The difficult-sample procedure selects 15 reference images and 10 distorted images for FR, and 150 images for NR; its diversity measure uses CLIP-visual-encoder feature-space MSE.Source: §4.1 · p. 9

4.2 Comparison of Nine Prompting Systems

  • On uniformly sampled images, the three open-source MLLMs perform unsatisfactorily in FR; LLaVA-v1.6 and mPLUG-Owl2 can disregard the second image despite explicit prompting.In NR single-stimulus standard prompting, those two models improve, while InternLM-XComposer2-VL and GPT-4V handle realistic camera distortions on SPAQ remarkably well.Source: §4.2 · pp. 10–11, Table 1 · p. 10, Figure 5 · p. 11
  • Single-stimulus prompting suits the three open-source MLLMs, whereas GPT-4V performs optimally with double-stimulus chain-of-thought prompting.The authors note that recent claims of open-source models surpassing GPT-4V for IQA did not pair GPT-4V with its optimal prompting system.Source: §4.2 · p. 11
  • In-context prompting offers GPT-4V little benefit and performs poorly with multiple-stimulus prompting, while chain-of-thought consistently improves it across protocols and nearly all visual attributes.Adding a contextual image’s human quality score has minimal impact; processing many images, such as eight, causes a sharp performance drop.Source: §4.2 · pp. 11–12, Figure 6 · p. 12

4.3 Further Testing on Difficult Data

  • On difficult samples, GPT-4V matches FSIM on FR-KADID, trails DISTS on Aug-KADID and TQD, struggles with color differences, and surpasses MUSIQ and LIQE on SPAQ.The comparisons cover full-reference and no-reference IQA; Tables 2 and 3 report those respective scenarios.Source: Table 2 · p. 13, Table 3 · p. 13, §4.3 · p. 13
  • GPT-4V generalizes remarkably to AI-generated images, while Q-Instruct improves IQA skills over LLaVA-v1.6.The authors identify potential use in guiding image-generation model optimization and attribute Q-Instruct’s improvement to visual instruction tuning.Source: §4.3 · pp. 13–14

4.4 Discussion and Limitation

  • The study identifies unoptimized prompts, reliance on human quality scores for sample selection, and lack of quantitative evaluation of generated text as limitations.It also leaves instruction tuning unaddressed; the authors suggest active continual learning or parameter-efficient tuning to balance IQA specificity and generality.Source: §4.4 · p. 14

5 Conclusion

The study combines psychophysics and NLP prompting for MLLM-based IQA, finding that different MLLMs perform best with different prompting systems. It also introduces difficult-sample selection and concludes that fine-grained discrimination and multiple-image quality analysis still have room for improvement.

5 Conclusion

  • Different MLLMs perform best with different prompting systems, prompting a re-evaluation of recent MLLM progress in IQA.Source: §5 · p. 14
  • MLLMs, including GPT-4V, still have room to improve in fine-grained quality discrimination and multiple-image quality analysis.Source: §5 · p. 14
  • The proposed difficult-sample sampler uses inference-efficient expert IQA models and balances sample diversity with uncertainty, without access to MLLM internals or input-output behavior.The paper frames the sampler as a form of black-box attack on MLLMs.Source: §5 · p. 14

A More Experimental Setups

This section provides more details regarding experimental setups.

A More Experimental Setups

A.1 Uniform Sample Selection

The uniform sample selection procedure differs by full-reference and no-reference evaluation scenario.

A.1 Uniform Sample Selection

  • For full-reference tests, 15 reference images are sampled, then 10 distorted images per reference; TQD is evaluated in full because it contains 150 images, while no-reference tests sample 150 distorted images per test dataset.The sampled distorted images cover the full quality spectrum in the full-reference scenario.Source: §A.1 · p. 20

A.2 Metrics

The evaluation uses SRCC and PLCC, with a logistic fit before PLCC and scenario-specific aggregation across images.

A.2 Metrics

  • PLCC is computed after fitting a four-parameter logistic function to compensate for prediction nonlinearity.Source: §A.2 · p. 20
  • For FR-IQA, correlations are computed across 10 distorted images per content and averaged over 15 references, except for SPCD; NR-IQA and SPCD use 150 sampled images.Source: §A.2 · p. 20

B Complete Input Prompts for MLLMs

The prompts operationalize full-reference and no-reference quality scoring, pairwise comparison, and ranking, with variants that add human-scored examples or request written quality analyses.

B.1 FR Scenario

  • FR single-stimulus prompts assign a 0–100 score to a distorted image against its reference; variants provide a human-scored example or request a written comparison.The comparison covers structure and texture, color and luminance, noise, contrast, sharpness, and other low-level distortions.Source: §B.1 · pp. 20–21
  • FR double-stimulus prompts compare two distorted images against one reference, coding the closer match as 1, the other as 0, and a tie as 2.In-context prompts supply a human comparison example, while chain-of-thought prompts request descriptions of both image-reference differences before the comparison.Source: §B.1 · pp. 21–23
  • FR multiple-stimulus prompts rank four distorted images against a reference from 0 for lowest quality to 3 for highest, allowing ties.The in-context variant provides a human ranking example, and the chain-of-thought variant requests descriptions of each image-reference difference before ranking.Source: §B.1 · pp. 23–24

B.2 NR Scenario

  • NR single-stimulus prompts score one image from 0 to 100; variants provide a human-scored example or request a written quality analysis before scoring.The prompts assess structure and texture, color and luminance, noise, contrast, sharpness, and other low-level distortions.Source: §B.2 · pp. 24–25
  • NR double-stimulus prompts compare two images, coding the better-quality image as 1, the other as 0, and equal quality as 2.In-context prompts give a human comparison example; chain-of-thought prompts request a description of the pair before the comparison.Source: §B.2 · pp. 25–26
  • NR multiple-stimulus prompts rank four images from 0 for lowest quality to 3 for highest, with in-context examples or written comparison before ranking.The standard prompt requests only four ranking scores; the chain-of-thought variant also requests a concise description of the ranking.Source: §B.2 · pp. 26–27
Loading 2403.10854v3…