Source-linked AI summary
Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models
Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, Chao Dong
TL;DR
Traditional IQA methods primarily provide scores, limiting their ability to express distortions and human reasoning. DepictQA introduces a language-based MLLM approach with hierarchical tasks and a multi-modal dataset, achieving strong benchmark performance and more accurate descriptive reasoning than general MLLMs. The authors also report limitations in data coverage, task performance, distortion modeling, voting efficiency, and the unresolved role of MLLM-based IQA relative to score-based methods.
Problem
Score-based IQA cannot adequately express multiple distortions or capture the human reasoning process used to assess image quality.
Method
DepictQA uses an MLLM with hierarchical description, comparison, and reasoning tasks, trained on the M-BAPPS multi-modal dataset with additional multi-source data.
Results
DepictQA achieves state-of-the-art performance on multiple IQA benchmarks and generates more accurate distortion descriptions and comparison reasoning than general-purpose MLLMs.
Takeaways & Limitations
Language-based IQA can depict image distortions and comparison reasoning in human-interpretable form while aligning with human judgments across benchmark evaluations.
Takeaways & Limitations
The authors report insufficient data coverage, unsatisfying description and reasoning performance, incomplete distortion modeling, time-consuming pairwise voting, and an open question about replacing score-based IQA.
Abstract
from arXiv · showhide
We introduce a Depicted image Quality Assessment method (DepictQA), overcoming the constraints of traditional score-based methods. DepictQA allows for detailed, language-based, human-like evaluation of image quality by leveraging Multi-modal Large Language Models (MLLMs). Unlike conventional Image Quality Assessment (IQA) methods relying on scores, DepictQA interprets image content and distortions descriptively and comparatively, aligning closely with humans' reasoning process. To build the DepictQA model, we establish a hierarchical task framework, and collect a multi-modal IQA training dataset. To tackle the challenges of limited training data and multi-image processing, we propose to use multi-source training data and specialized image tags. These designs result in a better performance of DepictQA than score-based approaches on multiple benchmarks. Moreover, compared with general MLLMs, DepictQA can generate more accurate reasoning descriptive languages. We also demonstrate that our full-reference dataset can be extended to non-reference applications. These results showcase the research potential of multi-modal IQA methods. Codes and datasets are available in https://depictqa.github.io.
1 The Chinese University of Hong Kong
The passage identifies the authorship context and the paper’s focus on image quality assessment using multi-modal language models.
- The project page for the work is depictqa.github.io.
- The passage marks some contributors as corresponding authors.
- The paper lists Image Quality Assessment and Multi-modal Language Models as keywords.
1 Introduction
The paper replaces score-only IQA with language-based, human-like descriptions and comparisons, supported by a hierarchical task design, the M-BAPPS dataset, and multi-source training. DepictQA reports stronger alignment with human judgments and more accurate descriptive reasoning than conventional methods and general MLLMs, while its broader applicability remains limited by data and task challenges.
- Existing IQA methods reduce image quality to scores, which cannot express multiple distortions or model human reasoning adequately.
- DepictQA takes images and a question as input, then describes quality, weighs distortion importance, and produces a human-interpretable comparison.
- DepictQA organizes evaluation into Quality Description, Quality Comparison, and Comparison Reasoning, following the human assessment process.
- M-BAPPS contains 5,104 detailed and 115,646 brief descriptions collected from the BAPPS IQA dataset.
- Multi-source training addresses limited high-quality descriptions, while experiments report state-of-the-art benchmark performance and stronger descriptive ability than general-purpose MLLMs.
2 Related Works
Related work covers score-based IQA methods, general multi-modal language models, and emerging MLLM-based approaches for low-level image perception and quality assessment.
- Score-based IQA methods assess quality through full-reference or non-reference scores, using traditional metrics or learned similarity models.
- MLLMs extend large language models with vision and support tasks including captioning, visual question answering, and document understanding.
- Recent MLLM-based IQA work includes benchmarks, instruction datasets, and text-guided quality-score regression, while DepictQA distinguishes itself from these approaches.
3 DepictQA Task and Dataset
DepictQA reformulates IQA as a hierarchy of language-based description, comparison, and reasoning tasks that follows human evaluation. The accompanying M-BAPPS dataset combines diverse distorted-image pairs with structured and textual annotations for training these tasks.
- Dataset Construction: The response pipeline collects quality-related information through questionnaires, converts annotations into natural-language text with GPT-4, and has annotators correct and enrich the outputs.This procedure is used to produce structured, higher-quality descriptive training responses.
- Task Description: DepictQA identifies distortions, compares images, and weighs multiple quality factors through three hierarchical tasks: description, comparison, and reasoning.The hierarchy follows the human process of perceiving distortions, comparing alternatives, and judging their relative impacts.
- Dataset Construction: M-BAPPS uses 197k diverse BAPPS samples, each pairing a reference image with two distorted versions and existing human quality judgments.These samples support dataset construction and validation of newly annotated data.
- Dataset Construction: The dataset construction uses randomized question variants and assisted response collection to improve question robustness and reduce biases in manually written texts.For each task, GPT-4-generated questions are filtered into ten-question sets, and training responses are structured through questionnaires.
- Task Description: Task 3 records five-level comparisons across distortion types and overall distortion, ranging from superior to inferior.The annotated overall-comparison judgments reach 84.3% consistency with BAPPS ground-truth judgments.
- Dataset Statistics: The dataset contains 5,104 detailed high-quality samples and 115,646 brief templated samples across the task-specific splits.Detailed samples cover Tasks 1 and 3, while brief templated samples form the Task-2 training set.
4 DepictQA Framework
DepictQA fuses image and question tokens in a multimodal language model that generates descriptive IQA responses. Its framework addresses multi-image input and limited high-quality data through specialized image tags, multi-source training, and parameter-efficient adaptation.
- Framework: DepictQA encodes images as visual tokens, projects them into textual space, fuses them with question tokens, and processes the sequence with an LLM.The image encoder is frozen, while the image projector is trainable.
- Response Generation: LoRA fine-tunes selected self-attention projection layers instead of fully fine-tuning the language model.This addresses the resource and data demands of complete LLM fine-tuning.
- Multi-image Processing: Specialized reference, Image A, and Image B tags distinguish multiple images more reliably than short textual cues alone.The tags are introduced because existing single-image MLLMs can confuse images in multi-image inputs.
- Training Data: Multi-source training combines 115,646 brief comparison samples, duplicated high-quality description and reasoning data, and unrelated content-description data.The additional sources increase training data and help bridge images with descriptive language.
- Training Scheme: Training uses next-token cross-entropy on response tokens while optimizing only the image projection and LoRA parameters, totaling 0.25% of the 7B model.Question and image tokens do not contribute directly to the loss computation.
5 Experiments
Experiments evaluate DepictQA with task-specific metrics, showing strong comparison performance, improved results over general MLLMs, and benefits from multi-source data, unique tags, larger datasets, and initialization choices.
- Evaluation Metrics: The evaluation uses accuracy for comparison, SRCC and PLCC for quality alignment, GPT-4 scores for descriptions and reasoning, and human reasonable rates as a complementary measure.Human evaluators require identifying the major distortion, avoiding severe comparison mistakes, and maintaining self-consistency.
- Comparison with Score-based IQA Methods: DepictQA surpasses FSIM by approximately 11% and LPIPS and DISTS by more than 2.6% on BAPPS quality comparison, approaching human performance.The comparison uses traditional and deep-learning-based score methods as baselines.
- Comparison with General Multi-modal LLMs: General MLLMs perform poorly on IQA, whereas fine-tuning on M-BAPPS substantially improves DepictQA; single-image evaluation raises LLaVA-1.5 from 18% to 24% but remains sub-optimal.The results suggest that general MLLMs struggle particularly with multi-image inputs, while fine-tuning improves task performance.
- Ablation Studies: Task-2 data improves comparison accuracy, Task-3 data strengthens reasoning, and Task-1 data aids distortion identification but can harm comparison performance.Content-description data counteracts this negative effect and regularizes training while enriching text diversity.
- Ablation Studies: A unique tag reduces image-confusion errors from 24% to 12%, while comparison performance increases gradually with training-data size.The unique-tag design distinguishes images through textual cues; alternative image-embedding and projector methods are also studied.
- Initialization and Extensions: DepictQA remains compatible with different LLMs, benefits slightly from more advanced models and perceptual pre-training, but pre-training nearly triples training time.The model also retains high performance in the non-reference setting after combining non-reference data with the original training data.
6 Conclusions and Limitations
The paper presents DepictQA as a preliminary demonstration that image quality can be described in language, while identifying substantial limitations before real-world application.
- Conclusion: DepictQA demonstrates the possibility of depicting image quality with language, but the authors describe real-world application as still distant.The conclusion frames the work as a preliminary attempt rather than a finished practical system.
- Limitations: Insufficient data amount and coverage limit generalization, while description and reasoning performance remains unsatisfying.These limitations are explicitly identified as future-work priorities.
- Limitations: The method currently assumes up to five predefined distortion types, lacks fine-grained local comparisons, and uses time-consuming pair-wise voting.The authors also leave replacement of score-based IQA methods by MLLM-based methods as an open question.
A1 Overview
The appendix provides dataset, methodology, ablation, quantitative, qualitative, and failure-case details following the main paper.
- Appendix Organization: The appendix describes the dataset first, then methodology details, additional ablations, and further quantitative, qualitative, and failure-case results.These materials are organized into Sections A2 through A5.
A2 Dataset Details
Appendix dataset details document M-BAPPS response statistics, questionnaire-option distributions, question variation, and the process used to produce high-quality responses.
- Response Construction: GPT-4 converts human-annotated questionnaire options into paragraphs, with two human-written examples used to reduce verbosity and misleading responses.The examples align generated responses with a consistent writing style and empirically improve response quality.
- Question Design: M-BAPPS includes varied question formulations for quality description, quality comparison, and comparison reasoning tasks.The examples support that users can express similar questions in different sentences.
- Response Statistics: Response statistics cover word count and string length, while quality-comparison validation statistics are omitted because only binary ground truth is used.The dataset’s word-length distribution is also shown in Fig. A1.
- Questionnaire Statistics: Questionnaire statistics break down texture-related options and distortion-related options for quality description and comparison reasoning.No questionnaire statistic is required for quality comparison because that task has no questionnaire.
- Dataset Vocabulary: The dataset wordcloud highlights terms such as “quality,” “overall distortion,” “color distortion,” and “artifact” as frequent image-quality vocabulary.The constant terms “Image A” and “Image B” are excluded from the wordcloud.
A3 Methodology Details
This section covers the model architecture, metric calculation, and complexity and efficiency analysis of DepictQA.
- The section details DepictQA’s model architecture.
- It explains how the evaluation metrics are calculated.
- It analyzes the model’s complexity and efficiency.
A3.1 Architecture
The appendix reports implementation details, ablations, qualitative evaluations, human-consistency analyses, and failure cases for DepictQA and GPT-4V.
- Architecture: DepictQA uses visual tokens from CLIP’s ViT-L/14 encoder after resizing and patching input images.Images are padded to dimensions divisible by 14 and partitioned into 14 × 14 patches.
- Efficiency: DepictQA has 16M trainable parameters, 0.25% of its 7B total, and trains in around 12 hours on 8 GPUs.
- Ablations: Local feature maps outperform singular global feature tokens, particularly on reasoning tasks.The appendix attributes this difference to the global token’s limited visual representation.
- Ablations: LoRA rank 8 causes divergence, while convergent ranks 12–32 yield performance insensitive to rank; rank 16 is the default.
- Data Ablations: Content description data improves performance and expands recognized texture diversity beyond predefined training categories.The model correctly identifies fine-grained textures such as trees and flowers, usually annotated as plants.
- Results and Limitations: DepictQA aligns with human judgments in qualitative comparisons, but fails on some distortion types, attribution cases, and unseen distortions.The unique-tag approach does not completely resolve confusion, and complete semantic content washout was absent from training data.
- GPT-4V Comparison: GPT-4V produces detailed comparison reasoning but can still misdescribe distortions, texture damage, and comparison outcomes.
- Extensions: DepictQA can adapt comparison results to specified human preferences, although improving this customization remains future work.