PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution

Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, Chao Dong

arXiv:2501.11561v3cs.CV

TL;DR

MLLMs show promise for linguistic image-quality description but remain challenged in accurate numerical scoring because discrete token outputs do not match continuous quality-score distributions. DeQA-Score discretizes score distributions into soft labels and adds Thurstone-model fidelity loss for dataset variation, outperforming baselines across benchmarks while predicting distributions aligned with human annotations.

  • Problem

    Accurate MLLM-based image-quality scoring is limited by the mismatch between continuous quality-score distributions and discrete token outputs, while one-hot labels lose distributional information.

  • Method

    DeQA-Score uses distribution-based soft labels and Thurstone-model fidelity loss to preserve score relationships and support co-training across varied IQA datasets.

  • Results

    DeQA-Score stably outperforms baselines in score regression across multiple benchmarks and predicts score distributions closely aligned with human annotations.

  • Takeaways & Limitations

    Quality-score distributions provide a basis for accurate MLLM score regression while retaining inter-image relationships and supporting multi-dataset training.

  • Takeaways & Limitations

    The score-regression setting normalizes mean scores to [1] [5] and normalizes variances accordingly, with dataset distortion differences still affecting performance.

Abstract

from arXiv · show

With the rapid advancement of Multi-modal Large Language Models (MLLMs), MLLM-based Image Quality Assessment (IQA) methods have shown promising performance in linguistic quality description. However, current methods still fall short in accurately scoring image quality. In this work, we aim to leverage MLLMs to regress accurate quality scores. A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs. This mismatch necessitates score discretization. Previous approaches discretize the mean score into a one-hot label, resulting in information loss and failing to capture inter-image relationships. We propose a distribution-based approach that discretizes the score distribution into a soft label. This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships. Moreover, to address dataset variation, where different IQA datasets exhibit various distributions, we introduce a fidelity loss based on Thurstone's model. This loss captures intra-dataset relationships, facilitating co-training across multiple IQA datasets. With these designs, we develop the distribution-based Depicted image Quality Assessment model for Score regression (DeQA-Score). Experiments across multiple benchmarks show that DeQA-Score stably outperforms baselines in score regression. Also, DeQA-Score can predict the score distribution that closely aligns with human annotations. Codes and model weights have been released in https://depictqa.github.io/deqa-score/.

1. Introduction

MLLMs describe image quality effectively but remain less accurate at numerical scoring because discrete outputs mismatch continuous quality-score distributions. DeQA-Score addresses this mismatch with distribution-based soft labels and fidelity loss, improving score regression across datasets.

  • Motivation: MLLM-based IQA methods describe image quality well but score it less accurately than traditional IQA methods.This limits their use in real-world IQA scenarios.
  • Challenges: Quality scores are continuous distributions, whereas MLLMs generate discrete tokens; one-hot mean-score discretization loses information and distorts inter-image relationships.It can make lower-quality Image A and Image B share labels while separating similarly rated Images B and C.
  • Method: Distribution-based soft labels discretize the full score distribution, preserving inter-image relationships while enabling recovery of continuous score distributions at inference.The approach approximates scores with Gaussian distributions and uses discrete level distributions as training labels.
  • Challenges: Different IQA datasets have distinct score distributions, making straightforward multi-dataset co-training difficult.Images with nearly identical linearly rescaled scores can still have drastically different perceptual quality.
  • Method: Fidelity loss based on Thurstone's model captures intra-dataset relationships and adapts multi-dataset co-training to MLLM-based quality scoring.It becomes applicable because the soft-label model can predict accurate score distributions.
  • Results: 1.3% PLCC improvement on KonIQ and 4.6% on LIVE-WILD show DeQA-Score outperforming baselines on in-distribution and out-of-distribution evaluation.The model also predicts score distributions closely aligned with extensive human annotations.

2. Related Works

IQA methods assess image quality through quality scores, using reference images in full-reference methods or direct regression in non-reference methods. Recent MLLM-based approaches extend IQA toward language-based and broader multimodal quality assessment.

  • IQA Taxonomy: IQA methods are categorized as full-reference or non-reference according to whether a high-quality reference image is available.Both categories mainly rely on quality scores to assess image quality.
  • Full-Reference IQA: Full-reference methods compute similarity between a distorted image and a high-quality reference using hand-designed or learned metrics.Classical examples include structural similarity, phase congruency with gradient magnitude, and image information.
  • Non-Reference IQA: Non-reference methods regress quality directly without a reference, evolving from handcrafted natural-image statistics to deep models trained on human-annotated datasets.These methods learn quality priors from annotated IQA data.
  • MLLM-Based IQA: MLLMs support multimodal tasks such as visual question answering, document understanding, and image captioning, motivating their use for low-level perception and IQA.MLLM-based IQA methods leverage foundational knowledge for improved performance or more detailed assessment results.

3. Soft Label Construction

The paper replaces one-hot score discretization with a soft label derived from the full Gaussian quality-score distribution. Post-adjustment preserves the original expectation and enables recovery of a distribution closely aligned with human annotations.

  • One-hot label limitations: One-hot labels discretize only the MOS into five intervals, treating “bad”, “poor”, “fair”, “good”, and “excellent” as independent levels.This loses relationships between neighboring levels, ignores score variance, and can assign similar images to different levels near interval borders.
  • Distribution-based discretization: The soft label discretizes the Gaussian score distribution across five central points c_i ∈ {1, 2, 3, 4, 5}, with each level region having width d = 1.Each discrete-level probability is obtained from the Gaussian probability mass within its corresponding interval, using the same textual tokens as Q-Align.
  • Post-adjustment: Truncation outside the represented score intervals otherwise prevents the rough probabilities from forming a valid distribution and shifts their expectation away from the original mean.The omitted regions are x < 0.5 and x > 5.5, which create discretization errors that post-adjustment corrects.
  • Post-adjustment: Post-adjustment applies a linear transformation so the discrete probabilities sum to 1 and their recovered expectation equals the original mean score.The transformation parameters α and β are determined by these two constraints, and their average values are close to 1 and 0, respectively.
  • Discretization precision: The adjusted soft label reduces MOS discretization error by approximately 10× to 35× versus Q-Align’s one-hot label while maintaining PLCC and SRCC of 1.000.Its recovered distribution is also closely aligned with the human-annotated score distribution, unlike the one-hot label.

4. DeQA-Score Model

DeQA-Score combines a vision-language architecture with distribution-based score prediction and fidelity loss for multi-dataset training. It estimates score-distribution statistics and uses them to preserve within-dataset image-quality relationships.

  • Model architecture: The model encodes images into visual tokens, projects them into textual space, reduces them to 64 tokens, and feeds them with text into an LLM.A vision encoder, vision abstractor, and LLM form the adopted MLLM architecture.
  • Training with soft labels: For the <level> token, KL divergence trains predicted probabilities against the soft label, while other response tokens use cross-entropy.The soft-label loss applies to the quality-level output, whereas ordinary response generation retains next-token prediction loss.
  • Score estimation during inference: During inference, predicted probabilities over the five levels yield the expectation and variance of each image’s quality-score distribution.The method can use either a close-set softmax over the five levels or, with nearly identical results, a softmax over all textual tokens.
  • Joint training on multiple IQA datasets: Fidelity loss compares two images sampled from the same dataset using their predicted probability that Image A exceeds Image B.The comparison is based on predicted expectations and variances and is matched to the ground-truth probability.
  • Joint training on multiple IQA datasets: The final objective combines fidelity loss with cross-entropy and KL-divergence losses through a weighting term γ.This design preserves intra-dataset ranking relationships while jointly optimizing response prediction and soft-label fitting.

5. Experiments

Experiments evaluate DeQA-Score on diverse IQA datasets for score regression, score distribution prediction, ablations, and throughput. DeQA-Score generally improves over Q-Align and benefits from distribution-based soft labels, fidelity loss, and multi-dataset co-training.

  • Score regression results of single-dataset training: DeQA-Score outperforms baseline methods across nearly all single-dataset score regression benchmarks.Models are trained on KonIQ and evaluated on in-distribution and out-of-distribution datasets using PLCC and SRCC.
  • Score regression results of single-dataset training: 1.3% improvement in PLCC over Q-Align on KonIQ and 4.6% improvement in PLCC on LIVE-Wild demonstrate gains in both in-distribution and out-of-distribution evaluation.
  • Score regression results of multi-dataset co-training: Multi-dataset co-training consistently improves DeQA-Score over Q-Align across different training combinations.Adding KADID improves performance on TID2013 and CSIQ, while adding PIPAL greatly improves PIPAL results but reduces performance on some other datasets.
  • Score distribution prediction: DeQA-Score predicts score distributions more closely aligned with human annotations than Q-Align.The evaluation uses JS divergence and Wasserstein distance; qualitative results also report smaller KL divergence.
  • Ablation studies: Soft labels outperform one-hot labels across benchmarks, and fidelity loss further improves soft-label performance.Adding uncertainty without modeling the score distribution does not yield stable or significant improvements, while one-hot labels do not combine well with fidelity loss.
  • Ablation studies: Common level texts achieve the best performance, while the model remains adaptable to alternative level-text relationships.The study evaluates random, reversed, numeric, irrelevant, and mixed level texts.
  • Inference latency and throughput: 32.29 images per second are processed on a single RTX A6000 GPU at batch size 256.The reported throughput is presented as suitable for real-time deployment.

6. Conclusions

DeQA-Score is presented as a distribution-based MLLM image quality scorer for both score regression and score distribution prediction. Its design emphasizes the importance of modeling quality score distributions in MLLM-based IQA.

  • DeQA-Score combines distribution-based soft labels and fidelity loss for MLLM-based image quality scoring.
  • The model supports both score regression and score distribution prediction.
  • The conclusion highlights quality score distribution as important for MLLM-based IQA methods.

B.1. Details of Dataset Construction

The dataset-construction details address PIPAL’s missing variance information and the numerical behavior of soft-label construction for very small score variance. PIPAL receives a pseudo variance, while small-variance cases use linear interpolation between neighboring levels.

  • Special designs for PIPAL dataset: PIPAL lacks variance information because it uses pair-wise comparisons and Elo ratings instead of conventional five-point ratings.
  • Special designs for PIPAL dataset: A pseudo variance for PIPAL is assigned using statistics derived from other training datasets.
  • Degradation to linear interpolation when the variance is quite small: When score variance is extremely small, the Gaussian score distribution is approximated by a unit impulse function.This avoids unreasonable adjusted probabilities caused by direct solutions to the post-adjustment constraints.
  • Degradation to linear interpolation when the variance is quite small: The resulting soft label uses linear interpolation between the two nearest center points.If cj < µ ≤ cj+1, the neighboring levels receive the interpolated weights and other levels receive zero.
  • Degradation to linear interpolation when the variance is quite small: The small-variance criterion is 3σ ≤ d/2, simplifying to σ ≤ 0.17 when d = 1; the relaxed normalized-variance threshold is (0.2)^2.

B.2. Details of Methodology

DeQA-Score adopts an mPLUG-Owl2-style architecture that converts images and text into tokens, fuses them, and uses an LLM to generate the final response.

  • Tokenizing input images and texts: A CLIP-pretrained ViT-L/14 image encoder converts images into visual tokens, while SentencePiece tokenization converts text into textual tokens.Visual tokens have channel size 1024 and textual tokens have channel size 4096.
  • Tokenizing input images and texts: A trainable six-layer transformer image abstractor maps visual tokens to the LLM’s hidden dimension and reduces them to 64 tokens.The LLM hidden dimension is 4096.
  • Token fusion: Visual tokens are inserted into predefined positions within textual tokens through token fusion.
  • Response generation using LLM: The fused tokens are processed by a LLaMA-2-7B LLM to generate the final response.

B.3. Details of Training and Inference

The appendix reports the training configuration and inference details for DeQA-Score, including optimization settings, component tuning, and five-level probability computation.

  • Training Setup: Training uses mPLUG-Owl2 initialization, AdamW, a 2e-5 cosine-decayed learning rate, 0.03 warmup, batch size 64, and three epochs.The loss weighting term is 0.05, bringing the two loss terms to roughly the same scale.
  • Dataset Variation: Images from different IQA datasets can have similar linearly normalized mean scores despite substantially different visual quality.The KonIQ example has the best quality among the images shown.
  • Training Setup: Both the vision encoder and abstractor are trainable, while the LLM is fully tuned unless otherwise specified.LoRA tuning achieves comparable performance to fully tuning the LLM and offers an option for limited-computation settings.
  • Inference: Inference applies a closed-set softmax over five quality levels to compute predicted probabilities.The five-level probability sum after training is very close to 1, so softmax over the five levels or all textual tokens yields nearly identical results.

C. More Results

Additional experiments evaluate score-distribution prediction, unseen AI-generated images, multi-dataset co-training, level counts, component tuning, and qualitative alignment with human annotations.

  • Score Distribution Prediction: DeQA-Score achieves much closer alignment with human annotations than Q-Align under KL divergence evaluation.The appendix attributes Q-Align’s large KL divergence to predictions concentrated on a single level and therefore very small variance.
  • AI-Generated Images: DeQA-Score consistently outperforms Q-Align on additional unseen AI-generated-image datasets after training on KonIQ, SPAQ, and KADID.The reported evaluation uses PLCC / SRCC metrics on AIGCIQA2023, AGIN, and AGIQA-1K.
  • Multi-Dataset Training: DeQA-Score consistently outperforms LIQE and Compare2Score in multi-dataset co-training comparisons.The models are trained on the KonIQ, SPAQ, and KADID datasets.
  • Level-Number Ablation: Performance improves with moderately larger level counts of 6–8 but declines with 10–12 levels.Moderate increases improve discretization accuracy, whereas more categories make level prediction harder.
  • Component Ablation: Training the vision encoder and abstractor significantly improves performance, while LoRA-tuned and fully tuned LLMs achieve similar results.LoRA tuning provides an alternative when computation resources are limited.
  • Qualitative Results: Qualitative results show DeQA-Score’s predicted score distributions align well with human annotations across authentic, artificial, and AI-generated distortions.Q-Align tends to predict a single level, producing higher KL divergence.

D. Extensions

The extensions examine whether score regression transfers to language-based quality description and whether quality description can be jointly trained with accurate scoring.

  • Score Regression Helps Quality Description: Co-training with or pre-training on score regression tasks consistently improves low-level perception performance on Q-Bench.The experiments train score regression on KonIQ, SPAQ, and KADID and evaluate low-level perception tasks on Q-Bench.
  • Score Regression Helps Quality Description: Score regression provides greater benefits than high-level tasks, likely because it is more closely related to low-level perception.This comparison is reported for both co-training and pre-training strategies.
  • Score Regression Helps Quality Description: Pre-training on score regression tasks performs better than co-training for improving low-level perception.The comparison is reported between the corresponding score-regression training configurations in Tab. A8.
  • Discretization Error: When the variance is very small, discretization can increase the recovered variance and flatten the resulting density curve.For N(3.5, 0.252), equal probabilities for “fair” and “good” recover N(3.5, 0.52).
  • Quality Description Harms Score Regression: Co-training with Q-Instruct produces a noticeable decrease in score regression performance.The appendix leaves better strategies for combining language-based quality description and score regression as future work.

E. Limitations and Future Works

The limitations concern joint training with quality description and discretization errors for very low-variance samples, while qualitative examples cover several IQA dataset types.

  • Quality Description Co-Training: Simple co-training with quality description cannot improve score regression, leaving better task-combination strategies for future work.The co-trained model remains reasonable and higher than many non-MLLM-based IQA methods.
  • Discretization Limitation: Discretization introduces variance errors when score distributions have very small variance.Such samples are rare: only 0.29% of KonIQ samples have variance smaller than 0.5^2, and better preservation remains future work.
  • Qualitative Results: The appendix provides qualitative results for in-the-wild IQA datasets sampled from KonIQ, SPAQ, and LIVE-Wild.These examples complement the stated limitation and future-work discussion with visual evaluations across dataset types.
  • Qualitative Results: Additional qualitative results cover artificial distortions from KADID and CSIQ and AI-generated images from AGIQA-3K.The figures extend the qualitative evaluation beyond in-the-wild images.
Loading 2501.11561v3…