PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-SiT: Teaching LMMs for Image Quality Scoring and Interpreting

Zicheng Zhang, Haoning Wu, Ziheng Jia, Weisi Lin, Guangtao Zhai

arXiv:2503.09197v2cs.CV

TL;DR

Image quality scoring and interpreting have often been treated separately, despite the paper's framing of them as connected tasks with different abilities and formats. Q-SIT jointly trains an LMM using scoring and interpreting data and a strategy for balancing their mixture. The authors report that the framework maintains performance across both tasks, with a lightweight variant offering a cost-effective deployment alternative.

  • Problem

    Image quality scoring and interpreting have often been addressed separately, despite differing abilities and formats and the challenge of limited low-level visual training data.

    Source: §I · pp. 2–3

  • Method

    Q-SIT unifies scoring and interpreting in one LMM, using scoring data, human-annotated low-level interpreting data, and a data-mix strategy selected with a small model.

    Source: §I · p. 3

  • Results

    The authors report that their balance strategy mitigates task interference and supports strong performance across scoring and interpreting, while Q-SiT-mini maintains competitive performance with reduced inference costs and time.

    Source: §I · p. 3

  • Takeaways & Limitations

    Q-SiT-mini is presented as a cost-effective alternative for practical deployment, and the findings support the feasibility of unifying scoring and interpreting in one model.

    Source: §V · p. 12

Abstract

from arXiv · show

Image quality scoring and interpreting are two fundamental components of Image Quality Assessment (IQA). The former quantifies image quality, while the latter enables descriptive question answering about image quality. Traditionally, these two tasks have been addressed independently. However, image-quality-specific psychophysical studies suggest that these two tasks are conceptually interconnected: interpreting explicitly represents perceived quality attributes whereas scoring summarizes such evidence into an overall quality judgment. Thus, unifying these capabilities within a single model is both intuitive and logically coherent. In this paper, we propose Q-SiT (Quality Scoring and Interpreting joint Teaching), a unified framework that enables large multimodal models (LMMs) to learn both image quality scoring and interpreting simultaneously. We achieve this by transforming conventional IQA datasets into learnable question-answering datasets and incorporating human-annotated quality interpreting data for training. Furthermore, we introduce an efficient scoring \& interpreting balance strategy, which first determines the optimal data mix ratio on lightweight LMMs and then maps this ratio to primary LMMs for fine-tuning adjustment. This strategy not only mitigates task interference and enhances cross-task knowledge transfer but also significantly reduces computational costs compared to direct optimization on full-scale LMMs. With this joint learning framework and corresponding training strategy, we develop Q-SiT, the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini. Experimental results demonstrate that Q-SiT achieves strong performance in both tasks with superior generalization IQA abilities, while Q-SiT-mini significantly reduces computational overhead while maintaining competitive performance.

I. INTRODUCTION

Image quality scoring and interpreting are presented as interconnected parts of a unified evaluation process, although prior work generally treated them separately. Q-SiT jointly teaches both tasks using scoring and interpreting pipelines and a data-balancing strategy.

I. INTRODUCTION

  • The paper frames interpreting as perception and scoring as decision-making, treating them as interconnected components of image quality evaluation.It argues that subjective quality evaluation involves describing perceived attributes and forming an overall quality decision.Source: §I · p. 1
  • Scoring-only training can degrade interpreting, while interpreting training can improve scoring to some extent, motivating a strategy for balancing the tasks.The tasks differ in required abilities and formats, and joint training also risks degrading semantic understanding when it relies exclusively on low-level data.Source: §I · p. 3
  • Q-SIT uses a small LMM to identify a data mix ratio, then transfers that ratio and loss information to train the primary model, reducing the need for full-scale optimization.The training data are divided into scoring, interpreting, and general-understanding datasets.Source: §I · p. 3
  • The proposed unified framework enables a single LMM to perform both image quality scoring and interpreting, tasks previously handled by separate models.The paper also introduces a lightweight model intended to reduce inference costs and time while preserving competitive performance.Source: §I · p. 3
  • The scoring pipeline trains LMMs to assign probabilities to text-defined quality levels, while the interpreting pipeline uses human annotations to inject low-level knowledge.The human feedback includes descriptions of attributes such as blur, noise, clarity, color, and brightness, together with an overall quality conclusion.Source: §I · pp. 2–3

II. RELATED WORKS

The related-work passages describe multimodal-model development through visual-input integration and instruction tuning, alongside GPT-assisted generation of vision-language training data. The supplied figure caption outlines a three-stage human-rating process used to motivate an LMM scoring syllabus.

A. Large Multi-modality Models

  • OpenFlamingo integrated gated cross-attention blocks into pretrained language encoder layers, while InstructBLIP extended BLIP-2 through vision-language instruction tuning.The passage situates these systems among LMMs used for captioning, visual question answering, segmentation, classification, and reasoning.Source: A. Large Multi-modality Models · p. 3
  • The depicted rating process trains raters on text-defined levels, collects categorical or slider ratings, then converts them to MOS by weighted averaging.The proposed LMM syllabus simulates the training stage, and probability-based inference is used to predict final scores.Source: Figure 3 · p. 4
  • GPT-4-assisted data generation contributed to the LLaVA series of open-source LMMs.The cited passage describes this as part of efforts to advance open-source LMMs.Source: A. Large Multi-modality Models · p. 4

B. Image Quality Assessment (IQA).

Prior IQA work spans handcrafted, deep-learning, vision-language, and LMM approaches, but scoring and interpreting remain separate. The Q-SIT section introduces separate pipelines followed by a balance strategy, while its rating-process discussion describes how human judgments become MOS.

B. Image Quality Assessment (IQA).

  • Prior IQA methods include handcrafted features, deep networks, vision-language approaches, and LMMs, but scoring and interpreting still rely on separate models without a unified framework or joint training strategy.The passage names NIMA, DBCNN, HyperIQA, MUSIQ, CLIP-IQA+, and LIQE among earlier approaches.Source: B. Image Quality Assessment (IQA) · p. 4

III. THE Q-SIT

  • The Q-SIT section first introduces scoring and interpreting pipelines separately, then presents a strategy to balance the two tasks.This passage states the section's organization but does not specify the strategy's mechanics.Source: §III · p. 4

1) [Insight 1] How Do Humans Rate?:

  • Human rating proceeds from aligning raters to rating levels, through categorical or slider judgments, to averaging those judgments into MOS.Raters do not directly provide numerical scores, and they do not participate in the final averaging stage.Source: 1) [Insight 1] How Do Humans Rate? · p. 4
  • The paper says qualitative feedback reflects everyday responses and can reduce cognitive load while improving subjective-evaluation accuracy.Raters are neither trained nor instructed to predict precise numerical scores.Source: 1) [Insight 1] How Do Humans Rate? · p. 4

2) [Insight 2] How Do LMMs Rate?:

The paper finds that unaligned LMMs tend to express image ratings with qualitative adjectives, motivating text-defined rating levels. It converts scores into five such levels for training and reports that this conversion retains strong linear correlation with original scores.

2) [Insight 2] How Do LMMs Rate?:

3) Conversion Between Rating Levels and Scores:

4) [Training] Scores →Rating Levels:

  • Training conversion divides the range from the highest to lowest score into five equal intervals and assigns each interval a standard text rating level.The authors report that this multi-to-one conversion retains strong linear correlation with the original scores across five IQA datasets.Source: 4) [Training] Scores →Rating Levels · p. 5

5) [Inference] Rating Levels →Scores:

At inference, the model’s predicted rating levels are converted back into scores using a reverse mapping and level probabilities. The resulting expectation follows the weighted-average form used to calculate human MOS and generalizes a binary-softmax strategy.

B. Teach LMMs for Image Quality Interpreting

The interpreting-data pipeline combines human pathway feedback with question-answering and conversation examples. It uses a curated image set and a controlled study to build detailed low-level visual annotations, then converts those annotations into additional VQA items.

B. Teach LMMs for Image Quality Interpreting

  • The interpreting training data combines human pathway feedback with VQA and extended-conversation subsets to support varied queries and low-level visual discussion.Source: B. Teach LMMs for Image Quality… · p. 5

1) Preparation of Images:

2) Task Definition of Pathway Feedbacks:

  • Pathway feedback pairs detailed natural-language descriptions of low-level attributes with an overall quality conclusion, preserving context such as where brightness or darkness occurs.The authors argue that this structured format better captures perception than multidimensional scores alone and follows human reasoning from attributes to a conclusion.Source: 2) Task Definition of Pathway Feedbacks · pp. 5–6

3) The subjective study process.:

  • A controlled laboratory study with 39 trained subjects collected 58K pathway feedbacks, using calibration materials and limits on consecutive images to reduce fatigue.Source: 3) The subjective study process · p. 6

4) Visual Question Answering (VQA):

  • GPT-generated VQA examples convert pathway feedback into 76K questions, 57K balanced yes-or-no questions, and an additional multiple-choice format with distractors.Questions include opinion adjectives, attribute or context nouns, and binary judgments.Source: 4) Visual Question Answering (VQA) · p. 6

5) Extended Conversations:

The paper describes a two-stage search on smaller models to estimate data-mixing ratios, then considers how the joint tasks’ different demands and formats affect performance. The cited comparison shows interpreting-only training can support scoring, while scoring-only training performs worse on interpreting.

5) Extended Conversations:

  • The proposed ratio search tests D2:D3 mixes on a validation set, then tests the resulting mixture against D1 and fits regressions to estimate coarse-grained ratios.The figure plots LLVisionQA accuracy across D2:D3 ratios and the mean of LLVisionQA accuracy and IQA correlation metrics across the combined-data-to-D1 ratio.Source: 5) Extended Conversations · p. 6, Figure 4 · p. 7
  • The 12K GPT-generated extended conversations cover low-level image analysis, photography suggestions, editing tools, consumer recommendations, and related discussion.Source: 5) Extended Conversations · p. 6

1) Why Mixing All Doesn’t Work:

  • Scoring and interpreting differ in ability demands and response formats: scoring makes a quantitative judgment, while interpreting uses fine-grained perception and varied question forms.Source: 1) Why Mixing All Doesn’t Work · p. 6
  • Q-Instruct, trained only on interpreting, achieves balanced scoring performance across IQA datasets, whereas scoring-only Q-Align declines substantially on interpreting.Source: 1) Why Mixing All Doesn’t Work · pp. 6–7

2) Data Utilization:

The section defines three training-data types and describes a small-model procedure for selecting their mixture. Regression first balances interpreting and general-understanding data, then incorporates scoring data.

2) Data Utilization:

  • D1 contains reformatted quality-assessment data for scoring, D2 contains image-quality interpreting data, and D3 uses general-understanding data to preserve semantic cognition.A system prefix is prepended to D1 examples to help distinguish scoring from interpreting.Source: 2) Data Utilization · p. 7

3) Predicting the Coarse-Grained Optimal Data Mix Ratio:

  • A small LMM is evaluated on varying D2:D3 proportions, and regression model A predicts the mix that balances scoring and interpreting performance.The validation scores used are mean performance scores for both tasks.Source: 3) Predicting the Coarse-Grained… · p. 7
  • The selected D2:D3 mix is combined with D1 at varying proportions, and regression model B predicts the optimal proportion between D1 and the D2+D3 mixture.The passage defines this combined dataset as |D2*+D3*|.Source: 3) Predicting the Coarse-Grained… · p. 7
  • The resulting D1:D2:D3 mix is the Coarse-Grained Optimal Data Mix Ratio, paired with a reference ratio of validation losses for scoring and interpreting.The reference ratio uses scoring loss LS and interpreting loss LI.Source: 3) Predicting the Coarse-Grained… · p. 7

4) Adjusting the Fine-Grained Optimal Data Mix Ratio:

The fine-grained procedure transfers a coarse data mix to the primary model and adjusts dataset sizes according to validation losses. The section also specifies training and evaluation settings and identifies the accompanying performance visuals.

4) Adjusting the Fine-Grained Optimal Data Mix Ratio:

  • When a task’s validation-loss proportion deviates significantly from the reference ratio, the primary model receives more data for that task in the next epoch until the ratio aligns.The paper describes this adjustment as a way to balance scoring and interpreting performance.Source: 4) Adjusting the Fine-Grained Optimal… · pp. 7–8

A. Experiment Settings

  • Table IV reports scoring performance for models trained on KonIQ and SPAQ, with average performance computed across IQA datasets.Cross-set evaluations are labeled CROSS.Source: Table IV · p. 8
  • Training uses LLaVA-Onevision-0.5B for the small model and LLaVA-Onevision-7B for the primary model, with batch size 64, learning rate 1e-5, and a default of three primary-model epochs.Training runs on eight NVIDIA A800 80G GPUs; inference latency is measured on one such GPU.Source: A. Experiment Settings · p. 8
  • Fig. 5 summarizes image-quality LMM performance using normalized (SRCC+PLCC)/2 scores and reports each model’s average across datasets.The normalization is relative to the maximum performance observed for each dataset.Source: Figure 5 · p. 8
  • Scoring tests use KonIQ, SPAQ, LIVEC, AGIQA-3K, and KADID-10k, while interpreting tests use LLVisionQA-test and LLDescribe, with train/test overlap removed.Validation uses 10% of KonIQ and SPAQ training data for scoring and LLVisionQA-dev for interpreting.Source: A. Experiment Settings · p. 9

B. Details about Data Mix Ratio

The authors estimate the coarse data mix on a lightweight model, then use it to initialize fine-grained adjustment on the primary model. They report predicted ratios, a reference loss ratio, and example visualizations.

B. Details about Data Mix Ratio

  • The fitted polynomial models predict D2:D3 = 2.42 and |D2*+D3*|:D1 = 3.54 as the coarse mix ratios.The first ratio is estimated using LLVisionQA-dev; the second uses LLVisionQA-dev and average IQA performance.Source: B. Details about Data Mix Ratio · p. 9
  • The resulting approximate D1:D2:D3 mix is 1.00:2.50:1.04, and the scoring-to-interpreting validation-loss ratio is approximately 1:4.66.The ratio is used to train the LLaVA-OneVision-7B primary Q-SIT model.Source: B. Details about Data Mix Ratio · p. 9
  • Fig. 6 visualizes Q-SIT results for one low-quality image and one high-quality image.The supplied table caption identifies Table VI as results on image-quality description interpreting ability.Source: Figure 6 · p. 10, Table VI · p. 10
  • After each training iteration, the primary model’s scoring and interpreting validation losses guide dynamic data-mix adjustment, usually over no more than three epochs.The procedure follows the fine-grained adjustment strategy described in Algorithm 1.Source: B. Details about Data Mix Ratio · p. 10

C. Quick Look of the Performance

Q-SIT is reported to balance image-quality scoring and interpreting, with gains in cross-dataset scoring and strong interpreting performance. The paper also reports competitive mixed-dataset scoring and a lightweight model that retains competitive description ability.

C. Quick Look of the Performance

  • Q-SIT delivers highly competitive, well-balanced performance across scoring and interpreting, unlike Q-Align’s reported interpreting decline after scoring-only training.Q-Instruct, trained only on interpreting data, has relatively balanced performance but does not stand out on IQA tasks.Source: C. Quick Look of the Performance · p. 10

D. Image Quality Scoring Performance

  • Q-SIT significantly improves cross-dataset scoring over visual-only methods by over 10%, and surpasses CLIP-IQA+ by 8% and LIQE by 6%.It slightly underperforms Q-Align in intra-dataset evaluations but significantly outperforms it in cross-dataset settings.Source: D. Image Quality Scoring Performance · pp. 10–11
  • With mixed-dataset scoring training, Q-SIT performs best on nearly all datasets except KonIQ and improves across all IQA datasets, including cross-dataset settings.Q-SIT-MINI is weaker than Q-SIT and Q-Align but outperforms other baseline models and shows cross-dataset generalization.Source: D. Image Quality Scoring Performance · p. 11

E. Image Quality Interpreting Performance

  • Q-SIT and Q-SIT-MINI achieve the best LLVisionQA performance within their respective 0.5B and 7B parameter groups, and excel in Yes-or-No and What questions.Both models also show a significant lead in the distortion category.Source: E. Image Quality Interpreting… · p. 11
  • Q-SIT-MINI’s LLDescribe performance is comparable to mid-tier 7B LMMs, while Q-SIT ranks second among 7B-level models.Under the same large-model backbone, Q-SIT also outperforms Q-Instruct on LLVisionQA and LLDescribe.Source: E. Image Quality Interpreting… · p. 11

F. Qualitative Analysis

Q-SIT performs strongly across scoring and interpreting, while Q-SIT-MINI offers a more efficient alternative. Ablations support the value of task-specific datasets and the proposed data-mixing strategy, and qualitative examples show both scoring and descriptive outputs.

F. Qualitative Analysis

  • In the qualitative example, Q-SIT rates image B higher than A, identifies A's quality issues, and perceives the flying bird's details in B.Source: F. Qualitative Analysis · p. 11

G. Ablation Study

  • Removing D1 degrades scoring and excluding D2 impacts interpreting; combining D2 and D3 further improves interpreting, while progressively applying mix-ratio strategies improves both tasks.Simply mixing data without a training strategy yields suboptimal results.Source: G. Ablation Study · p. 11

H. Computational Efficiency

  • The lightweight-model ratio search and 7B fine-tuning strategy reduces training cost by approximately 72% versus direct mix-ratio search on the 7B model.The reported estimates are approximately 720 A800 GPU hours for the 0.5B model plus 72 hours for Q-SIT, versus approximately 2,800 hours for direct search.Source: H. Computational Efficiency · p. 11
  • 49% lower inference latency, 25% of Q-SIT's VRAM use, and approximately 92% fewer FLOPs let Q-SIT-MINI retain balanced scoring and interpreting performance.Fig. 7 compares latency, VRAM, and FLOPs averaged across test tasks on one NVIDIA A800 80G GPU.Source: Figure 7 · p. 12, H. Computational Efficiency · p. 12

V. CONCLUSION

  • Q-SIT achieves state-of-the-art performance in image quality scoring and interpreting, while Q-SIT-MINI reduces computational overhead and retains competitive performance.The conclusion presents Q-SIT-MINI as a cost-effective alternative for practical deployment.Source: §V · p. 12
Loading 2503.09197v2…