PaperlayerView exact revision on arXiv

Source-linked AI summary

VGIQA: Fine-Grained Image Quality Assessment with Visual Grounding Rewards

Weichang Wang, Liyu Chen, Shuyue Guan, Jie Hu, Fei Gao, Wei Li, Nannan Wang

arXiv:2610.37186v1cs.CV

TL;DR

Existing MLLM-based IQA methods mainly provide holistic image-level evaluations rather than fine-grained localized assessments. VGIQA combines local and global assessment through two-stage training, and reports improved degradation perception and stronger local assessment than the cited baselines.

  • Problem

    Existing MLLM-based IQA methods are predominantly holistic and lack fine-grained localized assessment, despite spatially inhomogeneous artifacts in images.

    Source: §1 · p. 2

  • Method

    VGIQA combines a Cold Start phase with multi-task GRPO, jointly training local quality assessment, degradation perception, and score regression with task-specific rewards.

    Source: §3.1 · p. 5, Rewards for Local Quality Assessment · p. 6

  • Results

    VGIQA improves degradation perception over Q-Insight and outperforms Q-Insight and larger general-purpose MLLMs on local assessment, with a VGSR-2K F1 Score of 0.3024.

    Source: §1 · p. 3, §5.2 · p. 10, Degradation Perception · p. 10

  • Takeaways & Limitations

    The framework unifies local and global image quality assessment, while local assessment is reported to enhance degradation perception.

    Source: Degradation Perception · p. 10, §6 · p. 11

Abstract

from arXiv · show

Image quality assessment (IQA) aims to evaluate the perceptual visual quality of images. Recently, the rapid development of Multi-modal Large Language Models (MLLMs) has significantly advanced the IQA field. By integrating chain-of-thought (CoT) reasoning and reinforcement learning (RL), these models can generate more accurate and interpretable results while relying on less training data. However, existing MLLM-based methods are predominantly constrained to holistic, image-level evaluations and lack the capability to produce fine-grained, localized assessments, which substantially limits their practical applicability. To address this limitation, we propose VGIQA, a novel IQA model endowed with both local and global quality assessment capabilities. VGIQA is trained via a two-stage paradigm comprising a cold-start phase followed by reinforcement learning. Driven by carefully designed local quality assessment reward functions, our model exhibits exceptional image quality understanding on VGSR-2K, a new dataset we propose specifically for local quality evaluation, significantly outperforming existing open-source MLLMs such as Qwen2.5-VL-72B-Instruct and Qwen3-VL-32B-Instruct. Furthermore, the local quality evaluation task can effectively enhance the model's performance on the degradation perception task.

1 Introduction

Image quality assessment has advanced from handcrafted metrics to deep and MLLM-based methods, but existing approaches still lack fine-grained localization of image defects. VGIQA combines local and global assessment, and its experiments report stronger local assessment than existing IQA and general-purpose MLLM baselines.

1 Introduction

  • Earlier handcrafted and deep IQA methods generally produce holistic scores, leaving spatially inhomogeneous artifacts such as local blur or structural distortions unlocalized.The introduction identifies these defects as problematic in images that otherwise retain high fidelity.Source: §1 · pp. 1–2
  • VGIQA combines visual grounding with fine-grained IQA, using a Cold Start phase and GRPO to ground global quality scoring in verifiable local evidence.Its reward system uses geometric precision and semantic alignment signals to encourage a “look and then judge” process.Source: §1 · pp. 2–3
  • VGSR-2K is a reference-based dataset with global and coordinate-grounded local evaluations, built from public datasets and human voting and correction of model-generated baselines.The dataset is presented as the first local quality assessment dataset aligned with human preferences.Source: §1 · p. 3
  • Joint training significantly improves degradation perception over Q-insight, while local assessment outperforms Q-insight and Qwen2.5-VL-72B-Instruct [1] and Qwen3-VL-32B-Instruct [2].Figure 1 qualitatively compares VGIQA with Q-insight and Qwen2.5-VL-72B-Instruct [1].Source: Figure 1 · p. 2, §1 · p. 3

2 Related Works

IQA methods have progressed from handcrafted features to deep models and MLLMs, but the reviewed approaches largely evaluate images holistically. VGIQA’s framework adds fine-grained local assessment alongside global tasks.

2 Related Works

  • Handcrafted FR-IQA and NR-IQA methods use reference comparisons or designed statistical features, but struggle with complex scenarios involving local or real-world distortions.Examples include SSIM, VIF, and FSIM for FR-IQA, and BRISQUE and BLIINDS for NR-IQA.Source: §2 · p. 3
  • Deep IQA models use learned features, multi-scale processing, attention, or image-text alignment, yet typically return an overall score without locating degradation.The passage discusses NIMA, ManIQA, MUSIQ, and CLIP-IQA.Source: §2 · pp. 3–4
  • VGIQA’s illustrated framework adds local quality assessment to holistic score regression and degradation perception, using visual grounding to localize artifacts before judgment.The overview describes multi-task reinforcement learning and a “look and then judge” paradigm.Source: Figure 2 · p. 4, Figure 3 · p. 5
  • MLLM-based IQA methods use supervised fine-tuning or reinforcement learning, but the reviewed methods still lack fine-grained localization and evaluation of local defects.The passage describes Q-Align [41] and DepictQA among SFT methods, and Q-Insight [18], Q-Hawkeye [43], and Refine-IQA [11] among RL methods.Source: §2 · p. 4

3 Methodology

VGIQA combines a Cold Start phase with multi-task GRPO to support local and global quality assessment. Its rewards isolate local spatial grounding from holistic evaluation while guiding precise localization.

3.1 Overview

  • VGIQA uses a Cold Start phase followed by multi-task GRPO, with local grounding rewards routed only to local assessment to prevent interference with global tasks.Global tasks include degradation perception and score regression.Source: §3.1 · p. 5

3.2 Cold Start: Initializing IQA-Specific CoT Reasoning

  • Cold Start uses supervised fine-tuning to align structured reasoning and inject IQA knowledge, initializing the model for subsequent multi-task GRPO.The objective is described as supporting interpretable step-by-step reasoning and professional terminology.Source: §3.2 · pp. 5–6

Rewards for Local Quality Assessment

  • Local rewards classify empty, whole-image, and partial-artifact scenes, then use Hungarian-matched DIoU and a dynamic weight to guide localization and limit trivial predictions.The class-balanced base reward addresses spatial imbalance; the dynamic coefficient prevents collapse into trivial solutions such as predicting only empty boxes early in training.Source: Rewards for Local Quality Assessment · p. 6

Overall Multi-Task Reward Formulation

  • Global-task rewards cover format, score regression, distortion type, and severity, while task-specific routing decouples local spatial reasoning from holistic evaluation.The score reward activates when a predicted score falls within an ε-threshold of ground-truth MOS.Source: Overall Multi-Task Reward Formulation · p. 6

4 Proposed Dataset: VGSR-2K

VGSR-2K is a dataset for fine-grained local quality assessment, with spatial boxes grounding erroneous regions in paired images. Its human-in-the-loop pipeline refines annotations and adds pristine negative samples to calibrate spatial predictions.

4.1 Overview of VGSR-2K

  • VGSR-2K pairs low-resolution reference images with super-resolved images or high-resolution ground truth, using bounding boxes to locate erroneous regions.Source: §4.1 · p. 7

4.2 Pipeline

  • The pipeline annotates generation artifacts, semantic inconsistencies, and unrealistic hallucinations through four human-in-the-loop stages.Source: §4.2 · p. 7, Figure 4 · p. 8

Candidate Generation.

Double-Blind Selection.

  • Two independent annotator groups select the most consistent candidate; experts resolve disagreements between their selections.Source: Double-Blind Selection · p. 7

Double-Blind Correction and Merging.

  • Annotators independently remove incorrect or add missing boxes; retained boxes are intersected, while supplemented boxes are matched by negative DIoU and merged as MBRs when DIoU exceeds 0.5.Source: Double-Blind Correction and Merging · p. 7

Final Review & Balancing.

4.3 Dataset Statistics and Distribution

  • VGIQA significantly outperforms large-scale general MLLMs and the IQA-specific baseline on VGSR-2K local quality assessment.Source: Table 1 · p. 9
  • The dataset has two subsets: DiffIQA uses 800 training and 118 testing samples, while SRBench allocates 80 scenes to training and 20 exclusively to testing (220 samples).Source: §4.3 · pp. 8–9

5 Experiments

VGIQA is evaluated on local localization and global quality tasks, with ablations examining its training stages and joint-task training. It excels at local assessment, improves degradation perception, and remains competitive in score regression.

Datasets.

  • Local training subsets are mixed for training and evaluated separately; global score regression follows established benchmark splits, while degradation perception uses DQ-495K training data and 1,000 sampled test images.The score-regression benchmarks include KonIQ, SPAQ, KADID, PIPAL, LIVE-Wild, AGIQA-3K, and CSIQ.Source: Datasets · p. 9

Evaluation Metrics.

  • Local localization is measured with F1 Score and DIoU, score regression with PLCC and SRCC, and degradation perception with category and level accuracy.These metrics are selected to match each task's evaluation target.Source: Evaluation Metrics · p. 9

5.2 Local Quality Assessment

  • F1 Score 0.3024 and DIoU 0.3235; VGIQA substantially outperforms Q-Insight and general-purpose MLLMs, including a model with 10 times more parameters.The comparison is on VGSR-2K; the passage identifies Qwen2.5-VL-72B-Instruct as the 10-times-larger example.Source: §5.2 · p. 10

Degradation Perception

  • VGIQA improves degradation perception over Q-Insight across both accuracy measures in nearly all categories, with a slight JPEG Compression level-accuracy drop.The table caption highlights particularly strong gains for Noise and Null (no degradation).Source: Degradation Perception · p. 10, Table 3 · p. 10

Score Regression

  • VGIQA scores slightly below Q-Insight on PLCC and SRCC but remains competitive and broadly on par with DeQA.The paper attributes the correlation fluctuation to greater sensitivity to local imperfections, which can lower or destabilize predicted scores.Source: Score Regression · p. 10

Ablation of Each Training Stage

  • Cold Start alone yields only marginal F1 Score and DIoU changes; reward-guided GRPO is needed for precise visual grounding.Cold Start provides format alignment and domain priors, but the passage says these are insufficient for precise localization.Source: Ablation of Each Training Stage · p. 11

6 Conclusion

VGIQA combines local and global image quality assessment through a two-stage training framework and a visual-grounding dataset. It substantially outperforms much larger general-purpose MLLMs on local assessment, boosts degradation perception, and remains competitive on score regression.

6 Conclusion

  • VGIQA combines bounding-box localization of erroneous regions against pristine references, VGSR-2K, designed rewards, and cold-start followed by multi-task GRPO training.Experiments report exceptional local understanding versus general-purpose MLLMs with ten times the parameter scale, plus improved degradation perception and competitive score regression.Source: §6 · p. 11

A Ablation of Multi-Task Training

The multi-task comparison finds that adding local quality assessment improves degradation perception relative to global-only training.

A Ablation of Multi-Task Training

  • Joint training with local assessment improves Deg.Acc. and Lev.Acc. over the Global-only baseline across almost all degradation categories.The authors describe localization of microscopic errors as prompting spatial features that help capture global physical degradation clues.Source: A Ablation of Multi-Task Training · p. 15

B More Methodological Details

The appendix details GRPO and the reward system used to train VGIQA. It explains group-relative response scoring and the rewards for global assessment and local spatial grounding.

B.1 Preliminaries of Group Relative Policy Optimization

  • GRPO scores multiple candidate responses per query with task-specific rewards, then computes each response’s relative advantage from rewards within the group rather than using a separate critic.The policy objective also uses a clipped surrogate and KL penalty for stable training.Source: §B.1 · p. 15

B.2 More Details of Rewards Design

  • The reward system combines global task and format rewards with a local visual-grounding reward based on spatial matching.It reuses Q-Insight’s format, score-regression, and degradation-perception rewards; format reward checks the <think>/<answer> structure, while global score and degradation rewards check score error and distortion labels or severity.Source: §B.2 · p. 16
  • Local predictions and ground-truth boxes are matched by the Hungarian algorithm using negative DIoU costs, with a DIoU threshold of 0.5 defining true positives.Spatial precision and recall yield an F1 reward that evaluates localization and the number of predicted artifacts.Source: §B.2 · pp. 16–17

C.1 Data Composition and Source Distributions

This appendix describes the sources and sampling used to construct VGSR-2K, along with the candidate-generation examples and the scope of additional experimental setup details.

C.1 Data Composition and Source Distributions

  • VGSR-2K draws on SRBench’s diverse super-resolution outputs and DiffIQA’s generative-enhancement artifacts, with sampled image pairs including pristine references to balance positive and negative examples.The pristine pairs are intended to prevent the policy from exploiting trivial over-generation during reinforcement learning.Source: §C.1 · p. 17

C.2 Qualitative Examples of Candidate Generation

  • Two multimodal foundation models generate diverse candidate boxes for human screening and cross-correction in the spatial annotation pipeline.Fig. A shows a reference/generated image pair alongside six candidate-generation results.Source: §C.2 · p. 17

D Additional Experimental Setup

  • The appendix’s additional experimental setup section covers evaluation benchmarks, baseline selection, and implementation configurations for multi-task training.It directs readers to the main paper for final data ratios and distributions.Source: D Additional Experimental Setup · p. 17, §C.1 · p. 17

D.1 Evaluation Datasets

The evaluation setup spans score-regression benchmarks, a degradation-perception benchmark, and comparisons against handcrafted, deep-learning, and MLLM-based baselines.

D.1 Evaluation Datasets

  • Score regression is evaluated across in-the-wild, synthetic-distortion, model-processed, and AI-generated image datasets, with KonIQ used for training and the remaining datasets for OOD evaluation.The described groups include KonIQ, SPAQ, LIVE-Wild, KADID, CSIQ, PIPAL, and AGIQA.Source: §D.1 · p. 17, Qwen3-VL-Plus · p. 18
  • Degradation perception uses 7,000 single-distortion DQ-495K images for training and 1,000 non-overlapping, identically distributed images for testing because Q-Insight’s test set is unreleased.Source: Qwen3-VL-Plus · p. 18

D.2 Baseline Models

  • Score-regression comparisons include handcrafted, non-MLLM deep-learning, and MLLM-based methods, with non-handcrafted methods trained on KonIQ and MLLM baselines using approximately 7B parameters.The listed MLLM baselines include C2Score, Q-Align, DeQA-Score, Q-Insight, and a supervised fine-tuned Qwen.Source: §D.2 · p. 18

D.3 Implementation Details

The appendix gives joint-training settings and prompt designs for VGIQA’s assessment tasks. Its new local-quality prompt compares a low-resolution reference with a super-resolved output to localize and explain problematic regions.

D.3 Implementation Details

  • Training uses three cold-start epochs followed by ten GRPO epochs, with task-specific batch sizes of 8 for local assessment and 16 for degradation perception and score regression.The local reward weight decays from 5.0 to 1.0; training runs on eight 80 GB compute nodes.Source: §D.3 · pp. 18–19

E Designed Prompts

  • The framework uses Q-Insight’s holistic-task prompts and adds a local-assessment prompt with <think>/<answer> reasoning, which the authors say enhances spatial-grounding interpretability and final-judgment accuracy.Table B identifies the local prompt as the newly designed core contribution.Source: E Designed Prompts · p. 19
  • For local assessment, the prompt asks the model to compare an LR reference with an SR output, localize degraded or hallucinated regions in bounding boxes, and explain each issue in JSON.The score-regression prompt requests a 1–5 rating; the degradation prompt requests a distortion class and severity.Source: E Designed Prompts · pp. 19–20

Related papers

Loading 2610.37186v1…