PaperlayerView exact revision on arXiv

Source-linked AI summary

Of Chains and Gains: What Reinforcement Learning changes on Reasoning-based Image Quality Assessment

Yixuan Li, Yuxuan Jiang, Hanwei Zhu, Hancheng Zhu, Fan Zhang, Shiqi Wang, David Bull

arXiv:2610.33203v1cs.CVcs.CL

TL;DR

The paper asks whether RLVR's IQA gains reflect better reasoning or instead sampling effects and scores that bypass the chain. Using sampling-aware evaluation, chain interventions, and shortcut experiments, it finds improved scoring and more useful chains without stronger chain reliance, alongside gains from MOS-correlated textual cues.

  • Problem

    Because RLVR rewards score agreement with MOS rather than reasoning, it remains unclear whether IQA gains come through better chains or from sampling effects, a direct scoring path, or shortcuts.

    Source: §1 · p. 2

  • Method

    The study uses maxg @k evaluation and chain perturbation and transplantation to assess sampling effects and chain dependence, alongside controlled tests of MOS-correlated cues.

    Source: §4.1 · p. 4, §5.1 · p. 6, §6.2 · p. 9

  • Results

    RL-trained models improve quality prediction beyond sampling effects mainly in-distribution, produce more useful chains, and rely less on them during inference; correlated textual cues can also improve scores.

    Source: §1 · p. 3, §6.2 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    Higher IQA scores do not establish stronger reliance on explicit reasoning, and MOS-correlated cues can contribute to apparent scoring gains.

    Source: §1 · p. 3

    The study restricts its scope to explicit reasoning that requires models to output readable reasoning.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the performance of large Vision-Language Models (VLMs) on image quality assessment (IQA). Existing approaches often attribute these gains to improved reasoning before scoring, supported by increasingly plausible Chain-of-Thought (CoT). However, whether these gains exceed sampling effects and how they depend on explicit reasoning remain unclear. We rigorously investigate these questions through sampling-aware evaluation, controlled reasoning interventions, and shortcut experiments. We find that RL-trained models outperform base models under large-budget $\operatorname{max}_g@k$ evaluation, while improvements beyond averaging are concentrated in-distribution. RLVR produces chains that raise base-model performance when transplanted and improves scoring with the chain held fixed. Yet scoring becomes less dependent on the chain during inference, more clearly at 7B than at 32B. Steering the probed quality direction in thinking tokens also has little effect on scoring. Finally, RL can improve scoring accuracy by exploiting textual cues correlated with mean opinion scores (MOSs), which could inflate benchmark performance. Together, these findings show that RLVR raises quality scoring and produces more useful chains, but that better scoring does not imply a stronger reliance on explicit reasoning, while score-based rewards can also encourage reliance on quality-irrelevant shortcuts. We hope this preliminary study can enlighten more and more concrete reasoning schemes in IQA.

1 INTRODUCTION

The introduction asks whether RLVR improves image-quality reasoning or can raise scores through sampling effects, a direct image-to-score path, or MOS-correlated shortcuts. Controlled sampling and chain interventions find stronger scoring and more useful chains, but not greater reliance on explicit reasoning at inference.

1 INTRODUCTION

  • The paper asks whether better scores follow from better chains or can arise through sampling effects, a direct image-to-score path, or shortcuts correlated with MOS.Because the reward verifies score agreement rather than reasoning, it does not establish that score improvements are obtained through the chain.Source: §1 · p. 2
  • The experiments compare RL-trained and base models under maxg @k, intervene on chains, and train with textual cues whose correlation with MOS is controlled.Interventions include swapping or removing chain content, transplanting chains across models, and steering a hidden representation along a probed quality direction.Source: §1 · pp. 2–3
  • The study restricts its scope to explicit reasoning that requires models to output readable reasoning.Source: §1 · p. 2
  • RLVR improves quality prediction beyond sampling effects, with gains beyond averaging concentrated in-distribution and potentially absent out-of-distribution.The introduction also reports that RL can improve scores by exploiting textual cues correlated with MOS, including cues without quality semantics.Source: §1 · p. 3
  • RLVR produces more useful chains and improves the scorer, yet RL-trained models rely less on the chain during inference than base models.The conclusion attributes gains to both the chain and an improved mapping from image and chain to score, while emphasizing the gap between accurate and explainable IQA.Source: §1 · p. 3

2 RELATED WORKS

Prior IQA work reports gains from reasoning-based methods, but whether scores depend on chains, gains persist under large sampling budgets, or rewards exploit shortcuts remains unclear. This paper brings CoT faithfulness questions to IQA, where scores rely on dense visual evidence and lack exact correctness.

CoT faithfulness and mechanistic interpretability

  • Prior CoT-faithfulness studies focus on tasks with essential chains and exact correctness, whereas IQA relies on dense visual evidence and has no exact correctness.Source: §2 · p. 3

RLVR for IQA

  • Reasoning-based IQA methods report significant improvements, but whether scores depend on chains, gains persist at large sampling budgets, or rewards exploit shortcuts remains unclear.Source: §2 · p. 3
  • Textual reasoning can suffice to support an IQA score under essential conditions, but that does not establish that the score must pass through the chain.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe RLVR scoring and the think-then-score structure used to study reasoning-based IQA. They define the model response and rewards, then frame chain interventions and the model and benchmark setup.

Reinforcement learning with verifiable rewards

  • A VLM generates a reasoning chain followed by a quality score, and training maximizes expected reward with GRPO without explicit supervision of reasoning content.The verifier is the image's MOS; score-based reward checks whether the score falls within a tolerance, while rank-based reward measures agreement between image preference and MOS order.Source: §3 · pp. 3–4

Reasoning paradigms for IQA

  • In think-then-score IQA, the image can affect the score through the chain, directly, or through both paths, motivating interventions on the chain.Source: §3 · p. 4

Models and benchmarks

  • Experiments use Qwen2.5-VL-Instruct at 7B and 32B, training on KADID-10k or KonIQ-10k and evaluating on their test sets plus unseen datasets.The named unseen datasets are SPAQ, AGIQA-3K, and CSIQ.Source: §3 · p. 4

4 RLVR’S EFFECT ON IQA REASONING CAPACITY OF VLMS

Across sampling budgets, RL-trained models score above base models under maxg @k; averaging recovers some gains, but the remaining mapping advantage is concentrated in-distribution. Perplexity analysis also finds score-reward chains less typical of the base model, especially at 32B.

4.1 SETUPS

  • Models were sampled 512 times per image; maxg @k estimates the best SRCC among k complete sampled test-set predictions, using MOS only to select candidates.The estimate equals mean candidate SRCC at k=1 and the largest observed SRCC at k=512.Source: §4.1 · p. 4

4.2 IQA REASONING CAPACITY RESULTS

  • At k=1, RL-trained models lead by 0.03 to 0.46 SRCC; at k=512, every comparison remains positive and even the best base candidate trails one RL draw.Figure 2 shows maxg @k curves across test sets and model scales.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
  • Averaging recovers denoising, while RL's mapping gain is large in-distribution but small or absent out-of-distribution, where averaged base models may reach or cross single RL draws.Figure 3 presents avg @k curves; the panels follow Figure 2's columns.Source: §4.2 · pp. 5–6, Figure 3 · p. 5
  • Perplexity indicates rank-reward chains resemble base-model outputs, while score-reward chains shift further outside the base distribution at 32B than at 7B.Neither reward targets reasoning text, and the scores change substantially in both cases.Source: §4.2 · p. 6

5 CAUSAL ROLE OF COT IN IQA

The experiments test whether RLVR’s IQA gains flow through explicit chains, directly from images to scores, or through both paths. RLVR improves scoring and chain usefulness, but scoring becomes less dependent on chains at inference, with scale-dependent effects.

5.1 SETUPS

  • The setup distinguishes mediated, direct, and joint gain hypotheses, then tests chain swaps, content removal, cross-model transplants, and hidden-state steering.The interventions hold the image and original prompt fixed while changing the chain or its hidden state.Source: §5.1 · pp. 6–7

5.2 REASONING PERTURBATION RESULTS

  • After RLVR, scores follow the input image more than a swapped chain, especially at 7B, while removing chain content barely changes performance and 32B models can be more reliant.These interventions concern chain use at inference, not its role during training.Source: §5.2 · pp. 7–8
  • RL-trained chains improve base-model scoring when transplanted, while RL-trained models generally score better with the chain fixed; the balance differs between 7B and 32B.At 7B, RL models score almost as well with base-model chains as with their own; at 32B, they lose part of the gain without their own chains.Source: §5.2 · pp. 7–8, Table 1 · p. 7
  • Shifting thinking-token states by up to one probe-reading point changes SRCC by at most 0.005, no more than an equal-norm orthogonal control.The probe direction encodes quality, but the result indicates that informative hidden states do not mean scoring depends on them.Source: §5.2 · p. 8, Figure 7 · p. 8

6 SHORTCUTS TO THE QUALITY REWARD

The experiments test whether RLVR can exploit prompt cues correlated with MOS rather than assess image quality. Correlated training improves aligned-test SRCC, but the advantage reverses when cue assignments are reversed; both quality-stating and nonsemantic cues can be exploited.

6.1 SETUPS

  • The study compares Qwen2.5-VL-7B models trained with cues shuffled across images or paired with high- and low-MOS images, then tests aligned, shuffled, reversed, and removed cues.The images and MOSs remain unchanged; only the textual cue changes.Source: §6.1 · pp. 8–9

6.2 SHORTCUT RESULTS

  • 0.026 to 0.069 SRCC: correlated training beat independent controls under aligned cues, but every correlated model fell below its control under reversed assignments.The reported interactions were 0.046 for A/B identifiers, 0.903 for source descriptions, 0.892 for numerical hints, and 0.300 for text length.Source: Table 2 · p. 9, §6.2 · p. 9
  • Reversing source descriptions and numerical hints lowered correlated-model SRCC from 0.881 and 0.878 to −0.027 and −0.011; A/B identifiers and repeated neutral text were exploited more weakly.Removing either nonsemantic cue returned the correlated model to its control’s level.Source: §6.2 · p. 9
  • Independent training reduced cue shifts below 0.01, whereas correlated training strengthened them above 1.3 and gained an advantage whenever the cue predicted MOS.Base models already responded to source descriptions and numerical hints, so cue dependence was not wholly absent before RL.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion separates RLVR’s scoring gains from sampling effects and tests whether they depend on explicit reasoning. It finds more useful chains and better scoring, alongside weaker inference-time chain dependence and vulnerability to MOS-correlated textual shortcuts.

7 CONCLUSION

  • RL-trained models outperform base models across sampling budgets, while gains beyond averaging are concentrated in-distribution and small or absent out-of-distribution.Averaging base-model draws recovers the denoising component of the gain.Source: §7 · p. 10
  • RLVR improves chain usefulness and scoring simultaneously, but better scores do not imply stronger chain dependence at inference, especially at 7B compared with 32B.The conclusion says the interventions address inference-time use of chains, not their role during training.Source: §7 · p. 10
  • Textual cues correlated with MOS can improve scores, including cues without quality semantics, so the conclusion calls for evaluating chain dependence and shortcut robustness alongside reasoning-linked rewards.The paper presents this as a preliminary study.Source: §7 · p. 10

Related papers

Loading 2610.33203v1…