PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Rewarding Vision-Language Model via Reasoning Decomposition

Zongxia Li, Wenhao Yu, Chengsong Huang, Zhenwen Liang, Rui Liu, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, Dong Yu

arXiv:2508.19652v2cs.CV

TL;DR

VLMs often hallucinate or rely on language shortcuts because post-training supervises final answers more directly than visual reasoning. Vision-SR1 uses self-contained visual reasoning, same-model self-rewarding, and decoupled multi-reward optimization; it improves visual reasoning and reduces shortcuts across diverse tasks without additional reward-model GPUs.

  • Problem

    VLM post-training often rewards only answer matching, leaving intermediate visual reasoning unsupervised and enabling reliance on language priors.

  • Method

    Vision-SR1 decomposes reasoning into visual perception and language reasoning, verifies self-contained perception with a second same-VLM rollout, and optimizes separate reward advantages.

  • Results

    Vision-SR1 improves visual reasoning, mitigates hallucinations, and reduces language shortcuts across diverse vision-language tasks while avoiding additional reward-model GPU overhead.

  • Takeaways & Limitations

    Self-generated perception rewards provide a practical way to strengthen visual grounding and language reasoning without external visual supervision.

  • Takeaways & Limitations

    Some mathematical gains from VLM reinforcement learning may reflect recalibrated language distributions or language-shortcut exploitation rather than genuine visual grounding.

Abstract

from arXiv · show

Vision-Language Models (VLMs) often suffer from visual hallucinations: generating things that are not consistent with visual inputs and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language based reasoning over visual perception. We introduce Vision SR1, a three stage self rewarding reinforcement learning method that improves visual reasoning without relying on external visual supervision. Vision SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning, where the model is first prompted to produce self-contained visual descriptions sufficient to answer the question without referring back to the input image, before jointly optimizing both visual and language reasoning through our multi reward loss objective. To validate this self containment, the same VLM model is reprompted to perform language reasoning using only the generated visual reasoning as input to compute visual reward. The final reward is computed through a decoupled reward-advantage framework, where visual reward and language reasoning reward each have their advantages calculated separately. Our experiments show that Vision SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision SR1 introduces no extra GPU overhead beyond that of standard training.

1 INTRODUCTION

Vision-SR1 addresses visual hallucinations and language shortcuts by decomposing VLM reasoning into self-contained visual perception and language reasoning, then training them with separate reward signals. Its two-pass self-rewarding process verifies visual sufficiency using the same VLM and combines distinct optimization signals.

  • Motivation: VLMs can hallucinate visual content or bypass visual understanding by relying on language priors.Existing reinforcement-learning methods may encourage language reasoning while demoting visual perception.
  • Motivation: Existing post-training methods usually match only final answers, leaving intermediate visual reasoning without explicit supervision.Human annotations and externally distilled labels provide alternatives but introduce external supervision.
  • Vision-SR1: Vision-SR1 separates visual perception from language reasoning and requires the visual output to contain all information needed to answer without the image.The framework computes separate advantages, rollouts, policy losses, and KL terms before combining them into one objective.
  • Vision-SR1: The first rollout generates visual perception, chain-of-thought reasoning, and an answer from the image and query, while an accuracy reward evaluates the final answer.The model’s structured output explicitly distinguishes visual perception, reasoning, and answer components.
  • Vision-SR1: A second rollout uses only the query and generated visual perception, assigning self-visual reward when the model derives the correct answer.This tests whether the visual perception is faithful and self-contained without re-accessing the original image.
  • Vision-SR1: Multi-reward policy optimization gives visual reasoning quality and answer accuracy distinct feedback through separate advantages and rollout-specific loss terms.The resulting objective provides balanced training signals for visual perception and language reasoning without relying on additional reward models.

2 METHOD

Vision-SR1 decomposes VLM reasoning into self-contained visual perception and language reasoning, then optimizes their rewards separately through two rollouts and a multi-reward objective.

  • Two-rollout framework: The first rollout processes the image and query to produce visual perception, chain-of-thought reasoning, and a final answer scored against ground truth.The answer reward combines final-answer correctness with format reward and implicitly rewards the reasoning trace.
  • Two-rollout framework: The second rollout uses only the query and generated visual perception; deriving the correct answer assigns a self-visual reward for visual faithfulness.This treats visual perception as a text-only proxy for the image and avoids external reward models.
  • Multi-reward optimization: Answer and visual rewards receive separate log probabilities, advantages, and KL penalties before their gradients are combined.Decoupling prevents a sparse, entangled signal from obscuring which rollout produced the reward.
  • Multi-reward optimization: The multi-reward loss combines component-specific policy gradients and regularization to jointly optimize answer accuracy and self-contained visual reasoning.Group-wise advantages follow GRPO, while separate coefficients control answer and visual contributions.
  • Reasoning decomposition: Vision-SR1 addresses sparse visual supervision and language shortcuts by isolating visual perception from language reasoning during reinforcement learning.Existing answer-only rewards provide no direct supervision for intermediate visual reasoning, allowing language priors to dominate.
  • Computational efficiency: Two-stage rollout training is reported as 10-15% more expensive than standard GRPO without extra GPU computation.Both stages reuse the same VLM and do not load additional models or call external APIs.

3 EXPERIMENTS

Experiments evaluate Vision-SR1 across visual understanding, multimodal mathematics, hallucination detection, spatial reasoning, and language-shortcut robustness. Across these settings, Vision-SR1 improves benchmark performance and visual grounding while requiring no external reward model.

  • Evaluation: The evaluation spans general visual understanding, multimodal mathematical reasoning, visual hallucination detection, spatial reasoning, and language-shortcut robustness.Benchmarks include MMMU, MMMU-Pro, RealWorldQA, MathVerse, MATH-Vision, HallusionBench, MMSI-Bench, OmniSpatial, and ViLP.
  • Benchmark Results: 40.7 on MMMU-Pro and 52.2 on MMMU outperform Vision-R1 fair-comparison scores of 34.9 and 42.8, respectively, with the Qwen2.5VL-7B backbone.Vision-SR1 also improves average scores across 72B, 3B, and Mimi-VL-7B settings.
  • Spatial and Shortcut Robustness: Vision-SR1 generalizes to spatial reasoning and substantially improves robustness to visual-language shortcuts, with explicit visual descriptions helping avoid shortcut behavior.The method is evaluated on MMSI-Bench, OmniSpatial, and ViLP.
  • Shortcut Analysis: The Language Shortcut Rate measures correct answers paired with incorrect visual reasoning, and lower values indicate fewer language shortcuts.An LSR of 0 means every correct answer is supported by correct, self-contained visual reasoning.
  • Visual Attention Analysis: Post-training increases visual attention in early and late layers while decreasing it in middle layers, peaking at +10.2% in Layer 6 and +9.2% in Layer 20.These changes correspond to richer early visual feature extraction and later visual re-engagement during integration and output generation.

4 RELATED WORK

Related work spans post-training alignment, external visual supervision, self-rewarding reinforcement learning, and AI feedback for multimodal models. These approaches motivate Vision-SR1’s attempt to reduce reward-model dependence while explicitly evaluating visual reasoning.

  • Post-Training Alignment: Vision-language post-training commonly uses instruction tuning and reinforcement learning to improve general-purpose multimodal performance.
  • AI Feedback: AI feedback has been used as a preference-based alignment signal that can reduce object hallucination in vision-language models.
  • Self-Rewarding Reinforcement Learning: RL with verifiable rewards often depends on high-quality reward models or human feedback, creating a scalability bottleneck.Self-rewarding approaches instead use intrinsic reward signals generated by the model during reinforcement-learning post-training.
  • Multimodal Self-Rewarding: Self-generated rewards remain less developed for VLMs because visual perception makes reward definition and evaluation more complex.Existing multimodal self-rewarding work includes methods with visually constrained, step-wise self-scoring.

5 CONCLUSION AND FUTURE WORK

Vision-SR1 decomposes vision-language understanding into visual and language reasoning, using self-generated rewards and separate optimization signals. The paper reports stronger visual perception, fewer language shortcuts, and improved performance across vision-language tasks, while identifying unresolved efficiency and shortcut-learning questions.

  • Vision-SR1 decomposes vision-language understanding into visual reasoning and language reasoning components.
  • The framework uses the VLM itself to generate explicit visual-understanding rewards and Multi-Reward Policy Optimization for separate gradient attribution.
  • Vision-SR1 strengthens visual perception, reduces language shortcuts, and improves VLM performance across several vision-language task domains.
  • The proposed LSR metric indicates that perception rewards lower models’ tendency to answer through language shortcuts rather than genuine visual reasoning.
  • Future work should reduce decoded tokens by treating visual reasoning as latent thinking while retaining reward attribution to latent visual processes.
  • The paper cautions that some mathematical RL gains may reflect language-shortcut exploitation rather than genuine visual grounding, motivating further disentanglement analysis.

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

The paper used large language models only as assistive tools for grammar refinement and language clarity, with authors reviewing and verifying all outputs.

  • LLMs were used only to refine grammar and improve language clarity.
  • The authors reviewed, revised, and verified all LLM-assisted outputs and retained responsibility for the paper’s content.
  • The stated LLM use concerned writing assistance rather than generating or validating the research results.

B.1 PROMPT TEMPLATES

The appendix specifies prompt templates for visual description, reasoning, answers, caption-based self-reward, and LLM-as-a-Judge evaluation. These templates enforce structured outputs and use placeholders for task-specific inputs.

  • B.1 PROMPT TEMPLATES: The See-Think prompt template asks the model to analyze an image or video and produce a detailed self-contained description.
  • B.1 PROMPT TEMPLATES: The template then requests step-by-step reasoning with self-reflection or verification based on the visual description and image or video.
  • B.1 PROMPT TEMPLATES: The final response is a single-word or phrase answer enclosed in boxed formatting, following the description and reasoning stages.
  • B.1 PROMPT TEMPLATES: Question placeholders are used in the See-Think prompt, while description placeholders represent the actual text caption in text-only reasoning.
  • B.1 PROMPT TEMPLATES: The Caption-Reasoner self-reward template supplies a text description as the model’s input.
  • B.1 PROMPT TEMPLATES: Table 5 attributes improved general reasoning and reduced mathematical degradation to implicit text-only reasoning rewards from self-reward.
  • B.2 LLM-AS-A-JUDGE PROMPT: The LLM-as-a-Judge prompt identifies Gemini-2.5-flash as the judging model and asks it to assess candidate-answer correctness against a gold answer.
  • B.2 LLM-AS-A-JUDGE PROMPT: The judge returns its correctness judgment inside judgment tags, with placeholders for the question, reference answer, and candidate response.

B.2.1 ANALYSIS ON TEXT-ONLY REASONING

The text-only analysis evaluates whether multimodal RL preserves language reasoning across general and mathematical benchmarks. It reports mathematical degradation after multimodal RL, but better preservation and broader gains for the proposed method than Vision-R1.

  • The analysis asks whether decoupled visual and language rewards allow the two reasoning abilities to mutually reinforce one another.
  • Text-only evaluation covers MMLU-Pro, SuperGPQA, MATH-500, and GSM8K, comparing Vision-R1, the proposed method, and pre-RL baselines.
  • Multimodal RL training degrades text-only reasoning on GSM8K and MATH-500 for both Vision-R1 and the proposed method.
  • Table 6 presents an ablation comparing Vision-SR1 with a version without visual-perception self-reward.
  • Compared with Vision-R1, the proposed method better mitigates degradation on mathematical benchmarks and yields larger gains on MMLU-Pro and SuperGPQA.
  • The authors interpret separated optimization signals as helping preserve text-only competencies while retaining improvements from multimodal training.

B.2.2 ABLATION STUDY ON SELF-REWARD

The ablation removes self-visual reward and Multi-Reward Policy Optimization while retaining structured outputs, isolating the contribution of self-reward training.

  • The control model retains structured visual perception, chain-of-thought reasoning, and answer outputs.
  • It is optimized only with answer and format rewards, without self-visual reward or Multi-Reward Policy Optimization.
Loading 2508.19652v2…