PaperlayerView exact revision on arXiv

Source-linked AI summary

SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward

Kaixuan Fan, Kaituo Feng, Haoming Lyu, Dongzhan Zhou, Xiangyu Yue

arXiv:2505.17018v2cs.CV

TL;DR

Rule-based reinforcement learning can elicit strong MLLM reasoning but does not supervise the thinking process, allowing sub-optimal reasoning strategies. SophiaVL-R1 adds holistic thinking rewards with Trust-GRPO reliability weighting and annealing, and it consistently outperforms existing MLLMs across benchmarks, including a smaller model surpassing a much larger baseline.

  • Problem

    Outcome rewards supervise final correctness but not the thinking process, so models may learn sub-optimal reasoning strategies that hinder generalization.

  • Method

    SophiaVL-R1 combines a holistic thinking reward model with Trust-GRPO reliability weighting and time-based annealing alongside rule-based outcome rewards.

  • Results

    SophiaVL-R1 consistently outperforms existing MLLMs across multiple benchmarks, and SophiaVL-R1-7B outperforms LLaVA-OneVision-72B on most benchmarks.

  • Takeaways & Limitations

    The results support adding supervision for thinking processes beyond final correctness when developing multimodal reasoning models.

Abstract

from arXiv · show

Recent advances have shown success in eliciting strong reasoning abilities in multimodal large language models (MLLMs) through rule-based reinforcement learning (RL) with outcome rewards. However, this paradigm typically lacks supervision over the thinking process leading to the final outcome. As a result, the model may learn sub-optimal reasoning strategies, which can hinder its generalization ability. In light of this, we propose SophiaVL-R1, as an attempt to add reward signals for the thinking process in this paradigm. To achieve this, we first train a thinking reward model that evaluates the quality of the entire thinking process. Given that the thinking reward may be unreliable for certain samples due to reward hacking, we propose the Trust-GRPO method, which assigns a trustworthiness weight to the thinking reward during training. This weight is computed based on the thinking reward comparison of responses leading to correct answers versus incorrect answers, helping to mitigate the impact of potentially unreliable thinking rewards. Moreover, we design an annealing training strategy that gradually reduces the thinking reward over time, allowing the model to rely more on the accurate rule-based outcome reward in later training stages. Experiments show that our SophiaVL-R1 surpasses a series of reasoning MLLMs on various benchmarks (e.g., MathVisita, MMMU), demonstrating strong reasoning and generalization capabilities. Notably, our SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite the latter having 10 times more parameters. All code, models, and datasets are made publicly available at https://github.com/kxfan2002/SophiaVL-R1.

1 INTRODUCTION

The paper argues that outcome-only rewards can reinforce correct answers produced by flawed reasoning, limiting generalization. SophiaVL-R1 adds holistic thinking rewards, Trust-GRPO reliability weighting, and achieves strong benchmark performance.

  • Motivation: Outcome-only rewards can encourage correct answers from flawed reasoning, allowing sub-optimal strategies that generalize poorly.The paper motivates explicit supervision of the thinking process during GRPO training.
  • Method: SophiaVL-R1 evaluates entire reasoning processes with a thinking reward model covering logical soundness, cross-step consistency, and redundancy.The model is trained on annotated reasoning responses collected from GRPO trajectories.
  • Method: Trust-GRPO weights thinking rewards according to their reliability across responses producing correct versus incorrect answers.This weighting is designed to reduce reward hacking without extra computational overhead.
  • Results: SophiaVL-R1-7B consistently outperforms existing MLLMs across diverse benchmarks, including MathVista and MMMU.The reported results emphasize both reasoning and generalization abilities.

2 RELATED WORK

MLLM reasoning research has progressed from fine-grained process supervision and learned reward models toward rule-based reinforcement learning. DeepSeek-R1 showed that rule-based rewards can elicit strong reasoning without dense supervision.

  • Process supervision: Process supervision provides feedback at intermediate reasoning steps through mechanisms such as step-wise Process Reward Models.These approaches have been applied notably to mathematical reasoning.
  • MLLM reasoning: MLLM reasoning targets models that understand and reason across multiple modalities on complex tasks.Earlier approaches commonly used step-level supervision or learned reward models.
  • Rule-based reinforcement learning: DeepSeek-R1 demonstrated that reinforcement learning with rule-based rewards can incentivize strong reasoning without dense supervision.This result motivates extending the paradigm to multimodal reasoning.

3 METHOD

SophiaVL-R1 combines holistic thinking rewards with rule-based outcome rewards in Trust-GRPO, using group-level trust estimation and time-based annealing to limit unreliable reward influence.

  • Dataset composition: SophiaVL-R1-130k combines public text-only and multimodal datasets across five categories for reasoning and general vision-language tasks.The dataset supports both the thinking reward model and SophiaVL-R1 training.
  • Thinking reward: The thinking reward model scores intermediate reasoning quality from 0 to 1 independently of final-answer correctness.It is trained on filtered and balanced annotations from GRPO-generated responses, producing SophiaVL-R1-Thinking-156k.
  • Reward construction: The combined reward adds the trust-weighted thinking reward to the rule-based outcome reward, with α controlling the thinking reward’s impact.The outcome reward is derived from task-specific comparisons with ground-truth answers.
  • Trust-GRPO: Trust-GRPO groups responses by outcome correctness and compares their average thinking rewards to estimate each group’s trustworthiness.A lower γ indicates misalignment between thinking and outcome rewards, reducing the thinking reward’s influence without extra computational overhead.
  • Annealing: Time-based annealing gradually reduces thinking-reward influence as training progresses, increasing reliance on the more accurate outcome reward.The decay depends on the current global step and total training steps.
  • Training behavior: Trust-GRPO contrasts correct and incorrect responses to improve reward reliability and encourage more generalizable reasoning behavior.A training analysis reports that direct reward combination without trust weighting and annealing performs worse.

4 EXPERIMENT

SophiaVL-R1 is evaluated on mathematical and general multimodal reasoning benchmarks, plus reward-model quality. It achieves competitive benchmark performance and its 3B thinking reward model performs strongly on VLRewardBench.

  • The evaluation covers MathVista and MathVerse for multimodal mathematics, and MMMU, MME, MMStar, ChartQA, and MMBench for general multimodal capabilities.
  • 71.3% accuracy on MathVista surpasses Qwen2.5-VL-7B-Instruct models trained with GRPO or SFT+GRPO and LLaVA-OneVision-72B.
  • 18.1 points on MathVerse separates SophiaVL-R1-7B from VisualPRM, with scores of 48.8 versus 30.7 and improvements across all subtasks.
  • 4.5 points on MMMU separate SophiaVL-R1-7B from LLaVA-OneVision-72B, while SophiaVL-R1-7B remains strong across general benchmarks.
  • The 3B thinking reward model achieves higher VLRewardBench performance despite fewer parameters and strongly detects hallucinations.

5 ABLATION STUDY

Ablations show that the trained thinking reward model, trustworthiness weighting, and time-based annealing each contribute to SophiaVL-R1’s performance and training behavior.

  • The ablation compares removing the trained thinking reward model, removing trustworthiness weighting and annealing, removing only trustworthiness weighting, and using Qwen2.5-VL-7B with GRPO.
  • Effect of the Thinking Reward Model: The untrained reward-model variant consistently underperforms SophiaVL-R1 and performs comparably to Qwen2.5-VL-7B+GRPO.
  • Effect of the Trustworthiness Weight: Removing the trustworthiness weight causes performance drops across all benchmarks relative to full SophiaVL-R1.
  • Effect of the Time-based Annealing Strategy: The variant without trustworthiness weighting and annealing generally performs worse than the variant retaining annealing, consistent with over-exploitation of potentially unreliable thinking rewards.
  • Training Curve Analysis: SophiaVL-R1 reaches the highest mean outcome reward and improves faster during training, while directly combining rewards performs worse.

6 CONCLUSION

The paper presents SophiaVL-R1 as an MLLM that combines holistic thinking-process supervision with rule-based outcome rewards through Trust-GRPO. Its experiments support thinking supervision, trustworthiness weighting, and annealing as useful design elements for reasoning models.

  • SophiaVL-R1 integrates model-generated thinking rewards with rule-based outcome rewards through the Trust-GRPO algorithm.
  • A holistic-level thinking reward model assesses reasoning quality, while trustworthiness weighting and time-based annealing address reward hacking and reduce thinking-reward influence over training.
  • Experiments across multiple benchmarks show that SophiaVL-R1 consistently outperforms existing multimodal language models.
  • The findings highlight the value of supervising thinking processes beyond final correctness for developing reasoning models.

B TRAINING DETAILS

The section points to the paper’s training hyperparameters in Table 6.

  • Table 6 contains the training hyperparameters.

C ERROR PATTERNS OBSERVED IN GRPO TRAINING

GRPO training exhibited recurring reasoning errors, including misinterpreting visual conditions, extracting incorrect relationships, and performing faulty calculations. The experiments examine Trust-GRPO’s trustworthiness formulation and annealing schedule as key algorithmic choices.

  • Error patterns: The model confused edge-length information with angle values, producing incorrect reasoning in the Error Identification pattern.
  • Error patterns: Logical Soundness errors involved incorrectly extracting angle relationships and performing faulty equation calculations.
  • Error patterns: The section presents additional examples of error patterns observed during GRPO training.
  • Algorithmic design choices: The experiments examine Trust-GRPO’s trustworthiness-weight formulation and annealing schedule for thinking rewards.

E.1 AVERAGE REWARD-BASED TRUSTWORTHINESS WEIGHT DESIGN

The proposed average reward-based trustworthiness weight estimates reward reliability without extra computation, while annealing balances thinking and outcome rewards during training. Experiments report that exponential decay is slightly better than linear decay, though both improve performance over the instruct baseline.

  • Trustworthiness weight: The average reward-based trustworthiness weight estimates reliability without introducing additional computation.The design is tailored to GRPO, where MLLM training and inference are costly.
  • Trustworthiness weight: The variance-based alternative underperforms the original average reward-based method and incurs additional computation.The comparison is reported on MathVista (Math) and MMBench (General).
  • Annealing schedule: The thinking reward is gradually reduced because it is especially useful early, whereas outcome rewards become more reliable as training progresses.
  • Annealing schedule: Both exponential and linear decay improve performance relative to the instruct baseline, with exponential decay yielding slightly better experimental results.The comparable gains from linear decay suggest that gradual reduction matters more than the precise functional form.

F EVALUATION DETAILS

The evaluation uses VLMEvalKit with recommended package versions, leaderboard metrics for baselines, default prompts, and model-specific answer extraction. Benchmark results use specified splits and subsets for MathVista, MathVerse, MMMU, ChartQA, and MMBench.

  • Evaluation protocol: Most evaluations use VLMEvalKit, while baseline metrics come from the OpenVLM leaderboard.Default prompts are used, and answer extraction is adapted to each model’s output format.
  • Evaluation datasets: MathVista uses testmini, MMMU uses mmmu dev val, ChartQA uses test, and MMBench uses MMBench Dev EN.
  • Evaluation datasets: MathVerse performance averages vision-only, vision-dominant, vision-intensive, text-dominant, and text-lite subsets.

G CASE STUDY OF γ

Case studies show that thinking rewards can mis-rank reasoning quality, including rewarding less complete reasoning more highly than a correct response. Trust-GRPO uses γ to down-weight suspicious signals and stabilize reinforcement learning.

  • Case studies: The case studies include text-only mathematical, free-form, MathVista, MME, MMMU, and MMStar reasoning examples.
  • Text-only mathematical case: In a text-only mathematical case, Trust-GRPO identifies unreliable thinking rewards by comparing responses with correct and incorrect answers.The method adapts γ to downscale unreliable reward influence during training.
  • Free-form reasoning case: A free-form example shows RESPONSE[8] identifying eight key points versus six for RESPONSE[6], yet receiving thinking reward 0.0 versus 0.8.
  • Free-form reasoning case: The mismatch demonstrates how γ dynamically identifies unreliable thinking rewards and adaptively down-weights suspicious signals.The reported consequence is more stable reinforcement learning and mitigation of reward hacking.
Loading 2505.17018v2…