PaperlayerView exact revision on arXiv

Source-linked AI summary

SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

Mingyuan Wu, Jingcheng Yang, Shengyi Qian, Xudong Wang, Jize Jiang, Qifan Wang, Aashu Singh, Khoi Pham, Fei Liu, Zhaolun Su, Zhuokai Zhao, Klara Nahrstedt, Jianyu Wang, Hanchao Yu

arXiv:2607.10966v1cs.AI

TL;DR

Multimodal self-rethinking remains challenging and underexplored, motivating SVR-R1, which integrates same-model binary self-verification and rethinking into GRPO training. Across multimodal reasoning benchmarks, it significantly outperforms standard GRPO while using fewer verification rounds during training and maintaining improved accuracy.

  • Problem

    Effective self-rethinking remains challenging and underexplored for multimodal reasoning, despite evidence that models can sometimes self-verify their outputs.

  • Method

    SVR-R1 interleaves same-model answer generation, binary self-verification, and conditional rethinking within multi-turn GRPO rollouts, optimizing only the final outcome-based reward.

  • Results

    SVR-R1 significantly outperforms standard GRPO across multimodal reasoning benchmarks while models perform fewer verification rounds during training and maintain improved accuracy.

  • Takeaways & Limitations

    SVR-R1 shows that VLMs can leverage inherent verification capabilities for self-improvement within the RL training loop without external data or model supervision.

Abstract

from arXiv · show

We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source \textbf{SVR-R1} to facilitate future research in VLMs.

1 Introduction

SVR-R1 explores a middle ground between prompting and RL fine-tuning by integrating a VLM’s self-verification into multi-turn reasoning. Across multimodal and general reasoning tasks, it improves performance while training leads the policy to use fewer verification rounds, advancing self-improvement without external supervision.

  • Method: SVR-R1 integrates model self-verification rounds into VLM reasoning, prompting a rethink when the tentative answer is judged wrong and finalizing it when judged correct.The verification rounds share the same model copy, placing self-verification within the reasoning process.
  • Motivation: The framework targets an underexplored middle ground between explicit prompting for self-rethinking and reinforcement-learning fine-tuning.Prior work found that prompting models to second-guess answers could improve inference reasoning, while multimodal self-rethinking remained challenging and underexplored.
  • Results: SVR-R1 significantly improves vision-language reasoning across challenging multimodal table and chart benchmarks and general reasoning tasks.The reported evaluation spans multiple multimodal table and chart reasoning benchmarks as well as general reasoning tasks.
  • Results: During training, the policy gradually performs fewer verification rounds, eventually almost always affirming its answer immediately in a single round.This training dynamic accompanies the reported improvements in vision-language reasoning performance.
  • Contribution: SVR-R1 advances self-improvement without external supervision by integrating self-generated reasoning traces into an iterative reinforcement-learning pipeline.Earlier self-improvement approaches primarily used high-confidence self-generated traces for supervised fine-tuning rather than fully integrating them into iterative RL.

2 Related Work

Related work shows RL evolving from preference alignment toward efficient training, reasoning, and self-correction, while self-improvement has progressed from prompting and trace-based fine-tuning into vision-language settings. However, multimodal self-improvement remains challenging because of constrained settings, multimodal pretraining difficulties, and limited high-quality reasoning data.

  • Reinforcement learning for language models: RL entered mainstream LLM development through RLHF, which uses human-preference reward models with PPO, followed by more computationally efficient PPO variants.RLHF optimizes the LLM policy via Proximal Policy Optimization (PPO).
  • Reinforcement learning for language models: RL has improved LLM reasoning and self-correction, with RL fine-tuning studies reporting emergent behaviors such as “aha moments.”The passage frames these behaviors as intrinsic properties enabling effective self-improvement.
  • Self-improvement methods: Earlier self-improvement methods used generated feedback and response revision, prompt-level re-verification, or next-token fine-tuning on filtered reasoning traces.These approaches operated at inference or prompt level, or trained on selected high-quality responses.
  • Vision-language self-improvement: Vision-language self-improvement studies have expanded recently but remain constrained, partly because multimodal pretraining is difficult and high-quality reasoning data is scarce.The passage contrasts this data limitation with the pure language setting.

3 Method

SVR-R1 trains a VLM to interleave self-generation with binary self-verification, regenerating answers judged incorrect until verification succeeds or a turn limit is reached. Its multi-turn GRPO objective optimizes the finalized response while masking verification tokens and constraining updates against a frozen reference model.

  • Self-verification and generation: SVR-R1 interleaves generation and verification with shared weights but distinct prompts, repeating the process until the maximum number of rounds is reached.The verifier and generator are the same VLM policy copy, despite receiving different textual instructions.
  • Self-verification and generation: The verifier emits only Yes or No; No triggers a prompted rethink and regenerated answer, while Yes or the turn limit finalizes the response for accuracy or reward.Regeneration recursively preserves prior conversational context and adds a rethink trigger.
  • Multi-turn RL optimization: SVR-R1 optimizes multi-turn rollouts with the policy itself serving as verifier and applies a KL penalty against the frozen reference model.The objective uses trainable policy πθ, frozen reference model πref, reward function rϕ, and coefficient β > 0.
  • Multi-turn RL optimization: Verification tokens are masked from the RL loss so training focuses on final generation and avoids conflicting optimization dynamics between verification and generation.The method masks self-verification tokens such as Yes or No during loss calculation to stabilize updates.
  • GRPO with self-verification: SVR-R1 builds on GRPO, which updates the policy from group-relative advantages computed by normalizing outcome rewards across sampled rollouts without a learned value function.The formulation uses Monte Carlo response groups, clipping, and peer-relative improvement to encourage diverse reasoning while maintaining stability.

4 Experiment

Experiments evaluate SVR-R1 on challenging multimodal table and chart reasoning tasks using GRPO-based multi-turn fine-tuning. SVR-R1 consistently outperforms standard GRPO, accelerates learning, and achieves robust gains while reducing verification turns and entropy.

  • Implementation: SVR-R1 is implemented with GRPO in VeRL using Qwen-VL 2.5 at 3B and 7B scales for multimodal, multi-turn RL fine-tuning.Experiments use identical datasets and hyperparameters for standard GRPO and SVR-R1.
  • Main Results: SVR-R1 consistently outperforms baseline GRPO methods at both 3B and 7B scales on challenging visual table and chart reasoning tasks.The comparison uses the same amount of data, while larger ThinkLite-trained models improve performance on more general reasoning tasks.
  • Training Dynamics: Self-verification accelerates learning under the same data budget and improves final saturated performance compared with standard GRPO.Chart reasoning shows a more stable curve with more data, whereas table reasoning has a noisier curve because its training set contains only several hundred samples.
  • Training Dynamics: Verification turns gradually decrease during training and converge to approximately 2, corresponding to one generation step followed by a YES verifier response.The models become increasingly confident in their initial answers and tend to affirm them.
  • Entropy Analysis: SVR-R1 produces relatively lower entropy than standard GRPO, while increasing entropy does not improve results in the evaluated tasks.The study uses an entropy-controlled technique from DAPO to test whether higher entropy could improve chart-task performance.
  • Self-Reflection: When the self-verifier returns NO, a rethinking trigger prompts explicit self-reflection that can correct an initially wrong answer.An appendix validation example demonstrates correction in a cat breed identification task.

5 Conclusion · A Appendix

SVR-R1 integrates self-verification rounds into GRPO training to bootstrap VLM reasoning, outperforming standard GRPO on multimodal benchmarks with the same data and hyperparameters. During training, models use fewer verification rounds while maintaining improved accuracy, indicating that they learn to produce initial answers that pass self-verification.

  • 5 Conclusion: SVR-R1 integrates self-verification rounds into GRPO training to bootstrap VLM reasoning capabilities.The framework uses models’ inherent verification capabilities within the reinforcement-learning loop.
  • 5 Conclusion: SVR-R1 significantly outperforms standard GRPO on multimodal reasoning benchmarks using exactly the same data and hyperparameter setup.The comparison isolates the contribution of the self-verification training approach.
  • 5 Conclusion: The framework advances multimodal reasoning through self-verification-based reinforcement learning.This conclusion follows from the reported benchmark gains and training dynamics.
  • 5 Conclusion: Models gradually perform fewer verification rounds during training while maintaining improved accuracy.The observed training dynamic combines reduced verification frequency with continued accuracy gains.
  • 5 Conclusion: The training dynamics suggest that models learn to close the generation-verification gap.They increasingly produce initial answers that pass self-verification.
  • 5 Conclusion: SVR-R1 demonstrates that models can leverage inherent verification capabilities for self-improvement within the RL training loop.Self-verification functions as part of the training process rather than requiring an external verification mechanism.

A.1 Qualitative Example · A.2 Full Prompt for VLM Rollout and Inference · A.3 Full Prompt for Reward Judge

The appendix provides a qualitative self-reflection example, specifies the prompts governing SVR-R1 rollout and inference, and describes binary LLM judging for semi-open visual question answering. Together, these materials detail the example format, prompt requirements, and reward-evaluation procedure.

  • A.1 Qualitative Example: A.1 Qualitative Example illustrates a model identifying a pictured cat as a domestic shorthair.The response explains that domestic shorthairs can have varied coat patterns and colors.
  • A.2 Full Prompt for VLM Rollout and Inference: A.2 Full Prompt for VLM Rollout and Inference uses carefully designed requirements, explicit formatting instructions, and a one-shot example with detailed reasoning steps.The prompt structure is intended to enhance multimodal reasoning with chain-of-thoughts, drawing inspiration from ReFocus.
  • A.2 Full Prompt for VLM Rollout and Inference: A.2 Full Prompt for VLM Rollout and Inference defines separate self-verifier and self-generator turns, including a rethinking trigger and previous queries and responses.These turn structures are depicted in Figures 8 and 9.
  • A.2 Full Prompt for VLM Rollout and Inference: A.2 Full Prompt for VLM Rollout and Inference instructs the model to answer table-figure questions using information from the provided image.The initial prompt addresses a table figure, an image, and a user question.
  • A.2 Full Prompt for VLM Rollout and Inference: A.2 Full Prompt for VLM Rollout and Inference requires generated actions to resolve the user request perfectly and directs the model to act when it has an answer.The requirements state that the request is reasonable and solvable and encourage the model to try its best.
  • A.3 Full Prompt for Reward Judge: A.3 Full Prompt for Reward Judge uses an outcome-based binary reward and an LLM judge to compare final predictions with ground-truth answers on complex semi-open visual question answering tasks.The judge is instructed to output a binary decision based solely on the prediction and ground-truth answer; SVR-R1 uses gpt-oss-120b.
  • A.3 Full Prompt for Reward Judge: A.3 Full Prompt for Reward Judge presents GPT-OSS judgments with accompanying step-by-step reasoning.This behavior is illustrated in Figure 12.

B ThinkLite-VL · B.1 Full Prompt for VLM Rollout and Inference

The section extends evaluation beyond table and chart tasks to general multimodal reasoning using ThinkLite-VL-70K, while specifying a standardized prompt for both VLM inference and RL rollout. The prompt separates reasoning from the final answer and presents that answer in a boxed layout.

  • B ThinkLite-VL: ThinkLite-VL-70K broadens evaluation beyond specific table and chart tasks to general multimodal reasoning.The dataset includes multimodal reasoning, natural image understanding, and chart interpretation tasks.
  • B ThinkLite-VL: The dataset covers multimodal reasoning tasks drawn from prior multimodal reasoning work.The cited task sources include Lu et al. (2021), Chen et al. (2022), and Seo et al. (2015).
  • B ThinkLite-VL: ThinkLite-VL-70K also includes natural image understanding tasks.The cited sources include Kahou et al. (2018), Lu et al. (2022a), and Marino et al. (2019).
  • B ThinkLite-VL: Chart interpretation is another task category represented in ThinkLite-VL-70K.The cited sources include Lu et al. (2022b; 2023).
  • B.1 Full Prompt for VLM Rollout and Inference: For fair comparison with Wang et al. (2025b), the study uses the same prompt for VLM inference and RL rollout.This follows the standard RL-VLM setup shown in Figure 13.
  • B.1 Full Prompt for VLM Rollout and Inference: The prompt format clearly separates reasoning steps from the final answer.This formatting is used during both inference and rollout in RL training.
  • B.1 Full Prompt for VLM Rollout and Inference: The final answer is presented in a boxed layout.The boxed answer format is part of the standardized VLM inference and rollout prompt.

B.2 Experiment

The experiment uses a 70K multimodal training split spanning geometry, image, science, visual-question, chart, and table understanding tasks, alongside an 11K difficult subset. Training follows the Thinking-VL/EasyR1 setup with Qwen-2.5-VL 7B on 8×8 A100 GPUs and specified AdamW optimization settings.

  • Dataset Split: The 70K training split combines Geometry3K, GeoQA, GEOS, FigureQA, ScienceQA, OK-VQA, IconQA, and TabMWP examples across several multimodal reasoning domains.The data cover math questions with image input, image understanding, and chart understanding.
  • Difficult Data Selection: An 11K high-difficulty subset is selected from the full 70K dataset using MCTS to improve sample efficiency and test accuracy relative to the 70K split.The difficulty-based selection follows Wang et al. (2025b).
  • Experiment Setup: The experiment strictly follows Wang et al. (2025b)'s Thinking-VL hyperparameters, which adhere to the EasyR1 default RL-VLM setup.This establishes the configuration used for the experiment.
  • Experiment Setup: Training uses AdamW with an initial learning rate of 1 × 10−6, micro-batch size 4 per GPU, mini-batch size 128 per update, and overall batch size 512.These are the reported optimization and batching settings.
  • Experiment Setup: Qwen-2.5-VL 7B is trained on 8×8 A100 GPUs with 80GB memory per GPU using bf16 precision.The passage specifies the model, hardware, and numerical precision.

B.3 Supplementary Findings

SVR-R1 does not improve reasoning accuracy on the MCTS-selected 11K difficult split, where repeated self-verification appears unproductive because answers may remain unreachable despite unlimited rethinking.

  • Training limitations: SVR-R1 shows no improvement in reasoning accuracy when trained on the MCTS-selected 11K difficult split.This contrasts with training on the 70K split, as noted in the passage.
  • Training limitations: Verification turns grow dramatically during RL training on the 11K difficult split, despite the lack of accuracy gains.The passage attributes this pattern to repeatedly rethinking questions whose correct answers may remain unreachable even with unlimited self-verification turns.

B.4 Implementation Details

The implementation uses AdamW with a 1 × 10−6 initial learning rate, small per-GPU micro-batches, and task-specific mini-batches. Training uses 8×8 A100 GPUs with 80GB memory, bf16 precision, and temperature 1.0 for exploratory rollouts.

  • Optimization and batching: AdamW training uses an initial learning rate of 1 × 10−6, micro-batch size 2 per GPU, and mini-batch sizes of 256 for charts or 128 for tables.The setup accommodates image inputs and prompts up to 16k tokens.
  • Hardware and rollout decoding: The 3B and 7B models train on 8×8 A100 GPUs with 80GB memory using bf16 precision and decoding temperature 1.0 during rollouts.The temperature is set to 1.0 to encourage exploration.

B.5 Computation

SVR-R1 reuses the same model parameters for generation and verification, avoiding GPU weight transfers and adding only 10% wall-clock time when properly configured.

  • Limited Computational Overhead: 10% wall-clock time is the additional computational overhead when generation and verification reuse the same model parameters and the framework is properly configured.Parameter reuse eliminates the need to move weights to or from the GPU.
Loading 2607.10966v1…