Source-linked AI summary
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, Kai-Wei Chang
TL;DR
OpenVLThinker targets the limited reasoning abilities of open-source LVLMs and the difficulty of transferring text-only reasoning because of visual-grounding problems. It alternates SFT and RL in an iterative self-improvement process, producing stronger reasoning and benchmark performance across six visual-reasoning tasks. The approach is demonstrated as a proof of concept only on a 7B model and six established benchmarks.
Problem
Open-source LVLMs rarely exhibit advanced reasoning, while naive text-only reasoning distillation can degrade performance because of imprecise visual grounding.
Method
OpenVLThinker alternates lightweight SFT and RL, using reasoning traces and progressively refined model outputs to guide later iterations.
Results
OpenVLThinker-7B consistently reaches the best or second-best scores among comparable open-source LVLMs across six benchmarks, averaging 46.6% accuracy and improving 2.7% on HallusionBench over its base model.
Takeaways & Limitations
The results provide evidence that iterative SFT-RL can support complex R1-style reasoning in multimodal models with 12K training examples and three cycles.
Takeaways & Limitations
The method was evaluated on six established benchmarks and only a 7B model, leaving robustness in other settings and scaling to larger backbones untested.
Abstract
from arXiv · showhide
We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gains on challenging visual reasoning tasks. While text-based reasoning models (e.g., Deepseek R1) show promising results in text-only tasks, distilling their reasoning into LVLMs via supervised fine-tuning (SFT) often results in performance degradation due to imprecise visual grounding. Conversely, purely reinforcement learning (RL)-based methods face a large search space, hindering the emergence of reflective behaviors in smaller models (e.g., 7B LVLMs). Surprisingly, alternating between SFT and RL ultimately results in significant performance improvements after a few iterations. Our analysis reveals that the base model rarely exhibits reasoning behaviors initially, but SFT effectively surfaces these latent actions and narrows the RL search space, accelerating the development of reasoning capabilities. Each subsequent RL stage further refines the model's reasoning skills, producing higher-quality SFT data for continued self-improvement. OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning, notably improving MathVista by 3.8%, EMMA by 2.4%, and HallusionBench by 1.6%. Beyond demonstrating the synergy between SFT and RL for complex reasoning tasks, our findings provide early evidence towards achieving R1-style reasoning in multimodal contexts. The code, model and data are held at https://github.com/yihedeng9/OpenVLThinker.
1. Introduction
OpenVLThinker addresses the difficulty of eliciting sophisticated reasoning in open-source LVLMs by alternating SFT and RL, using each stage to improve the next. This cycle eventually improves reasoning depth and answer accuracy despite an initial SFT decline.
- Open-source LVLMs rarely demonstrate advanced planning, reflection, and verification despite strong visual instruction-following abilities.
- Naively distilling text-only R1 reasoning into LVLMs causes a performance drop because image-caption conversion lacks precise visual grounding.
- OpenVLThinker-7B alternates lightweight SFT with curriculum RL, using progressively harder reasoning data generated from prior iterations.
- Despite an initial SFT performance drop, iterative SFT-RL eventually improves reasoning depth and answer accuracy.
- SFT surfaces reasoning actions such as “first,” “wait,” and “check,” while RL refines reasoning and supports the next SFT iteration.
- The model consistently improves across six challenging benchmarks, including mathematical reasoning and hallucination evaluation.
2. Related Work
Prior work established strong text-based reasoning and explored visual reasoning through prompting, fine-tuning, distillation, and RL. OpenVLThinker is situated among recent efforts to transfer R1-style reasoning into LVLMs.
- DeepSeek-R1 and related work showed that RL with verifiable rewards can promote advanced chain-of-thought reasoning and reflection in open-weight language models.
- Recent LVLM research has elicited chain-of-thought reasoning through prompting and fine-tuning, while visual reasoning models remain an active area of development.
- Concurrent studies explore R1-like LVLM reasoning through text-only distillation or direct RL-based self-exploration.
- The related-work landscape is moving from early exploration toward more effective complex vision-language reasoning.
3. Preliminaries
This section introduces language-model generation, supervised fine-tuning, and reinforcement-learning objectives used as preliminaries for OpenVLThinker.
- An LLM models responses as token sequences sampled from a conditional distribution factorized across successive output tokens.The model is parameterized by weights θ and conditions each token on the prompt and preceding response tokens.
- Supervised Fine-Tuning (SFT): SFT trains a model on prompt-response pairs by maximizing the likelihood of desired responses.Minimizing the associated loss aligns generated responses with labeled examples.
- Reinforcement Learning (RL): RL fine-tunes language models using preference relationships between preferred and dispreferred responses under a Bradley-Terry formulation.The preference probability uses a sigmoid function and a reward-model objective.
- Reinforcement Learning (RL): GRPO samples groups of response trajectories from an old policy and optimizes a clipped objective with KL regularization.The normalized group-level reward serves as the advantage signal, while β controls the KL penalty.
4. OpenVLThinker: Iterative Self-improvement on Curriculum Data
OpenVLThinker improves complex vision-language reasoning by analyzing reasoning actions and iterating between SFT and curriculum RL. SFT surfaces useful behaviors, while later model-generated demonstrations support progressive self-improvement.
- 4.1. The Role of SFT and RL: The approach studies how SFT and RL affect reasoning-related keywords before introducing iterative SFT-RL cycles and curriculum RL.The section frames keyword analysis as motivation for the proposed training pipeline.
- Initial SFT Data: SFT-Iter1 data uses image captions as proxies for visual inputs to obtain text-based reasoning chains, selecting the shortest correct chain for distillation.The target model generates detailed captions, and QwQ-32B produces candidate chains from those captions.
- Impact of SFT and RL on Model Reasoning Actions: The base model rarely exhibits planning, reflection, or alternative-solution actions, whereas SFT guided by R1-like traces surfaces these behaviors.The analysis identifies representative keywords for perception, question comprehension, planning, reflection, and alternatives.
- Impact of SFT and RL on Model Reasoning Actions: 5.2%: GRPO after SFT improves MathVista performance while largely preserving the initial reasoning-action distribution.The reported refinement includes less repetitive reflection despite the limited distributional change.
- Impact of SFT and RL on Model Reasoning Actions: Direct RL without prior SFT struggles to induce reflection keywords efficiently, supporting SFT’s role in highlighting desirable reasoning actions.Equivalent training volume can still leave reflection behavior absent without the preceding SFT step.
- Iterative Self-improvement: GRPO-Iter1 improves complex reasoning and visual-input reliability, making it a stronger source of reasoning demonstrations for iterative self-improvement.Each iteration samples enhanced traces from the preceding model and retrains the base model, producing successive performance gains.
- Curriculum RL: Curriculum RL evaluates data-source difficulty so exploration remains challenging but appropriate for the model’s proficiency.GPT-4o rates representative examples from sources including FigureQA, MapQA, and GeoQA.
5. Experiments
Experiments evaluate OpenVLThinker across six visual reasoning benchmarks and analyze its data curriculum, iterative progression, and restart strategy. The model achieves strong cross-benchmark performance, while harder data and restarting iterations improve generalization.
- Evaluation: Evaluation covers visual mathematics, general reasoning, and perception across six established benchmarks.The benchmarks include MathVista, MathVerse, MathVision, MMMU-Pro, EMMA, and HallusionBench.
- Main results: OpenVLThinker-7B reaches 46.6% average accuracy, improving 2% over the base model and matching proprietary-model performance.It also improves HallusionBench accuracy by 2.7% over the base model while using one-tenth the data of concurrent single-iteration approaches.
- Main results: OpenVLThinker-3B achieves the best MathVista performance among compared 3B models and surpasses state-of-the-art 3B reasoning models.The comparison includes Qwen2.5-VL-3B and VLAA-Thinker-3B.
- Analysis: Filtering repetitive reflection keywords during initial distillation mitigates repetitive reasoning after SFT.The adopted strategy truncates overly repetitive reflection segments while preserving the final answer.
- Analysis: Medium-to-hard curriculum RL yields larger gains on difficult benchmarks such as MathVision without significantly changing easier-benchmark performance such as MathVista.Starting with medium-difficulty data and progressing to hard data outperforms training solely on hard data.
- Analysis: Reasoning length increases most at Iteration 1, then becomes progressively more concise without excessive repetition.The reasoning model uses more inference-time words than the base non-reasoning model.
- Analysis: Restarting from the base model at each iteration improves stability, limits overfitting, and supports transfer to unseen tasks.Continued training degraded HallusionBench performance, suggesting overfitting to the previous iteration’s data.
6. Conclusion
The paper frames reasoning as inference-time actions surfaced by SFT and refined by RL, and presents OpenVLThinker-7B as an iterative multimodal reasoning model. Across six benchmarks, three SFT-RL cycles and 12K examples raise average accuracy to 46.6%, while evaluation remains limited in breadth and scale.
- Conclusion: OpenVLThinker interprets reasoning as inference-time actions, with SFT surfacing actions and RL improving them through iterative self-improvement.The paper highlights tokens such as “wait” as markers of reasoning behavior.
- Conclusion: Three SFT-RL cycles and 12K training examples raise six-benchmark average accuracy to 46.6%, a 2% absolute gain over the base model.The result is reported as on par with proprietary systems such as GPT-4o.
- Limitations: The evaluation covers six benchmarks and validates the method only on a 7B model, leaving broader robustness and scaling unexplored.Testing larger backbones would require substantially greater computational resources.
A.1. Additional evaluation benchmarks.
Additional evaluation clarifies benchmark coverage across established and newer multimodal tasks. EMMA results are broken down by subject, while MM-Star and WeMath extend the evaluation beyond the main six benchmarks.
- Additional evaluation benchmarks: The main evaluation spans six vision-language benchmarks covering mathematical reasoning, general reasoning, and perceptual reliability.The setup matches benchmarks used in recent proprietary and open-source model reports.
- Additional evaluation benchmarks: EMMA results are reported separately for Math, Chemistry, Physics, and Code subsets.This provides subject-level coverage within the general reasoning evaluation.
- Additional evaluation benchmarks: The expanded evaluation adds MM-Star and WeMath as two recent benchmarks.MM-Star assesses fine-grained perception, mathematics, science and technology, and logical reasoning among other capabilities.
A.2. Computational Cost of the Iterative SFT–RL Loop
The iterative SFT-RL loop uses modest SFT data, staged RL, and practical academic-scale compute. Comparisons with single-stage baselines indicate that iterative refinement, rather than merely combining SFT and RL, drives the strongest performance.
- Computational cost: Hard-stage RL has the highest cost, but total compute remains comparable to contemporary RL-based post-training methods.Preceding SFT and medium RL accelerate convergence during the final RL stage.
- Computational cost: The RL-only baseline requires approximately 16 GPU-hours on an 8×H100 node, comparable to OpenVLThinker’s cumulative training time.The experiments use an 8×H100 or equivalent GPU node.
- Single-stage baselines: OpenVLThinker consistently outperforms the single-stage SFT-only and RL-only baselines under comparable training conditions.This supports improvement from iterative refinement rather than simply combining SFT and RL.
A.4. Impact of Caption Quality on Iterative Training
Caption quality materially shapes the iterative training process: stronger captions improve visual grounding, reasoning-trace precision, and the quality of data available for later iterations.
- Higher-quality captions increased the likelihood of correct reasoning traces, expanding the effective training pool.This effect was observed when iteration-1 data were filtered by final-answer correctness.
- Caption quality was compared across training iterations on MathVista using weaker and stronger caption generators under identical k=4 rejection sampling.
- Better caption quality improves visual grounding and reasoning-trace precision in early stages, producing higher-quality data for subsequent iterations.
B. Additional Empirical Study
Additional experiments show that complex reasoning benefits visual question answering when captions are sufficiently detailed, while a single caption-refinement pass provides limited gains.
- R1-style reasoning outperforms standard methods when models receive identical captioned inputs.DeepSeek-R1-Distill-14B achieved higher accuracy than Qwen2.5-14B-Instruct and could match or surpass its captioning model’s direct QA accuracy.
- Increasing sampled reasoning chains to k=2 or k=4 benefits R1 models more than standard Qwen models.The results indicate that multi-step reasoning can better converge on correct solutions when multiple hypotheses are explored.
- Richer and more precise captions significantly improve the effectiveness of complex reasoning for visual tasks.The study varied caption generators, sampled reasoning paths, and included direct image-based QA as a baseline.
- More detailed captions, such as those from GPT-4o, make the gains from complex reasoning especially pronounced.
- A single caption-feedback pass did not produce major accuracy improvements, suggesting that repeated or more elaborate refinement may be necessary.The corresponding table reports a slight pass@4 increase while pass@1 and pass@2 remain largely unchanged.
- The experiments support complex multi-step reasoning as a contributor to improved VQA accuracy relative to standard instruction-tuned reasoning.
C. Experiment Details
The paper documents its training and inference setup, including SFT and GRPO resources, hyperparameters, caption prompts, and qualitative output examples illustrating correct and incorrect reasoning.
- Implementation: The experiments used open-source LLaMA-Factory and EasyR1 frameworks for SFT and GRPO.
- Implementation: Each iteration required about 30 minutes for SFT, 20 hours for RL, and 8 hours for verified distillation-data generation on comparable H100 80GB GPU clusters.
- Implementation: The appendix provides inference, SFT, and GRPO hyperparameters together with prompts for generating image captions.
- Output examples: Qualitative examples compare repetitive or incorrect SFT outputs with later GRPO outputs on visual reasoning questions.
- Output examples: The examples include derivative comparison, angle measurement, route identification, and food-web reasoning tasks.
- Output examples: The output examples also show alternative reasoning paths for rectangle-angle and sign-direction questions.