PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, Cihang Xie

arXiv:2504.11468v1cs.CL

TL;DR

The paper investigates whether the standard SFT-then-RL paradigm effectively develops reasoning in LVLMs. It introduces VLAA-Thinking and evaluates SFT, RL, and combined training with mixed-reward GRPO. The results show that SFT can hinder subsequent RL, while direct RL achieves strong reasoning performance, including a 1.8% lead over the previous state of the art for 4B-scale LVLMs.

  • Problem

    The paper examines whether the widely adopted SFT-then-RL paradigm is necessary and beneficial for developing reasoning-capable LVLMs.

  • Method

    The study constructs VLAA-Thinking and compares SFT, RL, and combined training using GRPO with mixed perception and cognition rewards.

  • Results

    SFT can induce imitative pseudo-reasoning and hurt subsequent RL, while direct RL yields stronger performance; VLAA-Thinker surpasses the previous state of the art by 1.8% among 4B-scale LVLMs.

  • Takeaways & Limitations

    Reasoning appears to emerge more natively from RL, whereas SFT primarily teaches reasoning formats and can limit exploration during later RL.

  • Takeaways & Limitations

    SFT may be suitable for learning reasoning formats but falls short of fostering inherent self-reflection in multimodal reasoning.

Abstract

from arXiv · show

This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent RL by inducing ``pseudo reasoning paths'' imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged, hesitant, less informative steps, and incorrect reasoning. To systematically study this effect, we introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs. Constructed via a six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building on the Group Relative Policy Optimization (GRPO) with a novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard) among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope our findings provide valuable insights in developing reasoning-capable LVLMs and can inform future research in this area.

1 Introduction

The paper examines whether the standard SFT-then-RL recipe transfers reasoning effectively to LVLMs. Experiments find that SFT can induce imitative pseudo-reasoning that undermines later RL, whereas direct RL with mixed rewards supports more adaptive reasoning and strong benchmark performance.

  • Research questions and setup: The study tests SFT, RL, and their combination for developing reasoning-capable LVLMs using the VLAA-Thinking dataset.The dataset provides step-by-step multimodal reasoning traces for SFT and a harder split for RL, generated through a six-stage pipeline.
  • RL design: The mixed reward design combines perception and cognition signals within GRPO, with rule-based math and multiple-choice rewards especially beneficial.Combining rule-based and open-ended rewards yields the best performance in the reported ablations.
  • Findings on SFT: SFT improves standard-task performance but can produce pseudo reasoning paths with superficial self-reflective cues instead of genuine reasoning.These imitative patterns resemble expert reasoning while often containing prolonged or incorrect steps.
  • Findings on SFT: 47% relative performance drop occurs on 7B models when imitative reasoning patterns hinder genuine reasoning advancement.The figure contrasts SFT responses with pseudo traces against RL responses exhibiting native reasoning and authentic aha-moments.
  • SFT before RL: An average 12.7% performance drop follows applying SFT before GRPO on aligned models, and smaller-scale SFT produces a similar decline.Larger models also remain vulnerable, with 7B models showing nearly the same degeneration as smaller models.
  • Main outcome: VLAA-Thinker-Qwen2.5VL-3B achieves top-1 performance among 4B-scale LVLMs on the Open LMM Reasoning Leaderboard, surpassing the previous state of the art by 1.8%.The authors also report more concise and effective reasoning traces in model answers.

2 The VLAA-Thinking Dataset

VLAA-Thinking is a multimodal reasoning dataset built through captioning, reasoning distillation, rewriting, verification, and split curation. It provides separate SFT and RL subsets, with the RL subset emphasizing more challenging examples.

  • Dataset structure: VLAA-Thinking contains non-overlapping SFT and RL training sets designed for visual reasoning.The SFT split contains step-by-step reasoning grounded in visual inputs, while the RL split contains more challenging samples intended to elicit deeper reasoning.
  • Data preparation: The pipeline collects metadata from nine vision-language datasets and adds image captions and dataset-specific information before reasoning generation.GPT-4o produces detailed structured captions, while the pipeline also uses available knowledge beyond the captions.
  • Reasoning distillation: DeepSeek-R1 generates structured rationales and final answers from image captions, visual questions, and supplementary dataset information.The rationale follows a think-then-speak format enclosed by <think> and </think> tags.
  • Rewriting and filtering: GPT-3.5-turbo rewrites raw R1 outputs to remove modality-specific artifacts and enforce a clean instruction-following format.Samples with a sentence-length gap larger than 15 words are filtered out.
  • Verification: Only rewritten reasoning answers verified as correct against ground-truth answers are retained as final training data.An automated verifier compares each generated answer with the visual question’s ground truth.
  • Dataset statistics: The final dataset contains 126,413 SFT samples and 25,195 RL samples in the format {image, question, reasoning, answer}.GeoQA170K is included only in the RL split because of persistent captioning hallucination issues, using verifiable answers.

3 Investigating The Role of SFT for Multimodal Reasoning

This section examines how SFT data composition, scale, and model capacity affect multimodal reasoning. Across these settings, SFT can degrade reasoning performance, particularly when training includes aha moments or larger datasets, and larger models are not immune.

  • SFT with Aha Moments Degrades Performance: 55K examples containing aha moments reduce Qwen-2.5-VL-3B performance by an average of 10.5% versus the base model.The results suggest that SFT teaches imitation of self-reflective reasoning rather than genuine generalization.
  • SFT with Aha Moments Degrades Performance: Straightforward SFT causes average performance drops of 10.2% and 19.1% when trained on 25K and 126K multimodal reasoning samples, respectively.These results come from six reasoning benchmarks for Qwen-2.5-VL-3B.
  • More SFT Data, Worse Performance: Increasing the SFT dataset from 25K to 126K samples often harms performance, with a relative drop of over 14% across model and task settings.For example, the reported relative drops are 32.2% for 25K training and 47.0% for 126K training.
  • More SFT Data, Worse Performance: On WeMath and DynaMath, the relative decrease from larger SFT datasets reaches 97.9% over Qwen2.5-VL models on average.The 126K models also underperform untrained models by an average of 28.6% on MathVision and MathVerse across four models.
  • Larger Models Are Not Immune to SFT Degeneration: Larger models remain vulnerable: 126K-example SFT produces drops of 47.2% for smaller models and 45.4% for larger models versus base models.Qwen2.5-VL-7B additionally shows an average decline of 52.5% on the reasoning tasks.
  • Implications: SFT may teach reasoning formats but falls short of fostering inherent self-reflection, motivating investigation of more advanced methods such as RL.The section frames this as a limitation of SFT for enhancing multimodal reasoning.

4 Improving Multimodal Reasoning with Mixed Rewards

The paper adapts GRPO to multimodal reasoning with mixed perception and cognition rewards, then shows that preceding SFT can impair GRPO’s learning and exploration. Direct GRPO improves reasoning across model scales without additional SFT.

  • GRPO training: GRPO is applied to vision-language reasoning because prior results indicate it can enhance reasoning more effectively and efficiently than PPO.The objective samples grouped outputs and uses estimated advantages with a KL penalty between current, old, and reference policies.
  • Mixed reward design: The mixed reward framework combines rule-based and open-ended formats across verifiable perception and reasoning signals.Its verifiable rewards include digit, multiple-choice, math, IoU, and general reasoning signals; invalid extracted responses receive zero reward.
  • SFT before GRPO: 8.9% average performance drop occurs when SFT precedes GRPO across Qwen2VL-Base and Qwen2VL-Inst, relative to GRPO alone.The degradation affects both backbones and is larger for the instruction model, suggesting that SFT can compromise instruction-following ability relevant to GRPO.
  • SFT dataset scale: 13.5% average performance drop occurs with 5K SFT examples before GRPO, while increasing SFT to 10K yields only a 0.8% marginal improvement.Both comparisons use Qwen2.5-VL-3B and contrast SFT-plus-GRPO with GRPO alone.
  • Learning dynamics: SFT-ed models can start with higher rewards yet improve less, consistent with constrained exploration and imitative pseudo-reasoning rather than generalizable reasoning.The 10K SFT+GRPO model starts near 0.20 versus roughly 0.05 for GRPO-only, but GRPO-only later surpasses it.
  • Direct GRPO results: 36.5% average score at 7B represents a 2.0% improvement over the 34.5% base-model average after direct GRPO training.The 3B model also surpasses VLM-R1-Math by 1.1% on average, while stronger instruction tuning improves post-GRPO reasoning.
  • Emergent reflection: Aha moments emerge during GRPO without distilled reasoning paths, but their count does not directly correlate with overall reasoning performance.This supports treating reflective cues as an observed behavior rather than a sufficient performance indicator.

6 Mixed Reward ✓ ✓ ✓ ✓ ✓ 24.4 36.4 33.8

The mixed-reward ablation evaluates reward components and training hyperparameters, finding that combined reward signals perform best and KL constraints improve selected benchmarks. A qualitative example contrasts lengthy incorrect reasoning with a more concise, mostly correct trace.

  • Mixed Reward: A combination of rule-based and open-ended rewards yields a significant performance boost in the mixed-reward ablation.Table 5 evaluates MathVision, MathVerse, and WeMath.
  • Mixed Reward: 6.2% average improvement over the baseline is achieved by the Mixed Reward model across three math reasoning benchmarks.Using only rule-based rewards also appears in the ablation, but the combined design achieves the best overall performance.
  • Hyperparameters: 1.1% and 3.2% improvements on MathVerse and DynaMath, respectively, result from introducing KL constraints.The experiments compare learning-rate and KL settings on Qwen2.5-VL-3B.
  • Qualitative comparison: VLAA-Thinker-Qwen2.5VL-7B follows the correct reasoning track with only a minor late mistake, unlike the original model’s lengthy incorrect trace.The example notes that self-reflective wording in the original response does not prevent a wrong final answer.

5 Related Work

Prior vision-language reasoning methods build on text-only reasoning advances, while recent work increasingly uses structured SFT and RL to improve multimodal reasoning. Reward modeling similarly balances flexible scoring against risks such as reward hacking.

  • Vision-language reasoning models: Earlier vision-language methods such as few-shot prompting and chain-of-thought provided limited visual reasoning, while LLaVA-CoT introduced structured SFT outputs.The related-work discussion situates these methods after text-only systems such as OpenAI’s o1 and DeepSeek-R1.
  • Vision-language reasoning models: Recent vision-language systems increasingly complement or replace SFT with RL to enhance complex reasoning abilities.This trend follows the broader success of reasoning-oriented post-training in text-only models.
  • Reward modeling in reinforcement learning: Model-based rewards offer flexibility but are prone to reward hacking, whereas recent vision-language models often use binary correctness rewards for math or question-answering tasks.The contrast motivates careful reward design for reasoning-oriented reinforcement learning.

6 Conclusion

The paper compares SFT and RL for developing reasoning-capable LVLMs, concluding that SFT transfers pseudo-reasoning while GRPO-based RL yields stronger reasoning ability. It also documents prompts used across the dataset-construction pipeline.

  • The study compares SFT and GRPO-based RL for building LVLMs with strong reasoning ability.
  • Distilling reasoning data and applying SFT is described as a deficient way to transfer reasoning ability across modalities.
  • The proposed mixed reward objective for GRPO produces substantial improvement over baseline models.
  • The paper reports findings on combining SFT and GRPO and on correlations among reward, response length, and final performance.
  • The appendix shows prompts for captioning, R1 answer distillation, rewriting, and verification.

A.3 Sample Demonstration for VLAA-Thinking-SFT-126K

This section presents VLAA-Thinking-SFT-126K examples and specifies a rewriting procedure that makes minimal wording changes while preserving the original meaning.

  • The appendix provides several sample demonstrations from VLAA-Thinking-SFT-126K.
  • The rewriting task produces a nearly identical version of each text with minimal changes.
  • Rewriting replaces references to description, caption, and rationale with natural wording referring to the image.
  • The appendix includes the prompt used for answer rewriting with GPT-4-Turbo.
  • It also includes the prompt used for verification with GPT-3.5-Turbo.

B.1 Training

Training uses task-specific prompts, fixed batch and step schedules, and evaluation settings aligned with training. Experiments are evaluated across six visual-mathematical reasoning benchmarks using specified splits and metrics.

  • Training setup: Task-specific instructions for multiple-choice and short-answer tasks are appended to questions, using the system prompt shown in Figure 12.The global batch size is 128.
  • Training setup: Models train for 190 steps on 25K samples and 985 steps on 126K samples across 8 H100 GPUs.
  • Training setup: 25K SFT training on Qwen2-VL-7B sometimes produces loss spikes and model collapse, so evaluation uses a checkpoint obtained after repeated runs with a normal loss curve.
  • Evaluation setup: Evaluation uses VLMEvalKit with custom prompts disabled, a maximum of 256*32*32 pixels, and at most 800 new tokens.The training system prompt is reused for consistent training-test behavior; other toolkit hyperparameters remain default.
  • Evaluation setup: The benchmark suite covers MathVista, MathVision, MathVerse, DynaMath, WeMath, and LogicVista with their specified test splits and accuracy or strict-score metrics.MathVerse reports Vision Only accuracy, while WeMath reports Score (Strict).

C.1 Training

This section describes implementation choices and dataset-quality checks for training and evaluation, including CPU offloading and rule-based or GPT-assisted correction of dataset issues.

  • The reward model is offloaded to CPU and moved to GPU during rollouts and scoring to save GPU memory and accelerate training.
  • Dataset-specific inspection identifies formatting and answer-content issues in several datasets.
  • Rule-based filtering and GPT-assisted rewriting are used to fix identified issues and improve VL-Thinking dataset quality.
  • Models are evaluated using the identical setting described in Appendix B.2.

C.3 Case Study

The case study contrasts the backbone’s lengthy, incorrect reasoning with VLAA-Thinker’s mostly correct reasoning, while highlighting an intermediate angle-interpretation error.

  • C.3 Case Study: The geometry problem asks for angle C when angle A is 36° and AB is tangent to circle O at B.The listed answer choices include 54° as option B.
  • C.3 Case Study: Because a tangent is perpendicular to the radius, ∠OBA is 90°, enabling the triangle-angle calculation for the relevant central angle.The solution applies the tangent-radius perpendicularity and the 180° angle sum in triangle OAB.
  • C.3 Case Study: The inscribed-angle relation doubles ∠OAB to obtain ∠BOC = 108° before evaluating the remaining triangle.The cited reasoning states ∠BOC = 2 × ∠OAB = 2 × 54° = 108°.
  • C.3 Case Study: The reasoning initially risks confusing ∠OCB with the requested circumference angle, then reinterprets angle C as half the central angle and selects 54°, option B.This reinterpretation is explicitly identified as necessary before the final answer.

GeoQA

The GeoQA material presents a semicircle problem whose solution uses arc midpoint, intersecting-chords, and inscribed-angle relations, while also exposing uncertainty and an unrelated answer example.

  • GeoQA: The problem asks for tan(angle ADC) in a semicircle with diameter AB, intersecting chords AC and BD, ∠DPB = 60°, and D as the midpoint of arc BC.The task includes multiple-choice answers, though several answer-value fragments are incomplete in the passages.
  • GeoQA: The reasoning includes uncertainty about the exact point positions, arc measures, and applicability of cyclic-quadrilateral properties.Several intermediate attempts explicitly say the geometry may be tricky or that the writer is unsure.
  • GeoQA: The worked derivation obtains x = 60°, giving arcs BD = DC = 60°, arc BC = 120°, and arc AC = 60°.These values are reported after applying the arc and chord-angle relations.
  • GeoQA: Since angle ADC subtends arc AC, the derivation sets angle ADC to 30° and identifies the corresponding tangent value as option C.The passages state both the half-arc relation and the resulting 30° angle.
  • GeoQA: D being the midpoint of arc BC implies equal arcs BD and DC, while the intersecting-chords theorem relates ∠DPB to arcs DB and AC.The solution also invokes the inscribed-angle theorem to connect arc AC with angle ADC.
  • GeoQA: The supplied figures are labeled as VL-Thinking samples from GeoQA170K and Math PUMA, while another figure caption identifies a MathVerse testmini case.The captions do not provide additional quantitative comparisons.
  • GeoQA: A separate example asks how many objects remain after subtracting all balls and gives the answer as 3 objects.This example is unrelated to the semicircle derivation.
Loading 2504.11468v1…