Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models

1 Introduction

• Neural-symbolic methods rely heavily on program generation and are complex, while SFT requires high-quality CoT annotations and careful data mixing, with overfitting and limited domain-shift adaptability.Source: §1 · pp. 1–2
• Although RL offers exploration and feedback-driven optimization with limited labeled data, pure RL often lacks robustness to visual domain shifts.Source: §1 · p. 2
• Reason-RFT outperforms mainstream VLMs across three visual reasoning tasks and exceeds SFT-only and RL-only baselines under domain shift.The tasks are visual counting, structure perception, and spatial transformation.Source: §1 · p. 2
• With less than 5% of the data, Reason-RFT reaches over 90% of SFT-only performance.Source: §1 · p. 2
• Reason-RFT combines SFT-based and RL-based methods to strengthen VLM visual reasoning by leveraging their complementary strengths.Source: §1 · p. 2
• The reconstructed benchmark covers visual counting, structure perception, and spatial transformation to assess visual cognition, geometric understanding, and spatial generalization.Source: §1 · p. 3

2 Related Work

• Visual reasoning methods include program generation [36] [62] [63] and neural-symbolic approaches [32] [34] [35], alongside reasoning with VLMs.Source: §2 · p. 3
• LLaVA-CoT [37] uses multi-stage SFT with CoT prompting [64], Insight-V [65] combines SFT and RL, and DeepSeek-R1-Zero [66] applies rule-guided RL.Source: §2 · p. 3
• Post-training adapts LLMs and VLMs through SFT [71] [72] or RL [73] [74] [75] [76] [77], with RLHF [74] using human-preference feedback.Source: §2 · p. 3
• This work adapts R1 methods to VLMs and systematically compares SFT-based and RL-based paradigms for visual reasoning.Source: §2 · p. 3

3 Methodology

• Reason-RFT uses CoT-based SFT to activate visual reasoning, followed by GRPO with rule-based rewards to further develop the model’s reasoning potential.The framework is presented as a two-stage strategy for complex visual reasoning tasks.Source: §3 · p. 3

3.1 STAGE 1: SFT-based Reasoning Activation

• SFT trains on step-by-step visual reasoning data to decompose complex tasks, and the resulting πCoT model initializes the RL stage.Each sample includes an image, question, reasoning steps, and final answer; the objective maximizes the likelihood of generating the reasoning and answer given the image and question.Source: §3.1 · p. 3

3.2 STAGE 2: RL-based Reasoning Enhancement

• GRPO samples multiple actions from the initialized policy, evaluates them with verifiable rewards, and updates the policy using group-relative advantages while constraining divergence from a reference policy.The method describes group sampling as promoting diverse responses and exploration, while the KL constraint is used to maintain training stability.Source: §3.2 · p. 4

3.3 Reward Design for Visual Reasoning Tasks

• The reward design pairs a shared structured-output format reward with accuracy rewards tailored to the different visual reasoning tasks.Strict adherence to the prescribed response template earns the format reward; accuracy mechanisms are designed for discrete-valued, mathematical, and function-based problems.Source: §3.3 · pp. 4–5
• Discrete-valued rewards penalize deviations from ground truth, while mathematical rewards allow partial credit between exact-match and zero-reward tolerance thresholds.The mathematical reward gives 1 when relative error is below ϵ1 and 0 when it exceeds ϵ2; the passage gives 0.05 and 0.20 as example thresholds.Source: §3.3 · p. 5
• For spatial transformations, the accuracy reward aligns predicted and ground-truth function sequences, with weighting coefficients for partial matches.The reward description distinguishes complete, partial, and function-only matches.Source: §3.3 · p. 5

4.1 Experimental Details

• Evaluation restructures six existing datasets into counting, structure-perception, and spatial-transformation tasks, measuring accuracy according to answer type.Source: §4.1 · p. 6

4.2 Overall Evaluation of Reason-RFT Framework (RQ1)

• Under domain shift, Reason-RFT beats ANS-SFT by 10.95% (2B) and 13.93% (7B) in visual counting, and GPT-4o [95] by 31.7% in 2B spatial transformation.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• 70% of Reason-RFT-Zero’s final spatial-transformation performance is reached by 2B Reason-RFT with 3% of training data (1,600 samples).With 9% of data, the 2B model reaches 82.5%; the 7B model exceeds 92% of Reason-RFT-Zero’s performance with 3%.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3 Effect of STAGE 1 on Initialization (RQ2)

• CoT-SFT boosts Reason-RFT over Reason-RFT-Zero across tasks and scales, including 33.84% for 2B spatial transformation and 31.6% for 2B visual counting under DS-M.Source: §4.3 · pp. 7–8

4.4 Effect of STAGE 2 on Generalization (RQ3)

• 12.8% higher combined DS-D and DS-M score for 7B visual counting, and 21.04% higher average DS-L and DS-R score for 2B spatial transformation, versus CoT-SFT.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 Exploration on Reward Design (RQ4)

• Explicit visual-format tags benefit Reason-RFT-Zero more; partial credit helps ID but harms generalization, whereas penalizing partial matches improves domain-shift performance.Source: Table 2 · p. 8, §4.5 · p. 8, Table 3 · p. 8

4.6 Training Dynamics and Reasoning Behavior Analysis (RQ5)

• Reason-RFT-Zero’s reasoning length first falls, then rises and stabilizes as format reward plateaus and accuracy reward rises; early structured-trace training briefly drops ID and DS accuracy.Source: §4.6 · pp. 8–9, Figure 5 · p. 9, Figure 6 · p. 9
• On structure perception, Reason-RFT and Reason-RFT-Zero reach similar accuracy, but Reason-RFT produces longer reasoning traces that persist without length control.Source: Figure 7 · p. 9, §4.6 · p. 9

• Reason-RFT combines CoT reasoning-activation data in SFT with GRPO-based reinforcement learning to address overfitting and cognitive rigidity and improve cross-domain transfer.Source: §5 · p. 9
• The authors reconstruct a benchmark spanning visual counting, structure perception, and spatial transformation for evaluation across diverse scenarios.Source: §5 · p. 9
• Extensive experiments are reported to demonstrate the framework’s effectiveness and offer insights for visual reasoning research.Source: §5 · p. 9

NeurIPS Paper Checklist

• The authors state that the paper has no theoretical results or proofs, involves no human-subject research, and does not use LLMs in its core methodology or experiments.Source: NeurIPS Paper Checklist · pp. 18, 22, 23
• The authors mark experimental results as reproducible and direct readers to Section 4 for the supporting information.Source: NeurIPS Paper Checklist · p. 18
• For open access to data and code, the checklist directs readers to Section 1 and supplementary material without specifying access details here.Source: NeurIPS Paper Checklist · pp. 17–19
• Experiments use five training seeds, with average accuracy reported to avoid randomness.Source: NeurIPS Paper Checklist · p. 19

Appendix

• The appendix covers task evaluation and dataset reconstruction, model training details, experimental results, and CoT data generation.Source: Appendix · p. 24
• Further appendix sections detail mixed CoT datasets, pre- and post-RL CoT quality, visualizations, and limitations and societal impact.Source: Appendix · p. 24

A.1 Visual Counting

• Visual Counting tests attribute-based arithmetic in dynamic 3D-block scenes through subtraction, addition, adversarial, and multi-hop reasoning.Source: §A.1 · p. 24
• The counting benchmark refines CLEVR-Math into 35K training and 1K test samples, then adds Super-CLEVR-Math for domain-shift evaluation, with format and numerical-accuracy rewards.Source: §A.1 · pp. 24–25

A.2 Structure Perception

• Structure Perception covers geometric relations and calculations, including shape similarity, perimeter, area, angles, and transformations.Source: §A.2 · p. 25
• GeoMath-4K5 supplies 4.5K training and 820 test samples, while 800 Geometry3K samples assess domain shift; rewards support answer formats and tolerate small numerical deviations.Source: §A.2 · p. 25

A.3 Spatial Transformation

• Spatial Transformation infers single- or multi-step object-property changes from initial and final views, using five transformation functions.Source: §A.3 · p. 25
• After filtering, the spatial dataset has 60K training and 6K test samples, with center-view training and left- and right-view domain-shift sets; its reward gives credit for exact and partial function-sequence matches.Source: §A.3 · p. 26

• The experiments use Qwen2-VL-2B and Qwen2-VL-7B backbones, Open-R1 and vLLM, and servers equipped with 8×A800 GPUs.Source: B Details of Models and Training · p. 26
• Visual Counting and Spatial Transformation models train for one epoch, while Structure Perception models train for five epochs because its GeoMath training set has 4,500 samples.Source: B Details of Models and Training · p. 27
• All models first undergo CoT activation with 1,600 samples before reinforcement learning; Table 4 summarizes training-stage configurations for both model sizes and all three tasks.Source: B Details of Models and Training · p. 27, Table 4 · p. 27

Exploration on CoT Activation Data

• Reducing the in-domain share of CoT activation data lowers task performance, and more diverse visual-reasoning data may also reduce domain-specific performance.The comparison concerns Reason-RFT trained on Structure Perception with mixed domain-specific versus mixed general-domain activation data.Source: C More Experiment Results · p. 27, Table 5 · p. 27

Results on Different Backbones

• Across Qwen2.5-VL-3B and InternVL3-2B, Reason-RFT achieves strong averages across tasks, with especially large gains on domain-shifted splits.On Qwen2.5-VL-3B, it reaches T1 AVG 74.1 and T3 AVG 65.4; on InternVL3-2B, it reaches T1 AVG 74.93.Source: C More Experiment Results · pp. 27–28, Table 6 · p. 28

Evaluation on General Benchmarks

• On general benchmarks, Reason-RFT maintains or improves performance across tasks and model scales, with no observable performance degradation reported.Examples include 2B scores of 41.14 on MMMU and 75.24 on AI2D, and 7B scores of 61.31 on RealWorldQA and 50.04 on MMMU.Source: Table 7 · p. 28, C More Experiment Results · p. 28

Performance at Different Training Steps

• With 1,600 samples, Reason-RFT reaches about 70% of Reason-RFT-Zero performance, and with 9% of training data it reaches 82.5%.For the 7B model, it achieves over 92% of Reason-RFT-Zero performance on domain-shifted scenarios using 3% of training data.Source: C More Experiment Results · pp. 28–29

Automated Generation

• CoT drafts use reasoning-guided prompts, model sampling, and subtask-specific few-shot examples to encourage explicit intermediate justifications and error checking.GPT-4o and Gemini-Pro are sampled at temperature 0.7, top-k 50, and top-p 0.9.Source: D More Details on CoT Data Construction · p. 29

Automated Filtering

• Automated filters retain trajectories within a target length range and discard responses that contradict ground truth or contain inconsistent reasoning.Target length is based on a balanced mixture of human-written and model-generated samples; the 0.6 and 1.4 factors were selected in a pilot study.Source: D More Details on CoT Data Construction · p. 29

Human Verification

• A random 10% of drafts from each task undergo human review for coherence, logical validity, and alignment between reasoning and answer.The audit found critical logical flaws in about 3.8% of samples before verification and a residual error rate below 1% afterward.Source: D More Details on CoT Data Construction · p. 29

Discussion

• Generation, filtering, and targeted review together yield a CoT-SFT dataset described as more coherent and faithful while remaining scalable and reproducible.The filters address overly terse or verbose drafts and self-contradictory reasoning, while human review suppresses high-severity errors.Source: D More Details on CoT Data Construction · p. 29

Main Experiment

• The main experiments use Visual-Counting, Structure-Perception, and Spatial-Transformation datasets with 35,000, 4,500, and 60,000 samples, respectively.The datasets target quantitative analysis, structural understanding, and spatial reasoning; the overview is presented in Table 8.Source: E Detail on Mixed CoT Datasets · p. 30, Table 8 · p. 30

Ablation Studies

• The mixed general-domain ablation set combines 11 CoT-annotated datasets spanning scientific reasoning, graph topology, and pattern recognition.Examples include AI2D, ScienceQA, the GVLQA series, PuzzleVQA, IconQA, and Raven.Source: E Detail on Mixed CoT Datasets · p. 30, Table 8 · p. 30
• The mixed specific-domain ablation set focuses on geometric reasoning through GeoQA, GeomVerse, and Geometry3K, and all datasets are standardized for training compatibility.The three geometry datasets feature progressively complex problem structures.Source: E Detail on Mixed CoT Datasets · p. 30, Table 8 · p. 30

F Comparison of CoT Quality Before and After RL

• Post-RL outputs show stronger logical coherence, fewer broken or skipped inference chains, and more reflective self-checks than pre-RL outputs.One example is a pre-RL response that inferred a formula but omitted the subsequent multiplication by 2.Source: F Comparison of CoT Quality Before and… · p. 30
• The CoT comparison assesses reasoning-step count, prompting and connective usage, and answer accuracy.Step counts are obtained by automatic counting with GPT-4o; lexical categories include prompting expressions and logical connectives.Source: F Comparison of CoT Quality Before and… · pp. 30–31
• Post-RL training increases CoT step count, prompting and connective use, and final-answer accuracy on Structure Perception.The comparison uses the Qwen2VL-3B backbone on 100 randomly sampled problem instances; positive values in Table 9 indicate post-RL improvements.Source: F Comparison of CoT Quality Before and… · pp. 30–31, Table 9 · p. 30

G Visualization

• Across supplementary visualizations, Reason-RFT is reported to outperform CoT-SFT in logical consistency, reasoning quality, and correctness.The cited CoT-SFT flaws include incorrect assumptions and misinterpretations.Source: G Visualization · p. 31

Limitations

• Reason-RFT’s evaluation across computer-vision models, larger architectures, and MoE models remains future work.The authors also identify 32B/72B scaling as a future direction.Source: H Limitations and Societal Impact · p. 31
• Testing Reason-RFT in embodied AI and autonomous driving remains a future step toward assessing real-world use.The authors describe these as complex downstream scenarios requiring sophisticated visual reasoning and real-time decision-making.Source: H Limitations and Societal Impact · p. 31

Societal Impact

• The authors identify medical imaging, autonomous driving, and assistive technologies for visually impaired people as potential application areas.They state that stronger generalization and cross-domain transferability can improve AI applications in these areas.Source: H Limitations and Societal Impact · p. 31
• The reconstructed benchmark dataset is presented as supporting fair evaluation and promoting robust-AI research.Source: H Limitations and Societal Impact · p. 31
• Biases in training data and responsible deployment are ethical considerations that the authors say must be addressed to prevent misuse.Source: H Limitations and Societal Impact · p. 31