Source-linked AI summary
Visual-RFT: Visual Reinforcement Fine-Tuning
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang
TL;DR
Prior reinforcement fine-tuning with verifiable rewards showed promise for data-scarce language reasoning, but its use for visual perception and reasoning in LVLMs remained limited. Visual-RFT adapts this paradigm with multiple generated reasoning responses and task-specific visual rewards, achieving improvements over SFT across visual tasks and strong generalization with minimal data.
Problem
Reinforcement-learning reasoning research has largely remained language-focused, with limited exploration of improving LVLM reasoning and visual perception in multi-modal settings.
Method
Visual-RFT generates multiple LVLM responses with reasoning and final answers, then applies task-specific rule-based verifiable rewards with policy optimization such as GRPO.
Results
Visual-RFT outperforms SFT across fine-grained classification, few-shot detection, reasoning grounding, and open-vocabulary detection, with improvements reported under minimal-data settings.
Takeaways & Limitations
Visual-RFT demonstrates the potential of verifiable-reward reinforcement learning to enhance LVLM visual perception, grounding, reasoning, and generalization with limited data.
Abstract
from arXiv · showhide
Reinforcement Fine-Tuning (RFT) in Large Reasoning Models like OpenAI o1 learns from feedback on its answers, which is especially useful in applications when fine-tuning data is scarce. Recent open-source work like DeepSeek-R1 demonstrates that reinforcement learning with verifiable reward is one key direction in reproducing o1. While the R1-style model has demonstrated success in language models, its application in multi-modal domains remains under-explored. This work introduces Visual Reinforcement Fine-Tuning (Visual-RFT), which further extends the application areas of RFT on visual tasks. Specifically, Visual-RFT first uses Large Vision-Language Models (LVLMs) to generate multiple responses containing reasoning tokens and final answers for each input, and then uses our proposed visual perception verifiable reward functions to update the model via the policy optimization algorithm such as Group Relative Policy Optimization (GRPO). We design different verifiable reward functions for different perception tasks, such as the Intersection over Union (IoU) reward for object detection. Experimental results on fine-grained image classification, few-shot object detection, reasoning grounding, as well as open-vocabulary object detection benchmarks show the competitive performance and advanced generalization ability of Visual-RFT compared with Supervised Fine-tuning (SFT). For example, Visual-RFT improves accuracy by $24.3\%$ over the baseline in one-shot fine-grained image classification with around 100 samples. In few-shot object detection, Visual-RFT also exceeds the baseline by $21.9$ on COCO's two-shot setting and $15.4$ on LVIS. Our Visual-RFT represents a paradigm shift in fine-tuning LVLMs, offering a data-efficient, reward-driven approach that enhances reasoning and adaptability for domain-specific tasks.
1. Introduction
Visual-RFT extends verifiable-reward reinforcement fine-tuning from language reasoning to visual perception, targeting data-scarce settings. It uses task-specific rewards and shows improvements across classification, detection, grounding, and open-vocabulary recognition.
- RFT evaluates response correctness and learns through trial and error, making it suitable for domains with scarce fine-tuning data.
- Visual-RFT generates multiple LVLM responses containing reasoning tokens and final answers, then uses task-specific verifiable rewards to guide policy optimization.The approach includes rewards such as IoU for object detection and uses algorithms such as GRPO.
- 24.3% accuracy improvement over the baseline is achieved in one-shot fine-grained image classification with around 100 samples, while SFT drops 4.3%.
- Visual-RFT improves open-vocabulary detection mAP from 9.8 to 31.3 on new COCO classes and from 2.7 to 20.7 on selected rare LVIS classes.The reported evaluations also cover few-shot object detection and reasoning grounding.
- The paper introduces Visual-RFT, designs low-cost task-specific rewards, and evaluates it across four visual perception task families against SFT.The evaluated tasks are fine-grained classification, few-shot object detection, reasoning grounding, and open-vocabulary object detection.
- Visual-RFT releases its training code, training data, and evaluation scripts openly to support further research.
2. Related Work
Prior work established LVLMs through visual-text integration and post-training, while reinforcement-learning research largely remained focused on language reasoning. Multi-modal RL for improving LVLM reasoning and visual perception was comparatively under-explored.
- LVLMs integrate visual and textual data to understand complex multi-modal inputs, with training generally consisting of pre-training and post-training.Post-training includes supervised fine-tuning and reinforcement learning.
- Table 1 lists separate detection and classification prompts used to construct the dataset.
- DeepSeek-R1-style reinforcement-learning reasoning research has largely remained in the language domain, leaving limited exploration in multi-modal settings.
3. Methodology
Visual-RFT adapts reinforcement learning with verifiable rewards to visual perception by generating multiple multimodal responses and optimizing them with task-specific rewards. Its method combines group-based response comparison, KL-constrained policy updates, structured reasoning outputs, and perception-specific reward components.
- Reinforcement Learning with Verifiable Rewards: RLVR evaluates model responses with direct verification functions rather than a trained reward model for objectively verifiable outcomes.The reward function takes the question and output pair as inputs and checks whether the prediction matches the ground-truth answer.
- Visual-RFT Framework: GRPO compares groups of candidate responses directly, normalizing their rewards to estimate relative answer quality without requiring a separate critic model.Responses with higher within-group rewards receive greater preference during policy optimization.
- Visual-RFT Framework: Visual-RFT generates multiple reasoning-and-answer trajectories for each image-question input, scores them with verifiable rewards, and updates the policy using optimization such as GRPO.KL divergence limits the difference between the optimized policy and the reference model during training.
- Qualitative Effects: Qualitative results report that thinking improves fine-grained image classification and reasoning grounding ability with Visual-RFT.The grounding results are presented on LISA, and the classification figure associates the thinking process with higher image classification performance.
- Task-Specific Verifiable Rewards: Detection reward combines average IoU, confidence, and format components to evaluate predicted bounding boxes against ground-truth boxes.The method sorts boxes by confidence, matches them using an IoU threshold, and assigns zero IoU to unmatched boxes.
- Task-Specific Verifiable Rewards: Classification reward combines accuracy and format terms, while structured <think> and <answer> tags guide the model to produce reasoning before its final answer.Correct classification receives an accuracy reward of 1, whereas incorrect classification receives 0.
4. Experiments
Experiments evaluate Visual-RFT across few-shot classification, object detection, reasoning grounding, and open-vocabulary detection, generally comparing it with SFT under limited-data settings. Across these tasks, Visual-RFT improves performance and demonstrates transfer to difficult or novel categories.
- Experimental Setup: Visual-RFT is evaluated on fine-grained classification, few-shot detection, reasoning grounding, and open-vocabulary detection using limited training data.The experiments include Qwen2-VL models, four fine-grained classification datasets, COCO and LVIS detection settings, LISA grounding, and the out-of-domain MG dataset.
- Few-Shot Classification: +24.3% accuracy is achieved by Visual-RFT in one-shot fine-grained classification, while SFT declines by -4.3%.Under 4-shot classification, Visual-RFT achieves an average improvement of 25.9, while SFT remains slightly below baseline.
- Reasoning Grounding: Visual-RFT significantly improves reasoning grounding over SFT on the LISA dataset using 239 training images.Qualitative results indicate that the model’s thinking process improves reasoning and grounding accuracy.
- Few-Shot Object Detection: Visual-RFT consistently outperforms SFT in few-shot object detection on COCO and six rare LVIS categories.As training data increases on COCO, SFT reaches approximately 31 average mAP, while reinforcement fine-tuning approaches 47.
- Out-of-Domain Evaluation: Visual-RFT surpasses SFT on the out-of-domain MG dataset under 4-shot and 16-shot settings.The MG dataset contains abstract anime-style monster-girl categories intended to increase recognition and reasoning difficulty.
5. Conclusion
The paper introduces Visual-RFT to adapt GRPO-based reinforcement learning with rule-based verifiable rewards to visual perception and grounding in LVLMs. Across several visual tasks, it reports improvements over SFT with minimal data and strong generalization.
- Conclusion: Visual-RFT adapts GRPO-based reinforcement learning to improve LVLM visual perception and grounding.The method uses rule-based verifiable rewards tailored to visual tasks.
- Conclusion: Rule-based verifiable rewards reduce manual labeling and simplify reward computation for visual-task fine-tuning.The paper describes reward computation as efficient and low cost.
- Conclusion: Visual-RFT outperforms SFT with minimal data across fine-grained classification, open-vocabulary detection, reasoning grounding, and few-shot learning.The reported experiments also show strong generalization across these visual perception tasks.