Source-linked AI summary
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, Jinqiao Wang
TL;DR
LVLM preference optimization is limited by the cost of high-quality human-annotated preference data and robust reward models. Vision-R1 uses vision-guided, rule-based reinforcement learning with criterion-driven rewards and progressive refinement, yielding consistent gains across in-distribution and out-of-distribution tasks, including up to 50% improvement and results surpassing a state-of-the-art 10x larger model.
Problem
Constructing high-quality vision-language preference datasets and reliable reward models for LVLM post-training remains resource-intensive and challenging.
Method
Vision-R1 applies vision-guided R1-like reinforcement learning using criterion-driven multidimensional rewards and progressive rule refinement on curated instruction data, without specialized reward models or handcrafted preference datasets.
Results
Vision-R1 produces consistent gains across in-domain and out-of-domain benchmarks, with up to 50% improvement and performance surpassing the state-of-the-art Qwen2.5-VL-72B model.
Takeaways & Limitations
Vision-R1 provides a human-annotating-free approach that improves LVLM object localization across diverse models and scenarios while retaining advanced QA capabilities.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) typically follow a two-stage training paradigm-pretraining and supervised fine-tuning. Recently, preference optimization, derived from the language domain, has emerged as an effective post-training reinforcement strategy to enhance capabilities of LVLMs. However, constructing high-quality human-annotated preference data and developing robust reward models to mimic these preferences are both costly and challenging. Motivated by this observation, we propose Vision-R1, a novel vision-guided R1-like reinforcement learning algorithm for LVLMs that rewards models with definitive vision feedback. It only leverages curated instruction data, eliminating the need for specialized reward models and handcrafted preference datasets. We incorporate a criterion-driven reward function that further integrates multi-dimensional feedback to evaluate model completions comprehensively based on the vision task logic. Furthermore, we introduce a progressive rule refinement strategy that dynamically adjusts the reward criteria during training, enabling continuous model improvement and mitigating reward hacking. Extensive experiments on both in-distribution and out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with Vision-R1 achieves consistent performance gains, with even up to 50% improvement and surpassing the state-of-the-art 10x size model.
1. Introduction
Vision-R1 addresses costly preference-data and reward-model requirements in LVLM post-training with vision-guided reinforcement learning using curated instruction data. Its criterion-driven rewards and progressive rule refinement improve performance across localization and QA evaluations, including unseen scenarios.
- LVLM preference optimization remains costly because high-quality vision-language preference datasets and reliable reward models are difficult to construct.
- Vision-R1 uses curated instruction data and vision feedback to eliminate specialized reward models and handcrafted preference datasets.Its criterion-driven reward function evaluates completions with multidimensional task-specific signals, including box-coordinate precision.
- Progressive rule refinement dynamically adjusts reward criteria during training to support continuous improvement and mitigate reward hacking.
- Vision-R1 improves diverse tasks, including wild visual grounding and dense object detection, and surpasses the Qwen2.5-VL-72B model.
- 6% average improvement on unseen scenarios is achieved compared with SFT while maintaining advanced question-answering capabilities.
2. Related Works
Related work shows that LVLMs have advanced substantially but remain challenged by demanding object-localization tasks. Reinforcement-learning alignment methods have therefore extended language-model preference optimization approaches toward vision-language systems.
- LVLMs have expanded capabilities in question answering and reasoning, but complex and dense object detection still trails specialized vision models.
- Reinforcement learning methods for LVLMs seek improved alignment with human preferences, including RLHF for reducing hallucinations and DPO for simplifying training.
3. Vision-R1
Vision-R1 extends rule-based GRPO to LVLMs by using vision-grounded, criterion-driven rewards instead of reward models or handcrafted preference data. Its reward design addresses formatting, recall, and localization precision, while progressive refinement tightens criteria during training.
- Vision-R1: Vision-R1 extends GRPO to vision-language learning and avoids specialized reward models and manually annotated preference data.GRPO computes relative advantages across groups of completions, while Vision-R1 bases rewards on definitive visual feedback.
- Criteria-Driven Reward Function: Object localization motivates three reward targets: formatting correctness, identifying all relevant objects, and accurate bounding boxes.The framework analyzes long-sequence formatting errors, insufficient valid predictions, and difficult-object localization failures.
- Criteria-Driven Reward Function: The criteria-driven function converts textual coordinate outputs into visual feedback and evaluates predictions using dual-format, recall, and precision rewards.Box-prioritized matching associates predicted instances with coordinates, category labels, and IoU scores before reward calculation.
- Criteria-Driven Reward Function: The overall completion reward sums the three criteria-based rewards to comprehensively assess each output against visual task requirements.Dual-format reward requires both template and coordinate-content validity, while recall and precision measure coverage and box quality.
- Progressive Rule Refinement Strategy: Progressive rule refinement dynamically changes reward criteria during training to increase reward contrast, support continuous improvement, and mitigate reward hacking.Differentiation penalizes low recall and IoU while granting full rewards above thresholds; staged progression tightens thresholds from 0.5, 0.5, 0.75 to 0.75, 0.75, 0.9.
4. Experiments
Experiments evaluate Vision-R1 with two 7B LVLMs across in-domain and out-of-domain localization, general QA, and ablation settings. Results report stronger localization and generalization, while progressive reward adjustment affects outcomes differently across models.
- Experimental Setup: Vision-R1 is evaluated with Qwen2.5-VL-7B and Griffon-G-7B across object localization, general QA, and diverse benchmarks.The experiments include in-domain and out-of-domain datasets, with ablations analyzing matcher, reward-function, progressive-refinement, and QA-generalization design choices.
- In-domain Object Localization: Vision-R1 improves Griffon-G by 1.8 on COCO and 2.5 average mAP on ODINW-13.These improvements outperform Qwen2.5-VL-72B on ODINW-13 and bring Griffon-G-7B closer to specialized vision models.
- In-domain Object Localization: Vision-R1 improves Qwen2.5-VL-7B by 8.9 points on COCO and 8.7 points on ODINW, surpassing its 72B counterpart.Compared with supervised fine-tuning, Vision-R1 averages 1.25 and 7 points higher on the two models, respectively.
- Out-of-domain Object Localization: Vision-R1 gains 7.1 and 4.8 on out-of-domain localization for Griffon-G-7B and Qwen2.5-VL-7B, respectively.The setting treats either the object category or scene as absent from post-training, and Vision-R1 surpasses expert models on BoggleBoards and MountainDewCommercial.
- Ablation Studies: Box-only matching performs slightly better than matching boxes with category correctness, with limited overall difference between approaches.The authors attribute this to strong LVLM classification and greater object recall when matching solely by bounding boxes.
- Ablation Studies: STEP = 1/2 gives Griffon-G-7B the best COCO performance, while STEP = 1 falls below baseline without progressive reward adjustment.Without adjustment, Griffon-G reaches AR100 of 56.7 but produces lower-quality boxes and more false positives; the strategy suppresses these boxes.
5. Conclusion
The conclusion presents Vision-R1 as a human-annotation-free reinforcement learning method for LVLM object localization. It combines vision-based criteria rewards with progressive rule refinement and shows generalized effectiveness across diverse benchmarks.
- Conclusion: Vision-R1 combines a vision criterion-driven reward function with progressive rule refinement to improve LVLM object localization.The approach uses subjective and definite responses in abundant instruction data without human annotation.
- Conclusion: Comprehensive evaluation across diverse benchmarks and scenarios demonstrates the generalized effectiveness of Vision-R1.The stated scope is precise object localization for complex tasks and real-life applications.
Supplementary Material
The supplementary material provides additional methodological and experimental details. It covers dataset construction, task templates, evaluation data, detailed results, and further analysis of progressive adjustment.
- Supplementary Material: The supplementary material details dataset construction, task templates, evaluation data, and experimental results.It also analyzes the progressive adjustment strategy in greater depth.
1. Training Data Construction
Vision-R1 is trained without human preference data using curated localization instruction data with precise answer annotations. The resulting dataset balances diverse and challenging detection, grounding, and referring-expression examples.
- Training Data Construction: Vision-R1 trains directly on question-answer pairs with precise annotations instead of human-annotated preference data.Samples are curated from open-source localization instruction datasets according to diversity and challenge.
- Object Detection: Detection data comes from MS COCO, including difficult images defined as those containing more than 10 instances.One-third of the sampled COCO data is drawn from difficult and easy samples each according to the raw-data distribution.
- Visual Grounding: Visual grounding uses ODINW and V3Det data covering over 13K categories, with 5K and 4K samples collected from the two datasets.The datasets are converted into a consistent visual-grounding format, with some multi-category V3Det annotations retained.
- Referring Expression Comprehension: Referring-expression comprehension uses 5K RefCOCO samples plus multi-object expressions extracted from Visual Genome.The construction also considers cases where one referring expression corresponds to multiple objects through GRefCOCO-style examples.
- Training Data Construction: The selection process is intended to provide a balanced training foundation for challenging object localization tasks.The stated foundation combines detection, grounding, and referring-expression data.
2. Task Templates
The paper creates instruction templates for object localization tasks by following selected models’ fine-tuning formats and adapting related formats where needed.
- Five instruction templates were created for model training and evaluation.The templates are summarized in Table 8.
3. Detailed Results
Vision-R1 improves both evaluated models across most ODINW-13 datasets by large margins, demonstrating effectiveness on this benchmark.
- Vision-R1 improves Griffon-G-7B and Qwen2.5-VL-7B on most ODINW-13 datasets by a large margin.The detailed results for all 13 datasets are listed in Table 9.
4. Additional Ablation Studies
Additional analyses examine progressive rule refinement, task templates, generalization QA, and qualitative evidence. The results show that refinement settings should match model capability, while localization improves without heavily compromising general QA.
- Progressive Rule Refinement Strategy: Progressive rule refinement prevents reward hacking and supports continuous performance improvement.For Griffon-G-7B, increasing recall without better box quality can leave precision unchanged or slightly lower; refinement assigns full rewards to high-quality boxes, improving mAP.
- Task Templates: The study uses five instruction templates covering training and evaluation for each model and task.The templates are summarized in Table 8.
- Progressive Rule Refinement Strategy: For weaker Qwen2.5-VL-7B, STEP = 1 yields the best performance because midway criterion changes reduce training efficiency and performance.All tested settings still outperform the baseline.
- Progressive Rule Refinement Strategy: The STEP hyperparameter adapts progressive rule refinement to models with different capability levels.Stronger models can use progressive adjustment, whereas weaker models may need later transitions or STEP = 1.
- General VQAs: The method significantly enhances object localization without heavily compromising Qwen2.5-VL-7B’s general QA abilities.The result is reported in an ablation of generalization QA capabilities under the same VLMEvalKit setting.
5. Qualitative Analysis
Qualitative analysis shows that Vision-R1 improves Qwen2.5-VL-7B’s detection outputs by reducing redundancy and missed detections while producing more precise localization.
- Vision-R1 eliminates redundant and invalid outputs, improves recall, and produces more precise bounding-box localization.The original model generated redundant outputs, missed detections, and imprecise boxes in detection tasks.