PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, Tiancheng Zhao

arXiv:2504.07615v2cs.CVcs.CL

TL;DR

Visual understanding tasks often provide precise annotations that could support stable rule-based reinforcement-learning rewards, but applying R1-style RL to VLMs requires a suitable framework. VLM-R1 supplies that framework and evaluates it on REC and OVD, where RL improves performance and out-of-domain generalization over SFT while revealing reward-design and scaling insights.

  • Problem

    Visual understanding tasks have precise ground-truth annotations suitable for rule-based rewards, motivating investigation of whether R1-style reinforcement learning can improve VLM reasoning.

  • Method

    VLM-R1 is a unified, extensible framework that applies GRPO-based reinforcement learning to VLMs across flexible data, model, reward, and training configurations.

  • Results

    RL improves performance and out-of-domain generalization over SFT on REC and OVD, including 63.16 versus 54.82 on LISA-Grounding for 3B REC models.

  • Takeaways & Limitations

    Applying RL to VLMs requires careful reward engineering, training-data selection, and attention to model scale, especially for complex visual tasks.

  • Takeaways & Limitations

    VLM-R1 currently supports only the GRPO algorithm, with additional reinforcement-learning algorithms planned for future work.

Abstract

from arXiv · show

Recently DeepSeek R1 has shown that reinforcement learning (RL) can substantially improve the reasoning capabilities of Large Language Models (LLMs) through a simple yet effective design. The core of R1 lies in its rule-based reward formulation, which leverages tasks with deterministic ground-truth answers to enable precise and stable reward computation. In the visual domain, we similarly observe that a wide range of visual understanding tasks are inherently equipped with well-defined ground-truth annotations. This property makes them naturally compatible with rule-based reward mechanisms. Motivated by this observation, we investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs), aiming to enhance their visual reasoning capabilities. To this end, we develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks. Using this framework, we further explore the feasibility of applying RL to visual domain. Experimental results indicate that the RL-based model not only delivers competitive performance on visual understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in generalization ability. Furthermore, we conduct comprehensive ablation studies that uncover a series of noteworthy insights, including the presence of reward hacking in object detection, the emergence of the "OD aha moment", the impact of training data quality, and the scaling behavior of RL across different model sizes. Through these analyses, we aim to deepen the understanding of how reinforcement learning enhances the capabilities of vision-language models, and we hope our findings and open-source contributions will support continued progress in the vision-language RL community. Our code and model are available at https://github.com/om-ai-lab/VLM-R1

1 Zhejiang University 2 Om AI Research 3 Binjiang Institute of Zhejiang University

Figure 1 presents VLM-R1 as a standard reinforcement-learning pipeline for enhancing base vision-language models.

  • VLM-R1 uses a standard pipeline to enhance base vision-language models with reinforcement learning.

1. Introduction

The paper extends R1-style reinforcement learning to vision-language models because visual tasks often provide precise ground-truth annotations suitable for stable, rule-based rewards. VLM-R1 applies this approach to REC and OVD, where RL improves performance and out-of-domain generalization over SFT, while ablations examine reward design, data quality, and model scale.

  • Rule-based rewards are stable and interpretable for visual tasks with deterministic ground-truth annotations, such as IoU-based REC rewards.
  • VLM-R1 is an extensible framework for applying reinforcement learning to general vision-language tasks with flexible configurations and scalable training.
  • The study evaluates RL on REC and OVD, which share bounding-box outputs but differ in complexity from single-target localization to multiple-target detection.
  • RL improves visual understanding and out-of-domain generalization over SFT, including 63.16 versus 54.82 on LISA-Grounding for 3B REC models.
  • For OVD, the 3B RL model reaches 21.1 AP on COCO and 31.01 nms-AP on OVDEval, exceeding the cited SFT and 7B baselines.
  • Ablations identify reward hacking in object detection, an “OD aha moment,” training-data quality effects, and model-size-dependent RL behavior.

2. Related Work

Prior work applies R1-style methods to VLMs, especially multimodal mathematics and visual perception. VLM-R1 distinguishes itself by studying visual understanding more broadly and analyzing reward engineering and training-data selection beyond SFT-versus-RL comparisons.

  • Earlier VLM research advanced visual dialogue, reasoning, and instruction-data training, while image-input resolution remained a limitation.
  • R1-OneVision builds a cross-modal reasoning pipeline and dataset before reinforcement learning, whereas R1-V introduces GRPO into VLM training.
  • Most related studies target multimodal mathematics, while Visual-RFT applies reinforcement learning to visual perception tasks.
  • VLM-R1 goes beyond SFT-versus-RL comparisons by analyzing reward engineering and careful training-data selection.

3. VLM-R1 Framework

VLM-R1 extends Open-R1 into a modular GRPO pipeline for vision-language models. Its preparation and training stages support custom rewards, multiple model and training configurations, and standardized interfaces across VLM architectures.

  • VLM-R1 extends the Open-R1 framework to the vision-language domain.
  • 3. VLM-R1 Framework: Figure 2 depicts a two-stage pipeline in which grpo jsonl.py prepares data and rewards, while grpo trainer.py manages GRPO training.
  • 3. VLM-R1 Framework: The framework currently focuses exclusively on GRPO, with additional reinforcement-learning algorithms planned for future work.
  • 3. VLM-R1 Framework: The preparation stage supports custom reward functions, task-tailored data, and modular integration of various VLMs.
  • 3. VLM-R1 Framework: Training supports LoRA fine-tuning, vision-tower freezing, and full-parameter training configurations.
  • 3. VLM-R1 Framework: VLM-R1 combines GRPO support with flexible reward design, model selection, and optimization strategies for vision-language reinforcement-learning research.
  • 3.2. VLM Module: The VLM Module standardizes model-specific operations so GRPOTrainer can interact with different VLMs without handling separate implementations.

4. Reward Design

VLM-R1 integrates GRPO with task-specific accuracy and format rewards for REC and OVD, using shared bounding-box outputs while accommodating their differing complexity. GRPO compares sampled responses through normalized rewards, and OVD additionally penalizes redundant predictions.

  • Task selection: REC and OVD are selected because they share bounding-box outputs but differ in complexity, enabling RL analysis across tasks.REC predicts one query-conditioned box, whereas OVD predicts boxes and class labels for every queried target.
  • GRPO abstraction: GRPO samples candidate responses, evaluates them with rewards, and updates the policy toward responses with higher within-group advantages.Advantages are computed from rewards normalized by their group statistics, without requiring a separate critic model.
  • Reward components: VLM-R1 combines accuracy and format rewards for the REC and OVD tasks.The framework integrates these reward types into GRPO for visual-language optimization.
  • REC reward: REC accuracy rewards bounding boxes by their IoU with the ground-truth box, while format rewards return 1 or 0 for compliance with the required tagged JSON structure.The REC output must contain a bounding box in the specified <answer> format.
  • OVD reward: OVD accuracy rewards predicted box-label combinations using mAP and applies a penalty factor to redundant predictions; its format reward checks markdown-style JSON compliance.The prediction list contains bounding boxes and corresponding class labels, and the redundancy penalty was reported helpful for OVD performance.

5.1. Implementation details

Implementation uses Qwen2.5VL models with GRPO and SFT training settings, alongside task-specific prompts and evaluation distinctions between in-domain and out-of-domain REC data. Out-of-domain REC requires recognizing object roles using open-world knowledge before localization.

  • Models: Qwen2.5VL-3B-Instruct is the base model, with 7B and 32B variants used to study model-size effects.The base model was selected for its expected vision-language potential under reinforcement learning.
  • Hyperparameters: REC RL training uses GRPO with N=8, temperature 0.9, one iteration, β=0.04, two epochs, and learning rate 1e-6; OVD changes β to 0.The learning rate is 1e-6 for both RL and SFT, while other OVD settings remain identical.
  • Prompts: The REC prompt asks for bounding-box coordinates of the region described by a query.This prompt directly targets localization of the queried region.
  • Evaluation data: Out-of-domain REC requires recognizing the role of a soccer goalkeeper using open-world knowledge before locating it, unlike in-domain attribute-based descriptions.In-domain examples describe spatial or appearance attributes of the object.
  • Prompts: The OVD prompt requests detected target boxes and labels in JSON, with None specified when no targets appear.Each detected target is represented by bounding-box coordinates and its target name.
  • Output format: The shared instruction format places the reasoning process in <think> tags and the final answer in <answer> tags.This structure supports the required answer formatting during task prompting.

5.2. Main results

VLM-R1 evaluates reinforcement learning against supervised fine-tuning on referring expression comprehension and open-vocabulary detection. Across in-domain and especially challenging out-of-domain settings, RL shows stronger and more balanced generalization.

  • Evaluation setup: VLM-R1 compares RL and SFT using Qwen2.5VL-3B-Instruct trained on RefCOCO/+/g and evaluates both in-domain and out-of-domain performance.
  • Referring expression comprehension: RL consistently gains on in-domain referring expression comprehension, while SFT shows limited improvement across training steps.
  • Referring expression comprehension: RL maintains stable and superior out-of-domain performance as training progresses, whereas SFT slightly degrades on the challenging LISA-Grounding evaluation.LISA-Grounding requires fine-grained perception, precise understanding of referring expressions, and relational reasoning.
  • Open-vocabulary object detection: On OVDEval, RL outperforms SFT in 7 of 9 detection categories, including Position (+9.2 points), Relationship (+8.4 points), and Negation (+3.3 points).
  • Open-vocabulary object detection: RL improves average nms-ap by 4.51 points over SFT, reaching 31.01 versus 26.50, while providing more comprehensive gains across visual tasks.

5.3. Ablations & Extended experiments

Ablations examine reward design, training data, and model scale. They identify reward hacking under native detection rewards, an emergent two-step detection behavior with odLength, stronger results from semantically richer data, and scale-dependent RL effects.

  • Investigation about reward hacking: Native AP rewards can induce overlong, redundant object predictions, revealing reward hacking in object detection.The model may exploit reward-function loopholes by enumerating excessive categories or duplicated boxes without fulfilling the intended detection objective.
  • Investigation about reward hacking: The odLength reward produces shorter outputs and suppresses unnecessary predictions compared with native AP50 and mAP rewards.
  • OD aha moment: With odLength, the model exhibits an “OD aha moment”: it first reasons about object presence and then predicts accurate bounding boxes.Without odLength, duplicated correct and incorrect detections can still receive relatively high native mAP rewards.
  • Training data quality: D3-trained models significantly outperform COCO-trained models, including on the in-domain COCOfiltered evaluation set.The authors hypothesize that D3’s semantically richer, sentence-level queries encourage more robust reasoning chains than simple COCO labels.
  • RL effects across model scales: RL boosts the Relation sub-task across all tested model sizes, with scores changing 13.1→21.5, 16.2→20.1, and 20.6→27.0.
  • RL effects across model scales: Negation improves for 7B and 32B models but declines slightly for 3B, consistent with the authors’ view that RL reinforces existing reasoning patterns.Reported changes are 39.0→43.1, 47.0→51.7, and 38.7→37.7, respectively.

6. Discussion

RL improves VLM generalization across REC and OVD, but the more complex OVD task requires task-specific reward and data optimization. The experiments also show that reward design and training-data quality shape whether reasoning behaviors emerge.

  • RL transfers reasoning patterns from non-reasoning REC training data to challenging out-of-domain settings.
  • RL models outperform SFT counterparts in most subtasks on the complex OVDEval benchmark, with benefits across nearly all model sizes.
  • Reward engineering: Naive mAP rewards cause OVD reward hacking through excessive predictions, while odLength suppresses redundancy and supports object-presence reasoning before box prediction.
  • Training and evaluation data: Complex training samples can elicit reasoning behaviors, whereas low-quality or overly simple data may hinder learning and reduce generalization.
  • Training and evaluation data: LISA-Grounding and OVDEval are selected because they probe semantic understanding and generalization in challenging real-world scenarios.
  • Task-specific optimization: Applying RL to OVD requires customized rewards and more appropriate training data, unlike the simpler REC setting.

7. Conclusion

VLM-R1 brings R1-style reinforcement learning to visual understanding through a flexible, modular, and scalable framework. Across REC and OVD, it delivers gains in task performance and out-of-domain generalization while yielding practical insights for reward engineering, data selection, and model scaling.

  • VLM-R1 unifies R1-style reinforcement learning for vision-language models with flexible data definition, model modularity, and scalable training.
  • Experiments on REC and OVD show substantial gains in task performance and out-of-domain generalization.
  • The study identifies reward engineering, data selection, and model scaling as practical considerations for applying RL to complex vision-language tasks.
Loading 2504.07615v2…