Source-linked AI summary
Perception-R1: Pioneering Perception Policy with Reinforcement Learning
En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, Wenbing Tao
TL;DR
The paper asks how rule-based RL should be applied to visual perception, where thinking-based RL does not consistently improve all tasks. It analyzes task complexity and reward design, then proposes Perception-R1 using GRPO for scalable MLLM post-training. Perception-R1 reports gains across several perception benchmarks, including 31.9% mAP on COCO2017 val for object detection.
Problem
The role of RL in MLLM visual perception remains insufficiently understood because language-focused reasoning approaches do not consistently improve visual perception tasks.
Method
Perception-R1 uses rule-based GRPO during MLLM post-training, with reward matching and task-specific rewards for visual perception.
Results
Perception-R1 achieves improvements on RefCOCO+, PixMo-Count, and PageOCR, and reaches 31.9% mAP on COCO2017 val for object detection.
Takeaways & Limitations
RL is more effective for higher-complexity perception tasks, while reward design plays a pivotal role in learning stronger perception policies.
Takeaways & Limitations
Many current perception tasks are overly simplistic, limiting RL’s exploration space and the possibility of discovering perceptual patterns through thinking processes.
Abstract
from arXiv · showhide
Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual complexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2.5-VL-3B-Instruct, Perception-R1 achieves +4.2% on RefCOCO+, +17.9% on PixMo-Count, +4.2% on PageOCR, and notably, 31.9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.
1 Introduction
This work examines how rule-based RL can support perception policy learning in MLLMs, motivated by limited evidence from language-focused reasoning approaches. It finds that RL effectiveness depends on perceptual complexity and reward design, and introduces Perception-R1 as a scalable framework.
- Motivation: Current RL explorations primarily focus on linguistic or language-centric multimodal tasks, leaving perception policy learning in MLLMs less understood.The paper argues that visual perception has distinct properties requiring a revised understanding of RL.
- Motivation: Visual perception concerns physical truth values such as points, lines, and bounding boxes, while often involving direct single-step predictions rather than structured reasoning.These characteristics distinguish perception tasks from natural-language reasoning tasks.
- Findings: Explicit thinking during RL is not necessary for current perception policy, and models without it can perform better.The paper reports that adding a thinking process does not consistently improve visual perception performance.
- Findings: Perceptual complexity determines RL’s advantage over SFT, with larger gains on complex visual tasks such as object detection.The paper identifies perceptual perplexity as a factor influencing RL effectiveness.
- Framework and Results: Perception-R1 applies rule-based GRPO during MLLM post-training and achieves improvements across RefCOCO+, PixMo-Count, PageOCR, and COCO2017 detection.Reported gains include +4.2% on RefCOCO+, +17.9% on PixMo-Count, +4.2% F1-score on PageOCR, and 31.9% mAP on COCO2017 val.
2 Related Works
Related work develops multimodal foundation models and visual perception capabilities through architectures, modeling strategies, and post-training. This paper extends that line of work by studying RL as a way to enhance visual perception.
- Multimodal Foundation and Reasoning Models: Multimodal foundation models combine visual encoders and adapters with autoregressive language-model decoders to process visual and textual information.Large-scale pretraining and visual instruction tuning support their visual comprehension and generation capabilities.
- Visual Perception in Multimodal Models: Visual perception in MLLMs supports integrating, comprehending, and reasoning about information from images or videos.Existing approaches improve it through visual perception architectures, visual-language modeling strategies, and post-training techniques.
- RL-based Post-training in LLMs and MLLMs: RL-based post-training has improved LLM and MLLM capabilities through human-preference and task-specific objectives, including RLHF and DPO.The related-work discussion positions rule-based RL as a newer direction in post-training.
3 Preliminaries
The paper defines perception policy as a progression from extracting visual information to reasoning over it and completing tasks or interacting with the environment. It uses GRPO to optimize MLLM post-training through group-relative rewards without a separate critic model.
- Perception Policy Definition: Perception policy enables a model to extract and understand visual information, reason about it, and complete tasks or interact with the environment.The paper applies this concept to pure visual tasks such as counting and detection and visual-language tasks such as grounding.
- Group Relative Policy Optimization: GRPO is a rule-based RL algorithm for LLM post-training that uses group-relative rewards instead of a separate critic model.It samples multiple outputs for the same input, averages their rewards as a baseline, and uses relative rewards for policy updates.
- Group Relative Policy Optimization: GRPO computes advantages from groups of rewards associated with outputs sampled for the same input.The formulation includes hyper-parameters ϵ and β.
4 Perception-R1
Perception-R1 applies rule-based GRPO reinforcement learning during MLLM post-training, using task-specific visual rewards and matching procedures for perception tasks. Its configuration spans visual grounding, OCR, counting, and object detection, with curriculum sampling and constrained coordinate outputs where needed.
- Rule-based Reward Modeling: Perception-R1 applies rule-based GRPO during MLLM post-training and optimizes reward modeling for perception policy learning.GRPO uses multiple sampled outputs and group-relative rewards without a separate critic model.
- Rule-based Reward Modeling: Perception-R1 constrains coordinate outputs to the [x1,y1,x2,y2] structure for visual perception tasks requiring bounding boxes, lines, or points.This format supplements the final-answer and intermediate-reasoning format conventions used in existing RL methods.
- Multi-Subject Reward Matching: Reward matching models multi-subject predictions and ground truths as bipartite graph nodes, then uses the Hungarian algorithm to maximize total reward.The method ensures predicted attributes are paired with corresponding ground truths before answer rewards are aggregated.
- Multi-Subject Reward Matching: Perception-R1 primarily uses reward matching for visual counting and object detection, where images contain multiple objects.The matching problem addresses the need to assign each predicted subject attribute to the appropriate ground-truth reference.
- Perception-R1 Configuration: Experiments cover visual grounding, OCR, visual counting, and object detection using 5k ∼10k samples per task, with Qwen2-VL variants as baseline models.Training uses RL post-training with an initial learning rate of 1e−6, eight rollouts by default, and curriculum sampling from easier to harder examples.
- Rule-based Reward Modeling: Visual perception rewards use task-specific discriminative signals, including IoU for grounding, Euclidean distance for counting, and edit distance for OCR.These rewards provide explicit, quantifiable feedback based on visual ground truths.
5 Experiments
Experiments evaluate Perception-R1 across visual grounding, counting, OCR, object detection, and general image understanding, then analyze its components, rewards, and rollout scaling. Results show strong task performance, benefits from reward design and rollout increases, and task-dependent RL effectiveness.
- 5.1 Performance Landscape in Perception Tasks: Perception-R1 is evaluated on visual grounding, counting, OCR, object detection, and general image understanding benchmarks.The experiments use mainstream perception tasks and additional multimodal comprehension benchmarks.
- 5.1 Performance Landscape in Perception Tasks: 98.1 F1-score on PageOCR exceeds GOT at 97.2 and LLaVA-NeXT at 64.7, including zero-shot performance on the Chinese metric.Perception-R1 does not use Chinese OCR data for training.
- 5.1 Performance Landscape in Perception Tasks: 17.9% improvement over Qwen2-VL on PixMo val demonstrates strong visual counting performance using a detect-then-count formulation.The paper attributes the improvement to exploration of intrinsic visual-logic mechanisms.
- 5.1 Performance Landscape in Perception Tasks: 31.9 AP makes Perception-R1 the first pure MLLM to surpass 30 AP on COCO2017 val for general object detection.The result matches or exceeds specialized expert models according to the paper.
- 5.2 Ablation Study of Perception-R1: Reward matching improves multi-subject counting and detection, while explicit thinking during training and inference degrades performance across all four evaluated perception tasks.Bipartite matching expands reward-assignment possibilities relative to sequential matching; explicit thinking is described as non-essential.
- 5.2 Ablation Study of Perception-R1: RL outperforms SFT on high-perceptual-perplexity counting and detection tasks but underperforms SFT or SFT+RL on low-perplexity grounding and OCR tasks.The paper links this pattern to the larger exploration space of higher-perplexity perception tasks.
- 5.3 More In-depth Analysis: Progressively adding refined answer rewards to format rewards consistently improves object-detection performance and ultimately exceeds expert-model performance.The reward-design analysis identifies task-specific reward refinement as a promising direction.
- 5.3 More In-depth Analysis: Increasing rollout count improves reward optimization and final performance on visual grounding and counting.The rollout-scaling experiments use 5k sampled data, with 8 as the default 1× rollout setting.
6 Limitation and Conclusion
The paper identifies overly simplistic perception tasks as limiting RL exploration and the emergence of perceptual breakthroughs. It concludes that Perception-R1 demonstrates strong potential for scalable perception policy learning across visual tasks.
- Limitation: Overly simplistic perception tasks limit RL exploration and restrict the possibility of MLLMs achieving a perceptual “Aha moment” through thinking.The authors suggest that finding more appropriate perception tasks, or meta tasks, may address this limitation.
- Conclusion: Perception-R1 is a simple, effective, and scalable rule-based RL framework for MLLM perception policy learning.The framework is developed from the paper’s experimental findings about perception policy learning with RL.
- Conclusion: Perception-R1 sets new state-of-the-art results across multiple visual perception tasks, particularly object detection.The paper reports that the framework achieves and surpasses expert-model performance in some tasks.
A Appendix
The appendix supplies additional Perception-R1 details omitted from the nine-page main paper, including dataset, training, and experimental information.
- Appendix: The appendix adds detailed dataset and training settings, along with additional experimental results.Sections A.1 and A.2 cover these supplementary materials.
A.1 Additional Details about Experimental Setting
The appendix expands the experimental setting with dataset statistics and task-specific prompts. Training data are kept separate across perception tasks because their rewards differ.
- Dataset: Table 7 provides detailed training-dataset statistics for the perception tasks used in Perception-R1’s RL post-training.The appendix expands on the data introduced in Section 4.3.
- Dataset: Perception-R1 does not mix data from different perception tasks during joint training because their rewards vary.This design choice is stated in the Table 7 caption.
- Prompts: Table 8 presents diverse prompts used for distinct perception tasks in Perception-R1.The system prompt follows Qwen2-VL and Qwen2.5-VL.
A.2 Additional Experimental Results
The appendix provides qualitative analyses of Perception-R1 across visual counting, OCR, visual grounding, and general object detection.
- Qualitative Analysis: Qualitative cases evaluate Perception-R1 on visual counting, OCR, visual grounding, and general object detection.The selected examples are shown in Figures 3–6.
- Qualitative Analysis: In general object detection examples, bounding-box colors correspond to object categories.This encoding is specified in Figure 6.