VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning

1 INTRODUCTION

• Detection, segmentation, and counting share a multi-object cognition structure, suggesting they can be handled within a unified framework.Prior RL approaches often train task-specifically with different data for different tasks, which may limit scalability and generalizability.Source: §1 · p. 1
• VisionReasoner combines structured-thinking and non-repeat format rewards with multi-object IoU and L1 accuracy rewards, using batch computing plus Hungarian matching for prediction-to-ground-truth assignment.The matching pipeline is designed to improve computational efficiency while maintaining matching accuracy.Source: §1 · p. 1
• VisionReasoner reports relative gains of 29.1% on COCO-val (detection), 22.1% on ReasonSeg-test (segmentation), and 13.2% on CountBench-test (counting).It is evaluated across 10 tasks in detection, segmentation, and counting.Source: §1 · p. 2, Figure 1 · p. 2
• Human evaluation finds VisionReasoner’s reasoning faithful and reliable even without annotated reasoning training data.Source: §1 · p. 2
• The paper presents VisionReasoner as a unified visual-perception framework whose rewards and training strategy support diverse tasks within one shared model.It also identifies ablation studies as validating the design and offering insights into RL in LVLMs.Source: §1 · p. 2

2.1 LARGE VISION-LANGUAGE MODELS

• LVLMs have been extended to visual grounding and reasoning segmentation, but their performance has not been systematically evaluated under a unified visual-perception framework.Source: §2.1 · pp. 2–3

2.2 REINFORCEMENT LEARNING IN LARGE MODELS

• Related RL work includes RLHF (Ouyang et al., 2022), DPO (Rafailov et al., 2023), PPO (Schulman et al., 2017), and LVLM methods such as Visual-RFT (Liu et al., 2025b), EasyR1, and Seg-Zero (Liu et al., 2025a).The cited LVLM efforts are described as exhibiting strong reasoning capabilities and impressive performance.Source: §2.2 · p. 3

3.1 PRELIMINARY

• The method is motivated by the view that visual perception tasks share a multi-object cognition structure, while prior RL approaches often train task-specifically.The authors contrast reasoning-intensive instructions with traditional vision methods that struggle with complex or compositional queries.Source: §3.1 · p. 3, §3.5 · p. 5
• The method uses Group Relative Policy Optimization, an on-policy reinforcement-learning algorithm that generates groups of rollouts for each input before reward-based optimization.The method overview also identifies task reformulation, model architecture, a unified reward mechanism, and multi-object cognition training as its main components.Source: §3.1 · p. 3, §3 · p. 3

3.2 TASK REFORMULATION AND CATEGORIZATION

• Detection predicts bounding boxes, segmentation predicts binary masks through detect-then-segment, and counting estimates target-object numbers through detect-then-count.These categories include visual grounding and object detection, referring expression and reasoning segmentation, and object counting.Source: §3.2 · p. 4

3.3 VISIONREASONER MODEL

• VisionReasoner uses a reasoning module to locate objects and a segmentation module to produce masks when needed, supporting detection, segmentation, and counting in one shared framework.At inference, users provide an image, text prompt, and task type; the framework can also be extended to other visual perception tasks.Source: §3.3 · p. 4

3.4 UNIFIED REWARD MECHANISM

• The unified reward combines format and accuracy signals, using target bounding boxes and center points to reinforce structured outputs and multi-object recognition.It rewards thinking and answer formats, non-repetitive reasoning, and matched box and point accuracy.Source: §3.4 · p. 5

3.5 MULTI-OBJECT COGNITION IN LVLMS

• Batch reward computation and the Hungarian algorithm produce optimal one-to-one assignments between predicted objects and ground truth for box-IoU, box-L1, and point-L1 rewards.The matching strategy is designed to guarantee optimal assignment while maintaining computational efficiency.Source: §3.5 · pp. 5–6

4.1 EXPERIMENTAL SETTINGS

• Evaluation spans ten benchmarks and three task types; VisionReasoner is tested zero-shot after training on approximately 7k samples from four datasets.It is initialized with Qwen2.5-VL and SAM2.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 MAIN RESULTS

• VisionReasoner outperforms LVLMs in detection, reaches state-of-the-art performance in segmentation and counting, but trails some task-specific detection models on COCO.The segmentation and counting comparison uses SAM2 for LVLMs when needed.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDY

• For 30 objects, optimized matching takes 5 × 10^-4 seconds versus 2 × 10^-3 seconds without batching; reasoning length increases with query complexity.The reasoning is shorter for simple COCO class names and RefCOCOg phrases, and longer for ReasonSeg queries.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7
• Non-repeat rewards improve performance and shorten reasoning, while increased sampling first helps and then declines at larger sampling numbers.The non-repeat comparison uses 2,000 RefCOCOg samples; the sampling study uses all 7k training samples.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 9
• Performance improves with both GRPO and DAPO and with each of four training datasets; VQA performance gains slightly without VQA training.The RL-algorithm comparison uses 2,000 RefCOCOg samples.Source: Table 5 · p. 7, §4.3 · p. 8, Table 6 · p. 8
• Both reasoning and no-reasoning variants outperform the baseline, while the reasoning-enhanced model gains significantly on intricate reasoning segmentation data.The ablation removes the thinking instruction and thinking reward to eliminate the reasoning process.Source: §4.3 · p. 9, Figure 6 · p. 9

4.4 HUMAN EVALUATION ON REASONING PROCESS

• 97.0% image consistency accuracy and 90.5% answer consistency accuracy are reported; problematic traces are concentrated below IoU 0.25 despite no annotated reasoning training.Experts assess whether traces match image content and the final predicted answer.Source: §4.4 · p. 9, Table 8 · p. 9

4.5 QUALITATIVE RESULTS

• Figure 7 presents VisionReasoner addressing several perception tasks in one shared model, with further results and reasoning traces in Section G.Source: §4.5 · p. 9

• Across ten tasks, VisionReasoner reports relative gains of 29.1% on COCO detection, 22.1% on ReasonSeg segmentation, and 13.2% on CountBench counting.Source: §5 · p. 9
• The conclusion characterizes VisionReasoner as a unified vision-language framework that analyzes visual inputs, generates structured reasoning, and delivers task-specific outputs.Source: §5 · p. 9
• Human evaluation finds the reasoning traces well-grounded and faithful.Source: §5 · p. 9

• COCO and RefCOCO(+/g) supply detection evaluation, RefCOCO(+/g) and ReasonSeg supply segmentation evaluation, and PixMo-Count and CountBench supply counting evaluation.Source: B DETAILS OF EVALUATION BENCHMARKS · p. 14
• Evaluation preprocessing converts data to a unified multimodal conversation format and removes potential leakage, including numeric class labels in COCO and explicit counts in CountBench.Source: B DETAILS OF EVALUATION BENCHMARKS · p. 14
• COCO detection uses AP, RefCOCO(+/g) grounding uses bbox AP at IoU 0.5, segmentation uses gIoU, and counting uses count accuracy.Source: B DETAILS OF EVALUATION BENCHMARKS · p. 14
• The ten benchmarks cover detection, segmentation, and counting, with 66,023 test samples across ten specific tasks.Source: B DETAILS OF EVALUATION BENCHMARKS · p. 14, Table 9 · p. 14

• Training takes six hours on one node with eight GPUs, with peak GPU memory around 80 GB; reward converges around 100 steps and the best checkpoint is typically around 200.Peak memory can be adjusted through hyperparameters such as VeRL's memory_utilization.Source: C MORE TRAINING DETAILS · p. 14

• Expression-level results are additionally reported for RefCOCO(+/g), whose benchmarks provide multiple expressions per image.Source: D EXPERISSION LEVEL EVALUATION ON… · p. 15
• Table 10 compares expression-level performance on RefCOCO(+/g) tasks.Source: Table 10 · p. 15
• Results marked with an asterisk are cited from the Qwen2.5-VL report but were not reproducible in the authors’ environment.Source: Table 10 · p. 15

E TASK ROUTER

• TaskRouter semantically classifies textual instructions into one of four predefined task categories.It is a pure language model trained to identify instructions automatically during inference.Source: E TASK ROUTER · p. 15
• On a 20,000-sample training and 4,000-sample test dataset, Qwen2.5 accuracy falls below 50%, while the reinforcement-learning-trained router performs significantly better.The dataset combines original instructions and categories with ChatGPT-generated examples.Source: E TASK ROUTER · p. 15

F USER PROMPT TEMPLATE

• The policy-model prompt template is inspired by DeepSeek-R1-Zero and Seg-Zero; Table 12 indicates that “{Question}” is replaced during training and inference.Source: F USER PROMPT TEMPLATE · p. 15

G QUALITATIVE RESULTS

• VisionReasoner distinguishes similar objects in visual grounding and referring expression segmentation, and localizes multiple targets in detection and counting examples.Its reasoning is more detailed for intricate image-query pairs and concise for simpler inputs.Source: G QUALITATIVE RESULTS · p. 15, Figure 9 · p. 16

H EXTENSION

• Lightweight output-format modules extend VisionReasoner to referring-expression pose and depth estimation, with intermediate outputs serving as bridges to other modules.Figure 10 illustrates these extensions.Source: H EXTENSION · p. 16, Figure 10 · p. 17
• A hybrid mode pairs traditional visual models such as Yolo-World with VisionReasoner: the former handles simple categorical instructions and the latter complex instructions.Source: H EXTENSION · p. 17

I CONCEPTS CLARIFICATION

• The paper introduces its key terminology using a hierarchical task formulation and COCO as a representative example.Table 13 presents the terminology and definitions.Source: I CONCEPTS CLARIFICATION · p. 17, Table 13 · p. 17

• Although the taxonomy covers a broad range of task types, VisionReasoner is evaluated on only 10 representative tasks, with comprehensive evaluation of all types reserved for future research.Source: J DETAILS OF TASK REFORMULATION · p. 17
• Table 14 presents the fundamental task types of counting and visual question answering.Source: Table 14 · p. 17
• Table 15 presents the fundamental task types of detection and segmentation.Source: Table 15 · p. 18