PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions

Motivation

• Perceptual errors can invalidate otherwise correct reasoning, while changing post-training data mixtures and objectives can dilute caption-based perception supervision.Direct caption optimization is difficult because captions vary in granularity and style and bundle many claims into an open-ended description.Source: §1 · p. 1

Perception RL

• Perception RL audits dense captions, cross-verifies detected errors, and composes verified facts into fixed-label QA, combining unrelated errors in multiple-choice questions and correlated errors in identification chains.Specialized pipelines target selected high-error domains, while a general route covers failures beyond those domains.Source: §1 · p. 2

Perception Agentic RL

• Perception Agentic RL trains multi-turn visual inspection for small objects, cluttered scenes, and details needing magnification; Coordinate Injection corrects cold-start coordinates and a grounding-aware reward supervises intermediate spatial fidelity.Incorrect crops or annotations can distort later observations and propagate as coordinate drift.Source: §1 · p. 2, Figure 1 · p. 2

Training

• Training uses shared cold-start SFT followed by single-turn Perception RL and then multi-turn Perception Agentic RL, with each stage continuing from the preceding checkpoint.No additional SFT stage separates the two RL stages.Source: §1 · pp. 2–3

Contributions

• Across eight perception benchmarks, Perception RL raises average accuracy from 61.9% after cold-start SFT to 69.8%.It converts verified caption-error facts into fixed-label QA pairs with rule-based outcome rewards.Source: §1 · p. 3
• 71.6% average accuracy across seven benchmarks; PerceptRL exceeds the same-tool base model by 15.7 and the strongest evaluated frontier VLM by 5.2 percentage points.The shared tool-use protocol comparison is reported for the full pipeline.Source: §1 · p. 3

2.1 Fine-Grained Visual Perception

• Prior fine-grained perception work spans object detection, segmentation, text and document perception, and medical multimodal modeling, often aligning visual representations with localized outputs.The cited approaches include Grounding DINO, SAM, Florence-2, and methods for dense text, documents, and segmentation.Source: §2.1 · p. 3

2.2 Reinforcement Learning for VLMs

• VLM reinforcement-learning work primarily targets reasoning, while perception-focused methods provide explicit perception rewards, implicit losses, or task-specific detection and grounding rewards.The section contrasts reasoning-focused methods with perception-focused work including Perception-R1, PAPO, V-Triune, CodePercept, and CapRL.Source: §2.2 · p. 3

2.3 Thinking with Python

• Visual-tool methods learn cropping and code-execution policies, while PerceptRL adds error-mined verifiable QA and combines corrected trajectories with intermediate grounding-aware process rewards.Earlier tool-use approaches make spatial accuracy a complementary concern because later interactions depend on earlier crops and annotations.Source: §2.3 · pp. 3–4

3.1 Overview

• Perception RL targets verifiable single-turn answers, while Perception Agentic RL uses code-interpreter operations to inspect small, cluttered, or magnification-dependent image details.Incorrect crops or annotations can distort later evidence and cause coordinate drift.Source: §3.1 · p. 4
• Training proceeds from shared cold-start SFT on perception QA and coordinate-corrected agentic trajectories to Level-1 outcome-reward RL and Level-2 multi-turn RL with grounding-aware process rewards.Each stage starts from the preceding checkpoint; the two levels refer to task settings, whereas the three stages refer to the training sequence.Source: §3.1 · p. 4

3.2 Perception RL

• Level 1 organizes fine-grained abilities, turns the model’s perceptual errors into deterministic QA, and trains on them with rule-based reward RL.Source: §3.2 · p. 4, §3.2.2 · p. 4, §3.2.3 · p. 6

3.2.1 Perception Tree Construction

• The three-level tree groups visual attributes, spatial relations, and physical understanding; probing guides sampling, with specialized pipelines for four domains and a general route for other errors.The specialized domains are counting, color, spatial, and state.Source: §3.2.1 · p. 4

3.2.2 Rewardable QA Construction

• Agents audit dense captions, independently confirm errors, and compose unrelated errors as MCQs or correlated errors as multi-step chains.Quality gates and manual inspection screen QA pairs before their labels are used for rewards.Source: §3.2.2 · pp. 5–6

3.2.3 Rule-Based Reward for RL

• Fixed atomic labels enable exact-match outcome rewards: incorrect or unparseable answers receive zero.For Level 1, every response token shares an advantage determined by answer correctness relative to other responses in the group.Source: §3.2.3 · p. 6

3.3 Perception Agentic RL

• Coordinate drift can distort later observations and spatial decisions; Coordinate Injection corrects spatial coordinates in cold-start trajectories, while grounding-aware rewards supervise intermediate fidelity during RL.The token-level design adds each turn’s reward only to the tokens produced in that turn, alongside the final-answer reward applied across the trajectory.Source: §3.3 · p. 6, §3.3.1 · pp. 6–7, §3.3.2 · p. 8
• For difficult queries, the model crops, resizes, annotates, and measures images over multiple turns before answering.This targets small objects, cluttered scenes, and subtle color or state differences that may not be evident in a single view.Source: §3.3 · p. 6

3.3.1 Coordinate Injection

• Coordinate Injection extracts each spatial operation’s intended target, corrects its coordinates with a VLM judge or specialized detector, and re-executes the code before the next turn.The corrected trajectories become cold-start SFT data; the VLM judge handles most domains, while detectors can target precisely localizable objects such as those in dense counting.Source: Figure 3 · p. 7, §3.3.1 · pp. 7–8

3.3.2 Grounding-Aware Reward for RL

• A grounding-aware reward evaluates declared boxes and points against their intended targets, providing spatial-fidelity supervision alongside final-answer correctness.Operations are scored together in one composite image, with each grounding turn summarized by its least accurate operation.Source: §3.3.2 · p. 8

3.3.3 Token-Level Credit Assignment

• Final-answer reward supplies across-trajectory credit, while per-turn rewards distinguish grounding quality among turns within the same trajectory.The final-answer reward is broadcast to every token; each turn’s reward reaches only the tokens produced in that turn.Source: §3.3.3 · p. 8

4.1 Experimental Setup

• Training uses three sequential stages, and the 19,810-query Agentic QA pool is selected as a subset of the 36,445-pair single-turn pool; evaluation benchmarks are held out from training data.The evaluation covers eight perception benchmarks in the setup described for Table 1.Source: §4.1 · p. 9, Table 3 · p. 10

4.2 Main Results

• 61.9% mean accuracy after shared SFT (versus 61.7% for the base) rises to 69.8% after single-turn Perception RL, a 7.9-point gain.The Perception RL gain occurs before Level-2 tool interactions.Source: §4.2 · pp. 9–10
• 71.6% mean accuracy under shared tool use exceeds Gemini-3-Flash by 5.2 percentage points; the full pipeline gains 15.7 points over the same-tool base.Table 2 compares models using the same tool interface.Source: Table 2 · p. 10, §4.2 · p. 10
• Perception RL improves counting and embodied perception most: CountQA rises from 57.9% to 71.9%, RealWorldCounting from 29.3% to 42.1%, and ERQA from 53.8% to 62.8%.Gains also extend to TreeBench, BLINK, RefSpatial, RealWorldQA, and MMStar.Source: §4.2 · p. 10
• 53.9% on RealWorldCounting is Level 2’s largest improvement over Level 1; Level 2 also has the best reported RealWorldQA and TreeBench results, but trails Level 1 on MMStar, ERQA, and RefSpatial.Table 2 reports results under the shared tool-use protocol.Source: §4.2 · p. 10, Table 2 · p. 10

5.1 Coverage of the Two Construction Routes

• 71.9% on CountQA and 42.1% on RealWorldCounting are achieved when both construction routes are combined; the routes show complementary benchmark strengths under a matched query budget.The specialized route leads on TreeBench, ERQA, RefSpatial, and the counting benchmarks, while the general route leads on RealWorldQA, MMStar, and BLINK.Source: §5.1 · p. 10, Table 4 · p. 11

5.2 Comparison with Specialized Tool-Use Models

• Table 7 compares specialized visual tool-use models on V*, HR-Bench, MMStar, and TreeBench, but published scores retain their source evaluation settings.The base model with a code interpreter and PerceptRL use the paper’s Section 4.1 protocol.Source: §5.2 · pp. 10–11, Table 7 · p. 11

5.3 The Effectiveness of Coordinate Injections

• 45.8% on RealWorldCounting with inference-time Coordinate Injection compares with 37.4% using uncorrected coordinates on Gemini-3.5-Flash without training.Coordinate Injection further improves accuracy across all four evaluated benchmarks.Source: §5.3 · p. 11, Table 5 · p. 11

5.4 The Effectiveness of Grounding-Aware Process Reward

• 0.81 grounding fidelity at step 200 with the full reward compares with 0.63 for outcome-only RL; the full reward also improves accuracy on all four reported benchmarks.Accuracy gains over outcome-only RL are 2.0, 3.0, 5.1, and 8.4 percentage points on RealWorldQA, BLINK, TreeBench, and RealWorldCounting, respectively.Source: §5.4 · p. 11, Table 6 · p. 11

• PerceptRL improves aggregate performance across eight perception benchmarks, demonstrating that visual perception can be optimized during post-training.Source: §6 · p. 12
• The process-reward ablation improves grounding fidelity during training and final-answer accuracy across all four evaluated benchmarks.Source: §6 · p. 12
• The framework converts perceptual errors into verifiable QA and combines Coordinate Injection-corrected trajectories with grounding-aware process rewards for single-turn perception and multi-turn spatial grounding.Source: §6 · pp. 11–12