PaperlayerView exact revision on arXiv

Source-linked AI summary

Region-Level Policy Optimization for Fine-grained MLLM Perception

Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu

arXiv:2609.19745v1cs.CV

TL;DR

Fine-grained MLLM perception is costly when higher image resolution adds visual tokens. Vision-RL2 trains a region proposal predictor using a frozen MLLM reader, outperforming the base model and supervised predecessor at every token budget across fine-grained benchmarks and backbones.

  • Problem

    Fine-grained perception raises visual-encoding and language-model costs as higher image resolution adds tokens, while answer-relevant evidence may occupy only a small part of the image.

    Source: §1 · p. 1

  • Method

    Vision-RL2 trains a region proposal predictor with region-level reinforcement learning, using a frozen MLLM reader to score coherent regions by their effect on gold-answer likelihood.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across fine-grained benchmarks and backbones, Vision-RL2 outperforms the base model and supervised predecessor at every token budget; at matched survival, localization tolerates roughly 3–4× stronger token compression than recognition.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The measured asymmetry supports localizing from a coarse view and reserving higher resolution for selected evidence.

    Source: §1 · p. 2

    Because the reader remains frozen, the method cannot repair recognition failures that full-model fine-tuning can.

    Source: D LIMITATIONS · p. 18

Abstract

from arXiv · show

Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual tokens inflate vision-encoding and language-model prefilling costs. We show that the two operations underlying fine-grained perception, localizing the region of interest (RoI) and recognizing its content, have different resolution requirements. In a controlled diagnostic, localization tolerates roughly 3 to 4 times stronger token compression than recognition, which motivates localizing from a coarse view and concentrating resolution on the selected evidence. Decoding coordinates with the MLLM can be trained end-to-end from answers, but costs a full model pass per query and depends on grounding ability. A lightweight proposal network distilled from the model's attention is fast, but inherits the noise of its attention targets. The RoI from the proposal network reaches the answer through a discrete region choice, so its faithfulness to the answer cannot supervise the network. We therefore optimize the proposal network with region-level reinforcement learning, which we call Vision-RL2. It treats coherent regions as actions, and a frozen MLLM reader scores each one by how its removal changes the answer likelihood. Complementary subtractive and additive objectives suppress distracting proposals and recover missing evidence, updating only the predictor without region annotations, response sampling, or reasoning trajectories. The refined proposal further enables a sparse encoding that magnifies evidence and excludes background tokens. Across six fine-grained benchmarks and four MLLM backbones, Vision-RL2 improves accuracy over the base model at every token budget and surpasses its largest-budget accuracy with about 4 times fewer visual tokens. Code is available at https://github.com/YuHengsss/VisionRL2 .

1 INTRODUCTION

Fine-grained MLLM perception is costly because uniformly increasing image resolution adds visual tokens, even though localization tolerates more compression than recognition. Vision-RL2 uses a region-level answer signal to refine efficient RoI proposals and focus higher-resolution processing on selected evidence.

1 INTRODUCTION

  • 4.2× fewer visual tokens match SD-RPN accuracy on Qwen3.5-4B; across fine-grained benchmarks and backbones, Vision-RL2 beats the base model and SD-RPN at every shared token budget.At full resolution, it is also competitive with recent state-of-the-art methods that fully finetune the MLLM, while updating only the proposal network.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • At matched survival, localization tolerates roughly 3–4× stronger token compression than recognition, motivating coarse-view localization followed by high-resolution processing of selected evidence.The diagnostic re-predicts the box from a compressed scene for localization, but freezes the box and compresses its crop for recognition.Source: §1 · pp. 1–2
  • SD-RPN offers single-pass RoI proposals, but attention-derived pseudo-labels can retain distracting regions or miss needed evidence, and discrete region choice blocks answer-level gradients to the proposal map.Geometric supervision is also insufficient because fine-grained VQA may have multiple question-relevant regions rather than one uniquely annotated RoI.Source: §1 · p. 2
  • Vision-RL2 treats coherent regions as actions and uses a frozen MLLM reader’s answer-likelihood response to region removal to refine the proposal network without region annotations.Its subtractive objective prunes proposals below a calibrated noise margin, while its additive objective recovers missed evidence; it builds on trained SD-RPN to retain single-pass efficiency.Source: §1 · pp. 2–3
  • Sparse visual encoding processes selected evidence at finer token-level granularity, concentrating resolution on relevant regions rather than background.The paper presents the refined predictor and sparse encoding as a route to improved fine-grained accuracy and efficiency across models and benchmarks.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work improves fine-grained MLLM perception through higher-resolution encoding, answer-aware zooming, distillation, or separate localization, but these approaches incur compute, training, or supervision limitations. SD-RPN offers lightweight answer-free localization, yet its proposed regions are not verified against the answer.

Related approaches

  • Tiled and dynamic-resolution encoding raises visual-token use, while thinking-with-images RL and latent variants seek answer-aware zooming but incur full-model training or repeated inference costs.Uniform scaling spends compute regardless of evidence location; analysis suggests gains from thinking-with-images methods are dominated by the RL-optimized model rather than interleaved tool use.Source: §2 · p. 3
  • Privileged-view distillation transfers answer-level signals through full-backbone finetuning, while training-free attention or search-tree zooming needs multiple passes and transfers poorly.Supervised RoI predictors instead depend on curated coordinate annotations and costly data curation and finetuning.Source: §2 · p. 3
  • SD-RPN self-distills response-to-image attention into a lightweight, answer-free RoI predictor, but its token-wise surrogate supervision does not verify that proposed regions support the answer.It avoids both coordinate annotations and full-model finetuning by predicting RoIs from intermediate MLLM features.Source: §2 · p. 3

3 METHOD

Vision-RL2 refines an answer-free region proposal by scoring coherent regions with a frozen MLLM reader, then uses the proposal to concentrate visual tokens on foreground evidence. Its subtractive and additive objectives remove distracting regions and recover missing evidence without updating the MLLM.

3.1 PRELIMINARIES

  • SD-RPN distills response-to-image attention into an answer-free proposal, but token-wise supervision can leave spurious regions or miss necessary evidence without task-level feedback.Region-level reinforcement learning is introduced to optimize the proposal for minimal sufficient answer support.Source: §3.1 · pp. 3–4

3.2 REGIONS, MASKS, AND FUNCTIONAL CONTRIBUTIONS

  • The method treats connected visual regions as actions and scores each mask by how well a frozen, teacher-forced MLLM reader preserves the gold answer.Leave-one-out contributions require n + 1 reader evaluations for n regions, without response sampling or a reasoning trajectory.Source: §3.2 · pp. 4–5

3.3 FUNCTIONAL REGION-LEVEL POLICY OPTIMIZATION

  • Vision-RL2 optimizes predicted regions by removing low-contribution proposals and adding positively contributing missing regions, updating only the RPN while the MLLM remains frozen.Subtractive removals are calibrated against control-region noise; attainability weighting and an initial-SD-RPN KL anchor stabilize training.Source: §3.3 · pp. 5–6

3.4 SPARSE VISUAL ENCODING

  • Sparse visual encoding retains foreground tokens within the predicted bounding box, excludes background tokens, and views evidence at finer resolution for approximately the dense-crop token budget.For multiple regions, one enclosing crop preserves their shared positional frame, and source-image tokens are not re-encoded.Source: §3.4 · p. 6, Figure 3 · p. 6

4 EXPERIMENTS

Across four MLLM backbones and six fine-grained benchmarks, Vision-RL2 improves benchmark performance while enabling lower-token, lower-latency inference; ablations attribute gains to region-level RL and sparse encoding.

4.1 EXPERIMENTAL SETUP

  • Experiments train the predictor on four MLLM backbones using 7K QA pairs and evaluate fine-grained perception under both 16,384-token and training-aligned protocols.RL starts from the supervised SD-RPN checkpoint, with a frozen MLLM reader and 576-token source images during training. The training-aligned protocol uses short answers and rule-based scoring at 576 tokens, extended to 1,024, 2,048, and 4,096 for token-budget curves; the main protocol uses free-form responses and rule-based parsing with an LLM judge for remaining cases.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Vision-RL2-9B achieves the table’s highest average, ahead of Gemini-3.1-Pro and Vision-OPD-9B, with the best V* Bench and ZoomBench scores overall.It leads Vision-OPD-9B on every other benchmark except HR-Bench 4K. The 4B model surpasses Vision-OPD-4B on V* Bench, ZoomBench, and both HR-Bench splits but trails on MME-RealWorld; Qwen2.5-VL-7B has the highest average in its block, ahead of ZwZ-7B’s 69.9, while Gemma-4-12B gains 3.8 points over SD-RPN and 9.5 over the base model on average at 1,120 tokens.Source: §4.2 · pp. 7–8

4.3 EFFICIENCY ANALYSIS

  • At 576 tokens, both scales beat the base model at 4,096 tokens by more than three points while using about one-quarter as many visual tokens.The 4B model matches SD-RPN’s 4,096-token accuracy with 4.2× fewer visual tokens, while the 9B model comes within half a point with 2.5× fewer; Vision-RL2 dominates the accuracy–latency plane at both scales. At 1,024 tokens, the 4B model exceeds SD-RPN at 4,096 tokens with 2.1× lower latency, and the 4B model at 576 tokens beats Vision-OPD-4B at its largest budget while responding 2.6× faster.Source: §4.3 · p. 8, Figure 4 · p. 8
  • The RPN head routes in 31–49 ms, roughly 30× cheaper than attention routing or coordinate decoding, while achieving the best accuracy at both scales.Attention routing and coordinate decoding take 1.1–1.8 seconds per query, or 2.1–2.2× the RPN head’s end-to-end latency.Source: Figure 5 · p. 8, §4.3 · p. 9

4.4 ABLATION STUDIES

  • Region-level RL adds +3.0 six-benchmark average over SD-RPN, sparse encoding adds +1.5, and together the full system gains +14.9 over the frozen base model.RL improves which evidence is proposed, while sparse encoding improves how the crop budget is spent. With sparse encoding disabled, design ablations found costs of −2.3 for generation accuracy, −1.5 for raw mean log-probability, −1.2 for cell-level actions, −1.1 without the control-region margin, and −0.9 without additive recovery; pair removals matched the full recipe at extra cost.Source: §4.4 · p. 9, Table 2 · p. 9

5 CONCLUSION

Vision-RL2 uses answer-accountable region-level reinforcement learning to turn the localization–recognition compression asymmetry into an efficient, reusable RoI predictor. It outperforms the base model and its supervised predecessor at every token budget.

5 CONCLUSION

  • Vision-RL2 outperforms the base model and supervised predecessor at every token budget with a reusable, decode-free RoI predictor trained by frozen-reader region-level reinforcement learning.The frozen MLLM reader scores each coherent region by its functional contribution to the gold answer, without region annotations or response sampling.Source: §5 · p. 9

A METHOD DETAILS … A.3 STABILIZATION DETAILS

The method constructs candidate regions from proposal and reader-attention maps, then stabilizes region-level policy updates with anchor losses and an attainability weight. These details specify how regions are extracted, how proposals are regularized, and when policy losses are applied.

A.1 REGION CONSTRUCTION

  • Both candidate sources use a shared map-to-region operator that smooths maps, thresholds relative to the peak, extracts connected regions, and discards diffuse activations.The operator uses a Gaussian kernel, peak-relative threshold, and connected-component extraction.Source: §A.1 · p. 13
  • For subtractive proposals, region extraction and top-Kmax ranking are non-differentiable, but retained regions’ confidences remain differentiable with respect to their logits.Source: §A.1 · p. 13
  • For additive proposals, response-to-image maps from six evenly spaced MLLM layers are processed after subtracting the predicted mask, then overlapping residual regions are merged and ranked.The layers are 7, 11, 15, 19, 23, and 27, since answer-relevant attention can emerge at different depths across samples.Source: §A.1 · p. 13

A.2 ANCHOR LOSSES

  • A per-cell Bernoulli KL anchor to the frozen initial SD-RPN map is applied to all samples to limit drift on prompts the policy gradient cannot reach.Source: §A.2 · p. 13
  • When the component set has one region, the policy loss is zero and binary cross-entropy toward the binarized reference foreground preserves the proposal.With K = 1, no valid relative removal exists.Source: §A.2 · p. 13

A.3 STABILIZATION DETAILS

  • Both policy losses are scaled by a detached attainability weight that suppresses samples when the frozen reader cannot produce the gold answer under tested interventions.The subtractive group used for this weighting comprises the intact predicted mask and retained singleton-removal masks.Source: §A.3 · p. 13

B EXPERIMENTAL DETAILS · B.1 HYPERPARAMETERS · B.2 EVALUATION PROMPTS

Training uses a shared configuration with model-specific predictor depth and control margins, while the appendix lists benchmark prompts for both main and training-aligned evaluation protocols.

B.1 HYPERPARAMETERS

  • The predictor adds three trainable blocks after B frozen MLLM blocks, with B set per backbone and κ set to 1.25 for Qwen3.5-4B and 1.0 for the other three models.The three models using κ = 1.0 are encoded at the 560-token tier, nearest the 576-token limit.Source: §B.1 · p. 13
  • Table 3 provides training hyperparameters shared across all four backbones.Source: §B.1 · p. 13, Table 3 · p. 14

B.2 EVALUATION PROMPTS

  • MME-RealWorld-CN uses a Chinese translation of the EN template, while MME-RealWorld-Lite reuses that template with a trailing short-answer line under the training-aligned protocol.Source: §B.2 · p. 14
  • For the main protocol, V* Bench and HR-Bench use the same prompt asking users to select from the listed choices.Source: §B.2 · p. 14
  • The main-protocol ZoomBench prompt places the question and options after the image without additional answer-format instructions.Source: §B.2 · p. 14
  • The main-protocol MME-RealWorld EN prompt requests the correct option letter only, from A through E.Source: §B.2 · p. 14
  • Under the training-aligned protocol, V* Bench, HR-Bench, and ZoomBench prompts request an option letter directly, with wording varying by benchmark.Source: §B.2 · pp. 14–15
  • The training-aligned InfoVQA prompt asks for an answer in a single word or phrase.Source: §B.2 · p. 15

B.3 PER-BENCHMARK TOKEN CURVES · B.4 GENERALIZATION TO AN ENCODER-FREE BACKBONE

Across all six benchmarks, the proposed model stays above the base model and SD-RPN pipeline at every source-image token limit and both scales. The encoder-free Gemma-4-12B evaluation adapts the pipeline to tiered visual-token budgets and sparse crop encoding.

B.3 PER-BENCHMARK TOKEN CURVES

  • Across all six benchmarks, our model outperforms the base model and SD-RPN pipeline at every source-image token limit and both scales.Figure 6 plots accuracy against measured visual tokens per sample for source-image limits of 576, 1,024, 2,048, and 4,096 under the short-answer protocol, with Qwen3.5-4B and Qwen3.5-9B results in separate rows.Source: Figure 6 · p. 15, §B.3 · p. 15

B.4 GENERALIZATION TO AN ENCODER-FREE BACKBONE

  • For encoder-free Gemma-4-12B (Team et al., 2026), images are embedded directly into the language model at one of five visual-token tiers, from 70 to 1,120 tokens.The SD-RPN predictor uses the Qwen3.5 recipe on a Gemma-regenerated corpus at 560 tokens, and RL uses its own 7K pool at that tier. At test time, the source image uses 1,120 tokens, the crop uses the tier quantizing its crop budget, and sparse encoding drops background crop tokens before embedding.Source: §B.4 · p. 15

B.5 COMPARISON WITH VISUAL TOKEN PRUNING · B.6 QUALITATIVE COMPARISON

The pruning comparison finds that Vision-RL2 exceeds full-budget reference accuracy on the high-resolution suite at both reported tiers, while pruning struggles on text-dense benchmarks. The qualitative comparison shows RL refinement suppressing irrelevant regions and retaining answer-relevant evidence at the same token budget.

B.5 COMPARISON WITH VISUAL TOKEN PRUNING

  • The comparison uses Qwen2.5-VL-7B, released VisionZip and DART implementations, identical prompts, and a shared evaluation harness across a 4-to-4,096-token pixel range.VisionZip retains dominant-plus-contextual tokens after the vision encoder; DART prunes to a nominal fraction at the second LLM layer.Source: §B.5 · p. 16
  • For the document suite, source-plus-crop totals average 46.6% and 27.2% of native tokens at the two tiers.The method downsamples the source by factors of 3 and 6 and caps the crop at half the downsampled source.Source: §B.5 · p. 16
  • On the high-resolution suite, Vision-RL2 exceeds full-budget reference accuracy at both reported tiers and lies above the reference across the plotted budgets.The reported source-plus-crop totals average 41.2% and 24.7% of the reference at the 1,024- and 576-token rungs.Source: §B.5 · p. 16, Figure 7 · p. 16, Table 4 · p. 17
  • At the 25% tier, VisionZip retains 73.3% on InfoVQA and 56.5% on OCRBench, while Vision-RL2 holds 94–99% on both.DART falls further on these text-dense benchmarks.Source: §B.5 · p. 16, Table 4 · p. 17
  • ChartQA is the exception: at the 25% tier, pruning wins, while Vision-RL2 reduces both vision-encoder and LLM budgets and pruning saves only LLM-side tokens.The passage attributes the ChartQA result to global-layout dependence: uniform token selection at one-sixth source resolution preserves more structure than region re-allocation.Source: §B.5 · p. 16

B.6 QUALITATIVE COMPARISON

  • In Figure 8, RL-refined predictions suppress spurious regions and retain required evidence, enabling correct reader answers at the same token budget where supervised SD-RPN fails.The examples include a sundial plaque, gated archways, and all four bridge pillars.Source: §B.6 · p. 16, Figure 8 · p. 17
  • The refinement rewards regions for their functional contribution to the answer, teaching the predictor to retain answer-dependent evidence rather than regions that merely correlate with teacher attention.This addresses SD-RPN's pseudo-label-surrogate failures: spreading probability over question-irrelevant regions and only partially covering evidence.Source: §B.6 · pp. 16–17

C LOCALIZATION-VS-RECOGNITION PROBE: PROTOCOL AND CONTROLS

The probe compares localization from compressed scenes with recognition from compressed evidence crops. Localization retains an advantage even when crop budgets are fixed, while its boxes become less precise but continue to cover the evidence.

C LOCALIZATION-VS-RECOGNITION PROBE: PROTOCOL AND CONTROLS

  • The Qwen3.5-4B probe uses ZoomBench (Wei et al., 2026), filters to 276 eligible samples, and evaluates survival on 238 anchor-correct samples.The model greedily predicts evidence boxes; eligibility requires ground-truth-crop answerability robust to 15% area-preserving jitter and excludes confident text-only answers.Source: C LOCALIZATION-VS-RECOGNITION PROBE:… · p. 18
  • Localization downsamples the scene and reads a native-resolution crop from each re-predicted box; recognition freezes the anchor box and downsamples its crop.Recognition crops hit a processor floor of about six tokens from r≥3, making deep-rung survival potentially optimistic.Source: C LOCALIZATION-VS-RECOGNITION PROBE:… · p. 18
  • At r=6, localization retains an 18.2-point survival advantage over recognition with both crops capped at 128 tokens.The control addresses the larger native-resolution crops produced by re-predicted boxes as scene compression increases.Source: C LOCALIZATION-VS-RECOGNITION PROBE:… · p. 18, Table 5 · p. 18
  • From r=1.5 to r=6, median IoU falls from 0.73 to 0.11, while median ground-truth coverage rises from 0.44 to 1.00 and hit rate from 47% to 75%.The passage characterizes this as graceful soft failure: imprecise localization is recovered by downstream cropping, rather than demonstrating precision robustness.Source: C LOCALIZATION-VS-RECOGNITION PROBE:… · p. 18

D LIMITATIONS

Vision-RL2’s frozen reader cannot repair recognition failures, and its compression-asymmetry diagnostic has limited scope. Training requires question–answer pairs, though not region annotations.

D LIMITATIONS

  • The frozen reader cannot repair recognition failures that full-model finetuning can, as seen on MME-RealWorld at the smallest scale.This limitation follows from keeping the reader frozen.Source: D LIMITATIONS · p. 18
  • The compression-asymmetry probe is diagnostic evidence from one backbone and one benchmark, not a general law.Source: D LIMITATIONS · p. 18
  • Training requires question–answer pairs to score contributions, but does not require region annotations.Source: D LIMITATIONS · p. 18
Loading 2609.19745v1…