PaperlayerView exact revision on arXiv

Source-linked AI summary

Look at Where You Think: Localization-Head Alignment for Fine-Grained Perception in Thinking VLMs

Yuan Wang, Wei Wu, Hongning Wang

arXiv:2610.39649v1cs.CV

TL;DR

Thinking VLMs still miss small details in high-resolution images, and outcome rewards do not specify where to look. LHA supervises localization-head attention at visual-fact tokens during SFT and RL. It raises Qwen3.5-4B to 76.6% accuracy as the most accurate single-pass 4B model evaluated, with gains also reported for Qwen2.5-VL-3B.

  • Problem

    Thinking VLMs can reason for thousands of tokens yet still miss small details, while outcome-reward RL does not tell attention heads where to look during reasoning.

    Source: §1 · p. 1

  • Method

    LHA applies an auxiliary attention objective to localization heads at reasoning tokens stating visual facts, directing them toward annotated evidence in both SFT and RL.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    76.6%: LHA is the most accurate single-pass Qwen3.5-4B model evaluated, and its gains also carry over to direct-answer Qwen2.5-VL-3B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LHA provides a regional attention signal alongside outcome rewards, with the resulting model outperforming inference-time zooming and larger released models in a single pass.

    Source: §8 · p. 8

    Evaluation covers three high-resolution multiple-choice benchmarks; larger models, other model families, and open-ended tasks remain to be explored.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Thinking vision-language models reason for thousands of tokens, yet still miss small details in high-resolution images. Zooming methods pay for extra inference passes, and outcome rewards never tell the model where to look at. In Qwen3.5-4B, we find that the bottleneck is selection, not seeing: a few attention heads, which we call localization heads, pick the evidence out of thousands of visual tokens; ablating four of them costs 13.2 points more than ablating four random heads, and their focus fades as the reasoning grows. We therefore propose Localization-Head Alignment (LHA), which makes attention a training signal: whenever the reasoning states a visual fact, these heads learn to look at the focal region rather than the background, as an auxiliary loss in both SFT and RL training. On benchmarks of V*, HR-Bench 4K and TreeBench, LHA lifts Qwen3.5-4B from 74.1% to 76.6% accuracy, the best single-pass 4B result we evaluate, ahead of multi-pass zooming and released 7B models. Alignment alone adds 1.8 points over otherwise identical training while shortening reasoning by 20%, and the gains carry over to Qwen2.5-VL-3B.

1 Introduction

Thinking VLMs still miss small details because evidence competes with thousands of visual tokens and growing text. The paper diagnoses evidence selection as the bottleneck and introduces LHA to supervise localization-head attention during reasoning, improving single-pass accuracy.

Motivation

  • Small visual details remain difficult because they occupy only a handful of tokens among thousands, while language priors, unrelated regions, and growing text compete for attention.The introduction gives a kite in a 4K image as an example of this competition.Source: §1 · p. 1

Diagnosis

  • Ablating four localization heads costs 13.2 points more than ablating four random heads, and their preference for evidence fades as reasoning grows.Across 12 model–dataset settings, correct answers also involve more attention to evidence than incorrect answers.Source: §1 · pp. 1–2

Method

  • LHA supervises localization heads at reasoning tokens that state visual facts, training them to attend to annotated evidence rather than equally sized background regions.The auxiliary attention objective applies in both SFT and RL without changing inference; outcome rewards alone do not tell the heads where to look.Source: §1 · pp. 1–2

Main results

  • 76.6%: LHA is the most accurate single-pass Qwen3.5-4B model evaluated, beating ViCrop with two extra passes, EASE by 3.0 points, and released perception models up to 7B.Figure 1 compares accuracy with mean generated tokens and marks methods using extra passes or crops.Source: Figure 1 · p. 1, §1 · p. 2
  • 1.8 points: alignment SFT’s gain over cross-entropy SFT grows to this margin after identical outcome RL, with the greatest gains on questions relating several objects.The 1.8-point gap has a 95% CI of [0.5, 2.9].Source: §1 · p. 2

2 Related Work

Prior approaches either add inference-time zooming or supervise other attention units and signals. LHA instead supervises independently selected localization heads at the native queries that predict visual facts.

Inference-time methods and perception RL

  • Inference-time guided search, attention-selected crops, and learned crop tools recover small details at the cost of extra passes.Perception RL rewards correct answers, grounded boxes, or stated perception, acting on policy outputs.Source: §2 · p. 2

Attention supervision

  • EASE matches a head-averaged map to an evidence prior, Faithful-MR1 supervises an inserted focus token, and AVAR rewards visual attention after a cold start.These approaches differ from LHA’s direct supervision of independently selected localization heads at visual-fact queries.Source: §2 · p. 2

LHA’s distinction

  • LHA contrasts evidence attention with an equally large background, canceling the attention softmax normalizer and giving alignment SFT and attention RL a shared regional-preference objective.The objective does not change inference.Source: §2 · p. 2

3 Setup

The study evaluates thinking-mode Qwen3.5-4B on high-resolution multiple-choice benchmarks and compares its results with released models. The protocol uses common scoring rules and a fixed base-model reference.

Model

  • The study examines the 128 full-attention heads of Qwen3.5-4B, which directly attend to visual tokens, rather than its linear-attention layers.The model interleaves Gated DeltaNet linear-attention layers with eight full-attention layers, each having 16 query heads.Source: §3 · p. 3

Benchmarks and protocol

  • Evaluation covers 1,396 multiple-choice questions from V*, HR-Bench 4K, and TreeBench, with Qwen3.5-4B decoded in thinking mode under a 32K-token budget.All models use the same scoring rules and judge; reported evaluations were checked by hand against manual labels.Source: §3 · p. 3

4 Where Thinking VLMs Select the Evidence

The evidence survives image encoding but is buried among visual tokens, and a few localization heads select it. Those heads’ evidence routing weakens during reasoning and is associated with answer correctness.

4.1 The answer hinges on small evidence

  • 22.1 points: masking the question’s ground-truth region costs Qwen3.5-4B more than masking an equal-area region elsewhere; smaller evidence incurs a larger cost.This frames fine-grained perception as finding a few relevant tokens among thousands.Source: §4.1 · p. 3

4.2 The evidence survives encoding but is buried

  • 60.7% vs. 60.1%: Qwen3.5-4B answers V* and TreeBench questions about as accurately using only evidence-box tokens as it does from the full image.An equal-area set of other tokens yields 38.0%, while appending box tokens to the full image adds almost nothing.Source: §4.2 · p. 3

4.3 A few localization heads select the evidence

  • 13.2 points: zeroing the top four localization heads costs more than zeroing four random heads from the same layers, with large drops on every benchmark.Head ranking uses held-out grounding questions; in a no-reasoning probe, ablating eight selected heads lowers accuracy from 52.6% to 36.8%, while random heads leave it at 52.6%.Source: §4.3 · p. 3

4.4 Evidence routing predicts correctness and fades during reasoning

  • The localization heads’ share of attention on evidence falls by a quarter from the beginning to the end of the reasoning chain, while V* routing at the question’s end separates correct from incorrect answers.The decline also appears on V* and TreeBench rollouts; restoring the amount of attention reaching the image does not help.Source: §4.4 · pp. 3–4, Figure 2 · p. 4

5 Localization-Head Alignment

Localization-Head Alignment (LHA) applies a routing loss to localization heads where reasoning states visual facts, first in alignment SFT and then in attention-supervised RL. For direct-answer models, the adapted method supervises the answer position.

5 Localization-Head Alignment

  • LHA uses one routing loss across two phases: alignment SFT on the model’s own correct trajectories, followed by outcome-reward RL with that loss on correct rollouts.Source: §5 · p. 4
  • For direct-answer Qwen2.5-VL-3B, the adapted LHA supervises the answer position rather than fact positions.Source: §5 · p. 4

5.1 Routing loss on localization heads

  • At queries predicting annotated fact tokens, LHA contrasts localization-head attention on box-covered visual tokens with an equally sized outside-box background, shifting preference toward evidence.Source: §5.1 · pp. 4–5

5.2 Phase 1: alignment SFT

  • Alignment SFT combines the routing loss with whole-trajectory cross-entropy on the base model’s own correct best-of-five samples, without an external teacher.Training updates only the supervised layers’ attention projections, 2.6% of language-model parameters.Source: §5.2 · pp. 5–6

5.3 Phase 2: attention-supervised RL

  • Attention RL starts from the aligned model, adds the same loss to GRPO, and supervises fact queries only on correct rollouts using the union of referred boxes.The attention gradient flows through all parameters; unlike SFT, fact queries in a rollout share one evidence set.Source: §5.3 · p. 6

6 Experiments

Across the evaluated baselines, LHA is the most accurate single-pass Qwen3.5-4B model, and alignment gains persist through outcome RL. The experiments also compare attention RL, released perception models, and an adapted implementation on Qwen2.5-VL-3B.

6.1 Baselines and controls

  • Baselines include released models on their own backbones, inference-time interventions on Qwen3.5-4B, and EASE; CE and alignment SFT share trajectories and training setup before identical outcome RL.LHA continues alignment SFT with attention RL, while the β=0 control uses the same RL without that term.Source: §6.1 · p. 6

6.2 Attention supervision makes perception more accurate

  • 76.6% accuracy makes LHA the most accurate single-pass Qwen3.5-4B model, 2.5 points above the base model with 24% fewer tokens, and ahead of ViCrop’s two extra attention passes and crop.Table 1 covers V*, HR-Bench 4K, TreeBench, Overall accuracy across 1,396 questions, and mean completion length.Source: §6.2 · p. 6, Table 1 · p. 7
  • 1.8 points is the post-RL alignment advantage over CE SFT on identical trajectories, with 20% fewer completion tokens; alignment SFT alone reaches 76.1%, 1.4 points above CE SFT.The post-RL difference has 95% CI [0.5, 2.9].Source: §6.2 · p. 6
  • 0.9 points separate LHA from the β=0 control after the same RL, while the attention term sharpens evidence preference rather than leaving it flat.The advantage is 1.8 points on V* and 1.6 on TreeBench.Source: §6.2 · pp. 6–7
  • 3.0 points separate LHA’s 76.6% from EASE’s 73.6% on the same backbone; LHA supervises localization heads at fact queries, unlike EASE’s layer-averaged map.LHA uses a normalizer-free contrast.Source: §6.2 · p. 7
  • 3.8 points over ZwZ-4B and 9.8 over TreeVGR-7B put the 4B LHA model ahead of every evaluated released perception model overall.LHA also leads both models on HR-Bench 4K and TreeBench.Source: §6.2 · p. 7
  • 2.5 points is the aligned pipeline’s lead after the same RL stage on Qwen2.5-VL-3B, with alignment SFT already beating CE SFT on V* and TreeBench.This is an adapted direct-answer implementation of regional supervision.Source: §6.2 · p. 7, Table 2 · p. 7

7 Analysis

The analysis finds that the attention term supplies a consistent routing signal that outcome-reward gradients do not. Evidence preference strengthens during training, transfers to V*, and accompanies shorter, faster RL rollouts.

7 Analysis

  • The attention term raises routing score s in all eight active groups, while the larger outcome gradient is nearly orthogonal to routing and flips sign across groups.In all-correct groups, outcome advantages are zero, while the attention term still raises s.Source: §7 · pp. 7–8, Table 3 · p. 8
  • 0.760 to 0.860 is the supervised heads’ evidence preference increase over 50 steps with attention, while the β=0 control stays at 0.761.On V* object words, the preference is .895 after attention RL versus .842 for the control; in-box attention share rises from 0.076 to 0.126.Source: §7 · p. 8, Table 3 · p. 8
  • On Qwen2.5-VL-3B, alignment SFT nearly triples visual attention inside the V* evidence box at the answer, and the change survives RL.CE SFT barely moves that attention.Source: §7 · p. 8
  • 19% shorter rollouts make each RL step 15% faster than the CE-initialized policy; LHA runs 8% faster per step than its β=0 control.At test time, alignment SFT→RL generates 20% fewer tokens than CE SFT→RL, and LHA generates 24% fewer than the base model.Source: §7 · p. 8

8 Conclusion

LHA addresses the missing where-to-look training signal by directly supervising localization heads at fact tokens in both SFT and RL. It produces a single-pass 4B model that beats evaluated zooming methods and larger released models, with gains also carrying to a direct-answer model.

8 Conclusion

  • LHA supervises localization heads at reasoning fact tokens with a regional contrastive loss in SFT and RL, using boxes during training but not inference.The conclusion presents localization heads as a practical handle for training perception.Source: §8 · p. 8

Limitations

The evaluation is limited to two small Qwen backbones and three high-resolution multiple-choice benchmarks, with box annotation and grounding requirements for training. The related-work discussion distinguishes LHA by its focus on selected functional heads at native fact-prediction positions.

Limitations

  • The study trains two Qwen backbones at 3–4B scale and evaluates three high-resolution multiple-choice benchmarks; larger models, other families, and open-ended tasks remain unexplored.The cited open-ended examples include charts, documents, and video.Source: Limitations · p. 8
  • LHA relies on box annotations and grounding questions to identify heads for each new model; domains without region annotations would require automatic grounding.Fact-token matching also leaves facts phrased in other words unsupervised.Source: Limitations · pp. 8–9

A Extended Related Work

  • LHA targets functional heads at native reasoning positions that predict visual facts, contrasting evidence with equally sized background rather than supervising total visual attention.The same regional auxiliary loss is used for SFT and RL.Source: A Extended Related Work · p. 12

B Evaluation Protocol

The evaluation uses native-resolution benchmarks and controlled interventions to distinguish evidence loss from selection among post-encoder visual tokens. The diagnostics show that the evidence survives encoding and identify a small set of localization heads using held-out grounding data.

B Evaluation Protocol

  • The suite contains 1,396 native-resolution questions; trained Qwen3.5-4B results use two-decode means where available because same-checkpoint decodes differ by about 1.1 points.Rules extract committed options, with an independent judge used only when rules cannot parse a response; manual checks agreed with all automatic scores.Source: B Evaluation Protocol · pp. 12–13

C.1 Evidence masking

  • Masking the control region changes accuracy by −1.1 points relative to the original images (CI [−4.4, 2.3]); the drop is largest on V* and grows as the evidence box shrinks.Without the evidence, mean response length increases from 5,784 to 9,251 tokens.Source: §C.1 · p. 13

C.2 Encoder-side token controls

  • On 499 synthetic small-object scenes, evidence-box tokens alone raise accuracy by 16.8 points (CI [12.6, 21.0]), while appending them to the full image adds 1.8 points (CI [−1.0, 4.8]).A re-encoded box crop raises accuracy by 27.7 points (CI [23.2, 32.1]).Source: §C.2 · p. 13
  • On natural-image V* and TreeBench questions, evidence-box tokens alone preserve accuracy (60.7% vs. 60.1%); equal-area control tokens reduce it to 38.0%.Appending box tokens to the full image changes accuracy by only +0.8 points; on V*, box-only tokens cost 6.3 points, while TreeBench accuracy rises by 4.5 points.Source: §C.2 · p. 14

C.3 Localization head selection

  • The four highest-scoring localization heads are L19H14, L15H11, L19H2, and L23H9, selected by attention to evidence boxes versus equal-area controls on 96 held-out grounding questions.The V* ranking correlates with this ranking (Spearman 0.80); LHA supervises the top ten heads.Source: §C.3 · p. 14

C.4 Localization-head ablation

Localization heads read visual evidence at fact-stating tokens, but their attention to the evidence declines as reasoning grows. The section separates this drift into text-driven dilution and weakening image preference, and tests whether restoring image attention alone improves accuracy.

C.4 Localization-head ablation

  • Zeroing selected heads reduces held-out forced-choice accuracy from 52.6% to 36.8%, while layer-matched random heads leave it at 52.6%.The generation ablation’s drop appears on all three benchmarks.Source: §C.4 · p. 14

C.5 Where the image is read: direct answers and thinking

  • Across 12 model–dataset settings, correct direct answers have higher evidence share than incorrect ones by 0.040 to 0.113; confidence intervals exclude zero in nine settings.The five top-ranked localization heads per model are measured at the first answer token.Source: §C.5 · p. 14
  • At fact tokens, evidence share is 0.77 in the first half of reasoning and 0.72 in the second, but 0.50 at the final-answer token.Median enrichment at fact tokens is 3–3.6; 0.50 is the value of uniform attention.Source: §C.5 · p. 15
  • The evidence share falls from 0.71 in the first tenth of benchmark reasoning to 0.61 in the last and 0.59 at the answer, in both correct and incorrect rollouts.Across the chain, the decline appears in 220 base rollouts of 110 V* and TreeBench questions.Source: §C.5 · p. 15

C.6 Decomposing the drift of visual attention

  • The log visual share falls with slope −1.09 against log ntext, while image affinity also declines (slope −0.43, CI [−0.46, −0.40]).Within-image evidence-box share drops from 0.53 to 0.40 between the first and last quarter of the chain; localization heads drift faster than the average head.Source: §C.6 · p. 15
  • Accuracy stays at the base level when a log(ntext/n0) bias offsets dilution, so restoring attention to the image does not specify where within it the heads should look.LHA instead supervises preference for evidence regions.Source: §C.6 · p. 15

D Localization Heads in Qwen2.5-VL-3B

In Qwen2.5-VL-3B, evidence-selective heads are sparse and deep, and evidence share tracks answer correctness. Across Qwen3.5-4B alignment and RL analyses, the attention term shifts routing where outcome gradients do not.

D Localization Heads in Qwen2.5-VL-3B

  • 17 of 576 heads place more than 5% of visual attention on evidence boxes averaging 0.83% of visual tokens; the strongest head, L27H1, reaches 17.9%.The five strongest heads lie in layers 24–27; on one question, evidence share peaks at 0.45 in layer 27.Source: D Localization Heads in Qwen2.5-VL-3B · p. 16
  • 22.1% evidence share accompanies correct answers versus 11.0% for wrong answers, while accuracy rises from 63.4% to 90.7% across share quartiles.The correct-versus-wrong difference has Mann–Whitney p=0.006; the quartile trend has point-biserial r=0.24.Source: D Localization Heads in Qwen2.5-VL-3B · p. 16, Figure 5 · p. 16

E.1 Alignment SFT on Qwen3.5-4B

  • 65 correct self-generated trajectories suffice for alignment SFT, which supervises 44 fact-token predictors across 15 trajectories for 32 updates.The 65 trajectories are drawn from a pool of 82, with 17 held out for routing measurements.Source: §E.1 · p. 16

E.2 Attention RL on Qwen3.5-4B

  • The RL attention term supervises up to 64 fact positions per correct rollout, contrasting evidence boxes with an equally large random background disjoint from every box.It uses attention weights from ten supervised heads and leaves advantages unchanged.Source: §E.2 · p. 17
  • 0.860 evidence share after 50 RL steps with the attention term compares with 0.761 without it, a paired difference of +0.10 [0.08, 0.12].The attention term starts from the same alignment-SFT checkpoint; 95% of trajectories have higher routing share with the term.Source: §E.2 · p. 17, Figure 6 · p. 17

E.3 Gradient decomposition of the RL objective

  • 0.18 mean routing gain comes from the attention term even in all-correct groups, where outcome gradients vanish; in mixed groups, outcome gradients are about 18 times larger but nearly orthogonal.The attention term's first-order routing change ranges from 0.02 to 0.37 across active groups.Source: Figure 7 · p. 17, §E.3 · p. 18

E.4 Qwen2.5-VL-3B implementation

The Qwen2.5-VL-3B adaptation applies regional supervision at answer prediction, with alignment raising evidence share beyond cross-entropy training. Additional evaluations report greedy-decoding gains, improvements across evidence sizes, and shorter responses.

E.4 Qwen2.5-VL-3B implementation

  • Mean evidence share rises from 0.21 after CE SFT→RL to 0.54 after alignment SFT→RL, compared with 0.19 for the base model.Alignment SFT alone reaches 0.56; the focused fraction is 64.5% after alignment SFT→RL versus 33.7% after CE SFT→RL. The adaptation measures attention at the query predicting the first answer token and uses five heads selected per question.Source: §E.4 · p. 18, Figure 8 · p. 18

F Additional Results

  • In 44 of 60 pairs where only one model is truncated, the aligned model finishes; on V*, HR-Bench 4K, and TreeBench it answers 12, 27, and 10 more questions with 23.4%, 18.0%, and 21.3% fewer tokens.Pairs where both models finish account for 86.8% of the token saving.Source: F Additional Results · p. 19

F.1 Inference cost

  • Table 10 reports per-question forward passes, extra image encodings, and overall accuracy for inference-cost comparisons.The passage identifies the table’s measures but supplies no cell values.Source: Table 10 · p. 19, §F.1 · p. 19

F.2 Greedy decoding

  • Under greedy decoding, alignment SFT→RL beats CE SFT→RL by +2.1 points (95% CI [0.4, 4.0]) and generates 24.1% fewer completion tokens.The base model has 266 budget-truncated responses (19.1%) under greedy decoding, compared with 37 at temperature 1.Source: §F.2 · p. 19, Table 12 · p. 21

F.3 Gains by evidence size

  • Alignment SFT→RL gains +2.6 points for evidence boxes below 0.25% of the image, +1.1 for the middle tertile, and +2.0 for boxes above 2.6%.On HR-Bench 4K it gains 1.7 points (CI [0.2, 3.2]).Source: §F.3 · p. 19

F.4 Response length

  • Across both decodes, alignment SFT→RL generates 8.69M tokens versus 10.82M for CE SFT→RL, 19.7% fewer overall.The response-length reduction extends across the distribution, not only the truncated tail.Source: §F.4 · p. 19, Figure 9 · p. 21

G All Endpoints

  • Tables 11 and 12 collect the paper’s thinking-mode endpoints and greedy-decoding evaluation, respectively.Table 11 includes two independent decodes per trained model; Table 12 reports greedy decoding and paired differences to the base model.Source: G All Endpoints · p. 20, Table 11 · p. 20, Table 12 · p. 21

H Full-Width Figures

  • Appendix figures cover encoder-side token controls, attention drift and localization, and crop examples where attention shifts to evidence and answers change.The TreeBench example changes from answer D to A as evidence share rises from 7.1% to 59.7%; the RefChartQA example changes from wrong to correct as share rises from 2.8% to 40.3%.Source: Figure 10 · p. 21, Figure 9 · p. 21, Figure 11 · p. 22

Related papers

Loading 2610.39649v1…