PaperlayerView exact revision on arXiv

Source-linked AI summary

The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets

Zhensheng Hu, Guanglong Xu, Guanqiang Hu

arXiv:2610.32664v1cs.CV

TL;DR

Small visual-token budgets can make text-rich images illegible, while choosing a useful zoom region requires localization before the answer. LookZoom distills the model’s correct-answer attention into a dedicated pointer that selects how to look; across three Qwen3-VL scales and five benchmarks, it outperforms zooming baselines at low budgets.

  • Problem

    Under per-image visual-token budgets, downsampling can make the fine text in documents, charts, and infographics illegible, while the region for zooming must be chosen before answering.

    Source: §1 · pp. 1–2

  • Method

    LookZoom uses a dedicated <look> token to distill localization from the model’s correct answers in one prefill and select a view without answer generation, box annotations, or external localizers.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    At 98 tokens per view, LookZoom beats the strongest baselines by 4.8–9.1 points and attention-guided zooming by 11.1–17.0 points across three model scales and five benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis finds that answer-time attention localizes evidence more clearly than pre-answer attention, and that the value and best form of zooming depend on each image’s information gap.

    Source: §1 · p. 2, §5.2 · p. 7

    The study focuses on text-rich images; zooming into several regions at once and extending the token to video frames are left for future work.

    Source: F. Limitations · pp. 17–18

Abstract

from arXiv · show

Multimodal large language models (MLLMs) read images through visual tokens, and under the small per-image budgets that efficient deployment requires, the fine text of documents, charts and infographics becomes illegible. Zooming in recovers the evidence, but the region must be chosen before the model answers. We find that MLLMs know where the evidence is, but not in a form that can be read before they answer: the answer tokens localize it from the middle layers on, whereas the attention available before answering does so only in the top layers, at a depth that varies across models, and diffusely. Supervising the pre-answer attention to point degrades reading, and the benefit of zooming is governed by each image's information gap. Building on these findings, we introduce ours, which lets an MLLM look where its answer would. A dedicated look token, processed in a single prefill and kept out of the answering pass, learns in hindsight to attend as the model's own correct answers do, extracting this knowledge into a concentrated pointer before any answer exists. Its hidden state then selects how to look, from no zoom to crops, square windows, windows that adapt to the pointer and foveated warps, supervised by the counterfactual outcome of every view. ours needs no box annotations, per-model layer search, external localizers or generation before the answer. On Qwen3-VL at three scales and five text-rich benchmarks, it surpasses the strongest baseline by 4.8-9.1 points at 98 visual tokens per view and attention-guided zooming by 11.1-17.0 points, while running faster than attention-guided and grounding-based zooming.

1. Introduction

Small per-image visual-token budgets can make text in documents, charts, and infographics illegible, so zooming requires choosing evidence before answering. The paper analyzes this timing problem and introduces LookZoom, which extracts answer-time localization into a dedicated look token and uses it to select a view.

1. Introduction

  • Under visual-token budgets, image downsampling can make the text needed to answer questions illegible; at 98 tokens, Qwen3-VL-8B averages 48.2 on five text-rich benchmarks versus 82.2 at 392 tokens.Zooming must recover the relevant region within the per-view budget, and that region must be chosen before answering.Source: §1 · pp. 1–2
  • The answer tokens localize evidence better than the pre-answer position, but pre-answer localization catches up only in top layers, varies by model, and remains diffuse.The reported answer-token attention is 1.5–1.8× higher on evidence than the answer-start position; supervising pre-answer attention to point sharpens it but costs reading accuracy.Source: §1 · p. 2
  • LookZoom adds a dedicated <look> token that distills localization from the model’s correct answers in one prefill, then selects how to look without box annotations, layer search, external networks, or answer generation first.The token is removed from the answering pass; its hidden state selects among no zoom, crops, windows, pointer-adapted windows, and foveated warps.Source: §1 · p. 2
  • At 98 tokens per view, LookZoom exceeds the strongest baselines by 4.8–9.1 points and attention-guided zooming by 11.1–17.0 points, while running faster than attention-guided and grounding-based zooming.It achieves the best average accuracy at 98 and 196 tokens per view across three model scales and five benchmarks.Source: §1 · p. 2
  • LookZoom’s replace variant is 5.1 points more accurate than AttWarp at the same number of visual tokens.In the replace variant, the answer uses the selected view alone.Source: §1 · p. 2

2. Related work

Prior work re-encodes relevant image regions using either text-generated boxes or model attention. LookZoom instead uses a dedicated in-model pointer to choose among view types without generating an answer or coordinates first.

2. Related work

  • Region-selection methods either decode boxes or other text before reading a zoomed view, or derive regions from attention, including answer-start attention and generated-answer attention.Examples include Zoom-Refine, ZoomEye, RL-trained agents, ViCrop, and AttWarp.Source: §2 · p. 2
  • LookZoom computes a dedicated pointer in the question prefill, distills it from attention to the model’s own correct answers, and selects among view types using counterfactual outcomes.Unlike box-supervised or external-grounder approaches, it uses no box annotations, external grounders, or per-model layer search.Source: §2 · p. 3
  • Prior attention-shaping approaches train attention against annotated boxes or external grounding cues, whereas LookZoom gives pointing its own pathway rather than steering shared attention.The paper reports that supervising shared attention trades reading for pointing.Source: §2 · p. 3

3. Preliminary analysis

The preliminary analysis finds that evidence localization is available in answer-token attention but is difficult to read before answering, and that zooming’s value depends on the image’s information gap. These findings motivate extracting a pointer before answering and choosing views per sample.

3. Preliminary analysis

  • At 196 tokens, answer-token attention places 1.5–1.8× more mass on evidence than the pre-answer position, while comparable pre-answer localization appears only in model-dependent top layers and remains diffuse.For 2B, 4B, and 8B, middle-layer pre-answer evidence mass is 0.107 / 0.097 / 0.095, versus 0.167 / 0.142 / 0.144 for answer tokens.Source: §3 · p. 3, Figure 2 · p. 3
  • Supervising pre-answer attention sharpens pointing but reduces single-view reading accuracy in 8 of 11 benchmark–budget settings, by up to 2.7 points.This motivates giving pointing its own pathway.Source: §3 · p. 3
  • Zooming gain rises with information gap ρ: it costs 1.3–3.6 points when ρ < 1.6 and brings 10.3–15.1 points when ρ > 18.The trend also holds within benchmarks as the budget changes, rather than reflecting which datasets contain larger images.Source: §3 · p. 3, Figure 3 · p. 4
  • The best view changes with ρ, shifting from gentle views below ρ = 2.2 to stronger magnification at larger gaps; zooming adds no detail when ρ ≤ 1.For a zoomed view of relative area A, the pixel-density gain is bounded by min(ρ, 1/A).Source: §3 · p. 4

4. Method

LookZoom uses a dedicated token to form a pointer and select a view before answering, then trains the reader and view-selection head using answer rollouts and counterfactual view outcomes. The method operates under a fixed visual-token budget per view and can choose not to zoom.

4.1. Problem setup

  • With budget B per view, the model chooses where to look and which of eight transforms to apply, from no second view to crops, windows, and foveated warping.The transformed full-resolution image is re-encoded with at most B tokens; the answer uses the low-resolution view and transformed view, or only the original when no zoom is selected.Source: §4.1 · p. 4

4.2. A dedicated look token

  • A dedicated <look> token forms an image pointer in one prefill; its token-specific LoRA is excluded from the answering pass, separating pointing from reading.The pointer averages attention over a band of middle layers, normalized over image tokens and smoothed; reader-parameter gradients are discarded after pointer-loss backpropagation.Source: §4.2 · p. 4

4.3. Where to look: hindsight distillation

  • Hindsight distillation trains <look> to match sharpened answer-time attention from correct rollouts, extracting localization before an answer exists without boxes or an external localizer.For up to two correct rollouts, answer-token attention is aggregated over tokens, heads, and selected layers; squaring the target concentrates it on peaks.Source: §4.3 · pp. 4–5

4.4. How to look: counterfactual view supervision

  • A head predicts the view from <look> features and image, budget, and pointer features, trained on the recorded outcomes of every view across 6,000 further training images.It maximizes centered scores and selects the highest-scoring action at test time; choosing no second view saves visual tokens.Source: §4.4 · p. 5

4.5. Training the reader and inference

  • The reader trains with GRPO on uniformly sampled view types; inference uses one prefill to select a view, then one answering pass without <look>.The replace variant answers from the transformed view alone, using the same answering-pass visual-token count as a single view and AttWarp.Source: §4.5 · p. 5, Figure 4 · p. 5

5. Experiments

The experiments evaluate LookZoom under per-view visual-token budgets, compare it with attention-guided, grounding-based, and learned zooming baselines, and test how its pointer and viewing choices affect results. LookZoom’s strongest gains occur at tighter budgets, while ablations and analysis examine pointer concentration, reading, and evidence localization.

5.1. Setup

  • The study uses Qwen3-VL-Instruct at 2B, 4B, and 8B, budgets of 98, 196, and 392 tokens, and five text-rich benchmarks, with 500-question public-benchmark subsets.Training jointly adapts the reader and <look> token on 4,500 questions without evidence boxes.Source: §5.1 · pp. 5–6

5.2. Main results

  • At 98 tokens per view, LookZoom beats the strongest baselines by +9.1 / +5.3 / +4.8 points (2B / 4B / 8B), and ViCrop and AttWarp by +17.0 / +11.1 / +13.4 points.At 196 tokens, its lead over the strongest baseline is +5.5 / +4.0 / +4.6 points; at 392 tokens, it remains within −0.2 to +2.7 points.Source: Table 1 · p. 6, §5.2 · p. 7

5.3. Ablations

  • At 98 tokens, pointer-sized adaptive windows gain +9.9 points over same-layer pre-answer attention and +5.7 over best top-layer attention; the action head also improves on fixed views.Sharpening the pointer target raises evidence mass from 0.152 to 0.229; without sharpening, adaptive-window accuracy falls 3.8 points on average. Shared-attention training lowers reading accuracy by up to 2.0 points, while restricting targets to correct answers matters little.Source: §5.3 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8

5.4. Analysis

  • At 196 tokens on V-CoT-Doc, the <look> token puts 0.229 / 0.201 / 0.216 of its attention on evidence (2B / 4B / 8B), twice same-layer pre-answer attention.At 98 tokens, its selected region contains 68–81% of the evidence area, compared with 33–39% for Ground-then-crop; replacing its pointer and action head with the named box lowers accuracy by 3.6–4.7 points.Source: §5.4 · p. 8

6. Conclusion

The conclusion presents LookZoom as a way to extract diffuse, late evidence localization into a dedicated look token that can choose how to inspect an image without interfering with reading. It identifies multi-step looking, multi-image inputs, and video as next steps.

6. Conclusion

  • LookZoom uses a dedicated look token to extract evidence localization and decide how to look in one prefill, without interfering with reading.The conclusion says it outperforms attention-guided, grounding-based, and learned zooming.Source: §6 · p. 9
  • The conclusion characterizes pre-answer evidence localization as late in the network and diffuse.Source: §6 · p. 9
  • Multi-step looking, multi-image inputs, and video are identified as natural next steps.Source: §6 · p. 9

Supplementary Material

The supplementary material specifies implementation and baseline procedures, including how candidate views are generated and how comparison methods are reimplemented. It also reports that LookZoom’s selected regions cover more evidence than named boxes at smaller budgets.

A. Implementation details

  • The adaptive window spans the central 80% of pointer mass along each axis, with a minimum side of 12% of the image; foveated views use separable inverse-CDF resampling.The replace variant answers from the selected view alone and trains its look token with hindsight distillation.Source: A. Implementation details · p. 11

B. Baselines

  • ViCrop and AttWarp are reimplemented with their official settings, while GapSight is reimplemented from its paper because no code is public.GapSight’s router is trained on the same 6,000 images as the action head, using candidate-crop loss gaps as supervision.Source: B. Baselines · pp. 11–12
  • At 98 tokens, GapSight’s crop covers 51–60%; the gap closes at 392 tokens, where the text becomes legible enough to be named.Source: B. Baselines · p. 12

C. Full results

The supplementary results extend the budget comparison and report a full-evaluation-set check for Qwen3-VL-4B. They also include a five-benchmark average plot through 784 tokens.

C. Full results

  • The supplementary plot shows five-benchmark average accuracy across visual-token budgets, including 784 tokens, where the information gap is small and methods are close.Source: C. Full results · p. 12, Figure S1 · p. 13
  • On complete evaluation sets at 196 tokens, LookZoom averages 75.5, versus 71.3 for Ground-then-crop and 71.2 for GapSight.The reported margins are close to those on the 500-question subsets.Source: C. Full results · p. 12
  • The plot’s dashed line is the replace variant at AttWarp’s token cost; GapSight† denotes the authors’ reimplementation.Source: Figure S1 · p. 13

D. Additional analyses

Additional analyses show that zooming helps when the selected view is useful, while pointer choice and information gap shape the gains. The analyses also characterize how the action head changes its views and where evidence appears across layers.

D. Additional analyses

  • Pointed windows help substantially more than random windows, while ground-truth-centered windows indicate remaining placement headroom.Table S10 compares random, pre-answer-attention, <look>-token, and ground-truth-centered windows on V-CoT-Doc.Source: D. Additional analyses · p. 13
  • At 98 and 196 tokens, LookZoom corrects 37.9–47.6% of single-view errors and breaks 5.4–9.5% of correct answers, with methods closer at 392 tokens.At 2B and 196 tokens, GapSight breaks 6.9% versus LookZoom’s 7.1%; against the RL-adapted reader, correction and harm counts barely change.Source: D. Additional analyses · p. 13
  • At 98 tokens, the two-view adaptive window driven by <look> beats the same window driven by pre-answer attention by 9.3–10.7 points across scales.In the replace protocol, <look> also improves square windows and foveated warps over same-layer pre-answer attention at 98, 196, and 392 tokens.Source: D. Additional analyses · p. 13
  • As the information gap rises, the 4B action head zooms on more questions—74% in the lowest quintile versus 99% in the highest—and median window area falls from 0.38 to 0.17.The 8B head instead adapts view type, choosing foveated warps for 67% of questions in the lowest quintile and at most 2% in the top three.Source: D. Additional analyses · p. 14
  • Answer-token attention localizes evidence from the middle layers, but pre-answer attention does so only in top layers whose peaks vary by model scale.The <look> token is read from a fixed middle-layer band and exceeds both comparison bands there.Source: D. Additional analyses · pp. 14–15

E. Theoretical analysis

The theoretical analysis formalizes how information gap and pointer concentration shape the value and size of a zoomed view. It also explains how hindsight distillation and separated pointer training preserve reading while guiding view selection.

E. Theoretical analysis

  • For ρ ≤1, zoom adds no detail; below ρ=1.6 it can cost points, while useful windows shrink as the information gap grows.For a fixed window, the density gain grows with ρ until ρ=1/A; shrinking below A=1/ρ adds no detail but increases the risk of missing evidence.Source: E. Theoretical analysis · p. 15
  • Pointed windows perform far better than random windows, indicating that the model relies on the zoomed-view content.The comparison uses the same square-window size and includes ground-truth-centered windows as an upper bound on V-CoT-Doc.Source: Table S10 · p. 16
  • As pointer mass q concentrates, the window shrinks from p to evidence width w at q⋆, then can clip the evidence.Pointers with the same peak location can produce different windows; fixed windows are insensitive to this concentration.Source: E. Theoretical analysis · p. 16
  • Hindsight distillation minimizes expected KL loss at the conditional mean of sampled answer-attention targets, training <look> to predict where correct answers attend on average.At test time, the token extracts that localization from the prompt without sampling or decoding an answer.Source: E. Theoretical analysis · pp. 16–17
  • In pointer-training development runs, <look> reaches the highest evidence mass without affecting reading, whereas supervising shared attention degrades reading.The table reports evidence mass at 196 and 392 tokens and reading-accuracy change across 11 benchmark–budget settings.Source: Table S13 · p. 17
  • Pointer parameters do not affect earlier prefill states or the answer distribution for a given view because <look> is absent from the answering pass.The reader update also discards gradients from the pointer loss, so pointing affects reading through view choice rather than changing the reader.Source: E. Theoretical analysis · p. 17
  • Executing every view gives the action head the exact gradient of expected score, unlike a one-view policy-gradient estimate with nonzero variance.The proposition gives the gradient identity for the head’s objective.Source: E. Theoretical analysis · p. 17

F. Limitations

The evaluation focuses on text-rich images, and extending the approach to multiple regions or video frames remains future work.

F. Limitations

  • The <look> token is trained separately for each model, replacing per-model calibration of a training-free raw-attention pointer.Its gains concentrate at large information gaps; at 392 tokens it is on par with the strongest baseline, and at 784 tokens methods are close.Source: F. Limitations · p. 17
  • The study leaves zooming into several regions at once and extending the token to video frames for future work.The reported focus is text-rich images, where tight budgets make evidence illegible.Source: F. Limitations · pp. 17–18

G. Qualitative results

Qualitative failures show that selecting a window does not guarantee that the evidence will be found or read correctly. The examples contrast successful and failed views and answers.

G. Qualitative results

  • Failures include distractor-driven placement, border-clipped words, wrong-line reading in dense tables, and tall infographics whose evidence window cannot be magnified.A plausible low-resolution distractor can draw the window to a heading instead of the factory name.Source: G. Qualitative results · p. 19
  • Figure S4 pairs selected windows and evidence boxes with the zoomed view and answer correctness; Figure S5 uses the same layout for failures.In S4, correct answers are green and wrong answers red.Source: Figure S4 · p. 20, Figure S5 · p. 21

Related papers

Loading 2610.32664v1…