Attention Sinks in VLMs Are Borrowed, Not Seen

1 INTRODUCTION

• Visual attention sinks absorb image attention regardless of the question, making attention maps misleading for methods that treat them as records of where a model looks.Prior work had not established what makes an image token become a visual sink.Source: §1 · p. 1
• Centered cosine similarity between image-token keys and the text-sink key detects sinks in every sink-bearing model, with AUC ≥0.99.Hidden-state norm and key and value norms detect sinks in only some model families.Source: §1 · p. 2
• Moving flagged keys’ text-sink-direction component to the median image-key value returns sink attention to the text-sink while leaving answers nearly unchanged.The attention map changes, often placing its most-attended image token on the queried object.Source: §1 · p. 2
• The entropy-triggered re-look improves accuracy on 39 model-benchmark pairs, with a mean gain of +4.3 points.The evaluation covers 10 models and V*, HR-Bench (4K and 8K), and MME-RealWorld-Lite.Source: §1 · p. 2

2.1 ATTENTION SINKS

• Earlier work identifies sinks through massive activations, attributes them to several sources, and either redistributes or exploits their attention.Proposed sources include the vision encoder, language model, and rotary position encoding.Source: §2.1 · pp. 2–3
• In full-attention models, removing visual sinks returns their attention to the text-sink, not the target, while answers stay nearly the same.The altered map matters to methods that read it, including attention-guided cropping, token pruning, and grounding.Source: §2.1 · p. 3

2.2 VISUAL TOOLS IN MULTI-MODAL REASONING

• The re-look reads a sink-cleaned attention map, inserts a fresh crop as a second image block, and triggers at the trace’s first uncertain step.This contrasts with approaches that use encoded tokens or a phrase, or that train crop or zoom actions into the trace.Source: §2.2 · p. 3

3.1 DEFINITIONS

• A visual sink receives at least 100× its uniform attention share outside the target box; seven of ten models are sink-bearing, with sinks holding a quarter to four fifths of in-image attention.The other three models are sink-free under the study’s definition, with sink shares of at most 12%.Source: §3.1 · p. 4, §3.2 · p. 4, Table 1 · p. 4

3.3 A SINK IS AN IMAGE KEY ALIGNED WITH THE TEXT-SINK KEY

• Gemma-4’s sliding-window layers can form sinks when the question cannot reach the text-sink; shrinking the image moves a third to half of question attention back to text.Unlike sinks in full-attention models, Gemma-4’s image sinks have ordinary-sized values and dominate their layers’ attention output, so the model reads them.Source: §3.2 · p. 4, §3.3 · p. 6
• In edited full-attention models, removing the text-sink direction eliminates sink attention, while assigning it to random image keys captures 91% to 97% of in-image attention.Projecting out a random direction leaves sink attention unchanged; the authors conclude the text-sink alignment is necessary and sufficient for sink attention in the edited models.Source: §3.3 · pp. 5–6

3.4 REMOVING THE SINKS LEAVES THE MODEL UNCHANGED

• In four sink-bearing full-attention models, editing raises the text-sink’s share of question attention by 0.10 to 0.21, while accuracy shifts by at most three points; Gemma-4 is an output-changing exception.Removing Gemma-4’s sinks changes output KL by up to 0.53, so the authors do not apply the edit to that model.Source: Table 2 · p. 6, §3.4 · pp. 6–7

3.3 A SINK IS AN IMAGE KEY ALIGNED WITH THE TEXT-SINK KEY

• Only centered cosine separates sinks in every sink-bearing model, with a minimum AUC of 99.0; prior activation and norm features are family-specific.Raw cosine drops to 94.0 on Gemma-4-E2B, while the target and other image tokens stay near cosine 0.Source: §3.3 · p. 5, Table 3 · p. 6

3.4 REMOVING THE SINKS LEAVES THE MODEL UNCHANGED

• The edit gates tokens whose centered cosine exceeds τc = 0.5 and moves their text-sink-direction component to the median over image keys, leaving queries and values untouched.Layers are left unedited when more than ρ = 0.02 of image tokens pass, because the alignment then belongs to the image as a whole.Source: §3.4 · p. 6
• The re-look cleans the prefill attention map, selects the 1/3-grid window with the most attention, and inserts its crop when next-token entropy first exceeds H0 after c thought tokens.The edit is applied in a separate prefill; the model that writes the answer is unedited.Source: Figure 4 · p. 7, §3.4 · p. 7
• In four sink-bearing full-attention models, the largest single-token in-image attention share falls by an order of magnitude, and the most-attended token lands in the target box on 16% to 72% of images.The target gains no attention; it becomes the most-attended token because sinks no longer outrank it.Source: §3.4 · p. 7

4 METHOD

• The re-look cleans the map in a separate prefill, selects a region from that map, and inserts a crop when uncertainty first crosses a threshold during reasoning.The answer-generating model remains unedited; running the same procedure on the raw map isolates the effect of sinks.Source: §4 · pp. 7–8

4.1 SELECTING THE WINDOW

• The selector averages attention from two layers near half depth, then chooses the highest-attention window spanning one-third of the image-token grid.Gemma-4 instead uses two full-attention layers and skips the edit; the selected window is mapped to pixels, padded by 20 pixels, and fixed for the question.Source: §4.1 · p. 8

4.2 TRIGGERING AND INSERTING THE CROP

• The trigger inserts the crop at the first decoding step with next-token entropy above 1 nat, after at least 32 thought tokens and while still in the thought channel.It fires at most once; the crop comes from the original pixels and is encoded as a second image, while untriggered questions decode exactly as baseline.Source: §4.2 · p. 8
• Each question requires one extra prefill, and each triggered question adds one image encoding and one image-token block without changing the weights.The crop and transition text are the only prompt additions.Source: §4.2 · p. 8

5.1 SETUP

• The evaluation used ten models with frozen weights on V∗, HR-Bench 4K and 8K, and MME-RealWorld-Lite, with paired sampling seeds and exact McNemar tests.Raw-map runs isolated the effect of sinks on crop selection.Source: §5.1 · p. 8

5.2 RESULTS

• +4.3 points on average across all but one of 40 pairs; gains averaged 5.0 points on V∗ and HR-Bench versus 2.1 on MME-RealWorld-Lite.Table 4 reports baseline accuracy and paired re-look gains.Source: §5.2 · p. 8, Table 4 · p. 9
• Across 28 pairs with both runs, cleaned maps gained 4.1 points versus 2.9 for raw maps; on Qwen3-VL-8B V∗, crops missed targets 10% versus 78%.The cleaned-map advantage came from models with sinks; sink-free models selected nearly the same crops.Source: §5.2 · p. 9
• On V∗ and HR-Bench, Qwen3-VL gained about 3 points on average, versus about 7 for InternVL3.5 and Gemma-4-12B.All Qwen3-VL sizes gained significantly on HR-8K, where the first pass sees about one-third of the pixels, and none did on V∗, where the image fits the budget.Source: §5.2 · p. 9
• On V∗ and HR-Bench, the trigger fired on 60%–78% of Qwen3-VL questions, 79% or more for InternVL3.5, and 91%–96% or more for the other listed models.On MME-RealWorld-Lite, it fired on at least 91% of questions for every model.Source: §5.2 · p. 9

6 CONCLUSION

• A visual attention sink is an image token whose key aligns with the language model’s text-sink key; removing that direction returns attention to the text-sink without changing model output.The paper describes cleaning as a single cheap key edit.Source: §6 · p. 9
• The re-look crops the cleaned map’s most-attended region at a high-entropy thought step and inserts it into the trace, improving accuracy where raw-map crops do not.The authors state that cleaning costs almost nothing and can provide a grounding signal inside training pipelines at no additional cost.Source: §6 · pp. 9–10

AI DISCLOSURE

• Generative AI tools were used to polish author-written text and to support retrieval and discovery of related work.References surfaced by AI-assisted search were independently located, read, and verified.Source: AI DISCLOSURE · p. 10
• The authors state that they did not use generative AI for research ideation or execution.Source: AI DISCLOSURE · p. 10
• The authors say they reviewed all AI-assisted work and take responsibility for the final content and artifacts.Source: AI DISCLOSURE · p. 10

• Attention measurements use each model’s specified prompt and image budget, so reported numbers depend on that prompt.The V∗ image is about 3,300 tokens for Qwen3-VL and an HR-Bench image about 12,400; the stated greedy, letter-only accuracies are not comparable with chain-of-thought accuracies.Source: §A.1 · p. 15
• The text-sink position depends on the backbone, and its share of question attention is .62 to .77 in full-attention models except Pixtral-12B at .49.Gemma-4 has only seven or eight full-attention layers, where the sink takes .10 to .22 of question attention.Source: §A.1 · p. 15
• Seven models have strong visual sinks, while three are classified as sink-free under the peak-layer rule; sink shares range from a quarter to four fifths in sink-bearing models.Qwen3-VL-2B has a 12% share, and the other two sink-free models have no signature or only a faint version.Source: §A.1 · pp. 15–16
• Replacing the V∗ question with a generic prompt leaves the late-layer sink set almost unchanged, with Jaccard overlap of .73 to .92.Source: §A.1 · p. 16
• Sink depth differs by family: Qwen3 sinks persist from near half depth, Gemma-4 sinks appear only in sliding-window layers, and Pixtral-12B’s register persists across nearly all layers.Sinks collapse in the last layer for every model except Pixtral-12B; its register retains a third of in-image attention there.Source: §A.1 · p. 16, Figure 5 · p. 16
• At peak depth, raw-map localization is .00 in six of seven sink-bearing models and .14 in InternVL3.5-8B, but five models reach .53 to .82 at their best layer.Gemma-4-E2B reaches .24 at its best layer; Pixtral-12B’s register stays on top at every layer.Source: §A.1 · p. 16

A.2 THE FIRST QUARTER OF DEPTH

• In the first quarter of depth, localization is weak: the most-attended token lies in the target box in 0 to 9% of images, despite tokens above 100× uniform appearing in every model.Without the peak-layer rule, four models would peak in the first quarter; the per-layer gate and mid-depth selector leave the method unaffected.Source: §A.2 · pp. 16–17
• Gemma-4’s first-quarter sliding-window sinks have centered cosine below .4 and are not flagged by the gate in Gemma-4-12B; whether the direction carries through the residual stream is untested.The selector does not read Gemma-4 sliding-window layers.Source: §A.2 · p. 17

A.3 DETECTORS

• The centered key cosine is the only feature with AUC at least .99 in every sink-bearing model, and it also names individual sinks in full-attention families.In Gemma-4, AUC remains high but precision is low; other tokens can rank above sinks.Source: §A.3 · p. 17
• Hidden-state and value norms detect sinks in Qwen3 backbones and Pixtral-12B but fail in Gemma-4; key norm and logit spread are also family-limited.Key norm is unusually small for sinks in some models, while logit spread separates them only in Pixtral-12B; the three sink-free models do not share one centered-cosine pattern.Source: §A.3 · pp. 17–18

A.4 KEY INTERVENTIONS

• Projecting out the text-sink direction removes sink attention in the three Qwen3 backbones, while a random direction does not; aligning random keys captures nearly all in-image attention.For Pixtral-12B, moving the register’s component to the population median removes its share (.49 to .00), whereas projection to zero only halves it (.49 to .25).Source: §A.4 · p. 18, Table 7 · p. 18

A.5 GEMMA-4: WHAT THE SLIDING WINDOW CANNOT REACH

• When Gemma-4’s whole prompt fits inside the sliding window, .37 to .49 of question attention goes to text before the image and the strongest image token’s absolute attention drops three- to five-fold.The 100×-uniform token count is not comparable across budgets because both the uniform-share threshold and image coarseness change.Source: §A.5 · p. 19, Table 8 · p. 19

A.6 HEADS AND THE GATE THRESHOLD

• Heads with higher sink share have lower target attention in every sink-bearing model; heads that sink less can localize at 10–30× uniform.The target box covers 0.3% of the image; heads devoting 0.8 of image attention to sinks put the remaining 0.2 on the target at 0.1× uniform, not the possible 50×.Source: §A.6 · p. 19
• In full-attention families, the gate has 1.0 recall and .6 precision on Qwen3-VL and .8 on InternVL3.5 from thresholds 0.4–0.9; below 0.4, precision collapses.At 0.5 it admits about seven tokens per Qwen3-VL-8B image, including three aligned tokens below the 100× sink cutoff that the authors also want removed.Source: §A.6 · p. 19, Figure 6 · p. 20

A.7 WHAT THE SINKS CONTRIBUTE TO THE OUTPUT

• In full-attention families, sinks take .29–.77 of in-image attention but supply .03–.16 of output norm; Gemma-4 sinks supply the dominant output share and their removal rotates it.Full-attention sink values have under a tenth of the image-median norm and .7–.8 cosine with the text-sink value direction; Gemma-4’s sliding-window layers cannot reach the text-sink.Source: §A.7 · p. 20

A.8 MASKING INSTEAD OF EDITING

• Masking raw-map tokens recovers less than half the key edit’s localization gain in Qwen3 backbones and can mask target tokens, while the key edit improves Gemma-4 full-attention localization.For Gemma-4 full-attention layers, the key edit raises localization to .17–.31; the 10× mask lowers it to .04–.08.Source: §A.8 · pp. 20–21, Table 10 · p. 21

A.9 REDISTRIBUTION INSTEAD OF REMOVAL

• Redistributing sink attention raises target attention by up to 2.6× on Qwen3-VL-8B and doubles or triples it on InternVL3.5-8B, but accuracy stays flat and outputs shift more than with the key edit.Source: §A.9 · pp. 21–22

A.10 THE EDIT ALONG A CHAIN OF THOUGHT

• Across greedy traces of up to 192 tokens, teacher-forced KL stays at a few 10^-3 nats, about an order of magnitude above the kernel control, with no upward trend.Source: §A.10 · p. 22, Figure 7 · p. 22

A.11 THE 2B MODELS

• All-layer editing costs Qwen3-VL-2B seven points on V*; restricting its edit to the upper three quarters restores accuracy to .65, within run-to-run variation.The layer-2 gate admits weakly aligned tokens whose shift creates a new concentration near 1,000× uniform; the authors recommend depth restriction for small models.Source: §A.11 · p. 22

A.12 THE SAME MEASUREMENTS ON HR-BENCH

• The HR-Bench replication uses the first 50 distinct images from each resolution set; the two sets’ 800 questions share 151 images and have no target boxes.Source: §A.12 · p. 22, Table 11 · p. 23
• Qwen3-VL-2B’s peak sink share rises to .17 and .20 on HR-4K and HR-8K, crossing the 15% sink-bearing cutoff and accompanying larger all-layer edit losses.Source: §A.12 · p. 23
• The centered key cosine retains near-perfect detector AUC on every sink-bearing model, and sink output shares match V* within .01, including Gemma-4’s large contribution.Source: §A.12 · p. 23
• On four sink-bearing full-attention models, the key edit clears the map and leaves answers alone; freed attention raises the text-sink share by .10–.21.The exceptions are Gemma-4, whose map is unchanged, and Qwen3-VL-2B, whose degradation grows with input length.Source: §A.12 · p. 23, Table 12 · p. 23

B SELECTOR LAYERS

• Qwen3-VL-8B’s selector uses layers 18 and 21, where overlap between maps from real and unrelated prompts is lowest; other full-attention models use matching relative depths.The sweep covers 150 images across five benchmarks, and its layer choice predates the main experiments.Source: B SELECTOR LAYERS · p. 24, Table 13 · p. 24
• Gemma-4’s selector layers are chosen using 60 V* evaluation images, so its V* results are not fully held out; HR-Bench and MME-RealWorld-Lite remain held out.Source: B SELECTOR LAYERS · p. 24
• At a selector layer, every full-attention model but Pixtral-12B localizes the top-attended token in the target box on .68–.84 of V* images; Pixtral-12B reaches .37 at layer 20.Pixtral-12B’s best layer reaches .45 at layer 19; the selected values are within .05 of the best layer past the first quarter for the other full-attention models.Source: B SELECTOR LAYERS · p. 24
• Even after editing, late-layer top-attended tokens land on grid borders several times more often than chance, especially in Pixtral-12B; the paper does not correct for this.Source: B SELECTOR LAYERS · p. 24

C EXPERIMENTAL DETAILS

• Prompts use benchmark questions and answer options verbatim, while thinking behavior varies by model family; image inputs range from Qwen3-VL’s 16,384 patches to Gemma-4’s 1,120 soft tokens.InternVL3.5 uses its R1 system prompt, Gemma-4 enables thinking in its template, and InternVL3-8B and Pixtral-12B receive step-by-step instructions.Source: C EXPERIMENTAL DETAILS · pp. 24–25, Figure 9 · p. 25
• The re-look triggers at entropy 1 nat after at least 32 thought tokens, selects a crop using two mid-depth layers, and applies the key edit only during crop selection.Gemma-4 uses two full-attention layers without the edit; the crop window spans a third of the grid on each side with 20-pixel padding.Source: C EXPERIMENTAL DETAILS · p. 25
• Six of ten models trigger on more than 90% of questions on every benchmark, making the method close to an unconditional re-look on those models.The study reports no always-trigger or random-step control.Source: C EXPERIMENTAL DETAILS · p. 26, Table 14 · p. 26
• Across 30 model-benchmark pairs, the cleaned-map re-look wins 1,997 discordant questions and loses 1,181 against the base run, a 1.7 ratio.On Qwen3-VL-8B, cleaned-map crops hit .90 of triggered V* questions versus .22 for raw-map crops.Source: C EXPERIMENTAL DETAILS · p. 26, Table 15 · p. 27