Segment Where You Look: Attention as the Interface Between Language Models and Segmenters

1 INTRODUCTION

• The dominant [SEG] interface compresses the spatial message into one d-dimensional vector, leaving a decoder without the expression to infer the referent’s granularity.The released Sa2VA-4B scores 90.8 gIoU on stuff, 49.8 on single objects, and 17.8 on object parts.Source: §1 · p. 2
• With the same U-Net probe, [SEG] attention yields 18.2 more gIoU on GS-Eval and 21.5 more on ReasonSeg than the hidden state.Adding RGB improves the hidden-state probe by 4 to 7 gIoU but leaves the attention probe unchanged; localization is concentrated in late layers.Source: §1 · p. 2
• ATTNSEG replaces the [SEG] embedding with a learned, signed combination of its attention maps as a dense decoder prompt, directly supervised by attention guidance.Training first teaches the VLM to localize without the decoder, then trains the decoder to refine the frozen map, within a standard fine-tuning budget.Source: §1 · pp. 2–3
• At 4B, ATTNSEG improves ReasonSeg by 6.4 gIoU and GS-Eval by 10.2; at 14B, the gains are 8.8 and 11.3.At 4B, it also improves by 4.2 and 9.6 over the same model fine-tuned on the same data for the same budget; gains are 13 to 18 gIoU on parts, multiple objects, and single objects, versus under 1.5 on stuff.Source: §1 · pp. 2–3
• The paper’s probing analysis finds that [SEG] attention carries more localization than the hidden state and that this signal is concentrated in late layers.The contribution summary describes the attention-guidance objective as accounting for most of the gain and making attention directly readable.Source: §1 · pp. 2–3

2 RELATED WORK

• Text-mask methods serialize masks as polygons, codewords, or patch labels, while LISA prompts SAM with a [SEG] hidden state (Lai et al., 2024).These text-mask methods trail decoder-based systems in pixel accuracy; Text4Seg gains 5.7 cIoU on RefCOCO when SAM refines its labels (Lan et al., 2025b).Source: §2 · p. 3
• Prior attention analyses report that localization can reside in a few heads, while other attention mass falls on sink tokens (Kang et al., 2025a; Kang et al., 2025b).These findings motivate ATTNSEG’s selective learned aggregation.Source: §2 · p. 3
• VL-SAM (Lin et al., 2024) and DecAF (Han et al., 2025) sample attention-derived points, while READ (Qian et al., 2025) samples points from an embedding-space similarity map.ATTNSEG differs by using the [SEG] token’s own attention, preserving it as a dense map, and training the VLM to localize with it.Source: §2 · p. 3

3.1 PROBLEM SETUP AND HYPOTHESIS

• The [SEG] vector can point to a referent but cannot specify its granularity, leaving the decoder—which never sees the expression—to infer whether an object or its part is meant.This motivates the hypothesis that compressing the referent into the hidden state discards spatial information the VLM has computed.Source: §3.1 · p. 4

3.2 WHAT THE MODEL ALREADY KNOWS, AND WHERE IT KEEPS IT

• 18.2 and 21.5 more gIoU: a U-Net probe reads more localization from [SEG] attention than from the hidden state on GS-Eval and ReasonSeg.The comparison uses the same probe on a frozen Sa2VA-4B.Source: Figure 2 · p. 5, §3.2 · p. 5
• Adding RGB improves the hidden-state probe by 4.4 and 7.3 gIoU but changes the attention probe by −0.3 and +0.2, consistent with attention already encoding referent extent.The image is supplied as an additional U-Net input.Source: §3.2 · p. 5
• Attention localization is concentrated late: roughly the first 40% of layers carry almost none, while one late five-layer window matches a probe using every layer.The transition appears at the same depth for both probe capacities.Source: §3.2 · p. 5

3.3 ATTNSEG: SEGMENTING WHERE THE MODEL LOOKS

• ATTNSEG folds the [SEG] token’s image attention onto the patch grid, combines per-head maps with learned signed weights, and uses the resulting dense map as the decoder prompt.The map is not thresholded or sampled; negative weights can subtract misleading heads.Source: §3.3 · pp. 5–7, Figure 3 · p. 6
• Two-stage training first tunes the VLM and map weights with attention guidance, then freezes them while training the decoder to refine the map.The first stage excludes decoder mask loss from the map.Source: §3.3 · p. 6
• Attention guidance supervises the patch-grid map against an area-pooled mask and favors the referent’s full extent rather than only its peak location.The weighted cosine penalizes concentrated off-target responses more than diffuse background activity; target weights also vary with area.Source: §3.3 · pp. 6–7

4.1 EXPERIMENTAL SETUP

• The evaluation uses two Sa2VA scales and the same thirteen-dataset mixture for both training stages; their combined budget equals one epoch of baseline finetuning.ReasonSeg tests referring expressions, while GS-Eval varies target granularity; neither benchmark's training data is in the mixture.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

• At 4B, ATTNSEG gains 6.41 gIoU on ReasonSeg and 10.20 on GS-Eval; at 14B, gains are 8.83 and 11.26, respectively.Against the 4B base model finetuned on the same data and budget, gains are 4.19 and 9.56 gIoU.Source: §4.2 · pp. 8–9, Table 1 · p. 8
• At 4B, gains are 17.49, 13.04, and 13.17 gIoU on object parts, multiple objects, and single objects, versus 0.40 on stuff.At 14B, the corresponding gains are 18.04, 15.42, and 13.76, versus 1.40 on stuff.Source: §4.2 · pp. 8–9, Table 1 · p. 8
• On the same Qwen3-VL-4B base, SAM3 raises ReasonSeg from 58.67 to 64.50 but lowers GS-Eval from 65.41 to 60.29; ATTNSEG improves both benchmarks over its base.ATTNSEG on InternVL2.5-4B also exceeds SAMTok by 5.03 gIoU on ReasonSeg and 10.29 on GS-Eval.Source: Table 1 · p. 8, §4.2 · p. 9

4.3 ANALYSIS

• Weighted cosine beats Dice by 2.20 and 2.55 gIoU and peak ratio by 3.98 and 6.12 on ReasonSeg and GS-Eval, respectively.The cosine balances pointing and extent, while peak ratio points well but specifies little extent and Dice specifies extent but points worst.Source: §4.3 · p. 9, Table 2 · p. 9
• Attention guidance contributes 3.93 and 8.47 gIoU on ReasonSeg and GS-Eval, respectively—61% and 83% of total gains—while the attention interface alone adds less.The interface's value is that the map shaped by attention guidance is the map the decoder receives.Source: §4.3 · pp. 9–10, Table 3 · p. 9
• Figure 4 shows each query's aggregated attention map beside the mask refined from it, allowing direct comparison of the decoder's prompt and output.The map is the decoder's only prompt; further examples are in Appendix G.Source: Figure 4 · p. 10, §4.3 · p. 10
• After training, a 560-parameter gate gains 18.04 gIoU on ReasonSeg and 17.86 on GS-Eval, while the U-Net gains 1.91 and 3.95.The gate can only reweight existing attention maps, and adding the RGB image to the trained attention probe adds just 0.14 and 0.30 gIoU.Source: Table 4 · p. 9, §4.3 · p. 10

5 CONCLUSION

• At both model scales and on both benchmarks, gains concentrate on object parts, multiple objects, and single objects, while remaining negligible on stuff regions.The conclusion attributes this pattern to the need to single out the referent in the former categories and says the decoder recovers stuff regions unaided.Source: §5 · p. 10
• ATTNSEG replaces the [SEG] embedding with a learned, signed combination of [SEG] attention maps, directly supervised by an attention-guidance objective.It uses a two-stage recipe in which the VLM localizes and the decoder refines, within the budget of standard fine-tuning.Source: §5 · p. 10
• Probing a frozen segmentation VLM found more localization in the [SEG] token’s attention than in its hidden state, concentrated in late layers.The conclusion argues that the decoder should receive the localization computation in the form in which the model makes it, and that it should be supervised there.Source: §5 · p. 10

AI USE STATEMENT

• Generative AI tools helped draft and polish text, including redrafting author-supplied content, correcting grammar and typography, and condensing sections.The authors also report using the tools to debug and clean up parts of the code.Source: AI USE STATEMENT · p. 11
• AI assistance was limited to writing support and help with parts of the implementation, according to the authors.The statement names drafting, editing, debugging, and code cleanup as the reported tasks.Source: AI USE STATEMENT · p. 11
• The authors state that no AI system is an author and that they reviewed all AI-assisted text and code.They take full responsibility for the final content, including every claim, number, and artifact.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

• The study uses public datasets and benchmarks as released, collects no new data, and involves no human subjects.Both models it builds on are publicly released checkpoints.Source: ETHICS STATEMENT · p. 11
• Moving localization to the language model means its referent-selection errors and biases are inherited, and a confidently wrong attention map may be harder for the decoder to correct.The authors say benchmark gIoU improvements do not establish correct handling of ambiguous or contested descriptions and call for task-specific validation before safety-critical deployment.Source: ETHICS STATEMENT · p. 11
• Segmenting arbitrary objects from free-form language lowers the barrier to fine-grained visual search and carries dual-use considerations.Source: ETHICS STATEMENT · p. 11
• The authors report that the interface substitution alone contributes almost nothing to the end task and that a stronger-decoder baseline exceeds ATTNSEG at 4B on ReasonSeg.They also report computational cost added by the method and describe how they reduce it.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The training configuration is identical across model scales and stages, uses no per-benchmark tuning, and applies the same seed to every run.It uses a thirteen-dataset Sa2VA mixture, LoRA on the language model, AdamW, a cosine schedule with warmup, gradient clipping, bfloat16, and half of a single epoch per stage.Source: REPRODUCIBILITY STATEMENT · p. 11

A EXTENDED RELATED WORK

• SAM’s dense prompt channel accepts a soft, full-field spatial distribution, so an attention map can enter without thresholding.The appendix notes that this channel is rarely used by multimodal segmenters, which generally leave it at the learned no-mask default.Source: A EXTENDED RELATED WORK · p. 15
• Text-based mask interfaces face coarse output grids and tokenization errors for spatial deviations, while a SAM refiner can improve Text4Seg’s boundary from its descriptors.Text4Seg++ with an InternVL2-8B backbone reports 75.4 average cIoU on RefCOCO, versus 69.9 for LISA and 75.6 for GLaMM.Source: A EXTENDED RELATED WORK · p. 15

B PROBING PROTOCOL

• The probe appendix compares frozen-VLM attention and embedding signals using probes of increasing capacity, including a gate, convolution, and U-Net.The probes train for three epochs with the decoder unused; GS-Eval and ReasonSeg are out-of-domain for the training mixture.Source: Table 5 · p. 16, B PROBING PROTOCOL · p. 16
• The gate is evaluated with a swept threshold because a fixed 0.5 threshold would make its normalized mask nearly all foreground, so its recovered-accuracy fractions are upper bounds.Other probes use a 0.5 threshold.Source: B PROBING PROTOCOL · p. 16

C ADDITIONAL METHOD DETAILS

• The 1×1 aggregation convolution uses signed, unnormalized weights, allowing misleading attention heads to be subtracted rather than only downweighted.The appendix contrasts this with a softmax gate, which cannot subtract heads and may collapse onto a handful during training.Source: C ADDITIONAL METHOD DETAILS · p. 16
• The peak-ratio objective keeps applying pressure for any finite gap, whereas a fixed-margin objective can stop producing gradients once its threshold is met.With τ = 0.2, the fraction of samples still producing a gradient fell from 0.34 to zero within the first fifth of training.Source: C ADDITIONAL METHOD DETAILS · p. 17
• Stage one trains the language model and aggregation weights with attention guidance; stage two freezes both and trains only the decoder’s mask branch.The normalized attention map becomes the decoder’s only prompt, with no thresholding or sampling; the aggregation weights must be explicitly frozen because they sit outside the language model.Source: C ADDITIONAL METHOD DETAILS · p. 17

• Both stages and model scales use the same thirteen-dataset mixture, sampling factors, and seed; finetuned baselines match its mixture and budget.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 17
• ReasonSeg reports short-phrase, full-sentence, and union results, while GS-Eval separates stuff, object parts, multiple instances, and single objects.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 17
• Both stages use AdamW at 4 × 10^-5, with LoRA rank 128 on the language model and the vision encoder and several other components frozen.Each stage takes half a single epoch; the same settings apply at both model scales.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 17

Experimental setup

• All three objectives use the same two-stage recipe and area weighting, differing only in the stage-one map they inherit.Table 7 compares these objectives per split on Sa2VA-InternVL2.5-4B.Source: E CHOICE OF THE ATTENTION-GUIDANCE… · p. 17, Table 7 · p. 18

Training mixture

• The thirteen-dataset training mixture and its sampling weights are identical across both stages and model scales.Factors above one oversample a dataset; factors below one indicate a seeded subsample.Source: Table 6 · p. 18

Per-split results

• The cosine objective leads the alternatives by 2.4 to 10.4 gIoU on three GS-Eval splits, while objectives tie within 0.3 on stuff.Its advantage over both alternatives is significant on both benchmarks at 95% confidence; the objective ordering persists across four stage-two checkpoints.Source: E CHOICE OF THE ATTENTION-GUIDANCE… · p. 18, Table 7 · p. 18

Where the misplaced attention sits

• The cosine objective leaves more off-target mass than Dice but spreads it more diffusely: the strongest 1% of background patches hold 8.9 to 9.3% versus Dice’s 37 to 38%.Its participation ratio is three to four times higher; the passage characterizes this as redistribution, not removal, of misplaced attention.Source: E CHOICE OF THE ATTENTION-GUIDANCE… · p. 18

Reading the attention

• At 14B, storing all attention weights for backward pass would require roughly 44 GB in bfloat16 at a typical length of 3400.The method recomputes only the [SEG] rows and image columns while retaining fused kernels for layer outputs, reducing stored tensor size by seq/R.Source: F LIMITATIONS: READING THE ATTENTION · p. 18

GS-Eval

• The additional GS-Eval examples show each query alongside its aggregated [SEG] attention map and refined mask.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 19, Figure 5 · p. 20

ReasonSeg

• The additional ReasonSeg examples show each query alongside its aggregated [SEG] attention map and refined mask.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 19, Figure 6 · p. 21

Qualitative results

• The displayed sequence is query, aggregated map, then the mask refined by the decoder.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 19, Figure 5 · p. 20, Figure 6 · p. 21