PaperlayerView exact revision on arXiv

Source-linked AI summary

What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu, Yi-Zhe Song

arXiv:2610.34037v1cs.CV

TL;DR

Standard LVLM benchmarks score answers but do not show which human visual conventions models retain or where those conventions reside. This paper probes frozen models with sketches using temporal, spatial, and mechanistic readouts; part order tracks human convention, attention heads recover regions, and grounding localises to a mid-network band.

  • Problem

    Standard multimodal benchmarks score final answers but leave unresolved which visual conventions LVLMs retain and where those conventions are carried.

    Source: §1 · p. 1

  • Method

    The paper uses sketches as controlled abstractions in a training-free probe of temporal part order, spatial attention-based grounding, and grounding’s mechanistic locus.

    Source: §1 · pp. 2–3

  • Results

    Across the tested models, part order tracks human convention, frozen attention heads recover part and object regions, and grounding heads cluster in base-model-dependent mid-network bands.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sketches serve as diagnostic inputs for what LVLMs retain, while the identified layer band acts as a head-selection prior for grounding.

    Source: §1 · p. 3, §6 · p. 10

    The probes assume queried parts are present, and attention readouts cover only two base language model families.

    Source: §5 · p. 10

Abstract

from arXiv · show

LVLM benchmarks tell us whether a model answers correctly, but not which visual conventions it has retained, nor where those conventions live. Sketches expose this hidden structure. A sketch removes texture, lighting, and photographic context while preserving the strokes humans judged sufficient for recognition. If an LVLM can operate on that abstraction, the surviving signal is structural rather than merely photometric. We introduce a training-free sketch probe with three readouts: temporal part-construction order, spatial attention-based grounding, and the mechanistic layer/head locus of grounding. We evaluate nine LVLMs on part order and five on the attention readouts, over two part-annotated sketch datasets. Three findings emerge. First, LVLM part order tracks human convention: models reproduce canonical orderings where human drawings converge, and become correspondingly variable where human orderings vary. Controls that blank, withhold, and mismatch the sketch locate this convention in the language model. Second, part-level grounding survives the abstraction: attention heads selected from the frozen model with an existing recipe recover part and object regions on object and scene sketches, with no grounding supervision. Third, the grounding signal concentrates in a narrow mid-network band whose location follows the base language model rather than parameter count in the two families we compare, and remains stable across object and scene sketches. The result is a practical diagnostic for sketch-driven LVLMs: no fine-tuning, no auxiliary segmenter, and a layer band that doubles as a head-selection prior, matching or improving grounding mAcc@0.1 in every model and dataset we tested.

1 Introduction

Standard benchmarks score LVLM answers but leave unclear which human visual conventions models retain and where those conventions reside. Using sketches as controlled abstractions, the paper probes temporal part order, spatial grounding, and grounding’s network location.

1 Introduction

  • The probe uses sketches to preserve human-selected structural strokes while removing texture and context, reading temporal, spatial, and mechanistic information from frozen LVLMs.Its components include shuffled-list ordering, the Localization Heads recipe, and classical sequence metrics.Source: §1 · p. 2
  • LVLMs reproduce canonical part orderings where human drawings converge and become more variable where human orderings do not.Controls that blank, withhold, and mismatch the sketch locate this convention in the language model.Source: §1 · p. 2
  • Frozen cross-attention heads recover part and object regions on object and scene sketches without grounding supervision, overlapping target regions more often than trained grounders at coarse and moderate thresholds.At scale, the two readouts couple class by class, with LLaVA-1.5-13B Pearson r = +0.40.Source: §1 · p. 2
  • Grounding heads cluster in a narrow mid-network band whose location follows the base language model across the two compared families and remains conserved across object and scene domains (∆= +0.004).The same heads recur across domains; ablating the three most frequently selected lowers the downstream localisation-answer margin by 0.118, versus 0.010 for same-layer heads.Source: §1 · p. 2
  • The findings suggest sketch-based retrieval, segmentation, editing, and AR/UI pipelines could read region grounding directly from an LVLM rather than adding a dedicated segmenter.The band may also guide backbone selection and provide an intervention point for attention-steering methods.Source: §1 · p. 3
  • The training-free probe’s mid-network head-search band matches or improves grounding mAcc@0.1 across every tested model and dataset.The band’s location follows the base language model in the two families compared.Source: §1 · p. 3

2 Related Work

Prior work studies sketch understanding and model interpretability, but LVLM sketch benchmarks largely treat drawings as a domain shift rather than a probe of internal representation. This paper uses sketches to examine whether internal structure survives abstraction.

2 Related Work

  • Interpretability work covers generative grounding [44] [60], gradient and attribution saliency [13] [53] [62] [64], and attention-based localisation [12] [30].This paper uses attention-based localisation without coordinate decoding, gradients, or an auxiliary segmenter.Source: §2 · p. 3
  • Sketch research spans recognition, retrieval [8] [51], grouping [45], and generation [27], while LVLM line-drawing benchmarks [11] [19] [17] evaluate sketches as a domain shift.Sketch explainability [46], saliency [9], and detection also use drawings, while cognitive studies examine visual concept organisation [25] [41].Source: §2 · p. 3
  • Although sketch studies recover stable construction trajectories, how LVLMs internalise comparable structure remains untested; this work treats sketches as probes rather than evaluation domains.Source: §2 · pp. 3–4

3 Method

Section 3 motivates and formalizes three complementary readouts of frozen LVLMs, then instantiates training-free probes on object and scene sketches. The pilot contrasts temporal ordering with attention-based spatial grounding under increasing abstraction.

3.1 Pilot study: part-construction ordering and grounding under abstraction

  • Under increasing abstraction, spatial grounding accuracy declines while temporal structure and broad attention shape are preserved, motivating separate probes for these signals.The pilot reports that attention masks track ground truth on less abstract inputs but degrade as texture and context are removed.Source: §3.1 · p. 4

3.2 Problem formulation and method overview

  • The framework reads temporal order, spatial grounding, and the layer/head locus of grounding from forward passes of a frozen LVLM, without fine-tuning or auxiliary networks.The input pairs a sketch with a target part or object; attention is read across image patches.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Sketch-based probes of part-construction ordering and grounding

  • The temporal probe asks the model to order a shuffled part list as drawn and compares its trajectory with human sketches using edit distance, longest common subsequence, and pairwise-order agreement.The parts are not re-rendered during the protocol, and the human-order comparison is limited to objects.Source: §3.3 · p. 5
  • The spatial probe scores heads by visual attention mass and spatial concentration, combines the top three into a thresholded mask, and evaluates it against part or object masks.The mechanistic readout records selected head locations and per-sample spatial entropy; the probes use no fine-tuning, auxiliary heads, or intermediate rendering.Source: §3.3 · p. 5

4 Experiments

The experiments use complementary temporal and attention-based probes across model sets chosen for their access requirements. They find that sketch-order agreement tracks human consensus, grounding is recoverable from frozen attention, and its useful layer band varies by language-model family.

4.1 Model sets, protocols, and why they differ across probes

  • Temporal probing covers nine models, while attention grounding and mechanistic analysis use five open models across two visual stacks and two language-model families.The temporal protocol uses deterministic decoding and 100 strict-mode-matching sketches per category; attention readouts use a transferred Localization Heads recipe and a single forward pass without fine-tuning.Source: §4.1 · p. 6

4.2 Main results

  • Across nine models, per-category part-order agreement correlates with human consensus (mean Pearson r = +0.58), significant in eight models.Canonical categories elicit near-unanimous orders, while categories with lower human consensus show correspondingly lower LVLM agreement.Source: §4.2 · pp. 6–7, Figure 4 · p. 6, Figure 5 · p. 8
  • Blanking, withholding, or mismatching the sketch does not improve temporal ordering: none of 18 comparisons favours the real sketch, locating the convention in the language model.The only significant comparison favours no image (Pair, p = 0.04).Source: §4.2 · p. 7, Table 3 · p. 8
  • LLaVA-1.5-13B reaches 0.73 mAcc@0.1 on SPG and 0.78 on FS-COCO, exceeding trained grounders on mAcc@0.1 while trailing CLIPSeg on pointing accuracy.The caption identifies Table 2 as the comparison of pointing accuracy, mAcc@0.1, and spatial entropy on object and scene sketches.Source: §4.2 · p. 7, Table 2 · p. 7
  • The grounding band is narrow and mid-network, at fractional depth 0.35–0.45 for Vicuna models and 0.50–0.65 for DeepSeek models, with a +0.004 mean shift across object and scene sketches.The same heads recur across domains; combined spatial entropy is concentrated between 0.78 and 0.95.Source: §4.2 · pp. 8–9, Figure 6 · p. 9

4.3 The layer band is predictive, not merely descriptive

  • Restricting head selection to the family-specific band matches or exceeds full-grid scoring in every model–dataset cell, with the largest gains on smaller models.LLaVA-1.5-7B gains +0.08/+0.09 mAcc@0.1 on SPG/FS-COCO; high-scoring heads outside the modal range generalise less reliably.Source: §4.3 · p. 9, Figure 7 · p. 10
  • Ablating the three most frequently selected LLaVA-1.5-13B heads lowers the correct-side logit margin by 0.118, versus 0.010 for three heads from the same layers.Side accuracy barely moves; the layer-matched control ties the margin effect to the selected heads rather than depth.Source: §4.3 · p. 10
  • Layer-band location follows model family rather than parameter count: within-family SD is 0.049 for Vicuna and 0.040 for DeepSeek, versus a 0.175 cross-family gap.The DeepSeek pair reverses the Vicuna scaling pattern on SPG, with the smaller model outperforming the larger; modal layers are identical across SPG and FS-COCO to three decimal places.Source: §4.3 · p. 10, Figure 7 · p. 10

5 Discussion

The readouts reveal structure hidden by aggregate scores: convention-following varies with human consensus, and grounding failures are graded; the probes also have explicit scope limits.

5 Discussion

  • Agreement rises and falls with human consensus, making variability in ambiguous categories informative rather than simply a lower score.DeepSeek-VL-7B trails DeepSeek-VL-1.3B on object grounding, although both place their grounding heads in the same band.Source: §5 · p. 10
  • Under abstraction, models become wrong about part location before their attention becomes diffuse.Source: §5 · p. 10
  • The probes assume queried parts are present and do not signal when a named part is absent, so pipelines need a separate presence check.Attention readouts cover only two base language model families; further families remain future work.Source: §5 · p. 10

6 Conclusion

The conclusion presents sketches as a controlled abstraction test and a training-free diagnostic of what LVLMs retain and where grounding signals reside.

6 Conclusion

  • The probe combines temporal order, frozen-attention grounding, and mechanistic readouts to test LVLMs on sketches stripped of texture and context.Part order tracks human convention where stable and is carried by the language model; frozen attention heads recover part regions without grounding supervision.Source: §6 · p. 10
  • A narrow mid-network band set by the base language model acts as a head-selection prior and improves grounding without retraining.The conclusion also reports that the temporal and grounding readouts couple at scale.Source: §6 · p. 10
  • Sketches serve as diagnostic inputs rather than domain-shift benchmarks, exposing what LVLMs retain, when it becomes unreliable, and which layers carry it.Source: §6 · p. 10

A Implementation details

The appendix specifies the temporal prompt, the attention-to-mask grounding pipeline, and the frozen-model compute setup.

A.1 Temporal probe prompt

  • The temporal probe presents part names in a fixed shuffled order, parses the response’s order field, and matches returned names after normalising case, underscores, and hyphens.The prompt asks for earliest-to-latest drawing order, restricts answers to the supplied parts, and requires each part exactly once.Source: §A.1 · p. 16

A.2 Grounding pipeline

  • Grounding masks combine the top three attention maps after processing and threshold their maximum at the 80th percentile, covering the top 20% of patches.Ground truth is the SPG part-stroke convex hull or the FS-COCO-Seg object mask.Source: §A.2 · p. 16

A.3 Compute

  • Experiments use frozen-model forward passes on one NVIDIA A100 GPU, with closed-source models queried through vendor APIs and no training or fine-tuning.Source: §A.3 · p. 16

B Human consensus and LVLM agreement

Across SPG categories, model agreement with human part-order trajectories generally increases with human consensus, while an independent model-consensus check shows a similar association.

B Human consensus and LVLM agreement

  • Eight of nine models show a significant positive correlation between human consensus and per-category LCS; their high- versus low-consensus agreement gap is 0.09 to 0.24 LCS.LLaVA-1.5-7B is the exception, with agreement at random-permutation level; Table 5 reports the human-consensus/LVLM-agreement correlation over 25 categories.Source: B Human consensus and LVLM agreement · p. 16, Table 5 · p. 17
  • Model-prediction consensus correlates with human consensus in all nine models (r = +0.35 to +0.79), significantly in eight.With 25 categories, both analyses are modestly powered and are presented as evidence rather than proof.Source: B Human consensus and LVLM agreement · p. 17

C Image controls for the temporal probe

The controls test whether temporal part-order predictions depend on the sketch or on conventions retained by the language model. Neither blank, no-image, nor mismatched-sketch conditions underperform the real sketch, and the deviation and forced-choice tests provide little evidence that models use sketch-specific order.

C Image controls for the temporal probe

  • None of the 18 control comparisons favours the real sketch; the only significant difference favours no-image for Qwen2.5-VL-7B on Pair (p = 0.04).Each control uses the same shuffled parts and true-category prompt; the comparisons cover two models, three controls, and three metrics.Source: C Image controls for the temporal probe · p. 17
  • On deviation-set sketches, predictions shift toward the sketch’s own order per sketch, but not after clustering by category (p = 0.26, 17 categories).These sketches retain the modal-order parts but change their drawing order, so the image changes along with the correct answer.Source: C Image controls for the temporal probe · p. 17
  • Qwen2.5-VL-7B chooses between matched candidate orderings at chance (50%) both with and without the sketch.The candidates are matched for edit distance from the modal order, making convention-only reasoning unable to prefer either.Source: C Image controls for the temporal probe · p. 17

D Grounding at stricter overlap

At mAcc@0.1, LLaVA-1.5-13B exceeds both trained grounders on both datasets, with significant paired comparisons. Grounding-head selections also show closely aligned depth and substantial overlap between object and scene sketches, while stricter-overlap evaluation is sensitive to mask coverage.

D Grounding at stricter overlap

  • At mAcc@0.1, LLaVA-1.5-13B is above both trained grounders on both datasets, and all four comparisons are significant (p < 0.002).At mAcc@0.3 it remains above both, significantly in three of four comparisons; at mAcc@0.5 CLIPSeg is nominally ahead on both datasets and GroupViT on SPG, with no significant differences.Source: D Grounding at stricter overlap · p. 17, Table 6 · p. 18
  • At the 80th-percentile threshold, LLaVA-1.5-13B’s mask has precision 0.35, recall 0.54, and about 1.5 times the ground-truth area; lowering the threshold raises mAcc@0.5.The authors attribute the strict-overlap shortfall to under-coverage at the fixed threshold; fixed-area LVLM masks and larger, smoother SPG convex-hull ground truth also affect evaluation.Source: D Grounding at stricter overlap · p. 18

E Object and scene head overlap

  • Across object and scene sketches, every model has identical modal head-selection depth to three decimal places, and an average of 5.4 top-ten heads are shared.The comparison concerns selection distributions rather than paired queries; unrelated top-ten sets would share fewer than one head in expectation.Source: Table 7 · p. 18, E Object and scene head overlap · p. 18

F Head ablation

The ablation tests whether selected attention heads carry localization-relevant signal by measuring changes in a correct-side logit margin after head removal. The top-three selected heads reduce that margin more than random or same-layer alternatives, although side accuracy barely changes.

F Head ablation

  • The margin is the correct side’s next-token logit minus the other side’s, with the correct side defined by the ground-truth part-mask centroid.Parts whose centroid lies within 0.12 image-width of the midline are excluded; logits are read without generating a response.Source: F Head ablation · p. 18
  • Ablating the top three selected heads reduces the correct-side logit margin more than random heads (p = 0.0002) or layer-matched heads (p = 0.0003).Comparisons use per-category mean margins against the unablated model, with a Wilcoxon signed-rank test over 24 categories and a 95% bootstrap interval.Source: F Head ablation · pp. 18–19, Table 8 · p. 19
  • Side accuracy barely changes under any ablation, while the margin indicates that selected heads carry signal not carried by other heads at the same depth.The majority-class rate is 0.586, and three of 1,600 heads are too few to flip many decisions.Source: F Head ablation · p. 19, Table 8 · p. 19

G Phantom parts

Phantom-part queries show that the probe localizes named parts without checking whether they exist. In a separate newer-model comparison, Qwen3-VL improves over Qwen2.5-VL more at the smaller scale, while its two tested sizes are nearly indistinguishable.

G Phantom parts

  • The model never signals that queried phantom parts are absent, and its answers give them locations like those given to real parts.Examples include locating an airplane’s “bell” at the top of its wing and an alarm clock’s “stamens” at the top of the clock.Source: G Phantom parts · p. 19
  • The probe localizes a named part but does not verify its presence, so pipelines where parts may be absent need a separate presence check.Source: G Phantom parts · p. 19

H Newer open models

  • Qwen3-VL-4B gains +0.16 Edit and +0.20 Pair over Qwen2.5-VL-3B, while Qwen3-VL-8B gains +0.03 on both metrics over Qwen2.5-VL-7B.The two Qwen3-VL sizes are almost indistinguishable; Qwen3-VL-4B matches or exceeds GPT-5 (nano) and Gemini 2.5 Flash-Lite on all three metrics, and narrows the gap to Claude Haiku 4.5 to 0.04 LCS.Source: Table 9 · p. 19, H Newer open models · p. 19

Related papers

Loading 2610.34037v1…