PaperlayerView exact revision on arXiv

Source-linked AI summary

What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit

Alfredo F. Frontera Del Valle

arXiv:2609.27408v1cs.CVcs.AIcs.CL

TL;DR

Benchmarks report model scores without recording the answer convention, although conventions can change measured outcomes. This paper holds the image, question, and alternatives fixed while changing how choices are described, and tests whether models read those conventions. On COCO location selection, accuracy differs sharply between English cell names and pixel coordinates.

  • Problem

    Benchmarks treat the answer convention as neutral even though they report scores as properties of models without recording that convention.

    Source: §1 · p. 1

  • Method

    The paper holds the image, question, candidates, order, and output format fixed while swapping candidate descriptions, and uses wrong-name probes to test whether models follow the convention.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across color and location evaluations, conventions affect scores and can reverse rankings; on 200 COCO photographs, Qwen3-VL-4B scores 68.5% with English cells versus 20.0% with pixel coordinates.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Report the convention alongside a localization score and compare the same alternatives expressed in plain language before interpreting differences.

    Source: §10 · p. 9

    Real-image evidence is limited to COCO and coarse grid-cell selection, not free-form box grounding or box regression.

    Source: §9 · p. 9

Abstract

from arXiv · show

Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is treated as neutral. We find it is not, and that the limits a benchmark reports can belong to the readout rather than to the model. On 200 COCO photographs, Qwen3-VL-4B picks the correct one of nine locations for a named object 68.5% of the time when the locations are given in English and 20.0% when the same locations are given as pixel coordinates. Chance is 11.1%. The cost arises when the answer options are coordinates; giving the model a coordinate in the question instead costs 3.5 points and is not significant. The gap holds on a 4x4 grid, under 8-bit rather than 4-bit quantization, and in every slice by object size, boundary distance and category. It also decides which model wins. Two models that tie under English names differ by 39 points in one coordinate system and by 54 in the other, in opposite directions. On the color task, three of the four open models capable of the task show the penalty; on photographs, two of three open models do, and so does Gemini, at 11.1 points on parseable answers (p = 1e-4). GPT-4o does not. To ask whether a model reads a coordinate at all, we attach the wrong name to each one and record which the model follows. Color options written as hue angles are followed below chance; a normalized pixel convention is followed at four times chance. This tells apart conventions a model can use from ones it cannot, though it did not predict accuracy on two untried conventions. Five models also name the same color wheel five different ways, so a fixed answer vocabulary is not neutral across models either. Five times during this work we measured a capable model as incapable because our scorer and the model disagreed about what an answer looks like. We report each case. They are the phenomenon in miniature.

1 INTRODUCTION

Answer conventions are not neutral: on the same COCO photographs, Qwen3-VL-4B’s location accuracy changes sharply when answer choices switch from English cell names to pixel coordinates. The convention can also reverse model rankings, and the paper tests whether models read a convention rather than merely follow its associated name.

  • 68.5% with English cell names versus 20.0% with pixel coordinates: Qwen3-VL-4B’s accuracy on 200 COCO photographs choosing among nine locations.Chance is 11.1%; the convention alone accounts for 48.5 points.Source: §1 · p. 1
  • Putting a pixel coordinate in the question instead costs 3.5 points and is not significant; the larger penalty occurs when coordinates are answer options.A model scoring poorly on coordinate answers may still know where the object is.Source: §1 · pp. 1–2
  • Qwen2.5-VL-7B and Qwen3-VL-4B tie under English cell names, but pixel and normalized coordinates produce significant leads in opposite directions.Their English-name scores are 68.0% and 68.5%; the coordinate-system leads are 39 and 54 points, respectively.Source: §1 · p. 2
  • Wrong-name probes show hue angles followed below chance and normalized pixels followed four times more often than chance; readership distinguishes readable conventions but did not predict accuracy on two untried conventions.Source: §1 · p. 2
  • Five models give a calibrated color wheel five different stable vocabularies, so a fixed set of color names is not neutral across models.Source: §1 · p. 2
  • The convention swap is applied to color and location tasks, while frontier-model replication finds the gap in Gemini but not GPT-4o; the paper also reports five scorer/model disagreements.Source: §1 · p. 2
  • The authors do not claim that all benchmarks are wrong or that every coordinate system is unreadable.Source: §1 · p. 2

2 RELATED WORK

Prior work studies answer modality, question phrasing, option-selection bias, and coordinate grounding; this paper holds the text interface fixed while changing how alternatives are described. Its method calibrates each model’s answer vocabulary, tests matched conventions and whether they are read, and gates and evaluates the comparisons on controlled data.

2 RELATED WORK

3 METHOD

  • The convention swap changes only candidate descriptions while holding the image, question, candidate set, order, and output format fixed.The method also uses a model-calibrated response vocabulary, a capability gate, and a held-out split.Source: §3 · p. 3

3.1 CALIBRATING THE RESPONSE VOCABULARY

  • The model-specific color vocabulary retains angles consistently named across three renderings, using anchors at centers of unanimous runs separated by at least 40°.Models produce different stable vocabularies; candidate token surfaces are probed, and a restricted first-token readout is refused if candidate probability mass never exceeds 0.5.Source: §3.1 · p. 3

3.2 THE CONVENTION SWAP

  • Each trial presents the same K referents and changes only their surface descriptions, measuring digit-emission cost separately from convention cost.Emitting a digit rather than a word costs 1 point.Source: §3.2 · p. 4

3.3 VISIBLE AND CACHED CONDITIONS, AND THE CUE CROSSOVER

  • Color trials use visible and cached arrays; a cue crossover swaps two locations’ colors to test whether cached answers depend on the cued binding.The untouched location should show no shift, while a location-blind account predicts no sign reversal.Source: §3.3 · p. 4

3.4 THE MISPAIRED GLOSS, AND READERSHIP

  • The mispaired-gloss test breaks every coordinate–name pairing, then scores each response against the correct-coordinate and correct-name options.Readership is the share following the coordinate relative to chance; values below 1 indicate active avoidance.Source: §3.4 · p. 4

3.5 CAPABILITY GATE

  • A convention gap is assessed only when the control exceeds chance by 20 points on the same readout used for the contrast.A null contrast without demonstrated competence on that readout is uninformative.Source: §3.5 · p. 4

3.6 HELD-OUT SPLIT

  • The held-out split uses angles 10–20° from each anchor, off the calibration grid and at least 20° from other anchors.The split rule was committed before the generator was written because evaluation on selected calibration angles could not separate model representation from selection.Source: §3.6 · p. 4

4 COLOR: THE CONVENTION GRADIENT

In the controlled color task, changing how the same alternatives are described produces a large accuracy difference. The cue-crossover and mispaired-gloss tests further distinguish remembered bindings from answer substitution.

4 COLOR: THE CONVENTION GRADIENT

  • 30.1 points: replacing color words with hue angles lowers accuracy (175 label-only correct against 19 numeric-only; exact McNemar p = 8.7 × 10−33, n = 519).Changing the output from a color word to an integer costs 1 point; ungrounded hue angles score 21.8% against 16.7% chance.Source: §4 · p. 5
  • Accuracy and cue-crossover interaction move together across all five rows, while the untouched location is null and cached trials show the same gradient.Word conditions score near 99% on calibration angles and 84% on held-out angles.Source: §4 · p. 5
  • 75.1% of cells follow the name and 5.1% the angle when glosses are deranged, against 16.7% chance for the coordinate (p = 7.7 × 10−21, n = 692).The apparent recovery from attaching names to angles is substitution: the model follows the name rather than the coordinate.Source: §4 · p. 5

5 SPACE, AND REAL PHOTOGRAPHS

On COCO photographs, spatial accuracy depends on how answer locations are written, especially when coordinates are answer options. The gap persists across grid granularity and object slices, while the preferred coordinate convention varies by model.

  • The COCO comparison varies only the descriptions of nine location options, after filtering for unique instances, smaller segmentation area, and centers away from cell borders.The task uses one object per image and 3 × 3 grids.Source: §5 · p. 5
  • 20.0% with pixel options versus 68.5% with English cell names: LOCATE accuracy drops 48.5 points on 200 photographs (chance 11.1%).Table 2 covers LOCATE and IDENTIFY; the answer-side drop contrasts with a 3.5-point, nonsignificant cost when a pixel coordinate is supplied in the question.Source: §5 · p. 5, Table 2 · p. 6
  • On a 4 × 4 grid, pixel options score 7.5% versus 26.0% for cell names; the gap also holds across object-size and boundary-distance terciles and eleven of twelve categories.The finer-grid chance rate is 6.25%.Source: §5 · p. 6
  • In the synthetic spatial task, normalized-coordinate prompts are longer than pixel prompts (187 versus 183 tokens) yet score 15.8 points higher.The authors use this comparison to rule out prompt length as the explanation within that design.Source: §5 · p. 6
  • The normalized 0–999 convention scores 59.5% for Qwen3-VL-4B versus 20.0% for pixels, but only 19.0% versus 13.5% for InternVL3-2B.The reported advantage does not transfer between these models.Source: §5 · p. 6

6 READERSHIP: IS THE CONVENTION READ AT ALL?

A mispaired-name test distinguishes whether a model follows a coordinate from whether it merely gets the answer right. Its readership measure separates conventions the model reads from those it avoids, but does not predict accuracy reliably.

  • The test deranges coordinate-name pairings and scores each response against both the correct-coordinate and correct-name keys; readership measures coordinate-following relative to chance.The measure is designed to distinguish using a convention from reaching the right answer by another route.Source: §6 · pp. 6–7
  • With a wrong name attached, the correct hue angle is chosen on 5.1% of cells against 16.7% chance, indicating active avoidance rather than indifference.The result also holds when the pairing is stated once in a legend rather than inline.Source: §6 · p. 6
  • The normalized 0–999 convention is followed on 45.0% of mispaired trials against 11.1% chance, while the name is followed on 14.0%.The model follows the coordinate despite the contradictory name.Source: §6 · p. 7
  • Readership did not predict accuracy for two further conventions: grid references scored 31.0% and percentages 23.0%, contrary to the predicted under-20.0% scores.The authors report readership as a test of whether a convention is read, not as an accuracy predictor.Source: §6 · p. 7

7 ACROSS MODELS

Answer conventions alter both measured performance and model rankings across color and real-photograph tasks. The readable convention differs by model, and the observed penalties vary across open and frontier models.

  • Five models give the same calibrated color wheel different stable vocabularies, including divergent red and teal names and four names for LLaVA-1.5 versus six for others.Only green near 145° is a shared anchor across all five.Source: §7 · p. 7
  • Three of four capable open models show a color word-to-numeric penalty: 30.1 points for Qwen3-VL-4B, 23.9 for Qwen2.5-VL-7B, and 19.7 for InternVL3-2B; LLaVA-1.5-7B shows none.SmolVLM-2B did not clear the capability gate and is uninformative.Source: §7 · p. 7
  • Running Qwen3-VL-4B at 8-bit rather than 4-bit widens the coordinate gap from 48.5 to 52.0 points while raising every condition.The reported result indicates the effect is not an artifact of low precision.Source: §7 · p. 7
  • On photographs, Gemini loses 11.1 points to pixel coordinates on parseable answers, while GPT-4o shows no significant gap; two of three open models show the penalty.LLaVA-1.5 and SmolVLM are absent because the harness could not score them with the first-token method.Source: §7 · p. 7
  • On COCO photographs, cell-name minus pixel gaps are −48.5 for Qwen3-VL-4B, −32.5 for InternVL3-2B, −9.0 for Qwen2.5-VL-7B, −15.5 for Gemini, and −5.0 for GPT-4o.The Figure 1 comparison uses the same 200 photographs; GPT-4o’s reported interval spans zero.Source: Figure 1 · p. 8
  • Two of three model pairs invert rankings across conventions: Qwen2.5-VL-7B and Qwen3-VL-4B tie on English names, then lead by 39 points and 54 points in opposite directions under pixel and normalized coordinates.The models lead in the coordinate convention their reports say they were trained in.Source: §7 · pp. 7–8, Table 4 · p. 8

8 A READOUT AUDIT OF OUR OWN EXPERIMENTS

Five scoring failures made capable models appear near chance or silent because the scorer and model disagreed about answer surfaces or output handling. The authors report the cases and describe audit checks added to address them.

  • SmolVLM-2B scored 16–30% against 16.7% chance on every color condition despite candidate mass 0.00, emitting Green (token 4910) while the scorer read green (token 9298).On a green swatch, its top three first tokens were Green, Red, Blue, with the correct one far ahead; the authors recommend resolving each model’s emitted surface by measurement.Source: §8 · p. 9
  • For SmolVLM and LLaVA-1.5, shared leading-space tokens make restricted first-token scores over integer labels unavailable, so the harness refuses those scores.The models are instead scored by generating and parsing.Source: §8 · p. 9
  • Seven of 24 held-out color angles fell in categories discarded during calibration, making the model right and the key wrong; the defect conservatively penalized the word condition.The authors retain the registered analysis while acknowledging the rule was wrong.Source: §8 · p. 9
  • A cross-model parser indexed shuffled options while the target indexed the canonical alphabet, putting every word score at chance; after correction, readouts agree on 97–100% of trials.The label rungs escaped the bug because both sides used slot indices.Source: §8 · p. 9
  • A frontier run returned empty strings on all ten calls because its eight-token output budget ran out during reasoning before the answer.Without inspecting raw text, a capable model would have been recorded as producing nothing.Source: §8 · p. 9
  • The audits parse prompts against targets, recompute labels from raw output, break trial-to-answer pairing and require chance, and refuse alphabets the model does not emit.The authors say all five failures were found by examining model outputs rather than scores.Source: §8 · pp. 8–9

9 LIMITATIONS

The real-image evidence is limited to one person-skewed dataset and a grid-cell selection task; several model, stimulus, and evaluation settings also constrain the findings.

9 LIMITATIONS

  • The real-image results use person-skewed COCO and grid-cell selection for named objects, not free-form box output or grounding.The 3 × 3 and 4 × 4 grids are coarse, and the results do not license claims about box regression.Source: §9 · p. 9
  • Color results use synthetic squares on gray, while the tested frontier-model sample is small and only one of two shows the effect.InternVL3-2B shares a Qwen language backbone, making the cross-family replication partial.Source: §9 · p. 9
  • The primary model runs use 4-bit precision, greedy decoding, and prompts that state each convention once; full precision, sampling, and longer instructions were not tested.Eight-bit precision widens the gap, but full precision was not run.Source: §9 · p. 9
  • Frontier models are scored by generation and parsing, whereas local models use first-token scoring; the instruments agree where both are available but are not identical.The paper reports that five scoring errors made a capable model appear incapable when its answer format disagreed with the scorer.Source: §9 · p. 9
  • Readership tests whether a convention is read, but it failed as a predictor of accuracy on every part of the pre-registered out-of-sample test.The paper describes readership as a valid test of reading, not a predictor of accuracy.Source: §9 · p. 9

10 WHAT A BENCHMARK AUTHOR SHOULD DO

The paper recommends treating answer conventions and scoring as part of benchmark design, and documents its ethics, reproducibility practices, and use of AI assistance.

10 WHAT A BENCHMARK AUTHOR SHOULD DO

  • Report the answer convention alongside localization scores and run a same-alternatives convention swap, reporting both results.The paper recommends a plain-language alternative and says differences of more than a few points belong next to the result.Source: §10 · p. 9
  • Test whether models read coordinate conventions, avoid glossing coordinates, separate parse failures from wrong answers, and calibrate answer vocabularies per model.The authors also recommend checking the scorer against raw outputs because errors in Section 8 were obvious in the text but invisible from scores.Source: §10 · pp. 9–10

ETHICS STATEMENT

  • The study uses public COCO data and synthetic stimuli, with no human subjects or personal data.Its stated intended effect is more accurate evaluation of publicly available models.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Scripts generate every reported number and figure from raw trial logs, with checks for prompt-target alignment, labels, chance-level pairing breaks, and emitted alphabets.The authors say the analysis plan, code, logs, prompts, and audits are available on request and will be released publicly.Source: REPRODUCIBILITY STATEMENT · p. 10

USE OF AI ASSISTANCE

  • The author directed the research and reviewed delegated AI-assisted coding, trial runs, references, and drafts; the author retained the final wording, structure, and claims.Five scoring errors from AI-assisted implementation were caught by the required audits.Source: USE OF AI ASSISTANCE · p. 10

A ADDITIONAL OBSERVATIONS

Additional analyses compare coordinate readership across models and examine a post hoc explanation for below-chance performance on one convention.

Readership across models

  • Pixel readership ranges from 1.12× for InternVL3-2B to 6.62× for Gemini; two models that barely read pixels lose 85–93% of headroom, versus 16–19% for three that do.With five points, the headroom relationship is an observation, not a fit, and was not pre-registered.Source: A ADDITIONAL OBSERVATIONS · p. 11

The number below chance

  • Qwen2.5-VL-7B chose the option nearest the object center when its normalized numbers were read as pixels on 53.0% of trials, against an 11.8% null.That option was the true target on only 2.5% of trials; the test was post hoc.Source: A ADDITIONAL OBSERVATIONS · p. 12
  • The pixel-number misreading test was significant for Qwen3-VL-4B at 28.0% versus 11.4%, but null for InternVL3-2B and for Gemini and GPT-4o.The authors say the misreading accounts for open-model inversions, not frontier gaps.Source: A ADDITIONAL OBSERVATIONS · p. 12

B PROMPT TEMPLATES, OPTION ORDER, GATE SENSITIVITY, AND INSTRUMENT AGREEMENT

The appendix records prompt templates and paired option-order procedures, then reports sensitivity checks for the model gate and scoring instruments.

Prompt templates

  • The photograph examples show cell-name, absolute-pixel, and normalized-coordinate prompts for choosing among nine locations, with a fixed per-image cell order.The examples are one-turn text prompts under the image and come from a 640 × 426 COCO photograph.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 12
  • The location templates also include glossed and mispaired pixel labels, while identify prompts place a location in the question and offer object names as options.The mispaired example attaches incorrect cell names to the pixel coordinates.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 12
  • Color templates use calibrated color words, integer labels, and hue angles, including glossed and mispaired variants.The color ladder uses each model's calibrated angle for its words and asks about held-out angles.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 13

Option order

  • Option order is a deterministic trial-hash permutation shared across conventions; mispaired glosses use a derangement with no correctly paired option.The paired design removes order effects from within-item contrasts, while orders are not resampled.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 13

Gate sensitivity

  • The gate sensitivity analysis found the set of penalty models unchanged through thresholds of 38.2 points for color and 34.9 points for photographs.No conclusion depends on the chosen 20-point gate.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 13

Instrument agreement

  • First-token argmax and parsed generation agree on 100.0% of tested trials for Qwen2.5-VL-7B and InternVL3-2B, and on 97.2% for SmolVLM's word rung.SmolVLM fails the gate and contributes no contrast.Source: B PROMPT TEMPLATES, OPTION ORDER, GATE… · p. 13

C ADDITIONAL TABLES

The additional tables document calibrated color vocabularies, COCO location results, robustness slices, and synthetic spatial-array trials.

C ADDITIONAL TABLES

  • Table 5 lists calibrated color vocabularies, with each anchor marking the center of a wheel-angle run named unanimously across three renders.The vocabularies differ across models, including whether red or teal appears as a stable name.Source: Table 5 · p. 13
  • Gemini’s cell-name minus pixel gap is −11.1 points on parseable answers and −15.5 when unparseable answers count as wrong.The comparison uses the same 200 COCO photographs and is paired per image.Source: Table 6 · p. 14
  • Table 7 reports the Qwen3-VL-4B cell-name minus pixel gap across terciles of object size and distance from the nearest cell boundary.It gives the number of images per slice and uses exact McNemar tests.Source: Table 7 · p. 14
  • Table 8 covers synthetic spatial-array LOCATE and IDENTIFY trials for Qwen3-VL-4B, with visible and cached conditions and their chance rates.It specifies 120 trials using four colored squares in distinct cells of a 3 × 3 grid.Source: Table 8 · p. 14
Loading 2609.27408v1…