PaperlayerView exact revision on arXiv

Source-linked AI summary

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem, Lotem Peled-Cohen

arXiv:2609.37863v1cs.CLcs.AIcs.CV

TL;DR

Substitutability tests give VLM judges a verdict from a single evaluation configuration, leaving unclear whether it holds when supposedly irrelevant context changes. MIST tests sentence-based idiom labels with aligned, misleading, or absent images; images changed labels at similar rates regardless of which reading they depicted.

  • Problem

    Substitutability tests return a single-run verdict, leaving unclear whether it is robust to context that guidelines declare irrelevant.

    Source: §1 · p. 1

  • Method

    MIST uses 200 English sentence–target items whose labels are determined by the sentence, presented with aligned, misleading, or no image.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    20.5% aligned-image and 19.4% misleading-image change rates across all thirteen judges were close for every judge and exceeded the image-fixed prompt-edit control.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    A substitutability verdict describes the judge and its evaluation configuration, since aggregate agreement can conceal instance-level instability.

    Source: §4 · p. 4

    Both tested images depict a reading of the target phrase, so the study does not test whether an unrelated image would move labels similarly.

    Source: B Limitations · p. 7

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.

1 Introduction

The paper asks whether substitutability tests measure a VLM judge or the particular context in which it is evaluated. MIST tests whether images that guidelines say to ignore change sentence-based labels.

1 Introduction

  • Existing substitutability tests give a single-run verdict, leaving unclear whether it holds when supposedly irrelevant context changes.This matters because human labels may become evaluation sets and training data, where errors can propagate into flawed comparisons and false conclusions.Source: §1 · p. 1
  • Idioms allow context to change while the correct label stays fixed: a phrase can have literal and figurative readings, but its sentence determines which applies.The same phrase can be figurative in one sentence and literal in another.Source: §1 · pp. 1–2
  • Unlike multimodal tasks where image changes legitimately alter answers, MIST distinguishes appropriate context use from influence by context the judge should ignore.Related work on irrelevant context measures task accuracy rather than agreement with the human annotators a model would replace.Source: §1 · p. 2
  • MIST tests whether aligned, misleading, or absent images change labels that should be decided from the sentence alone.It contains 200 English items built around potentially idiomatic phrases.Source: §1 · p. 2, Figure 1 · p. 2
  • Aligned and misleading images change similar numbers of labels, and neither reliably shifts labels toward the interpretation it depicts.The paper reports that ignored context destabilises judgments without informing them.Source: §1 · p. 2

2 MIST: The Misleading-Image Stress Test

MIST pairs sentence-based idiom labels with aligned, misleading, or absent images. Its design keeps the correct answer fixed while varying image context.

2 MIST: The Misleading-Image Stress Test

  • The 200-item test samples 200 expressions from a release containing 551 English potentially idiomatic expressions, retaining one figurative or literal sentence per expression.The discarded sentence row’s image remains available, allowing either image to accompany each retained sentence.Source: §2 · p. 2
  • Each item is presented with an aligned image, a misleading image depicting the opposite sentence sense, or no image.Crossing figurative and literal sentence senses with image type yields four conditions, with 50 expressions each; VLM judges see all three inputs per item.Source: §2 · p. 2
  • Labels describe how the written phrase reads in its sentence, using four categories: Fully Literal, Figurative and Literal, Weak Figurative, and Fully Figurative.The first two categories distinguish literal use with or without a figurative reading; the latter two distinguish whether the phrase retains a semantic link to its literal words.Source: §2 · p. 3
  • Six fluent English speakers annotated disjoint item sets in informed and blind trios, which reached moderate agreement without a significant difference.Their Fleiss κ values were 0.65 and 0.57 (p = 0.22).Source: §2 · p. 3

3 Experiments

The experiments test whether image context changes VLM labels, whether changes follow image content, and whether agreement with human annotators shifts. Across judges, images change labels more than an image-fixed prompt edit, without a corresponding directional pull toward the depicted sense.

3.1 Experiments Setup

  • Four prompt techniques were crossed with explicit and silent image-instruction arms, producing 20 labels per item and 52,000 in total.Thirteen VLMs were evaluated; seven passed the alt-test in at least one configuration and six never passed.Source: §3.1 · p. 3, §3.2 · p. 3
  • Change rate tracks exact label differences between inputs; direction tracks movement toward the depicted sense, while agreement is exact match with the human majority.Substitutability uses the alt-test, run per human trio at its tier-specific slack.Source: §3.1 · p. 3

3.2 Results

  • 15.9% aligned-image and 14.7% misleading-image change rates were reported for the seven judges that passed the alt-test in at least one configuration.Table 1 pools change rates over the four prompts for those seven judges.Source: §3.2 · p. 3, Table 1 · p. 4
  • 20.5% aligned-image and 19.4% misleading-image change rates across all thirteen judges were close for every judge and exceeded the prompt-edit control.The control held the image fixed and removed only the instruction to ignore it.Source: §3.2 · p. 4
  • 37% of label changes between the two images moved toward the sense depicted, while 63% moved away; every alt-test passer was below chance.Among the seven judges, 71% of movement stayed on the same side of the figurative–literal divide.Source: §3.2 · p. 4
  • 54.4% agreement with the human majority with no image, 53.7% aligned, and 54.7% misleading show no aggregate accuracy reduction under images.Only 5 of 13 judges lost accuracy under an image; changed items could differ even when overall agreement did not.Source: §3.2 · p. 4

4 Discussion

Images destabilize VLM labels without pulling them toward the depicted sense, while agreement with human annotators stays unchanged. The results suggest that substitutability depends on the judge’s configuration as well as the judge.

4 Discussion

  • 15.9% of labels change when an image is attached, but movement does not track its content and human agreement is unchanged.The effect is smaller for judges that pass the alt-test than for those that do not, but is present in both groups.Source: §4 · p. 4
  • A substitutability verdict can conceal instance-level instability because aggregate agreement does not reveal whether individual labels change across configurations.The paper recommends reporting the prompt and any irrelevant context attached.Source: §4 · p. 4
  • MIST tests image-context invariance, not whether a judge changes appropriately when a sentence genuinely changes reading.A judge could pass the invariance test without reading its input; the evaluated judges agreed with the human majority at 54.4%.Source: §4 · p. 4

A Related Work

Prior substitutability protocols give one verdict for a single input and configuration, while related work on unwanted context generally measures task accuracy rather than agreement with human annotators. MIST applies an invariance test to judges and asks which measurement conditions their verdicts depend on.

A Related Work

  • Existing substitutability tests assess a judge in one input and configuration, not whether its verdict holds under another.The alt-test compares agreement with a human panel against a withheld human; He et al. test statistical indistinguishability.Source: A Related Work · p. 7
  • Prior unwanted-context studies measure task accuracy, while MIST evaluates agreement with the annotators a model would replace.ID10M-JAM also uses conflicting cues for idiomatic expressions, but its input-length perturbation and two-annotator raw-agreement baseline differ from MIST.Source: A Related Work · p. 7
  • In existing figurative benchmarks, images are part of the decision; MIST instead scores a text label that guidelines say must ignore the image.Examples include IRFL, V-FLUTE, and AdMIRe.Source: A Related Work · p. 7
  • To the authors’ knowledge, no prior work tests which measurement conditions a judge’s substitutability verdict depends on.Source: A Related Work · p. 7

B Limitations

MIST’s limitations include testing only phrase-relevant images, using a partially ordinal scale, and not measuring repeated human labels. Its guidelines define sentence-based labels and specify how annotators should handle ambiguity and image distraction.

B Limitations

  • Only two phrase-relevant images are tested, leaving the effect of an image unrelated to the sentence untested.Source: B Limitations · p. 7
  • The label scale combines different judgments at its two boundaries, so net shifts average unlike transitions; exact-match change rates avoid that issue.The reported 71% of movement staying on one side of the first-step boundary uses the distinction the scale encodes cleanly.Source: B Limitations · p. 7
  • Because each human annotator saw each expression in only one condition, the study cannot measure how often human labels change across conditions.The prompt-edit control provides a within-judge baseline, not a human change-rate measurement.Source: B Limitations · p. 7
  • The study covers English items from one corpus and disables reasoning modes, though the effect appears across thirteen judges and seven model families.Source: B Limitations · p. 7

C Annotation Guidelines

  • Annotators judge the highlighted phrase from the full sentence, treating the image as silent context that must not change the sentence-based interpretation.Source: C Annotation Guidelines · p. 8
  • The four labels distinguish whether the literal meaning is active in the sentence from whether it remains conceptually linked to the figurative meaning.Annotators first assess the sentence-specific literal reading; if absent, they judge the phrase’s literal–figurative connection to distinguish WF from FF.Source: C Annotation Guidelines · p. 8

D Construction Details

MIST samples compounds that have both figurative and literal rows, selects one sentence per compound, and pairs it with a sense-matched or opposite-sense image. The construction also marks each target phrase in the sentence.

D Construction Details

  • 200 compounds were sampled uniformly from eligible source compounds with both figurative and literal rows, then split into four disjoint groups of 50.The sampling used random.seed(123), and each compound appeared in exactly one condition.Source: D Construction Details · p. 8
  • Sentence selection favors shorter candidates through word-count tiers, and all 200 selected sentences contain the compound exactly or as a morphological variant.Of the final sentences, 164 (82%) use an exact substring and 36 (18%) a morphological variant.Source: D Construction Details · p. 8
  • Each item uses the relevant row’s correct_image for same-sense conditions and the opposite row’s image for cross-sense conditions.The images are from the source release; no further curation or new image generation is performed.Source: D Construction Details · p. 9
  • Targets are marked with double asterisks, and Figure 2 shows how T1 and T4 cross-pair sentence and image rows while T2 and T3 use one row.Source: D Construction Details · p. 9, Figure 2 · p. 9

E Data Statistics

MIST uses 200 distinct compounds across four sentence/image conditions, with three independent human labels per item. Label distributions support the intended sentence readings, while human agreement contrasts between conditions remain unresolved.

Composition

  • 200 items use 200 distinct compounds, with 50 items per condition and three independent human labels per item.The informed trio labelled 100 items and the blind trio the other 100, with no item labelled by both.Source: E Data Statistics · p. 9

Label usage

  • 87% of annotations on figurative-sentence items and 86% on literal-sentence items fall on the corresponding sentence-sense side, confirming the sentence fixes the reading as intended.Within-condition label distributions are severe by construction, so per-class figures rest on small cells and aligned labels leave less room for shifts toward the middle.Source: E Data Statistics · p. 9

Trio reliability

  • The two human trios have close agreement intervals that overlap, so their reliability difference is not treated as a quality difference.The pooled row averages two disjoint trios and is not an estimate of either trio’s reliability.Source: E Data Statistics · p. 10, Table 3 · p. 10

Agreement by condition

  • The informed trio is flat (∆κ = −0.00), while the blind trio moves from 0.62 to 0.52; neither aligned-versus-misleading shift is statistically resolvable.The trios are reported separately; condition cells contain roughly 25 items each and are noisy.Source: E Data Statistics · p. 10, Table 4 · p. 10

E.1 Are the human contrasts resolvable?

The section tests whether human trios differ in reliability and whether misleading images reduce agreement relative to aligned images. Neither trio shows a statistically resolvable alignment effect, and the available sample has limited power for such contrasts.

Procedures

  • The analysis compares 50 aligned with 50 misleading items using both bootstrap intervals for Fleiss κ and a two-sided Welch t-test on per-item agreement.The bootstrap resamples items 10,000 times; per-item agreement is not chance-corrected, and the groups’ label distributions are close.Source: §E.1 · pp. 10–11, Table 5 · p. 11

Trio effect

  • The informed trio has Fleiss κ = 0.65 [0.56, 0.74] and the blind trio 0.57 [0.46, 0.67], but their difference is not significant (p = 0.22).The estimated difference is +0.08 [−0.05, +0.22], so neither trio is designated the reference.Source: §E.1 · p. 11

Alignment effect

  • Neither trio shows a significant aligned-versus-misleading difference: blind p = 0.27 and informed p = 0.92, with every interval containing zero.At 50 items per condition, the design resolves only differences of about 0.15 or larger in per-item agreement and 0.20 or larger in Fleiss κ.Source: §E.1 · p. 11, Table 5 · p. 11

Interpretation and scope

  • The null human contrasts do not establish either robustness or an image effect, and the blind-trio interval still admits a shift twice its point estimate.At 50–100 annotated items, a substitutability verdict may be reachable even though a condition-level contrast has little power to show whether that verdict changes.Source: §E.1 · p. 11

F Change rates, all thirteen judges

The full change-rate table covers all thirteen judges, including six that never pass the alt-test. Across both judge groups, image-related change rates exceed each judge’s prompt-edit control.

Alt-test coverage

Full judge set

  • Every judge, in both alt-test groups, exceeds their own prompt-edit control rate in both image-change columns.Table 6 is the full version of Table 1; each judge contributes equally to columns 1–3, whose means are also pooled rates.Source: Table 6 · p. 12

Pooling note

  • The pooled rate for Table 6’s fourth column is 37%, rather than the 34% shown, because that column’s denominator varies by judge.The denominator variation does not apply to columns 1–3.Source: Table 6 · p. 12

G Full drift grid

The full drift grid shows prompt-dependent, often anti-image-following changes rather than a stable pull toward depicted senses. Intervals mostly include zero, and the selected prompt can differ by sentence type.

Prompt selection

  • The two sentence-type selections disagree for 8 of 13 judges when choosing the prompt most favourable to image-following.Table 7 shows drift for each judge under each prompt, with the selected cells marked in bold.Source: G Full drift grid · p. 12, Table 7 · p. 13

Drift patterns

  • On literal sentences, total shift is negative for 12 of 13 judges, while the aligned-to-misleading term runs opposite for 10; only Q2-7B points toward image-following.Under Few-Shot + CoT, only one judge has the aligned-to-misleading term in the predicted direction on both sentence types, and 179 of 200 majority-vote labels remain unchanged when the image is replaced.Source: G Full drift grid · p. 12, Table 7 · p. 13

Prompt sensitivity

  • On literal sentences, aligned-to-misleading shifts vary sharply by prompt: K-VL ranges from +0.01 to +0.32 and Q2.5-7B from −0.32 to +0.36.The prompt ordering is not consistent across judges.Source: G Full drift grid · p. 12, Table 7 · p. 13

Intervals on the content term

  • 23 of 26 selected-prompt intervals contain zero; under Few-Shot + CoT, 22 of 26 contain zero and all four exceptions are anti-image-following on literal sentences.The initial interval exceptions are close to what selecting a maximum across four prompts per cell would produce.Source: G Full drift grid · p. 12, Table 7 · p. 13

Direction of label changes

  • For Few-Shot + CoT, only Q2.5-7B exceeds F/S = 0.5 (0.60), while six judges fall below 0.4.F/S is the share of changed text-only-to-aligned labels that move toward the aligned image’s sense; below 0.5, changes run counter to the image.Source: G Full drift grid · p. 13, Table 8 · p. 14

H Alt-test grid

The alt-test result depends on the annotator trio and slack, while condition-specific passes are unstable. The prompt keeps experimental factors fixed, and its labeling rules separate sentence-level literal plausibility from phrase-level figurative connections.

H Alt-test grid

  • Aligned and misleading half-slices produce unstable passes, while lower annotator agreement on the blind trio lowers the withheld-human bar and makes judges look more substitutable.The authors interpret the half-slice reversals as noise, not a condition effect.Source: H Alt-test grid · p. 14
  • No informed-trio cell passes; on the blind trio, 2 judges pass at ε = 0.15 and 7 at ε = 0.20, with identical passing sets under ACC and MAE.The table covers both trios, conditions, prompts, slacks, and scoring functions; single T-types are omitted because n ≈25 falls below the recommended minimum per annotator.Source: Table 9 · p. 14, H Alt-test grid · p. 14

I The prompt

  • The prompt is assembled from fixed base and output blocks plus an image clause only when an image is attached; that clause says the image is context, not the annotation target.The output block varies with whether chain-of-thought is requested.Source: I The prompt · p. 15, SYSTEM · p. 16

SYSTEM

  • The labeling rules use four categories and first test literal plausibility in the exact sentence, then whether the figurative meaning connects to the phrase’s literal words.The guidance also says to read the full sentence and not assume a famous idiom is figurative.Source: SYSTEM · p. 16
  • Few-shot demonstrations are fixed across judges, drawn from unanimously labeled items, and excluded from the target; the silent arm removes only the image clause.The four demonstrations include one per label.Source: SYSTEM · p. 17
Loading 2609.37863v1…