PaperlayerView exact revision on arXiv

Source-linked AI summary

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Shubh Garg, Chen Shani

arXiv:2610.13910v1cs.CL

TL;DR

The paper asks whether VLMs faithfully represent the structured relations needed for reasoning, a question not answered by output-based benchmarks alone. It introduces CASP-VL and EICS to test representation changes under controlled concept interventions. Spatial relations are less faithfully represented than lexical information, and relation information present at predicate tokens is lost in pooled representations.

  • Problem

    It is unclear whether VLMs that answer spatial questions correctly represent the relation itself or rely on statistical associations between input words.

    Source: §1 · p. 1

  • Method

    The paper introduces CASP-VL, controlled one-concept caption interventions from scene graphs, and EICS, which tests whether an original-caption-trained probe predicts the changed label.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 3

  • Results

    Across VLMs, spatial relations are less faithfully represented than lexical information despite above-chance compositional scores; CLIP predicate tokens reach EICS +0.930, but pooled representations lose much of the relation.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    Output-based evaluation can miss systematic failures in how VLMs represent structured concepts.

    Source: §1 · p. 1

    EICS probes only text-encoder representations, and its caption interventions leave the image unchanged, so it does not establish how image or joint representations encode spatial arrangement.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models), a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across multiple VLM architectures, we find that spatial relations are represented substantially less faithfully than lexical information, even when the same models score well above chance on compositional benchmarks. Further analysis shows that the relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation, whether through the (EOS) token or by mean pooling. These findings reveal a systematic failure invisible to output-based evaluation: VLMs do not preserve, in the representation used downstream, relational information that is present in their own input.

1 Introduction

The paper asks whether VLMs internally represent concepts needed for spatial reasoning, beyond producing correct outputs. It introduces CASP-VL and EICS and finds that spatial relations are less faithfully represented than lexical information.

1 Introduction

  • Behavioral success does not necessarily imply faithful internal representations, and relevant information may be encoded even when a model’s final prediction fails.Understanding representation is complementary to measuring predictions for systems expected to reason about spatial structure under changing conditions.Source: §1 · p. 1
  • CASP-VL uses controlled spatial perturbations, while EICS measures whether representations track one-concept interventions beyond surface lexical cues.The paper compares VLM representations with a bag-of-words baseline and evaluates representations across layers, unlike benchmarks that score model outputs.Source: §1 · pp. 1–2, Table 1 · p. 2
  • VLMs represent spatial relations less faithfully than BoW despite above-chance compositional scores; in CLIP, the relation is identifiable at predicate tokens but largely lost in pooled sentence representations.Output-based evaluation can miss this representational failure.Source: §1 · pp. 1–2

2 Related Work

Prior work evaluates compositional behavior through model outputs, studies structural training methods and spatial reasoning errors, and probes internal representations. This paper positions EICS as a complementary representation-level evaluation.

Compositional benchmarks

  • Prior compositional benchmarks score model outputs, and language priors can make hard negatives exploitable (Hsieh et al., 2023).Related work also reports failures on relational expressions, visio-linguistic compositionality (Thrush et al., 2022), and basic spatial terms (Kamath et al., 2023).Source: §2 · p. 2

Training-time compositional methods

  • Training methods inject structure through scene graphs, structured hard negatives, or cross-modal masking and reconstruction; whether they raise EICS remains open.These approaches are evaluated by whether the correct caption outscores a hard negative, whereas EICS probes consistent, decodable representational changes.Source: §2 · p. 2

Spatial reasoning

  • Prior work proposes explicit spatial supervision (Chen et al., 2024) and links spatial reasoning errors to attention on irrelevant image regions (Chen et al., 2025).This paper adds a representation-level account: spatial relations are suppressed below what input tokens alone predict.Source: §2 · p. 2

Probing and causal methods

  • Probing classifiers characterize encoded properties (Belinkov, 2022), while other work studies representation drift and causal spatial IDs in VLMs (Wani et al., 2026; Kang et al., 2026).EICS asks whether the representation changes consistently when only a relation changes, rather than evaluating output behavior.Source: §2 · p. 2

Information Bottleneck

3 CASP-VL and EICS

CASP-VL constructs controlled one-concept caption perturbations from Visual Genome scene graphs. EICS tests whether a probe trained on original captions predicts the changed concept in perturbed captions, with word counts as a lexical baseline.

3.1 Benchmark Construction

  • CASP-VL creates image-caption triplets by changing exactly one concept in a Visual Genome scene graph while holding the image fixed.The perturbed caption is a counterfactual description of that same image.Source: §3.1 · pp. 2–3
  • The benchmark uses 108,077 Visual Genome images; relation flips change one sentence, while color flips change an object instance’s color in every sentence mentioning it.Scene-graph edits replace a spatial predicate with another from a fixed alternative list or an attribute with one of the same type.Source: §3.1 · p. 3
  • CASP-VL includes 2,400 color triplets across 12 classes and 1,491 spatial triplets across 16 classes, with balanced class distributions.It also includes 4,200 cross-category triplets for taxonomy analysis.Source: §3.1 · p. 3

3.2 EICS Metric

  • EICS tests whether a probe trained only on original captions predicts the changed label for perturbed captions, with zero indicating chance-level performance.The probe uses the text encoder’s [EOS]-pooled hidden state at a given layer.Source: §3.2 · p. 3
  • The BoW baseline applies the same procedure to normalized word counts, measuring information available from word identity without contextual encoding.Figure 2 schematically contrasts a color flip that crosses the probe boundary with a spatial flip that remains in the original-label cluster.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

4 Experiments

The experiments use image-grouped held-out perturbations to evaluate representational sensitivity, decodability, localization, and agreement with behavioral benchmarks across VLMs.

4.2 Do Representations Faithfully Encode Spatial Relations?

  • At CLIP L12, spatial EICS is +0.070 ± 0.012 (19% of BoW’s +0.372 ± 0.022), while color reaches +0.312 ± 0.019 (94% of BoW’s +0.332 ± 0.016).The spatial gap persists with wider CLIP encoders and alternative caption styles; reducing captions to the edited sentence narrows, but does not close, it.Source: Figure 3 · p. 5, §4.2 · p. 5

4.1 Do VLM Representations Respond to Spatial Changes?

  • At CLIP L12, original and perturbed [EOS] states have mean cosine similarity 0.992 for relation flips and 0.943 for color flips.The text representation is therefore less sensitive to spatial-relation changes than to comparable attribute changes.Source: §4.1 · p. 5

4.2 Do Representations Faithfully Encode Spatial Relations?

  • Across CLIP, SigLIP, and LLaVA, spatial EICS remains below the BoW spatial baseline of +0.359: model scores are +0.066, +0.053, +0.038, and +0.115, respectively.For LLaVA, the authors do not claim a color–spatial gap; its predicate tokens still identify the edited word at EICS of at least +0.86.Source: §4.2 · pp. 5–6

4.3 Is Spatial Structure Encoded at All?

  • Predicate-token pooling reaches spatial EICS +0.930, near its +0.938 ceiling, versus +0.070 at [EOS]; mean pooling reaches only +0.093, below BoW’s +0.372.Neither full-sentence pooling method preserves the relation well, though mean pooling retains more spatial signal for single-sentence captions than [EOS].Source: Table 2 · p. 6, §4.3 · p. 6
  • At L12, CLIP distinguishes original from argument-swapped captions with 0.574 accuracy (95% CI [0.535, 0.610]), above BoW’s 0.500 by construction.This is only a weak trace of argument order and may partly reflect the plausibility of the annotated order.Source: §4.3 · p. 6

4.4 Does Benchmark Accuracy Reflect Representational Faithfulness?

  • SigLIP pairs the highest SugarCrepe relation accuracy (70.6%) with the lowest spatial EICS (+0.038); CLIP reaches 69.3% accuracy but spatial EICS is +0.070 versus BoW’s +0.372.Argument-order results were weak: contrastive models selected correctly on 60–62% of SugarCrepe swaps, and CLIP’s probe reached 0.574 accuracy.Source: Table 3 · p. 7, §4.4 · p. 7

5 Analysis and Discussion

The analysis finds spatial encoding weaker than lexical encoding and tests what that gap means for relational structure and representation geometry. Greater representational movement and visual grounding do not reliably recover the intended relation.

5 Analysis and Discussion

  • CLIP L12 spatial EICS is +0.070 versus BoW +0.372; a two-hidden-layer MLP reaches only +0.086, still below the linear BoW baseline.The experiments establish a consistent gap between spatial and lexical encoding.Source: §5 · p. 7
  • With the predicate and token multiset fixed, CLIP separates original from swapped captions at 0.574 accuracy, a weak trace insufficient to recover who relates to whom.Color shows neither this failure nor the failure to track which relation holds.Source: §5 · p. 7
  • Last-layer representations move less under spatial than color edits: mean cosine similarity is 0.992 for CLIP and 0.984 for SigLIP, yet greater movement does not imply more faithful spatial encoding.SigLIP moves twice as far as CLIP under spatial edits but has no higher spatial EICS.Source: §5 · p. 8
  • After a spatial flip, image–caption similarity prefers the original caption in only 34% of cases, while visual grounding improves discrimination for color and size changes.The visual modality does not reliably recover the intended spatial relation.Source: §5 · p. 8
  • A three-way probe finds that part-whole and action relations collapse into a broad spatial region rather than occupying separable representational directions.Unperturbed captions are often labeled spatial, and edits toward part-whole or action relations almost never change the prediction.Source: §5 · p. 8
  • Structural supervision that penalizes argument-order errors may help preserve subject–predicate–object structure; useful pooling may need to weight predicates and arguments.Mean pooling does not recover the relation, and objective changes alone are insufficient.Source: §5 · p. 8

6 Conclusion

The conclusion reports that spatial relations are not stably encoded in the pooled representation used downstream, despite being identifiable at predicate tokens. EICS and CASP-VL are introduced to measure representational faithfulness under controlled concept changes.

6 Conclusion

  • The paper introduces EICS and CASP-VL to measure representational faithfulness in VLMs.CASP-VL is the benchmark and EICS the metric, as specified in the paper context.Source: §6 · p. 8
  • Spatial relations have [EOS] EICS +0.070, under a fifth of the BoW baseline, while color has EICS +0.312, close to BoW +0.332.The conclusion characterizes spatial relations as not encoded in a comparably stable, decodable form at the downstream [EOS] position.Source: §6 · p. 8
  • The predicate’s own tokens identify the relation at EICS +0.930, but [EOS] and mean pooling lose much of that signal.Visual grounding and objective changes do not resolve the failure, and other relation types collapse onto the spatial region.Source: §6 · p. 8

Limitations

The evaluation is limited to a small set of text encoders and caption-based interventions, with several caption and probing choices affecting the measured spatial signal. Additional checks indicate that these choices do not fully account for the color–spatial gap.

Limitations

  • The primary analysis uses CLIP ViT-B/32, with SigLIP-B/16 and LLaVA-1.5-7B as comparisons; larger, newer, and richer-spatial-supervision models remain untested.CLIP ViT-L/14@336 appears only in model-scaling analysis, and the authors note that explicit spatial grounding could yield higher EICS.Source: Limitations · p. 8
  • EICS probes text-encoder states after caption edits, not image-encoder or joint representations, and the image itself is never changed.The results therefore do not establish whether image representations encode spatial arrangements more faithfully than text representations.Source: Limitations · p. 8
  • Multi-sentence captions reduce the measured spatial signal, but the color–spatial gap remains with only the edited sentence.In that setting, CLIP keeps 52% of the BoW signal for spatial relations versus 98% for color.Source: Limitations · p. 9, Q3. Is this an artifact of template… · p. 10
  • Template captions may lower absolute scores, but human-written descriptions and fluent paraphrases retain the color–spatial gap.CLIP keeps 97% and 87% of the lexical signal for color, versus 31% and 15% for spatial relations, respectively.Source: Limitations · p. 9, Q3. Is this an artifact of template… · p. 10
  • A linear probe’s near-zero EICS cannot exclude nonlinear encoding, though an MLP raises L12 spatial EICS only from +0.066 to +0.086.The MLP does not reach the linear BoW baseline at any layer, but nonlinear explanations remain possible for untested architectures.Source: Limitations · p. 9, Q5. Could the relation be encoded… · p. 10

A Frequently Asked Questions

  • Across five image-grouped splits, CLIP keeps 16–22% of the BoW signal for spatial relations; widening the encoder does not improve spatial EICS.Spatial EICS is +0.066, +0.053, and +0.045 for ViT-B/32, ViT-L/14, and ViT-L/14@336, respectively.Source: Q5. Could the relation be encoded… · p. 10
  • A 69% SugarCrepe relation score is compatible with weak EICS because caption preference and representation tracking assess different things.After a spatial edit, the image prefers the original caption in only 34% of cases.Source: Q5. Could the relation be encoded… · p. 11

B Cosine Similarity Under Concept Interventions

D Dataset Statistics

CASP-VL edits scene-graph concepts to create balanced color and spatial triplets, with exclusions for some relation flips that cannot be scored under the main protocol.

D Dataset Statistics

  • The 300 flips (20% of relation flips) producing far from and away from, labels absent from original labels, are excluded from the main protocol because probes trained on original labels cannot score them.Source: D Dataset Statistics · p. 11
  • Each relation edit changes exactly one sentence, while 19% of color edits change several because the recolored object may be mentioned repeatedly.Captions render five scene-graph relationships in most cases, and a third repeat a sentence when Visual Genome annotates the same relationship more than once.Source: D Dataset Statistics · p. 11
  • The main protocol excludes 300 flips to two never-original labels, far from and away from, because probes trained on original labels cannot score them.These flips account for 20% of relation flips.Source: D Dataset Statistics · p. 11

E Main Evaluation Protocol

The main protocol groups splits by image and excludes unseen relation labels, then evaluates edits using text-encoder representations. Robustness checks examine split choice, predicate grouping, model comparisons, and unchanged object priors.

E Main Evaluation Protocol

  • Image-grouped splits prevent images from appearing in both train and test, and the protocol excludes test flips whose new label is unseen in training.The resulting test sets contain 482 color triplets and 242 spatial triplets; 53 of 295 spatial test triplets are excluded as one-directional antonym flips.Source: E Main Evaluation Protocol · p. 11
  • Across five image-grouped splits, CLIP keeps 81–102% of the BoW signal for color and 16–22% for spatial relations.Split 42 is least favorable for color, while all five splits agree on the spatial range.Source: E Main Evaluation Protocol · p. 11
  • Merging near-synonymous predicates yields spatial EICS of +0.227 for CLIP versus +0.375 for BoW; excluding polysemous predicates yields +0.120 versus +0.305.The authors report that the gap is not an artifact of synonym labels or polysemous predicates.Source: E Main Evaluation Protocol · pp. 11–12
  • SigLIP beats CLIP ViT-B/32 by 5–14 points on attribute and object subtasks, but not on relational subtasks.The relational differences are +1.3 on replace_rel and −1.2 on swap_obj; Table 6 reports EICS with 95% bootstrap confidence intervals and notes a slightly different SigLIP split.Source: Table 5 · p. 12, Table 6 · p. 12
  • Unchanged subject and object nouns leave both probes predicting the old label more often than chance, pulling EICS below its ceiling.The old-label excess is +0.08 for CLIP and +0.12 for BoW on spatial edits, and +0.11 and +0.09 on color edits, respectively.Source: E Main Evaluation Protocol · p. 12

F Error Analysis

The error analysis finds that spatial edits usually leave CLIP’s prediction unchanged, unlike color edits. Spatial mistakes are broadly distributed rather than near-misses, and removing captions with leftover old labels does not close the spatial gap with BoW.

F Error Analysis

  • For spatial edits, 13% reach the new relation, 14% keep the old one, and 73% go elsewhere; predictions stay unchanged for 86% of triplets, versus 51% for color.Among originals the probe gets right, 4 of 50 spatial triplets move to the new relation while 35 keep the old one; for color, 140 of 313 move to the new color.Source: F Error Analysis · p. 12, Table 11 · p. 14, Table 12 · p. 14
  • Spatial errors are not mostly near-misses: merging synonym groups raises CLIP’s EICS from +0.066 to +0.227, but BoW rises to +0.375.No row puts half its error mass on one predicted class, except the outside row, which keeps the old label inside half the time.Source: F Error Analysis · pp. 12–13
  • Across the splits shown in Table 10, all EICS values have 95% confidence intervals above zero.Table 9 gives confidence intervals for split 42’s 482 color and 242 spatial held-out triplets.Source: Table 10 · p. 13, Table 9 · p. 13
  • After removing triplets whose perturbed captions retain the old label, color EICS is +0.391 for CLIP versus +0.326 for BoW, while spatial EICS is +0.074 versus +0.366.The retained old label occurs in 35.8% of perturbed color captions and 36.9% of relation captions; removing it leaves spatial EICS low either way.Source: F Error Analysis · pp. 13–14

G BoW Robustness Check

The BoW comparison shows that CLIP’s spatial representations preserve less information than word identities alone. More flexible probing, alternative caption grounding, and image–text alignment analyses do not eliminate the observed spatial weakness.

G BoW Robustness Check

  • Under the main protocol, spatial EICS is +0.066 for CLIP versus +0.359 for BoW; under the triplet-level split, it is +0.040 versus +0.173.The BoW probe uses normalized word-count vectors; counting CLIP BPE tokens instead gives a spatial baseline of +0.347 rather than +0.359.Source: G BoW Robustness Check · pp. 14–15

H Linear vs. MLP Probe Comparison

  • At no layer does the MLP-probed spatial EICS reach the linear BoW baseline of +0.359, despite rising to +0.214 at L02.At the last layer, MLP probing changes spatial EICS from +0.066 to +0.086; the authors conclude the late-layer result is not a linear-probe artifact.Source: H Linear vs. MLP Probe Comparison · p. 15, Table 14 · p. 15

I Visual Grounding

  • For spatial flips, images prefer the original caption only 34.0% of the time (∆ = −0.0014), and the table caption reports that they prefer the wrong caption 66% of the time.The authors suggest the image encoder may capture object co-occurrence more than arrangement.Source: I Visual Grounding · p. 15

J Relation Taxonomy Analysis

Cross-category probing shows an asymmetry: spatial predicates dominate the representation, so edits from spatial relations to other relation types are rarely detected.

J Relation Taxonomy Analysis

  • Representations are distinguishable using 4,200 cross-category triplets, and the analysis applies a majority-class baseline correction; the passage associates this result with Table 16.Source: J Relation Taxonomy Analysis · p. 15
  • After spatial→part-whole edits, the probe predicts part-whole for only 1.8% of perturbed captions; after part-whole→spatial edits, spatial predictions rise 13.3 points over baseline.The table reports corrected flip rates and a baseline spatial prediction rate on unperturbed source-class captions.Source: J Relation Taxonomy Analysis · p. 16
  • The probe predicts spatial for 67.5% of unperturbed part-whole captions, indicating that part-whole and action relations occupy the same representational region.The authors conclude that spatial predicates dominate the geometry enough to make other relation types invisible to linear probes.Source: J Relation Taxonomy Analysis · p. 16

K Caption-Style Robustness

Caption-style tests reproduce the color–spatial gap across human-written and fluent paraphrased captions, while single-sentence pooling narrows but does not close it. Increasing text-encoder width does not raise spatial EICS.

K Caption-Style Robustness

  • The style tests use human-written Visual Genome descriptions, single-sentence captions, and LLM paraphrases that preserve the one-concept edit.The paraphrase subset contains 2,182 triplets after filtering rewrites that alter marked or other label words.Source: K Caption-Style Robustness · p. 16
  • On VG regions, CLIP keeps 97% of lexical signal for color but 31% for spatial; on LLM paraphrases, it keeps 87% for color and 15% for spatial.Fluent paraphrases reproduce the template result almost exactly; templates lower absolute scores but do not create the gap.Source: K Caption-Style Robustness · pp. 16–17, Table 17 · p. 17
  • With one sentence per caption, CLIP keeps 52% of lexical signal for spatial relations, versus 18% in the main results; mean pooling reaches +0.702 and predicate-token pooling +0.938.Removing leftover old labels lowers the single-sentence retained spatial signal to 49%.Source: K Caption-Style Robustness · p. 17, Table 17 · p. 17

L Model Size Scaling

  • Across the tested CLIP widths, spatial EICS does not rise (+0.066, +0.053, +0.045), while color EICS changes only slightly (+0.292, +0.303, +0.298).Both larger models reach the predicate-span ceiling (+0.933); the tested width increase from 512 to 768 dimensions does not help spatial encoding.Source: L Model Size Scaling · p. 17, Table 18 · p. 18

Related papers

Loading 2610.13910v1…