PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

Md Tanvirul Alam

arXiv:2604.12119v1cs.CVcs.LG

TL;DR

VLMs can rely on familiar semantic priors even when prompts redefine the correct mapping, but existing evaluations may conflate this with perception failures. The paper introduces VLM-Fix, which compares identical boards under standard and inverse rules, and finds consistent standard-rule advantages across models. Neutral aliases, joint-rule training, VLMBias interventions, and late-layer steering respectively narrow, broaden, externally support, or partially recover performance under remapped semantics.

  • Problem

    Existing evaluations do not cleanly separate VLM perception failures from failures to remap rules when prompts specify alternative meanings.

  • Method

    VLM-Fix evaluates identical terminal board states from four abstract games under paired standard and inverse rules, with prompt, post-training, external-benchmark, and activation-steering interventions.

  • Results

    Standard-rule accuracy consistently exceeds inverse-rule accuracy across 14 VLMs; neutral aliases narrow the gap, joint-rule training improves broader transfer, VLMBias shows a similar pattern, and late-layer steering partially recovers performance.

  • Takeaways & Limitations

    Semantic fixation is partly associated with late semantic readouts and is partly editable, while broader transfer benefits from training on both rule mappings.

  • Takeaways & Limitations

    The conclusions are bounded because VLM-Fix is synthetic and VLMBias provides support from a related but not identical task family.

Abstract

from arXiv · show

Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.

1 Introduction

The paper defines semantic fixation as defaulting to familiar meanings despite alternative prompt-specified mappings, and introduces VLM-Fix to isolate this behavior from visual perception. Across models, rule remapping produces a consistent gap, while prompting, training, external validation, and activation steering probe its causes and editability.

  • Motivation: Semantic fixation describes VLMs defaulting to familiar meanings even when prompts specify alternative mappings.The concept targets rigid semantic expectations rather than changes in the perceptual input.
  • Benchmark: VLM-Fix evaluates identical terminal boards under standard and inverse rules across four abstract strategy games, isolating semantic interpretation from perception.The benchmark covers Tic-Tac-Toe, Connect Four, Reversi, and Dots and Boxes.
  • Main findings: Inverse rules reduce accuracy despite identical visual states, revealing a robust semantic-fixation gap across games and models.This comparison directly tests whether models can remap decisions when only rule semantics change.
  • Main findings: Neutral aliases narrow the inverse-rule gap, whereas semantically loaded aliases restore it.The same underlying task is reframed first with neutral tags and then with tags carrying semantic valence.
  • Post-training: Rule-aligned post-training improves same-rule performance but harms opposite-rule transfer, while joint-rule training improves generalization across both rules.These findings indicate that training exposure to one mapping can specialize behavior to that mapping.
  • External validity and editability: Late-layer activation steering partially recovers degraded inverse-rule performance, and similar qualitative effects appear on VLMBias beyond synthetic games.Together, these results suggest late semantic representations are partly editable and the pattern extends to a related benchmark.

2 Related Work

Related work documents systematic bias, brittle grounding, fixation-like reasoning, and anchoring in multimodal and language models. Existing mitigation strategies use decoding, prompting, counterfactual learning, and counterfactual data interventions to reduce biased or spurious behavior.

  • VLM reliability: Prior VLM evaluations report social bias, representational skew, brittleness under distribution shift, and weak grounding under challenging visual conditions.These findings motivate studying reliability beyond aggregate benchmark performance.
  • Fixation and anchoring: Language-model studies connect fixation and anchoring to persistent reliance on familiar or initial semantic mappings.The related literature frames inflexible reasoning as a prior-driven behavior.
  • Mitigation: Bias-mitigation research explores decoding, prompting, self-debiasing, causal interventions, counterfactual prompt learning, and counterfactual data interventions.These approaches target harmful continuations, biased reasoning, or spurious semantic–visual correlations.

3 VLM-Fix: Dataset and Evaluation Setup

VLM-Fix is a controlled synthetic benchmark that holds terminal board states fixed while varying rule semantics, visual rendering, and prompt framing. Its matched conditions support attribution of performance differences to semantic interpretation rather than board-state difficulty or visual changes.

  • Dataset construction: VLM-Fix covers Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes using terminal board states.Each game has 300 unique terminal base states, excluding draws and balancing the canonical winner across players.
  • Rule conditions: Standard rules use each game’s canonical objective, while inverse rules reverse the usual winning condition.Winner and loser queries are evaluated under both conditions.
  • Rendering variants: The benchmark uses base, checkerboard, and glyph renderings to vary visual presentation while preserving the underlying board structure.Glyph rendering replaces canonical player symbols with deterministically sampled alphabetic glyphs.
  • Prompt variants: Base, Alias, and SemAlias prompts vary semantic framing by using canonical terms, arbitrary tags, or explicitly defined tag meanings.The design separates lexical familiarity from explicit semantic grounding and avoids naming games verbatim.
  • Evaluation scale: Each image–prompt configuration yields 2,400 examples per game before Direct and Chain-of-Thought evaluation, and 12,000 examples across five main configurations.The 2,400 count combines 300 states, two rules, two query targets, and two multimodal orderings.

4 Experiment & Results

Across 14 VLMs, standard-rule accuracy substantially exceeds inverse-rule accuracy on identical boards, while semantic and prompting controls reveal that the gap is rule- and framing-sensitive.

  • Baseline Performance: 67.1% standard accuracy versus 52.5% inverse accuracy yields a 14.6-point gap across games and models.The gap is largest for Dots and Boxes: 73.8% versus 50.0%, a 23.8-point difference.
  • Baseline Performance: 13 of 14 models show lower inverse-rule accuracy than standard-rule accuracy.Sonnet-4.5 is the only exception.
  • Input Interventions: Alias raises inverse accuracy to 63.08 and reduces the standard–inverse gap to 2.29 points.SemAlias lowers inverse accuracy to 53.51 and reopens the gap, whereas visual perturbations produce only modest changes.
  • Control Settings: Descriptive prompting produces more balanced results than baseline, at 71.0/68.8 standard/inverse accuracy.Text-only evaluation retains an inverse drop of 69.2/58.6, indicating the effect persists without image input.
  • Control Settings: Under chain-of-thought prompting, Base retains an 85.0 versus 74.1 standard–inverse gap, while Alias largely closes it at 84.8/84.1.SemAlias partially reopens the gap to 85.5/77.3.
  • Input Order: Text-first prompting amplifies the direct Base gap from 13.2 points to 21.4 points, while chain-of-thought largely removes this order effect.The direct image-first and text-first results are 60.2/47.0 and 66.2/44.8, respectively.

5 External Validity on VLMBias

VLMBias tests whether defamiliarizing images or lexical prompts transfers the semantic-fixation intervention beyond synthetic rule-switching games. Combined visual and lexical defamiliarization produces the strongest aggregate improvement.

  • External Validity: The VLMBias pattern supports external validity beyond the synthetic game setting.The benchmark is used as an external transfer test rather than as a second direct measurement of semantic fixation.
  • Intervention Design: Flip vertically alters orientation cues while preserving task-relevant pixels and scene complexity, whereas Alias replaces the target object term with a generic ITEM token.Flip targets visual familiarity; Alias tests lexical defamiliarization without explicit rule semantics.
  • Results: Flip+Alias raises aggregate accuracy from 11.6% to 20.7% and reduces bias from 76.7% to 58.9%.The combined intervention is strongest overall across the four VLMBias subsets.
  • Results: Alias alone improves accuracy to 15.0% and reduces bias to 69.5%, while Flip alone reaches 13.3% accuracy and 70.9% bias.Both single perturbations improve on Base, but less than their combination.
  • Results: Animals shows the largest task-level gain under Flip+Alias, increasing from 3.6% to 22.2%.The pooled summary reports improvement in accuracy and reduced bias for all 14 models.

6 Training Interventions and Transfer

SFT and RLVR improve transfer when training and evaluation use the same rule, but opposite-rule transfer degrades; exposure to both rules improves cross-game generalization.

  • VLM-Fix Rule-Transfer Splits: D1–D3 evaluate same-rule, opposite-rule, and held-out cross-game transfer across standard and inverse formulations.D3 trains on Tic-Tac-Toe and Reversi before testing on Connect Four and Dots and Boxes under both rules.
  • VLM-Fix Rule-Transfer Splits: Post-training improves held-out accuracy when evaluation matches the training rule but drops performance below base when the rule mapping is flipped.SFT is typically strongest on same-rule transfer, while RLVR is less brittle under the shift.
  • VLM-Fix Rule-Transfer Splits: Training on both rules improves transfer to held-out Connect Four and Dots and Boxes.RLVR is stronger on standard-rule evaluation, whereas SFT is stronger on inverse-rule evaluation, especially for Qwen2.5-VL-7B.
  • Synthetic Leg-Count Transfer: Synthetic leg-count supervision raises Qwen2.5-VL-3B Animals accuracy from 0% to 34.6% with SFT and 23.1% with RLVR.For Qwen2.5-VL-7B, the corresponding gains are 21.8% and 22.2%; gains on VLMBias Other are smaller and less consistent.
  • Synthetic Leg-Count Transfer: VLMBias Counting (Unbiased) changes only minorly after post-training because base models are already near ceiling on Animals and around the mid-60s on Other.The transfer benefit is concentrated on the challenging VLMBias Animals slice.

7 Activation Steering Analysis

Late-layer donor-based activation steering partially edits rule-sensitive and counting representations without retraining. Improvements depend on layer, task difficulty, architecture, routing reliability, and donor traces.

  • VLM-Fix Steering: Steering patches one of the final 12 decoder layers at the query token and moves target activations toward matched donor representations.A lightweight router predicts the relevant rule/answer bucket before patching.
  • VLM-Fix Steering: Late-layer steering improves patched VLM-Fix accuracy when routing is reliable, with strongest gains in Reversi and Dots and Boxes.Gains concentrate in later layers and are limited in harder settings such as Connect Four.
  • VLMBias Steering: In VLMBias Animals, donor-based steering improves patched accuracy, with clearest late-layer gains in the Qwen models.The SFT models provide donor traces for steering corresponding base models in the controlled 2-leg versus 3-leg setting.
  • VLMBias Steering: The relevant counting representation is partly editable, but accessibility varies across architectures and depends on reliable donor traces.Additional steering results are reported for Molmo2-4B and InternVL3.5-4B.

8 Discussion and Limitations

The paper concludes that semantic fixation extends beyond synthetic VLM-Fix results, while acknowledging that the evidence remains bounded by task scope and incomplete mechanistic understanding.

  • VLM-Fix and VLMBias show consistent trends suggesting that defamiliarizing image–prompt associations can reduce prior-sensitive behavior.
  • The conclusions are bounded because VLM-Fix is synthetic and VLMBias evaluates a related but not identical task family.
  • Late-layer activation steering partially recovers degraded performance, but the formation and use of semantic representations during inference remain incompletely explained.

A Dataset Details

The dataset details define controlled terminal-state evaluations across four games, multiple renderings and prompts, and standard versus inverse rules. Results show that neutral aliasing most reliably narrows rule gaps, while descriptive and text-only settings retain distinct performance patterns.

  • Dataset construction: Each game contributes 300 unique terminal states, excluding draws and balancing canonical winners evenly across players.
  • Dataset construction: The four games use game-specific terminal-state procedures, including legal-play constraints for Tic-Tac-Toe, Reversi, and Connect Four, plus synthetic claimed-cell outcomes for Dots and Boxes.
  • Prompt and rendering configurations: The benchmark varies Base, Glyph, Checkerboard, Alias, and SemAlias configurations while testing direct and CoT response formats.
  • Direct and CoT results: Alias consistently closes the standard–inverse gap across all four games and is the strongest intervention for mitigating rule-inversion sensitivity.
  • Descriptive prompting: Descriptive prompting yields an overall mean of 70.99%/68.79%, with Dots and Boxes easiest and Connect Four most challenging.
  • Text-only inputs: Text-only inputs show an overall standard/inverse asymmetry of 69.22%/58.62%, while image-based CoT and descriptive prompting perform better.
  • Input ordering: Text-first ordering tends to amplify standard–inverse gaps for Base, Glyph, and Checkerboard, whereas Alias and descriptive prompting remain more balanced.

B.6 VLMBias Results

VLMBias performance varies substantially by subset and generally improves with alias-based prompting, although prompt and image perturbations do not fully close accuracy gaps.

  • 25.89, 24.23, 30.57, and 29.02: Flags has the highest averaged accuracy across the four input configurations.
  • 3.62/5.70: Animals is the most difficult subset under base settings but reaches 22.17 with Flip+Alias.
  • 11.39 to 16.16: Game Boards improves with prompting and perturbations but remains in a mid-low accuracy range.
  • 13.85 to 16.65/15.72: Logos likewise improves across configurations without fully closing the subset-level gap.
  • Accuracy and Bias are reported across Animals, Flags, Game Boards, and Logos for four input configurations.

C.2.1 Dataset construction

The study constructs synthetic leg-counting data and documents supplementary evaluation and steering artifacts for post-training transfer analyses.

  • 8192 image–text pairs: the synthetic counting dataset is balanced 50/50 between bird glyphs and quadruped-style animal glyphs.
  • Bird counts use {1, 2, 3}, whereas quadruped counts use {3, 4, 5}, with procedural variation in pose, shape, color, and background.
  • The shared instruction requires counting legs and answering only with a number in curly brackets.
  • Additional transfer results are provided for Molmo2-4B and InternVL3.5-4B on D1–D3 and synthetic-leg transfer.

D.1 VLM-Fix

VLM-Fix steering evaluates whether late decoder representations can be edited to address rule-remapping errors across four abstract games.

  • 100 underlying board states per game: paired variants cover standard/inverse rules, winner/loser queries, and image-first/text-first order.
  • 70/30 state-level splits repeated over three random splits prevent leakage across paired variants.
  • At each of the final 12 decoder layers, a two-stage router classifies rules and rule-conditional answers before a single query-token patch.
  • α = 1.0: steering uses a fixed intervention scale, with curves reporting means and standard deviations across three splits.
  • Additional steering results span Qwen2.5-VL-3B, Molmo2-4B, Molmo2-8B, and InternVL3.5-4B.
Loading 2604.12119v1…