PaperlayerView exact revision on arXiv

Source-linked AI summary

Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

Arun Josephraj Arokiaraj, Chenyi Huang, Zekun Wu, Adriano Koshiyama

arXiv:2610.03445v2cs.CVcs.AI

TL;DR

VLMs may show near-zero teacher-forced loss for a target caption while freely generating the original description, leaving unclear which model stage determines attack success. Using controlled PGD and layer-wise logit-lens analysis on Qwen2.5-VL-7B-Instruct, the paper finds a fixed single-step failure and decoder-level divergence between susceptible and resistant images.

  • Problem

    It is unclear which stages of a VLM’s visual encoder, cross-modal merger, and autoregressive decoder determine whether an adversarial perturbation changes free-form output.

    Source: §1 · pp. 1–2

  • Method

    The authors apply controlled two-stage PGD to 200 held-out COCO images and use the logit lens to track target-token ranks through the model.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    At the merger output, target-token rank drops comparably across groups, but by the final layer Group A shifts −3,754 while Group C shifts +1,580 (Mann-Whitney p < 0.001, rank-biserial r = 0.579).

    Source: §6 · p. 5

  • Takeaways & Limitations

    The results locate the arbitration of adversarial outcomes in the language decoder rather than the visual encoder, and motivate measuring semantic drift and internal corruption in faithfulness evaluations.

    Source: §1 · p. 2, §7 · p. 6

    Findings are correlational and limited to one model and one target caption, so the paper does not claim architecture-general or target-general conclusions.

    Source: §8 · p. 6

Abstract

from arXiv · show

A targeted adversarial perturbation can drive a vision-language model's (VLM's) teacher-forced training loss for a fixed target caption to near zero, yet the same model, allowed to generate freely, produces the original, correct description with no trace of the target. We call this dissociation the train/inference gap, and give it a precise mechanistic account on Qwen2.5-VL-7B-Instruct using a controlled two-stage PGD attack on 200 held-out COCO images. First, we show that image-level pixel statistics, including a correctly re-implemented, texture-based attackability measure from the CNN robustness literature, have essentially no predictive power over which images are corrupted (best predictor r=-0.050, p=0.484; ridge regression R^2=0.069). Second, using the logit lens, we localise the gap to a single autoregressive step: the rank of the target token, conditioned on the correct first token already being generated, is fixed at exactly 3,488 out of 152,064 vocabulary entries for every image and every condition, with zero variance. Third, tracking target-token rank across all 28 LLM decoder layers reveals that the visual encoder corrupts every image's representation by a comparable margin regardless of eventual outcome, but the language model decoder then differentially arbitrates: amplifying the corrupted signal for susceptible images and actively suppressing it, past its clean-image baseline, for resistant ones (p<0.001, rank-biserial r=0.579). A linear probe on the merger hidden state separates these two outcomes with AUC=0.858, though we flag a circularity concern in this estimate. Together these results argue that adversarial robustness in autoregressive VLMs is substantially a property of the language decoder's prior, not the visual encoder, with direct implications for where faithfulness evaluations and defenses for deployed VLM systems should be targeted.

1 Introduction

The paper studies why a targeted perturbation can nearly eliminate teacher-forced loss while free generation retains the original description. It uses token-level and layer-wise analysis to locate the gap and examine which model stages determine attack outcomes.

1 Introduction

  • PGD can drive teacher-forced loss for a fixed target caption near zero even as free generation produces the original description almost unchanged.The paper treats this train/inference gap as its central object and gives it a token-level, mechanistic explanation.Source: §1 · pp. 1–2
  • The study asks whether pixel statistics predict image corruption, where the train/inference gap occurs, and whether the decoder passively misses or actively suppresses corrupted visual signals.It analyzes 200 held-out COCO val2017 images with controlled two-stage PGD and the logit lens at each vision-encoder and language-decoder layer.Source: §1 · p. 2
  • The authors argue that robustness is not primarily a visual-encoder property and that pixel- or encoder-level evaluation alone may miss where the decision is made.They frame this as relevant to faithful, grounded VLM deployment.Source: §1 · p. 2
  • The paper reports three contributions: testing texture-based attackability prediction, localizing the gap to one generation step, and tracing decoder amplification or suppression of corrupted signals.The authors describe the texture measure as correctly re-implemented and the layer-wise account as showing the decoder, rather than the visual encoder, as the locus of adversarial resistance.Source: §1 · p. 2

2 Related Work

Prior work studies image hijacks, pixel-level predictors of CNN susceptibility, and interpretability tools for visual-language models. This paper builds on those lines by measuring the train/inference gap token-by-token and applying the logit lens to adversarial visual tokens.

2 Related Work

  • Bailey et al. [2024] observed qualitatively that teacher-forced convergence need not yield free-generation success on LLaVA-13B, with a wider gap for longer target sequences.The paper quantifies the gap at the single-token level rather than reporting only aggregate attack success rate.Source: §2 · p. 2
  • CNN-focused attackability work includes Object Texture Intensity and 72 GLCM features, but these measures were developed and validated on convolutional image classifiers.The paper tests whether those predictors transfer to an autoregressive VLM’s visual encoder.Source: §2 · p. 2
  • The logit lens projects intermediate hidden states through the unembedding matrix, and prior work found visual object information readable by middle-to-late layers on clean images.This paper applies the lens to adversarially perturbed visual tokens at encoder and decoder levels and relates layer-wise signals to attack outcome.Source: §2 · p. 2
  • Linear probes test whether properties are linearly encoded in hidden states, while FFN sublayers have been described as key-value memories that inject language-prior information into the residual stream.That prior motivates the hypothesis that late-layer FFN-mediated computation may suppress corrupted visual signals.Source: §2 · pp. 2–3

3 Experimental Setup

Experiments use a frozen Qwen2.5-VL-7B-Instruct model and 200 stratified COCO val2017 images with a fixed three-token target caption. A two-stage PGD protocol and layer-wise instrumentation track how the target signal moves through the model.

3 Experimental Setup

  • The study uses frozen Qwen2.5-VL-7B-Instruct, with a 32-block ViT encoder, a 28-layer LLM decoder, and a vocabulary of 152,064 tokens.The merger reduces 1,024 patch tokens to 256 visual tokens in the LLM embedding space.Source: §3 · p. 3
  • The dataset comprises 200 COCO val2017 images, stratified by supercategory and resized to 448×448, with “A coconut.” as the fixed three-token target caption.The caption was chosen for semantic distance from every COCO category so that the target concept’s appearance is unambiguous.Source: §3 · p. 3
  • A two-stage momentum PGD attack first targets the visual encoder alone, then refines the perturbation with the target caption teacher-forced under full captioning context.The protocol uses μ = 0.9, ε = 16/255, α = 4/255, and 500 iterations per stage; groups are defined by Stage-1 loss and free-generation target hits.Source: §3 · p. 3
  • Hooks on all 32 ViT blocks and 28 decoder blocks support logit-lens tracking from the merger output through visual-token positions and the final generation position.The analysis tracks target-token rank in the full vocabulary distribution.Source: §3 · p. 3

4 RQ1: Encoder Susceptibility Is Not Predictable from Pixels

The paper tests whether image-level pixel features predict susceptibility to encoder-targeted attacks and finds little predictive signal. The reported pattern instead varies with semantic category, while an included per-token analysis reports a fixed target-token rank across images and conditions.

4 RQ1: Encoder Susceptibility Is Not Predictable from Pixels

  • The OTI reimplementation uses COCO polygon masks, channel-wise Sobel filtering, and normalization by C×H×W; a naive implementation produced a misleadingly larger correlation (r = −0.218).The fingerprint also includes texture, frequency-band energy, structural regularity, object size and count, colour entropy, and edge density.Source: §4 · p. 3
  • The correctly re-implemented OTI measure has essentially zero correlation with Stage-1 loss on Qwen2.5-VL-7B (r = −0.050, p = 0.484).Adding 72 GLCM features and using a Random Forest degrades accuracy from 69.0% to 64.0%, while ridge R2 falls from +0.069 to −0.055.Source: §4 · p. 3
  • Susceptibility follows semantic-category structure rather than measured pixel features: trucks and trains are disproportionately susceptible (50–67%), while birds and beds are fully resistant (0%).The contrasting categories have comparable or higher texture intensity in the resistant examples.Source: §4 · p. 4
  • The null predictor results rule out a simple texture-based image heuristic for pre-screening attack targets or hardening images.The authors use this result to motivate examining the model rather than its input.Source: §4 · p. 4

5 RQ2: The Gap Is an Architectural Constant, Not a Statistical Tendency

The target token’s rank is invariant at the second generation step, exposing a structural failure hidden by the reduced teacher-forced loss. Across decoder layers, susceptible and resistant images then follow different target-rank trajectories.

5 RQ2: The Gap Is an Architectural Constant, Not a Statistical Tendency

  • 3,488 of 152,064 vocabulary entries is the target token’s rank at the second step for every image and condition, with zero variance.Greedy target choice at this step is 0.0% across the 200 images.Source: §5 · p. 4
  • Across 28 decoder layers, Group A remains shifted toward the target while Group C crosses into suppression below its clean baseline around layers 15–18.The per-layer logging used a random 50-image subsample of the 153 Group C images.Source: Figure 2 · p. 5, §5 · p. 5
  • The three-token average loss can fall because later-token losses decrease even though the perturbation fails identically at the second token.At this context, the rank is an architectural constant invariant to the ε = 16/255 perturbation.Source: §5 · pp. 4–5

6 RQ3: The Language Decoder, Not the Encoder, Arbitrates the Outcome

The merger output shows comparable corruption across groups, but their trajectories diverge in the LLM decoder. A merger-state probe separates susceptible from resistant images, though shared dependence between labels and inputs makes that evidence suggestive rather than conclusive.

6 RQ3: The Language Decoder, Not the Encoder, Arbitrates the Outcome

  • At the merger, target-token rank shifts are −5,724 for Group A and −3,619 for Group C, but final-layer shifts diverge to −3,754 and +1,580, respectively.The final-layer difference is significant (Mann–Whitney p < 0.001, rank-biserial r = 0.579); the Group C shift is beyond its clean-image baseline.Source: §6 · p. 5
  • Group C’s target-rank shift crosses into suppression around decoder layers 15–18, while Group A maintains a positive shift toward the target.The authors note that this depth is consistent with FFN sublayers injecting corrective language-prior information, but the layer-wise evidence is correlational.Source: §6 · p. 5
  • The merger-state linear probe achieves AUC = 0.858 ± 0.076 on adversarial features, versus 0.630 ± 0.034 on clean features of the same images.The outcome is linearly decodable from internal state before generation begins.Source: §6 · p. 5
  • The probe’s separability is suggestive, not conclusive, because its labels depend on Stage-1 loss computed from features of the representation it probes.This shared dependence may account for part of the reported separability.Source: §6 · p. 6
  • Pixel statistics, token rank, layer-wise logit lens, and probing converge on susceptibility being invisible at input, uniform at the encoder, and decided at the decoder..Source: §6 · p. 6

7 Discussion: Implications for Grounded and Faithful VLM Deployment

The discussion argues that pixel- or encoder-level defenses may miss where the model’s behavior is decided, and that robustness evaluation should look beyond exact target-string matches. It also cautions that the language prior’s protection may be weaker for cluttered scenes, an implication not tested in deployment.

7 Discussion: Implications for Grounded and Faithful VLM Deployment

  • Pixel- or encoder-only defenses are likely insufficient; the authors suggest placing a lightweight anomaly detector after the merger and before the LLM decoder.The proposed location is where the representational difference between eventually amplified and suppressed images is largest.Source: §7 · p. 6
  • The language prior overrides corruption for canonical semantic content but not reliably for cluttered or multi-element scenes, with implications for agentic settings left untested.The paper gives birds and beds as canonical examples, and trucks, trains, and crowded interiors as cluttered examples.Source: §7 · p. 6
  • Faithfulness evaluations should measure semantic drift and internal representational corruption, not only exact-match success against a fixed target phrase.An exact-match-only protocol would report 0.0% attack success here while missing corruption of the visual encoder for every tested image.Source: §7 · p. 6

8 Limitations

The mechanistic findings are scoped to one model and target caption, and the logit-lens and probing results are correlational rather than causal.

8 Limitations

  • The logit-lens and probing results are correlational; activation patching to test causal attribution for the layer 15–18 crossover was not technically feasible.The findings use one model, Qwen2.5-VL-7B-Instruct, and one target caption on 200 images; transfer, ensemble, and agentic-deployment results are not reported.Source: §8 · p. 6

Ethics and Responsible Disclosure

The work characterizes an existing vulnerability class rather than introducing a materially more effective attack. It releases mechanistic analysis to support defense research without releasing optimized attack checkpoints.

Ethics and Responsible Disclosure

Related papers

Loading 2610.03445v2…