PaperlayerView exact revision on arXiv

Source-linked AI summary

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

Drandreb Earl O. Juanico, Rowel O. Atienza

arXiv:2609.30993v1cs.CVcs.AI

TL;DR

Behavioral changes under internal intervention in open-weight VLMs are ambiguous: they may reflect better visual evidence use, instability, or degradation. FLIP applies elementwise flooring to the normalized final state and evaluates intervention strength with a probe-and-sweep protocol. Its bounded response and control-sensitive signature support using it as a screen for mechanistic follow-up, not as a steering method.

  • Problem

    Behavioral change under intervention in open-weight VLMs may reflect improved visual evidence use, generic instability, or degradation, so a single intervention effect is mechanistically ambiguous.

    Source: §1 · p. 1

  • Method

    FLIP floors the post-normalization final hidden state before logit computation and evaluates intervention effects across a strength sweep using a four-criterion validation protocol.

    Source: §1 · p. 1

  • Results

    The Final-site flooring sweep yields a bounded, grounding-proxy-aligned response, while raw decoder-layer interventions and the left/right negative control do not reproduce its signature.

    Source: §1 · p. 1, §8 · pp. 5, 8

  • Takeaways & Limitations

    FLIP is a population-level screen for deciding when downstream patching, circuit tracing, or causal-abstraction analysis is warranted, not a circuit identification result or steering method.

    Source: §9 · p. 8, §8 · p. 8

    FLIP does not identify the circuit, and its grounding-proxy analysis is scoped to detection/counting, where ∆R50 is observable but not a literal within-pass mediator.

    Source: §8 · p. 8

Abstract

from arXiv · show

We present FLIP, a final-layer inference-time probe for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation. Behavioral change under internal intervention is otherwise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hidden state before logit computation, leaving parameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negligible change, a bounded interior regime in which detection recall at IoU 0.50 ($R_{50}$) improves while tolerant counting error ($\mathcal{E}_{\mathrm{count}}$) falls, and over-suppression. We formalize a four-criterion probe-and-sweep protocol for disciplining the interpretation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions, including the last-block output before final normalization, and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multiple VLMs replicate it. FLIP is therefore a validation step for intervention-based mechanistic interpretability, not a steering method.

1. Introduction

FLIP tests whether intervention effects at a VLM’s logit-facing site reflect structured, task-linked computation rather than generic perturbation. The paper proposes a probe-and-sweep protocol and applies it to a final-layer flooring probe.

1. Introduction

  • Behavioral changes under intervention can reflect improved visual evidence use, generic instability, or degradation, making single-point effects ambiguous.The protocol evaluates intervention sites across intervention strengths rather than interpreting a single behavioral change in isolation.Source: §1 · p. 1
  • A bounded interior flooring regime improves R50 and reduces Ecount, between negligible change and over-suppression.The authors frame this controlled detection/counting response as evidence to evaluate through the full probe-and-sweep protocol.Source: §1 · p. 1
  • FLIP floors the post-normalization Final-site hidden state before logit computation, using the standard output-head baseline to interpret the intervention.The probe-and-sweep protocol treats intervention effects as requiring validation, not as self-interpreting performance changes.Source: §1 · p. 1
  • A positive effect must also align with grounding proxies, depend on feature coherence, contrast with a task-specific negative control, and be specific to the logit-facing site.The protocol-level signature is replicated across same-site magnitude-bounding operators and multiple VLMs.Source: §1 · pp. 1–2

2. Background and Related Work

Prior interpretability approaches use interventions to study causal variables, implicated components, or computational graphs, while VLM work examines visual processing and latent representations. FLIP addresses the preceding question of whether a candidate site merits mechanistic follow-up.

2. Background and Related Work

  • Causal abstraction, mediation, patching, circuit discovery, and tracing investigate internal variables, components, or computational graphs through interventions.The passage cites causal abstraction [Geiger et al., 2021], causal mediation, activation patching and circuit discovery [Zhang and Nanda, 2024; Conmy et al., 2023], and circuit tracing.Source: §2 · p. 2
  • Mechanistic conclusions can depend on the selected behavior, metric, examples, and ablation design.The passage notes this dependence for stronger mechanistic follow-up methods.Source: §2 · p. 2
  • FLIP asks whether a candidate site has a population-level, task-linked, control-sensitive signature that warrants mechanistic follow-up.Related VLM work also studies visual-token processing, interpretable feature bases, and latent or hidden-state steering [Neo et al., 2024; Pach et al., 2025; Yang et al., 2026; Juanico et al., 2025; Li et al., 2025].Source: §2 · p. 2

3. FLIP: Final Layer Inference-Time Probing

FLIP applies elementwise flooring to the normalized final hidden state immediately before output logits are computed. It is an inference-time probe, not an optimization method.

3.1. Operational Definition

  • The tensor z(t) is the post-normalization Final-site state passed to the output head, with a prompt/image prefill sequence and single-position cached decoding steps.FLIP leaves weights, prompts, tokenization, decoding rules, and KV caches unchanged.Source: §3.1 · p. 2
  • Components below strength ϑ are clamped elementwise in the post-normalization Final-site tensor before logit computation.The operator applies during generation, covering prefill logits and subsequent autoregressive decoding steps.Source: §3.1 · p. 2, §3 · p. 2
  • The probed Final site is distinct from raw decoder-layer outputs, including the pre-normalization last-block output; FLIP is not an optimization method.The main experiments target the post-normalization final hidden state passed to the output head.Source: §3.1 · p. 2

4. Tasks and Vision-Centric Evaluation Setting

The evaluation uses localization and counting as vision-centric probes, alongside additional vision-language benchmarks. The main-text experiments generally use Qwen3-VL-4B.

4. Tasks and Vision-Centric Evaluation Setting

  • Unless otherwise stated, main-text experiments use Qwen3-VL-4B, with additional model results reported in the appendix.Source: §4 · p. 2

4.1. Intermediate Visual Signals

  • Localization prompts ask the model to detect an object and provide bounding boxes, which are parsed into a common pixel-space representation.The task uses 7,056 image–prompt pairs from an image-clustered MS COCO val2017 subset, with one query per present object category [Lin et al., 2014].Source: §4.1 · p. 2

4.2. Reasoning-Based Counting

  • The counting prompt requests an integer object count to isolate errors attributed to visual instance discrimination rather than formatting.Source: §4.2 · p. 2

4.3. Vision-Centric Benchmarks

  • MMStar, MindCube, and NaturalBench are evaluated through LMMS-Eval as intervention probes rather than optimization targets.Source: §4.3 · p. 2

5. Theory

The theory frames FLIP as a simple flooring probe whose effects need interpretation against the output head, not as a new representation theory. Flooring defines a sweep-strength axis, while useful changes must project onto task-relevant output directions.

5. Theory

  • FLIP’s operator facts provide a baseline for interpreting the probe, not the paper’s main novelty.Because it is not trained, task-conditioned, or optimized, any useful effect must align its induced displacement with task-relevant directions already present in the logit-facing representation.Source: §5 · pp. 2–3

5.1. Componentwise Flooring and Perturbation Growth

  • As ϑ increases, flooring moves from near-inactivity through a bounded intermediate region to widespread clamping.This geometry defines the probe’s strength axis but does not by itself determine decoded task metrics.Source: §5.1 · p. 3, Figure 1 · p. 3

5.2. Standard Output-Head Alignment Baseline

  • The output-head baseline makes task relevance a projection question: an object-agnostic clamp is useful only insofar as it projects onto task-relevant output directions.The protocol tests whether that necessary condition is accompanied by grounding-proxy alignment, feature-coherence dependence, logit-facing site specificity, and negative-control contrast.Source: §5.2 · p. 3

6. Grounding-Proxy Association Analysis

The analysis treats detection recall as a proxy for an unobserved grounding state and examines its association with tolerant counting error. Because localization and counting use separate prompts and outputs, the analysis does not establish within-pass mediation.

6. Grounding-Proxy Association Analysis

  • The analysis defines ΔR50 as R50−E[R50 | 1FLIP = 0]; for counting, y is the parsed model count and y⋆ is the ground-truth count.Source: §6 · p. 3
  • Separate localization and counting prompts make ΔR50 an observable proxy for an unobserved grounding state, not a literal within-pass mediator of Ecount.The outputs are scored independently, so the association does not establish that one output directly causes the other.Source: §6 · p. 3
  • The product summary combines a Gaussian GLM for the a-term with a Poisson GLM on strict counting responses for the b-term.Confidence intervals are large-sample clustered intervals over image-level units, not intervals requiring normally distributed individual errors.Source: §6 · p. 4

7. Experiments

The experiments test FLIP on a clustered COCO detection-and-counting probe and evaluate its response across intervention strengths. A bounded interior regime aligns increased detection recall with reduced tolerant counting error, while controls and site comparisons constrain how that pattern is interpreted.

7.1. Experimental Setup

  • The core setup uses 7,056 clustered COCO image–query pairs with Qwen3-VL-4B-Instruct, a fixed pre-specified ϑ grid, and image-cluster-level uncertainty.Pointwise tests across ϑ are descriptive screening; regime identification uses the full sweep and checks subsample stability.Source: §7.1 · p. 4, §7 · p. 4

7.2. Metrics

  • Detection recall and tolerant counting error are the primary quantities; relevance mass reallocation is an appendix-side redistribution diagnostic, not a causal measure.RMR tracks RMR+, RMR−, and RMRobj.Source: §7.2 · p. 4

7.3. Results

  • Intermediate flooring raises ΔR50 while reducing Ecount, and over-suppression reverses the effect; Table 1’s product summary is non-zero in the recall-peak ϑ window.The separate localization and counting passes support a shared grounding-proxy interpretation, not direct within-pass causal mediation.Source: §7.3 · p. 4, Table 1 · p. 5
  • Only the post-normalization Final site exhibits the full interior regime; raw decoder-layer and percentile-matched sweeps do not recover its dose–response.The raw-layer contrast is not explained by thresholds that are too weak or poorly scaled.Source: §7.3 · p. 5
  • Disrupting feature coherence removes the positive interior window and leads to earlier collapse, though the nonzero-λ settings are not strictly ordered.The control retains any permutation baseline damage rather than subtracting it away.Source: §7.3 · p. 5
  • Flooring, symmetric clipping, and additive shift each produce regime-shaped primary-task curves, so curve shape or peak ΔR50 alone is not specific evidence.The full operator replication evaluates the joint signature, including grounding-proxy alignment, coherence dependence, site specificity, and the left/right control.Source: §7.3 · p. 5
  • Qwen3-VL-8B, Qwen3-VL-4B, and Kimi-VL-A3B show the qualitative three-phase structure, while the singleton-pair left/right control does not reproduce a coherent positive interior regime.The left/right response stays near baseline or drifts downward before late collapse.Source: §7.3 · p. 5
  • For the lower-amplitude Qwen3-VL-8B case, RMR+ tracks the interior regime, while RMR− and RMRobj mark transition to over-suppression.Additional structured sweeps are reported on MMStar, MindCube, and NaturalBench; these remain secondary to the mediator-aligned protocol evidence.Source: §7.3 · p. 5

8. Discussion

The discussion treats FLIP’s Final-site response as credible only through a joint signature of task alignment, feature-coherence dependence, control contrast, and readout-site specificity. It positions FLIP as a screening step for circuit-level follow-up, not as circuit identification or proof of understanding.

8. Discussion

  • Figure 3 shows a bounded interior gain in detection recall at the Final site, while the left/right negative control lacks the same positive regime.The control stays near baseline over most non-collapse strengths and degrades under stronger intervention.Source: §8 · p. 5, Figure 3 · p. 6
  • Figure 4 shows that raw decoder-layer and percentile-matched sweeps do not reproduce the post-normalization Final site's full dose–response.The comparison argues against a simple raw-threshold calibration explanation under the tested sweeps.Source: §8 · p. 5, Figure 4 · p. 6
  • Figure 5 shows that disrupting feature coherence makes the positive interior window disappear earlier and collapse sooner than the unpermuted condition.All curves use the same unpermuted no-intervention baseline, retaining any baseline damage from permutation.Source: §8 · p. 5, Figure 5 · p. 7
  • Figure 6 shows the qualitative three-phase response across three VLMs, but the left/right control has no reproducible bounded interior optimum across models.Qwen3-VL-8B has the lower-amplitude detection response; Qwen3-VL-4B and Kimi-VL-A3B have larger peaks.Source: §8 · p. 5, Figure 6 · p. 7
  • FLIP is population-level triage, not circuit identification, a deployment-safety certificate, an optimization method, or proof of model understanding.The grounding-proxy account is scoped to detection/counting, and the raw-layer result is a readout-space contrast under the implemented sweep, not evidence that earlier layers lack task-relevant information.Source: §8 · pp. 5, 8

9. Conclusion

FLIP is a population-level validation screen for intervention effects, using a non-targeted flooring probe and independent checks rather than treating performance gains as self-interpreting. Its interpretation is limited to a grounding-proxy account and does not identify a circuit or establish direct mediation.

9. Conclusion

  • The conclusion frames FLIP as a lightweight screen for deciding when downstream patching, circuit tracing, or causal-abstraction analysis is warranted.The protocol requires grounding-proxy alignment, feature-coherence dependence, performance-based negative-control contrast, and logit-facing readout specificity.Source: §9 · p. 8

A. Generic Perturbation: Definition and Scope

  • Generic perturbation means behavioral change from non-specific representational disruption rather than alignment with task-relevant intermediate computations.The paper operationalizes the distinction through grounding-proxy alignment, feature-coherence controls, and negative-control tasks.Source: A. Generic Perturbation: Definition… · p. 10

B. Detection Matching and Mediator Construction

  • Detection matching uses one-to-one assignment at IoU 0.50 to obtain per-sample TP50 counts, from which recall R50 is computed.Predicted-box area weights overlap only to break matching ties; mean IoU is not the primary mediator.Source: B. Detection Matching and Mediator… · p. 10

C. Output-Head Alignment: Scope and Interpretation

  • Because flooring is not task-conditioned, useful effects depend on its displacement projecting onto task-relevant output-head directions.Decoded R50 and Ecount need not vary continuously or monotonically with intervention strength because decoding and scoring include discontinuous operations.Source: C. Output-Head Alignment: Scope and… · p. 10

D. Grounding-Proxy Interpretation and Limitations

  • The association between localization recall and counting error is grounding-proxy evidence, not proof of direct within-pass mediation or causal sufficiency.Separate prompts prevent the localization output from literally causing the separately generated counting output; a shared representational cause remains possible, though constrained by controls.Source: D. Grounding-Proxy Interpretation and… · pp. 10–11

E. FLIP Deployment and Runtime Intervention

  • FLIP floors the post-normalization Final-site tensor before output projection, with externally supplied strength enabling fixed-grid sweeps without changing parameters, tokenization, or decoding.The Final-site patch leaves early visual feature extraction unchanged and perturbs the normalized readout state where image/text evidence is mapped to logits; core sweeps used one A100 GPU.Source: E. FLIP Deployment and Runtime… · p. 11

F. Cross-Architecture Validation

Cross-architecture tests find a shared qualitative dose–response pattern across three VLMs, with differences in peak size. The auxiliary indoor/outdoor diagnostic measures answer instability rather than task performance, while mediation replications retain the reported qualitative pattern.

F. Cross-Architecture Validation

  • All three models show the same qualitative three-phase detection response, with larger interior peaks for Kimi-VL-A3B and Qwen3-VL-4B than for Qwen3-VL-8B.Kimi-VL-A3B differs from the Qwen family in visual encoder, cross-modal fusion, and training recipe.Source: F. Cross-Architecture Validation · p. 11
  • The indoor/outdoor response-switch diagnostic measures answer changes relative to baseline, so it checks generic output instability rather than task performance.The performance-based negative-control contrast comes from left/right accuracy results.Source: F. Cross-Architecture Validation · p. 11
  • Cross-architecture mediation summaries preserve the qualitative interpretation of an interior regime with IRRindirect < 1 and reversal under over-suppression.The reported replications cover Kimi-VL-A3B and Qwen3-VL-8B.Source: F. Cross-Architecture Validation · p. 11

G. Raw Decoder-Layer Checks and Logit-Facing Specificity

Raw decoder-layer interventions do not reproduce the complete Final-site signature, even when intervention scale is matched. Redistribution diagnostics offer a comparative view of the sweep’s regimes, but are not causal evidence or a replacement for the primary behavioral analysis.

G. Raw Decoder-Layer Checks and Logit-Facing Specificity

  • Raw decoder-layer sites do not reproduce the full feature-coherence-dependent Final-site signature under percentile-matched sweeps, though late layers may show partial ∆R50 movement.Layers 8 and 16 have matched active-set fractions but much smaller clamp magnitudes at ϑ = 0, followed by rapid saturation; Layer 32 is closer but still lacks the full signature.Source: G. Raw Decoder-Layer Checks and… · pp. 11, 13, Table 4 · p. 13, Figure 8 · p. 14

H.1. Target-Conditioned Patch Relevance and RMR Computation

  • RMR summarizes target-conditioned redistribution toward object patches, away from them, and in object-versus-background contrast.These diagnostics refine the sweep’s interpretation but do not replace the primary evidence based on ∆R50 and Ecount.Source: §H.1 · p. 15, §H.3 · p. 15

H.2. Comparative role of rollout

  • Rollout relevance is most reliable for comparisons across ϑ under the same model, prompt, and target; absolute saliency magnitudes should not be over-interpreted.If a target token is absent at a given ϑ, that sample is marked missing for RMR computation at that strength.Source: §H.2 · p. 15

H.3. Regime interpretation of the RMR diagnostics

  • At very negative ϑ, small ∆R50 and RMR+ accompany comparatively modest RMR− and RMRobj near zero, marking the negligible-change regime.Source: §H.3 · pp. 15–16
  • In the interior regime RMR+ co-varies with ∆R50, while over-suppression brings declining RMR+, sharply rising RMR−, and a late RMRobj spike.The late RMRobj rise is more consistent with suppression-dominant contrast enhancement than with a stronger constructive interior regime.Source: §H.3 · p. 16

I. Vision-Centric Benchmark Consistency

This section examines FLIP’s structured, non-monotonic responses on vision-centric benchmarks as consistency checks, not mediator-aligned or causal evidence. It formalizes flooring and task-relevant output-head alignment, contrasts FLIP with prompt control, scopes the core experiments to Qwen3-VL-4B-Instruct, and covers negative controls, operators, feature coherence, and robustness diagnostics.

I. Vision-Centric Benchmark Consistency

  • MMStar, MindCube, and NaturalBench all show structured, non-monotonic responses, with benchmark-specific differences in their interior regimes.MMStar has a broad elevated plateau, MindCube the clearest three-phase pattern, and NaturalBench an interior peak followed by degradation. These are regime-consistency checks, not mediator-aligned evidence; their error bars reflect LMMS-Eval run variability.Source: §I · p. 16, Figure 10 · p. 18

J. Negative-Control Task Design

  • The accuracy-scored singleton-pair left/right task tests whether a distinct spatial judgment reproduces the detection/counting pathway’s positive interior regime, rather than merely changing answers.Pairs are drawn from singleton object categories with horizontal separation and approximately balanced yes/no prompts; the control is not designed to isolate the detection-derived mediator.Source: J. Negative-Control Task Design · p. 17, §J.1 · pp. 17–18

K. Baseline Centering, Paired Designs, and Heterogeneity

  • Baseline-centered and paired-difference analyses target the same treatment contrast, while both instance-count strata retain positive interior effects and reverse under over-suppression.Large-count gains attenuate earlier and are non-significant at ϑ = 0.5; the reversal at ϑ = 1.0 is significant in both strata. The analysis addresses instance count, not all possible sources of heterogeneity.Source: K. Baseline Centering, Paired Designs,… · pp. 18–19, L. Regression Specifications and… · p. 19

L. Regression Specifications and Robust Inference

  • Image-clustered standard errors account for dependence among queries and matched baseline/intervention observations; Gaussian and Poisson GLMs supply the reported a-term and b-term, respectively.The auxiliary ZeroInflatedPoisson model is used only for the Ecombined format-failure/tolerant-error decomposition, not the reported product summary.Source: L. Regression Specifications and… · p. 19

M.1. Full Four-Criterion Replication Across Same-Site Operators

  • Same-site flooring, symmetric clipping, and additive shift retain a bounded positive primary-task regime, while the left/right accuracy control does not reproduce a comparable positive interior regime.The operator comparison concerns displacement geometry at the logit-facing representation, not theoretical identity among operators; the protocol tests mediator alignment, feature-coherence dependence, site localization, and negative-control behavior.Source: M. Operator-Comparison Baselines and… · p. 19, §M.1 · p. 20, §M.3 · pp. 20–21

M.2. Graduated Permutation Control

  • Compared with λ = 0, all nonzero λ settings lose the positive regime earlier and enter collapse sooner, although ordering among nonzero λ values is not strictly monotonic.The graduated permutation control keeps the flooring instantiation fixed and tests whether its positive interior regime depends on feature coherence.Source: §M.2 · p. 20

N. Subsample Stability of the Interior Regime

  • Image-cluster subsampling preserves the qualitative interior regime, with means tracking the full-sample curve and variability decreasing as sample size grows.The analysis treats the ϑ grid as descriptive and checks whether the FWHM regime window remains visible under subsampling; Figure 14 provides a representative visual of negligible change, interior improvement, and over-suppression.Source: N. Subsample Stability of the Interior… · p. 21, Figure 14 · p. 21

P.1. Componentwise Flooring Identity and Monotonicity

  • Componentwise flooring makes each intervention entry, its active support, and its p-norm non-decreasing as the flooring strength increases.Source: §P.1 · p. 22

P.2. Output-Head Margin Expansion

  • At the post-normalization Final site, a first-order logit-margin increase requires positive alignment between the FLIP displacement and task-relevant output-head directions.Source: §P.2 · p. 22

Q. FLIP versus Prompt-Based Control

  • Unlike prompt engineering, which steers through input distributions and learned priors, FLIP intervenes after tokenization and cross-modal fusion on the post-normalization Final-site state.FLIP cannot inject information absent from internal representations and fails gracefully when evidence is weak or over-suppressed.Source: Q. FLIP versus Prompt-Based Control · p. 22

R. Model Choice, Scope, and Claims

  • The core detection/counting and mediator-regime claims are scoped to Qwen3-VL-4B-Instruct, the post-normalization Final site, and the evaluation protocol.The stated contribution is the methodology of regime-structured sweeps, a mediator-based probe, and redistribution diagnostics; cross-architecture validation is reported in Appendix F.Source: R. Model Choice, Scope, and Claims · p. 22

S. Auxiliary Instability Diagnostic: Global Scene Classification

  • Across bounded interventions, the primary task shows an interior positive regime, but the left/right control does not reproduce it and feature-coherence disruption destabilizes the effect.Flooring has the largest interior peak; symmetric clipping collapses earliest, while additive shift stays positive over a broader range before delayed collapse. The interior regime remains qualitatively stable across image-cluster subsamples, whose means track the full-sample curve closely and whose variability decreases with scale.Source: Figure 11 · p. 24, Figure 12 · p. 25, Figure 13 · p. 25
Loading 2609.30993v1…