PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMProbe: An Open-source Library for VLM Probing and Interpretation

Gexin Huang, Haisheng Fu, Yanting Yang, Chen Zhao, Ruinan Jin, Jiaqi Liu, Xiaoxiao Li

arXiv:2610.33072v1cs.CVcs.AI

TL;DR

Vision-language model responses do not reveal how visual information is encoded or used, motivating analysis that connects representations with spatial evidence and behavioral effects. vlmprobe brings complementary methods into a shared framework and evaluates them through matched cases and controlled dictionary studies. The comparisons find that interpretation depends on analysis choices and that different measures can provide conflicting or distinct evidence.

  • Problem

    VLM responses do not reveal how visual information is encoded, transported, or used to select an answer, making it important to connect representations with spatial evidence and computational effects.

    Source: §1 · p. 1

  • Method

    vlmprobe integrates lens readouts, attention attribution, causal interventions, and SAE analysis in a common interface with shared adapters, evaluation pipelines, and visualizations.

    Source: §1 · pp. 1–2

  • Results

    Readouts and spatial explanations depend on objectives and observation sites, local attribution can disagree with measured intervention effects, and SAE evaluations expose distinct trade-offs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support combining multiple forms of evidence and validating explanatory hypotheses against model behavior.

    Source: §1 · p. 2

    The demonstrated implementation provides semantic lexical-neighbor readout, but not the complete sink/dead/alive classification and pruning analysis.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Understanding how vision-language models use visual information requires connecting internal representations to model behavior. We present vlmprobe, an open-source library that integrates lens readouts, attention attribution, causal interventions, and sparse autoencoder (SAE) analysis within a common model interface. Shared adapters, calibration and checkpoint utilities, evaluation pipelines, and visualizations support systematic analysis across architectures. Using this framework, we compare methods across vision-language, language, and vision models through matched case studies and controlled evaluations of pretrained sparse dictionaries. The comparisons show that representation readouts and spatial explanations depend on the analysis objective and observation site, and that local attribution estimates can disagree with measured intervention effects. SAE evaluations further expose distinct trade-offs in reconstruction fidelity, concept readout, and spatial localization. Together, these results support combining multiple forms of evidence to evaluate an interpretation. vlmprobe provides a reusable basis for selecting methods, comparing their conclusions, and testing hypotheses about model behavior. The code is available on github.com/ubc-tea/vlm-probe

1 Introduction

The paper introduces vlmprobe to connect internal representations with spatial evidence and the effects of changing model computation. Its matched comparisons show that interpretation depends on the method, objective, and observation site, supporting the use of multiple forms of evidence.

1 Introduction

  • Understanding VLM behavior requires connecting internal representations to spatial evidence and to the effects of changing model computation.Responses alone do not reveal how visual information is encoded, transported, or used to select an answer.Source: §1 · p. 1
  • vlmprobe integrates lens, attention, causal, and SAE methods through a common model interface with shared evaluation and visualization utilities.Adapters preserve native activation sites and image–text token layouts while calibration and checkpoint utilities support analysis across architectures.Source: §1 · pp. 1–2
  • Matched cases and controlled pretrained-SAE studies make inputs, analysis units, targets, and controls explicit for comparisons among compatible methods.The studies examine reconstruction, concept readout, spatial localization, and decoder answer evidence.Source: §1 · p. 2
  • Readouts and spatial explanations depend on objectives and observation sites, local attributions can disagree with measured intervention effects, and SAE evaluations reveal distinct trade-offs.These findings motivate combining measurements and validating explanatory hypotheses against model behavior.Source: §1 · p. 2

2 Library and evaluation protocol

The library uses a shared contract for native model inputs, explicit analysis settings, and method outputs while retaining model-specific details. Its evaluation protocol organizes matched cases and compatible measures without treating distinct analysis units as a universal leaderboard.

2.1 Model adapters and method configuration

  • Native model tensors and coordinates, method settings, and outputs are retained together, allowing shared preparation and visualization without assuming identical architectures.Settings include methods, activation sites, positions, and target scores; paired inputs and calibrated artifacts are supplied when required.Source: §2.1 · p. 2, Figure 1 · p. 3
  • The worked example combines representation reading, attention-gradient explanation, and exact versus first-order residual replacement using shared method interfaces.Teacher-forced scoring uses a fixed answer, and a reference supplies aligned source activations.Source: §2.1 · p. 2

2.2 Models, inputs, and comparison groups

  • Shared VLM cases use predetermined inputs and preserve semantic targets across models while retaining native tokenization, image geometry, and token positions.Displayed maps are transformed back to the original photograph and its bounding-box coordinates.Source: §2.2 · p. 3, Table 2 · p. 4
  • The fixed-answer score includes every native answer piece except EOS, while absolute scores are unsuitable for cross-model calibration ranking because tokenizations differ.A negative ΔS indicates reduced support for the fixed answer; native generation is shown separately from explanations of forced answers.Source: §2.2 · pp. 3–4
  • Comparison groups organize matched inputs and compatible measures, while GPT-2 and ViT studies complement shared VLM cases rather than form a universal leaderboard.The SAE studies use frozen dictionaries, independent calibration and test data, reconstruction and spatial-shuffle controls, and source-native transforms and patch coordinates.Source: Table 3 · p. 4, §2.2 · p. 4

3 Lens methods

Lens readouts expose different aspects of model decisions: residual projections, head messages, and visual-token neighbors are not interchangeable evidence of recognition. Transfer readouts also depend on the decoding context, and likelihood changes alone do not establish successful information transfer.

3.2 Lexical neighbors reveal associations rather than object labels

  • EmbedLens neighbors associate a black-shirt patch with “black” but a yellow-shirt patch with “Blue,” and neither cosine similarity nor a large norm is a calibrated object prediction.The implemented semantic readout does not include the paper’s complete sink/dead/alive classification and pruning analysis.Source: §3.2 · pp. 5, 7, Figure 4 · p. 8

3.1 Residual and head readouts expose different levels of a decision

  • Across the IOI case, residual lenses select Danielle at layer 9, but layer-10 Attention Lens readouts vary: only heads 0, 2, 6, and 10 decode her.On the GQA example, Qwen3-VL generates Black while the other models generate Yellow; early vocabulary projections alone do not establish shirt recognition or locate a causal loss of visual information.Source: §3.1 · p. 5, Figure 2 · p. 6, Figure 3 · p. 7

3.3 Transfer readouts depend on the decoding prompt

  • In the color-copy context, a yellow-source residual raises Yellow’s complete-word log probability by 0.419 at layer 15 relative to zero injection, yet the top prediction is “is.”Both sources produce “is” at layers 7, 15, and 23, and “->” at layer 31; source site, mapping, and target prompt need validation.Source: §3.3 · pp. 7–8, Figure 5 · p. 9

4 Attention explanations

Across fixed inputs and targets, attention methods produce different spatial explanations, and a zero local map need not mean visual irrelevance. Matched deletion tests expose cases where plausible maps disagree with intervention effects.

4.1 Propagation rules change the map for a fixed answer

  • Although maps share an output shape, their scores are not equivalent: rollout follows residual-augmented attention, Grad-CAM uses local target gradients, and ATTATTR integrates path sensitivity.Raw attention, Grad-CAM, and ATTATTR inspect the last decoder attention layer, while propagation methods span the decoder attention stack.Source: §4.1 · p. 9, Figure 6 · p. 10
  • Across displayed models, generic relevance concentrates more within the shirt than rollout, but sharper maps do not establish greater faithfulness.These box fractions describe one input, and each map is independently normalized.Source: §4.1 · p. 9, Figure 6 · p. 10
  • ATTATTR is predominantly negative in four models but more mixed in InternVL; positive-only maps would conceal this sign distinction.Source: §4.1 · p. 10

4.2 A zero local map can coexist with visual dependence

  • SmolVLM, Qwen3-VL, and Pixtral have zero last-layer Grad-CAM maps despite nonzero propagation maps; residual replacement still changes SmolVLM’s answer score.Adapters expose attention probabilities used by supported eager decoders for ATTATTR, and a measured zero remains a result rather than an unavailable method.Source: §4.2 · p. 10

4.3 Deletion tests reveal disagreements hidden by plausible heatmaps

  • Deleting 39 of 196 patches leaves cat probability at 65.1% under rollout versus 0.27% under Beyond-C; dog Grad-CAM deletion raises dog probability from 2.99% to 29.3%.Deletion is sensitive to the replacement baseline and removal of competing objects.Source: §4.3 · p. 10, Figure 7 · p. 11

5 Causal interventions

The section compares replacement, knockout, path-patching, edge-retention, and steering interventions, showing that measured effects depend on the intervention site and analysis setup. Local attribution estimates can diverge from replacement effects, and steering outcomes vary by model and strength.

5.1 Attribution patching can miss large replacement effects

  • SmolVLM’s layer-5 visual replacement changed log P(Yellow) by −0.498 versus an estimate of −0.067, while LLaVA’s −1.646 effect was estimated as +0.100.The LLaVA estimate reverses the measured effect’s sign; the authors recommend testing important candidate sites with actual replacement.Source: §5.1 · p. 11, Figure 8 · p. 12

5.2 The intervention unit determines the apparent locus of dependence

  • Image-slot effects peak early, while prediction-slot effects are larger late; LLaVA’s prediction-slot effect is −1.646 on Yellow versus −0.172 on the black-shirt case.The final visual-slot effect is zero because no remaining decoder layer can transport the change to an answer position; these cases do not establish a universal pathway or robustness ranking.Source: §5.2 · pp. 12–13, Figure 9 · p. 13

5.3 Knockout tests information routes, while path patching isolates senders

  • On the yellow-shirt example, image-to-question knockout disrupts answer support more than direct image-to-prediction blockade; blocking other-region keys can instead increase support.The target region has 20 keys versus 556 outside it, so a size-matched control is needed to attribute the difference specifically to grounding.Source: §5.3 · pp. 13–14, Figure 10 · p. 14
  • Controlled path patching identifies influential sender heads only under its specified receiver and frozen-head rule, not a complete circuit or functional labels.The largest negative effects include Qwen3-VL L23.H6 (−0.054), Pixtral L34.H12 (−0.080), and LLaVA L14.H3 (−0.064).Source: §5.3 · p. 14, Figure 11 · p. 15

5.4 Circuit attribution requires measured recovery

  • At 184 retained edges, EAP recovers 0.258 of the original-reference margin difference, while EAP-IG recovers 0.045.This single-pair, graph-specific evaluation separates attribution ranking from validation by testing actual edge-retention interventions.Source: Figure 12 · p. 15, §5.4 · p. 16

5.5 Steering depends on both model and strength

  • The same raw steering scale changes different claims across models: LLaVA mentions visible rocks at 0.02 but invents other people at 0.05, while SmolVLM and Qwen3-VL change descriptions differently.Added visual detail and unsupported content can occur within the same description; identical scales need not produce equivalent perturbations across models.Source: §5.5 · pp. 16–17
  • At strength 0.17, person’s logit increases by 4.19 and car’s decreases by 4.38 despite repetitive free generation, so favorable token responses do not ensure grounded, fluent descriptions.This VSV steering study uses the raw block-residual rule without SLA and therefore is not a complete VISTA reproduction.Source: Figure 14 · p. 17, §5.5 · p. 17

6 Sparse features connect reconstruction, concepts and spatial structure

The section evaluates sparse autoencoders (SAEs) through reconstruction, concept readout, and spatial evidence, finding that performance depends on the measured objective and observation site. Decoder-feature maps also vary with the answer objective and identify answer-sensitive directions rather than object segmentations.

6 Sparse features connect reconstruction, concepts and spatial structure

  • The studies use frozen pretrained dictionaries and assess reconstruction, behavioral fidelity, concept readout, and localization; SAE feature indices gain interpretations from measurements on independent inputs rather than predefined labels.The language comparison uses six SAE families and matched Pythia-160M block-8 dictionaries, with shared evaluation tools and controls.Source: §6 · p. 18, §6.1 · p. 18

6.1 Matched language dictionaries expose separate trade-offs

  • At approximately 80 active features per token, BatchTopK has FVU 0.0648 and ΔNLL = 0.1945 nats, versus ReLU’s 0.1078 and 0.5796 nats at L0 = 77.71.TopK at 320 active features reaches FVU 0.0182 and an NLL increase of 0.0341 nats.Source: §6.1 · p. 18, Figure 15 · p. 19
  • Near 20 active features, Matryoshka’s one-feature AUROC is 0.9065 versus TopK’s 0.8412; with five features, PCA reaches 0.9481 versus the best SAE probe’s 0.9380.Matryoshka’s FVU is higher than TopK’s in the one-feature comparison, 0.1166 versus 0.1081.Source: Figure 15 · p. 19, §6.1 · p. 19

6.2 Visual features expose spatial structure across models and methods

  • For matched CLIP-B/32 observations, TopK uses 64 active features per position versus ReLU’s 1,413, but has higher FVU and lower Point@1.TopK has FVU 0.2163 versus 0.0152 and Point@1 44.7% versus 54.5%; its active-map mass inside boxes is higher, while coverage is lower.Source: Table 4 · p. 20, §6.2 · pp. 20–21
  • For DINO-B/16, ReLU reaches 52.7% Point@1 versus TopK’s 35.5%, with similar activity but higher response coverage.The pointing rates conditional on response are nearly equal, while coverage is 97.9% for ReLU and 65.4% for TopK.Source: Table 4 · p. 20, §6.2 · pp. 20–21
  • CLIP-L/14 Matryoshka uses 23.09 active features with 83.1% coverage and 62.5% Point@1; across configurations, reconstruction and localization need not agree.DINOv2 BatchTopK has FVU 6.6730 but 66.8% Point@1 versus 23.8% shuffled, while RA-SAE has 70.6% active-map mass but 36.4% coverage and 29.0% overall Point@1.Source: Table 4 · p. 20, §6.2 · pp. 20–21
  • All twelve visual SAE configurations exceed spatially shuffled controls, but object-associated response coverage ranges from 36.4% to 100%.The comparisons include six visual backbones and four photographs, with observations before language decoding.Source: §6.2 · p. 21, Figure 17 · p. 21

6.3 Decoder features probe question-conditioned answer evidence

  • The same image yields different decoder-feature spatial patterns across configurations and answer objectives; the maps locate answer-sensitive directions, not queried-object segments.In Qwen and LLaVA, image-prefix states cannot read the later question; question conditioning enters through the downstream answer objective.Source: §6.3 · p. 22

Data resources

The data-resources section lists source works spanning language modeling, visual question answering, spatial reasoning, and visual explanation.

Data resources

  • Other entries cover VisOnlyQA, visual spatial reasoning, and visual shortcomings of multimodal language models.Source: Data resources · p. 24
  • The bibliography also includes works on language modeling, text classification, and vision-related research.Source: Data resources · p. 24
  • The listed resources include The Pile and works on visual explanations and visual question answering, including GQA.Source: Data resources · p. 24

A Experimental details

The appendix records checkpoint and input contracts, describes hybrid-attention boundaries in Qwen3.5, and specifies lens fitting and readout scope.

A.1 Checkpoints and input contracts

  • Shared VLM runs pin software, checkpoint revisions, and model inputs; GQA, MMVP, VSR, and VisOnlyQA use specified source cases and prompt conventions.Checkpoint identities and revisions are listed in Table 5.Source: §A.1 · p. 25, Table 5 · p. 25

A.2 Hybrid attention in Qwen3.5

  • Qwen3.5’s 24 DeltaNet blocks prevent standard attention rollout from representing full-stack propagation by simply skipping those blocks.Extending rollout requires defining and validating propagation through DeltaNet blocks; the current adapter also lacks the controlled output-freezing contract required for IOI-style path patching.Source: §A.2 · p. 25

A.3 Lens fitting and readout scope

  • VLM Tuned Lenses are fitted on COCO caption positions with frozen models, and Table 6 reports held-out calibration fidelity rather than answer accuracy.Shared report cases are excluded from fitting; the table evaluates initial and selected translators on the same held-out positions.Source: §A.3 · pp. 25–26, Table 6 · p. 26
  • The GPT-2 lens artifacts cover text studies, not a five-VLM calibrated Jacobian/Attention Lens comparison.Jacobian Lens estimates causal transport into the final residual basis; its scores are not calibrated next-token probabilities.Source: §A.3 · p. 26
  • EmbedLens and Patchscope are readout studies, not completed pruning or cross-model transfer trials.EmbedLens reports lexical neighbors without the full paper’s category thresholds, while Patchscope uses specified source layers and a zero-vector control.Source: §A.3 · p. 26

A.4 Attention implementations and spatial display

This appendix section specifies which attention methods apply to hybrid architectures and how spatial attribution maps are constructed and displayed.

A.4 Attention implementations and spatial display

  • Qwen3.5 has 24 DeltaNet layers between eight full-attention layers, so skipping those layers is not full-stack rollout.Qwen3-VL propagation also omits the vision encoder, projector, and DeepStack injections as separate graph edges.Source: §A.4 · p. 26
  • TAM and Beyond use 20 right-Riemann samples, while ATTATTR retains the official-code left-sample convention with 20 samples.The zero-pixel endpoint is excluded for SmolVLM because it denotes a padding-image sentinel.Source: §A.4 · p. 26
  • Spatial maps are normalized on their native grids before cropping, and display transformations do not change archived scores.The display uses an opacity overlay and a power transform; zero maps are labeled explicitly.Source: §A.4 · p. 26
  • The native ViT control uses converted timm/JAX weights; its illustrative images are not a benchmark subset.Deletion sets selected processed patches to zero, and ten seeded random orders define the displayed envelope.Source: §A.4 · p. 26

A.5 Intervention contracts and validation

This appendix specifies intervention and attribution contracts, documents matched validation cases, and checks that pinned dictionary conversions reproduce their source inference functions. Numerical parity verifies execution, but not the uniqueness of an interpretation.

A.5 Intervention contracts and validation

  • The matched intervention setup uses a gray reference image, sampled layers for patching, all 32 layers for attribution, and separate visual and answer-prediction intervention sets.Four matched runs require 658 forwards, 132 activation backwards, and 18.17 GiB peak allocation.Source: §A.5 · p. 27
  • The LLaVA knockout masks a clipped nine-layer head window and measures token-ID 612 probability; its second case starts at P(Black) = 0.9982 and falls by at most 9.43%.The metric is not full-word Yellow probability; shirt/other masks contain 20/556 keys, and already-forbidden causal connections serve as a null control.Source: §A.5 · p. 27
  • Path patching fixes other attention-head outputs to a base cache while recomputing MLP and LayerNorm; complete sweeps cover declared sender heads, but Qwen3.5 DeltaNet freezing is unavailable.Selected paths are user-configured, not automatically assigned semantic component names.Source: §A.5 · p. 27
  • The GPT-2 EAP experiment compares clean-gradient EAP with five-sample EAP-IG on an editable pre-LayerNorm message graph, and every retained-edge curve point is an actual forward pass.Its BOS-prepended IOI margins are 6.1643/3.2137 and should not be numerically pooled with the separately prepared text-lens readout.Source: §A.5 · p. 27
  • The VSV steering comparison replays greedy full prefixes at seven retained scales and archives outputs and controls; large-scale repetition is specific to this configuration, not complete VISTA.The study omits the released implementation’s PCA, normalized MLP-output rule, and SLA.Source: §A.5 · p. 27
  • Same-input replacement and baseline restoration are exact; final-site BF16 readouts differ by at most 0.125, and FP32 graph parity is within 1.26 × 10−4.These checks verify execution, not that an interpretation is unique.Source: §A.5 · p. 27

A.6.1 Pinned dictionaries and inference functions

  • The 18 Pythia and three CLIP dictionary conversions agree with independent source equations, with maximum relative L2 errors of 2.45 × 10−7 for encoding and 2.29 × 10−7 for reconstruction.The checks establish numerical agreement for declared functions, not semantic interpretation; the language dictionaries use trainer indices 0, 2, and 4, which are sparsity settings rather than independent training seeds.Source: §A.6.1 · p. 28

A.6.2 Cross-backbone patch dictionaries

The cross-backbone patch evaluation binds published sparse dictionaries to their native observations and preserves each source’s inference and preprocessing choices. Validation checks support the declared conversions, while the scope varies across dictionaries and backbones.

A.6.2 Cross-backbone patch dictionaries

  • The four ViT-Prisma spatial dictionaries were trained with different token scopes and sparsity settings, but evaluation uses patches only without retuning preprocessing or sparsity budgets on COCO.CLIP ReLU, DINO ReLU, and DINO TopK trained on CLS plus patches; CLIP TopK trained on patches only.Source: §A.6.2 · p. 28, Table 7 · p. 29
  • Prisma’s three normalized dictionaries normalize each token before encoding and rescale decoded outputs by the token standard deviation; the CLIP ReLU checkpoint instead uses unnormalized activations.Feature order, biases, and decoder scales are preserved.Source: §A.6.2 · p. 29
  • The Gated CLIP-B/32 SAE observes block-9 MLP outputs before residual addition, and its converted patch features match the author’s equations exactly.Independent MLP observations agree at relative L2 error 2.18×10−6, while reconstruction relative L2 error is 7.76 × 10−8.Source: §A.6.2 · pp. 29–30
  • ViSAEBench dictionaries use fixed saved thresholds after rectification, so their training target k = 64 does not force 64 active features on every COCO patch.The three thresholds are 0.54186058 for CLIP, 0.63923049 for DINOv2, and 0.66128719 for SigLIP.Source: §A.6.2 · p. 30
  • Qwen2.5 retains the 100 largest encoder values without rectification, whereas Qwen3 uses a rectified, thresholded level-1 SAE from a jointly trained hierarchy.Qwen2.5 selection is by value rather than absolute magnitude; Qwen3 evaluation does not treat level 2 as a second independent patch method or reproduce the full hierarchy objective.Source: §A.6.2 · p. 30
  • The Qwen3 evaluation observes block 23 before spatial merging, inverts grouped patches to raster order, and does not instantiate the language model.The frozen level-1 inference uses weights trained jointly within the SAE++ hierarchy.Source: §A.6.2 · pp. 30–31
  • RA-SAE preserves its nonlinear encoder and relaxed archetypal decoder, with exact native/source feature agreement and at most 8.94×10−8 row-wise dictionary difference.The spatial evaluation selects patches 5–260 after final LayerNorm, excluding CLS and register activations.Source: §A.6.2 · p. 31

A.6.3 Text selection, metrics and probe controls

The text evaluations define held-out Pile reconstruction and AG News concept-probe comparisons with controls fitted on training data. The procedures distinguish native-zero-residual loss from SAE-feature ablation and keep test observations out of probe selection and scaling.

A.6.2 Text selection, metrics and probe controls

  • Pile reconstruction uses 128 calibration contexts and 512 test contexts from held-out files, but historical overlap with training documents is unknown, so it is not a verified unseen-training-data test.Each context uses the first 1,024 tokens of one document, without concatenating documents.Source: §A.6.3 · p. 31
  • The reconstruction evaluation scores 65,472 shifted next-token predictions, using identical native positions for test reconstructions and a globally summed fixed-reference denominator.The original NLL is 2.3461 nats per prediction and the native-zero-residual NLL is 11.9161; the zero control removes selected native residual vectors, not merely SAE features.Source: §A.6.3 · p. 31
  • AG News probes rank features using training labels, fit scaling on training documents, and select regularization on development data before final metrics on 1,000 untouched test documents.The controls compare 768 raw neurons, 128 training-fitted principal components, and a seed-17 Gaussian projection into 4,096 directions, with matched per-class feature budgets.Source: §A.6.3 · pp. 31–32
  • The language dictionary table reports published sparsity penalty, measured Pile L0 and FVU, ΔNLL, and AG News AUROC at one or five selected features per class.The matched probe-control table reports AUROC at one, two, or five features and F1 at five.Source: Table 8 · p. 32, Table 9 · p. 32

A.6.4 Image geometry and feature selection

The image evaluation fixes preprocessing, native patch geometry, and calibration-based feature selection before validation localization. Its comparisons use paired image-level resampling and explicitly limit what can be inferred from pooled categories and atlas examples.

A.6.4 Image geometry and feature selection

  • The CLIP ViT-L/14-336 cohort uses 576 spatial patches per image and separately evaluates CLS reconstruction; annotation boxes follow the same resize and crop as the pixels.The cohort excludes crowd boxes and boxes entirely outside the crop, and uses no square padding.Source: §A.6.4 · p. 33
  • Cross-backbone comparisons preserve each processor’s crop, normalization, and native grid, so differing source crops can change visible regions and eligible pair membership.Qwen2.5 is observed at decoder block 34 after spatial merging, while Qwen3 is observed at vision block 23 before merging and is restored to raster order.Source: §A.6.4 · p. 33
  • Feature selection maximizes calibration positive-evidence contrast subject to category-image activity and visibility requirements; selected features are frozen before validation.Each dictionary selects its own feature, and categories failing the same eligibility conditions are recorded as ineligible rather than filled using test data.Source: §A.6.4 · p. 33
  • Localization pools image–category pairs, so common categories contribute more pairs; shuffled controls preserve feature values and activity while removing measured spatial arrangement.Targets are category-box unions, not pixel-level segmentation masks, and activation mass is averaged only over positive-active pairs with coverage reported separately.Source: §A.6.4 · p. 34
  • The compact atlas spans different backbones and observation stages, so its encoder-only examples do not isolate an SAE algorithm while holding weights and hooks fixed.RA-SAE’s person and dog maps retain measured zero response without per-image text labels; the selected car feature remains active.Source: §A.6.4 · p. 34
  • Real-minus-shuffled point@1 differs by 44.0 percentage points (interval 41.5–46.4), and active-only mass differs by 42.1 points (interval 40.5–43.6).Intervals condition on fixed dictionaries, feature selection, splits, and the recorded shuffle; they do not estimate SAE training-seed variability.Source: §A.6.4 · p. 35

A.6.5 Additional CLIP ViT-L/14-336 CLS dictionaries

The additional CLIP ViT-L/14-336 CLS dictionaries retain high final projected image-embedding similarity, while the section also documents reproducible, fixed-checkpoint evaluation and rendering workflows.

A.6.5 Additional CLIP ViT-L/14-336 CLS dictionaries

  • Final projected image-embedding cosine similarity is 0.9709 for BatchTopK and 0.9696 for Matryoshka, measuring representation retention rather than classification accuracy.Table 12 covers 1,000 held-out COCO images; its FVU reference mean comes from 500 calibration images.Source: Table 12 · p. 35, §A.6.5 · p. 35
  • Across 1,534 held-out image–category pairs, patch-feature coverage is 83.1%, point@1 is 62.5% versus 18.5% shuffled, and positive-activity box mass is 64.9% versus 22.8%.The dictionary supplies the CLIP-L/14-336 maps in Figure 17, using the same independently selected feature IDs as the quantitative evaluation.Source: §A.6.5 · pp. 35–36

A.6.6 Reproducibility

  • Saved records preserve source revisions, checksums, samples, selections, configurations, and results; reported checkpoints are fixed, so the comparisons do not estimate SAE training-seed variability.Rendering reads measured maps and predeclared cases; tutorial-subset scores are separate from report measurements.Source: §A.6.6 · pp. 36–37

A.7 Pretrained multimodal decoder SAE protocol

The protocol evaluates pretrained decoder SAEs through fixed answer contrasts, feature attributions, and interventions. Results reveal limits in reconstruction fidelity and differences between attribution, deletion effects, and free generation.

A.7 Pretrained multimodal decoder SAE protocol

  • The protocol applies five published decoder dictionaries to eight fixed questions; model-specific observation depths vary, and PaliGemma’s text-position dictionary is transferred to visual positions.The PaliGemma cases are not an independent held-out VQAv2 evaluation.Source: §A.7 · pp. 37–38
  • Scores contrast mean log probabilities for fixed reference and foil continuations; up to eight positive-contribution SAE features are selected per case, not by object boxes.Canonical and foil continuations share a scoring prefix but are differentiated separately.Source: §A.7 · p. 39
  • Deletion removes selected features at valid spatial image-token positions while retaining reconstruction error; count-matched random groups do not match perturbation energy.For large car and dog boxes, translated pixel controls overlap the target and cannot establish object-exclusive causal dependence.Source: §A.7 · p. 40
  • Jacket deletion shifts contrast from 0.649 to −0.252 in LLaVA-Mistral and 0.083 to −0.703 in PaliGemma, yet both generate “brown”; Qwen2.5 generates “black” despite positive blue-versus-red contrast.These cases distinguish sensitivity of a fixed answer pair from unrestricted generation.Source: §A.7 · p. 40
  • Qwen3.5 FVU is 0.407–0.489 versus PaliGemma’s 1.814–3.040, above the within-image variance baseline; Qwen2.5 tag attribution 0.261 differs from deletion 0.572.PaliGemma’s nonzero attribution and deletion effects therefore do not establish a faithful sparse decomposition of image states.Source: §A.7 · p. 41

A.8 Replaying figures and measurements

  • The archived lens, attention, and causal report replays bundled figures without weights or a GPU; live SAE notebooks acquire inputs, load checkpoints, and compute new measurements.Portable manifests record source hashes and exported-result equality.Source: §A.8 · p. 44

B.1 A second vision-classifier comparison

  • Figure 19 repeats elephant–zebra native ViT attribution with the cat–dog deletion protocol, so map differences can be checked against removing selected patches.The figure uses identical preprocessing and deletion budgets.Source: Figure 19 · p. 45, §B.1 · p. 45

B.2 Residual replacement on the black-shirt input

  • On the black-shirt case, LLaVA’s measured effects are smaller than on Yellow, while SmolVLM remains sensitive to the gray-image reference; exact text-position replacements locate question-word and prediction-position effects.Aligned first-order maps provide an approximation check, not a separate attribution target.Source: §B.2 · p. 45, Figure 20 · p. 46, Figure 21 · p. 47

Related papers

Loading 2610.33072v1…