GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

• VLMs often under-use visual information, raising whether their representations capture visual evidence beyond what text already provides.The introduction illustrates this with textual praise that conflicts with a disordered room image, where recognizing the mismatch supports a sarcastic interpretation.Source: §1 · pp. 1–2
• PID distinguishes redundant information shared by vision and language, modality-unique information, and synergy available only from their combination.The contrasting tidy-room text and disorderly image illustrate why their relationship can matter beyond either modality alone.Source: §1 · pp. 1–2
• Prior PID estimators quantify components without locating their subspaces, while training-free visual-grounding methods rely on heuristic signals.The introduction contrasts this with approaches requiring costly additional training.Source: §1 · p. 2
• GeoPID measures redundancy through principal angles, modality-unique directions through orthogonality, and synergy in the complement of the combined subspaces.It connects this geometric analysis to a concrete vision-unique intervention target.Source: §1 · pp. 2–3
• 7.63% average relative accuracy gain; across 14 benchmarks and 22 VLMs, correct answers on vision-requiring questions show stronger vision-unique components.The analysis also finds that shared geometry develops across early and intermediate layers, while later-layer modality-specific profiles differ.Source: §1 · p. 3
• Shared geometric structure alone does not establish effective visual use, and information components vary with the visual and cross-modal demands of each question.Visual-grounding questions exhibit the highest vision-unique component, while questions requiring visual grounding and image-text combination exhibit higher synergy.Source: §1 · p. 3

2.1 CHARACTERIZING MULTIMODAL INFORMATION

• GeoPID defines shared, vision-unique, text-unique, and synergy-evaluation geometries from the relationships among vision and text representation subspaces.The vision-unique geometry is orthogonal to the estimated text subspace; the synergy-evaluation geometry lies outside the combined principal span.Source: §2.1 · p. 4
• For diagnosis, GeoPID measures target-related information in projected representations with matrix-based Rényi-2 mutual information and defines redundancy and modality-specific uniqueness from those measurements.Synergy is evaluated in the combined-span complement by contrasting correctly paired inputs with image-mismatched conditions.Source: §2.1 · pp. 4–6
• It estimates modality subspaces using batch-wise PCA, with sample contributions normalized by token count, and identifies shared directions using principal-angle alignment above τ = 0.7.Vision-unique and text-unique geometries retain principal directions orthogonal to the other modality's estimated subspace.Source: §2.1 · pp. 4–5

2.2 ANALYZING VISUAL INFORMATION USE

• Across models, shared geometry generally grows in early layers, while later layers show contrasting modality-specific profiles; stronger text-unique profiles also tend to coincide with lower vision-unique information and synergy.The profiles differ particularly around normalized depths 0.8–0.95.Source: §2.2 · p. 6, Figure 2 · p. 6
• Visual-observation questions have the highest normalized vision-unique information (UV = 1.34), while V and J have higher synergy (S = 3.56 and 3.55) than T (2.24).The analysis covers 22 VLMs and 11 benchmarks with sufficient examples across at least two question types.Source: §2.2 · p. 7, Table 1 · p. 7
• Vision-unique components are stronger in correct answers to visually demanding V and J questions, while the pattern reverses for text-sufficient T and T− questions.This contrast motivates selective rather than indiscriminate visual amplification.Source: §2.2 · p. 7, Figure 3 · p. 7

2.3 TARGETED INTERVENTION IN VISUAL REPRESENTATIONS

• For each model–benchmark pair, a disjoint 100-question calibration subset selects intervention depth and amplification strength; the selected settings are fixed for evaluation without inference-time label-dependent gating.The vision-unique basis is estimated without ground-truth labels, while labels are used to select the calibration settings.Source: §2.3 · p. 7
• The intervention scales visual representations along the vision-unique subspace by 1 + α while preserving the orthogonal complement, after which remaining decoder layers process the modified representations.The intervention is applied to all vision tokens simultaneously.Source: §2.3 · pp. 7–8

3.1 CAN GEOPID IMPROVE PREDICTIONS?

• 7.63% average relative improvement followed from selecting intervention depth and strength on 100 calibration questions, then fixing them for evaluation across 308 model–dataset pairs.The calibration subset was disjoint from the final evaluation set, with no further tuning or label-dependent selection on that set.Source: §3.1 · p. 8, §3 · p. 8
• The largest improvement occurs on joint-evidence questions, while positive gains also occur on text-sufficient and text-prior questions.Joint-evidence questions require textual and visual information together.Source: §3.1 · p. 8

3.2 CAN GEOMETRY GUIDE INTERVENTION SETTINGS?

• Intervention gains vary across models and benchmarks, motivating the use of baseline geometric structure to guide intervention settings.The reported association between layer-wise geometric trajectories and intervention gain reaches Pearson r = 0.80.Source: §3.1 · p. 9, §3.2 · p. 9
• Strong late-layer text dominance favors early interventions with high amplification when shared geometry is strong or weak, whereas weak text dominance degrades under strong early intervention.Figure 4 contrasts strong text-unique dominance with high or low shared geometry against weak text-unique dominance.Source: Figure 4 · p. 9
• A fixed geometry-based rule yields a 4.62% relative accuracy gain across 308 model–benchmark pairs; GEOPID also achieves the highest overall accuracy and absolute gain with minimal damage among the compared baselines.The comparison includes training-free baselines and a training-based PID baseline; Table 5 reports accuracy gains for the fixed-rule intervention across 22 models and 14 datasets.Source: §3.2 · p. 9, Table 5 · p. 9

4 CONCLUSION

• Appropriately configuring subspace manipulation using early shared geometry and text dominance leads to consistent gains across diverse benchmarks without costly retraining.The conclusion describes GEOPID as a tool for analyzing multimodal representation spaces and addressing reasoning bottlenecks.Source: §4 · p. 10
• The source code is not publicly released, although the manuscript documents configurations, implementations, formulations, and hyperparameters for reconstruction.The evaluations use publicly available benchmarks and open-weight models.Source: REPRODUCIBILITY STATEMENT · p. 10

B.1 ANALYZING MULTIMODAL INTERACTION WITH PID

• Prior PID approaches use separate estimators and report improvements through additional training; GEOPID reads information from representation matrices and intervenes without training.The cited prior work includes decision-level and layer-wise PID, diagnosis, distillation, and fine-tuning approaches.Source: §B.1 · p. 16

B.2 REPRESENTATION GEOMETRY AND INFORMATION MEASUREMENT

• Overall representation rank does not distinguish directions shared across modalities from modality-specific directions, so GEOPID uses their geometric relationships to identify vision-unique intervention targets.The paper relates rank to representational diversity and measures answer-related information associated with the identified subspaces.Source: §B.2 · p. 16

B.3 TRAINING-FREE INTERVENTIONS FOR VISUAL INFORMATION USE

• Unlike training-free methods that adjust logits, activations, tokens, or information flow without explicit PID targets, GEOPID selectively amplifies visual components along the vision-unique subspace.The amplification occurs during inference.Source: §B.3 · pp. 16–17

C.1 MODELS

• The model suite spans general-purpose, reasoning-oriented, and explicitly grounded VLMs, with nominal sizes from approximately 2B to 8B parameters and weights fixed during experiments.The evaluated list includes ViGoRL variants developed for visually grounded reasoning.Source: §C.1 · pp. 17–18

C.2 BENCHMARKS

• The benchmark suite spans multimodal understanding, visual perception and hallucination, scientific, knowledge-based and medical reasoning, and generated-image comparison, using common answer-choice scoring.Multiple-choice scoring selects the answer choice with the highest output logit, with identical prompts before and after intervention.Source: §C.2 · p. 18, Table 7 · p. 18

D.1 DEFINITION

• Each question receives one evidence category: text-sufficient (T), text-prior (T−), visual-observation (V), or joint vision-and-text (J).The categories describe evidence requirements inferred from the question and its accompanying information.Source: D QUESTION CATEGORY ANNOTATION · p. 19, §D.1 · p. 19
• T permits answering from text and knowledge; T− suggests but does not establish an answer, V requires image observation, and J requires matching textual content to the image.For T−, the particular image could change the answer; for J, neither modality alone determines it.Source: §D.1 · p. 19

D.2 ANNOTATION PROTOCOL

• The annotated intervention pool contains 11,372 items across 14 benchmarks, and Claude Opus 5 assigned labels without seeing images, model predictions, or accuracy measurements.Each benchmark contributed up to 700 valid items; output files were checked for complete coverage and one label per item.Source: §D.2 · p. 20

D.3 ANNOTATION VALIDATION

• 87% of the 100 shared AI2D items received matching annotations across runs, with Cohen’s κ = 0.80.Disagreements occurred only at the T versus T− textual-sufficiency boundary.Source: §D.3 · p. 20
• 89.1% accuracy on T versus 29.2% on V with blank images supports the distinction between textual sufficiency and visual dependence.The V result is close to the four-option chance level of 25%.Source: §D.3 · p. 20
• 3.9 percentage points on T versus 35.4 on V and 28.9 on J: blanking images reduces accuracy much more for visually dependent categories.The table aggregates accuracy reduction over model-item pairs; larger reductions indicate greater sensitivity to image removal.Source: §D.3 · p. 20, Table 8 · p. 20

D.4 ANNOTATION RESULTS

• POPE and CV-Bench are entirely V, while GenAI-Bench and Rapidata-4o are entirely J; Reefknot is predominantly T and T−.Because pooled category comparisons can reflect benchmark differences, the analysis uses within-model-benchmark contrasts when separating evidence requirements from benchmark effects.Source: §D.4 · p. 21, Table 9 · p. 21

E.1 EQ. 1, 5, 6: DETERMINING rb, ru, rs IN BATCH-WISE PCA

• ρ = 1.000 across tested ru values preserves model ordering, while intervention rank 8 improves gain over rank 32 by a median 0.67 percentage points.The rank budgets are rb = 64, ru = 32, and rs = 128; the rank-8 intervention result suggests separately calibrating intervention rank may yield further gains.Source: §E.1 · p. 21, Table 10 · p. 22, Table 11 · p. 22

E.2 DETAILS OF SECTION 2.1

• Across decoder depth, the visual and textual representation distributions vary in orientation and spread, with Qwen2.5-VL-7B retaining a broader late-layer visual cloud than ViGoRL-3B.Figure 5 shows two-dimensional projections for both models at selected layers.Source: §E.2 · pp. 22–23, Figure 5 · p. 22
• The retained vision-unique and text-unique bases are orthogonal to the opposite modality’s estimated subspace.The appendix establishes this property for each unique basis by applying the same argument with vision and text exchanged.Source: §E.2 · p. 23
• The complementary representations can be computed without explicitly constructing the full complementary basis.The two implementations preserve identical pairwise inner products and norms.Source: §E.2 · p. 23

E.3 IMPLEMENTATION OF INFORMATION MEASUREMENT (EQ. 8)

• The implementation computes matrix-based Rényi-2 mutual information from squared cosine-kernel entries, including sums over sample pairs with matching labels, without eigendecomposition.The passage states this is equivalent to the trace-normalized Rényi-2 formulation.Source: §E.3 · p. 23

E.4 VALIDATING THE CORRESPONDENCE DEPENDENCE (EQ. 10) OF S

• 0.4150 answer information with the matched image falls to 0.0323 with a same-category substitute, close to 0.0274 with a different-category substitute.The other two reported examples show the same pattern, supporting sensitivity to specific image-text correspondence rather than category change alone.Source: §E.4 · p. 23, Table 12 · p. 24
• Image-pair preference tasks show a later rise in Ut than other task families, while MMBench and A-OKVQA have the largest reported S.Image-pair preference Ut remains near zero through much of the middle layers; Rapidata-4o and GenAI-Bench have smaller absolute S values of 0.0160 and 0.0088.Source: §E.4 · p. 24, Figure 6 · p. 24, Table 13 · p. 24
• Qwen2.5-VL-7B has higher S than Vision-R1-7B on every benchmark, while its I2 stays near 0.049–0.050 across ScienceQA, MME, and A-OKVQA despite differing S values.For Qwen2.5-VL-7B, S is 0.163, 0.201, and 0.289 on those three benchmarks, respectively.Source: §E.4 · pp. 24–25, Table 14 · p. 25

E.6 DETAILS OF TABLE 1: HOW S RELATES TO VISUAL INFORMATION USE

• Across pooled model–benchmark pairs, S correlates 0.73 with image-removal accuracy drop and 0.83 with base accuracy, slightly above I2(H; Y) at 0.70 and 0.82.After excluding one model family, the correlations remain positive at 0.68 across models and 0.67 across model–benchmark pairs.Source: §E.6 · p. 25, Table 15 · p. 25, Table 16 · p. 26

E.7 DETAILS OF FIGURE 3

• Correct V/J items have higher GUv than correct T items in all eight benchmarks with at least ten correct T items, and in nine of ten when sparse-T benchmarks are included.The reported difference remains significant under both inclusion criteria (p = 0.0002).Source: §E.7 · p. 26, Table 17 · p. 26

E.8 DETAILS OF EQ. 11

• At the intervention layer, the operator amplifies the vision-unique subspace while preserving text-basis coordinates, remains invertible for α ≥ 0, and removes no representation directions.The implementation applies the intervention without constructing a d × d projection matrix.Source: §E.8 · pp. 26–27

E.9 DETAILS OF SECTION 3.2

• Across 308 model-benchmark combinations, full layer-wise geometric trajectories predict intervention gains with correlation r = 0.80, versus 0.66 from static layer differences.The trajectory result uses gradient-boosted trees with 10-fold cross-validation.Source: §E.9 · p. 27, Table 18 · p. 27

F.1 QUALITATIVE RESULTS

• Models with weak vision-unique and dominant text-unique components shift toward lower answered-blind rates and higher overall accuracy after vision-unique amplification.Figure 7 shows this qualitative shift across 14 benchmarks.Source: §F.1 · p. 27, Figure 7 · p. 28
• In open-ended visual question answering, amplification of GUv overrides misleading text priors and produces more visually grounded responses.Examples include correcting “corn” to unripe bananas and identifying a German Shepherd rather than “chihuahua.”Source: §F.1 · pp. 27–28

F.2 ACCURACY GAIN BY QUESTION TYPES

• Joint-evidence questions show the largest gains; for V and J questions, gains with original images are 13.1 and 13.9 percentage points, versus 1.0 and −3.3 with mismatched images.Blank images yield little change for V and J questions, while T-question gains remain substantial with mismatched images.Source: §F.2 · pp. 28, 30, Table 19 · p. 28, Table 20 · p. 30

F.3 UNDERSTANDING THE ACCURACY GAINS

• The most frequently selected answer option accounts for 72% of predictions before intervention and 54% afterward, compared with a 39% ground-truth majority share.Among 678 initially incorrect model-item pairs with at least three options that changed answers, 74.0% of new predictions are correct, versus 34.1% expected under uniform switching.Source: §F.3 · p. 30, Table 21 · p. 30, Table 22 · p. 30

F.4 DECISION-STATE CHANGES AFTER INTERVENTION

• After intervention, image-text angular separation increases in all three displayed benchmark examples, including 10°→31° for ViGoRL-MCTS-3B on SEED-Bench.The displayed clouds move away from their original locations; the frame uses an image-conditioned mean displacement.Source: §F.4 · p. 30, Figure 9 · p. 31

G.1 ABLATION STUDIES ON ALTERNATIVE SUBSPACE TARGETS

• At normalized depth 0.6 and strength α = 1, GEOPID-selected vision-unique directions yield the highest average accuracy gain among the tested amplification targets.Source: §G.1 · p. 31, Table 23 · p. 31

G.2 ABLATION STUDIES ON DIFFERENT INTERVENTION SETTINGS

• The alternative policy assigns stronger amplification when at least 0.7 of probe answers remain unchanged after replacing the image with a blank image.Figure 10 presents accuracy trends across intervention depths and strengths for gated and non-gated cells.Source: §G.2 · p. 32, Table 24 · p. 32, Figure 10 · p. 32
• +4.62 percentage points is the reported mean gain for the Figure 4 intervention rule, versus +2.24 and +2.28 percentage points for the alternatives.The comparison covers 308 model-benchmark pairs.Source: §G.2 · p. 32, Table 25 · p. 32

G.3 ABLATION STUDIES ON DIFFERENT r, ℓ∗, AND α

• 29.67 percentage points is the largest tested mean gain, at (ru, ℓ⋆/L, α) = (16, 0.6, 4); with ru = 64 at the same depth and strength, the gain is 21.67 points.With ru = 16, increasing α from 1 to 4 improves the mean gain at all three tested depths.Source: §G.3 · p. 32, Table 26 · p. 33

H.1 ROBUSTNESS TO THE NUMBER OF IMAGE K IN EQ. 10

• Approximately 0.001 is the reported standard error of S due to mismatch randomness across 24 model-benchmark pairs when varying K ∈ {3, 5, 10}.Source: §H.1 · p. 33, Table 27 · p. 33

H.2 ROBUSTNESS TO KERNEL, R´ENYI ORDER, AND SAMPLE SIZE

• Ranking correlations with the cosine-kernel reference are 0.78 for RBF and 0.75 for standardized linear kernels, versus 0.996 and 0.997 for Rényi orders 1.5 and 4.Using 25%, 50%, or 75% of samples yields correlations of approximately 0.99, 0.97, and 1.00.Source: Table 28 · p. 33, §H.2 · p. 34