PaperlayerView exact revision on arXiv

Source-linked AI summary

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Abdullah All Tanvir, Xin Zhong

arXiv:2609.36458v1cs.LGcs.CLcs.CV

TL;DR

Semantic-preserving transformations can move representations, while equal Euclidean displacements can have unequal predictive effects, leaving open how to measure consequential local variation. Fisher-IRG contrasts local Fisher sensitivity across semantic-preserving and semantic-changing neighborhoods to recover invariant subspaces. Across language and vision models, it improves predictive selectivity and generally yields more reproducible subspaces than covariance-based geometry.

  • Problem

    Representation motion and predictive change can diverge: equal Euclidean perturbations may have different effects on a model’s output distribution.

    Source: §1 · p. 2

  • Method

    Fisher-IRG aggregates local Fisher information across semantic-preserving and semantic-changing neighborhoods and uses a contrastive generalized eigenproblem to identify invariant directions.

    Source: §3.2 · p. 4

  • Results

    Fisher-IRG consistently improves predictive selectivity across modalities and depth, with mean gains ranging from 0.170 for Gemma to 0.368 for Qwen2.5 across language models.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The findings support predictive sensitivity as a principled geometric basis for analyzing local semantic invariance in learned representations.

    Source: §1 · p. 2

    The local geometry assumes representations concentrate near a lower-dimensional manifold with smooth local structure.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Semantic-preserving transformations can induce substantial motion in learned representations, while small changes may strongly affect model predictions, raising a basic question: what local metric best captures semantically consequential variation? We propose Fisher-induced invariant representation geometry (Fisher-IRG), which measures local representation directions through their predictive sensitivity. Around each representation, we construct semantic-preserving and semantic-changing neighborhoods, aggregate their local Fisher information, and recover invariant directions through a contrastive generalized eigenvalue problem. Controlled displacement analyses first show that comparable Euclidean motion can have substantially different predictive consequences, supporting the need for a predictive geometry. Across language and vision models, Fisher-IRG yields stronger semantic-versus-nuisance predictive selectivity and generally more reproducible subspaces than covariance-based geometry, while recovering systematically distinct local directions. Representation interventions further localize semantic effects to the Fisher-derived subspace, and held-out separation and retrieval show that the recovered geometry generalizes beyond the discovery neighborhoods. These results support Fisher-IRG as a principled framework for characterizing local invariant representation geometry.

1 Introduction

Meaning-preserving variation need not leave every representation coordinate unchanged, and equal Euclidean motions can have unequal predictive effects. Fisher-IRG uses predictive sensitivity to identify local subspaces that distinguish semantic-changing from semantic-preserving variation.

1 Introduction

  • The central problem is measuring representation changes by their effects on model behavior, not requiring all meaning-preserving inputs to have unchanged coordinates.The question is studied in pretrained language and vision models.Source: §1 · p. 1
  • Equal Euclidean perturbation magnitudes can produce substantially different changes in a model’s output distribution.Covariance captures sampled directional motion but does not directly measure its predictive consequences.Source: §1 · p. 2
  • Prior work motivates contrasting semantic-preserving and semantic-changing neighborhoods while measuring their predictive effects with Fisher geometry.The cited foundations include covariance-based local invariant analysis [6] and pullback Fisher geometry for activation steering [18].Source: §1 · p. 2
  • Fisher-IRG contrasts semantic-preserving and semantic-changing neighborhoods using Fisher information as a measure of local predictive sensitivity.It recovers directions more predictively sensitive to semantic change than to meaning-preserving variation.Source: §1 · p. 2
  • Fisher-IRG yields stronger semantic-versus-nuisance predictive selectivity and generally more reproducible subspaces than covariance-based geometry.Interventions selectively affect semantic behavior, and held-out separation and retrieval assess generalization beyond discovery neighborhoods.Source: §1 · p. 2

2 Related Work

Related work establishes foundations in representation invariance, contrastive subspace analysis, and Fisher geometry. This paper builds on those foundations by using predictive sensitivity, rather than displacement statistics, to discover local invariant subspaces.

Representation Geometry and Contrastive Subspace Analysis

  • Fisher-based construction measures local predictive sensitivity, extending covariance-based invariant analysis beyond displacement statistics.Covariance-based local invariant-subspace discovery contrasts semantic-changing and semantic-preserving displacement statistics [6].Source: §2 · p. 2

Semantic Invariance in Language and Vision Representations

  • Prior work links learned representations to linguistic and visual invariance, nuisance minimality, representation comparison, and contrastive spectral analysis.The cited examples include [17], [11], [2], [10], and.Source: §2 · p. 2
  • Unlike contrastive representation-training objectives such as SimCLR [5] and SimCSE [9], this work analyzes pretrained models without introducing a training objective.It identifies local invariant subspaces through predictive effects of semantic-preserving and semantic-changing variation.Source: §2 · p. 2

Fisher Information and Predictive Representation Geometry

  • Fisher information connects local statistical-model changes to second-order KL divergence, while latent-geometry studies construct non-Euclidean metrics for generative models [3] [4].Related language-model work pulls output Fisher geometry back to Transformer activations for steering [18].Source: §2 · p. 3
  • The paper uses Fisher geometry for contrastive subspace discovery and stability evaluation, rather than parameter optimization, latent interpolation, or targeted activation steering.The related methods include natural-gradient foundations, latent metrics [3] [4], and activation steering [18].Source: §2 · p. 3

3 Fisher-Induced Local Invariant Representation Geometry

Fisher-IRG equips local representation directions with a predictive metric, then contrasts Fisher sensitivity across semantic-preserving and semantic-changing neighborhoods. A regularized generalized eigenproblem extracts directions selectively sensitive to semantic change, with first-order score gradients supporting computation.

3.1 Local Representation Geometry and Predictive Sensitivity

  • Representations are assumed to concentrate near a lower-dimensional manifold with smooth local structure, providing a local model without requiring global parameterization.The assumption is made for representations of a frozen model.Source: §3.1 · p. 3
  • A predictive metric reweights tangent directions by their effects on model behavior, producing anisotropic local geometry distinct from Euclidean geometry.Figure 2 depicts the representation manifold, tangent motion, and predictive metric around an anchor.Source: Figure 2 · p. 3
  • Fisher-IRG uses a model-induced quadratic form to measure local directions by their effects on predictive behavior.Unlike covariance geometry, it measures predictive sensitivity rather than only where sampled representations move.Source: §3.1 · pp. 3–4

3.2 Data-Driven Fisher Subspace Discovery

  • Semantic-preserving variants modify nuisance factors, while semantic-changing variants alter content; discovery neighborhoods do not assume semantic equivalence means Euclidean proximity.Locality refers to differential analysis around each sampled representation, not treating every finite anchor-to-variant displacement as infinitesimal.Source: §3.2 · p. 4
  • Fisher information is the local curvature of predictive distribution change, and its quadratic form approximates predictive KL divergence to second order.The derivatives are with respect to the representation rather than model parameters.Source: §3.2 · p. 4
  • A regularized generalized Rayleigh quotient contrasts semantic-changing and semantic-preserving Fisher operators to select relatively predictive-sensitive directions.The ridge term keeps the denominator positive definite when estimated Fisher matrices are rank deficient.Source: §3.2 · p. 4
  • The leading generalized eigenvectors define a basis-independent projection whose semantic interpretation is tested with held-out representation and intervention experiments.The subspace criterion alone does not establish stability under finite semantic-preserving transformations.Source: §3.2 · p. 4
  • First-order predictive score gradients estimate Fisher operators, and truncating output outcomes yields an approximation rather than the exact full Fisher.Approximation quality depends on omitted probability mass and associated score magnitudes; the construction also requires a specified predictive readout and intervention.Source: §3.2 · pp. 4–5

4 Experiments

Across controlled language and vision neighborhoods, Fisher-IRG is more predictively selective than Cov-IRG, generally more reproducible, and recovers systematically different local subspaces. The experiments also show that Euclidean displacement alone does not determine predictive consequence.

4.1 Datasets and Model Coverage

  • The experiments use controlled SP/SC neighborhoods around language prompts and Visual Genome images, evaluating seven language models and three vision encoders.The visual SC variants span easy, medium, and hard semantic changes; the model suite spans different architectures and learning objectives.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 Displacement-Consequence Separability

  • Distance-matched SP/SC perturbations induce sharply different predictive changes in every evaluated model, with mismatch cases defined by ρKL ≥2; this local effect spans all evaluated layers for several models, but not universally.Source: §4.2 · p. 6, Table 1

4.3 Predictive Selectivity and Subspace Stability

  • Fisher-IRG improves predictive selectivity over Cov-IRG at every reported depth across all seven language and three vision models.The strongest advantage occurs at different depths in different architectures.Source: Table 2 · p. 6, §4.3 · p. 6
  • Fisher-IRG generally yields lower Grassmann distances than Cov-IRG, indicating more reproducible subspaces across models, layers, and dimensions.The authors qualify this as a cross-model tendency, not uniform dominance in every finite-sample setting.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 Fisher and Covariance Induce Distinct Local Geometries

  • For the evaluated language models, the Fisher- and Cov-IRG cross-method projection distance d(F,C) ranges from 0.993 to 0.998, with reported values for Mistral, LLaMA, Gemma, Qwen, DeepSeek, Phi, and Falcon.Source: Table 4 · p. 7, §4.4 · p. 7

F UC, as a complementary measure

Interventions show that semantic effects are concentrated in the Fisher-IRG subspace, while held-out separation and retrieval tests show that its geometry generalizes beyond discovery neighborhoods. Fisher-IRG also recovers subspaces systematically distinct from Cov-IRG.

F UC, as a complementary measure

  • Cross-method distances exceed within-method distances for every evaluated semantic group, with near-zero mean principal cosines indicating little directional overlap.The pattern holds for language and vision models.Source: F UC, as a complementary measure · p. 7, §4.5 · p. 7

4.5 Causal Validation by Representation Intervention

  • Both causal-localization criteria are positive across all 21 language model–group and nine vision group–difficulty combinations.Within Fisher-IRG, SC replacement changes predictions substantially more than SP replacement, and the same SC donor has a larger effect through Fisher-IRG than through its complement.Source: Table 5 · p. 8, §4.5 · p. 8

4.6 Functional Validation of Fisher-IRG

  • Fisher-IRG achieves the largest held-out semantic-separation margin in all 21 language model–layer comparisons and all nine vision layer–difficulty settings.At the earliest evaluated vision layer, Fisher-IRG yields positive separation while the original and Cov-IRG representations have negative margins at all three SC difficulty levels.Source: Table 6 · p. 8, §4.6 · p. 8
  • Fisher-IRG improves original-representation pairwise accuracy from 0.949 to 0.959, MRR from 0.921 to 0.931, and Hit@1 from 0.873 to 0.885.It achieves the highest MRR for all seven language models and the highest pairwise accuracy and Hit@1 for six of seven.Source: Table 7 · p. 8, §4.6 · pp. 8–9

A Dataset Construction and Model Details

The study builds anchor-centered semantic-preserving and semantic-changing neighborhoods for language and vision, then evaluates Fisher-IRG and Cov-IRG under matched conditions. Disjoint discovery and evaluation samples support held-out testing.

A Dataset Construction and Model Details

B Additional Results for Displacement–Consequence Separability

Across language and vision models, distance-matched perturbations can produce substantially different predictive changes, showing that Euclidean displacement alone does not determine behavioral significance. The mismatch recurs across many language-model layers and every evaluated ViT-B/16 layer, while its prevalence and magnitude vary by layer.

B Additional Results for Displacement–Consequence Separability

  • Groups are Euclidean-matched at δL2 ≤0.10 and qualify at ρKL ≥2; median gaps and ratios summarize qualifying groups only.Because ρKL ≥2 defines the mismatch event, the median KL ratio is descriptive rather than an independent statistical test.Source: B Additional Results for… · p. 14

B.1 Language Models

  • Mistral and Gemma have qualifying groups at every evaluated layer, compared with 27 of 28 Qwen2.5 layers and 22 of 32 LLaMA-3 layers.The authors emphasize recurrence across substantial portions of networks, not qualification at every representation state for every architecture.Source: §B.1 · p. 15, Table 11 · p. 15

B.2 Vision Model

  • In ViT-B/16, qualifying groups occur at every layer, with Median L2 Gap ranging from 1.28% to 3.14% and Median KL Ratio from 2.43× to 3.59×.These results provide a vision counterpart to the language-model findings.Source: §B.2 · p. 17
  • Across modalities, distance-matched counterexamples recur across architectures and depths, but do not establish global decoupling of Euclidean and predictive geometry.The supported claim is that Euclidean magnitude alone is insufficient to determine the behavioral significance of a local representation change.Source: B Additional Results for… · p. 14, §B.2 · p. 17

C Full Layer-wise Predictive Selectivity Results

The layer-wise comparison finds positive mean predictive-selectivity advantage for Fisher-IRG over Cov-IRG at every reported depth across the tested language and vision models. The advantage’s magnitude and strongest depth vary by architecture, rather than showing uniform layer-wise dominance.

Evaluation protocol

  • The evaluation estimates invariant subspaces from discovery variants and compares their effects under semantic-preserving and semantic-changing interventions across five normalized depths.Language uses mean-pooled hidden representations, vision uses CLS-token representations, and the default subspace dimension is k = 16.Source: C Full Layer-wise Predictive… · p. 18

Complete layer-wise comparison

  • Fisher-IRG has positive mean ∆S at every displayed depth for all seven language models and all three vision models, with advantage magnitude varying by architecture.Table 13 reports ∆S as Fisher-IRG minus Cov-IRG, averaged across semantic groups; positive values indicate stronger semantic-versus-nuisance predictive selectivity under Fisher-IRG.Source: C Full Layer-wise Predictive… · pp. 18–19, Table 13 · p. 19

Language models

  • The strongest language-model gains occur at different normalized depths across architectures, while aggregate advantage remains positive at all five reported depths for every model.LLaMA-3-Instruct peaks near 0.50; Mistral-Instruct-v0.3, DeepSeek-MoE-Chat, and Phi-4-Mini near 0.25; Qwen2.5-Instruct at 0.75; Gemma-IT and Falcon3-Instruct near the earliest evaluated depth.Source: C Full Layer-wise Predictive… · p. 18

Vision models

  • ViT-B/16, DINOv2 ViT-B/14, and CLIP ViT-B/16 have positive ∆S across reported depths, with strong early and intermediate advantages persisting through later layers.The authors report consistency across supervised, self-supervised, and image–text contrastive training objectives; absolute values should not be compared directly across modalities.Source: C Full Layer-wise Predictive… · p. 19

D Additional Analysis of Invariant-Subspace Reproducibility

Across matched layer–dimension comparisons, Fisher-IRG is generally more reproducible than Cov-IRG in both language and vision models. Its recovered span is more reproducible than any particular ordered basis used to represent it.

Resampling protocol

  • Independent subspaces are estimated from separate samples of discovery variants and compared using Grassmann distance at matched layer and dimension.The evaluation considers k ∈ {8, 16, 32} and layers 3, 8, and 10; distances are not directly comparable across different k values.Source: D Additional Analysis of… · p. 19

Full language-model and vision-model results

Subspace stability versus individual-vector stability

  • Across seven language models and three evaluated dimensions, principal-cosine agreement ranges from 0.5730 to 0.7519, versus 0.1482 to 0.3856 for individual-vector cosine.Vision models show the same pattern: principal cosine ranges from approximately 0.53 to 0.68, versus approximately 0.16 to 0.37 for individual-vector cosine.Source: D Additional Analysis of… · p. 21, Table 17 · p. 22

Summary and interpretation

  • Fisher-IRG is more reproducible in the large majority of matched comparisons across language and vision architectures, but isolated reversals occur and no universal layer- or dimension-wise trend appears.The positive subspace–vector gap across configurations supports treating the invariant span, rather than a particular eigenvector basis, as the reproducible object.Source: D Additional Analysis of… · pp. 19–22

E Additional Analysis of Fisher–Covariance Geometric Separation

Fisher-IRG and Cov-IRG recover distinctly different local subspaces, with their separation exceeding within-method estimation variability across the evaluated language and vision models. The comparison is relative, not a measure of absolute subspace reproducibility.

Complete cross-model comparison

  • Across all seven language models and three vision models, Fisher–covariance separation exceeds both within-method distances for every evaluated semantic group.Table 18 reports within-method variation, cross-method separation, principal cosine, geometric margin, and the fraction of positive semantic groups.Source: E Additional Analysis of… · p. 23, Table 18 · p. 23

Language-model geometry

  • Language-model cross-method projection distance ranges from 0.993 to 0.998, with mean principal cosine from 0.056 to 0.071; geometric margins are positive across architectures and semantic groups.The geometric margin ranges from 0.484 for LLaMa-3-Instruct to 0.581 for Phi-4-Mini.Source: E Additional Analysis of… · p. 23

Vision-model geometry

  • Vision-model projection distances range from 0.992 to 0.997, with principal cosines from 0.065 to 0.080 and positive geometric margins for all three models.The margins are 0.479 for ViT-B/16, 0.529 for DINOv2 ViT-B/14, and 0.465 for CLIP ViT-B/16.Source: E Additional Analysis of… · p. 23

Why geometric distinctness matters

  • Cov-IRG measures sampled representation displacement, whereas Fisher-IRG measures how local directions affect predictive distributions; the resulting geometries are not merely alternative estimates of one subspace.Low principal cosines and large projection distances indicate that predictive weighting changes which directions are geometrically important.Source: E Additional Analysis of… · p. 24

Relation to the reproducibility analysis

  • This comparison establishes relative separation from within-method variability, not absolute subspace reproducibility, which is evaluated separately using Grassmann distance.Within-method projection distances can themselves be substantial, especially for language models.Source: E Additional Analysis of… · p. 24

F Additional Causal Validation by Representation Intervention

Representation interventions test whether the Fisher-IRG subspace has a functional role in prediction. Across the reported language and vision settings, semantic-changing replacements produce stronger output changes through the Fisher component than semantic-preserving replacements or changes through its complement, although semantic information is not exclusive to that subspace.

Intervention protocol

  • The causal gap compares semantic-changing with meaning-preserving replacement of the Fisher component; the localization gap compares semantic-changing replacement through that component versus its complement.Positive values for both contrasts provide the expected causal-localization pattern.Source: F Additional Causal Validation by… · p. 24

Language-model interventions

  • All 21 language model–group combinations show positive semantic-selectivity and localization gaps, with small predictive changes for meaning-preserving replacement of the Fisher component.The complete language results use three shared semantic groups across seven architectures.Source: F Additional Causal Validation by… · p. 25, Table 19 · p. 25

Vision interventions

  • In ViT-B/16, all nine configurations show positive causal gaps of 0.104334 to 1.005070 and localization gaps of 0.069955 to 0.794108.The pattern persists across easy, medium, and hard semantic changes, but gap magnitude is not monotonic in difficulty because the conditions differ in modification type and composition.Source: F Additional Causal Validation by… · p. 25, Table 20 · p. 26

Interpretation

  • The intervention results support localization, not a perfectly orthogonal semantic–nuisance decomposition: complementary-component replacements can also produce non-negligible output changes.Across seven language models and the evaluated vision setting, every reported intervention satisfies both criteria.Source: F Additional Causal Validation by… · p. 26

G Additional Results for Functional Validation

Fisher-IRG’s semantic-versus-nuisance organization transfers from discovery neighborhoods to held-out separation and retrieval in language and vision. Retrieval results are strongest across metrics and baselines, while vision results show a depth-dependent emergence of positive separation.

Held-out semantic separation

Held-out visual separation

  • At ViT-B/16 layer 10, Fisher-IRG margins are 0.723853, 0.724569, and 0.678376 for easy, medium, and hard changes, respectively.At layer 3, the original representation and Cov-IRG have negative margins at all three difficulty levels, while Fisher-IRG has positive margins.Source: G Additional Results for Functional… · p. 27, Table 22 · p. 27

Held-out semantic retrieval

  • Fisher-IRG has the highest MRR for all seven architectures and the highest pairwise accuracy and Hit@1 for six of seven.For Llama-3-Instruct, the original representation has higher pairwise accuracy (0.978 versus 0.968); for Qwen2.5-Instruct, Cov-IRG has higher Hit@1 (0.825 versus 0.807).Source: Table 23 · p. 28, G Additional Results for Functional… · p. 29
  • Fisher-IRG achieves higher MRR than Random, PCA, cPCA, and Cov-IRG for every evaluated architecture.Its pairwise accuracy also exceeds these baselines except for cPCA on LLaMA-3-Instruct (0.968 versus 0.972).Source: G Additional Results for Functional… · p. 29

Summary

  • Held-out separation generalizes across language and vision, and retrieval shows that the recovered geometry supports semantic matching beyond discovery variants.Subspaces are estimated from discovery variants and evaluated on disjoint held-out examples.Source: G Additional Results for Functional… · pp. 26, 27, 29

H Implementation Details

The implementation freezes pretrained language and vision models and estimates local Fisher geometry using predictive readouts and low-rank operators. It also evaluates subspace reproducibility through independently resampled local neighborhoods.

Text experiments

  • Language experiments freeze seven pretrained models, use inputs truncated to 128 tokens, and generally mean-pool non-padding hidden states.The models run in evaluation mode.Source: H Implementation Details · p. 29

Predictive interventions

  • Interventions add a representation displacement to every token state, propagate the modified sequence through remaining blocks, and evaluate next-token probabilities at the final valid prompt position.For mean-pooled representations, the displacement is added identically to each non-padding token state.Source: H Implementation Details · p. 29

Fisher estimation

  • Fisher-IRG estimates local predictive sensitivity from gradients of next-token log probabilities, retaining and renormalizing the top-q outcomes under the local predictive distribution.The score gradients come from the original predictive distribution; truncation approximates the Fisher expectation over outcomes.Source: H Implementation Details · p. 29

Low-rank operators and solver

  • Fisher and covariance operators use low-rank factors, and the generalized eigenproblem is solved within their joint span with SP regularization of 10^-3 times its mean diagonal value.The main subspace dimension is k = 16; dimension and pooling analyses vary k and compare mean pooling with last-token pooling.Source: H Implementation Details · p. 29

Vision experiments

  • Vision experiments freeze ViT-B/16, DINOv2 ViT-B/14, and CLIP ViT-B/16, extracting CLS-token representations from standard-preprocessed images.Each Visual Genome neighborhood contains an anchor, 10 semantic-preserving variants, and 64 semantic-changing variants.Source: H Implementation Details · p. 30

Subspace stability

  • Subspace reproducibility is assessed by independently bootstrapping SP and SC variants and comparing replicate subspaces with principal-angle and projection measures.Three bootstrap replicates are used per group–layer configuration; higher principal cosine and lower Grassmann or projection distance indicate greater reproducibility.Source: H Implementation Details · p. 30

Computational resources

  • Experiments avoid dense Fisher matrices and use single precision for pooled representations and operators where needed for numerical stability.Model forward passes use half precision; representations are cached on CPU and transferred to the GPU as needed.Source: H Implementation Details · p. 30

Related papers

Loading 2609.36458v1…