PaperlayerView exact revision on arXiv

Source-linked AI summary

LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations

Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla, Will Walden, Andrew Yates, Reno Kriz, Sarah M. Erfani, James Bailey, Vishal M. Patel, Sanjeev Khudanpur

arXiv:2609.28086v2cs.LGcs.AIcs.CV

TL;DR

Downstream benchmarks provide limited insight into how learned representations develop across layers, and intermediate-layer evaluation can be costly and task-dependent. LAYERSCOPE compares layerwise representations with label-free geometric diagnostics across video and multimodal tasks. Intermediate layers often improve classification and clustering, while useful layer choices and geometric relationships vary by task and model.

  • Problem

    Benchmarks measure classification, clustering, and retrieval success but treat internal representations as a black box, while evaluating intermediate layers can be costly and task-dependent.

    Source: §1 · p. 1

  • Method

    LAYERSCOPE evaluates layerwise representations across video and multimodal tasks using complementary geometric diagnostics and examines their associations with performance and layer-selection regret.

    Source: §1 · pp. 2–3

  • Results

    Intermediate layers often improve classification and clustering, retrieval often performs well at final or near-final text–video pairs, and RankMe has the lowest mean selection regret among tested measures for classification and clustering but is not a universal selector.

    Source: §5 · p. 11

  • Takeaways & Limitations

    Layerwise geometry provides a label-free way to characterize representation changes, while correspondence-aware diagnostics reveal text–video structure that marginal distribution measures can miss.

    Source: §1 · p. 3, §5 · p. 11

    Geometric metrics do not replace task-specific layer evaluation, and the preferred representation depends on the model and task.

    Source: §5 · p. 11

Abstract

from arXiv · show

We propose LAYERSCOPE, a label-free, layerwise framework that aims to characterize a model's learned representations in video and multimodal settings. Evaluating downstream performance using representations from final or intermediate layers typically requires large amounts of labeled data, repeated task-specific evaluations, and substantial computation. To address these limitations, LAYERSCOPE uses local, global, distributional, and correspondence-based geometric metrics to compare layerwise representation structure within and across models without requiring task-specific labels. We evaluate seven architecturally diverse models across video and multimodal classification, clustering, and text-to-video retrieval tasks from MVEB/MVEB+. We find that intermediate-layer representations can outperform final-layer and model-default outputs. We also find that no single geometric metric consistently predicts downstream performance, but note that distinct layerwise geometric signatures emerge across model families. LID shows task-dependent relationships with performance, while RankMe provides the strongest measure for classification and clustering, but is not a universal layer selector. We also find that pairing-aware metrics explain retrieval better than distributional distances alone. LAYERSCOPE therefore offers a framework for comparing representations across models and layers, enabling a more systematic evaluation in video and multimodal settings.

1 Introduction

Downstream benchmarks reveal model success but offer limited insight into how representations develop across layers, while intermediate-layer evaluation can be costly and task-dependent. LAYERSCOPE uses label-free geometric diagnostics to characterize representations and examine when they relate to performance or help identify useful layers.

Motivation

  • Geometric diagnostics characterize local, global, distributional, and correspondence-sensitive properties that downstream scores do not expose.The metrics do not require downstream labels or task scores, although their relationship with performance may vary across tasks.Source: §1 · p. 1

Why layerwise representation geometry may help

  • Useful semantic information can emerge before the final layer, motivating evaluation of intermediate representations rather than only model outputs [10].Global measures describe dataset-level dimensionality or spectral utilization, while local measures such as LID characterize neighborhood-level manifold complexity.Source: §1 · p. 2

Contributions

  • LAYERSCOPE examines how representation geometry evolves with depth, whether geometric measures associate with downstream performance, and whether they identify high-performing representations.It studies video and multimodal models using downstream tasks and complementary local, global, and cross-modal diagnostics.Source: §1 · pp. 2–3

Evaluation questions

  • The framework investigates classification, clustering, zero-shot classification, and text-to-video retrieval, including when intermediate representations outperform final-layer or usual model outputs.It also compares layerwise geometric profiles across predictive, contrastive, and autoregressive model families.Source: §1 · p. 3

Contributions

  • LAYERSCOPE distinguishes marginal distributional proximity from pair-sensitive alignment to examine when geometric measures track retrieval behavior.Its stated contribution is to identify when correspondence-aware diagnostics reveal text–video structure that marginal distribution measures cannot capture.Source: §1 · p. 3

2 Related Work

Existing benchmarks measure downstream capability but generally leave within-network representation changes unexamined; prior work motivates complementary geometric measures and layerwise analysis. LAYERSCOPE extends that analysis to performance, layer selection, and cross-modal correspondence.

Embedding and multimodal evaluation benchmarks

  • MVEB and related suites benchmark diverse downstream tasks but generally use default outputs, leaving layer depth and high-performing internal representations open to investigation.LAYERSCOPE complements these benchmarks by testing label-free, layerwise geometric measures against downstream behavior.Source: §2 · p. 3

Geometry of learned representations

  • Geometric analysis is motivated by evidence that learned representations occupy lower-dimensional structures and can change across network layers.Local measurements capture neighborhood structure, while global measurements use the full embedding space.Source: §2 · pp. 3–4
  • Rank-based global measures correlate with downstream performance in S3Ms but do not reliably select the best task-specific layer; LID instead measures local neighborhood dimensionality.Neither LID nor effective rank is assumed to be a universal performance indicator.Source: §2 · p. 4

Intermediate representations and layer selection

  • Intermediate representations can retain task-relevant information weakened or specialized in final layers, while prior studies have examined similarity and task-specific layer selection.LAYERSCOPE instead evaluates label-free geometric diagnostics across classification, clustering, and complete text-video layer configurations.Source: §2 · p. 4

Cross-modal alignment and distributional proximity

  • Successful cross-modal matching does not require identical text and video distributions, and reducing modality gaps can help retrieval in some settings but harm it in others.Prior work also reports effects that differ between group-level and instance-level retrieval.Source: §2 · p. 4
  • LAYERSCOPE distinguishes marginal distributional proximity, paired cross-modal correspondence, and semantic alignment, using SW2 for the first and Procrustes error for paired correspondence.SW2 compares marginal distributions; Procrustes error tests the structure of corresponding text-video examples.Source: §2 · pp. 4–5

3 Methods

LAYERSCOPE evaluates layerwise representations across video and multimodal models and tasks, using geometric diagnostics to compare representation structure and assess layer selection. Its cross-modal analysis separates marginal distributional proximity from paired correspondence.

3 Methods

  • Across seven video and multimodal models, LAYERSCOPE caches layer representations, measures geometry, evaluates downstream performance, and uses regret to quantify geometry-based layer-selection cost.Model profiles are descriptive comparisons because the models differ in architecture, data, objectives, and default outputs.Source: §3 · p. 5, §3.2 · p. 5

3.1 Problem setting

  • For classification and clustering, regret compares geometry-selected layers with the oracle best layer; retrieval applies the comparison to selected text-video layer pairs and reports pair-selection regret separately from default and matched-depth baselines.Retrieval performance is mean nDCG@10, and minimum-SW2 selection includes all exact minimum-distance pairs.Source: §3 · p. 5, §3.1 · p. 5

3.2 Models, representations, and tasks

  • The study freezes seven diverse models and trains no layer-specific or cross-modal alignment adapters; it distinguishes final-block states from model-default outputs after any normalization, pooling, or projection.Classification and clustering use mean hidden states; retrieval representations are transformed into each model's shared text-video space.Source: §3.2 · pp. 5–6
  • Evaluation covers Breakfast and UCF101 classification, UCF101 clustering, HMDB51 replication, UCF101 zero-shot classification, and T2VR on TUNA-Bench and VATEX.Classification uses eight-shot linear probes and accuracy; clustering uses MiniBatchKMeans and V-measure, while retrieval uses cosine similarity and mean nDCG@10.Source: §3.2 · p. 6

3.3 Geometric measurements

  • LID summarizes local neighborhood complexity, while RankMe measures the native singular spectrum and covariance effective rank measures the standardized covariance spectrum.The LID layer value is the harmonic mean of finite, strictly positive pointwise estimates.Source: §3.3 · p. 6

3.4 Cross-modal geometry

  • Cross-modal analysis evaluates all text-video layer combinations and also matches layers at nearest normalized depths when the streams have different numbers of blocks.Normalized depth is each layer index divided by its stream's block count.Source: §3.4 · p. 6
  • SW2 compares marginal text-video distributions, whereas held-out orthogonal Procrustes error measures paired structure after fitting a rotation on training pairs.The analysis also permutes text-video correspondences while leaving both marginal distributions unchanged.Source: §3.4 · pp. 6–7

4 Results

Layerwise geometry varies across models, and no geometric selector or distributional distance alone reliably identifies the best task representation. Intermediate layers often improve single-modality performance, while retrieval generally favors late text–video pairs.

4.1 Layerwise geometry differs across models

  • UCF101 geometry profiles differ by model: V-JEPA 2 and X-CLIP peak early in GRIDS-LID but rise later in RankMe, while both PE-AV models increase in GRIDS-LID with depth.These are descriptive patterns, not established model-family effects, because the models also differ in architecture, data, scale, preprocessing, and output processing.Source: §4.1 · p. 7, Figure 2 · p. 7

4.2 No geometric measure identifies the best layer across all tasks

  • Across 21 comparisons, RankMe had the lowest mean layer-selection regret (1.43%), versus the final-layer baseline (2.49%) and covariance effective rank (3.99%).No geometric measure reliably chose the best layer across every comparison; GRIDS-LID’s favorable direction varied by dataset and task.Source: §4.2 · p. 8

4.3 Intermediate layers often improve single-modality downstream performance

  • Six of seven models had an intermediate layer that outperformed the final layer on at least one task; X-CLIP and LCO defaults beat every evaluated layer.Gemma 4 layer 13 improved UCF101 clustering V-measure by ∆= +0.1683 over the last layer.Source: §4.3 · pp. 8–9, Figure 4 · p. 9

4.4 Distributional distance tracks retrieval but does not determine it

  • SW2 was negatively associated with nDCG@10 in 8 of 10 evaluations, but minimum-SW2 pairs were poor selectors; Procrustes favored true pairings in 31 of 32 comparisons.Retrieval favored true pairings over randomized ones in 27 of 32 comparisons at p ≤0.05.Source: Figure 5 · p. 10, §4.4 · p. 10

4.5 Cross-modal retrieval peaks late, but the best text and video depths may differ

  • Retrieval usually favored final or near-final text–video pairs, often at matched depths; X-CLIP’s final-text/penultimate-video pair gained +0.0685 nDCG@10 on TUNA.The corresponding X-CLIP gain was +0.0198 on VATEX; LCO’s best VATEX pair used text layer 34 and video layer 35 (∆= +0.0062). Models with similar retrieval scores could have different cross-modal geometries.Source: §4.5 · p. 11

5 Discussion and Conclusion

LAYERSCOPE characterizes how representations and their geometry change across layers, while treating geometric measures as complements to task evaluation. Its results show that layer choice remains dependent on both model and task.

5 Discussion and Conclusion

  • GRIDS-LID relationships with downstream performance were task-dependent, while intermediate layers often improved single-modality tasks and retrieval often favored final or near-final pairs.The preferred representation depends on the model and task.Source: §5 · p. 11
  • Across the evaluated MVEB classification and clustering tasks, RankMe had the lowest mean regret among tested geometric selectors, but it was not a universal layer selector.Geometry characterizes representation changes; it does not replace task-specific layer evaluation.Source: §5 · p. 11
  • SW2 was associated with higher retrieval performance at lower distances but did not generally identify the best text–video pair; Procrustes identified paired structure in 31 of 32 comparisons.The paper calls for controlled comparisons to isolate how architecture, training objective, data scale, and output mapping shape layerwise profiles.Source: §5 · p. 11

A Experimental Details and Reproducibility

The experiments evaluate frozen models across specified layer ranges and downstream tasks, using distinct representation extraction and scoring procedures for single-modal and cross-modal evaluations.

A Experimental Details and Reproducibility

  • The experimental model roster and evaluated layer ranges are summarized in Tables 1 and 2, with retrieval eligibility reported alongside layer coverage.The seven models remain frozen, and model-default outputs are evaluated separately from final-layer representations because they may add normalization, pooling, or projection.Source: Table 1 · p. 16, Table 2 · p. 16, A Experimental Details and… · p. 16
  • For classification and clustering, each video layer uses averaged eligible hidden states; retrieval and zero-shot evaluation test eligible text–video layer pairs through the shared embedding interface.The paired vectors are compared using cosine similarity, with no layer-specific projections or cross-modal alignment adapters trained.Source: A Experimental Details and… · p. 16
  • Table 3 summarizes downstream evaluations: classification and clustering use ten evaluator seeds, while retrieval and zero-shot classification are deterministic for fixed representations.Geometric measures use training representations without class labels; retrieval geometry uses query and video representations without relevance judgments.Source: Table 3 · p. 16, A Experimental Details and… · p. 16

B Geometric measure definitions and implementation details

LAYERSCOPE computes geometric measures layer by layer, with preprocessing tailored to each measure and representation setting. It includes local and spectral geometry, distributional distances, correspondence-sensitive alignment measures, and permutation-based comparisons.

B Geometric measure definitions and implementation details

  • Each geometric measure is computed at every evaluated layer using the full representation pool, with preprocessing determined by the measure and task.Classification and clustering use pooled layer representations; retrieval uses the model-specific shared-space readout.Source: B Geometric measure definitions and… · p. 17

B.1 GRIDS-LID, depth roughness and neighborhood-size sensitivity

  • GRIDS-LID standardizes feature coordinates before neighbor search, aggregates finite positive pointwise estimates by harmonic mean, and evaluates both minimum and maximum layers.The kth neighbor sets the radius, while the estimator uses the first k −2 neighbor distances within it; sensitivity analyses repeat the calculation at k = 50 and 150.Source: §B.1 · p. 17, Figure 6 · p. 18
  • LID depth roughness measures local variation across neighboring layers, with the primary window spanning approximately 20% of model depth.Windows spanning 10% and 30% are used for sensitivity analysis; performance roughness is analyzed only for models with at least ten evaluated layers.Source: §B.1 · p. 17

B.2 Spectral measures

  • RankMe uses singular values of the native representation matrix without additional centering or coordinate standardization, whereas covariance effective rank uses standardized, centered representations.The analysis uses absolute values; display normalization is separate and does not remove RankMe’s feature scaling.Source: §B.2 · p. 18

B.3 Sliced Wasserstein distance and relative-depth alignment

  • Sliced 2-Wasserstein distance compares empirical text and video representation distributions without using item correspondence, so permuting pairings leaves it unchanged.The analysis uses 1,024 fixed unit Gaussian directions per model–task comparison and pairs layers by closest relative network depth when comparing corresponding depths.Source: §B.3 · p. 19

B.4 Correspondence-sensitive measures

  • Correspondence-sensitive measures use paired, row-normalized text and video representations, so shuffling pairings changes the measures.Held-out orthogonal Procrustes error fits an unscaled orthogonal map on an 80% split after centering by fit-set modality means, then measures root-mean-square residual on held-out pairs; lower error indicates better generalization to unseen pairs.Source: §B.4 · p. 19

B.5 Correspondence permutations

  • The correspondence permutation analysis shuffles text–video pairings while preserving both marginal representation sets, then compares observed measures with 1,000 shuffled values.The one-sided Monte Carlo value counts permutations at least as favorable as the observed measure; directional separation uses the permutation mean and standard deviation, with the sign set by whether larger or smaller values are favorable.Source: §B.5 · pp. 19–20

C.1 Sliced Wasserstein distance and retrieval

Across evaluated text–video layer pairs, lower SW2 often accompanies better retrieval, but the minimum-distance pair does not select the retrieval optimum.

C.1 Sliced Wasserstein distance and retrieval

  • The analysis evaluates all text–video layer pairs on TUNA and VATEX, with model-default outputs evaluated separately.It also compares pairs along a path matching text and video layers by closest relative network depth.Source: §C.1 · p. 20
  • The minimum-SW2 pair is never the best retrieval pair; its mean selection regret is 0.2994 versus 0.0097 for the model-default pair.SW2 is negatively associated with nDCG@10 in eight of ten model–task comparisons; mean Spearman correlation is −0.514, falling to −0.279 after accounting for relative depth.Source: §C.1 · p. 20
  • Table 4 reports the best layer pair, the best matched-depth score, and the model-default score relative to the retrieval oracle.The table caption defines the oracle as the pair with the highest nDCG@10 and identifies matched-depth selection as restricted to the normalized-depth path.Source: §C.1 · p. 20

C.2 Cross-modal correspondence across depth

Cross-modal analyses compare layerwise text–video alignment and test whether observed pairings outperform shuffled correspondence. Retrieval and zero-shot classification are also evaluated across layer pairs.

C.2 Cross-modal correspondence across depth

  • The cross-modal comparison aligns text and video layers by relative depth and includes all five shared-space models and every evaluated text layer.Figure 8 presents linear CKA and held-out orthogonal Procrustes error on TUNA and VATEX; higher CKA and lower Procrustes error are favorable.Source: §C.2 · pp. 20–21, Figure 8 · p. 21
  • Held-out Procrustes error beats all 1,000 permutation values in 31 of 32 comparisons, while nDCG@10 does so in 26 and reaches pMC ≤0.05 in 27.Linear CKA, matched-minus-unmatched cosine similarity, and neighbor overlap reach the minimum Monte Carlo value in all 32 comparisons; at final evaluated depth, paired measures and retrieval all favor observed correspondence.Source: §C.2 · pp. 21–22, Figure 9 · p. 22

C.3 UCF101 zero-shot classification

  • For the four-prompt ensemble, the best UCF101 zero-shot layer pair beats the model-default pair in four of five models, with gains up to 0.0082.X-CLIP’s best pair scores 0.0005 lower than its model-default pair; prompts were evaluated individually and as a fixed ensemble without selection based on evaluation accuracy.Source: §C.3 · p. 22, Table 5 · p. 22

D Layerwise classification and clustering performance

  • The layerwise classification and clustering analysis evaluates every video layer directly and evaluates the model-default output separately.The reported tasks include Breakfast classification and UCF101 classification and clustering.Source: D Layerwise classification and… · p. 22

D.1 Primary classification and clustering results

The section defines how best-layer scores are compared with final-layer and model-default scores, reports an HMDB51 replication, and examines within-task geometric associations with performance.

D.1 Primary classification and clustering results

  • The best layer is selected by ten-seed mean score, and its score differences from the final layer and model-default output are computed from unrounded scores.Positive differences favor the best layer; Table 6 covers Breakfast and UCF101 classification and clustering, with accuracy and V-measure respectively.Source: §D.1 · p. 22, Table 6 · p. 23

D.2 HMDB51 classification replication

  • An intermediate layer outperforms the final layer in five of seven HMDB51 models and the model-default output in four, while three models perform best with model-default outputs.The replication uses the same eight-shot layerwise evaluation protocol.Source: §D.2 · p. 23, Table 7 · p. 23

E Geometry and downstream performance

  • The primary geometry analysis covers 21 model–task comparisons across seven models and three tasks, correlating layerwise geometric measures with ten-seed mean downstream scores.Correlations are not reported for four-layer PE-AV models, and degenerate ranked variables yield undefined associations.Source: E Geometry and downstream performance · p. 23, §E.1 · p. 24, Figure 10 · p. 24

E.1 Geometry–performance associations

  • GRIDS-LID associations vary by task for LCO, while RankMe and standardized covariance effective rank are more often positive but remain inconsistent across models.For LCO, GRIDS-LID correlations are +0.94 on Breakfast, −0.37 on UCF101 classification, and −0.66 on UCF101 clustering; both spectral measures are negative for Gemma 4 on the UCF101 tasks.Source: §E.1 · p. 24

E.2 Layer-choice regret

Across the 21 classification and clustering comparisons, maximum RankMe has lower mean layer-choice regret than the last-layer baseline, while GRIDS-LID’s performance varies by task. The regret analysis compares geometry-selected layers with the best-performing layer determined separately for each evaluator seed.

E.2 Layer-choice regret

  • For each model–task comparison, geometry-based layer choices are fixed across evaluator seeds, while the best-performing layer is determined separately for each seed.Regret is computed over seeds 0–9 and averaged within each comparison.Source: §E.2 · p. 24
  • Across 21 primary classification and clustering comparisons, Maximum RankMe has the lowest mean regret, 0.0143 versus 0.0249 for the final-layer baseline, and improves on it in eight comparisons.Regret uses the native 0–1 task-score scale; lower values are better.Source: §E.2 · p. 24, Table 8 · p. 25
  • GRIDS-LID regret depends strongly on task: maximum GRIDS-LID incurs large UCF101 regret, particularly for V-JEPA 2 and X-CLIP, while minimum GRIDS-LID shows the opposite pattern in several cases.Source: Figure 11 · p. 25, §E.2 · p. 25
  • Maximum standardized covariance effective rank improves on the last-layer baseline in ten comparisons but has higher mean regret, 0.0399.Source: §E.2 · p. 25
  • Minimum GRIDS-LID has median regret 0.0063 but mean regret 0.0447, while maximum GRIDS-LID has the worst mean regret, 0.0997, and maximum regret, 0.4734.The low median alongside the higher mean for minimum GRIDS-LID reflects several large errors.Source: §E.2 · p. 25

Related papers

Loading 2609.28086v2…