PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

Kim-Cuc Nguyen, Ngai-Man Cheung

arXiv:2610.11205v1cs.CVcs.LG

TL;DR

Prior ViT proxy and architecture analyses leave module-level feature compatibility uncertain, while architecture search can be computationally costly. The paper analyzes feature information across ViT modules and develops proxies for generalization prediction; these improve correlation rankings and support higher-accuracy subnet selection with less computation.

  • Problem

    Prior ViT analyses assume whole-network features, skip connections, or later layers carry more information, while NAS also requires costly Supernet training and subnet evaluation.

    Source: §I · p. 1, §IV · pp. 2–3

  • Method

    The paper analyzes ViT feature information across architectural scales, reduces feature collapse, and uses entropy and minimum eigenvalue to characterize features for generalization prediction.

    Source: §I · p. 1

  • Results

    Correlation rankings improve 18%, 25%, and 48% over baselines for Tiny, Small, and Base ViTs, respectively; the proxies also identify higher-accuracy subnets with reduced search time.

    Source: COMBINATIONS SHOW STRONG CORRELATIONS… · p. 4, §VI · p. 4

  • Takeaways & Limitations

    The analysis provides a framework for understanding ViT features, predicting generalization, and guiding efficient architecture design.

    Source: §VII · p. 4

Abstract

from arXiv · show

Representation structure is crucial for understanding Vision Transformer (ViT) architectures and their generalization behavior. However, prior studies neither isolate nor analyze module-level features nor investigate how their interactions contribute to performance estimation. In this work, we conduct the first rigorous analysis of feature information across diverse architectural scales, empirically uncover the relationship between ViT representation and generalization behavior, and leverage these insights to guide efficient ViT design. Our contributions are fivefold: Across diverse architectural scales, 1) We identify feature collapse at initialization, which leads to redundancy, and propose a reduction scheme to mitigate this issue. 2) We quantify feature information using entropy and the minimum eigenvalue, demonstrating that these metrics serve as reliable indicators for generalization prediction. 3) We show that feature in the token space provides a more faithful representation than those in embedding space. 4) We discover an unexpected finding: features produced by linear submodules within ViT layers are critical for the prediction of generalization performance. 5) Our proposed proxy improves the correlation ranking by 18-48% over prior baselines and can effectively identify ViT architectures that achieve higher accuracy at lower or comparable computational cost.

I. INTRODUCTION

The introduction motivates module-level ViT feature analysis as a route to more flexible architecture design, while noting that existing performance-estimation approaches leave important gaps. It presents a feature-analysis framework and proxies intended to support generalization prediction and efficient design.

I. INTRODUCTION

  • Understanding ViT feature information may support more flexible architectures with stronger generalization, despite the reduced interpretability of representations from attention and MLP modules.The paper identifies multi-head self-attention and the MLP as critical ViT components.Source: §I · p. 1
  • Generalization potential is a model’s capacity to generalize before training and can estimate expected performance without optimization, making it valuable for ViT design.NAS seeks accurate architectures under computational constraints, but existing methods require extensive resources.Source: §I · p. 1
  • NAS methods incur high costs by training a Supernet and evaluating many subarchitectures, while training-free proxies estimate performance without exhaustively training candidates.The proxies are presented as a way to mitigate NAS search costs.Source: §I · p. 1
  • Prior work emphasizes overall architecture; CNN-derived proxies, [17]–[19] transfer poorly to ViTs, while ViT proxies –, overlook individual modules and features.ViT feature dimensionality and depth also make information difficult to quantify.Source: §I · p. 1
  • The paper proposes the first cross-scale analysis of ViT feature information, a reduction scheme for feature collapse, and entropy and minimum-eigenvalue measures for generalization prediction.It identifies outputs of linear submodules as critical and ubiquitous for prediction.Source: §I · p. 1

II. RELATED WORK

Related work covers feature collapse, feature selection, and ViT performance proxies. The paper distinguishes its analysis by examining feature collapse at initialization and information across ViT modules.

II. RELATED WORK

  • Prior work defines feature collapse as identical representations for words in the same concept [21]; another study introduces entropy collapse, while this paper observes feature collapse at ViT initialization.The paper proposes a reduction scheme to mitigate the resulting redundancy.Source: §II · pp. 1–2, Figure 1 · p. 2
  • ViT feature-selection methods prune redundant tokens, adapt the number of processed tokens, or select effective patches in final layers for earlier-layer selection.Source: §II · p. 2
  • Prior ViT proxies use synaptic diversity and saliency, search for proxies that generalize across domains and datasets, or target tiny datasets with knowledge distillation [27].Source: §II · p. 2

III. PRELIMINARIES

The preliminaries introduce Shannon entropy for discrete random variables and a Pearson correlation matrix for feature-vector variables.

III. PRELIMINARIES

  • Shannon entropy is defined for a discrete random variable with outcomes x1 through xn and probability function P(X).Source: §III · p. 2
  • For a feature matrix whose columns are feature vectors, the Pearson correlation matrix P ∈ R^D×D is defined to capture pairwise correlations.The matrix definition uses σ_Xi, the standard deviation of feature vector X_i.Source: §III · p. 2

IV. VISION TRANSFORMER FEATURE PROXIES

The section defines ViT features at token and embedding axes, then proposes feature-information proxies and reductions for analyzing their relationship to generalization. It also reports that common assumptions about which features matter do not hold for ViT generalization.

Feature Compatibility

  • Prior assumptions that the whole network, skip connections, or later layers contain more informative features do not hold for ViT generalization.The section frames feature compatibility in terms of how features are connected and combined without conflict.Source: §IV · pp. 2–3

Feature axes

  • Rows of the feature matrix are variables in token space, while columns are variables in embedding space.The input image is divided into patches, projected into embeddings, and combined with positional embeddings and a class token.Source: §IV · p. 3

ViT Feature

  • The analysis extracts representative output features from each ViT module during a forward pass, including attention, MLP, skip-connection, and normalization outputs.It also considers feature combinations, such as M1 + M2.Source: §IV · p. 3

Feature dimension reduction

  • The entropy proxy selects the u lowest-entropy row or column vectors to form a reduced representation, and the resulting information measure is scaled to compensate for dimension reduction.The scaling is intended to reflect the contribution of the entire feature space.Source: §IV · p. 3

Minimum Eigenvalue Proxy

  • The minimum-eigenvalue proxy uses a Pearson correlation matrix and rowwise sums of absolute correlations; its minimum eigenvalue is associated with feature diversity and independence.Higher diversity is reported to correlate with better classification accuracy, stronger generalization, and more stable training in ViTs.Source: §IV · p. 3

V. DISCOVERING THE IMPACT OF FEATURE INFORMATION ON VIT GENERALIZATION PREDICTORS

The experiments use 1,000 small ViT architectures with ImageNet-1K test accuracies to evaluate feature proxies. Proxy values are compared with test accuracy using rank-correlation metrics.

Procedure

  • The testbed samples 1,000 AutoFormer-Tiny architectures with 5–7 million parameters and collects their ImageNet-1K test accuracies using pretrained weights.Subnet test accuracy from supernet-inherited weights is reported as comparable to accuracy from independently trained subnets.Source: §V · p. 3
  • The unified framework computes Entropy and Minimum Eigenvalue proxies from ViT features.The section heading identifies the analysis as investigating feature collapse at initialization and feature information for generalization prediction.Source: §V · p. 3

Metrics

  • Proxy effectiveness is evaluated by computing Spearman and Kendall correlations between proxy values and test accuracy.Source: §V · p. 3

B. Comparison with State-of-the-Art Methods

The supplied passages describe token-entropy proxy variants, a feature-Q singular-value spectrum, and the scope of a proxy–accuracy correlation table.

B. Comparison with State-of-the-Art Methods

  • The reported token-entropy proxies with reduced dimension u = 1 include M1 + M2 and MHSAout + M1 + M2.Source: B. Comparison with State-of-the-Art… · p. 3
  • Most singular values of feature Q are near zero in token and embedding spaces, indicating low-rank structure and feature collapse.The figure plots variable indices against mean and standard deviation of logarithmic singular values across layers.Source: Figure 2 · p. 4
  • Table I covers Spearman ρ and Kendall τ correlations with test accuracy for 1,000 ViT architectures across 26 feature types, both spaces, and u = 1.Source: Table I · p. 4

PROXIES IN TABLE II. THE ENTROPY AND MINIMUM EIGENVALUE

Entropy and minimum-eigenvalue proxies indicate ViT generalization, with linear-submodule features and token-space features showing stronger relationships with test accuracy. Across three parameter ranges, the proposed proxies improve correlation rankings over baselines.

PROXIES IN TABLE II. THE ENTROPY AND MINIMUM EIGENVALUE

  • Entropy and minimum-eigenvalue proxies quantify feature information as indicators of ViT generalization, with linear-submodule outputs highlighted as important features.The passage names Q, K, V, V′, MHSAout, M1, and M2 among the linear-submodule outputs.Source: PROXIES IN TABLE II. THE ENTROPY AND… · p. 4

COMBINATIONS SHOW STRONG CORRELATIONS WITH TEST

VI. EFFICIENT VIT ARCHITECTURE DESIGN

The proposed proxies identify higher-accuracy ViT subnets while reducing search computation time. They also improve correlation rankings and are reported as indicators of ViT generalization.

VI. EFFICIENT VIT ARCHITECTURE DESIGN

  • Accuracy improves by 0.11%–0.17% over previous proxies when searching 1,000 ViT architectures in the 5–7M parameter range.The results are on ImageNet.Source: §VI · p. 4, Table III · p. 4
  • The proxies significantly improve correlation rankings and are described as strong indicators of ViT generalization.Source: §VI · p. 4
  • Search computation time falls by 87.5%–98.6% across 1,000 ViT architectures, requiring 0.06 hours versus 0.48–4.3 hours for previous proxies.Source: §VI · p. 4

VII. CONCLUSION

The paper analyzes how ViT feature representations relate to generalization and reports proxy improvements across parameter ranges. It also presents the approach as a way to support more efficient architecture design.

VII. CONCLUSION

  • The analysis relates ViT feature representations to generalization across feature information, axes, dimensions, and locations.Source: §VII · p. 4
  • The proposed feature proxies improve correlation ranking by 18–48% over baselines across parameter ranges.Source: §VII · p. 4
  • The method achieves higher accuracy across architectures while reducing training, search, and proxy computation costs.Future work applies the findings to design compact models for different tasks –.Source: §VII · p. 4
Loading 2610.11205v1…